<%BANNER%>

Assessing the Effectiveness of Supplemental Educational Services in Urban Florida School Districts

Permanent Link: http://ufdc.ufl.edu/UFE0021969/00001

Material Information

Title: Assessing the Effectiveness of Supplemental Educational Services in Urban Florida School Districts
Physical Description: 1 online resource (118 p.)
Language: english
Publisher: University of Florida
Place of Publication: Gainesville, Fla.
Publication Date: 2008

Subjects

Subjects / Keywords: academic, educational, esea, evaluation, nclb, school, ses, supplemental, title, tutoring, urban
Educational Administration and Policy -- Dissertations, Academic -- UF
Genre: Educational Leadership thesis, Ph.D.
bibliography   ( marcgt )
theses   ( marcgt )
government publication (state, provincial, terriorial, dependent)   ( marcgt )
born-digital   ( sobekcm )
Electronic Thesis or Dissertation

Notes

Abstract: Nationally, millions of federal dollars are given to private tutoring companies to provide tutoring to low income students who participate in the Supplemental Educational Services (SES) program in Title I schools. There is little or no research to prove the effectiveness of this program in improving the academic growth of these students. The purpose of this study was to evaluate the effectiveness of the SES program in six large urban Florida school districts by utilizing existing scores from the Florida Comprehensive Assessment Test (FCAT). A quasi-experimental design was used to analyze the FCAT developmental scale scores wherein the scores were divided into two groups of Title I students. Group One was composed of students who received SES in the 2006-07 school year and completed the FCAT in math and reading in both 2006 and 2007. Group Two was a control group composed of students who were enrolled in Title I schools, completed the FCAT in reading and math in 2006 and 2007, and were eligible for SES but did not participate in the program. The developmental scale scores from the 2006 and 2007 FCAT administration were used as pre- and post-tests. A one-way analysis of variance design was used to analyze the FCAT data. The analysis was completed using the Statistical Package for Social Sciences with a .05 level of significance. Results of this study found increased developmental scale scores for Title I students who received tutoring in math, but there were no statistically significant findings on increased academic achievement for students who received tutoring in reading only or in reading and math. This study also assessed the effectiveness of individual SES providers by analyzing the three largest SES providers in each of the six districts that participated in this study. Of those 18 providers, only 3 provided results that confirmed increased academic achievement.
General Note: In the series University of Florida Digital Collections.
General Note: Includes vita.
Bibliography: Includes bibliographical references.
Source of Description: Description based on online resource; title from PDF title page.
Source of Description: This bibliographic record is available under the Creative Commons CC0 public domain dedication. The University of Florida Libraries, as creator of this bibliographic record, has waived all rights to it worldwide under copyright law, including all related and neighboring rights, to the extent allowed by law.
Thesis: Thesis (Ph.D.)--University of Florida, 2008.
Local: Adviser: Quinn, David.

Record Information

Source Institution: UFRGP
Rights Management: Applicable rights reserved.
Classification: lcc - LD1780 2008
System ID: UFE0021969:00001

Permanent Link: http://ufdc.ufl.edu/UFE0021969/00001

Material Information

Title: Assessing the Effectiveness of Supplemental Educational Services in Urban Florida School Districts
Physical Description: 1 online resource (118 p.)
Language: english
Publisher: University of Florida
Place of Publication: Gainesville, Fla.
Publication Date: 2008

Subjects

Subjects / Keywords: academic, educational, esea, evaluation, nclb, school, ses, supplemental, title, tutoring, urban
Educational Administration and Policy -- Dissertations, Academic -- UF
Genre: Educational Leadership thesis, Ph.D.
bibliography   ( marcgt )
theses   ( marcgt )
government publication (state, provincial, terriorial, dependent)   ( marcgt )
born-digital   ( sobekcm )
Electronic Thesis or Dissertation

Notes

Abstract: Nationally, millions of federal dollars are given to private tutoring companies to provide tutoring to low income students who participate in the Supplemental Educational Services (SES) program in Title I schools. There is little or no research to prove the effectiveness of this program in improving the academic growth of these students. The purpose of this study was to evaluate the effectiveness of the SES program in six large urban Florida school districts by utilizing existing scores from the Florida Comprehensive Assessment Test (FCAT). A quasi-experimental design was used to analyze the FCAT developmental scale scores wherein the scores were divided into two groups of Title I students. Group One was composed of students who received SES in the 2006-07 school year and completed the FCAT in math and reading in both 2006 and 2007. Group Two was a control group composed of students who were enrolled in Title I schools, completed the FCAT in reading and math in 2006 and 2007, and were eligible for SES but did not participate in the program. The developmental scale scores from the 2006 and 2007 FCAT administration were used as pre- and post-tests. A one-way analysis of variance design was used to analyze the FCAT data. The analysis was completed using the Statistical Package for Social Sciences with a .05 level of significance. Results of this study found increased developmental scale scores for Title I students who received tutoring in math, but there were no statistically significant findings on increased academic achievement for students who received tutoring in reading only or in reading and math. This study also assessed the effectiveness of individual SES providers by analyzing the three largest SES providers in each of the six districts that participated in this study. Of those 18 providers, only 3 provided results that confirmed increased academic achievement.
General Note: In the series University of Florida Digital Collections.
General Note: Includes vita.
Bibliography: Includes bibliographical references.
Source of Description: Description based on online resource; title from PDF title page.
Source of Description: This bibliographic record is available under the Creative Commons CC0 public domain dedication. The University of Florida Libraries, as creator of this bibliographic record, has waived all rights to it worldwide under copyright law, including all related and neighboring rights, to the extent allowed by law.
Thesis: Thesis (Ph.D.)--University of Florida, 2008.
Local: Adviser: Quinn, David.

Record Information

Source Institution: UFRGP
Rights Management: Applicable rights reserved.
Classification: lcc - LD1780 2008
System ID: UFE0021969:00001


This item has the following downloads:


Full Text





ASSESSING THE EFFECTIVENESS OF SUPPLEMENTAL EDUCATIONAL SERVICES
IN URBAN FLORIDA SCHOOL DISTRICTS





















By

MYRNA L. ALLEN


A DISSERTATION PRESENTED TO THE GRADUATE SCHOOL
OF THE UNIVERSITY OF FLORIDA IN PARTIAL FULFILLMENT
OF THE REQUIREMENTS FOR THE DEGREE OF
DOCTOR OF PHILOSOPHY

UNIVERSITY OF FLORIDA

2008



































2008 Myma L. Allen

































To my children Terry, Christine, and Roger whose love, motivation, and encouragement enabled
me to make this milestone possible and to my grandson, Braeden, whose birth has been a
blessing to our family









ACKNOWLEDGMENTS

I would like to acknowledge and thank my children, Terry, Christine, and Roger for their

love, patience, and support while I pursued this degree. I could never have accomplished this

milestone without their love and support. By far the most important accomplishment in my life

has been raising my children to be loving, responsible, caring adults.

I would like to also express my gratitude to all of the people who helped me achieve my

goal of obtaining a Ph.D. This group includes Dr. Jim Doud, who made this program possible by

establishing the Jacksonville Cohort, as well as Dr. David Quinn for the support and patience he

provided as chair of my committee. I also give my deepest appreciation to Dr. Jean Crockett for

her positive attitude and editing skills, as well as Dr. Kathy Gratto for the words of

encouragement she shared with me on more than one occasion. I would also like to acknowledge

the help and dedication of Angela Rowe.

I thank the Title I Office for their motivation and enthusiasm as I pursued this goal,

especially Latosha Norman, Brenda Smith, Jennifer Militello, and Libby Marsh for their

encouragement and support that allowed me to leave the office in Jacksonville to attend classes

in Gainesville. My deepest gratitude to Frank Herrington whose support and confidence in me

never wavered, Lynette Weber whose friendship and editing skills were much appreciated, Dr.

Kathy Divine for her expert advice on data analysis, Pat Cascone for her editing advice, and to

my Associate Pastor, Bob Gauger, whose encouragement was invaluable. I am grateful to Tim

Allen for his steadfast confidence in my abilities and his words of encouragement for the last

three and a half years.

My final acknowledgement is extended to my friends in the Jacksonville/St. Augustine

cohort. The support and friendship of the cohort was instrumental in my success. I will always

have fond memories of the Fridays and Saturdays we spent in class. Special thanks to Lissa









Dunn whose friendship and gentle nature was much appreciated and to Ed Pratt-Dannals whose

research and perseverance brought the University of Florida to Jacksonville and made the cohort

possible.

I thank God for his blessings and guidance while pursuing this degree, as well as

throughout my life. He truly blessed me in making this degree possible.









TABLE OF CONTENTS

page

A CK N O W LED G M EN T S ................................................................. ........... ............. .....

L IS T O F T A B L E S ................................................................................. 8

LIST O F TER M S ...... ...................................................................................... .......................

A B S T R A C T ........................................... ................................................................. 1 1

CHAPTER

1 IN TR O D U C T IO N ................................................................................ 13

Supplem ental Educational Services (SES) ................................................................... 18
State ent of the Problem ................................................................................................22
P u rp o se o f th e S tu dy ...................................................................................................2 4
S ig n ifican ce of th e Stu dy ................................................................. ...............................2 5

2 L IT E R A TU R E R E V IE W ..............................................................................................27

History of Tutoring ....................................................................... ........ 27
Quality Indicators of Effective Tutoring .............................................................. ... 29
The Effectiveness of Other After-school Tutoring Programs ..............................................31
Current Status of Evaluating SES........................... ......... 34
Lack of Funding ......................... .................... ............36
The Increasing Number of SES Providers .................................. ........ ................... 37
Implementation in Rural Districts and Districts with Low Enrollment ...............................38
H ighly Qualified Teachers................................................... 39
Increasing Enrollm ent ............... .............................. ......................... ... .. 39
SES Coordination with Classroom Curriculum...... .................. ...............40
SE S Student-T teacher R atio .............................................................................. ....................42
E existing R research on SE S ........................................................................ .. ............42
Minneapolis Public Schools' SES Effectiveness......... ......................................... 44
Chicago Public Schools' SES Effectiveness .........................................................................45
Los Angeles Unified School District's SES Effectiveness ............... ...............47
Tennessee's SES Effectiveness Study ........................ ..... ..... ............. ........ .. ...... .... 47
Large Urban Districts' SES Effectiveness............................... ........48
SE S Evaluation Progress .......................................................... ......49
C challenges in E evaluating SE S .....................................................................................51
Recommendations to Strengthen and Evaluate SES .......................................................53
Summary ............................................... ..................55

3 STU D Y O V E R V IE W .................................................................................................57




6









Purpose of the Study ............... ...................................................... 58
Research Hypotheses .............. ................. ............ ........................ ... 59
O overview of the M ethod ............................................................................... .. ............. 60
2006-07 Participating D district Statistics ..................................................................... ...... 60
P ro c e d u re s .............................................................................................6 4
S u m m ary ................... ...................6...................5..........

4 R E SE A R C H FIN D IN G S ......................................................................... ........................67

Research Questions..................... ......... ... .. .... ....... .... .......... 68
Research Hypotheses .................. .................. ............... ............... .... ....... 68
Analysis and Quantitative Results .......................................................... ............... 69
S u m m ary ................... ...................7...................6..........

5 D ISC U S SIO N ............................................................................... 83

D discussion of the Findings .................. .......................... ............... .. ............ 84
Conclusions and R ecom m endations ................................................................................. 86
Im p licatio n s .........................................................................9 3
F utu re R research ................................................................9 5
S u m m ary ................... ...................9...................6..........

APPENDIX

A UNIVERSITY OF FLORIDA INSTITUTIONAL REVIEW BOARD ..............................100

B REQUEST FOR DISTRICT FCAT DATA ...................................................................... 101

C DISTRICT ONE INSTITUTIONAL REVIEW BOARD ..................................... 103

D DISTRICT TWO INSTITUTIONAL REVIEW BOARD .................. .......... 105

E DISTRICT THREE INSTITUTIONAL REVIEW BOARD ........................... 106

F DISTRICT FOUR INSTITUTIONAL REVIEW BOARD ........ ...... ...........108

G DISTRICT FIVE INSTITUTIONAL REVIEW BOARD ................................110

H DISTRICT SIX INSTITUTIONAL REVIEW BOARD .................................................111

L IST O F R E F E R E N C E S ....................................................................................................... 112

BIOGRAPHICAL SKETCH .........................................................................118









7









LIST OF TABLES

Table page

2-1 Quality Indicators of Effective Tutoring...................................... ................. .. ............. 56

3-1 Participating D district Statistics................................................. .............................. 66

4-1 SE S Students by G rade L evel ........................................ ..............................................77

4-2 SE S Students by D istrict.......................................................................... ....................77

4-3 A ctive SE S Providers by D istrict............................................................ .....................78

4-4 Total Students: Math Only Group by District and Grade ............................ ...............78

4-5 Total Students: Reading Only Group by District and Grade.................... ............ 79

4-6 Total Students: Reading and Math Group by District and Grade............... ........... 79

4-7 Analysis of Variance for Math Only Group ................. ................................ 80

4-8 Analysis of Variance for Reading Only Group ........................ ............... ..............80

4-9 Analysis of Variance for Reading and Math Tutoring Group .......................................80

4-10 A analysis of V ariance for D district 1 ......................................................... .....................80

4-11 Analysis of Variance for District 2 .............................................................................81

4-12 Analysis of Variance for District 3 ..............................................................................81

4-13 Analysis of Variance for District 4 .............................................................................81

4-14 Analysis of Variance for District 5 ..............................................................................81

4-15 Analysis of Variance for District 6 .............................................................................82

5-1 Summary of Findings for Hypotheses 1 through 3.....................................................98

5-2 Summary of Findings for Hypothesis 4 ............... ............. .. ...............99











8









LIST OF TERMS


Adequate Yearly Progress







The Elementary and Secondary Education Act


Eligible Students




Eligible School





No Child Left Behind




Provider


Public School Choice




State Assessments


(AYP) is the measure of the extent to which
students in a school taken as a whole, and certain
groups within the school, demonstrate
proficiency in at least reading/language arts and
mathematics. It also measures the progress of
schools under other academic indicators, such as
the graduation or school attendance rate.

(ESEA) was first enacted in 1965, is the principal
federal law funding K-12 education.

Students eligible for Supplemental Educational
Services are those students from low-income
families who attend Title I schools that are in
their third year or more of not making AYP.

An eligible school is a Title I school that has
students eligible for Supplemental Educational
Services. The school must be a Title I school that
has not made adequate yearly progress for three
years or more.

The federal No Child Left Behind Act, the most
recent authorization of the Elementary and
Secondary Education Act (ESEA), is the primary
federal law funding K-12 education.

A provider of Supplemental Educational Services
may be any public or private (non-profit or for-
profit) entity that meets the State's criteria for
approval. Potential providers include public
schools, including charter schools, private
schools, school districts, educational service
agencies, institutions of higher education, faith-
and community-based organizations, and private
businesses.

Students who attend a Title I school in their
second year of not making AYP are eligible to
transfer to another public school in the district,
including a public charter school.

Under the No Child Left Behind Act, state
assessments are aligned with academic standards.
Beginning in the 2002-03 school year, schools
were required to administer assessments in each









of three grade spans: grades 3-5, grades 6-9, and
grades 10-12 in all schools. Beginning in the
2005-06 school year, assessments were required
to be administered every year in grades 3 through
8 in math and reading. Beginning in the 2007-08
school year, science achievement was also
assessed.

Subgroup Subgroups are smaller groups of students
disaggregated from the whole group that may be
present in a school or school system. The
subgroups specified in the NCLB law are Native
American/Alaskan Native students, Asian/Pacific
Islander students, Black students, White students,
Hispanic students, students with limited English
proficiency, students who are economically
disadvantaged, and students with disabilities.

Supplemental Educational Services Supplemental Educational Services are
additional academic instruction designed to
increase the academic achievement of students
from low-income families who attend Title I
schools in their third year or more of not making
AYP.

Title I Title I provides federal funding under NCLB for
schools to ensure that all students have a fair,
equal, and significant opportunity to obtain a
high quality education.









Abstract of Dissertation Presented to the Graduate School
of the University of Florida in Partial Fulfillment of the
Requirements for the Degree of Doctor of Philosophy

ASSESSING THE EFFECTIVENESS OF SUPPLEMENTAL EDUCATIONAL SERVICES
IN URBAN FLORIDA SCHOOL DISTRICTS

By

Myrna L. Allen

May 2008

Chair: David Quinn
Major: Educational Leadership

Nationally, millions of federal dollars are given to private tutoring companies to provide

tutoring to low income students who participate in the Supplemental Educational Services (SES)

program in Title I schools. There is little or no research to prove the effectiveness of this

program in improving the academic growth of these students. The purpose of this study was to

evaluate the effectiveness of the SES program in six large urban Florida school districts by

utilizing existing scores from the Florida Comprehensive Assessment Test (FCAT).

A quasi-experimental design was used to analyze the FCAT developmental scale scores

wherein the scores were divided into two groups of Title I students. Group One was composed of

students who received SES in the 2006-07 school year and completed the FCAT in math and

reading in both 2006 and 2007. Group Two was a control group composed of students who were

enrolled in Title I schools, completed the FCAT in reading and math in 2006 and 2007, and were

eligible for SES but did not participate in the program. The developmental scale scores from the

2006 and 2007 FCAT administration were used as pre- and post-tests. A one-way analysis of

variance design was used to analyze the FCAT data. The analysis was completed using the

Statistical Package for Social Sciences with a .05 level of significance.









Results of this study found increased developmental scale scores for Title I students who

received tutoring in math, but there were no statistically significant findings on increased

academic achievement for students who received tutoring in reading only or in reading and math.

This study also assessed the effectiveness of individual SES providers by analyzing the three

largest SES providers in each of the six districts that participated in this study. Of those 18

providers, only 3 provided results that confirmed increased academic achievement.









CHAPTER 1
INTRODUCTION

The Elementary and Secondary Education Act (ESEA) of 1965 was signed into law by

President Lyndon B. Johnson as part of the War on Poverty and is still the principal federal law

funding K-12 education (Kennedy, 2005). ESEA created the Chapter I program, subsequently

renamed the Title I program, to ensure that students living in poverty have an equal opportunity

to obtain a high quality education and achieve the required academic growth to meet state

standards and assessments. Title I provides federal funding to help students who are either

behind academically or are at risk of falling behind academically. Funding is based on the

number of low-income children in a school, generally including those eligible for free or reduced

price lunch, and was intended to supplement, not replace, state and district funds. Schools

receiving Title I monies have been required to involve parents in deciding and evaluating how

these funds are spent and in the review process (Peterson & West, 2003). Title I sought to

strengthen America's schools by allocating substantial federal resources "to turn the door of the

schoolhouse into a true door to opportunity for all students" (Kennedy, 2005, p. 16). The goal

was to afford all students, even the most vulnerable, the opportunity to succeed by establishing

public school classrooms as a place of learning.

ESEA was most recently reauthorized on January 8, 2002 when President George W. Bush

signed the No ChildLeft BehindAct (NCLB) into law. NCLB is referred to as the most

comprehensive piece of federal legislation in 35 years (Peterson & West, 2003). This law

represents an extensive renovation of federal support for elementary and secondary education in

the United States and affects every federal program initially authorized under ESEA. As the

largest federal program supporting elementary and secondary education, NCLB focuses

resources where students have the greatest need (U. S. Department of Education, 2002).









NCLB was signed into law in response to demands for public education to meet the

challenges of America's workforce and changes occurring in America's industry. The economic

expansion of the 1990s gave birth to a new global economy, which increased the pressure on

America's public education system to set high standards, create stronger schools, and produce

better teachers. In response to this pressure, President Clinton proposed education reform in the

Goals 2000 Educate America Act, that became law on March 14, 1994. Goals 2000 was designed

to increase academic standards, measure student progress, and provide adequate support to

enable students to meet the increased academic standards. Although Federal funding for

education was increased from 1994 to 2000, Goals 2000 still fell short of meeting America's

educational needs (Kennedy, 2005).

Poverty and inequality in America's education system continued to be a roadblock to

opportunity and progress and became visible in achievement gaps between student groups.

African-American and Latino students continued to lag behind in academic achievement

compared to other students. For example, the math and reading skills of ethnic minority students

graduating from high school were equal to those of 13 year-old white students. Kennedy (2005)

reported that the high school dropout rate increased for African-American students at the rate of

1 dropout for every 20 black students, compared to 1 for every 30 white students. Achievement

gaps became especially obvious in high poverty schools, where in 2001 these students were 77%

more likely to be taught by a teacher who was teaching out-of-field as compared to students in

low poverty schools. Kennedy also contends that while the dropout rate increased for African-

American students, the number of students learning English as a second language also increased

by 1 million students, a 50% increase. America's schools were not equipped to meet the

educational needs of these students. The hope of Brown vs. the Board of Education (1954) to









provide an equal, quality education for all minority students was still not a reality in 2001.

America's ethnic minority and low-income students were not receiving a quality education from

qualified teachers (Kennedy, 2005). Furthermore, international educational comparisons revealed

that other countries' education systems were producing students who were academically superior

to students in the United States (Hess & Finn, 2004b).

Education remained foremost on the minds of the American people. During the 2000

presidential campaign both candidates pledged to improve the education system. George W.

Bush, the GOP candidate and governor of Texas, promoted the accountability system of Texas as

a national model. Upon taking office, President Bush made good on his campaign promise and

immediately sent to Congress legislation that created strong accountability for districts and

states, as well as a school choice option modeled after the system in place in Texas public

schools (Hess & Finn, 2004b).

Negotiations over this legislation lasted a year, as the White House and Congressional

leaders forged a bipartisan compromise that came to be known as the No ChildLeft BehindAct.

NCLB enjoyed strong support from Republicans and prominent Democrats such as

Massachusetts Senator Edward Kennedy and California Representative George Miller. However,

winning bipartisan support created many changes in the original Bush education plan. For

example, the White House compromised on its proposal of choice-based reforms, such as school

vouchers, and agreed to more restrictive options for students in low-performing schools. These

negotiations added to the complexity of the NCLB legislation and, as a result, the federal

government was challenged to work with and through states to make implementation meaningful

with the end result providing a quality education for all American students (Hess & Finn,

2004b).









For the first time the federal government tied federal dollars to consequences for schools

that failed to make adequate yearly progress. It gave parents and students in low-performing

schools the option to obtain additional educational assistance or move to another school. From

the time NCLB was signed into law it became evident that this complex law required extensive

monitoring and evaluation. The academic performance of students, individual schools, and

school districts would necessitate the use of best practices to turn NCLB into policy and

subsequently into practice in every school district in America (Hess & Finn, 2004b).

NCLB was charged with the reformation of the American education system which

included meeting the educational needs of low-income children, children with disabilities,

minority children, and English language learners. These students became the top priority of

school reform. NCLB called for improvement in America's schools and demanded

accountability for results in decreasing the achievement gap. The law required each state to

create content standards that specified what students should know and performance standards

that specified mastery levels for students. State tests were to be rigorous and aligned with the

standards, and teachers were to be of high quality for all students. The law created opportunities

for students to transfer to other schools, participate in after-school programs, and obtain free

tutoring and extra academic support. The law also required that parents be informed of the

academic performance of their child's school and be encouraged to become more involved in

their child's education. According to Kennedy (2005), the No Child Left BehindAct is "more

than a slogan," (p. 18). It is a commitment and a promise to offer each child a quality education

by improving public education.

The commitment to strengthen public education by improving the academic achievement

of all students was apparent in NCLB by its very specific requirements for accountability: annual









state assessment of the academic performance of students in grades 3 through 8 in math and

reading, with one additional test to be administered in high school. The state assessments must

be based on rigorous state standards and at least 95% of enrolled students in each school must

participate in the assessment. The results are reported by student subgroups which include

poverty levels, race, ethnicities, disabilities, and limited English proficiencies. These data are

converted into an annual school report card required by NCLB, and provide comparative

information on the quality of schools (U. S. Department of Education, 2002). School report cards

detail the performance of each subgroup and are required to be reported to the public. These

annual report cards must show adequate yearly progress (AYP) of the school toward the goal of

100% student proficiency by the end of the 2013-2014 school year. The AYP is defined by each

individual state and is evaluated most commonly in terms of benchmarks that increase on

predetermined intervals until the goal of 100% proficiency is reached in the 2013-14 school year.

If a school does not demonstrate AYP, a sanction process, described below, is initiated and

increases every year until the school makes AYP for two consecutive years (Peterson & West,

2003).

As well as addressing the academic performance of students, NCLB built upon and

expanded the Elementary and Secondary Education Act by creating ground breaking educational

mandates such as parental choice, high teacher quality, and Supplemental Educational Services

(SES). These programs were created to increase the quality and effectiveness of the Title I

program in raising the achievement of all students, especially those students achieving at the

lowest levels (Stullich, Eisner, McCrary, & Roney, 2006).

Promoting parental choice and increased involvement in their child's education is one of

the cornerstones of NCLB (Hess & Finn, 2004b). Under this legislation, local school districts









must set aside 20% of their Title I, Part A allocation. Title I is a federally funded formula grant

given to school districts to supplement the education of students who attend high poverty

schools. The 20% setaside is to be used to fund the first two sanctions under NCLB, public

school choice and SES. The first sanction, public school choice, must be offered to students who

attend schools that have failed to make AYP for two years. Part of the 20% is required to fund

the transportation of students from their home school to the school of their choice (U. S.

Department of Education, 2002).

Supplemental Educational Services (SES)

If schools fail to make AYP for three years, they are required to offer SES, the second

sanction under NCLB. SES is designed to increase the academic achievement of students from

low income families who attend Title I schools. NCLB defines SES as additional academic

assistance (e.g., tutoring, remediation, and other educational interventions) provided that these

approaches are consistent with the content and instruction used by the school district and are

aligned with the state's academic content standards. In addition to the NCLB sanctions of public

school choice and SES, additional sanctions are added in each subsequent year a school does not

make AYP. These additional sanctions include corrective action, planning for restructuring, and

implementation of restructuring (U. S. Department of Education, n.d.; Sunderman & Kim, 2004).

The SES program must be aligned with the state's academic standards and is required by federal

guidelines to be provided before school, after school, or on Saturday. Additionally, the SES

program must be of high quality, research based, and designed to increase student achievement.

This program is structured to create partnerships between public schools and a wide variety of

tutoring companies (U. S. Department of Education, 2005).

The Supplemental Education Services program evolved from President Bush's initial

proposal, when drafting NCLB, that a student in a school that did not make AYP for three years









would be offered a $1,500 exit voucher to be used toward a private school education. The exit

voucher was not well received by the Democrats, and Senator Kennedy adamantly refused to

support it. However, he did counter with a proposal to use these funds as "supplemental

services." The White House accepted this proposal feeling that it would be a sufficient incentive

to improve failing schools because the districts would have to pay for the tutoring out of their

Title I funds, thus decreasing the amount available for district use. Furthermore, President Bush

felt that private tutoring would give struggling students an educational opportunity that would

otherwise be unavailable to them (Hess & Finn, 2004b).

In Florida for the 2007-08 school year, the 20% setaside ranges from a low of $46,724 for

Liberty County Public Schools to a high of $25,308,760 for Miami-Dade County Public Schools,

with 22 Florida school districts encumbering over $1 million for this setaside. The total 20%

setaside encumbered to fund public school choice and SES in Florida is $115,140,533 for 67

school districts (Florida Department of Education, 2007a). Full implementation of SES

nationwide could total more than $2.5 billion of the annual Title I grant funds (Ascher, 2006;

Cohen, 2006).

The state, local school district, parents, and SES providers are all charged with

implementation responsibilies via the Supplemental Educational Services Non-Regulatory

Guidance (2005), developed by the United States Department of Education. Implementation of

SES begins with the state and includes a number of responsibilities in providing services to

eligible students. These responsibilities begin with the identification of providers, as well as

maintaining a list of approved providers, and monitoring the implementation and success of the

program. States start the SES process by developing and advertising the procedure by which a

tutoring company becomes state-approved to serve students. Once a provider is approved, the









state must notify individual school districts of the providers who are eligible to provide SES. The

state is also charged with the responsibility to develop, implement, and publicly report the

effectiveness of SES providers and is required to withdraw approval from providers that fail to

increase the academic performance of students for two consecutive years (U. S. Department of

Education, 2005).

The school district also has many responsibilities in implementing SES in Title I schools

that have not made AYP for three years. Districts must notify parents of eligible students

annually and must offer assistance, if requested, in helping parents choose a provider that will

meet their child's educational needs. Federal guidance mandates that the notification sent to

parents must include a comprehensive description of the SES program and a timeline that

explains the enrollment process. Districts are urged to reach the eligible population by notices

mailed to homes, as well as by newspaper, radio and TV ads, Internet sites, and notices at movie

theaters, shopping malls, beauty salons, and churches. To ensure the privacy of students, districts

are reminded that individual student identities must not be released to the public. Districts also

must draft and enter into a contract with the state-approved SES providers. If the demand for

services exceeds the available funding, districts must implement a prioritization plan (U. S.

Department of Education, 2005).

After the enrollment process is completed and contracts with SES providers are executed,

the district is responsible for implementation of the program. This includes supervising the

program in each school where services are provided, paying the SES providers, monitoring

student attendance, verifying that the tutors have clear background checks and meet education

requirements, as well as approving each enrolled student's academic plan (Florida Department of

Education, 2006b)









Parent responsibilities include choosing a provider from the state-approved list. Parents

are permitted to request assistance from the district in making this choice. They also are required

to be active participants in the SES program, which includes working with the district and

provider to create an academic plan appropriate for their child's needs and making sure that their

child attends the tutoring sessions on a consistent basis.

The SES provider responsibilities include providing the appropriate curriculum to enable

each child to succeed academically in the program. This includes monitoring student attendance,

measuring student progress, reporting student progress to the parents and district, following the

timeline set forth by the district, and ensuring that student privacy is maintained. Provider

responsibilities also include providing services that are secular, neutral, and non-religious (U. S.

Department of Education, 2005).

To fund the SES program each student is allotted a per pupil allocation for tutoring.

During the 2007-08 school year in Florida, per pupil allocations ranged from a low of $883 for

Nassau County Public Schools to a high of $1,467 for Miami-Dade Public Schools; the average

allocation was $1,124 per student (Florida Department of Education, 2007d). When the

allocation is depleted, services automatically end for each student. Providers set their hourly

rates within the parameters established by each state; in Florida, the hourly rate has ranged from

$5 to $80 per hour. The hourly rate in Florida for the 2007-08 school year ranges from a low of

$20 per hour to a high of $80 per hour, with an average of $60 an hour for the 201 state-

approved providers. SES providers also choose the student-to-teacher ratio they will utilize. The

Florida Department of Education allows providers to choose from small group instruction with a

5:1 student: tutor ratio, large group instruction that cannot exceed 10 students per tutor, or

individual tutoring. Providers are paid the hourly rate multiplied by the number of students being









tutored. For example, if a provider is tutoring 10 students at $80 an hour, their total hourly

income would be $800. It is also important to note that, in Florida, SES providers are only paid

for students who attend the tutoring sessions (Florida Department of Education, n.d.). The

federal guidance allows states to determine if payment for tutoring will be based on attendance.

The Florida policy requires that the student be present. Otherwise, the provider cannot charge the

district for services to the student on that day (Florida Department of Education, 2006a).

Statement of the Problem

Nationwide, each Title I school that has not made AYP for three years is required to

provide SES to its students from low-income families. In the 2004-05 school year, 19% of

eligible funded students participated nationwide in SES, which was a 7% increase over 2003-04,

when only 12% participated in this free tutoring program (Fusarelli, 2007). Locally, the Florida

Department of Education reported that in the 2006-07 school year 70,908 students participated in

SES. This is more than triple the enrollment of the 2005-06 school year when 23,187 students

received services (Florida Department of Education, 2007c).

The main goal of the Supplemental Educational Services program is to provide tutoring or

other supplemental academic enrichment services in reading, language arts, and mathematics to

increase the academic performance of students. These tutoring sessions must be offered before or

after school, and they must be research based (Florida Department of Education, 2006b). To fund

this program, districts are required to setaside 20% of their Title I, Part A funds and they have

been doing so since the 2002-03 school year (Cohen, 2006). School districts currently spend

billions of federal dollars on this program nationwide with funds that would have been allocated

to schools to supplement the curriculum before NCLB was signed into law (Cohen, 2006;

Sunderman, 2006).









Florida implemented the SES program in 2004-05 in 11 urban districts that had Title I

schools in their third year of non-adequate yearly progress. The number of districts required to

offer SES has increased every year until the 2006-07 school year, when SES was required in

every school district in Florida. In the 2007-08 school year, 201 Florida providers were approved

to tutor students (Florida Department of Education, n.d.).

Now that SES has been fully implemented in Florida, the focus now should turn from

implementation and move towards quality control of the services provided. This may be the most

challenging aspect of NCLB, and many states and districts are struggling with SES evaluation.

The entire concept of successful tutoring must be clarified. Evaluation must address what

specifically increases a student's academic performance, especially in schools where many

different programs and new curricula are being implemented to help low performing students to

achieve. If a student progresses to higher academic levels on the state assessment, it must be

determined if the increase is due to a few hours of tutoring per week or to a new curriculum

implemented school wide (Hess & Finn, 2004b).

Per the following excerpt from the NCLB Statute, Section 1116(e) 4, evaluating the

effectiveness of SES is clearly defined as a state responsibility:

4) STATE EDUCATIONAL AGENCY RESPONSIBILITIES- A State educational
agency shall-

(D) develop, implement, and publicly report on standards and techniques for monitoring
the quality and effectiveness of the services offered by approved providers under this
subsection, and for withdrawing approval from providers that fail, for two consecutive
years, to contribute to increasing the academic proficiency of students served.

States are beginning to focus on SES effectiveness, but there has been no definitive answer

from the federal government on what defines effectiveness (Hess & Finn, 2004b). Monitoring

the quality and effectiveness is a major challenge in many states. The Northwest Regional

Educational Laboratory (2004) sums up the importance of quality control by stating that the main









concern it is not the lack of evaluation, but rather the importance of knowing whether low

achieving Title I students are receiving high quality services that can actually improve their

academic performance. Although SES has been implemented for over four years and billions of

dollars have been spent, there is still little known about the effects of tutoring on improving

student achievement, the fundamental goal of SES (Ascher, 2006).

Purpose of the Study

The purpose of this analysis was to assess the effectiveness of the SES program in Florida

Title I urban district schools that were required to offer this program during the 2006-2007

school year.

To assess the effectiveness of SES, the following research questions will be addressed:

1. Do students who participated in Supplemental Educational Services exclusively in math for
10 or more hours demonstrate higher developmental scale score gains, as determined by the
FCAT, compared to students in a control group?

2. Do students who participated in Supplemental Educational Services exclusively in reading
for 10 or more hours demonstrate higher developmental scale score gains, as determined by
the FCAT, compared to students in a control group?

3. Do students who participated in Supplemental Educational Services in both math and
reading for 10 or more hours demonstrate higher developmental scale score gains, as
determined by the FCAT, compared to students in a control group?

4. Do students tutored by the three major SES providers in each district (i.e., those that
provided tutoring for the largest number of students) for 10 or more hours demonstrate
higher developmental scale score gains, as determined by the FCAT, compared to students
in a control group?

Null Hypothesis 1. There is no statistically significant difference between 2006 FCAT

developmental scale scores as the pre-test and 2007 FCAT developmental scale scores as the

post-test for students in Title I schools receiving at least 10 hours of tutoring in math.









Null Hypothesis 2. There is no statistically significant difference between 2006 FCAT

developmental scale scores as the pre-test and 2007 FCAT developmental scale scores as the

post-test for students in Title I schools receiving at least 10 hours of tutoring in reading.

Null Hypothesis 3. There is no statistically significant difference between 2006 FCAT

developmental scale scores as the pre-test and 2007 FCAT developmental scale scores as the

post-test for students in Title I schools receiving at least 10 hours of tutoring in math and

reading.

Null Hypothesis 4. There is no statistically significant difference between 2006 FCAT

developmental scale scores as the pre-test and 2007 FCAT developmental scale scores as the

post-test for students receiving at least 10 hours of tutoring from the three SES providers who

tutored the largest number of students in each district


Significance of the Study

The highest priority of the No ChildLeft BehindAct is to give low achieving students the

opportunity and ability to meet high academic standards and reach proficiency on demanding

state assessments. The fact that only 29% of all fourth grade students performed at or above the

proficient level on the National Assessment of Educational Progress in reading in 2000 clearly

indicated a need for nationwide education reformation (U. S. Department of Education, 2002).

The federal government chose the reauthorization of ESEA to implement education reform,

which funded Title I in 2007 in the amount of $12.7 billion nationwide. This reformation

requires accountability from states that received this federal aid (U. S. Department of Education,

2006).

Currently, Florida has no evaluation of the effectiveness of Supplemental Educational

Services which consume federal funds that have been diverted from Title I schools. While









Florida is required to prohibit providers from offering services to students if they fail to increase

student achievement for two years, the state has neglected to enforce this section of the law.

NCLB has been described as the accountability law, but this cornerstone program has not been

held accountable for results. SES cannot be improved if the weaknesses and deficiencies are not

identified. Evaluating SES effectiveness will allow states, districts, and lawmakers to strengthen

and improve these services.









CHAPTER 2
LITERATURE REVIEW

The purpose of this study is to assess the effectiveness of Supplemental Educational

Services, most often offered as a tutoring program, designed by NCLB to increase the academic

achievement of Title I students in urban Florida school districts. The purpose of this chapter is to

present a review of the literature pertaining to the history and best practices of tutoring, as well

as the effectiveness of other after-school tutoring programs. The current status of SES evaluation

efforts nationwide will then be reviewed, followed by existing research on the effectiveness of

SES. Finally, the literature relevant to SES evaluation challenges and recommendations to

strengthen and evaluate SES will be presented.

History of Tutoring

Tutoring is a form of education with a long, rich history that has been used to educate

children for centuries (Gordon, Morgan, Ponticell, & O'Malley, 2004). Tutoring is as old as

civilization itself, and its beginnings can be traced back to as early as 7000 B. C. when the family

was responsible for the education of its children, as parents tutored their children in life skills

and passed on their culture's folklore. History's most favored and honored teachers were tutors.

Socrates, Plato, and Aristotle taught their students one at a time by using the principal of

imitation, lectures, and posing problems to their students. Tutoring was the only form of

education in ancient Rome and Greece and can be traced through and beyond the medieval

transition in 500- 440. During the medieval period the Church controlled education; thus tutoring

focused on theological principles, and secular training was abandoned. During this time Britain

tutored girls in the nunneries and, at the same time, boys in Ireland were tutored by receiving

instruction in monasteries. As the century progressed, tutoring again returned to the home, but it









was only available for the children of royalty or noble descendants. Centuries passed, but the

primary subject of tutoring remained theology (Gordon & Gordon, 1990).

Tutoring persisted in Europe as the primary style of education, and when the colonists

came to America they brought tutoring with them. In America, education soon gave birth to the

public school, but tutoring was often the only form of education available in isolated, rural,

frontier settings. Education was valued in colonial America. Often when a child grew up in a

home of non-readers it was not unusual for neighbors to offer to tutor the child. New England

Puritans' concern for education centered on strong religious beliefs, and tutoring centered on

reading the Bible. Tutoring persisted throughout the frontier era, but the number of children

tutored at home declined as America was settled and more states passed compulsory school

attendance laws (Gordon & Gordon, 1990). As the 19th century transitioned to the early 20th

century, educators ignored tutoring as a part of education. They contended that due to student

socialization issues, a public school education for all children was superior to tutoring small

groups of isolated students (Gordon et al., 2004).

Interest in tutoring was revived in the 1960s when the public started to question the quality

of public education due to the poor academic achievement of many children. Peer tutoring was

introduced in Detroit Public Schools and with the Mobilization for Youth Program in New York

City. Peer tutors were soon joined by adult tutors and paid tutors. By 1973, the National School

Volunteer Program estimated that 2,000,000 tutors were tutoring 3,000,000 students in over

3,000 programs. Tutoring was commonly used to educate students with disabilities, as well as

gifted students. The academic benefits of tutoring children were seen in classrooms across

America in the 1960s and 1970s (Gordon & Gordon, 1990). Since the 1960s, each passing

decade has increased the demand for tutoring in each socioeconomic group (Gordon, 2003).









Presently, tutoring has become a permanent tool in educating students in the United

States. Gordon & Gordon (1990) predicted that the "future use of tutors across society will likely

increase as educators continue to focus on individual differences and the means to increase

productivity in all modes of learning" (p. 320).

Quality Indicators of Effective Tutoring

Tutoring has made a significant contribution to the education of children throughout

history. Tutoring adjusts for a child's individual differences and creates a more meaningful

education for students (Gordon & Gordon, 1990). There were several common themes identified

in the literature as quality indicators of effective tutoring. Small student-teacher ratio,

collaboration with classroom teachers, tutor qualifications, and tutor training were identified

repeatedly as quality indicators of tutoring.

One-to-one tutoring is considered as the most effective form of tutoring (Baker, Rieg, &

Clendaniel, 2006; Elbaum, Vaughn, Hughes, & Moody, 2000; Juel, 1996; Lauer et al., 2004;

Lauer, 2006; Wasik, 1998). Tutoring programs that provide one-to-one tutoring for at-risk

students were proven to increase the academic achievement of students in reading. Tutoring

programs that target reading as their primary goal should strive to provide one-to-one tutoring

(Elbaum et al., 2000; Lauer et al., 2004). Elbaum, et al. (2000) conducted a meta-analysis to

assess the effectiveness of one-to-one reading interventions for at-risk elementary students with

low reading skills. In this meta-analysis of 29 studies, the reading outcomes for 42 samples of

students (n = 1,539) were examined. It was discovered that students who received one-to-one

tutoring performed at a level that was two fifths of a standard deviation higher than the average

level of the control group. An important program component of any tutoring program is to keep

the student-teacher ratio low so students can obtain the best possible benefits (Sanderson, 2003).









Baker et al. (2006) suggested maintaining a 2:1 student teacher ratio or smaller for a successful

tutoring program.

Collaboration between the child's classroom teacher and the tutor also has been identified

as an important aspect of effective tutoring (Baker et al., 2006; Gordon, 2003; Gordon et al.,

2004; Wasik, 1998). Tutoring needs to be coordinated with classroom instruction to present the

student with supplementary tutoring on what is being taught in the classroom with the same

materials and taught in the same manner. This enables the student to have multiple opportunities

to work on challenging materials. As students master the material during tutoring, they are more

likely to perform better in class. This can be an effective motivator for future tutoring sessions

(Wasik, 1998). The tutor is able to provide individual, intense instruction that a classroom

teacher is unable to provide in a group setting and, in most cases, the classroom teacher is more

than willing to collaborate with the tutor (Gordon, 2003). Close tutor-teacher collaboration will

help make the most of each tutoring session (Gordon et al., 2004).

Tutoring best practices addresses the qualifications of tutors. Experienced, certified

teachers make the best tutors (Elbaum et al., 2000; Gordon, 2003; Gordon et al., 2004; Wasik,

1998). Teachers make effective tutors because they are specifically educated to teach, and

tutoring enables them to reach out and individualize instruction, which they are unable to do in a

classroom setting. Due to their education and prior professional experience, teachers can make a

major difference in a tutoring situation (Gordon et al., 2004).

Research also reveals that training tutors is important in effective tutoring (Baker et al.,

2006; Elbaum et al., 2000; Gordon et al., 2004; U. S. Department of Education, 1997; Vadasy,

Jenkins, Antil, Wayne, & O'Connor, 1997; Wasik, 1998). Extensive training stands out among

the features of tutoring programs that produce the most positive effects (U. S. Department of









Education, 1997). Training must be ongoing and include observing tutors and students during

tutoring sessions, followed by immediate feedback. Volunteer tutors also need to be trained to

have an understanding of the reading process and the complex cognitive interactions that

facilitate reading (Wasik, 1998). Specialized training as a tutor consistently produces higher

levels of student achievement than tutors with little or no special training (Gordon et al., 2004).

Other quality indicators of effective tutoring include a long term commitment sustained

over the entire school year with the same tutor working with the same student, intensive and

consistent tutoring, as well as using tutors that can motivate and encourage (Baker et al., 2006;

Gordon, 2003; Juel, 1996; Wasik, 1998).

The Effectiveness of Other After-school Tutoring Programs

There is existing research that assesses the effectiveness of other academic tutoring

programs provided to students outside their regular school day. One such research study

completed in 2002 evaluates the 21st Century Learning Community Centers, a nationally known

after-school program for children in urban and rural communities that includes academic

activities. The results of this evaluation have shown that the program had limited influence on

student achievement. At the elementary and middle school levels, test scores in reading and

grades in most subjects were not higher for 21st Century students than for similar students in a

control group. This after-school program had no impact on whether students completed their

homework or finished assignments to their teacher's expectations (U. S. Department of

Education, 2003).

A meta-analysis of 65 studies on school based tutoring programs confirmed that tutoring

has some positive effects on raising student academic achievement. Cohen, Kulik & Kulik

(1982) conducted this meta-analysis and reported on the academic achievement of tutored

students. Of the 65 studies, 45 reported that the tutored students made achievement gains, but









only 20 of the comparisons reported statistically significant effects for students receiving the

tutoring. Further analysis of the studies showed that students in more structured programs had

larger tutoring effects, as did students in tutoring programs of shorter duration. Effects were also

larger when lower level skills were taught and tested. Tutoring in math produced larger effects

than tutoring in reading. Students exhibited larger academic effects when tested on locally

developed tests rather than nationally standardized tests.

In 2001, the Corporation for National Service commissioned a report on the outcomes of

tutoring conducted by AmeriCorps. AmeriCorps is a network of local, state, and national service

programs that connects more than 70,000 Americans each year in intensive service to meet the

critical needs of our country in education, public safety, health, and the environment. The

AmeriCorps study conducted a meta-analysis of their tutoring programs that used adult

volunteers. The study sample included 869 students in 68 tutoring programs and assessed their

reading gains using the Woodcock Johnson Reading Test as the pre-test and post-test measure.

The 869 students were nearly evenly dispersed across grade levels with 294 in first grade, 292 in

second grade, and 283 in third grade. The number of boys and girls was almost evenly split in

grades two and three, however the first grade student sample was 59% female and 41% male.

Ethnically the sample was composed of 51% white students, 23% African-American, 20%

Hispanic, and 7% Native American, Pacific Islander, Asian, or mixed ethnicity. Findings

indicated that tutored students at all grade levels that began the program with below grade-level

academic achievement closed the gap and were reading at or near their grade-level by the end of

the school year as measured by the Woodcock Johnson Reading Test. These gains were large

enough to be statistically significant and represented meaningful changes in the students reading

performance. The reading gains were the same for students of different ethnic/racial









backgrounds. Surveyed teachers also agreed that tutored students improved their reading to some

degree (Moss, Swartz, Obeidallah, Stewar, & Green, 2001).

AmeriCorps program directors reported that more than a third of the tutoring programs

were conducted in one-to-one tutoring sessions and an additional 30% of the programs reported

small group tutoring consisting of two to four students. The length of the tutoring sessions lasted

from 30-60 minutes and totaled more than 1.5 hours a week. The report also stated that tutor

training was critical to a successful tutoring program. Moss et al. (2001) identified several

effective practices that, when implemented, were essential in showing significant reading gains

in the AmeriCorps program. These effective practices include:

1. Tutors receive training both prior to and during the course of tutoring;
2. Tutoring sessions occur at least three times a week;
3. Tutoring activities are monitored and evaluated. (p. 36)

In 2004, Lauer et al. conducted a research synthesis on the effectiveness of out-of-school-

time strategies for improving the academic performance of low-achieving students. The meta-

analysis included 53 studies conducted after 1984 that concentrated on the effectiveness of a

program that was delivered outside the regular school day. Program indicators included the

timeframe in which the program was offered (after school or summer), grade level of students,

focus of the program (academic or academic and social), duration of the program, and student

teacher ratio. Results from this research synthesis led to the following conclusions:

1. Out-of-school-time strategies can increase the achievement level of low-achieving
students in reading and math.

2. The times that out-of-school strategies are offered (after school or summer
school) do not impact the effectiveness.

3. Student in early grades had the greatest gains in reading. Students in secondary
school had the greatest benefit from math strategies.

4. These strategies have positive effects when the focus of the program is academic
or academic and social.









5. Administrators should closely monitor the implementation and ongoing student
learning to determine the appropriate duration for particular strategies and
activities.

6. Out-of-school-time strategies that provide one-to-one tutoring for low-achieving
or at-risk students have strong positive effects on reading achievement.

Current Status of Evaluating SES

One of the key provisions of NCLB is the implementation of SES, which is designed to

provide tutoring or other academic activities to students from low-income families who attend

Title I schools that have been identified as not making AYP for three years. As a result of not

making AYP, these schools are required to participate in NCLB sanctions which include SES.

The SES program is to be implemented in Title I schools in their second year of school

improvement, corrective action, or restructuring (Smole, 2004). SES is defined as additional

academic instruction designed to increase the academic achievement of students in low-

performing schools. These services are most commonly provided in the form of tutoring and

must be provided outside the regular school day by companies known as SES providers that are

approved by each state's Department of Education (Sunderman & Kim, 2004). Parents of

eligible Title I students select an SES provider from a state-approved list, and the local school

district contracts and implements the tutoring program with the selected provider. SES providers

are required to demonstrate that the tutoring provided increases student academic achievement in

order to retain their status as approved and remain on the state-approved list. Therefore, states

are required to evaluate SES providers to determine their effect on increasing student

achievement (Smole, 2004). This topic is addressed in the SES Non-Regulatory Guidance under

state responsibilities and monitoring requirements:

B. OVERVIEW OF STATE RESPONSIBILITIES

B-1. What is the responsibility of a State Educational Agency (SEA) in providing
Supplemental Educational Services?









The SEA has a number of responsibilities in ensuring that eligible students receive
additional academic assistance. The SEA must identify providers, maintain a list of
providers, and monitor services [Section 1116(e) (4)]. Specifically, the SEA must:

5. Develop, implement, and publicly report on standards and techniques for monitoring
the quality and effectiveness of services offered by approved Supplemental Educational
Service providers, and for withdrawing approval from providers that fail, for two
consecutive years, to contribute to increasing the academic proficiency of students served
by the providers (see Section D for additional information) (U. S. Department of
Education, 2005, p. 4).

D. MONITORING REQUIREMENTS

D-4. How may an SEA terminate approval of a provider that is not meeting the statutory
requirement to increase students' academic achievement?

An SEA must use a consistent policy for withdrawing Supplemental Educational Service
providers from the State-approved list. The statute requires an SEA to remove from the
approved list any provider that fails, for two consecutive years, to contribute to increased
student proficiency relative to State academic content and achievement standards [Section
111',., 1,-i)]. In addition, a provider must be removed from the list if, at any time, it fails
to provide Supplemental Educational Services consistent with applicable health, safety,
and civil rights requirements (U. S. Department of Education, 2005, p. 19).

The review of literature on the evaluation of SES programs reveals there have been

research studies and evaluations that have produced conflicting results of the effectiveness of

SES on increasing student academic growth. Additionally, in a report delivered to Congress in

August 2006, the Government Accountability Office concluded that no definitive evaluation had

been done to measure the effects of SES on student academic achievement (U. S. Government

Accountability Office, 2006). Although states are required by NCLB to evaluate SES providers

annually, surveys indicate that few states have developed and implemented evaluations (Reid,

2004). Critics contend that there are no thorough evaluations of SES programs in place in any

state and that the evaluations that do exist rely on the most basic evaluation methods (Sunderman

& Kim, 2004). Due to the absence of evaluation procedures, SES providers have no evaluation

data to show student academic growth on state assessments as is required of states and districts

by NCLB. Current state monitoring of SES providers consists of relying on questionnaires,









surveys, and data submitted by the providers themselves. Some states depend on SES providers

to evaluate their own effectiveness by using their pre- and post-test assessments to measure

increased academic growth of students (Reid, 2004).

Lack of Funding

One of the reasons cited for the lack of evaluation is that states lack the capacity and funds

to develop a monitoring system for tutoring (Reid, 2004). NCLB requires states to ensure that

Supplemental Educational Services are of high quality, research based, and designed to help

eligible children attain proficiency in meeting the state's academic achievement standards. States

and districts have been overwhelmed with designing an implementation plan for providing SES

in schools and communities nationwide (Cohen, 2006). Due to the increasing enrollment and

mandatory NCLB requirements, the SES program has created administrative and management

challenges at both state and district level. States and districts are forced to assume these

additional responsibilities with no additional funds to cover them (Ascher, 2006).

NCLB places a financial burden on local and state administrators with no additional

administrative funds designated to cover the costs of implementing and evaluating SES (Ascher,

2006; Burch, 2007). District administrative costs for implementing the SES program are

significant considering that 40% of all urban districts are required to offer SES (Ascher, 2006).

For example, the SES administrative budget of the Chicago School District was more than $2

million in the 2004-05 school year (Sunderman, 2006). Further evidence of escalating SES

administrative costs are demonstrated in Duval County Public Schools in Jacksonville, Florida,

where these costs have increased from $700,000 in 2006-07 to a projected amount of $1.1

million for the 2007-08 school year. This is in addition to the mandatory 20% setaside and

amounts to 3.5% of the total Title I allocation that must be used to administer SES that, in past

years, would have been allocated to Title I schools (Duval County Public Schools, 2007).









An example of the administrative challenges presented by SES can be found in Oakland,

California, where 500 parents failed to select an SES provider on their free tutoring application.

Per federal SES guidance, the SES coordinator was required to contact these parents to select one

of the 25 providers on the district's state-approved list. Duplicate applications also presented a

problem in Oakland where parents submitted one application at the district's SES Provider Fair

and then submitted a duplicate application with another provider who visited the student's home.

Again, the SES coordinator had to contact these parents to determine their provider choice.

Monitoring attendance, verifying student eligibility, and tracking which students are enrolled in

which programs are some of the administrative burdens experienced by districts nationwide

(Rentner et al., 2006).

The Increasing Number of SES Providers

The ever increasing number of approved SES providers nationwide heightens concern over

the lack of evaluation and inadequate monitoring (Cohen, 2006). As of February 2007, states

reported approving at total of 3,223 SES providers (U. S. Department of Education, 2007).

Stullich et al., (2006) reported this was over three times as many SES providers than had been

approved just two years earlier, when the approved list consisted of 997 providers. Many of the

approved providers are leaders in providing educational services, but others have little or no

track record of effectiveness in tutoring low-income, low-achieving students (Cohen, 2006). In

2004, the number of approved providers varied per state with a high of 216 in New York to a low

of 3 in Hawaii (Reid, 2004). The Bush Administration remains an enthusiastic supporter of the

program and has pushed states to increase the number of approved SES providers, even though

there is little evidence of their effectiveness (Sunderman, 2006).

SES providers can include nonprofit, for-profit, and faith-based organizations, as well as

school districts, charter schools, and private schools (Kasmin & Farmer, 2006). An annual









review conducted in 2005-06 by the Center on Education Policy, a national, independent

advocate for public education, reported 54% of all state-approved providers were for-profit, 21%

were nonprofit not affiliated with a religious group, 9% were districts, with the remaining 16%

made up of public entities, private faith-based organizations, and other types of organizations

(Ascher, 2006). A typical tutoring program provides 30 hours of free tutoring (Hess & Finn,

2004a). The mode of instruction can vary, including one-to-one tutoring, small group instruction

and online tutoring. The frequency or intensity of service can also vary because NCLB has no set

requirements except that help with homework is prohibited (Kasmin & Farmer, 2006).

Implementation in Rural Districts and Districts with Low Enrollment

Rural districts and districts with low enrollment face a different problem in attracting a

sufficient number of SES providers to serve their eligible students, thereby creating a lack of

services in their districts (Burch, 2007; Fusarelli, 2007). These districts disclosed that they failed

to implement SES due to a lack of state-approved SES providers, and in one case, there was no

approved provider within 200 miles of the school district (U. S. Government Accountability

Office, 2006). In the Grant Union School district in California, only 26 of the hundreds of state-

approved providers were interested in serving students in this rural district and only 11 attended

the SES Provider Fair for parents. SES providers require a minimum number of students to make

SES a profitable venture. A similar problem occurred in Kansas City, Missouri. Although Kansas

City Schools are considered an urban school district, half of the SES providers on the state-

approved list removed themselves from providing services in this school district due to low

enrollment because they would not be profitable. The only solution at this time for rural districts

or districts with low enrollment is to contract with online SES providers. This presents additional

barriers to serving students such as unreliable connectivity and limited access to computers

(Schwartzbeck, 2005).









Highly Qualified Teachers

Although NCLB mandates highly qualified teachers in every classroom taught during

regular school hours, the law does not specify any education or certification requirements for

SES tutors (Ascher, 2006; Burch, 2007; Kasmin & Farmer, 2006). Districts have expressed

concerns about programs that use tutors who lack teaching credentials (Sunderman, 2006).

Critics say that this is in conflict with the intention of NCLB, which strives to match the most

qualified teachers with students who have the most academic need (Kasmin & Farmer, 2006).

Ascher (2006) found that although most SES tutors are certified teachers employed by the

district, some were college students with no teaching experience, and 7% were high school

students. The American Institutes for Research and the Education Industry Association found

that a majority of SES tutors are certified teachers, while others are high school students and

college graduates without prior tutoring experience (Burch, 2007). The U. S. Department of

Education reported that 15 of 24 the providers interviewed for this case study of supplemental

services stated that they require their tutors to be certified teachers, the remaining 9 did not

require certification. For example, Huntington Learning Center does not require teacher

certification, while Princeton Review hires certified teachers employed in non-SES schools

(Anderson & Laguarda, 2005). Although training programs, typically 4 to 20 hours, are provided

by some SES providers, not all providers train their tutors. It should also be noted that some, but

not all, providers evaluate their tutors.

Increasing Enrollment

Due to the increasing number of schools being required to offer SES programs, the

number of Title I students participating in this free tutoring program increases dramatically each

school year. To fund this rapidly increasing program, local school districts are required to set

aside 20% of their annual Title I allocation (Cohen, 2006).









The lack of evaluation has created boon times for SES providers. The number of students

receiving SES in 2004-05 amounted to over 400,000 students. This number more than tripled

from the previous two years and student participation is still increasing rapidly ("Beyond the

pale: SES provisions emerge as 'loose cannon' reform," 2006). In the 2005-06 school year, the

number of students receiving this free tutoring program again increased to almost 500,000

students (U. S. Department of Education, 2007).

A substantial amount of money is at stake in funding this program. Per NCLB, a portion of

the 20% setaside of Title I funding is to provide SES in school districts (Borja, 2006). Chicago's

SES program served 60,000 students in 2004-05 and cost the district almost $50 million

(Chicago Public Schools, 2005). In 2005, SES providers nationwide received $400 million in

federal money for tutoring services (Borja, 2006). Under full implementation nationwide, SES

could amount to more than $2.5 billion of the almost $13 billion annual Title I grant funds

(Ascher, 2006; Cohen, 2006). SES providers contend that they are eager to have their programs

evaluated (Reid, 2004), but critics state that they operate as "loose cannons" in school districts

across the country with no accountability ("Beyond the pale: SES provisions emerge as 'loose

cannon' reform," 2006).

SES Coordination with Classroom Curriculum

Concerns over the lack of accountability also center on the lack of requirements to

coordinate SES with classroom curriculum or for providers to communicate with school

personnel. SES provides no procedure for providers to receive academic information from

teachers or vice versa. Critics contend that this lack of coordination between the school and SES

providers reverses a district's approach to a coordinated and comprehensive reform of Title I

schools. Some district officials have expressed concern that the curriculum of SES providers was

at odds with their efforts to create a complete academic program for students in these schools.









They feel that SES provisions weaken the ability of their schools to develop a consistent

instructional program by requiring tutoring to be offered outside the school day, further

discouraging coordination of the SES provider and classroom teacher ("Beyond the pale: SES

provisions emerge as 'loose cannon' reform," 2006; Sunderman, 2006). In Tennessee, many

teachers reported that they were not even aware that their students were being tutored by SES

providers, which meant that there was no collaboration regarding student academic needs.

Teachers also expressed frustration at the lack of collaboration with providers in developing

student learning plans (Potter, Ross, Paek, & McKay, 2006). Experts stated that academic gains

would be stronger if providers would collaborate with the students' teachers. They also contend

that SES will not meet its potential as long as the SES curriculum is disconnected from what is

being taught in the classroom (Viadero, 2007). Although there is no research that supports the

methodology behind SES, there is research that suggests that students benefit in schools that

support a comprehensive approach to educating students. This includes Title I curriculum that is

aligned with the regular curriculum and provides services that support the core instruction

(Sunderman & Kim, 2004).

The curriculum used during tutoring is up to the discretion of the SES provider and to-

date almost nothing is known about what students are actually being taught during tutoring

sessions beyond what is listed in their state application, on their website, or in their marketing

materials. Providers most commonly describe their curriculum as "literacy skills" and "problem

solving skills." While NCLB encourages SES providers to align their curriculum with state

standards, it clearly forbids states and districts from attempting to influence a provider to

implement a certain curriculum (Burch, 2007).









There is also speculation that SES decreases accountability by focusing on short term

individual student achievement rather than focusing on a broad range of school level outcomes

based on state standards. The focus is narrowed further by not only serving individual students,

but serving only students who request the free tutoring services (Sunderman, 2006).

SES Student-Teacher Ratio

The student-teacher ratio is established by SES providers and can vary significantly by

provider, district, and location (Burch, 2007). Although research strongly suggests that the

greatest academic gains can be obtained from one-to-one tutoring, current state and district

evaluations have found that most SES programs are significantly larger (Baker et al., 2006;

Burch, 2007; Elbaum et al., 2000; Juel, 1996; Lauer et al., 2004; Lauer, 2006; Wasik, 1998). The

Education Industry Association reports that SES is most commonly offered in small groups of

less than 10 students, but studies have shown that some providers have ratios of 1:10 or 1:12.

Providers with higher enrollment rates tend to have larger groups sizes of 1:8 to 1:10, while

providers with a smaller enrollment tend to tutor in groups of 1:1 to 1:3 (Burch, 2007).

Another noteworthy problem is that by funding SES with the 20% setaside of Title I

funds to pay providers, the allocations to schools have been drastically reduced. Rather than use

these funds to focus on school wide improvement tied to state standards, SES uses these funds to

focus on individual student academic gains for the students whose parents elect this free tutoring

program (Sunderman, 2006).

Existing Research on SES

In 2004, The Center for Education Policy conducted a study of the implementation of SES.

Six states and nine districts with SES implementation experience were selected. The study

consisted of telephone surveys to state administrators in charge of implementing SES statewide

and site visits to the nine districts to interview district staff, visit schools, conduct teacher and









parent focus groups, and hold personal interviews. This study reported that although parents had

been satisfied with the services delivered to their children, they wanted the services to start

sooner. Parents were concerned that their children were not receiving the maximum benefit of

the tutoring due to the amount of time it took to actually start tutoring after they had chosen a

provider. In some districts several weeks passed before students started receiving tutoring. The

study also stated that it was too early to assess the effectiveness of SES on student academic

achievement and that states needed guidance on how to assess SES effectiveness (Anderson &

Weiner, 2004).

In 2005, the U. S. Department of Education commissioned a follow-up study to the 2004

Center for Education Policy study. Once again, six states and nine districts were included in this

case study of SES. However, four of the nine districts included in the original study were no

longer providing SES because they no longer had schools in their second or higher year of school

improvement. To compensate for these four districts, four new districts in two additional states

were added. This study, conducted by Anderson & Laguarda (2005), noted a lack of coordination

between the SES provider and the teacher. Many teachers in the study reported that they did not

know which of their students were participating in the SES program and felt that collaboration

with the tutors would help their students. Teachers and parents also reported that they did not

receive progress reports on their students, although providers of these same students reported that

they sent regular progress reports to them. Parents also objected to the instructional approach that

providers took in teaching their children. Additionally, these parents stated that they were

disappointed with the quality of the tutoring provided and felt their children had made no

academic gain in reading and math. They also questioned whether just a few hours of tutoring

could actually make a difference for their children. This U. S. Department of Education case









study concluded that, as a result of the lack of evaluation of SES, very few providers have been

removed from state lists. In a few instances, providers were removed for financial irregularities

or because they did not offer the specified tutoring services to their students, but there were no

providers removed from state-approved lists because the state had determined that the quality of

their tutoring program was not adequate. Burch (2007) reported that 15 of the 30 states (50%)

who responded to a survey conducted through The Education Public Interest Center stated that

their state actually removed a provider from the state list because of a contract infraction or a

lack of effectiveness in the SES services provided.

Minneapolis Public Schools' SES Effectiveness

In 2005, Minneapolis Public Schools (MPS) conducted an internal SES study on the

effectiveness of the program in their district. Dr. David Heistad of the MPS Research,

Evaluation, and Assessment Office conducted this evaluation because he felt the state was ill-

equipped to conduct a controlled study. He stated that it may be incumbent upon large urban

districts to conduct their own evaluations of the SES program in an effort to provide feedback to

its stakeholders, including schools, parents, SES providers, and the Minnesota Department of

Education. Dr. Heistad conducted two different studies.

The first compared reading achievement gains for students in each SES program on the

Northwest Achievement Levels Tests (NALT). The second study matched students on the NALT

pre-tests and student demographic characteristics and calculated achievement on the Minnesota

Comprehensive Assessments (MCA) for the largest SES provider with a control group of

students who received no tutoring. Results of the first study found no statistical significant

difference among SES providers on NALT annual reading gains. No provider with at least 10

students receiving tutoring came close to averaging 100% of expected scale score growth based

on the NALT norms. For example, Education Station tutored 561 students who averaged 71% of









a year's growth, while 92 students tutored by Newton Learning Services averaged 67% of a

year's growth. Students in grades 3 and 7 who did not receive SES achieved at a higher level

than students they were paired with who received SES from Catapult Learning, outscoring them

by 19 points and 6 points, respectively. On the other hand, Catapult's 5th grade students

outscored their matched sample by 4 points. The achievement differences in both instances were

not statistically significant. Overall, the average growth for students in SES programs was 66%

of the national norm growth for all students (Burch, 2007; Heistad, 2006).

Results from the second study found no statistically significant difference in MCA reading

achievement for students who received tutoring from their largest SES provider compared to the

control group who received no tutoring. Dr. Heistad stated that the results of the second study

were disappointing when given the fact that 564 students who received tutoring from their largest

SES provider spent an additional 37 hours of instruction in the SES program. The report

concluded by saying that the SES program must be redesigned to better serve the needs of

Minneapolis students.

Chicago Public Schools' SES Effectiveness

Another evaluation of SES tutoring programs was completed in 2005 by the Chicago

Public Schools. This evaluation was done to determine if students who received SES had made

sufficient academic gains. Test score data from the Iowa Test of Basic Skills from 2003-04 and

2004-05 were analyzed and it was determined that students who received at least 40 hours of

tutoring in grades 4 8 achieved higher gain scores than students who did not receive SES. It

was also determined that students who received 40 total hours of tutoring achieved higher math

and reading scores than students who received less than 40 hours (Chicago Public Schools,

2005).









SES was again evaluated in the Chicago Public Schools in 2006. The results showed that

tutored elementary students made only slightly higher gains in reading than students who were

eligible for tutoring, but did not receive SES. In the 2005-06 school year, SES was offered in 324

schools by 41 private firms who were approved to provide SES by the Illinois Department of

Education. Chicago Public Schools spent $50 million for 56,000 students who received SES and

officials questioned whether the tutoring services were worth the money. Officials stated they

wanted to see higher achievement gains for the investment that was made (Grossman, 2007).

A policy brief from the Education Public Interest Center (EPIC) at Arizona State

University stated that although the studies from Minneapolis and Chicago made important

contributions to the evaluation of SES, they both contained methodological limitations that left

many aspects of the program unevaluated. In the Chicago study, the evaluation failed to consider

differences among student populations. It was also stated that the higher gain scores may have

resulted from an academic program other than SES tutoring. Selection bias could also account

for the higher gain scores. For example, parents who enroll their students in SES programs tend

to be better educated parents and take a more active role in their child's education. EPIC

contends that since these possibilities are not considered, the evaluation results are open to

question (Burch, 2007).

EPIC cited similar limitations in the design of the Minneapolis study. Minneapolis

compared their students with a nationally normed population, when in fact the Minneapolis

students were disproportionately low income, with 73% of the students classified as minority

students, with 76% being black, Hispanic, American-Indian or Asian. Therefore, while the study

compared its results to a national norm, significant differences in the two populations make the

findings questionable (Burch, 2007).









Los Angeles Unified School District's SES Effectiveness

In February of 2007, Los Angeles Unified School District released a study wherein they

duplicated a January 2006 study of the effectiveness of the 2004-05 SES program on increasing

the academic achievement of participating students. The January 2006 study found that SES had

no significant impact on student achievement on the California Standards Test in language arts

and math. The 2007 study analyzed the academic achievement of the 14,759 students who

participated in SES in the 2005-06 school year. The results of the study showed that students

who attended SES had a statistically higher, yet non-significant increase in their academic

achievement than did students who qualified for SES but did not participate. The SES program

was more effective for elementary students than secondary students and some providers had a

greater effect on increasing student academic achievement, while others had no effect (Rickles &

Barnhart, 2007).

Tennessee's SES Effectiveness Study

The effectiveness of SES was assessed statewide in Tennessee in a study completed by the

Center for Research in Educational Policy (CREP) in March 2007. The goal of this study was to

evaluate the academic achievement of students in grades 4 through 8 who received SES in the six

school districts during the 2005-06 school year that were required to offer these services. The

results of the study by CREP yielded no statistically reliable effects for increased student

achievement from the 33 SES providers included in the study. The results show no statistically

significant differences between students who received tutoring and those who did not. The

evaluation of two SES providers in this study resulted in negative effects on academic

achievement for students who enrolled in their programs (Potter et al., 2006).









Large Urban Districts' SES Effectiveness

The U. S. Department of Education released a report in June 2007, where data from the

2004-05 school year were analyzed from nine large urban school districts that had a large

number of students participating in the two choice options. The nine districts were Baltimore,

Chicago, Denver, Long Beach, Los Angeles, Palm Beach, Philadelphia, San Diego, and

Washington, DC. The first of the two choice options allowed parents to transfer their children to

another school in the district that had not been identified for school improvement and the second

choice option gives parents the opportunity to enroll their children in SES. Of the nine districts

included in this study, two districts were eliminated in the SES academic achievement portion

because they had less than 100 students participating in the tutoring program. Across the

remaining seven districts, students who participated in SES scored better in reading and math in

the first year and the higher academic achievement was deemed statistically significant. Students

also scored even higher in the subsequent year of SES (Zimmer, Gill, Razquin, Booker, &

Lockwood, 2007). Zimmer et al. (2007) stated that these findings were based on a small number

of school districts that are not nationally representative, and the results should not be viewed as

representative of the effectiveness of SES nationally. The results are important because this study

is one of the first to analyze the effectiveness of SES in multiple districts.

Contradictory evidence was presented in a study for the Los Angeles Unified School

District. Rickles & Barnhart (2007) stated that, in the 2004-05 school year, they found no

significant impact on the achievement of Los Angeles Unified School District SES students on

the California state test. Rickles & Barnhart's report contradicts the Zimmer et al. (2007) report

where it was stated that, as one of the nine districts that was included in their study, Los Angeles

Unified School District students who participated in SES made statistically significant academic

gains in reading and math in the 2004 -05 school year.









SES Evaluation Progress

Although states report that systems have been developed and implemented to evaluate

and monitor SES providers, as of 2005, 15 states had not established any monitoring process, 25

states had not established any standards for evaluating provider effectiveness, and none had

finalized their evaluation process. Seventeen (17) states plan to use student achievement on state

assessments as their evaluation tool for SES providers, although only one state planned to use a

control group. The most common evaluation method proposed by 25 states was to survey

districts about provider effectiveness, while 18 states plan on using providers' reports on student

academic progress (Stullich et al., 2006). In a policy brief from EPIC at Arizona State

University, 15 of the 30 states that responded to a survey stated that they did not use any form of

test data to evaluate SES provider quality, but instead relied on annual site visits (Burch, 2007).

Several states are making progress on evaluating SES providers. Louisiana has made

progress on its SES evaluation by using test scores from state assessments to monitor the

effectiveness of individual SES providers on student achievement. To accomplish this task

Louisiana developed a data system for all of its programs that take place outside the regular

school day. This data system communicates with the district's entire student data system and

compares the state test scores of students in after-school programs with the scores of those who

are not (Gewertz, 2005). Illinois, Ohio and Florida also were scheduled to implement their SES

evaluation program in the summer of 2006 (Pines, 2006). Illinois and New Jersey reported to the

United States Government Accountability Office that they were in the process of improving their

data collection system to effectively capture and examine needed data to determine SES provider

effectiveness. Several states said that while they have collected needed data to evaluate

providers, they have not completed the evaluations. Other states reported that additional federal

guidance on implementing effective evaluation models was needed because developing and









implementing effective evaluation models was time-consuming and costly (U. S. Government

Accountability Office, 2006).

Although state monitoring of SES implementation has been limited, states reported

making gains in monitoring both districts and providers in 2005-06. Districts have also increased

monitoring efforts of SES providers serving their students. New Mexico and Tennessee were the

only two states that had final or draft SES evaluation reports that attempt to assess the

effectiveness of SES providers on student academic achievement, but these evaluations did not

provide conclusive results. Although states are required to remove providers from approved lists

if they fail to increase student achievement for two years, most states have failed to implement

this section ofNCLB (U. S. Government Accountability Office, 2006).

The Center on Education Policy (CEP) published a report in 2007 on the status of state

implementation of SES. In their annual survey regarding NCLB implementation, CEP included

several questions regarding the state's role in overseeing SES. Two key findings center on a

state's ability to monitor SES providers effectively. It was found that 38 states were unable to

comprehensively monitor the quality and effectiveness of providers and only 10 states reported

that they adequately monitor SES. States reported that the biggest barriers to effective

monitoring were a lack of staff and inadequate funding at the district level to administer the

program (Minnici & Bartley, 2007).

The Education Industry Association (2005) has attempted to make SES providers

themselves responsible for monitoring their tutoring programs by creating high ethical standards

that its over 500 members have adopted as a code of professional conduct. The Supplemental

Educational Services Code of Conduct (2005) states that the importance of SES activities and the

complicated interactions of SES make it vital that members adhere to the highest standards of









professional conduct and ethics. This voluntary Code of Conduct describes key organizational

behaviors and policies that are to direct SES providers. It includes: (a) guidelines on how

providers should conduct business and fulfill responsibilities, (b) directives to consistently

implement the NCLB Supplemental Educational Services provisions and promote full access to

SES services, and (c) describes best practices for implementing SES. The Supplemental

Educational Services Code of Conduct is now a part of the state application process in several

states including Connecticut, New York, New Jersey, Maryland, Illinois, Ohio, Florida, Georgia,

and New Mexico ("Code of Professional Conduct," 2006).

Challenges in Evaluating SES

Many challenges are presented in the SES literature, but there are few suggestions to

guide states and districts on evaluating the benefits of SES, particularly in improving the

education of low-income and minority students who are to benefit from this program

(Sunderman, 2006). Three-fourths of states reported that they are experiencing challenges

evaluating SES, including designing evaluations to determine the academic gain of students,

having the personnel and funds to analyze data, and developing data systems to monitor SES

information (U. S. Government Accountability Office, 2006).

Due to the fact that states do not have the capacity or funding to monitor and evaluate SES

providers, many students end up in sub-standard programs (Rees, 2006). In March of 2006, CEP

published their fourth annual report, From the Capital to the Classroom: Year 4 of the No Child

Left BehindAct. Their findings were based on a survey of all 50 states, a nationally

representative survey of 299 school districts, case studies of 38 geographically diverse districts

and 42 schools, three national forums, and six special analyses of critical issues in implementing

NCLB. According to survey results, states consider the greatest challenge to implementing SES

is monitoring and evaluating the quality and effectiveness of the tutoring services provided.









Forty-one (41) states and 51% of school districts called this a moderate to serious challenge.

Current federal regulations restrict the ability of school districts to establish rules for SES

providers, but districts ultimately are responsible for allocating federal funds to providers and

increasing the academic performance of SES students. In response to these expectations, CEP

has made recommendations to help NCLB work better. One of these recommendations calls for

states and school districts to be given sufficient resources and authority to evaluate and monitor

SES providers' effectiveness in raising student achievement (Rentner et al., 2006).

Forty-nine (49) states reported that they used the criteria required by NCLB and federal

guidance to review and approve SES providers. These criteria include the assurances that

providers are financially sound, have a record of effectiveness, utilize research-based strategies,

provide services that are consistent with district instruction, and adhere to health, safety, and

civil rights laws (Minnici & Bartley, 2007). States also reported that the effectiveness of SES is

jeopardized because the federal government pushes states to increase the pool of providers with

little evidence of program effectiveness. Districts have raised additional concerns about what

they consider to be unethical practices some providers have used to attract students to their

tutoring program (Sunderman, 2006). Providers have used recruiting practices to entice parents

to enroll students in their program that are unrelated to the quality of their services. In Chicago,

parents were given over 40 SES providers from which to choose from; some of these providers

attempted to influence parents by offering cash payments, video game consoles, or other

enticements for enrolling their children in the provider's program (Rees, 2006). The use of

incentives by SES providers had become such a problem in Florida that the practice was

specifically addressed in a state statute that was approved by the Florida legislators in 2006.









Florida State Statute 1008.331 Supplemental Education Services in Title I schools states the

following:

(1) INCENTIVES. A provider or school district may not provide incentives to entice a
student or a student's parent to choose a provider. After a provider has been chosen, the
student may be awarded incentives for performance or attendance, the total value of which
may not exceed $50 per student per year (The 2006 Florida State Statutes, 2006).

Recommendations to Strengthen and Evaluate SES

Nothing is gained by avoiding the question of the effectiveness of NCLB or naively

assuming that well-intentioned efforts will be sufficient. As noted by Michael Kirst (2004), a

veteran policy analyst, it took more than 10 years and many legislative and administrative

adjustments before the initial Title I program in the 1965 Elementary and Secondary Education

Act met the legal intent of Title I. This is typical of aggressive new federal programs. These new

programs rarely work well at first; instead they bring a plethora of unforeseen problems,

consequences, loopholes, and impractical features. In fact, NCLB is immensely more ambitious

than the original ESEA because its goals, methods, and reforms do not conform to current

educational practices in America's public schools. NCLB requires states and districts to

participate in unfamiliar, untried activities, such as SES. The important question is whether SES,

in its current form, has a realistic chance of being effective given time and practice or whether

the law is impractical and unrealistic. If the latter is true, then NCLB needs rethinking and

revision, not just persistence and adjustments (Hess & Finn, 2004a).

A wealth of recommendations to strengthen and evaluate SES are presented in existing

literature. Recommendations from Steven Pines, Executive Director of the Education Industry

Association (2006) include the following:

1. States should directly involve SES providers and the local education agency in the
development of evaluation policies and specific methodologies.









2. States should ensure the fairness of the system particularly when the local
education agency is an approved provider.

3. States should separate the effects of SES from other variables that might affect a
student's achievement.

4. States should use extreme care or avoid using standardized tests for purposes
other than for which they were originally deemed reliable and valid.

5. States should include data from providers' pre-and post-testing of SES students in
their overall evaluation of provider effectiveness.

Hess & Finn (2004) made recommendations to strengthen and improve the existing SES

program. These recommendations include the following:

1. Guidance must be provided to states to ensure sound and rigorous evaluations
including careful planning, time, and adequate funding.

2. Better federal data must be provided on how SES is being utilized and how it is
working. For example, data might include tracking the number of students
receiving services, or which providers are serving how many students. This
information is vital and could be tracked by the National Center for Education.

3. Districts must be given additional money to support the administration and
evaluation of SES programs.

4. School districts need to function as either administrators of SES or SES providers,
not both.

Sunderman & Kim (2004) recommend that the federal government terminate the SES

program as a mandated sanction of NCLB for poorly performing Title I schools, while

suggestions made by the United States Government Accountability Office (2006) call for states

to be provided with needed assistance regarding methods for the evaluation of SES, including

additional and clearer guidance. States also have requested a forum to share SES best practices

with peers, such as meetings and conferences that could be facilitated by the federal government.

There is also a request for data collection on the amount of Title I funds spent on SES by districts

(U. S. Government Accountability Office, 2006).









Summary

NCLB legislation requires states to ensure that Supplemental Educational Services are of

high quality, research based, and designed to help eligible children attain proficiency in meeting

the state's academic achievement standards, yet the federal government has not partnered with

states to develop an adequate evaluation process.

Presently, $2.5 billion is being diverted from Title I funds annually to pay for this free

tutoring program with virtually no accountability. While states are required to remove providers

from their approved lists if they fail to increase student achievement for two years, states have

failed to implement this section of NCLB. The few states that are evaluating SES providers have

not yet produced reports that provide a decisive evaluation of their effect on student academic

gains. States also have expressed concern that additional federal guidance on implementing

effective evaluation models is needed because developing and implementing evaluation models

is time consuming and costly. In fact, 75% of states reported that they are experiencing

challenges evaluating SES, including designing evaluations to determine the academic gains of

students, having the personnel and funds to analyze data, and developing data systems to monitor

SES information (U. S. Government Accountability Office, 2006).

Current literature provides evidence that the Supplemental Educational Services provision

of NCLB has been in effect nationwide for several years with minimal and conflicting

evaluations of its effectiveness on increasing student achievement despite the NCLB requirement

for evaluation. The literature echoes a resounding demand to evaluate the effectiveness of SES to

determine the level of academic gain of Title I students, if any, that this program facilitates. With

billions of dollars set aside annually to fund the SES program, it is critical that it be evaluated

and adjusted, if necessary.









Table 2-1. Quality Indicators of Effective Tutoring
Indicator 1: The student-teacher ratio should be kept low. A 1:1 or 2:1 ratio is suggested for a
successful tutoring program.


Indicator 2:


Indicator 3:


Indicator 4:


Indicator 5:


Collaboration between the child's classroom teacher and the tutor has been
identified as an important aspect of effective tutoring.

Experienced, certified teachers make the best tutors. Teachers make effective
tutors because they are specifically educated to teach.

Extensive training of tutors produces the most positive effects. Training must be
ongoing and includes observing tutors and students during tutoring sessions.

A long term commitment sustained over the entire school year with the same tutor
working with the same student produces positive results. Tutoring should be
intensive and consistent.









CHAPTER 3
STUDY OVERVIEW

With the reauthorization of the Elementary and Secondary Act of 1965, the No Child Left

Behind Act of 2001 initiated unprecedented educational reform to ensure that all children have

the opportunity to obtain a high-quality education and reach proficiency on state standards and

assessments by increasing the accountability of states and districts (Stullich et al., 2006).

Supplemental Educational Services is one of the elements of this historic educational legislation

and is defined as academic activities provided outside normal school hours and typically offered

as an after-school tutoring program. The legislated goal of the SES program is to increase the

academic performance of low-income students who attend Title I schools that have not made

adequate yearly progress for three or more years. An SES program typically includes tutoring in

math, reading, and/or language arts, but specifically prohibits homework help (Smole, 2004).

NCLB requires states to ensure that Supplemental Educational Services are of high quality,

research based, and designed to help eligible children increase their academic achievement to

attain proficiency in meeting the state's academic achievement standards.

This free tutoring program is offered by SES providers who include nonprofit, for-profit,

and faith-based organizations, as well as school districts, charter schools, and private schools that

have been approved by state educational agencies (Kasmin & Farmer, 2006). SES is a parent-

driven program wherein providers are selected by parents from a state-approved list. SES

providers remain on this approved list unless they fail to demonstrate student academic gains for

two consecutive years. In order to demonstrate academic gain, providers are required to be

evaluated annually by the state educational agency. This evaluation is the specific responsibility

of the state, but currently no definitive evaluation has been completed on the effectiveness of

SES on raising student academic achievement in any state (Smole, 2004).









Annually as much as $2.5 billion is setaside nationwide to provide SES in Title I schools

for a program that has little or no research to prove its effectiveness in increasing academic

growth for enrolled students (Ascher, 2006; Cohen, 2006). While states are required to remove

providers from their approved lists if they fail to increase student achievement for two years,

states have failed to implement this section of NCLB. According to the U. S. Government

Accountability Office (2006) the few states that are evaluating SES providers have not generated

reports that provide a thorough evaluation of their effect on increasing academic achievement for

Title I students.

States have expressed concern that additional guidance is needed from the United States

Department of Education to construct and implement adequate evaluation models (U. S.

Government Accountability Office, 2006). At this time the federal government has not partnered

with states to develop an adequate evaluation process to ensure that these requirements are being

met (U. S. Government Accountability Office, 2006). Furthermore, states contend that

development and implementation of evaluation models is time consuming and costly. In fact,

75% of states reported that they are experiencing challenges evaluating SES, including designing

evaluations to determine the academic gains of students, having the expertise and funds to

analyze data, and developing data systems to examine SES information (U. S. Government

Accountability Office, 2006).

Purpose of the Study

The purpose of this analysis was to assess the effectiveness of the SES program in

Florida Title I urban district schools that were required to offer this program during the 2006-

2007 school year.









To assess the effectiveness of SES, the following research questions will be addressed:

1. Do students who participated in Supplemental Educational Services exclusively in math
for 10 or more hours demonstrate higher developmental scale score gains, as determined by
the FCAT, compared to students in a control group?

2. Do students who participated in Supplemental Educational Services exclusively in reading
for 10 or more hours demonstrate higher developmental scale score gains, as determined by
the FCAT, compared to students in a control group?

3. Do students who participated in Supplemental Educational Services in both math and
reading for 10 or more hours demonstrate higher developmental scale score gains, as
determined by the FCAT, compared to students in a control group?

4. Do students tutored by the three major SES providers in each district (i.e., those that
provided tutoring for the largest number of students) for 10 or more hours demonstrate
higher developmental scale score gains, as determined by the FCAT, compared to students
in a control group?
Research Hypotheses

The study was guided by the following research hypotheses:

Null Hypothesis 1. There is no statistically significant difference between 2006 FCAT

developmental scale scores as the pre-test and 2007 FCAT developmental scale scores as the

post-test for students in Title I schools receiving at least 10 hours of tutoring in math.

Null Hypothesis 2. There is no statistically significant difference between 2006 FCAT

developmental scale scores as the pre-test and 2007 FCAT developmental scale scores as the

post-test for students in Title I schools receiving at least 10 hours of tutoring in reading.

Null Hypothesis 3. There is no statistically significant difference between 2006 FCAT

developmental scale scores as the pre-test and 2007 FCAT developmental scale scores as the

post-test for students in Title I schools receiving at least 10 hours of tutoring in math and

reading.

Null Hypothesis 4. There is no statistically significant difference between 2006 FCAT

developmental scale scores as the pre-test and 2007 FCAT developmental scale scores as the









post-test for students receiving at least 10 hours of tutoring from the three SES providers who

tutored the largest number of students in each district.

Overview of the Method

This study utilized existing FCAT scores to assess the effectiveness of SES in grades 4 -

10 for six large urban Florida school districts in an effort to align this study with the anticipated

evaluation of SES by the State of Florida. A memo dated December 21, 2007, stated that the

Florida Department of Education will use FCAT scores in the 2006-07 evaluation of companies

providing SES to students in grades 4-10 (L. Dukes, personal communication, December 21,

2007).

Included in this study are six of the largest urban Florida school districts. The six districts

have been randomly assigned numbers one through six for identification purposes. This study

was limited to these large urban Florida school districts due to the unavailability of statewide

data.

2006-07 Participating District Statistics

District One has a total student enrollment of 362,050. The number of students who

receive free or reduced price lunch is 221,229 or 61% of the student population. Title I funds are

provided to schools with a 63% or higher rate of students eligible for free or reduced price lunch.

Of the 438 schools in the district, 47% (n = 204) are Title I schools that are required to offer

SES. The ethnic makeup of the student population district wide is comprised of 61% Hispanic,

27% Black, 9% White, and 3% Other, which includes American Indian, Asian, and Multiracial

categories. In grades 9-12, there are a total of 8,526 dropouts which represents a 6.7% annual

dropout rate. The four-year graduation rate in this district is 59.2%. The district employs 39,603

full time employees, including administrators, teachers, and support staff.









There are a total of 193,681 students enrolled in District Two. Free or reduced price lunch

is provided for 49% (n = 95,726) of the student population. In District Two, Title I funding is

provided to schools with a 57% or higher rate of students eligible for free or reduced price lunch.

This district is comprised of 279 schools of which 123 are Title I schools offering SES. The

district ethnic breakdown of the student population consists of 44% White, 22% Black, 26%

Hispanic, and 8% Other, which includes American Indian, Asian, and Multiracial categories. The

annual dropout rate in grades 9-12 is 2.1% (n =1,330), with a four year graduation rate of 77.3%.

A total of 17,272 full time employees are employed by this district, including administrators,

teachers, and support staff.

District Three has a total enrollment of 175,593 students. Students receiving free or

reduced price lunch total 81,245 students or 46% of the student population. Title I funds are

provided to schools with a 75% or higher rate of students eligible for free or reduced price lunch

This district is composed of 222 schools of which 39 schools or 18% are Title I schools offering

SES. The district wide ethnic distribution of students in District Three is 36% White, 28% Black,

29% Hispanic, and 7% Other, which includes American Indian, Asian, and Multiracial

categories. The annual dropout rate is 1.9% (n = 1,165) students in grades 9-12. The four year

graduation rate is 72.2%. There is a total of 22,361 full time staff members employed in the

district, including administrators, teachers, and support staff.

Student enrollment totals 174,861 students in District Four. Of the total enrollment, 72,947

or 42% of the district's students receive free or reduced price lunch. District Four provides Title I

funding to schools with a 44% or higher rate of students eligible for free or reduced price lunch

There are 265 schools in this district with 24% (n = 64) being Title I Schools offering SES. The

ethnic breakdown of the district enrollment is 42% White, 29% Black, 22% Hispanic, and 7%









Other, which includes American Indian, Asian, and Multiracial categories. The annual dropout

rate of students in District Four, grades 9-12, is 3% (n =1,715). The four year graduation rate in

this district is 69.3%. There are 19,696 full time staff members employed by the district,

including administrators, teachers, and support staff.

District Five has a total enrollment of 126,648 students. Free and reduced price lunch is

received by 52,709 students or 42% of the student enrollment in the district. This district

provides Title I funding to schools with a 62% or higher rate of students eligible for free or

reduced price lunch. Of the 182 schools in the district, 41 or 23% are Title I schools that are

required to offer SES. The ethnic composition of the student population district-wide is

comprised of 44% White, 43% Black, 6% Hispanic, and 7% Other, which includes American

Indian, Asian, and Multiracial categories. The annual dropout rate of students in grades 9-12 in

District Five is 6.6% (n = 2,780) with a four year graduation rate of 60.5%. A total of 12,958 full

time staff members are employed by the district, including administrators, teachers, and support

staff.

The final school district included in this evaluation, District Six, has a total student

enrollment is 112,150 with 40% (n = 45,217) receiving free or reduced price lunch. In District

Six, Title I funding is provided to schools with a 43% or higher rate of students eligible for free

or reduced price lunch This district is comprised of 279 schools of which 32 are Title I schools

offering SES. The ethnic distribution of students in this district is 64% White, 19% Black, 9%

Hispanic, and 8% Other, which includes American Indian, Asian, and Multiracial categories. In

grades 9-12, there are a total of 1,304 dropouts, which represents a 3% annual dropout rate. The

four-year graduation rate is 67%. A total of 14,409 full time staff members are employed by

District Six, including administrators, teachers, and support staff









FCAT data requested from the six Florida school districts was used in a quasi-experimental

design in this study. In each district, the data was divided into two groups of Title I students

receiving free or reduced price lunch in grades 4 10. Both groups of students had FCAT scores

for 2006 and 2007 for math and reading and attended schools that offered SES. Group One, the

experimental group, was composed of students who received at least 10 hours of tutoring in the

SES program during the 2006-07 school year. Group Two, the control group, was composed of

students who were eligible for SES, but not enrolled in the program.

When submitting the requested data, districts were asked to include FCAT developmental

scale scores, as well as FCAT achievement levels. In addition, demographic information was

requested including gender, ethnicity, student ESE codes, SES provider codes, as well as the

total hours of tutoring in math and/or reading.

Prior to submitting the Institutional Review Board application to each of the six urban

Florida school districts included in this study, permission to conduct research was obtained from

the University of Florida Institutional Review Board. The study did not involve direct student

contact, but evaluated existing FCAT scores and student identification numbers were deleted.

Therefore, individual students were not identifiable and informed consent was not required.

The methodology included using developmental scale scores from the 2006 and 2007

FCAT administrations. Developmental scale scores were selected as the unit of measure because

they are an effective tool in tracking student performance across grade levels. As a student

moves from grade to grade, his/her performance can be tracked and compared to the

performance of other students. Developmental scale scores enable the yearly progress of

individual students to be reported by the change in their scores (Florida Department of

Education, 2007b).









Procedures

The developmental scale score from the 2006 and 2007 FCAT administrations were used

as pre- and post-tests. A quasi-experimental design was utilized wherein the experimental group

participated in the SES program and students in the control group did not participate in the

tutoring program.

Students who did not participate in both the 2006 and 2007 administration of the FCAT

were eliminated from the study. In addition, the data file was scrubbed by eliminating students

who did not receive a minimum of 10 hours of tutoring. A minimum of 10 hours of tutoring was

consistent with criteria set by the Florida Department of Education's evaluation proposal that

only students who receive a minimum of 10 hours of tutoring be included. Students who had

received SES were placed in the experimental group. The control group was randomly selected

utilizing the Statistical Package for Social Sciences (SPSS) software and was re-sampled for

each analysis. In addition, the control group was matched by grade with students in the

experimental group. The control group was qualified to receive SES, but the parents of these

students chose not to enroll them.

Next, the experimental group was divided into three categories: students who received

tutoring in math only, students who received tutoring in reading only, and students who received

tutoring in both math and reading. The control group was subsequently divided into three

matching groups.

The null hypotheses in this study were tested using the ANOVA test of means to determine

if there was a statistically significant increase in FCAT developmental scale scores from the

2006 to 2007 administration. Computations were performed using SPSS software and tested at a

.05 level of significance. The effectiveness of SES was reported in three areas: reading only,









math only, and reading and math. SES effectiveness was also reported by individual SES

provider within each district.

Summary

This chapter provided an overview of the study by discussing the need to evaluate SES to

determine its success on improving the academic achievement of students receiving tutoring. An

overview of the method discussed the criteria used to select the six Florida urban school districts

included in this study, as well as the selection protocol used to form the control and experimental

groups. Finally, a discussion of the procedures was presented wherein scrubbing the data and the

formation of categories in the experimental and control group was explained as a preface to the

reporting of data in Chapter 4.










Table 3-1. Participating District Statistics
Statistics District District District District District District
One Two Three Four Five Six
Total
Enrollment 362,050 193,681 175,593 174,861 126,648 112,150

Students with
Free/Reduced
Lunch 221,229 95,726 81,245 72,947 52,709 45,217

Poverty rate for
Title I funding 63% 57% 75% 44% 62% 43%

Total
Schools 438 279 222 265 182 279

SES
Schools 204 123 39 64 41 32

Dropout
Rate 6.7% 2.1% 1.9% 3% 6.6% 3%

Graduation
Rate 59.2% 77.3% 72.2% 69.3% 60.5% 67%

Full Time
Staff 39,603 17,272 22,361 19,696 12,958 14,409









CHAPTER 4
RESEARCH FINDINGS

The purpose of this analysis was to assess the effectiveness of the SES program in

Florida urban school districts that were required to offer this tutoring program during the 2006-

07 school year. To determine effectiveness, FCAT data was obtained from six of the largest

Florida urban school districts. Initially the study was designed to include the seven largest urban

Florida school districts, but was revised to six school districts when obstacles in obtaining the

needed data arose from the seventh district. In reporting the results of this study, participating

school districts are not identified by district name. Districts were randomly numbered one

through six for identification purposes in this study.

This study utilized FCAT data from these districts in a quasi-experimental design. FCAT

developmental scale scores for 2006 and 2007 were used as pre- and post-tests. In each district,

the FCAT data from the Title I SES schools was divided into two groups receiving free or

reduced price lunch in grades 4 10. Both groups were required to have FCAT scores from the

2006 and 2007 administration for reading and math. Group one, the experimental group, was

composed of students who received at least 10 hours of tutoring in the 2006-07 school year.

Group two, the randomly selected control group, was composed of students who were eligible to

receive SES, but were not enrolled in the program.

Next, the experimental group was divided into three categories: students who received

tutoring in math only, students who received tutoring in reading only, and students who received

tutoring in math and reading. The control group was subsequently divided into three matching

groups. In the experimental group and the control group, each student's 2006 FCAT

developmental scale score was subtracted from their 2007 score and the net result was used in

the analysis.









The null hypotheses in this study were tested using a one-way analysis of variance design

to determine if there was an increase in FCAT developmental scale scores from the 2006 to 2007

administration for students who participated in SES. Computations were performed using SPSS

software testing at a .05 level of significance. The results reported in three areas: reading only,

math only, and reading and math. SES effectiveness was also reported by individual SES

providers in each district who served the largest number of students who received at least 10

hours of tutoring in math and/or reading.

Research Questions

The following research questions were addressed in the study:

1. Do students who participated in Supplemental Educational Services exclusively in math for
10 or more hours demonstrate higher developmental scale score gains, as determined by the
FCAT, compared to students in a control group?

2. Do students who participated in Supplemental Educational Services exclusively in reading
for 10 or more hours demonstrate higher developmental scale score gains, as determined by
the FCAT, compared to students in a control group?

3. Do students who participated in Supplemental Educational Services in both math and
reading for 10 or more hours demonstrate higher developmental scale score gains, as
determined by the FCAT, compared to students in a control group?

4. Do students tutored by the three major SES providers in each district (i.e., those that
provided tutoring for the largest number of students) for 10 or more hours demonstrate
higher developmental scale score gains, as determined by the FCAT, compared to students
in a control group?

Research Hypotheses

The study was guided by the following research hypotheses:

Null Hypothesis 1. There is no statistically significant difference between 2006 FCAT

developmental scale scores as the pre-test and 2007 FCAT developmental scale scores as the

post-test for students in Title I schools receiving at least 10 hours of tutoring in math.









Null Hypothesis 2. There is no statistically significant difference between 2006 FCAT

developmental scale scores as the pre-test and 2007 FCAT developmental scale scores as the

post-test for students in Title I schools receiving at least 10 hours of tutoring in reading.

Null Hypothesis 3. There is no statistically significant difference between 2006 FCAT

developmental scale scores as the pre-test and 2007 FCAT developmental scale scores as the

post-test for students in Title I schools receiving at least 10 hours of tutoring in math and

reading.

Null Hypothesis 4. There is no statistically significant difference between 2006 FCAT

developmental scale scores as the pre-test and 2007 FCAT developmental scale scores as the

post-test for students receiving at least 10 hours of tutoring from the three SES providers who

tutored the largest number of students in each district.

Analysis and Quantitative Results

FCAT data from the six districts participating in this study were combined to assess the

effectiveness of the SES program in raising student achievement. This FCAT data file contained

complete data by grade level for the 9,026 students who received SES during 2006-07 school

year. Table 4-1 reports the number of students represented at each grade level. Table 4-2 reports

the combined breakdown of students by grade level and school district. When assessing

academic gains for students, grade levels with less than 50 students were eliminated because it

was determined they did not adequately represent the student population. Grades 11 and 12 were

eliminated because of the small number of students represented in these grades (n= 17). As a

result, the experimental group used in this study contained 9,009 students each of whom received

at least 10 hours of tutoring in math and/or reading. Table 4-3 documents the number of state

approved SES providers who provided tutoring services within each district during the 2006-07









school year. Some providers served multiple districts; elimination of duplication resulted in a

total of 86 different providers assessed in this study.

The control group was comprised of students with similar demographic information as

represented in the experimental group. The control group population included 21,784 students

who attended Title I SES schools but did not participate in the SES tutoring program. Using the

SPSS software program, a random selection process was utilized to compile control groups that

matched the experimental groups. The control group was re-sampled for each analysis completed

in this study.

Research Question 1: Do students who participated in Supplemental Educational Services
exclusively in math for 10 or more hours demonstrate higher developmental scale score gains, as
determined by the FCAT, compared to students in a control group?

The math only group included 1,565 students in the experimental group and 1,565 students

in the corresponding control group. Table 4-4 illustrates the number of students in each grade

level per district that received tutoring in math. This group received a total of 24,590 hours of

math tutoring from 57 SES providers. The number of math tutoring hours provided per student

ranged from a minimum of 10 hours to a maximum of 56 hours. The developmental scale score

gain mean for the experimental group was 144.16 and 122.60 for the control group. Therefore, a

mean score difference of 21.56 was determined with the tutored group having the higher mean

score. Developmental scale scores gains ranged from a low of -828 to a high of 1046 for the

experimental group. In the control group, developmental scale scores gains ranged from a low of

-816 to a high of 1120.

Null Hypothesis 1: There is no statistically significant difference between 2006 FCAT
developmental scale scores as the pre-test and 2007 FCAT developmental scale scores as the
post-test for students in Title I schools receiving at least 10 hours of tutoring in math.









Results of the ANOVA test of the means for Null Hypothesis 1 (Table 4-7) found a

statistically significant difference between the math experimental group and the math control

group on the FCAT developmental scale score gains (f= 11.161; p = .001). Thus, Null

Hypothesis 1 stating that SES tutoring in math would have no statistically significant effect in

FCAT developmental scale scores was rejected.

Research Question 2: Do students who participated in Supplemental Educational Services
exclusively in reading for 10 or more hours demonstrate higher developmental scale score gains,
as determined by the FCAT, compared to students in a control group?

The reading only experimental group was comprised of 6,630 students who participated in

the reading tutoring program. The control group mirrored the grade levels and number of

students in the experimental group. Table 4-5 details the number of students per grade level and

district for the reading group. Students received 106,600 hours of tutoring in reading provided by

73 SES providers. The minimum number of per student tutoring hours in reading was 10 with the

maximum tutoring hours of 53. The developmental scale score gain mean for the experimental

group was 126.60 and the mean for the control group was 120.21 a mean score difference of 6.39

favoring the experimental group. Developmental scale score gains ranged from a low of -1250 to

a high of 1498 for the experimental group. The control group developmental scale scores gains

ranged from a low of -891 to a high of 1163.

Null Hypothesis 2: There is no statistically significant difference between 2006 FCAT
developmental scale scores as the pre-test and 2007 FCAT developmental scale scores as the
post-test for students in Title I schools receiving at least 10 hours of tutoring in reading.

Results of the ANOVA test of means (Table 4 8) for Null Hypothesis 2 found no

statistically significant difference between the FCAT developmental scale score gains of the

reading experimental group and the reading control group (f = 2.453; p = .117). As a result Null









Hypothesis 2, stating that there is no statistically significant difference between the 2006 and

2007 FCAT developmental scale scores for students tutoring in reading was accepted.

Research Question 3: Do students who participated in Supplemental Educational Services
in both math and reading for 10 or more hours demonstrate higher developmental scale score
gains, as determined by the FCAT, compared to students in a control group?

The reading and math experimental group was comprised of 1,084 students who received

tutoring in both subjects. The control group for this analysis also contained 1,084 students at the

same grade levels. Table 4-6 summarizes the number of students and the grade levels per district

for this group. These students received 6,488 hours of reading tutoring and 6,505 hours of math

tutoring from 38 SES providers. Individual students received a minimum of 10 and a maximum

of 36 hours of tutoring in reading. The same group of students received a minimum of 10 hours

and a maximum of 30 hours in math tutoring. The developmental scale score mean for the

experimental reading group was 126.95 and the control reading group was 122.29, a mean scale

score difference of 4.66 favoring the experimental reading group. The mean developmental scale

score gain for the experimental math group was 137.69 while the mean score gain for the control

math group was 128.44. The experimental math group produced a higher mean score gain

difference of 9.25. Developmental scale score gains ranged from a low in the reading

experimental group of-956 to a high of 1369. The math experimental group had developmental

scale score gains that varied from a low of -792 to a high of 1036. The control group

developmental scale scores in reading ranged from -823 to a high of 1231 and in math ranged

from a low of -816 to a high of 1120.

Null Hypothesis 3: There is no statistically significant difference between 2006 FCAT
developmental scale scores as the pre-test and 2007 FCAT developmental scale scores as the
post-test for students in Title I schools receiving at least 10 hours of tutoring in math and
reading.









The ANOVA for Null Hypothesis 3 (Table 4 9) resulted in a finding of no statistically

significant difference in mean scores between the experimental and control groups receiving

tutoring in reading and math (reading: f = .212; p = .646; math: f= 1.347; p = .246). The finding

of no statistically significant difference between the 2006 and 2007 FCAT developmental scale

score gains between the experimental and control groups resulted in acceptance of Null

Hypothesis 3.

Research Question 4: Do students tutored by the three major SES providers in each
district (i.e., those that provided tutoring for the largest number of students) for 10 or more hours
demonstrate higher developmental scale score gains, as determined by the FCAT, compared to
students in a control group?

An ANOVA was used to assess the effectiveness of individual SES providers that provided

tutoring in the six districts included in this study. The three SES providers that served the largest

number of students in each district were included in the analysis for a total of 18 SES providers.

Results are reported by district in Tables 4-10 through 4-15.

Of the 18 providers analyzed, three were found to have statistically significant results as

tested by the ANOVA. These SES providers were (a) Advanced Learning, (b) Rocket Learning,

and (c) Studentnest, Inc.

In District 2 (Table 4-11), Advanced Learning tutored 88 students in math and 97 students

in reading. Students received 1,857 hours of tutoring in reading and 1,782 in math. Reading

students received a minimum of 10 hours of tutoring and a maximum of 29. A minimum of 10

hours and a maximum of 34 hours was provided to students in math tutoring. The mean

developmental scale score for the math experimental group was 157.34 and the math control

group developmental scale score mean was 84.64. A mean score difference of 72.70 resulted

with the math experimental group having the higher mean score. The reading experimental group

in the analysis of Advanced Learning had a mean developmental scale score of 158.94 and the









control group had a mean score of 79.40 resulting in a 79.54 difference. Developmental scale

score gains in the experimental group ranged from a low in reading of -320 to a high of 483. In

math the scores ranged from a low of -547 to a high of 1433. The control group developmental

scale score gains ranged from a low in reading of -523 to a high of 445. In math the scores

ranged from a low of -681 to a high of 588.

In District 3 (Table 4-12), Rocket Learning provided tutoring for 47 students in reading.

This SES provider tutored students for a total of 1,120 hours. Students received a minimum of 10

hours of tutoring and a maximum of 34 hours. The mean developmental scale score for the

experimental group was 189.02 and the control group was 71.49, resulting in a difference in the

mean developmental scale scores of 117.53. In the experimental group developmental scale score

gains for Rocket Learning students ranged from a low -342 to a high of 818. Developmental

scale score gains for the control group ranged from a low -334 to a high of 357.

The final SES provider that had statistically significant ANOVA results was Studentnest in

District 4 (Table 4-13) who tutored 155 students in Math for a total of 3,151 hours. A minimum

of 10 hours of tutoring was provided to students with a maximum of 27 hours offered per

student. The control group had a mean developmental scale score of 96.14, and a mean of 151.13

was determined in the experimental group. Thus, a difference of 54.99 was found between the

experimental group and the control group. The analysis of Studentnest developmental scale score

gains in the experimental group resulted in a low of -543 to a high of 778. The control group

developmental scale score gains ranged from a low of -427 to a high of 696.

Null Hypothesis 4: There is no statistically significant difference between 2006 FCAT
developmental scale scores as the pre-test and 2007 FCAT developmental scale scores as the
post-test for students receiving at least 10 hours of tutoring from the three SES providers who
tutored the largest number of students in each district.









The ANOVA resulted in a finding of statistically significant difference for three of the SES

providers analyzed, but the remaining 15 SES providers resulted in a finding of no statistically

significant difference between 2006 FCAT developmental scale scores and 2007 FCAT

developmental scale scores. Tables 4-10 through 4-15 detail the statistical findings for the 18

SES providers analyzed. Null Hypothesis 4, which states that tutoring provided by the three

largest providers in each district would have no statistically significant results, was rejected for

the three providers who demonstrated statistically significant results: Advanced Learning,

Rocket Learning, and Studentnest, Inc. The ANOVA results for the remaining 15 SES providers

was retained for Null Hypothesis 4.

The ANOVA analysis for Advanced Learning resulted in a finding of a statistically

significant difference between the experimental group receiving tutoring in reading and math and

the matching control group on the developmental scale score gains on the FCAT reading (f=

4.688; p = .032) and math (f= 5.344; p = .022). This finding of a statistically significant

difference between the 2006 and 2007 FCAT developmental scale score gains in reading and

math resulted in rejection of Null Hypothesis 4 for Advanced Learning in District 2 (Table 4-11).

Results of the ANOVA (Table 4-12) for Rocket Learning in District 3 found a statistically

significant difference between the mean scores of the reading experimental group and control

group on FCAT developmental scale score gains (f= 7.193; p = .009). This finding resulted in

rejection of Null Hypothesis 4 for Rocket Learning in District 3.

The outcome of the ANOVA test of means (Table 4-13) for Studentnest in District 4 found

a statistically significant difference between the math experimental group and the math control

group on the FCAT developmental scale score gains (f = 8.237; p = .004). This finding resulted

in rejection of Null Hypothesis 4 for Studentnest in District 4.









Summary

The purpose of these analyses was to examine the effectiveness of the SES program in

Florida urban school districts' Title I schools that were required to offer the SES program in the

2006-07 school year. The ANOVA test of means was used to analyze groups of tutored students

that included students who were tutored in math only, reading only, and reading and math. Also

analyzed were the three SES providers who tutored the largest number of students in each

district. Of the 21 groups evaluated, 4 groups resulted in a statistically significant difference in

the mean scores of the experimental group and the control group. The analysis concluded that

students tutored in math only, as well as students tutored by Advanced Learning, Rocket

Learning, and Studentnest, Inc., significantly increased their academic achievement per the

developmental scale scores on the 2007 FCAT.

In Chapter 4 quantitative results of the study were presented. Chapter 5 will present a

discussion of the findings, conclusions and recommendations, as well as implications, future

research and summary.









Table 4-1. SES Students by Grade Level


Grade Level
4
5
6
7
8
9
10
11
12


Number of Students
3,188
2,854
1,312
710
720
121
104
12
5
TOTAL 9,026


Table 4-2. SES Students by District
Grade District District
One Two
4 1,458 585
5 1,158 503
6 712 171
7 404 79
8 427 123
9 44 15
10 44 20
11 3
12 2
Total 4,247 1,501


District
Three
27
262
104
30
14





437


N = 9,026


District
Four
527
428
302
182
146
62
40
9
3
1,699


District
Five
234
212
23
15
10





494


District
Six
357
291








648









Table 4-3. Active SES Providers by District
District District District District District District
One Two Three Four Five Six
Providers 35 22 15 28 20 20



Table 4-4. Total Students: Math Only Group by District and Grade
Grade District District District District District District
One Two Three Four Five Six


88
112
65
51
49


217
200
76
25
50


10
Total


N= 1,565









Table 4-5. Total Students: Reading Only Group by District and Grade
Grade District District District District District District
One Two Three Four Five Six


4
5
6
7
8
9
10
Total
N = 6,360


Table 4-6.
Grade


1,225
918
606
329
346
39
38
3501


366
302
95
54
73
10
15
915


26
229
95
25
12


387


238
204
132
73
62
46
28
783


226
165


192
166
7
10
8


Total Students: Reading and Math Group by District and Grade
District District District District District
One Two Three Four Five


District
Six


4 145 2 1 248 10 45
5 128 1 16 176 7 41
6 41 9 72 1
7 24 4 33
8 32 1 27 1
9 1 5
10 4 9
Total 375 3 31 570 19 86
N =1,084









Table 4-7. Analysis of Variance for Math Only Group


Subject df 1 df2 F p
Math 1 3128 11.1161 .001*
* Statistically significant




Table 4-8. Analysis of Variance for Reading Only Group
Subject df 1 df2 F p
Reading 1 12,718 2.453 .117



Table 4-9. Analysis of Variance for Reading and Math Tutoring Group
Subject df 1 df2 F p
Reading 1 2166 .212 .646
Math 1 2166 1.347 .246



Table 4-10. Analysis of Variance for District 1
SES provider Subject df 1 df 2 F p


Cool Kids
Rocket Learning
Rocket Learning
Education Station


Reading
Reading
Math
Reading


414
1536
118
2507


.000
.163
1.053
.219


.984
.687
.307
.640









Table 4-11. Analysis of Variance for District 2
SES provider Subject df 1


Academy of Success I
Academy of Success
Advanced Learning I
Advanced Learning
A+ Tutor U I
A+ Tutor U


leading
vlath
leading
vlath
leading
vlath


df2
206
206
192
174
274
224


F
.167
.004
4.688
5.344
.242
2.131


p
.684
.947
.032*
.022*
.623
.146


* Statistically Significant



Table 4-12. Analysis of Variance for District 3
SES provider Subject df 1 df 2 F p
Club Z! Reading 1 66 2.305 .134
Club Z! Math 1 107 3.656 .059
A+ Tutor U Reading 1 298 .256 .613
Rocket Learning Reading 1 92 7.193 .009*
* Statistically Significant



Table 4-13. Analysis of Variance for District 4
SES provider Subject df 1 df 2 F p
AlphaBEST Reading 1 534 .001 .978
AlphaBEST Math 1 534 1.592 .208
Huntington Learning Reading 1 314 1.985 .160
Studentnest Math 1 308 8.237 .004*
* Statistically Significant



Table 4-14. Analysis of Variance for District 5
SES provider Subject df 1 df 2 F p
"A" Wise Choice Reading 1 88 .644 .424
Huxtable Education Reading 1 116 .782 .378
A+ Tutor U Reading 1 164 .151 .698









Table 4-15. Analysis of Variance for District 6
SES provider Subject df 1 df 2 F p
ACES Reading 1 112 .103 .748
ACES Math 1 138 .170 .680
Achievia Reading 1 84 .323 .571
PCTA Reading 1 108 1.285 .260
PCTA Reading 1 108 1.329 .251









CHAPTER 5
DISCUSSION

Poverty and inequality in America's education system are barriers to opportunity and

progress. This is apparent in the achievement gaps between student groups with African-

American and Latino students continually lagging behind in academic achievement compared to

other students (Kennedy, 2005). The fact that only 29% of all fourth grade students performed at

or above the proficient level on the National Assessment of Educational Progress in reading in

2000 clearly indicated a need for nationwide education reformation and the federal government

enacted NCLB to implement this reform (U. S. Department of Education, 2002; U. S.

Department of Education, 2006).

The highest priority of NCLB is to give low achieving students the opportunity and ability

to meet high academic standards and reach proficiency on demanding state assessments. The

Title I program, one of the largest programs funded by NCLB, was designed to ensure that

students living in poverty have an equal opportunity to obtain a high quality education and

achieve the required academic growth to meet state standards and assessments. Title I provides

federal funding to help students who are either behind academically or are at risk of falling

behind academically.

NCLB introduced the SES program to Title I schools to provide additional academic

assistance (e.g., tutoring, remediation, and other educational interventions). The purpose of this

study was to assess the effectiveness of the SES program in Florida urban school districts' Title I

schools that were required to offer the program in the 2006-07 school year. Nationwide, each

Title I school that has not made AYP for three years is required to provide SES to students from

low-income families. This program is designed to increase the academic achievement of students

in eligible Title I schools. NCLB defines SES as additional academic assistance such as tutoring,









remediation, and other educational interventions, provided that these approaches are consistent

with the content and instruction used by the school district and are aligned with the state's

academic content standards (Kennedy, 2005). SES providers are required to demonstrate that the

tutoring provided increases student academic achievement in order to maintain their status as

approved and remain on the state-approved list (U. S. Department of Education, 2005).

Therefore, states are required to evaluate SES providers to determine their effect on increasing

student achievement (Smole, 2004).

While federal guidance places the responsibility of evaluating the effectiveness of the SES

program on the state, Florida has not evaluated this program since it began in the 2004-05 school

year. States are required to prohibit providers from offering services to students if they fail to

increase student achievement for two years, but Florida, as well as all other states, has neglected

to enact this section of the law.

Discussion of the Findings

This study included 9,009 students from 6 large urban Florida school districts in grades 4

through 10 who were tutored by 86 state-approved SES providers. These SES students received a

total of 113,148 hours of tutoring in reading and 31,128 hours of tutoring in math for a total of

144,276 hours of tutoring during the 2006-07 school year. Findings from this study demonstrated

that, although there was some increase in the academic achievement of students participating in

this program, the increase was limited to a relatively small number of students. Of the 21 tutoring

groups analyzed, only 4 groups demonstrated statistically significant academic gains on FCAT

developmental scale scores. These 4 groups include the math only tutoring group and 3

individual SES providers. Those providers are Advanced Learning, Rocket Learning, and

Studentnest, Inc. The results of the study are summarized in Tables 5-1 and 5-2.









The following research questions were addressed in the study.

1. Do students who participated in Supplemental Educational Services exclusively in math
for 10 or more hours demonstrate higher developmental scale score gains, as determined by the
FCAT, compared to students in a control group?

From an experimental group of 1,565 students, an ANOVA test of means evaluated the

developmental scale scores from the 2006 and 2007 FCAT to assess the academic gains of

students tutored in math. The results from the ANOVA were statistically significant with a

significance level of .001. Of the three subject assessments (tutoring in math only, tutoring in

reading only, and tutoring in reading and math) this was the only analysis that resulted in a

statistically significant difference in mean scores. Of the 9,009 students evaluated in this study,

only 1,565 students received benefits from the math only tutoring. Null Hypothesis 1 was

rejected for the math only tutoring group.

2. Do students who participated in Supplemental Educational Services exclusively in
reading for 10 or more hours demonstrate higher developmental scale score gains, as determined
by the FCAT, compared to students in a control group?

The number of students tutored in reading was considerably higher than the number

tutored in math. This analysis contained 6,360 students who were tutored in reading only. The

ANOVA test of mean scores was not statistically significant for the reading group with a

significance level of p = .117. Therefore, Null Hypothesis 2 was retained. The 106,600 hours of

tutoring this group received, was not effective in increasing their academic achievement in

reading.

3. Do students who participated in Supplemental Educational Services in both math and
reading for 10 or more hours demonstrate higher developmental scale score gains, as determined
by the FCAT, compared to students in a control group?

This group of 1,084 students received tutoring in both math and reading. Reading tutoring

totaled 6,488 hours and math tutoring totaled 6,505. The ANOVA test of mean scores resulted in









a finding of no statistical difference between the tutored experimental group and the control

group with a significance level of p = .246 in math and .646 in reading. This group also failed to

increase their academic achievement. Therefore, Null Hypothesis 3 was retained.

4. Do students tutored by the three major SES providers in each district (i.e., those that
provided tutoring for the largest number of students) for 10 or more hours demonstrate higher
developmental scale score gains, as determined by the FCAT, compared to students in a control
group?

Each district was represented by three SES providers who provided tutoring for the

largest number of students in that district. Of the 18 providers analyzed, 3 were found to have

produced statistically significant results per the ANOVA. As a result, the null hypothesis was

rejected for these 3 providers. The ANOVA results from the remaining 15 providers resulted in a

finding of no statistically significant difference in the mean scores between the tutored

experimental group and the control group. Therefore, Null Hypothesis 4 was retained for these

15 SES providers.

Conclusions and Recommendations

Sunderman & Kim (2004) defined the goal of SES as a program that was designed to

increase the academic achievement of students in low-performing schools. The results of this

study found minimal evidence to indicate that the SES tutoring program was producing

significant academic growth for Title I students. In fact, the study's statistical evidence clearly

demonstrates that the SES program had only a minimal impact on Title I students. More

extensive results should be expected for a program that consumes 20% of district's Title I

allocation.

This analysis indicated that despite 9,009 students in grades 4-10 received a total of

144,276 hours of tutoring, major academic growth accrued for a relatively small number of

students. Of the 6 districts assessed, only 3 of the 18 major SES providers had statistically









significant results on increasing the academic achievement of students as measured by FCAT

Developmental Scale Scores. The number of students tutored by the three providers was small (n

= 413) when compared to the total number of students (n = 9,009) who received tutoring in the

2006-07 school year. The only student group that was positively affected by the tutoring in math

totaled 1,565 students. The students who demonstrated increased academic achievement in at

least one subject totaled 1,978 (22%), leaving 7,051 (78%) students who received tutoring, but

did not demonstrate increased academic achievement.

Isolating the impact of the SES program on students who did make academic gains is

difficult. Rickles & Barnhart (2007) stated that positive outcomes of students who participated in

the SES program might be due to students who are more motivated and persistent in nature or

who have parents who are more involved in their children's education than those who did not

participate. Burch (2007) noted that that selection bias could also account for higher gain scores.

For example, parents who enroll their children in SES tend to be better educated and take a more

active role in their children's education. He also stated that academic gains may have resulted

from an academic program other than SES. Such selection bias was not apparent from the results

of this study.

The cost of the SES program also must be considered when assessing the effectiveness of

the program. SES is funded by a 20% setaside that is deducted from the school district's Title I

funds. This setaside is taken from grant funds that in past years were distributed to Title I

schools. Therefore, with the implementation of SES, school funding for Florida's Title I schools

was decreased by 20%. The setaside encumbered to fund SES and school choice in the 67

Florida school districts for 2007-08 amounted to $115,140,533. The 20% setaside for Miami

Dade County Public Schools alone amounted to $25,308,760, with an additional 21 Florida









school districts encumbering at least $1 million each to fund the SES program (Florida

Department of Education, 2007a). Full implementation of SES nationwide could total more than

$2.5 billion of the annual Title I grant funds (Ascher, 2006; Cohen, 2006). Given the minimal

number of Florida students in large urban school districts who benefit from SES, it must be

determined whether this is the most effective way to spend these funds.

Students in Title I schools that have not made AYP can be challenging learners. In past

years many research based, innovative programs have been utilized with these students to

increase their academic achievement. Despite these programs the achievement of these students

has not significantly increased. The question must be asked whether it is feasible for after-school

tutoring programs presented a few hours a week to increase the academic performance of these

students. SES is expected to enable significant academic strides in students who have continued

to be unsuccessful. Perhaps the expectations of the SES program are inappropriate.

The task of this analysis was to assess the effectiveness of the SES program in Florida

urban district schools that were required to offer this program in the 2006-2007 school year.

Through the literature review, data collection, and findings reported in this analysis, the

following conclusions were developed:

Conclusion 1:

According to SES federal guidance, the state is charged with the responsibility to develop,

implement, and publicly report the effectiveness of SES providers and is required to withdraw

approval from providers that fail to increase the academic performance of students for two

consecutive years (U. S. Department of Education, 2005). Hess & Finn (2004b) suggested that

guidance be provided to states to help develop sound and rigorous evaluations, including

planning and funding. The Northwest Regional Educational Laboratory (2004) summarized the









importance of evaluating SES programs by stating that it is not the lack of evaluation that is of

the utmost concern; it is the importance of knowing if SES students are actually receiving

services that increase their academic achievement.

Florida implemented the SES program in 2004-05 in 11 urban districts that had Title I

schools in their third year of non-adequate yearly progress. The number of SES districts

increased every year until 2006-07, when SES was required in every school district in Florida.

During the 2007-08 school year Florida approved 201 providers to provide SES (Florida

Department of Education, n.d.). SES has been fully implemented in Florida, but there has been

no evaluation of the effectiveness of the program in the state. This study confirmed that

relatively few providers were effective in raising the academic performance of students in large

urban Florida school districts. Effective providers need to be retained, but ineffective providers

need to be eliminated from the state-approved list. To meet the mandate of NCLB a

comprehensive evaluation is necessary so that ineffective providers can be removed from the list

of state-approved providers.

Conclusion 2:

Reid (2004) indicated that one of the reasons for the lack of evaluation is that states lack

the capacity and funding to develop an effective monitoring system and evaluation method. By

the very nature of the NCLB sanction process, SES enrollment continues to increase yearly. With

increased enrollment, additional administrative and management burdens are created. Districts

are forced to assume these responsibilities with no additional funding (Ascher, 2006). The Center

on Education Policy released a 2006 study based on survey results from 50 states, concluding

that NCLB created greater administrative burdens with inadequate funding. Rentner, et al. (2006)

concluded that districts and states lack funding and staff to manage the many demands of NCLB.









These financial burdens are not part of the 20% setaside, but are additional funds taken

from the Title I budget that would have been allocated to schools before NCLB was enacted. A

portion of the required setaside should be used to ease the administrative burden on districts

created by SES. Using 4% of the 20% setaside would cover administrative costs such as printing,

postage, parent outreach, monitoring, and personnel needs. Using this portion of the setaside for

SES administrative costs would ultimately increase the financial support of all Title I schools and

students if such costs were not taken from their allocation.

Conclusion 3:

Although there is research that identifies quality indicators (Table 2-1, p. 57) of effective

tutoring (i.e., student-teacher ratio, collaboration with the classroom teacher, tutor qualifications,

tutor training, and length of tutoring), NCLB does not require that they are used in SES tutoring

programs. The law vaguely states that SES programs must be of high quality, research based, and

designed to increase student achievement (U. S. Department of Education, 2005). For SES to be

effective in raising the academic achievement of students, established quality indicators of

effective tutoring must be required of SES providers and be part of the evaluation of

effectiveness.

The literature review conducted for this analysis indicated that low student-teacher ratio is

an important factor in effective tutoring. A 1:1 or 2:1 ratio is suggested for a successful tutoring

program. Research indicated that 1:1 tutoring is considered as the most effective form of tutoring

(Baker et al., 2006; Elbaum et al., 2000; Juel, 1996; Lauer, 2006; Wasik, 1998). Programs that

provide 1:1 tutoring for at-risk students were proven to increase the academic achievement of

students in reading. Currently, Florida allows SES providers to choose their student group size.

These choices include small group (not to exceed 5 students), large group (not to exceed 10









students), and individual tutoring. The allowed student grouping size for SES must be revised to

provide 1:1 tutoring to individualize instruction.

Another important aspect of successful tutoring is collaboration between the child's

classroom teacher and the tutor. Tutoring needs to be coordinated with classroom instruction to

provide the student with additional instruction that parallels what is being taught in the classroom

and that uses comparable materials. This enables the student to have multiple opportunities to

work on difficult materials. As students master the material during tutoring, they are more likely

to perform better in class (Wasik, 1998). Currently, the curriculum used during tutoring is left to

the discretion of the SES provider. Little is known about what students are actually being taught

during tutoring sessions beyond what is listed in the SES provider's state application, on the

provider's website, or in marketing materials. Although NCLB encourages SES providers to

align their curriculum with state standards, it clearly prohibits states and districts from trying to

influence a provider's curriculum (Burch, 2007). A study conducted by Anderson & Laguarda

(2005) noted a lack of coordination between the SES provider and the teacher. Many teachers in

the study reported that they did not know which of their students were participating in the SES

program and felt that collaboration with the tutors would help their students. Collaboration

between teachers and SES providers must be required so that tutoring will be meaningful to

students.

The next quality indicator that should be required is that of tutor qualifications.

Experienced, certified teachers are the best tutors (Elbaum et al., 2000; Gordon, 2003; Gordon et

al., 2004; Wasik, 1998). Teachers make effective tutors because they are specifically educated to

teach. Tutoring enables them to individualize instruction, which they are often unable to do in a

classroom setting. Their education and professional experience make teachers key in providing









effective tutoring (Gordon et al., 2004). Currently, the Florida Department of Education requires

that SES tutors meet the minimum standards for Title I paraprofessionals. These minimum

standards include a high school diploma and two years of college, an associate's (or higher)

degree, or a passing score on a test required of paraprofessionals who are employed in Title I

schools. The qualifications of tutors ought to be revised to prohibit anyone from tutoring who is

not a certified teacher. Teachers are specifically trained in differentiated instruction and can

individualize instruction based on a student's needs. Title I students have special academic needs

that are most likely to respond to the experience and expertise of certified teachers.

Tutors are most effective when they receive extensive training. Tutors with specialized

training consistently produce higher levels of student achievement than tutors with little or no

training (Gordon et al., 2004).Training must continue throughout the duration of the tutoring

services and must include monitoring of tutors and students during tutoring sessions, followed by

immediate feedback (Wasik, 1998). Although training programs which typically consist from 4

to 20 hours are provided by some SES providers, not all providers train their tutors (Ascher,

2006). Tutoring training is not required by NCLB and the Supplemental Educational Services

Non-Regulatory Guidance does not address it (U. S. Department of Education, 2005).

Curriculum implementation, behavior management, and individualized instruction are just a few

of many important tools necessary to deliver effective instruction in a tutoring environment. SES

tutors would benefit from specific, ongoing training to tutor SES students effectively.

Tutoring delivered over the entire school year with the same tutor working with the same

student was found to be most effective. Effective tutoring should be long term, intensive, and

consistent (Baker et al., 2006; Gordon, 2003; Juel, 1996; Wasik, 1998). The SES program is

designed to provide each student with a specific allocation to fund tutoring. When the allocation









is depleted, services automatically end for each student (U. S. Department of Education, 2005).

A typical tutoring program provides 30 hours of free tutoring (Hess & Finn, 2004a). It should

also be noted that the frequency or intensity of service can vary because NCLB has no set

requirements (Kasmin & Farmer, 2006). If tutoring is provided by a certificated teacher using

research based curriculum in a low student to teacher ratio, it may not be effective because of the

short length of time SES is offered. The U.S. Department of Education currently allows SES

providers to set their own hourly rate within the parameters established by each state. The hourly

rate in Florida for the 2007-08 school year ranges from a low of $20 per hour to a high of $80

per hour, with an average being $60 an hour (Florida Department of Education, n.d.). The

average student allocation for the 2006-07 school year was $1,124 (Florida Department of

Education, 2007d). Therefore, based on the parameters set for SES provider's hourly rates, a

student could be tutored for 56 hours at $20 per hour or 14 hours at $80 per hour. Hourly rates

need to be established by the Florida Department of Education to allow students to receive

consistent, long term tutoring services. In addition, to enable tutoring to be long term, the student

allocation must be increased. If hourly rates are established by the Florida Department of

Education and the student allocation is increased, more tutoring hours would be provided to each

student.

Implications

Per SES federal guidelines, it is the state's responsibility to evaluate SES providers to

ascertain whether they are effective in raising the academic achievement of students who receive

tutoring. Although billions of federal education dollars are pumped into this program yearly, no

state has completed a comprehensive evaluation of the effectiveness of SES. Research has shown

that states are struggling with evaluation design and funding for a comprehensive assessment of

SES effectiveness (Minnici & Bartley, 2007; U. S. Government Accountability Office, 2006).









SES evaluation is complicated and costly, and the U. S. Department of Education must take the

lead in designing, as well as funding, an effective SES evaluation process.

Current evaluations are typically conducted by large, urban school districts concerned

about the amount of money invested in a program with no proven results. There is considerable

research to confirm the effectiveness of tutoring, but most evaluations completed on SES do not

show positive results on increasing the academic achievement of students (Lauer et al., 2004;

Moss et. al., 2001). Although SES has been implemented in Florida since the 2004-2005 school

year, there has been no statewide evaluation of SES.

The only avenue to remove ineffective SES providers from the state-approved list is by

state evaluation (U. S. Department of Education, 2005). Lack of state evaluation forces school

districts to enter into contracts year after year with SES providers whose effectiveness is not

documented, and per the results of this study, are questionable, thereby wasting millions of

federal dollars. More importantly, students who participate in SES are not provided the help they

need to increase their academic skills. States must begin evaluating SES providers; those who are

effective should remain on the approved list and ineffective providers must be removed. If the

overall evaluation results indicate that there are no effective SES providers, then the focus must

shift to defining a program design that will increase student academic performance. A plethora of

research exists that addresses the effectiveness of tutoring, and this information must be used to

increase the effectiveness of this multibillion dollar program.

To facilitate additional planning, future researchers should be aware of the data collection

problems experienced in this study. This study was designed to be a statewide assessment of

SES, but the statewide data could not be obtained from the Florida Department of Education in a

timely manner. The Florida Department of Education puts data requests in a queue and the wait









time is 18 months to have a data request reviewed. Thus, the most current data is not available

for use in such a study. Although Florida is considered by many to have the most advanced data

system in the nation, essential data were not made available for this study. This roadblock

required redesign of this study to include the seven largest urban school districts in Florida. One

of the districts initially included in the study required payment of a substantial fee for processing

the needed data. After much thought and conversation, that district was excluded and the study

was again redesigned to include six urban Florida school districts. Timely receipt of data still

produced complications in that it took eight months to receive data from two of the districts.

Therefore, creating a timeline that allows ample time for redesign and data collection is essential.

Future Research

The goal of the SES program to increase the academic achievement of Title I students is

an important goal in today's education system. There are many problems facing educators in

schools today, and new ways to educate students effectively must be found. Decreasing the

dropout rate, eliminating the achievement gap, and educating students to succeed in a global

economy are worthy goals, and NCLB was implemented in response to these needs. NCLB

provisions are a starting point, but school districts, regardless of their level of effectiveness, are

currently unable to follow them. This study opens conversation on the effectiveness of SES, but

future research is needed. The following recommendations address needs for future research

regarding SES effectiveness.

1. A longitudinal study that follows SES students from year to year is needed to assess the
cumulative effect of long term tutoring. Consistency of results across multiple years would
strengthen conclusions regarding program effectiveness.

2. FCAT developmental scale scores are designed to track a student's progress from year to
year. Analyses that show statistically significant results should be compared to the
expected increase in scale scores from grade level to grade level to assess the actual
academic achievement.









3. In this study FCAT developmental scale scores were used as the measurement tool. From
an assessment perspective, this is a rather blunt instrument to use for this purpose. In future
research to better understand and interpret the results, additional measures are needed to
assess the effectiveness of SES tutoring.

4. Future evaluations should include systematic sampling to form a control group that groups
students by pre-test FCAT developmental scale scores, in addition to grade levels, that are
aligned with the experimental group.

5. Consideration must be given to the confounders in this study. For example, SES is a
parent-driven program in which a child can only receive tutoring if a parent completes and
returns the tutoring application. Analysis should be completed on the differences between
parents who request services and parents who do not. This would help in determining if the
differences in parental support are a contributing factor to the success or lack of success in
advancing the academic achievement of SES students.

6. Another confounder to address in future research is the individual attributes of each SES
school. Schools that have not made AYP for several years receive intense remediation with
several new programs and strategies every year. Adjustments in staff, curriculum, and
instruction are ongoing in an attempt to provide an educational program that is effective in
raising students' academic achievement. If an SES school increases the academic
achievement of its students, can this increase be entirely attributed to a tutoring program
that lasts a few weeks? If not, thought must be given to methods for isolating increased
achievement as a result of SES or other educational programs and strategies.

7. Tutoring is an effective tool in raising the academic performance of students (Moss et al.,
2001; Lauer et al., 2004). Future research should focus on the implementation and fidelity
of the SES program. Several quality indicators of successful tutoring (Table2-1) have been
identified and future research should focus on the implementation of these indicators in the
SES program.

Summary

The No Child Left BehindAct was implemented in 2002 in an effort to decrease the

achievement gap between student groups by providing additional academic opportunities for

America's Title I students in low achieving schools. Supplemental Educational Services was

included as one of these academic opportunities and was designed to provide students with

tutoring. Although SES is specifically designed to increase the academic achievement of

participating students through tutoring, there has been no evaluation of the success of this

program (U. S. Government Accountability Office, 2006). Federal guidance places the









responsibility of evaluating SES on the state, but there have been no evaluations completed by

the Florida Department of Education. Therefore, the goal of this study was to assess the

effectiveness of SES in increasing academic achievement for students

This study focused on assessing the increased academic performance of students on the

FCAT as measured by developmental scale scores. The results found increased developmental

scale scores for students who received tutoring in math, but there were no statistically significant

findings on increased achievement for students who received tutoring in reading or in reading

and math. This study also assessed 3 of the largest SES providers in each of the 6 districts that

participated in this study. Of those 18 providers assessed, only 3 provided results that confirmed

increased academic achievement.

Although there was some increased academic achievement found in this study, the number

of students benefiting from SES must be significantly increased to justify the time and money

spent on implementing this tutoring program in America's schools. At the time of this study, the

reauthorization of NCLB is only months away. Therefore, this is the optimal time to make

needed adjustments and revisions. Michael Kirst (2004), a veteran policy analyst, found that it

took more than 10 years and many legislative and administrative modifications before the initial

Title I program in the 1965 Elementary and Secondary Education Act achieved its intent. Kirst

stated that this is typical of aggressive new federal programs.

The overall goal of Title I is to increase the academic achievement of students who live in

poverty, and all programs funded through Title I must meet this goal. Therefore, SES must be

evaluated and subsequent action must be taken to ensure its success, or the program needs to be

abandoned and the funds returned to Title I schools where they can be used to meet the academic

needs of students.









Table 5-1. Summary of Findings for Hypotheses 1 through 3
Hypothesis Findings


Null Hypothesis 1


Null Hypothesis 2




Null Hypothesis 3


There is a relationship between SES
tutoring and increased FCAT
developmental scale scores in math
(p=.001)

There is no relationship between
SES tutoring and increased
FCAT developmental scale scores
in reading (p=. 117)

There is no relationship between
SES tutoring and increased
FCAT developmental scale
scores in math and reading
(Math p=.246; Reading p=.646)


Status of Null
Hypothesis
Rejected


Retained


Retained









Summary of Findings for Hypothesis 4*


District SES Provider Subject p = Status of Null
Hypothesis 4
1 Cool Kids Reading .984 Retained
Rocket Learning Reading .687 Retained
Rocket Learning Math .307 Retained
Education Station Reading .640 Retained

2 Academy of Success Reading .684 Retained
Academy of Success Math .947 Retained
Advanced Learning Reading .032** Rejected
Advanced Learning Math .022** Rejected
A+ Tutor U Reading .632 Retained
A+ Tutor U Math .146 Retained

3 Club Z! Reading .134 Retained
Club Z! Math .059 Retained
A+ Tutor U Reading .613 Retained
Rocket Learning Reading .009** Rejected

4 AlphaBEST Reading .978 Retained
AlphaBEST Math .208 Retained
Huntington Learning Reading .160 Retained
Studentnest Math .004** Rejected

5 "A" Wise Choice Reading .424 Retained
Huxtable Education Reading .378 Retained
A+ Tutor U Reading .698 Retained

6 ACES Reading .748 Retained
ACES Math .680 Retained
Achievia Reading .571 Retained
PCTA Reading .260 Retained
PCTA Math .251 Retained
Hypothesis 4 states that there is no statistically significant difference between 2006 FCAT
developmental scale scores as the pre-test and 2007 FCAT developmental scale scores as the post-test
for students receiving at least 10 hours of tutoring from the three SES providers who tutored the
largest number of students in each district.
** Statistically Significant


Table 5-2.











APPENDIX A
UNIVERSITY OF FLORIDA INSTITUTIONAL REVIEW BOARD


UF Institutional Review Board
UN IVERSITYof FLORIDA


DATE:


PO Box 112250
Gainesville, FL 32611-2250
352-392-0433 (Phone)
352-392-9234 (Fax)
irb2(u fl.edu


May 29, 2007


FROM:


Myrna Alien
1898 Village Glen Drive
Jacksonville, FL 32259
Ira S. Fischler, Chair 'M- Ca-
University of Florida (
Institutional Review Board


SUBJECT: Renewal of Protocol #2007-U-0530

TITLE: The Effectiveness of Supplemental Educational Services in Florida Urban School
Districts
SPONSOR: PI'S OWN FUNDS


Because only existing data will be used in this protocol, it is exempt from further review
by the Board in accordance with 45 CFR 46.101b(4), as no human participants are
involved in this research. It is understood that information will be recorded by the
investigator in such a manner that individuals cannot be identified, directly or through
identifying links.

Should the nature of your study change or if you need to revise this protocol in any
manner, please contact this office before implementing the changes.

IF:dl


An F ul Oprprlunlty InsILtulll'n









APPENDIX B
REQUEST FOR DISTRICT FCAT DATA


July 20, 2007

XXXX County Public Schools
Mr. XXX XXXX
Manager
Information Services
XXX XXXXXXX
XXXX, Florida XXXXX

Dear Mr. XXXXX:

Please find enclosed the required documents to gain permission from the XXXXXX
School District to obtain FCAT data for Title I students receiving Supplemental Educational
Services (SES). I am currently pursuing a doctoral degree at the University of Florida and
request permission from your district to analyze the requested data to determine the effectiveness
ofNCLB's SES program for Florida students. As well as being a UF doctoral student, I am an
administrator for Duval County Public Schools in the Title I Office where my job responsibilities
include coordinating the SES Program for our district. My dissertation will analyze FCAT scores
for Title I students in the seven largest urban Florida school districts, which includes:

XXXXX
XXXXX
XXXXX
XXXXX
XXXXX
XXXXX
XXXXX

This evaluation will be done by using FCAT test scores for two consecutive years (2006
and 2007) for students in urban Florida districts who have received SES services. FCAT
developmental scale score gains will be used to determine academic growth. I am required by the
University of Florida to submit my request to the University Institutional Review Board. I have
enclosed a copy of this form and subsequent approval letter for your review. I anticipate needing
this data in September 2007, as I would then officially be a Ph.D. candidate at the university and
would begin the data analysis of the requested FCAT scores. Dr. David Quinn is my dissertation
committee chair and will supervise the dissertation process for the University of Florida. I am
submitting my request early in an effort to meet your requests and requirements in a timely
manner.

I have discussed this study with Mr. XXX XXXXX, Director of Federal Programs of
XXXXXX County Public Schools and will be happy to share my results with his office, as well
as any other department in your school district. The results of this study will be useful in









enhancing and improving the SES Program in your district, as well as other districts in Florida. I
will also be submitting this study to the Florida Department of Education Bureau of Public
School Options, which implements SES for the entire state.

This request was also submitted to Ms. Teresa Miller, Director of Educational Policy, at
Integrated Education Data Systems at FDOE in an effort to obtain all needed FCAT scores from
her department, but was told that it would be at least an 18 month wait for this data. Due to my
dissertation timeline and requirements, I cannot wait that length of time to begin the data analysis
needed, so I am contacting each school district to gain permission to use this data.

Enclosed you will find your districts required research form, a proposal narrative, a
literature review, an example Excel spread sheet that lists the FCAT data needed, and the IRB
form submitted to the University of Florida, as well as the subsequent approval letter. Please
consider my request to use this student data in my research. If you have any questions please do
not hesitate to contact me. I can be reached at XXX-XXX-XXXX or on my district cell at XXX-
XXX-XXXX.

Thank you for your consideration.

Sincerely,




Myrna Allen
Supervisor of Supplemental Instruction
Title I Office
Duval County Public Schools
1701 Prudential Drive
Jacksonville, Fl 32207
Office: 904-390-2123 Fax: 904-390-2634
allenm2@educationcentral. org

cc: Dr. David Quinn
Mr. XXXXXX
Mr. XXXXXX











APPENDIX C
DISTRICT ONE INSTITUTIONAL REVIEW BOARD


Superintendent of Schools
Office of Program Evaluation






July 30, 2007

Ms. Myrna Allen
XXXXXXXXXX
Jacksonville, FL XXXXX

Dear Ms. Allen:
I am pleased to inform you that the Research Review Committee of the XXXXXX County Public
Schools has approved your request to conduct the study, "The Effectiveness of Supplemental
Educational Services in Urban Florida School Districts." The approval is granted with the
following conditions:

1. The anonymity and confidentiality of all subjects must be assured.

2. The computer-generated data which are provided by the XXXXX will be either
aggregated or coded to ensure the subjects' anonymity.

3. The study is based on anonymous student records, so parent permission forms are
not required.

4. The study will involve approximately 34,000 XXXXXX students.

It should be emphasized that the approval of the Research Review Committee does not
constitute an endorsement of the study. It is simply a permission to request the voluntary
cooperation in the study of individuals associated with the XXXXX. It is your responsibility to
ensure that appropriate procedures are followed in requesting an individual's cooperation, and
that all aspects of the study are conducted in a professional manner. With regard to the latter,
make certain that all documents and instruments distributed within the XXXXXXX as a part of
the study are carefully edited.

The approval number for your study is 1382. This number should be used in all
communications to clearly identify the study as approved by the Research Review Committee.
The approval expires on June 30, 2008. During the approval period, the study must adhere to
the design, procedures and instruments which were submitted.

The computer-generated data for the study will be provided by Mr. XXXXXX of the Office of
Program Evaluation of the XXXX. Contact him at XXX-XXX-XXXX to make the necessary
arrangements.










If there are any changes in the study as it relates to the MDCPS, it may be necessary to
resubmit your request to the committee. Failure to notify me of such a change may result in
the cancellation of the approval.

If you have any questions, please call me at XXX-XXX-XXXX. Finally, remember to forward an
abstract of the study when it is complete. On behalf of the Research Review Committee, I want
to wish you every success with your study.


Sincerely,



Chairperson
Research Review Committee

XXX:XX


cc: Mr. XXXXXXXX


APPROVAL EXPIRES: 6-30-08


APPROVAL NUMBER: 1382













APPENDIX D

DISTRICT TWO INSTITUTIONAL REVIEW BOARD


July 25, 2007 c

rj


Mrs. Myrna L. Allen
Duval County Public Schools
Title I Office
1701 Prudential Drive. Room 406
Jacksonville, Florida 32207


Dear Mrs. Alien:


1,

"i


Thea County Public School district has agreed to participate in your research proposal, The Effectiveness of
Supplemental Educational Services in Urban Florida School Districts. A copy of this letter MUST be presented to the principal and
participants in order to assure them your research has been approved by the district. Approval is given, however, under the
following conditions:

1) Participation by the school, its teachers, students, or parents is to be on a voluntary basis. That is. participation is NOT
MANDATORY and you must advise all participants that they are not obligated to participate in your study.

2) If the principal agrees the school will participate, it is up to you to find out what rules the school has for allowing people
on campus and you must abide by the school's "check in" policy. You will NOT BE ALLOWED On any school campus without
first following the school's rules for entering campus grounds.

3) Confidentiality must be assured for all. That is, ALL DATA MUST BE AGGREGATED SUCH THAT THE PARTICIPANTS CANNOT BE IDENTIFIED.
Participants include the district, parents, students, and administrators.

4) PARENT PERMISSION MUST BE OBTAINED FOR ALL STUDENTS INVOLVED IN YOUR RESEARCH. This includes both the sample group AND.
if you have one, the control group. You must indicate in your letter to the parent all the types of data you will be collecting
(i.e., race, gender, FCAT scores, etc.). It is appropriate to ask parents of participating students to respond only if they do
not want their child to participate-

5) Student data MUSI be uESTROYED when the project has been completed.









APPENDIX E
DISTRICT THREE INSTITUTIONAL REVIEW BOARD

RESPONSE OF XXXXXXX COUNTY PUBLIC SCHOOLS
INSTITUTIONAL REVIEW BOARD SUBMISSION
(CONTINUED ON NEXT PAGE)

August 7, 2007



Your research project is approved but please be aware that SES data are not readily
accessible in the district. It would require substantial work on your part to retrieve the
SES data. FCAT and demographic data could be provided.

The contact person for Title 1 SES Services is XXX (XXX) XXX-XXX.

Senior Director
Accountability, Research, and Assessment
Telephone: XXX-XXX-XXXX
Fax: XXX-XXX-XXXX

















Submit this form and a copy
of -v- if renncl o -


Your research proposal should
include: Project Title; Purpose
and Research Problem;
Instruments; Procedures and
Proposed Data Analysis


R nl. lstr'S Namo Myrna L. Allen


Address: Home
Rp_;_,, 1701 Prudential Drive, Rm 406. Jacksonville, FL


Date July 5, 2007

PhonI


32207
Project Director or Advisor Dr. David Phon

Address Please see attached UF IRB form.


Degree Sought: O Associate ID Bachelor's l Master's Specialist
(check one) E Doctorate 0 None


Project TitleThe Effectiveness of Supplemental Educational Services in Urban Florida School Districts


ESTIMATED INVOLVEMENT

PERSONNEL/CENTERS NUMBER AMOUNT OF TIME SPECIFY/DESCRIBE GRADES,
PERSONNELCEN S DAYS. HOURS. ETC I SCHOOLS SPECIAL NEEDS. ETC
Students .-L. 1 T.I E. no l.- -.r_. ru F .A..r .7 I :0-l L ,
Teachers N/A
Administrators N/A
Schools/Centers N/A
Others specifyl r ..

Spe., r, possible benefits to
studenls/schoot system ine re 'slts of this research stud.,- .-,iT 1 i ... -1 -,I -I _:1 .
FC.i Z. :;rij. to .L.i_ 1 enhance their Supplementr I .:r*i.. ~-, -, ,i : T
"-*-F !' "' -.11 ,*i.1. LL. r. o, ;; r. FC<' E ,- a.i 5- ,r r F.r L. r L AL, -p r._ ,_


is the offiep ree~noneihie For i-he sfei-eaide~ inn1 e.,.ernt. i-i noofC


ASSURANCE
Using the proposed procedures and instrument, I hereby agree to conduct research in accordance with the policies of
the Orange County Pu ij_,Sit.nools Dev;aions 9m I';aporoved procedures shail be cleared lhrou[gh the Snrior
Director of Acicountability. R rch. and Assess i R, fc nd mitenals snall be supplied as specifiea

Requester's Sijnalure i/ '.-. .t Y, .


Approval Granted: Yes 0 No Date: ,-- 7

Signature of the Senior Director
for Accountability, Research, and Assessment


NOTE TO REQUESTER When seeking apj
Accountability, Research. ana Assesrmenle .

Reference School Board Policy GCS, p 249


'n. signed
(


Dv the Senior Director.


IDO ;GB0103'2i3-1,tFY RE'.' 1.04


------~


'"';I


-VY~YYIY


E 2t J7


p1 e i .i i










APPENDIX F
DISTRICT FOUR INSTITUTIONAL REVIEW BOARD









November 9, 2007



Ms. Myra Allen
Duval County Public Schools
Title I Office
1701 Prudential Drive, Room 406
Jacksonville, FL 32207

Dear Ms. Allen:

The Superintendent's Research Review Committee approved your request to conduct research
concerning The Effectiveness of Supplemental Education Services (SES) in Urban Florida School
Districts, in the School District of XXXXXXX County.

The purpose of this study is to analyze FCAT scores for Title I students receiving Supplemental
Educational Services (SES) and a comparison group that did not receive services, but have the same
demographics. This report will assess whether the additional academic instruction given by SES providers
increases the academic achievement of students in low-performing schools in the seven largest urban
Florida school districts.


To conduct the survey, you will

SContact the Department of Supplemental Educational Services to identify a contact person;
Request the contact person to provide 2006 and 2007 FCA T scores in Math/Reading, as well as,
other needed data for all students who have received SES services and for a control group of
students who were not enrolled in SES;

Engage in no direct student contact;

Receive only student data with the identity of all students deleted.
As you conduct your research, please use the following guidelines:

Submit to this office, a signed Affidavit of Good Moral Character for each researcher before they
begin (A blank affidavit form is enclosed);

SObtain permission from the principal or department administrator before beginning;

SIn the case of student subjects, obtain written permission from the parent or guardian before
proceeding;

Provide a copy of all completed and signed parental/guardian consent forms to the principal or
principal's designee;













Page 2
Myra Allen
The Effectiveness of Supplemental Educational Services in Urban Florida School Districts



* All collection activities involving students must occur in the presence of school staff members;

* If your research requires the use of additional resources in the future, you must first submit a
written request to this office and then wait for a response before proceeding;

* One copy of the study results with an executive summary must be submitted to the Department of
Research and Evaluation no later than one month after completion of the research;

Your research activities at the school must not occur during the testing window of the Florida
Comprehensive Assessment Test (FCAT). The FCAT testing window includes pre-test,
administration, and post-test activities from February 1, 2008 through March 28, 2008.

According to our District's procedures, participation is voluntary. Thank you for your interest in our
school district.






XXXXXXXXX,Director
Research and Evaluation

XX:XX
Enclosure

c: XXXX
XXXX









APPENDIX G
DISTRICT FIVE INSTITUTIONAL REVIEW BOARD







September 11, 2007





Dear Ms. Allen:
Your request to conduct research in XXXX County Public Schools has been
approved. This approval applies to your project in the form and content as
submitted to this office for review. Any variations or modifications to the
approved protocol must be cleared with this office prior to implementing such
changes.

Due to the fact that you are actively employed by XXXX County Public
Schools, the data upon which your study is based has not been disguised in any
way. However, no child or group of children is to be mentioned directly or
made personally identifiable in the final product of your research.

Upon completion of the study, it is customary to forward a copy of the finished
report to the Office of Instructional Research and Accountability, XXXXXX
XXXXXXXXX, XXXXXXXX, Florida XXXXX. This office also shall be
notified, in advance, of the publication of any reports/articles in which XXXX
County is mentioned by name.


If you have questions or concerns, please don't hesitate to call me at XXX-
XXXX.





General Director
Instructional Research and Accountability











APPENDIX H
DISTRICT SIX INSTITUTIONAL REVIEW BOARD









August 13, 2007

Ms. Myrna Allen
Duval County Public Schools Title
1 Office
1701 Prudential Drive, Room 406
Jacksonville, Florida 32207

Dear Ms. Allen:

I received your request to conduct research in XXXXXXX County. Your study, "The
Effectiveness of Supplemental Educational Services in Urban Florida School Districts"
proposal number 070708-04 has been reviewed and the research has been approved.

Your research request does not require school; teacher or student's participation;
therefore please contact our office to make arrangement to receive data you have
requested.

If there are ay questions or if additional information is needed, please contact our
Research & Accountability Department at (XXX) XXX-XXXX.

Once the research is completed please forward a copy of the results to my office.

Sincerely,




XXXXXXXXXX, Ph. D.
Director, Program Evaluation

XX:x










LIST OF REFERENCES


The 2006 Florida State Statutes. (2006). Retrieved March 29, 2007 from http://flsenate.gov/
statutes/index.cfm?Appmode=Display_Statute& Search_String=&URL=CH1008/SEC33
1.HTM&TITLE=->2006->CH1008->Section%20331#1008.331.

Anderson, L. M., & Laguarda, K. G. (2005). Case studies of Supplemental Services under the No
ChildLeft BehindAct: Findings from 2003-04. Washington, DC: U. S. Department of
Education, Office of Planning, Evaluation and Policy Development.

Anderson, L.M., & Weiner, L. (2004). Early implementation of Supplemental Educational
Services under the No Child Left BehindAct. Washington, DC: U. S. Department of
Education, Office of the Under Secretary.

Ascher, C. (2006). NCLB's Supplemental Educational Services: Is this what our students need?
Phi Delta Kappan, 88, 136-141.

Baker, J. D., Rieg, S. A., & Clendaniel, T. (2006). An investigation of an after-school math
tutoring program: University tutors + elementary students = successful partnership.
Education, 127, 287-293.

Beyond the pale: SES provisions emerge as 'loose cannon' reform [Electronic (2006). Version].
American Teacher, 91, 7. Retrieved December 20, 2006 from http://www.aft.org/
pubs-reports/americanteacher/oct06/nclb.htm.

Borja, R. R. (2006). Market for NCLB tutoring falls short of expectations [Electronic Version].
Education Week, 26, 5-13.

Burch, P. (2007). Supplemental Education Services under NCLB: Emerging evidence and policy
issues. Tempe, AZ: Arizona State University.

Chicago Public Schools. (2005). SES tutoring programs: An evaluation of the second year -part
one ofa two part report. Chicago: Office of Research, Evaluation and Accountability.

Code ofprofessional conduct and business ethics for Supplemental Educational Services
provider. (2005). Retrieved December 20, 2006, from http:// www.educationindustry.org
/tier.asp?downloadid=66.

Cohen, J. H. (2006). 'Supplemental Services': Theory vs. practice. Education Week, 25, 34-35.

Cohen, P. A., Kulik, J. A., & Kulik, C. C. (1982). Educational outcomes of tutoring: A meta-
analysis of findings. American Educational Research Journal, 19, 237-248.

Duval County Public Schools. (2007). Title I, Part A budget for 2007-08. Unpublished
spreadsheet.









Elbaum, B., Vaughn, S., Hughes, M. T., & Moody, S. W. (2000). How effective are one-to-one
tutoring programs in reading for elementary students at risk for reading failure? A meta-
analysis of the intervention research. Journal of Education Psychology 92, 605-619.

Florida Department of Education. (n.d.). Bureau of Public School Options approved SES
providers. Retrieved May 10, 2007, from http://data.fldoe.org/ses/search.

Florida Department of Education. (2006a). NCLB monitoring work papers. Tallahassee, FL:
Office of Title I Programs and Academic Intervention Services.

Florida Department of Education. (2006b). NCLB school choice leadership summit: Building on
the school day. Tallahassee, FL: Bureau of Public School Choice.

Florida Department of Education. (2007a). 1st preliminary fiscal year 2007-08 Title I, Part A
allocations distribution table. Tallahassee, FL: Office of Title I Programs and Academic
Intervention Services.

Florida Department of Education. (2007b). Assessment and accountability briefing book.
Tallahassee, FL: Office of Accountability, Research, and Measurement.

Florida Department of Education. (2007c). NCLB choice: Supplemental Educational Services
update for 2006-2007 schoolyear. Tallahassee, FL: Bureau of Public School Choice.

Florida Department of Education. (2007d). SESper pupil allocation 2007-2008. Tallahassee, FL:
Office of Title I Programs and Academic Intervention Services.

Fusarelli, L. D. (2007). Restricted choices, limited options: Implementing choice and
Supplemental Educational Services in No Child Left Behind, Educational
Policy, 21(1), 132-154.

Gewertz, C. (2005). States wrestle with how to evaluate tutoring [Electronic Version]. Education
Week, 24, 5-5. Retrieved March 16, 2007.

Gordon, E. E., & Gordon, E. H. (1990). Centuries of tutoring: A history of alternative education
in American and western Europe. Lanham, MD: United Press of America.

Gordon, E. E. (2003). Looking beyond the stereotypes: Ensuring the true potential of tutoring.
Phi Delta Kappan, 84, 456-459.

Gordon, E. E., Morgan, R. R., Ponticell, J. A., & O'Malley, C. J. (2004). Tutoring solutions for
No Child Left Behind: Research, practice, and policy implications. NASSP Bulletin, 88,
59-68.

Grossman, K. N. (2007). $50 million for what? [Electronic Version]. Chicago Sun-Times.
Retrieved May 15, 2007 from http://www.suntimes.com/news/education/385515,CST-
NWS-tutor 15. article.










Heistad, D. D. (2006). Analysis of 2005 Supplemental Education Services in Minneapolis Public
Schools: An application of matched sample statistical design. Minneapolis, MN:
Minneapolis Public Schools.

Hess, F. M., & Finn, C. E. (2004a). Inflating the life rafts of NCLB: Making public school
Choice and Supplemental Services work for students in troubled schools. Phi Delta
Kappan, 86, 34-40.

Hess, F. M., & Finn, C. E. (2004b). Leaving no child behind? Optionsfor kids in failing schools.
New York: Palgrave Macmillan.

Juel, C. (1996). What makes literacy tutoring effective? Reading Research Quarterly, 31, 268-
289.

Kasmin, M. S., & Farmer, G. L. (2006). The promise of Supplemental Educational Services: Is
the policy failing? Children & Schools, 28, 181-185.

Kennedy, E. (2005). The No Child Left Behind Act. Human rights: Journal of the Section of
Individual Rights & Responsibilities, 32(4), 16-18.

Kirst, Michael. (2004). To glimpse NCLB's future, look to the past. The Education Gadfly, 4.
Retrieved December 12, 2007, http://www.edexcellence.net/foundation/
gadfly/issue.cfm?edition=&id=129#1610.

Lauer, P. A. (2006). Out-of-school-time programs: A meta-analysis of effects for at-risk
students. Review of Educational Research, 76, 275-313.

Lauer, P. A., Akiba, M., Wilkerson, S. B., Apthorp, H. S., Snow, D., & Martin-Glenn, M.
(2004). The effectiveness of out-of-school-time strategies in assisting low-achieving
students in reading and iti1heIuliIn \ A research ,yinhe\i\. Aurora, CO: Mid-continent
Research for Education and Learning.

Minnici, A., & Bartley, A. P. (2007). State implementation of Supplemental Educational Services
under the No ChildLeftBehindAct. Retrieved May 15, 2007 from http://www.cep-
dc.org/_data/n_0001/resources/live/CEP-SES.pdf

Moss, M., Swartz, J., Obeidallah, D., Stewar, G., & Green, D. (2001). AmeriCorps tutoring
outcomes study. Retrieved January 5, 2007 from http://www.abtassociates.com/
reports/tutoring_0201.pdf

Northwest Regional Educational Laboratory. (2004). Evaluating Supplemental Educational
Service providers: Issues and challenges. Retrieved January 12, 2007 from
http://www.nwrel.org/planning/reports/esesp/esesp.pdf









Peterson, P. E., & West, M. R. (2003). No child left behind? The politics and practice of school
accountability. Washington, DC: Brookings Institution Press.

Pines, S. (2006). Briefing by Mr. Steven Pines, Executive Director; Education Industry
Associationfor the Senate Subcommittee on Education and Early Childhood
Development. Retrieved January 12, 2007 from http://www.educationindustry.org/EIA/
files/ccLibraryFiles/Filename/000000000125/EIA%20Testimony%20to%20
Senate%20HELP%20Comm%207-25-06.pdf.

Potter, A., Ross, S. M., Paek, J., & McKay, D. (2006). Supplemental Educational Services in the
state of Tennessee: 2005 2006. Memphis, TN: The University of Memphis, Center for
Research in Educational Policy.

Rees, N. S. (2006). MIA on SES [Electronic Version]. The Education Gadfly. Retrieved January
12, 2007 from http://www.edexcellence.net/foundation/gadfly/issue.cfm?edition=
&id=237#2775.

Reid, K. S. (2004). Federal law spurs private companies to market tutoring [Electronic Version].
Education Week 24, 1, 18-19. Retrieved January 5, 2007 from http://edweek/org//ew/
articles/2004/12/08/15tutor.h24.html.

Rentner, D. S., Scott, C., Kober, N., Chudowsky, N., Chudowsky, V., Joftus, S., et al. (2006).
From the capital to the classroom: Year 4 of the No ChildLeft Behind Act. Retrieved
December 12, 2007 from http://www.cep-dc.org/_data/global/nidocs/CEP-NCLB-Report-
4.pdf

Rickles, J. H., & Barnhart, M. K. (2007). The impact of Supplemental Educational Services
participation on student achievement: 2005-06 (No. Planning, Assessment and Research
Division Publication No. 352): Los Angeles: Los Angeles Unified School District,
Program Evaluation and Research Branch.

Sanderson, D. R. (2003). Setting up a successful after-school tutorial program: One district's
journey. Reading Improvement, 40(1), 13-20.

Schwartzbeck, T. D. (2005). Understanding Supplemental Educational Services. School
Administrator, 62(5), 12-13.

Smole, D. P. (2004). Supplemental Educational Services for children from low-income families
under ESEA Title I-A (No. RL31329). Washington, DC: The Library of Congress.

Stullich, S., Eisner, E., McCrary, J., & Roney, C. (2006). NationalAssessment of Title I interim
report: Volume I: Implementation of Title I (No. NCEE 2006-4001. Washington, DC:
U. S. Department of Education, Institute of Education Sciences.

Sunderman, G. L. (2006). Do Supplemental Educational Services increase opportunities for
minority students? Phi Delta Kappan 88, 117-122.










Sunderman, G. L., & Kim, J. (2004). Increasing bureaucracy or increasing opportunities?
School district experience i ith Supplemental Educational Services. Cambridge, MA: The
Civil Rights Project at Harvard University.

U. S. Department of Education. (n.d.). Elementary and Secondary Education Part A Improving
basic programs operated by local educational agencies. Retrieved May 10, 2007 from
http://www.ed.gov/policy/elsec/leg/esea02/pg2.html#ses.

U. S. Department of Education. (1997). Evidence that tutoring works. Washington, DC: Office
of the Under Secretary Planning and Evaluation Service.

U. S. Department of Education. (2002). No ChildLeft Behind: A desktop reference. Washington,
DC: Office of Elementary and Secondary Education.

U. S. Department of Education Office. (2003). When schools stay open late: The national
evaluation of the 21st-Century Community Learning Centers Program. Washington, DC:
Office of the Under Secretary.

U. S. Department of Education. (2005). Supplemental Educational Services non-regulatory
guidance. Retrieved April 8, 2007 from http://www.ed.gov/policy/
elsec/guid/suppsvcsguid.doc.

U. S. Department of Education. (2006). Fiscal year 2007 budget summary -February 6, 2006.
Retrieved May 11, 2007, from http://www.ed.gov/about/overview/
budget/budget07/summary/edlite-section .html.

U. S. Department of Education. (2007). No Child Left Behind: Helping families by supporting
and expanding school choice. Retrieved March 23, 2007 from http://www.ed.gov/
nclb/choice/schools/choicefacts.html.

U. S. Government Accountability Office. (2006). No ChildLeft BehindAct: Education actions
needed to improve local implementation and state evaluation of Supplemental
Educational Services. (GAO Publication No. 06-758). Washington, DC: Government
Accountability Office.

Vadasy, P. F., Jenkins, J. R., Antil, L. R., Wayne, S. K., & O'Connor, R. E. (1997). The
effectiveness of one-to-one tutoring by community tutors for at-risk beginning readers.
Learning Disability Quarterly, 20, 129-139.

Viadero, D. (2007). Evidence thin on student gains from NCLB tutoring [Electronic Version].
Education Week. Retrieved June 12, 2007 from http://www.edweek.org/ew/
articles/2007/06/13/41 ses.h26.html.

Wasik, B. A. (1998). Using volunteers as reading tutors: Guidelines for successful practices.
The Reading Teacher, 51, 562-570.










Zimmer, R., Gill, B., Razquin, P., Booker, & Lockwood, J. R. I. (2007). State and local
implementation of the No ChildLeft BehindAct: Volume Title I choice, Supplemental
Educational Services, and student achievement. Washington, DC: Office of Planning,
Evaluation and Policy Development.









BIOGRAPHICAL SKETCH

Myrna L. Allen graduated from Coventry High School in 1977 in Coventry, Ohio. After

moving to Florida in 1988, she attended Florida Community College at Jacksonville and

graduated with her associate's degree in the spring of 1991. She then attended the University of

Florida in the fall of that year and in 1993 she graduated Summa Cum Laude with her Bachelor

of Arts degree in education.

Ms. Allen accepted her first teaching position at Switzerland Middle School in St. Johns

County teaching middle school students with varying exceptionalities for four years. During this

time, Ms. Allen received her Master of Arts degree in educational leadership and accepted her

first administrative position as Assistant Principal of Bannerman Learning Center, an alternative

school for students in grades 6 through 12, in Clay County Public Schools. After working at

Banner Learning Center for three years, she accepted a position in Duval County Public Schools

as Assistant Principal of Southside Middle School where she was the administrator for the sixth

grade. After several years at Southside Middle, Ms. Allen joined the district office and currently

holds a position in the Title I Office as the district's Supervisor of Supplemental Services.

Included in Ms. Allen's family is her son, Terry, daughter, Christine, son-in-law, Roger,

and grandson, Braeden. She has resided in St. Johns County since 1988 and, for leisure, she

enjoys reading, gardening, cheering for the Florida Gators, and spending time with her family.





PAGE 1

1 ASSESSING THE EFFECTIVENESS OF SUPPLEMENTAL EDUCATIONAL SERVICES IN URBAN FLORIDA SCHOOL DISTRICTS By MYRNA L. ALLEN A DISSERTATION PRESENTED TO THE GRADUATE SCHOOL OF THE UNIVERSITY OF FLOR IDA IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY UNIVERSITY OF FLORIDA 2008

PAGE 2

2 2008 Myrna L. Allen

PAGE 3

3 To my children Terry, Christine, and Roger w hose love, motivation, and encouragement enabled me to make this milestone possible and to my grandson, Braeden, whose birth has been a blessing to our family

PAGE 4

4 ACKNOWLEDGMENTS I would like to acknowledge and thank m y children, Terry, Christine, and Roger for their love, patience, and support while I pursued this degree. I could never have accomplished this milestone without their love and support. By far the most important accomplishment in my life has been raising my children to be loving, responsible, caring adults. I would like to also express my gratitude to all of the people who helped me achieve my goal of obtaining a Ph.D. This group includes Dr Jim Doud, who made this program possible by establishing the Jacksonville Cohor t, as well as Dr. David Quinn for the support and patience he provided as chair of my committee. I also give my deepest appreciation to Dr. Jean Crockett for her positive attitude and editing skills, as well as Dr. Kathy Gratto for the words of encouragement she shared with me on more than one occasion. I would also like to acknowledge the help and dedication of Angela Rowe. I thank the Title I Office fo r their motivation and enthusiasm as I pursued this goal, especially Latosha Norman, Brenda Smith, Jennifer Militello, and Libby Marsh for their encouragement and support that a llowed me to leave the office in Jacksonville to attend classes in Gainesville. My deepest gratitude to Fra nk Herrington whose support and confidence in me never wavered, Lynette Weber whose friendship a nd editing skills were much appreciated, Dr. Kathy Divine for her expert advice on data anal ysis, Pat Cascone for her editing advice, and to my Associate Pastor, Bob Gauger, whose encourag ement was invaluable. I am grateful to Tim Allen for his steadfast confidence in my abilitie s and his words of encouragement for the last three and a half years. My final acknowledgement is extended to my friends in the Jacksonville/St. Augustine cohort. The support and friendship of the cohort wa s instrumental in my success. I will always have fond memories of the Friday s and Saturdays we spent in class. Special thanks to Lissa

PAGE 5

5 Dunn whose friendship and gentle nature was much appreciated and to Ed Pratt-Dannals whose research and perseverance brought the University of Florida to Jacksonville and made the cohort possible. I thank God for his blessings and guidance while pursuing this degree, as well as throughout my life. He truly blessed me in making this degree possible.

PAGE 6

6 TABLE OF CONTENTS page ACKNOWLEDGMENTS...............................................................................................................4 LIST OF TABLES................................................................................................................. ..........8 LIST OF TERMS.............................................................................................................................9 ABSTRACT...................................................................................................................................11 CHAP TER 1 INTRODUCTION..................................................................................................................13 Supplemental Educational Services (SES)............................................................................. 18 Statement of the Problem....................................................................................................... .22 Purpose of the Study........................................................................................................... ....24 Significance of the Study........................................................................................................25 2 LITERATURE REVIEW.......................................................................................................27 History of Tutoring............................................................................................................ .....27 Quality Indicators of Effective Tutoring................................................................................ 29 The Effectiveness of Other After-school Tutoring Program s................................................. 31 Current Status of Evaluating SES...........................................................................................34 Lack of Funding......................................................................................................................36 The Increasing Number of SES Providers.............................................................................. 37 Implementation in Rural Districts an d Districts with Low Enrollm ent..................................38 Highly Qualified Teachers......................................................................................................39 Increasing Enrollment.......................................................................................................... ...39 SES Coordination with Classroom Curriculum...................................................................... 40 SES Student-Teacher Ratio.................................................................................................... 42 Existing Research on SES......................................................................................................42 Minneapolis Public Schools SES Effectiveness ....................................................................44 Chicago Public Schools SES Effectiveness.......................................................................... 45 Los Angeles Unified School Districts SES Effectiveness .....................................................47 Tennessees SES Effectiveness Study....................................................................................47 Large Urban Districts SES Effectiveness..............................................................................48 SES Evaluation Progress........................................................................................................49 Challenges in Evaluating SES................................................................................................ 51 Recommendations to Strengt hen and Evaluate SE S.............................................................. 53 Summary.................................................................................................................................55 3 STUDY OVERVIEW.............................................................................................................57

PAGE 7

7 Purpose of the Study........................................................................................................... ....58 Research Hypotheses............................................................................................................ ..59 Overview of the Method.........................................................................................................60 2006-07 Participating Di strict Statistics .................................................................................60 Procedures..................................................................................................................... ..........64 Summary.................................................................................................................................65 4 RESEARCH FINDINGS........................................................................................................67 Research Questions............................................................................................................. ....68 Research Hypotheses............................................................................................................ ..68 Analysis and Quantitative Results.......................................................................................... 69 Summary.................................................................................................................................76 5 DISCUSSION.........................................................................................................................83 Discussion of the Findings..................................................................................................... .84 Conclusions and Recommendations.......................................................................................86 Implications................................................................................................................... .........93 Future Research......................................................................................................................95 Summary.................................................................................................................................96 APPENDIX A UNIVERSITY OF FLORIDA INST ITUTIONAL REVIEW BOARD ............................... 100 B REQUEST FOR DISTRICT FCAT DATA......................................................................... 101 C DISTRICT ONE INSTITUTIONAL REVIEW BOARD.................................................... 103 D DISTRICT TWO INSTITUTIONAL REVIEW BOARD................................................... 105 E DISTRICT THREE INSTITUT IONAL REVIEW BOARD ...............................................106 F DISTRICT FOUR INSTITUT IONAL REVIEW BOARD ..................................................108 G DISTRICT FIVE INSTITUTIONAL REVIEW BOARD................................................... 110 H DISTRICT SIX INSTITUT IONAL REVIEW BOARD ...................................................... 111 LIST OF REFERENCES.............................................................................................................112 BIOGRAPHICAL SKETCH.......................................................................................................118

PAGE 8

8 LIST OF TABLES Table page 2-1 Quality Indicators of Effective Tutoring............................................................................ 56 3-1 Participating District Statistics...........................................................................................66 4-1 SES Students by Grade Level............................................................................................ 77 4-2 SES Students by District................................................................................................... .77 4-3 Active SES Providers by District....................................................................................... 78 4-4 Total Students: Math Only Group by District and Grade.................................................. 78 4-5 Total Students: Reading Only Group by District and Grade ............................................. 79 4-6 Total Students: Reading and Math G roup by District and Grade......................................79 4-7 Analysis of Variance for Math Only Group...................................................................... 80 4-8 Analysis of Variance for Reading Only Group..................................................................80 4-9 Analysis of Variance for Reading and Math Tutoring Group........................................... 80 4-10 Analysis of Variance for District 1.................................................................................... 80 4-11 Analysis of Variance for District 2.................................................................................... 81 4-12 Analysis of Variance for District 3.................................................................................... 81 4-13 Analysis of Variance for District 4.................................................................................... 81 4-14 Analysis of Variance for District 5.................................................................................... 81 4-15 Analysis of Variance for District 6.................................................................................... 82 5 Summary of Findings fo r Hypotheses 1 through 3 ............................................................ 98 5 Summary of Findings for Hypothesis 4.............................................................................99

PAGE 9

9 LIST OF TERMS Adequate Yearly Progress (AYP) is th e measure of the extent to which students in a school taken as a whole, and certain groups within the school, demonstrate proficiency in at least reading/language arts and mathematics. It also measures the progress of schools under other academic indicators, such as the graduation or school attendance rate. The Elementary and Secondary Education Act (E SEA) was first enacted in 1965, is the principal federal law funding K-12 education. Eligible Students Students eligible for Supplemental Educational Services are those students from low-income families who attend Title I schools that are in their third year or more of not making AYP. Eligible School An eligible school is a Title I school that has students eligible for Supplemental Educational Services. The school must be a Title I school that has not made adequate yearly progress for three years or more. No Child Left Behind The federal No Child Left Behind Act, the most recent authorization of the Elementary and Secondary Education Act (ESEA), is the primary federal law funding K-12 education. Provider A provider of Supplem ental Educational Services may be any public or private (non-profit or forprofit) entity that meets the States criteria for approval. Potential providers include public schools, including charter schools, private schools, school district s, educational service agencies, institutions of higher education, faithand community-based orga nizations, and private businesses. Public School Choice Students wh o attend a Title I school in their second year of not making AYP are eligible to transfer to another publi c school in the district, including a public charter school. State Assessments Under the No Child Left Behind Act, state assessments are aligned with academic standards. Beginning in the 2002-03 school year, schools were required to administer assessments in each

PAGE 10

10 of three grade spans: grades 3-5, grades 6-9, and grades 10-12 in all schools. Beginning in the 2005-06 school year, assessments were required to be administered every year in grades 3 through 8 in math and reading. Beginning in the 2007-08 school year, science achievement was also assessed. Subgroup Subgroups are smaller groups of students disaggregated from the whole group that may be present in a school or school system. The subgroups specified in the NCLB law are Native American/Alaskan Native students, Asian/Pacific Islander students, Black students, White students, Hispanic students, students with limited English proficiency, students who are economically disadvantaged, and students with disabilities. Supplemental Educational Services Supplemental Educational Services are additional academic instruction designed to increase the academic achievement of students from low-income families who attend Title I schools in their third year or more of not making AYP. Title I Title I provides fe deral funding under NCLB for schools to ensure that all students have a fair, equal, and significant opportunity to obtain a high quality education.

PAGE 11

11 Abstract of Dissertation Pres ented to the Graduate School of the University of Florida in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy ASSESSING THE EFFECTIVENESS OF SUPPLEMENTAL EDUCATIONAL SERVICES IN URBAN FLORIDA SCHOOL DISTRICTS By Myrna L. Allen May 2008 Chair: David Quinn Major: Educational Leadership Nationally, millions of federal dollars are given to private tutoring companies to provide tutoring to low income students who participate in the Supplemental Educational Services (SES) program in Title I schools. There is little or no research to prove the effectiveness of this program in improving the academic growth of these students. The purpose of this study was to evaluate the effectiveness of the SES program in six large ur ban Florida school districts by utilizing existing scores from the Florid a Comprehensive Assessment Test (FCAT). A quasi-experimental design was used to an alyze the FCAT developmental scale scores wherein the scores were divided into two groups of Title I studen ts. Group One was composed of students who received SES in the 2006-07 school year and completed the FCAT in math and reading in both 2006 and 2007. Group Two was a c ontrol group composed of students who were enrolled in Title I schools, completed the FCAT in reading and math in 2006 and 2007, and were eligible for SES but did not participate in the program. The developmental scale scores from the 2006 and 2007 FCAT administration were used as pr eand post-tests. A one-way analysis of variance design was used to analyze the FCAT data. The analysis was completed using the Statistical Package for Social Sciences with a .05 level of significance.

PAGE 12

12 Results of this study found increased developm ental scale scores for Title I students who received tutoring in math, but there were no statistically significant findings on increased academic achievement for students who received tutoring in reading only or in reading and math. This study also assessed the effectiveness of individual SES providers by analyzing the three largest SES providers in each of the six districts that particip ated in this study. Of those 18 providers, only 3 provided results that c onfirmed increased academic achievement.

PAGE 13

13 CHAPTER 1 INTRODUCTION The Elementary and Secondary Education Act (ESEA) of 1965 was signed into law by President Lyndon B. Johnson as part of the W ar on Poverty and is still th e principal federal law funding K-12 education (Kennedy, 2005). ESEA crea ted the Chapter I program, subsequently renamed the Title I program, to ensure that stud ents living in poverty have an equal opportunity to obtain a high quality education and achieve the required academic growth to meet state standards and assessments. Title I provides fede ral funding to help students who are either behind academically or are at risk of falling behind academically. Funding is based on the number of low-income children in a school, generall y including those eligible for free or reduced price lunch, and was intended to supplement, not replace, state and district funds. Schools receiving Title I monies have been required to involve parents in deciding and evaluating how these funds are spent and in th e review process (Peterson & West, 2003). Title I sought to strengthen Americas schools by al locating substantial federal reso urces to turn the door of the schoolhouse into a true door to opportunity for all students (Kennedy, 2005, p. 16). The goal was to afford all students, even the most vulne rable, the opportunity to succeed by establishing public school classrooms as a place of learning. ESEA was most recently reauthorized on January 8, 2002 when President George W. Bush signed the No Child Left Behind Act (NCLB) into law. NCLB is referred to as the most comprehensive piece of federal legislation in 35 years (Peterson & West, 2003). This law represents an extensive renovation of federal sup port for elementary and secondary education in the United States and affects every federal program initially authorized under ESEA. As the largest federal program supporting elementary and secondary education, NCLB focuses resources where students have the greatest need (U. S. Department of Education, 2002).

PAGE 14

14 NCLB was signed into law in response to demands for pub lic education to meet the challenges of Americas workforce and changes occurring in Americas industry. The economic expansion of the 1990s gave birth to a new global economy, which increased the pressure on Americas public education system to set high standards, create str onger schools, and produce better teachers. In response to this pressure, Presiden t Clinton proposed edu cation reform in the Goals 2000 Educate America Act that became law on March 14, 1994. Goals 2000 was designed to increase academic standards, measure studen t progress, and provide adequate support to enable students to meet the increased acad emic standards. Although Federal funding for education was increased from 1994 to 2000, Goal s 2000 still fell short of meeting Americas educational needs (Kennedy, 2005). Poverty and inequality in Am ericas education system con tinued to be a roadblock to opportunity and progress and became visible in achievement gaps between student groups. African-American and Latino students continued to lag behind in academic achievement compared to other students. For example, the math and reading skills of ethnic minority students graduating from high school were equal to those of 13 year-old white students. Kennedy (2005) reported that the high school dropout rate increased for African-Ame rican students at the rate of 1 dropout for every 20 black students, compared to 1 for every 30 white students. Achievement gaps became especially obvious in high poverty schools, where in 2001 these students were 77% more likely to be taught by a teacher who was t eaching out-of-field as compared to students in low poverty schools. Kennedy also contends that while the dropout rate increased for AfricanAmerican students, the number of students lear ning English as a second language also increased by 1 million students, a 50% increase. Americas schools were not equipped to meet the educational needs of these students. The hope of Brown vs. the Board of Education (1954) to

PAGE 15

15 provide an equal, quality education for all mi nority students was still not a reality in 2001. Americas ethnic minority and low-income students were not receiving a quality education from qualified teachers (Kennedy, 2005). Furthermore, in ternational educational comparisons revealed that other countries education systems were pr oducing students who were academically superior to students in the United St ates (Hess & Finn, 2004b). Education remained foremost on the minds of the American people. During the 2000 presidential campaign both candi dates pledged to improve the education system. George W. Bush, the GOP candidate and governor of Texas, promoted the accountability system of Texas as a national model. Upon taking office, Presiden t Bush made good on his campaign promise and immediately sent to Congress legislation that created strong accountability for districts and states, as well as a school choice option modele d after the system in place in Texas public schools (Hess & Finn, 2004b). Negotiations over this legisl ation lasted a year, as the White House and Congressional leaders forged a bipartisan compromise that came to be known as the No Child Left Behind Act. NCLB enjoyed strong support from Republi cans and prominent Democrats such as Massachusetts Senator Edward Kennedy and California Representative George Miller. However, winning bipartisan support created many changes in the original Bush education plan. For example, the White House compromised on its propos al of choice-based reforms, such as school vouchers, and agreed to more re strictive options for students in low-performing schools. These negotiations added to the complexity of the NCLB legislation and, as a result, the federal government was challenged to work with and th rough states to make implementation meaningful with the end result providing a quality edu cation for all American students (Hess & Finn, 2004b).

PAGE 16

16 For the first time the federal government tied federal dollars to cons equences for schools that failed to make adequate yearly progress. It gave parents and st udents in low-performing schools the option to obtain additio nal educational assistance or move to another school. From the time NCLB was signed into law it became ev ident that this complex law required extensive monitoring and evaluation. The academic performance of students, individual schools, and school districts would necessitate the use of best practices to turn NCLB into policy and subsequently into practice in every school district in America (Hess & Finn, 2004b). NCLB was charged with the reformation of the American education system which included meeting the educational needs of low-income children, children with disabilities, minority children, and English language learners. These students became the top priority of school reform. NCLB called for improveme nt in Americas schools and demanded accountability for results in decreasing the achie vement gap. The law required each state to create content standards that specified what students should know and performance standards that specified mastery levels for students. State tests were to be rigorou s and aligned with the standards, and teachers were to be of high qualit y for all students. The law created opportunities for students to transfer to other schools, participat e in after-school programs, and obtain free tutoring and extra academic support. The law also required that parents be informed of the academic performance of their childs school and be encouraged to become more involved in their childs education. Acco rding to Kennedy (2005), the No Child Left Behind Act is more than a slogan, (p. 18). It is a commitment and a promise to offer each child a quality education by improving public education. The commitment to strengthen public edu cation by improving the academic achievement of all students was apparent in NCLB by its very specific requirements for accountability: annual

PAGE 17

17 state assessment of the academic performance of students in grades 3 through 8 in math and reading, with one additional test to be administ ered in high school. The state assessments must be based on rigorous state standard s and at least 95% of enrolled students in each school must participate in the assessment. The results ar e reported by student subgroups which include poverty levels, race, ethnicities, disabilities, and limited English proficiencies. These data are converted into an annual school report card required by NC LB, and provide comparative information on the quality of schools (U. S. Depa rtment of Education, 2002). School report cards detail the performance of each subgroup and are required to be reported to the public. These annual report cards must show adequate yearly pr ogress (AYP) of the school toward the goal of 100% student proficiency by the end of the 2013-2014 school year. The AYP is defined by each individual state and is evalua ted most commonly in terms of benchmarks that increase on predetermined intervals until the goal of 100% proficiency is re ached in the 2013-14 school year. If a school does not demonstrate AYP, a sanction process, described below, is initiated and increases every year until the school makes AY P for two consecutive years (Peterson & West, 2003). As well as addressing the academic perf ormance of students, NCLB built upon and expanded the Elementary and Secondary Education Act by creating ground breaking educational mandates such as parental choice, high teacher quality, and Supplemental Educational Services (SES). These programs were created to increase the quality and effectiveness of the Title I program in raising the achievement of all stude nts, especially those students achieving at the lowest levels (Stullich, Eisner, McCrary, & Roney, 2006). Promoting parental choice and increased involv ement in their childs education is one of the cornerstones of NCLB (Hess & Finn, 2004b). U nder this legislation, local school districts

PAGE 18

18 must set aside 20% of their Title I, Part A allocation. Title I is a federally funded formula grant given to school districts to supplement th e education of students who attend high poverty schools. The 20% setaside is to be used to fund the first two sanctio ns under NCLB, public school choice and SES. The first sanction, public school choice, must be offered to students who attend schools that have failed to make AYP for two years. Part of the 20% is required to fund the transportation of students from their hom e school to the school of their choice (U. S. Department of Education, 2002). Supplemental Educational Services (SES) If schools fail to m ake AYP for three years, they are required to offer SES, the second sanction under NCLB. SES is designed to increas e the academic achievement of students from low income families who attend Title I schools. NCLB defines SES as additional academic assistance (e.g., tutoring, remediation, and other educational interventions) provided that these approaches are consistent with the content and instruction used by the school district and are aligned with the states academic content standard s. In addition to the NCLB sanctions of public school choice and SES, additional sanctions are added in each subsequent year a school does not make AYP. These additional sanctio ns include corrective action, pl anning for restructuring, and implementation of restructuring (U. S. Depart ment of Education, n.d.; Sunderman & Kim, 2004). The SES program must be aligned with the state s academic standards and is required by federal guidelines to be provided before school, afte r school, or on Saturday. Additionally, the SES program must be of high qualit y, research based, and designed to increase student achievement. This program is structured to create partnershi ps between public schools and a wide variety of tutoring companies (U. S. Department of Education, 2005). The Supplemental Education Services program evolved from President Bushs initial proposal, when drafting NCLB, that a student in a school that did not make AYP for three years

PAGE 19

19 would be offered a $1,500 exit voucher to be used toward a private school education. The exit voucher was not well received by the Democrats, and Senator Kennedy adamantly refused to support it. However, he di d counter with a proposal to use these funds as supplemental services The White House accepted this proposal fee ling that it would be a sufficient incentive to improve failing schools because the districts would have to pay for the tutoring out of their Title I funds, thus decreasing the amount available for district use. Furthermore, President Bush felt that private tutoring would gi ve struggling students an edu cational opportunity that would otherwise be unavailable to them (Hess & Finn, 2004b). In Florida for the 2007-08 school year, the 20% setaside ranges fr om a low of $46,724 for Liberty County Public Schools to a high of $25,308,760 for Miami-Dade County Public Schools, with 22 Florida school districts encumbering ove r $1 million for this setaside. The total 20% setaside encumbered to fund public school choice and SES in Florida is $115,140,533 for 67 school districts (Florida De partment of Education, 2007a). Full implementation of SES nationwide could total more than $2.5 billion of the annual Title I grant funds (Ascher, 2006; Cohen, 2006). The state, local school dist rict, parents, and SES provi ders are all charged with implementation responsibilies via the Supplem ental Educational Services Non-Regulatory Guidance (2005), developed by the United States Department of Education. Implementation of SES begins with the state and includes a numbe r of responsibilities in providing services to eligible students. These responsibilities begin w ith the identification of providers, as well as maintaining a list of approved providers, and mo nitoring the implementation and success of the program. States start the SES process by develo ping and advertising th e procedure by which a tutoring company becomes state-approved to serv e students. Once a provider is approved, the

PAGE 20

20 state must notify individual school districts of the providers who ar e eligible to provide SES. The state is also charged with the responsibility to develop, implement, a nd publicly report the effectiveness of SES providers a nd is required to withdraw approva l from providers that fail to increase the academic performance of students fo r two consecutive years (U. S. Department of Education, 2005). The school district also has many responsibilities in implementing SES in Title I schools that have not made AYP for three years. Dist ricts must notify parent s of eligible students annually and must offer assistan ce, if requested, in helping pa rents choose a provider that will meet their childs educational n eeds. Federal guidance mandates that the notification sent to parents must include a comprehensive descrip tion of the SES program and a timeline that explains the enrollment process. Districts are urged to reach the eligib le population by notices mailed to homes, as well as by newspaper, radio and TV ads, Internet sites, and notices at movie theaters, shopping malls, beauty sa lons, and churches. To ensure the privacy of students, districts are reminded that individual studen t identities must not be releas ed to the public. Districts also must draft and enter into a contract with the state-approved SES provi ders. If the demand for services exceeds the available funding, districts must implemen t a prioritization plan (U. S. Department of Education, 2005). After the enrollment process is completed and contracts with SES providers are executed, the district is responsible for implementation of the program. This in cludes supervising the program in each school where services are provided, paying the SES providers, monitoring student attendance, verifying that the tutors have clear background checks and meet education requirements, as well as approving each enrolled st udents academic plan (Florida Department of Education, 2006b)

PAGE 21

21 Parent responsibilities incl ude choosing a provider from the state-approved list. Parents are permitted to request assistance from the district in making this choice. They also are required to be active participants in the SES program, which includes working with the district and provider to create an academic plan appropriate fo r their childs needs and making sure that their child attends the tutoring sessi ons on a consistent basis. The SES provider responsibiliti es include providing the approp riate curriculum to enable each child to succeed academically in the program. This includes monitoring student attendance, measuring student progress, repor ting student progress to the pare nts and district, following the timeline set forth by the district, and ensuring that student privacy is maintained. Provider responsibilities also include providing services th at are secular, neutral, and non-religious (U. S. Department of Education, 2005). To fund the SES program each student is allotted a per pupil allocation for tutoring. During the 2007-08 school year in Florida, per pupil allocations range d from a low of $883 for Nassau County Public Schools to a high of $1,467 fo r Miami-Dade Public Schools; the average allocation was $1,124 per student (Florida De partment of Educa tion, 2007d). When the allocation is depleted, services automatically end for each student. Providers set their hourly rates within the parameters established by each stat e; in Florida, the hourly rate has ranged from $5 to $80 per hour. The hourly rate in Florida fo r the 2007-08 school year ranges from a low of $20 per hour to a high of $80 per hour, with an average of $60 an hour for the 201 stateapproved providers. SES providers al so choose the student-to-teacher ratio they will utilize. The Florida Department of Education allows providers to choose from small group instruction with a 5:1 student: tutor ratio, large group instruction th at cannot exceed 10 students per tutor, or individual tutoring. Providers are paid the hourly rate multiplied by the number of students being

PAGE 22

22 tutored. For example, if a provi der is tutoring 10 students at $ 80 an hour, their total hourly income would be $800. It is also important to not e that, in Florida, SES providers are only paid for students who attend the tutoring sessions (F lorida Department of Education, n.d.). The federal guidance allows states to determine if payment for tuto ring will be based on attendance. The Florida policy requires that th e student be present. Otherwise, the provider cannot charge the district for services to the student on that day (Florida De partment of Education, 2006a). Statement of the Problem Nationwide, each Title I school that has not made AYP for three years is required to provide SES to its students from low-income families. In the 2004-05 school year, 19% of eligible funded student s participated nationwide in SES, which was a 7% increase over 2003-04, when only 12% participated in this free tutoring program (Fusar elli, 2007). Loca lly, the Florida Department of Education reported that in the 2006-07 school year 70,908 stude nts participated in SES. This is more than triple the enrollm ent of the 2005-06 school year when 23,187 students received services (Florida Department of Education, 2007c). The main goal of the Supplemental Educational Services program is to provide tutoring or other supplemental academic enrichment services in reading, language arts, and mathematics to increase the academic performance of students. Thes e tutoring sessions must be offered before or after school, and they must be research based (Florida Departme nt of Education, 2006b). To fund this program, districts are required to setaside 20% of their Title I, Part A funds and they have been doing so since the 2002-03 school year (Cohen, 2006). School distri cts currently spend billions of federal dollars on this program nationwide with funds that would have been allocated to schools to supplement the curriculum be fore NCLB was signed into law (Cohen, 2006; Sunderman, 2006).

PAGE 23

23 Florida implemented the SES program in 200405 in 11 urban districts that had Title I schools in their third year of nonadequate yearly progress. The number of districts required to offer SES has increased every year until the 2006-07 school year, when SES was required in every school district in Florida. In the 2007-08 school year, 201 Florida providers were approved to tutor students (Florida Department of Education, n.d.). Now that SES has been fully implemented in Florida, the focus now should turn from implementation and move towards quality control of the services provided. This may be the most challenging aspect of NCLB, and many states an d districts are struggli ng with SES evaluation. The entire concept of successful tutoring must be clarified. Evaluation must address what specifically increases a students academic pe rformance, especially in schools where many different programs and new curricula are being implemented to help lo w performing students to achieve. If a student progresses to higher academic levels on the state assessment, it must be determined if the increase is due to a few hour s of tutoring per week or to a new curriculum implemented school wide (Hess & Finn, 2004b). Per the following excerpt from the NCLB Statute, Section 1116(e) 4, evaluating the effectiveness of SES is clearly de fined as a state responsibility: 4) STATE EDUCATIONAL AGENCY RESPONSIBILITIES A State educational agency shall ( D) develop, implement, and publicly report on standards and techniques for monitoring the quality and effectiveness of the servic es offered by approved providers under this subsection, and for withdrawing approval from providers that fail, for two consecutive years, to contribute to increasing the academic proficiency of students served. States are beginning to focus on SES effectiven ess, but there has been no definitive answer from the federal government on what defines effectiveness (Hess & Finn, 2004b). Monitoring the quality and effectiveness is a major challenge in many states. The Northwest Regional Educational Laboratory (2004) sums up the importan ce of quality control by st ating that the main

PAGE 24

24 concern it is not the lack of evaluation, but rather the im portance of knowing whether low achieving Title I students are receiving high quality services that can actually improve their academic performance. Although SES has been impl emented for over four years and billions of dollars have been spent, ther e is still little known about the effects of tutoring on improving student achievement, the fundamental goal of SES (Ascher, 2006). Purpose of the Study The purpose of this analysis was to assess the effectiv eness of the SES program in Florida Title I urban district schools that were re quired to offer this program during the 2006-2007 school year. To assess the effectiveness of SES, the follo wing research questions will be addressed: 1. Do students who participated in Supplemental Educational Services exclusively in math for 10 or more hours demonstrate higher developmen tal scale score gains, as determined by the FCAT, compared to students in a control group? 2. Do students who participated in Supplemental Educational Services exclusively in reading for 10 or more hours demonstrate higher developmental scale score gains, as determined by the FCAT, compared to students in a control group? 3. Do students who participated in Supplemental Educational Services in both math and reading for 10 or more hours demonstrate high er developmental scale score gains, as determined by the FCAT, compared to students in a control group? 4. Do students tutored by the three major SES pr oviders in each distri ct (i.e., those that provided tutoring for the largest number of students) for 10 or more hours demonstrate higher developmental scale score gains, as de termined by the FCAT, compared to students in a control group? Null Hypothesis 1 There is no statistically significant difference between 2006 FCAT developmental scale scores as the pre-test a nd 2007 FCAT developmental scale scores as the post-test for students in Title I schools rece iving at least 10 hours of tutoring in math.

PAGE 25

25 Null Hypothesis 2. There is no statistically significant difference between 2006 FCAT developmental scale scores as the pre-test a nd 2007 FCAT developmental scale scores as the post-test for students in Title I schools receiv ing at least 10 hours of tutoring in reading. Null Hypothesis 3. There is no statistically significant difference between 2006 FCAT developmental scale scores as the pre-test a nd 2007 FCAT developmental scale scores as the post-test for students in Title I schools receiv ing at least 10 hours of tutoring in math and reading. Null Hypothesis 4. There is no statistically significant difference between 2006 FCAT developmental scale scores as the pre-test a nd 2007 FCAT developmental scale scores as the post-test for students receiving at least 10 hours of tutoring from the three SES providers who tutored the largest number of students in each district Significance of the Study The highest priority of the No Child Left Behind Act is to give low achieving students the opportunity and ability to meet high academic standards and reach proficiency on demanding state assessments. The fact that only 29% of all fourth grade students performed at or above the proficient level on the National Assessment of E ducational Progress in reading in 2000 clearly indicated a need for nationwide education reformation (U. S. Department of Education, 2002). The federal government chose the reauthorizatio n of ESEA to implement education reform, which funded Title I in 2007 in the amount of $12.7 billion nationwide. This reformation requires accountability from states that received th is federal aid (U. S. Department of Education, 2006). Currently, Florida has no evaluation of the effectiveness of Supplemental Educational Services which consume federal funds that have been diverted from Title I schools. While

PAGE 26

26 Florida is required to prohibit provi ders from offering services to st udents if they fail to increase student achievement for two years, the state has neglected to enforce this section of the law. NCLB has been described as the accountability law, but this co rnerstone program has not been held accountable for results. SES cannot be improved if the weakne sses and deficiencies are not identified. Evaluating SES effectiveness will allow states, districts, and lawmakers to strengthen and improve these services.

PAGE 27

27 CHAPTER 2 LITERATURE REVIEW The purpose of this study is to assess th e effectiveness of Supplem ental Educational Services, most often offered as a tutoring prog ram, designed by NCLB to increase the academic achievement of Title I students in urban Florida sc hool districts. The purpose of this chapter is to present a review of the literature pertaining to the history and best practices of tutoring, as well as the effectiveness of other after-school tutori ng programs. The current st atus of SES evaluation efforts nationwide will then be reviewed, followed by existing research on the effectiveness of SES. Finally, the literature relevant to SE S evaluation challenges and recommendations to strengthen and evaluate SES will be presented. History of Tutoring Tutoring is a for m of educati on with a long, rich history that has been used to educate children for centuries (Gordon, Mo rgan, Ponticell, & O'Malley, 2004). Tutoring is as old as civilization itself, and its beginnings can be traced back to as early as 7000 B. C. when the family was responsible for the education of its children, as parents tutored their children in life skills and passed on their cultures folklore. History s most favored and honored teachers were tutors. Socrates, Plato, and Aristotle taught their students one at a time by using the principal of imitation, lectures, and posing problems to thei r students. Tutoring was the only form of education in ancient Rome and Greece and can be traced through and beyond the medieval transition in 500440. During the me dieval period the Church controlled education; thus tutoring focused on theological principles, and secular training was abandoned. During this time Britain tutored girls in the nunneries and, at the same time, boys in Ireland were tutored by receiving instruction in monasteries. As the century progressed, tutoring ag ain returned to the home, but it

PAGE 28

28 was only available for the children of royalty or noble descendants. Ce nturies passed, but the primary subject of tutoring rema ined theology (Gordon & Gordon, 1990). Tutoring persisted in Europe as the primary style of education, and when the colonists came to America they brought tutoring with them. In America, e ducation soon gave birth to the public school, but tutoring was often the only form of education available in isolated, rural, frontier settings. Education was valued in coloni al America. Often when a child grew up in a home of non-readers it was not unusual for neighb ors to offer to tutor the child. New England Puritans concern for educati on centered on strong religious be liefs, and tutoring centered on reading the Bible. Tutoring pers isted throughout the frontier era, but the number of children tutored at home declined as America was sett led and more states pa ssed compulsory school attendance laws (Gordon & Gordon, 1990). As the 19th century transitione d to the early 20th century, educators ignored tutoring as a part of education. They contended that due to student socialization issues, a public school education for all children was supe rior to tutoring small groups of isolated student s (Gordon et al., 2004). Interest in tutoring was revive d in the 1960s when the public st arted to question the quality of public education due to the poor academic achievement of many children. Peer tutoring was introduced in Detroit Public Sc hools and with the M obilization for Youth Program in New York City. Peer tutors were soon join ed by adult tutors and paid tutors. By 1973, the National School Volunteer Program estimated that 2,000,000 tuto rs were tutoring 3,000,000 students in over 3,000 programs. Tutoring was commonly used to edu cate students with disabilities, as well as gifted students. The academic benefits of tuto ring children were seen in classrooms across America in the 1960s and 1970s (Gordon & Gordon, 1990). Since the 1960s, each passing decade has increased the demand for tutoring in each socioeconomic group (Gordon, 2003).

PAGE 29

29 Presently, tutoring has become a permanent tool in educating students in the United States. Gordon & Gordon (1990) predicted that the f uture use of tutors across society will likely increase as educators continue to focus on individual differenc es and the means to increase productivity in all modes of learning (p. 320). Quality Indicators of Effective Tutoring Tutoring has m ade a significant contributi on to the education of children throughout history. Tutoring adjusts for a ch ilds individual differences a nd creates a more meaningful education for students (Gordon & Gordon, 1990). Th ere were several common themes identified in the literature as quality i ndicators of effective tutori ng. Small student-teacher ratio, collaboration with classroom teachers, tutor qu alifications, and tutor training were identified repeatedly as quality indicators of tutoring. One-to-one tutoring is considered as the most effective form of tutoring (Baker, Rieg, & Clendaniel, 2006; Elbaum, Va ughn, Hughes, & Moody, 2000; Juel, 1996; Lauer et al., 2004; Lauer, 2006; Wasik, 1998). Tutoring programs that provide one-to-one tutoring for at-risk students were proven to increase the academic achievement of students in reading. Tutoring programs that target reading as their primary goal should strive to provide one-to-one tutoring (Elbaum et al., 2000; Lauer et al., 2004). Elbaum et al. (2000) conducted a meta-analysis to assess the effectiveness of one-to-one reading inte rventions for at-risk elementary students with low reading skills. In this meta-analysis of 29 studies, the reading outcomes for 42 samples of students (n = 1,539) were examined. It was disc overed that students w ho received one-to-one tutoring performed at a level that was two fifths of a standard deviation higher than the average level of the control group. An im portant program component of any tutoring program is to keep the student-teacher ratio low so students can obtain the best possible benefits (Sanderson, 2003).

PAGE 30

30 Baker et al. (2006) suggested maintaining a 2:1 st udent teacher ratio or smaller for a successful tutoring program. Collaboration between the childs classroom teacher and the tutor also has been identified as an important aspect of effective tutori ng (Baker et al., 2006; Gordon, 2003; Gordon et al., 2004; Wasik, 1998). Tutoring needs to be coordinate d with classroom instruction to present the student with supplementary tutoring on what is being taught in the classroom with the same materials and taught in the same manner. This en ables the student to have multiple opportunities to work on challenging materials. As students ma ster the material during tutoring, they are more likely to perform better in class. This can be an effective motivator for future tutoring sessions (Wasik, 1998). The tutor is able to provide individual, intense instru ction that a classroom teacher is unable to provide in a group setting and, in most cases, the classroom teacher is more than willing to collaborate w ith the tutor (Gordon, 2003). Close tutor-teacher collaboration will help make the most of each tu toring session (Gordon et al., 2004). Tutoring best practices addre sses the qualifications of tu tors. Experienced, certified teachers make the best tutors (Elbaum et al., 2000; Gordon, 2003; Gordon et al., 2004; Wasik, 1998). Teachers make effective tutors because they are specifically educated to teach, and tutoring enables them to reach out and individua lize instruction, which they are unable to do in a classroom setting. Due to their education and prio r professional experience, teachers can make a major difference in a tutoring situation (Gordon et al., 2004). Research also reveals that trai ning tutors is important in e ffective tutoring (Baker et al., 2006; Elbaum et al., 2000; Gordon et al., 2004; U. S. Department of Education, 1997; Vadasy, Jenkins, Antil, Wayne, & O'Connor, 1997; Wasik, 1998). Extensive training stands out among the features of tutoring programs that produce th e most positive effects (U. S. Department of

PAGE 31

31 Education, 1997). Training must be ongoing and in clude observing tutors and students during tutoring sessions, followed by immediate feedback. Vo lunteer tutors also n eed to be trained to have an understanding of the reading process and the complex cognitive interactions that facilitate reading (Wasik, 1998). Specialized tr aining as a tutor consistently produces higher levels of student achievement than tutors with little or no special trai ning (Gordon et al., 2004). Other quality indicators of effective tutori ng include a long term commitment sustained over the entire school year with the same tuto r working with the same student, intensive and consistent tutoring, as well as us ing tutors that can motivate a nd encourage (Baker et al., 2006; Gordon, 2003; Juel, 1996; Wasik, 1998). The Effectiveness of Other After-school Tutoring Programs There is exis ting research that assesses the effectiveness of other academic tutoring programs provided to students outside their re gular school day. One such research study completed in 2002 evaluates the 21st Century Learning Community Centers, a nationally known after-school program for children in urban and rural communities that includes academic activities. The results of this evaluation have shown that th e program had limited influence on student achievement. At the elementary and middl e school levels, test scores in reading and grades in most subjects were not higher for 21st Century students than for similar students in a control group. This after-school program had no impact on whether students completed their homework or finished assignments to their teachers expectations (U. S. Department of Education, 2003). A meta-analysis of 65 studies on school based tutoring programs co nfirmed that tutoring has some positive effects on raising student academic achievement. Cohen, Kulik & Kulik (1982) conducted this meta-analysis and reported on the academic achievement of tutored students. Of the 65 studies, 45 reported that the tutored student s made achievement gains, but

PAGE 32

32 only 20 of the comparisons reported statistically significant effects fo r students receiving the tutoring. Further analysis of the studies showed that students in more structured programs had larger tutoring effects, as did students in tutoring programs of s horter duration. Effects were also larger when lower level skills were taught and te sted. Tutoring in math produced larger effects than tutoring in reading. Students exhibited larger academic effects when tested on locally developed tests rather than na tionally standardized tests. In 2001, the Corporation for National Servi ce commissioned a report on the outcomes of tutoring conducted by AmeriCorps. AmeriCorps is a network of local, stat e, and national service programs that connects more than 70,000 Americans each year in intensive service to meet the critical needs of our country in education, public safety, hea lth, and the environment. The AmeriCorps study conducted a meta-analysis of their tutoring programs that used adult volunteers. The study sample included 869 students in 68 tutoring programs and assessed their reading gains using the Woodcock Johnson Reading Te st as the pre-test a nd post-test measure. The 869 students were nearly evenly dispersed acro ss grade levels with 294 in first grade, 292 in second grade, and 283 in third grade. The number of boys and girls was almost evenly split in grades two and three, however the first grade student sample was 59% female and 41% male. Ethnically the sample was composed of 51% white students, 23% African-American, 20% Hispanic, and 7% Native American, Pacific Is lander, Asian, or mixed ethnicity. Findings indicated that tutored students at all grade levels that began th e program with below grade-level academic achievement closed the gap and were readi ng at or near their grade-level by the end of the school year as measured by the Woodcock Johnson Reading Test. These gains were large enough to be statistically significant and represented meaningful changes in the students reading performance. The reading gains were the same for students of di fferent ethnic/racial

PAGE 33

33 backgrounds. Surveyed teachers also agreed that tutored students improved their reading to some degree (Moss, Swartz, Obeidallah, Stewar, & Green, 2001). AmeriCorps program directors reported that more than a third of the tutoring programs were conducted in one-to-one tutoring sessions and an additional 30% of the programs reported small group tutoring consisting of two to four stud ents. The length of the tutoring sessions lasted from 30-60 minutes and totaled more than 1.5 hour s a week. The report also stated that tutor training was critical to a succe ssful tutoring program. Moss et al. (2001) identified several effective practices that, when implemented, were essential in showing significant reading gains in the AmeriCorps program. These effective practices include: 1. Tutors receive training both prior to and during the course of tutoring; 2. Tutoring sessions occur at least three times a week; 3. Tutoring activities are monitored and evaluated. (p. 36) In 2004, Lauer et al. conducted a research synt hesis on the effectiven ess of out-of-schooltime strategies for improving the academic perf ormance of low-achieving students. The metaanalysis included 53 studies c onducted after 1984 that concentrated on the effectiveness of a program that was delivered outside the regular school day. Program indicators included the timeframe in which the program was offered (aft er school or summer), grade level of students, focus of the program (academic or academic and social), duration of the program, and student teacher ratio. Results from this research synthesis led to the following conclusions: 1. Out-of-school-time strategies can increas e the achievement level of low-achieving students in reading and math. 2. The times that out-of-school strategies are offered (after school or summer school) do not impact the effectiveness. 3. Student in early grades had the greatest gains in reading. Students in secondary school had the greatest benefit from math strategies. 4. These strategies have positive effects when the focus of the program is academic or academic and social.

PAGE 34

34 5. Administrators should closely monito r the implementation and ongoing student learning to determine the appropriate dur ation for particular strategies and activities. 6. Out-of-school-time strategies that provi de one-to-one tutoring for low-achieving or at-risk students have strong posi tive effects on reading achievement. Current Status of Evaluating SES One of the key provisions of NCLB is the im plem entation of SES, which is designed to provide tutoring or other academic activities to students from low-income families who attend Title I schools that have been identified as not making AYP for three years. As a result of not making AYP, these schools are required to partic ipate in NCLB sanctions which include SES. The SES program is to be implemented in Ti tle I schools in their second year of school improvement, corrective action, or restructuri ng (Smole, 2004). SES is defined as additional academic instruction designed to increase the academic achie vement of students in lowperforming schools. These services are most commonly provided in the form of tutoring and must be provided outside the regular school da y by companies known as SES providers that are approved by each states Department of Education (Sunderman & Kim, 2004). Parents of eligible Title I students select an SES provider from a state-approved list, and the local school district contracts and implements the tutoring pr ogram with the selected provider. SES providers are required to demonstrate that the tutoring prov ided increases student academic achievement in order to retain their status as approved and rema in on the state-approved list. Therefore, states are required to evaluate SES providers to determine their effect on increasing student achievement (Smole, 2004). This topic is addr essed in the SES Non-Re gulatory Guidance under state responsibilities an d monitoring requirements: B. OVERVIEW OF STA TE RESPONSIBILITIES B-1. What is the responsib ility of a State Educationa l Agency (SEA) in providing Supplemental Educational Services ?

PAGE 35

35 The SEA has a number of res ponsibilities in ensuring that eligible students receive additional academic assistance. The SEA must identify providers, maintain a list of providers, and monitor services [Section 1116(e) (4)]. Specifically, the SEA must: 5. Develop, implement, and publicly report on standards and techniques for monitoring the quality and effectiveness of services offered by approved Supplemental Educational Service providers, and for withdrawing a pproval from providers that fail, for two consecutive years, to contribute to increasi ng the academic proficie ncy of students served by the providers (see Section D for additional information) (U. S. Department of Education, 2005, p. 4). D. MONITORING REQUIREMENTS D-4. How may an SEA terminate approval of a provider that is not meeting the statutory requirement to increase students academic achievement? An SEA must use a consistent policy for withdrawing Supplemental Educational Service providers from the State-approved list. The st atute requires an SEA to remove from the approved list any provider that fails, for two consecutive years, to contribute to increased student proficiency relative to State academic content and achievement standards [Section 1116(e)(4)(D)]. In addition, a provider must be remove d from the list if, at any time, it fails to provide Supplemental Educational Services consistent with applic able health, safety, and civil rights requirements (U. S. Department of Education, 2005, p. 19). The review of literature on the evaluation of SES programs reveals there have been research studies and evaluations that have produ ced conflicting results of the effectiveness of SES on increasing student academic growth. Additio nally, in a report delivered to Congress in August 2006, the Government Accountability Office concluded that no definitive evaluation had been done to measure the effects of SES on st udent academic achievement (U. S. Government Accountability Office, 2006). Although states are required by NCLB to ev aluate SES providers annually, surveys indicate that few states have developed and implemented evaluations (Reid, 2004). Critics contend that there are no thorough evaluations of SE S programs in place in any state and that the evaluations th at do exist rely on the most ba sic evaluation methods (Sunderman & Kim, 2004). Due to the absence of evaluati on procedures, SES provide rs have no evaluation data to show student academic grow th on state assessments as is required of states and districts by NCLB. Current state monitori ng of SES providers consists of relying on questionnaires,

PAGE 36

36 surveys, and data submitted by the providers themselves. Some states depend on SES providers to evaluate their own effectiveness by using th eir preand post-test assessments to measure increased academic growth of students (Reid, 2004). Lack of Funding One of the reasons cited for the lack of evaluation is that stat es lack the capacity and funds to develop a m onitoring system for tutoring (Rei d, 2004). NCLB requires states to ensure that Supplemental Educational Services are of high quality, research based, and designed to help eligible children attain proficiency in meeting th e states academic achievement standards. States and districts have been overwhelmed with desi gning an implementation plan for providing SES in schools and communities nationwide (Cohen, 2006). Due to the increasing enrollment and mandatory NCLB requirements, the SES program has created administrative and management challenges at both state and district level. States and district s are forced to assume these additional responsibilities w ith no additional funds to cover them (Ascher, 2006). NCLB places a financial burden on local and state administrators with no additional administrative funds designated to cover the cost s of implementing and evaluating SES (Ascher, 2006; Burch, 2007). District administrative costs for implementing the SES program are significant considering that 40% of all urban dist ricts are required to offer SES (Ascher, 2006). For example, the SES administrative budget of th e Chicago School District was more than $2 million in the 2004-05 school year (Sunderman, 2006). Further evidence of escalating SES administrative costs are demonstrated in Duval County Public Schools in Jacksonville, Florida, where these costs have increased from $700,000 in 2006-07 to a projected amount of $1.1 million for the 2007-08 school year. This is in addition to the mandatory 20% setaside and amounts to 3.5% of the total Title I allocation that must be used to administer SES that, in past years, would have been allocated to Title I schools (Duval County P ublic Schools, 2007).

PAGE 37

37 An example of the administrative challenge s presented by SES can be found in Oakland, California, where 500 parents failed to select an SES provider on their free tutoring application. Per federal SES guidance, the SES coordinator was required to contact these parents to select one of the 25 providers on the district s state-approved list. Duplicat e applications also presented a problem in Oakland where parents submitted one a pplication at the districts SES Provider Fair and then submitted a duplicate application with a nother provider who visited the students home. Again, the SES coordinator had to contact thes e parents to determine their provider choice. Monitoring attendance, verifying student eligibi lity, and tracking which students are enrolled in which programs are some of the administrative burdens experienced by districts nationwide (Rentner et al., 2006). The Increasing Number of SES Providers The ever increasing num ber of approved SES providers nationwide heightens concern over the lack of evaluation and in adequate monitoring (Cohen, 2006). As of February 2007, states reported approving at total of 3,223 SES providers (U. S. Department of Education, 2007). Stullich et al., (2006) reported this was over three times as many SES providers than had been approved just two years earlier, when the approv ed list consisted of 997 providers. Many of the approved providers are leaders in providing educational services, but others have little or no track record of effectivene ss in tutoring low-income, lowachieving students (Cohen, 2006). In 2004, the number of approved providers varied per state with a high of 216 in New York to a low of 3 in Hawaii (Reid, 2004). The Bush Administration remains an enthusiastic supporter of the program and has pushed states to increase the number of approved SES providers, even though there is little evidence of their effectiveness (Sunderman, 2006). SES providers can include nonprof it, for-profit, and faith-based organizations, as well as school districts, charter school s, and private schools (Kasmin & Farmer, 2006). An annual

PAGE 38

38 review conducted in 2005-06 by the Center on Education Policy, a national, independent advocate for public education, reported 54% of all state-approved providers were for-profit, 21% were nonprofit not affiliated with a religious group, 9% were dist ricts, with the remaining 16% made up of public entiti es, private faith-based organizations and other types of organizations (Ascher, 2006). A typical tutoring program provides 30 hours of free tutoring (Hess & Finn, 2004a). The mode of instruction can vary, incl uding one-to-one tutoring, small group instruction and online tutoring. The frequency or intensity of service can also vary because NCLB has no set requirements except that help with homewo rk is prohibited (Kasmin & Farmer, 2006). Implementation in Rural Districts and Districts w ith Low Enrollment Rural districts and districts with low enroll ment face a different problem in attracting a sufficient number of SES providers to serve their eligible studen ts, thereby creating a lack of services in their districts (Burc h, 2007; Fusarelli, 2007). These dist ricts disclosed that they failed to implement SES due to a lack of state-appr oved SES providers, and in one case, there was no approved provider within 200 miles of the school district (U. S. Government Accountability Office, 2006). In the Grant Union School district in California, only 26 of the hundreds of stateapproved providers were interested in serving studen ts in this rural distri ct and only 11 attended the SES Provider Fair for parents. SES providers require a minimum number of students to make SES a profitable venture. A similar problem occu rred in Kansas City, Missouri. Although Kansas City Schools are considered an urban school dist rict, half of the SES providers on the stateapproved list removed themselves from providing services in this school district due to low enrollment because they would not be profitable. The only solution at this time for rural districts or districts with low enrollment is to contract with online SES providers. This presents additional barriers to serving students such as unreliab le connectivity and limited access to computers (Schwartzbeck, 2005).

PAGE 39

39 Highly Qualified Teachers Although NCLB m andates highly qualified teachers in ever y classroom taught during regular school hours, the law does not specify any education or certification requirements for SES tutors (Ascher, 2006; Burch, 2007; Kasm in & Farmer, 2006). Districts have expressed concerns about programs that use tutors who lack teaching credentials (Sunderman, 2006). Critics say that this is in conflict with the in tention of NCLB, which strives to match the most qualified teachers with students who have th e most academic need (Kasmin & Farmer, 2006). Ascher (2006) found that although most SES tu tors are certified teachers employed by the district, some were college students with no teaching experi ence, and 7% were high school students. The American Institutes for Resear ch and the Education I ndustry Association found that a majority of SES tutors are certified t eachers, while others are high school students and college graduates without prior tutoring experience (B urch, 2007). The U. S. Department of Education reported that 15 of 24 the providers in terviewed for this case study of supplemental services stated that they require their tutors to be certified teachers, the remaining 9 did not require certification. For example, Huntington Learning Center does not require teacher certification, while Princeton Review hires certified teachers employed in non-SES schools (Anderson & Laguarda, 2005). Although training progr ams, typically 4 to 20 hours, are provided by some SES providers, not all providers train their tutors. It shoul d also be noted that some, but not all, providers ev aluate their tutors. Increasing Enrollment Due to the increasing number of schools being required to offer SES programs, the number of Title I students partic ipating in this free tutoring progr am increases dramatically each school year. To fund this rapidly increasing prog ram, local school districts are required to set aside 20% of their annual Ti tle I allocation (Cohen, 2006).

PAGE 40

40 The lack of evaluation has created boon times for SES providers. The number of students receiving SES in 2004-05 amounted to over 400,000 students. This number more than tripled from the previous two years and student par ticipation is still increas ing rapidly ("Beyond the pale: SES provisions emerge as 'loose cannon' reform," 2006). In the 2005-06 school year, the number of students receiving this free tutoring program again increased to almost 500,000 students (U. S. Department of Education, 2007). A substantial amount of money is at stake in funding this program. Per NCLB, a portion of the 20% setaside of Titl e I funding is to provide SES in sc hool districts (Borja, 2006). Chicagos SES program served 60,000 students in 2004-05 and cost the district almost $50 million (Chicago Public Schools, 2005). In 2005, SES providers nationw ide received $400 million in federal money for tutoring services (Borja, 2006). Under full implementation nationwide, SES could amount to more than $2.5 billion of the almost $13 billion annua l Title I grant funds (Ascher, 2006; Cohen, 2006). SES providers contend that they are eager to have their programs evaluated (Reid, 2004), but critics state that they operate as l oose cannons in school districts across the country with no accountability ("Beyond the pale: SES provisions emerge as 'loose cannon' reform," 2006). SES Coordination with Classroom Curriculum Concerns ov er the lack of accountability al so center on the lack of requirements to coordinate SES with classroom curriculum or for providers to communicate with school personnel. SES provides no procedure for provide rs to receive academic information from teachers or vice versa. Critics c ontend that this lack of coordi nation between the school and SES providers reverses a districts approach to a coordinated and comprehensive reform of Title I schools. Some district officials have expressed concern that the curriculum of SES providers was at odds with their efforts to create a complete academic program for students in these schools.

PAGE 41

41 They feel that SES provisions weaken the abil ity of their schools to develop a consistent instructional program by requi ring tutoring to be offered out side the school day, further discouraging coordination of the SES provider and classroom teacher ("Beyond the pale: SES provisions emerge as 'loose cannon' reform ," 2006; Sunderman, 2006). In Tennessee, many teachers reported that they were not even awar e that their students were being tutored by SES providers, which meant that there was no coll aboration regarding student academic needs. Teachers also expressed frustration at the lack of collaboration with providers in developing student learning plans (Potter, Ross, Paek, & Mc Kay, 2006). Experts stated that academic gains would be stronger if providers would collaborate with the students teache rs. They also contend that SES will not meet its potential as long as the SES curriculu m is disconnected from what is being taught in the classroom (Viadero, 2007). Al though there is no resear ch that supports the methodology behind SES, there is re search that suggests that stude nts benefit in schools that support a comprehensive approach to educating stude nts. This includes Title I curriculum that is aligned with the regula r curriculum and provides services that support the co re instruction (Sunderman & Kim, 2004). The curriculum used during tutoring is up to the discretion of the SES provider and todate almost nothing is known about what stude nts are actually being taught during tutoring sessions beyond what is listed in their state application, on their we bsite, or in their marketing materials. Providers most commonl y describe their curriculum as literacy skills and problem solving skills. While NCLB encourages SES pr oviders to align their curriculum with state standards, it clearly forbids states and districts from attempting to influence a provider to implement a certain curriculum (Burch, 2007).

PAGE 42

42 There is also speculation that SES decr eases accountability by focusing on short term individual student achievement rather than fo cusing on a broad range of school level outcomes based on state standards. The focus is narrowed further by not only servin g individual students, but serving only students who request the free tutoring services (Sunderman, 2006). SES Student-Teacher Ratio The student-teacher ratio is established by SE S providers and can vary significantly by provider, district, and locati on (Burch, 2007). Although research strongly suggests that the greatest academic gains can be obtained from on e-to-one tutoring, curren t state and district evaluations have found that most SES programs are significantly larger (Baker et al., 2006; Burch, 2007; Elbaum et al., 2000; Juel, 1996; Laue r et al., 2004; Lauer, 2006; Wasik, 1998). The Education Industry Association reports that SES is most commonly offered in small groups of less than 10 students, but studies have shown that some providers have ratios of 1:10 or 1:12. Providers with higher enrollment rates tend to ha ve larger groups sizes of 1:8 to 1:10, while providers with a smaller enrollment tend to tutor in groups of 1:1 to 1:3 (Burch, 2007). Another noteworthy problem is that by fundi ng SES with the 20% setaside of Title I funds to pay providers, the allocations to schools have been drastically reduced. Rather than use these funds to focus on school wide improvement tie d to state standards, SE S uses these funds to focus on individual student academic gains for the students whose parents elect this free tutoring program (Sunderman, 2006). Existing Research on SES In 2004, The Center for Education P olicy conducted a study of the implementation of SES. Six states and nine districts with SES implementation experience were selected. The study consisted of telephone surveys to state administra tors in charge of implementing SES statewide and site visits to the nine dist ricts to interview dist rict staff, visit schoo ls, conduct teacher and

PAGE 43

43 parent focus groups, and hold personal interviews This study reported that although parents had been satisfied with the services delivered to th eir children, they wanted the services to start sooner. Parents were concerned that their children were not re ceiving the maximum benefit of the tutoring due to the amount of time it took to actually start tu toring after they had chosen a provider. In some districts several weeks passe d before students starte d receiving tutoring. The study also stated that it was t oo early to assess the effectiveness of SES on student academic achievement and that states needed guidance on how to assess SES effectiveness (Anderson & Weiner, 2004). In 2005, the U. S. Department of Educati on commissioned a follow-up study to the 2004 Center for Education Policy study. Once again, six stat es and nine districts were included in this case study of SES. However, four of the nine districts included in th e original study were no longer providing SES because they no longer had schools in their second or higher year of school improvement. To compensate for these four district s, four new districts in two additional states were added. This study, conducted by Anderson & Laguarda (2005), noted a lack of coordination between the SES provider and the teacher. Many t eachers in the study reported that they did not know which of their students were participating in the SES program and felt that collaboration with the tutors would help thei r students. Teachers and parents al so reported that they did not receive progress reports on their st udents, although providers of th ese same students reported that they sent regular progress reports to them. Parent s also objected to the inst ructional approach that providers took in teaching their children. Additi onally, these parents stat ed that they were disappointed with the quality of the tutoring pr ovided and felt their children had made no academic gain in reading and math. They also questioned whether just a few hours of tutoring could actually make a difference for their childr en. This U. S. Department of Education case

PAGE 44

44 study concluded that, as a result of the lack of evaluation of SES, very few providers have been removed from state lists. In a few instances, provi ders were removed for fi nancial irregularities or because they did not offer the specified tutori ng services to their students, but there were no providers removed from state-approved lists because the state had determined that the quality of their tutoring program was not adequate. Burch (2007) reported that 15 of the 30 states (50%) who responded to a survey conducted through The Edu cation Public Interest Center stated that their state actually removed a provider from the st ate list because of a contract infraction or a lack of effectiveness in the SES services provided. Minneapolis Public Sch ools SES Effectiveness In 2005, Minneapolis Public Schools (MPS) conducted an internal SES study on the effectiveness of the program in their distri ct. Dr. David Heistad of the MPS Research, Evaluation, and Assessment Office conducted this evaluation because he felt the state was illequipped to conduct a controlled study. He stat ed that it may be incumbent upon large urban districts to conduct their own evaluations of the SE S program in an effort to provide feedback to its stakeholders, including schools, parents, SE S providers, and the Minn esota Department of Education. Dr. Heistad conducte d two different studies. The first compared reading achievement gains for students in each SES program on the Northwest Achievement Levels Tests (NALT). The second study matched students on the NALT pre-tests and student demographi c characteristics and calculated achievement on the Minnesota Comprehensive Assessments (MCA) for the larg est SES provider with a control group of students who received no tutoring. Results of the first study found no st atistical significant difference among SES providers on NALT annual reading gains. No provider with at least 10 students receiving tutoring came close to averagi ng 100% of expected scale score growth based on the NALT norms. For example, Education Sta tion tutored 561 students who averaged 71% of

PAGE 45

45 a years growth, while 92 students tutored by Newton Learning Services averaged 67% of a years growth. Students in grades 3 and 7 who did not receive SES achieved at a higher level than students they were paired with who received SES from Catapult Learning, outscoring them by 19 points and 6 points, respectivel y. On the other hand, Catapults 5th grade students outscored their matched sample by 4 points. The achievement differences in both instances were not statistically significant. Overall, the aver age growth for students in SES programs was 66% of the national norm growth for all students (Burch, 2007; Heistad, 2006). Results from the second study found no statistically significant difference in MCA reading achievement for students who received tutoring fro m their largest SES provider compared to the control group who received no tutoring. Dr. Heistad stated that the results of the second study were disappointing when given the fact that 564 students who received tutoring from their largest SES provider spent an additional 37 hours of instruction in the SES program. The report concluded by saying that the SES program must be redesigned to better serve the needs of Minneapolis students. Chicago Public Schools SES Effectiveness Another evaluation of SES tutoring progr am s was completed in 2005 by the Chicago Public Schools. This evaluation was done to de termine if students who received SES had made sufficient academic gains. Test score data from the Iowa Test of Basic Skills from 2003-04 and 2004-05 were analyzed and it was determined that students who received at least 40 hours of tutoring in grades 4 8 achieved higher gain sc ores than students who did not receive SES. It was also determined that students who received 40 total hours of tutoring achieved higher math and reading scores than student s who received less than 40 ho urs (Chicago Public Schools, 2005).

PAGE 46

46 SES was again evaluated in the Chicago Pub lic Schools in 2006. The results showed that tutored elementary students made only slightly higher gains in reading than students who were eligible for tutoring, but did not receive SES. In the 2005-06 school year, SES was offered in 324 schools by 41 private firms who were approved to provide SES by the Illinois Department of Education. Chicago Public Schools spent $50 million for 56,000 students who received SES and officials questioned whether the tutoring services were worth the money. Officials stated they wanted to see higher achievement gains for th e investment that was made (Grossman, 2007). A policy brief from the Education Public Interest Center (EPIC) at Arizona State University stated that although the studies fr om Minneapolis and Ch icago made important contributions to the evaluation of SES, they bot h contained methodological limitations that left many aspects of the program unevaluated. In the Ch icago study, the evaluation failed to consider differences among student populations It was also stated that th e higher gain scores may have resulted from an academic program other than SES tutoring. Selection bias could also account for the higher gain scores. For example, parents who enroll their students in SES programs tend to be better educated parents and take a more active role in their childs education. EPIC contends that since these possi bilities are not considered, the evaluation results are open to question (Burch, 2007). EPIC cited similar limitations in the de sign of the Minneapolis study. Minneapolis compared their students with a nationally norm ed population, when in fact the Minneapolis students were disproportionately low income, with 73% of the students classified as minority students, with 76% being black, Hispanic, Ameri can-Indian or Asian. Therefore, while the study compared its results to a national norm, significa nt differences in the two populations make the findings questionable (Burch, 2007).

PAGE 47

47 Los Angeles Unified School Di stricts SES Effectivenes s In February of 2007, Los Angeles Unified School District released a study wherein they duplicated a January 2006 study of the effectiv eness of the 2004-05 SES program on increasing the academic achievement of participating students. The January 2006 study found that SES had no significant impact on student achievement on th e California Standards Te st in language arts and math. The 2007 study analyzed the academ ic achievement of the 14,759 students who participated in SES in the 2005-06 school year. The results of the study showed that students who attended SES had a statistically higher, ye t non-significant increase in their academic achievement than did students who qualified for SES but did not participate. The SES program was more effective for elementa ry students than secondary stude nts and some providers had a greater effect on increasing student academic achievement, while others had no effect (Rickles & Barnhart, 2007). Tennessees SES Effectiveness Study The effectiveness of SES was assessed stat ewide in Tennessee in a study com pleted by the Center for Research in Educational Policy (CREP) in March 2007. The goal of this study was to evaluate the academic achievement of students in grades 4 through 8 who received SES in the six school districts during the 2005-06 sc hool year that were required to offer these services. The results of the study by CREP yielded no statistica lly reliable effects for increased student achievement from the 33 SES providers included in the study. The results show no statistically significant differences between students who re ceived tutoring and those who did not. The evaluation of two SES providers in this st udy resulted in negative effects on academic achievement for students who enrolled in their programs (Potter et al., 2006).

PAGE 48

48 Large Urban Districts SES Effectiveness The U. S. Departm ent of Education released a report in June 2007, where data from the 2004-05 school year were analyzed from nine la rge urban school distri cts that had a large number of students participating in the two choice options. The nine districts were Baltimore, Chicago, Denver, Long Beach, Los Angeles, Palm Beach, Philadelphia, San Diego, and Washington, DC. The first of the two choice options allowed parents to transfer their children to another school in the district that had not been identified for school improvement and the second choice option gives parents the opport unity to enroll their children in SES. Of the nine districts included in this study, two districts were elim inated in the SES academic achievement portion because they had less than 100 students partic ipating in the tutoring program. Across the remaining seven districts, student s who participated in SES scored better in reading and math in the first year and the higher academic achievement was deemed statistically significant. Students also scored even higher in the subsequent year of SES (Zimmer, Gill, Razquin, Booker, & Lockwood, 2007). Zimmer et al. (2007) stated that these findings were based on a small number of school districts that are not na tionally representative, and the results should not be viewed as representative of the effectiveness of SES nationa lly. The results are important because this study is one of the first to analyze the effectiveness of SES in multiple districts. Contradictory evidence was presented in a study for the Los Angeles Unified School District. Rickles & Barnhart (2007) stated that, in the 2004-05 school year, they found no significant impact on the achievement of Los An geles Unified School District SES students on the California state test. Rickles & Barnharts repo rt contradicts the Zimmer et al. (2007) report where it was stated that, as one of the nine districts that was included in their study, Los Angeles Unified School District students who participated in SES made statistically significant academic gains in reading and math in the 2004 -05 school year.

PAGE 49

49 SES Evaluation Progress Although states report that system s have been developed and implemented to evaluate and monitor SES providers, as of 2005, 15 states had not established any monitoring process, 25 states had not established any standards for evaluating provider effectiveness, and none had finalized their evaluation process. Seventeen (17) states plan to use student achievement on state assessments as their evaluation tool for SES pr oviders, although only one state planned to use a control group. The most common evaluation me thod proposed by 25 states was to survey districts about provider effectiveness, while 18 st ates plan on using provide rs reports on student academic progress (Stullich et al., 2006). In a policy brief from EPIC at Arizona State University, 15 of the 30 states that responded to a survey stated that they did not use any form of test data to evaluate SES provi der quality, but instead relied on annual site visits (Burch, 2007). Several states are making progress on eval uating SES providers. Louisiana has made progress on its SES evaluation by using test scor es from state assessments to monitor the effectiveness of individual SES providers on student achievement. To accomplish this task Louisiana developed a data system for all of its programs that take place outside the regular school day. This data system comm unicates with the districts en tire student data system and compares the state test scores of students in after-school programs with the scores of those who are not (Gewertz, 2005). Illinois, Ohio and Florida also were scheduled to implement their SES evaluation program in the summer of 2006 (Pines 2006). Illinois and New Jersey reported to the United States Government Accountability Office that they were in the pr ocess of improving their data collection system to effectively capture a nd examine needed data to determine SES provider effectiveness. Several states said that while they have collected needed data to evaluate providers, they have not complete d the evaluations. Other states re ported that additional federal guidance on implementing effective evaluation m odels was needed because developing and

PAGE 50

50 implementing effective evaluation models was time-consuming and costly (U. S. Government Accountability Office, 2006). Although state monitoring of SES implemen tation has been limited, states reported making gains in monitoring both districts and prov iders in 2005-06. Districts have also increased monitoring efforts of SES providers serving their students. New Mexico and Tennessee were the only two states that had final or draft SES evaluation reports that attempt to assess the effectiveness of SES providers on student academ ic achievement, but these evaluations did not provide conclusive results. Although states are required to remove providers from approved lists if they fail to increase student achievement for two years, most states have failed to implement this section of NCLB (U. S. Government Accountability Office, 2006). The Center on Education Policy (CEP) publishe d a report in 2007 on the status of state implementation of SES. In their annual surv ey regarding NCLB implementation, CEP included several questions regarding th e states role in overseeing SE S. Two key findings center on a states ability to monitor SES providers effectively. It was found that 38 states were unable to comprehensively monitor the quality and effectiv eness of providers and only 10 states reported that they adequately monitor SES. States re ported that the biggest barriers to effective monitoring were a lack of staff and inadequate funding at the district le vel to administer the program (Minnici & Bartley, 2007). The Education Industry Association (2005) has attempted to make SES providers themselves responsible for monitoring their tu toring programs by creating high ethical standards that its over 500 members have adopted as a c ode of professional conduct. The Supplemental Educational Services Code of C onduct (2005) states th at the importance of SES activities and the complicated interactions of SES make it vital th at members adhere to the highest standards of

PAGE 51

51 professional conduct and ethics. This voluntary Code of Conduc t describes key organizational behaviors and policies th at are to direct SES providers. It includes: (a) guidelines on how providers should conduct business and fulfill respons ibilities, (b) directiv es to consistently implement the NCLB Supplemental Educational Se rvices provisions and promote full access to SES services, and (c) describes best practices for implementing SES. The Supplemental Educational Services Code of C onduct is now a part of the state application pro cess in several states including Connecticut, New York, New Jers ey, Maryland, Illinois, Oh io, Florida, Georgia, and New Mexico (Code of Professional Conduct, 2006). Challenges in Evaluating SES Many challenges are p resented in the SES lite rature, but there are few suggestions to guide states and districts on ev aluating the benefits of SES, particularly in improving the education of low-income and minority students who are to benefit from this program (Sunderman, 2006). Three-fourths of states repo rted that they are experiencing challenges evaluating SES, including designing evaluations to determine the academic gain of students, having the personnel and funds to analyze data, and developing data systems to monitor SES information (U. S. Government Accountability Office, 2006). Due to the fact that states do not have the capacity or funding to monitor and evaluate SES providers, many students end up in sub-standa rd programs (Rees, 2006). In March of 2006, CEP published their fourth annual report, From the Capital to the Classr oom: Year 4 of the No Child Left Behind Act. Their findings were based on a surv ey of all 50 states, a nationally representative survey of 299 school districts, cas e studies of 38 geographi cally diverse districts and 42 schools, three national foru ms, and six special analyses of critical issues in implementing NCLB. According to survey results, states cons ider the greatest challenge to implementing SES is monitoring and evaluating the quality and e ffectiveness of the tuto ring services provided.

PAGE 52

52 Forty-one (41) states and 51% of school districts called this a moderate to serious challenge. Current federal regulations restri ct the ability of school districts to establish rules for SES providers, but districts ultimately are responsible for allocating federal funds to providers and increasing the academic performance of SES stude nts. In response to these expectations, CEP has made recommendations to help NCLB work better. One of these recommendations calls for states and school districts to be given sufficient resources and authority to evaluate and monitor SES providers effectiveness in raising student achievement (Rentner et al., 2006). Forty-nine (49) states reporte d that they used the criteria required by NCLB and federal guidance to review and approve SES providers. These criteria include the assurances that providers are financially sound, have a record of effectiveness, u tilize research-bas ed strategies, provide services that are consiste nt with district instruction, an d adhere to health, safety, and civil rights laws (Minnici & Ba rtley, 2007). States also reported th at the effectiveness of SES is jeopardized because the federal government pushes states to increase the pool of providers with little evidence of program effectiv eness. Districts have raised additional concerns about what they consider to be unethical practices some providers have used to attract students to their tutoring program (Sunderman, 2006). Providers have used recruiti ng practices to entice parents to enroll students in their program that are unrelat ed to the quality of their services. In Chicago, parents were given over 40 SES providers from wh ich to choose from; some of these providers attempted to influence parents by offering cash payments, video game consoles, or other enticements for enrolling their children in th e providers program (Rees, 2006). The use of incentives by SES providers had become such a problem in Florida that the practice was specifically addressed in a stat e statute that was approved by the Florida legislators in 2006.

PAGE 53

53 Florida State Statute 1008.331 Supplemental Educa tion Services in Titl e I schools states the following: (1) INCENTIVES. A provider or school district may not provide incentives to entice a student or a student's parent to choose a provider. After a provider has been chosen, the student may be awarded incentives for performa nce or attendance, the total value of which may not exceed $50 per student per year ( The 2006 Florida State Statutes 2006). Recommendations to Strengthen and Evaluate SES Nothing is gained by avoiding the question of the effectiveness of NCLB or naively assuming that well-intentioned efforts will be sufficient. As noted by Michael Kirst (2004), a veteran policy analyst, it took more than 10 years and many legislative and administrative adjustments before the initial Title I program in the 1965 Elementary and Secondary Education Act met the legal intent of Title I. This is typical of aggressive new federal programs. These new programs rarely work well at first; instead th ey bring a plethora of unforeseen problems, consequences, loopholes, and impractical features. In fact, NCLB is immensely more ambitious than the original ESEA because its goals, methods, and reforms do not conform to current educational practices in Americas public school s. NCLB requires states and districts to participate in unfamiliar, untried activities, such as SES. The important question is whether SES, in its current form, has a realistic chance of be ing effective given time and practice or whether the law is impractical and unrealistic. If the la tter is true, then NCLB needs rethinking and revision, not just persistence and adjustments (Hess & Finn, 2004a). A wealth of recommendations to strengthen and evaluate SES are presented in existing literature. Recommendations from Steven Pines, Executive Director of the Education Industry Association (2006) include the following: 1. States should directly invol ve SES providers and the local education agency in the development of evaluation polic ies and specific methodologies.

PAGE 54

54 2. States should ensure the fairness of the system particularly when the local education agency is an approved provider. 3. States should separate the effects of SES from other variables that might affect a students achievement. 4. States should use extreme care or avoi d using standardized tests for purposes other than for which they were orig inally deemed reliable and valid. 5. States should include data from providers pre-and post-testing of SES students in their overall evaluation of provider effectiveness. Hess & Finn (2004) made recommendations to strengthen and improve the existing SES program. These recommendations include the following: 1. Guidance must be provided to states to ensure sound and rigorous evaluations including careful planning, time, and adequate funding. 2. Better federal data must be provided on how SES is being utilized and how it is working. For example, data might in clude tracking the number of students receiving services, or which providers are serving how many students. This information is vital and could be trac ked by the National Center for Education. 3. Districts must be given additional money to support the administration and evaluation of SES programs. 4. School districts need to function as either administrators of SES or SES providers, not both. Sunderman & Kim (2004) recommend that the federal government terminate the SES program as a mandated sanction of NCLB fo r poorly performing Title I schools, while suggestions made by the United States Government Accountability Office (2006) call for states to be provided with needed a ssistance regarding methods for the evaluation of SES, including additional and clearer guidance. States also have requested a forum to share SES best practices with peers, such as meetings and conferences th at could be facilitated by the federal government. There is also a request for data collection on th e amount of Title I funds spent on SES by districts (U. S. Government Accountability Office, 2006).

PAGE 55

55 Summary NCLB legislation requires stat es to ensure that Supplementa l Educational Services are of high quality, research based, and designed to help eligible children attain proficiency in meeting the states academic achievement standards, yet the federal government has not partnered with states to develop an adequate evaluation process. Presently, $2.5 billion is bei ng diverted from Title I funds annually to pay for this free tutoring program with virtually no accountability. While states are required to remove providers from their approved lists if they fail to increase student achievement for two years, states have failed to implement this section of NCLB. The fe w states that are evalua ting SES providers have not yet produced reports that provide a decisive evaluation of their effect on student academic gains. States also have expressed concern that additional federal guidance on implementing effective evaluation models is needed because developing and implementing evaluation models is time consuming and costly. In fact, 75% of states reported that they are experiencing challenges evaluating SES, including designing ev aluations to determine the academic gains of students, having the personnel and funds to analyze data, and developing data systems to monitor SES information (U. S. Governme nt Accountability Office, 2006). Current literature provides evidence that the Su pplemental Educational Services provision of NCLB has been in effect nationwide for several years with mi nimal and conflicting evaluations of its effectiveness on increasing st udent achievement despite the NCLB requirement for evaluation. The literature echoes a resounding de mand to evaluate the effectiveness of SES to determine the level of academic gain of Title I studen ts, if any, that this pr ogram facilitates. With billions of dollars set aside annua lly to fund the SES program, it is critical that it be evaluated and adjusted, if necessary.

PAGE 56

56 Table 2-1. Quality Indicators of Effective Tutoring Indicator 1: The student-teacher ratio should be kept low. A 1:1 or 2:1 ratio is suggested for a successful tutoring program. Indicator 2: Collaboration be tween the childs classroom teacher and the tutor has been identified as an important aspect of effective tutoring. Indicator 3: Experienced, certified teachers ma ke the best tutors. Teachers make effective tutors because they are speci fically educated to teach. Indicator 4: Extensive training of tutors produces the most positive effects. Training must be ongoing and includes observing tutors a nd students during tutoring sessions. Indicator 5: A long term commitment sustained ov er the entire school year with the same tutor working with the same student produces positive results. Tutoring should be intensive and consistent.

PAGE 57

57 CHAPTER 3 STUDY OVERVIEW W ith the reauthorization of the Elementary and Secondary Act of 1965, the No Child Left Behind Act of 2001 initiated unprecede nted educational reform to en sure that all children have the opportunity to obtain a high-quality education and reach proficiency on state standards and assessments by increasing the accountability of states and districts (Stullich et al., 2006). Supplemental Educational Services is one of the el ements of this historic educational legislation and is defined as academic activities provided outside normal school hours and typically offered as an after-school tutoring program. The legislat ed goal of the SES program is to increase the academic performance of low-income students w ho attend Title I schools that have not made adequate yearly progress for three or more year s. An SES program typically includes tutoring in math, reading, and/or language arts, but specif ically prohibits homework help (Smole, 2004). NCLB requires states to ensure that Supplemental Educational Services are of high quality, research based, and designed to help eligible children increase their academic achievement to attain proficiency in meeting the states academic achievement standards. This free tutoring program is offered by SE S providers who include nonprofit, for-profit, and faith-based organizations, as well as school di stricts, charter schools, and private schools that have been approved by state educational agencies (Kasmin & Farmer, 2006). SES is a parentdriven program wherein providers are selected by parents fr om a state-approved list. SES providers remain on this approved list unless they fail to demonstrate student academic gains for two consecutive years. In order to demonstrate academic gain, providers are required to be evaluated annually by the state e ducational agency. This evaluati on is the specific responsibility of the state, but currently no definitive evalua tion has been completed on the effectiveness of SES on raising student academic achie vement in any state (Smole, 2004).

PAGE 58

58 Annually as much as $2.5 billion is setaside nationwide to provide SES in Title I schools for a program that has little or no research to prove its effectiveness in increasing academic growth for enrolled students (Ascher, 2006; C ohen, 2006). While states are required to remove providers from their approved lists if they fail to increase student achievement for two years, states have failed to implement this section of NCLB. According to the U. S. Government Accountability Office (2006) the few states that ar e evaluating SES providers have not generated reports that provide a thorough ev aluation of their effect on incr easing academic achievement for Title I students. States have expressed concern that additional guidance is needed from the United States Department of Education to construct and im plement adequate evaluation models (U. S. Government Accountability Office, 2006). At this time the federal government has not partnered with states to develop an adequa te evaluation process to ensure that these requirements are being met (U. S. Government Accountability Office, 2006). Furthermore, states contend that development and implementation of evaluation m odels is time consuming and costly. In fact, 75% of states reported that th ey are experiencing challenges ev aluating SES, including designing evaluations to determine the academic gains of students, having the expertise and funds to analyze data, and developing data systems to examine SES information (U. S. Government Accountability Office, 2006). Purpose of the Study The purpose of this analysis was to assess th e effectiveness of the SES program in Florida Title I urban district schools that were required to offer th is program during the 20062007 school year.

PAGE 59

59 To assess the effectiveness of SES, the follo wing research questions will be addressed: 1. Do students who participated in Supplement al Educational Services exclusively in math for 10 or more hours demonstrate higher developmental scale score gains, as determined by the FCAT, compared to students in a control group? 2. Do students who participated in Supplemental Educational Services exclusively in reading for 10 or more hours demonstrate higher developmental scale score gains, as determined by the FCAT, compared to students in a control group? 3. Do students who participated in Supplemental Educationa l Services in both math and reading for 10 or more hours demonstrate high er developmental scale score gains, as determined by the FCAT, compared to students in a control group? 4. Do students tutored by the three major SES pr oviders in each distri ct (i.e., those that provided tutoring for the largest number of students) for 10 or more hours demonstrate higher developmental scale score gains, as de termined by the FCAT, compared to students in a control group? Research Hypotheses The study was guided by the following research hypotheses: Null Hypot hesis 1. There is no statistically signifi cant difference between 2006 FCAT developmental scale scores as the pre-test a nd 2007 FCAT developmental scale scores as the post-test for students in Title I schools rece iving at least 10 hours of tutoring in math. Null Hypothesis 2. There is no statistically signifi cant difference between 2006 FCAT developmental scale scores as the pre-test a nd 2007 FCAT developmental scale scores as the post-test for students in Title I schools receiv ing at least 10 hours of tutoring in reading. Null Hypothesis 3. There is no statistically significant difference between 2006 FCAT developmental scale scores as the pre-test a nd 2007 FCAT developmental scale scores as the post-test for students in Title I schools receiv ing at least 10 hours of tutoring in math and reading. Null Hypothesis 4 There is no statistically significant difference between 2006 FCAT developmental scale scores as the pre-test a nd 2007 FCAT developmental scale scores as the

PAGE 60

60 post-test for students receiving at least 10 hours of tutoring from the three SES providers who tutored the largest number of students in each district. Overview of the Method This study utilized existing FCAT sc ores to a ssess the effectiveness of SES in grades 4 10 for six large urban Florida school districts in an effort to align this study with the anticipated evaluation of SES by the State of Florida. A memo dated December 21, 2007, stated that the Florida Department of Education will use FCAT scores in the 2006-07 ev aluation of companies providing SES to students in grades 4-10 (L Dukes, personal communication, December 21, 2007). Included in this study are six of the largest ur ban Florida school distri cts. The six districts have been randomly assigned numbers one thro ugh six for identifica tion purposes. This study was limited to these large urban Florida school di stricts due to the unavailability of statewide data. 2006-07 Participating Di strict Statistics District One has a total student enrollm ent of 362,050. The number of students who receive free or reduced price l unch is 221,229 or 61% of the student population. Title I funds are provided to schools with a 63% or higher rate of students eligible for free or reduced price lunch. Of the 438 schools in the district, 47% (n = 204) are Title I schools that are required to offer SES. The ethnic m akeup of the student population di strict wide is comprised of 61% Hispanic, 27% Black, 9% White, and 3% Other, which incl udes American Indian, Asian, and Multiracial categories. In grades 9-12, there are a tota l of 8,526 dropouts which represents a 6.7% annual dropout rate. The four-year graduation rate in th is district is 59.2%. The district employs 39,603 full time employees, including administra tors, teachers, and support staff.

PAGE 61

61 There are a total of 193,681 students enrolled in District Two. Free or reduced price lunch is provided for 49% (n = 95,726) of the student population. In District Two, Title I funding is provided to schools with a 57% or higher rate of students eligible for free or reduced price lunch. This district is comprised of 279 schools of which 123 are Title I schools offering SES. The district ethnic breakdown of the student population consists of 44% White, 22% Black, 26% Hispanic, and 8% Other, which includes American Indian, Asian, and Mul tiracial categories. The annual dropout rate in grades 9-12 is 2.1% (n =1 ,330), with a four year graduation rate of 77.3%. A total of 17,272 full time employees are employed by this district, including administrators, teachers, and support staff. District Three has a total enrollment of 175,593 students. Students receiving free or reduced price lunch total 81,245 students or 46 % of the student popula tion. Title I funds are provided to schools with a 75% or higher rate of students eligible for free or reduced price lunch This district is composed of 222 schools of wh ich 39 schools or 18% are Title I schools offering SES. The district wide ethnic di stribution of students in Distri ct Three is 36% White, 28% Black, 29% Hispanic, and 7% Other, which include s American Indian, Asian, and Multiracial categories. The annual dropout rate is 1.9% (n = 1,165) students in grades 9-12. The four year graduation rate is 72.2%. There is a total of 22,361 full time staff members employed in the district, including administrators teachers, and support staff. Student enrollment totals 174,861 students in Di strict Four. Of the total enrollment, 72,947 or 42% of the districts students receive free or reduced price lunch. District Four provides Title I funding to schools with a 44% or hi gher rate of students eligible for free or reduced price lunch There are 265 schools in this district with 24% (n = 64) being Title I Schools offering SES. The ethnic breakdown of the district enrollment is 42% White, 29% Black, 22% Hispanic, and 7%

PAGE 62

62 Other, which includes American Indian, Asian, and Multiracial categories. The annual dropout rate of students in District Four, grades 9-12, is 3% (n =1,715). The four year graduation rate in this district is 69.3%. There are 19,696 full time staff members employed by the district, including administrators, teachers, and support staff. District Five has a total enrollment of 126,648 students. Free and reduced price lunch is received by 52,709 students or 42% of the student enrollment in the district. This district provides Title I funding to schools with a 62% or higher rate of students eligible for free or reduced price lunch. Of the 182 schools in the di strict, 41 or 23% are Title I schools that are required to offer SES. The ethnic composition of the student populat ion district-wide is comprised of 44% White, 43% Black, 6% Hispan ic, and 7% Other, which includes American Indian, Asian, and Multiracial cate gories. The annual dropout rate of students in grades 9-12 in District Five is 6.6% (n = 2,780) with a four year graduation rate of 60.5%. A total of 12,958 full time staff members are employed by the district, including admini strators, teachers, and support staff. The final school district included in this evaluation, District Six, has a total student enrollment is 112,150 with 40% (n = 45,217) receivi ng free or reduced pri ce lunch. In District Six, Title I funding is provided to schools with a 43% or higher rate of students eligible for free or reduced price lunch This di strict is comprised of 279 school s of which 32 are Title I schools offering SES. The ethnic distribution of students in this district is 64% White, 19% Black, 9% Hispanic, and 8% Other, which includes American Indian, Asian, and Mul tiracial categories. In grades 9-12, there are a total of 1,304 dropouts, which represents a 3% annual dropout rate. The four-year graduation rate is 67%. A total of 14,409 full time staff members are employed by District Six, including administrato rs, teachers, and support staff.

PAGE 63

63 FCAT data requested from the six Florida schoo l districts was used in a quasi-experimental design in this study. In each district, the data was divided into two groups of Title I students receiving free or reduced price lunch in grades 4 10. Both groups of students had FCAT scores for 2006 and 2007 for math and reading and attende d schools that offered SES. Group One, the experimental group, was composed of students who received at least 10 h ours of tutoring in the SES program during the 2006-07 school year. Gr oup Two, the control group, was composed of students who were eligible for SES, but not enrolled in the program. When submitting the requested data, districts were asked to include FCAT developmental scale scores, as well as FCAT achievement leve ls. In addition, demographic information was requested including gender, ethnicity, student ES E codes, SES provider codes, as well as the total hours of tutoring in math and/or reading. Prior to submitting the Institutional Review Board application to each of the six urban Florida school districts included in this study, permission to c onduct research was obtained from the University of Florida Institutional Review Board. The study did not involve direct student contact, but evaluated existing FCAT scores and student identif ication numbers were deleted. Therefore, individual students were not identifiable and informed consent was not required. The methodology included using developm ental scale scores from the 2006 and 2007 FCAT administrations. Developmental scale scores were select ed as the unit of measure because they are an effective tool in tracking student performance across grade levels. As a student moves from grade to grade, his/her perfor mance can be tracked and compared to the performance of other students. Developmental scale scores enable th e yearly progress of individual students to be repor ted by the change in their scor es (Florida Department of Education, 2007b).

PAGE 64

64 Procedures The developmental scale score from the 2006 and 2007 FCAT administrations were used as preand post-tests. A quasi-experimental de sign was utilized wherein the experimental group participated in the SES program and students in the control group did not participate in the tutoring program. Students who did not particip ate in both the 2006 and 2007 administration of the FCAT were eliminated from the study. In addition, the data file was scrubbed by eliminating students who did not receive a minimum of 10 hours of tutoring. A minimum of 10 hours of tutoring was consistent with criteria set by the Florida Depa rtment of Educations evaluation proposal that only students who receive a minimum of 10 hours of tutoring be included. Students who had received SES were placed in the experimental group. The control group was randomly selected utilizing the Statistical Package for Social Sciences (SPSS) software and was re-sampled for each analysis. In addition, the control group was matched by grade with students in the experimental group. The control group was qualified to receive SES, but the parents of these students chose not to enroll them. Next, the experimental group was divided into three categories: st udents who received tutoring in math only, students who received tuto ring in reading only, and students who received tutoring in both math and reading. The contro l group was subsequently divided into three matching groups. The null hypotheses in this study were tested using the ANOVA test of means to determine if there was a statistically significant increase in FCAT developmental scale scores from the 2006 to 2007 administration. Computations were pe rformed using SPSS software and tested at a .05 level of significance. The eff ectiveness of SES was reported in three areas: reading only,

PAGE 65

65 math only, and reading and math. SES effectiv eness was also reported by individual SES provider within each district. Summary This chapter provided an overview of the study by discussing the need to evaluate SES to determ ine its success on improving the academic achievement of students receiving tutoring. An overview of the method discussed the criteria used to select the six Florida urban school districts included in this study, as well as th e selection protocol used to fo rm the control and experimental groups. Finally, a discussion of the procedures was presented wherein scrubbing the data and the formation of categories in the experimental and control group was explained as a preface to the reporting of data in Chapter 4.

PAGE 66

66 Table 3-1. Participating District Statistics Statistics District District Distri ct District Di strict District One Two Three Four Five Six Total Enrollment 362,050 193,681 175,593 174,861 126,648 112,150 Students with Free/Reduced Lunch 221,229 95,726 81,245 72,947 52,709 45,217 Poverty rate for Title I funding 63% 57% 75% 44% 62% 43% Total Schools 438 279 222 265 182 279 SES Schools 204 123 39 64 41 32 Dropout Rate 6.7% 2.1% 1.9% 3% 6.6% 3% Graduation Rate 59.2% 77.3% 72.2% 69.3% 60.5% 67% Full Time Staff 39,603 17,272 22,361 19,696 12,958 14,409

PAGE 67

67 CHAPTER 4 RESEARCH FINDINGS The purpose of this analysis was to asse ss the effectiveness of the SES program in Florida urban school districts that were requir ed to offer this tutoring program during the 200607 school year. To determine effectiveness, FCAT data was obtained from six of the largest Florida urban school districts. Initially the study was designed to include the seven largest urban Florida school districts, but was revised to si x school districts when obs tacles in obtaining the needed data arose from the seve nth district. In reporting the resu lts of this study, participating school districts are not identif ied by district name. Districts were randomly numbered one through six for identification purposes in this study. This study utilized FCAT data from these districts in a quasi-experimental design. FCAT developmental scale scores for 2006 and 2007 were us ed as preand post-tests. In each district, the FCAT data from the Title I SES schools wa s divided into two gr oups receiving free or reduced price lunch in grades 4 10. Both groups were required to have FCAT scores from the 2006 and 2007 administration for reading and ma th. Group one, the experimental group, was composed of students who receiv ed at least 10 hours of tutoring in the 2006-07 school year. Group two, the randomly selected control group, was composed of students who were eligible to receive SES, but were no t enrolled in the program. Next, the experimental group was divided in to three categories: students who received tutoring in math only, students who received tuto ring in reading only, and students who received tutoring in math and reading. The control group was subsequently divided into three matching groups. In the experimental group and the control group, each students 2006 FCAT developmental scale score was subtracted from their 2007 score and the ne t result was used in the analysis.

PAGE 68

68 The null hypotheses in this study were tested using a one-way analys is of variance design to determine if there was an increase in FCAT developmental scale scores from the 2006 to 2007 administration for students who participated in SES. Computations were performed using SPSS software testing at a .05 level of significance. Th e results reported in three areas: reading only, math only, and reading and math. SES effectiv eness was also reported by individual SES providers in each district who served the largest number of st udents who received at least 10 hours of tutoring in math and/or reading. Research Questions The following research questions w ere addressed in the study: 1. Do students who participated in Supplemental Educational Services exclusively in math for 10 or more hours demonstrate higher developmen tal scale score gains, as determined by the FCAT, compared to students in a control group? 2. Do students who participated in Supplemental Educational Services exclusively in reading for 10 or more hours demonstrate higher developmental scale score gains, as determined by the FCAT, compared to students in a control group? 3. Do students who participated in Supplemental Educationa l Services in both math and reading for 10 or more hours demonstrate high er developmental scale score gains, as determined by the FCAT, compared to students in a control group? 4. Do students tutored by the three major SES pr oviders in each distri ct (i.e., those that provided tutoring for the largest number of students) for 10 or more hours demonstrate higher developmental scale score gains, as de termined by the FCAT, compared to students in a control group? Research Hypotheses The study was guided by the following research hypotheses: Null Hypot hesis 1 There is no statistically significant difference between 2006 FCAT developmental scale scores as the pre-test a nd 2007 FCAT developmental scale scores as the post-test for students in Title I schools rece iving at least 10 hours of tutoring in math.

PAGE 69

69 Null Hypothesis 2 There is no statistically significant difference between 2006 FCAT developmental scale scores as the pre-test a nd 2007 FCAT developmental scale scores as the post-test for students in Title I schools receiv ing at least 10 hours of tutoring in reading. Null Hypothesis 3. There is no statistically significant difference between 2006 FCAT developmental scale scores as the pre-test a nd 2007 FCAT developmental scale scores as the post-test for students in Title I schools receiv ing at least 10 hours of tutoring in math and reading. Null Hypothesis 4. There is no statistically significant difference between 2006 FCAT developmental scale scores as the pre-test a nd 2007 FCAT developmental scale scores as the post-test for students receiving at least 10 hours of tutoring from the three SES providers who tutored the largest number of students in each district. Analysis and Quantitative Results FCAT data from the six districts participating in this study were combined to assess the effectiveness of the SES program in raising stud ent achievement. This FCAT data file contained complete data by grade level for the 9,026 st udents who received SES during 2006-07 school year. Table 4-1 reports the number of students represented at each grade level. Table 4-2 reports the combined breakdown of students by grade le vel and school district. When assessing academic gains for students, grade levels with less than 50 students were eliminated because it was determined they did not adequately repres ent the student population. Grades 11 and 12 were eliminated because of the small number of students represented in these grades (n= 17). As a result, the experimental group used in this st udy contained 9,009 students each of whom received at least 10 hours of tutoring in math and/or reading. Table 4-3 documents the number of state approved SES providers who provided tutoring se rvices within each district during the 2006-07

PAGE 70

70 school year. Some providers served multiple distri cts; elimination of duplication resulted in a total of 86 different provider s assessed in this study. The control group was comprised of students with similar demographic information as represented in the experimental group. The control group population included 21,784 students who attended Title I SES schools but did not par ticipate in the SES tutoring program. Using the SPSS software program, a random selection process was utilized to compile control groups that matched the experimental groups. The control gro up was re-sampled for each analysis completed in this study. Research Question 1: Do students who participated in Supplemental Educational Services exclusively in math for 10 or more hours demons trate higher developmental scale score gains, as determined by the FCAT, compared to students in a control group? The math only group included 1,565 students in the experimental group and 1,565 students in the corresponding control group. Table 4-4 illustrates the numb er of students in each grade level per district that received tutoring in math. This group received a total of 24,590 hours of math tutoring from 57 SES providers. The numbe r of math tutoring hours provided per student ranged from a minimum of 10 hours to a maximu m of 56 hours. The developmental scale score gain mean for the experimental group was 144.16 and 122.60 for the control group. Therefore, a mean score difference of 21.56 was determined w ith the tutored group having the higher mean score. Developmental scale scores gains range d from a low of -828 to a high of 1046 for the experimental group. In the control group, developm ental scale scores gains ranged from a low of -816 to a high of 1120. Null Hypothesis 1 : There is no statistically signi ficant difference between 2006 FCAT developmental scale scores as the pre-test a nd 2007 FCAT developmental scale scores as the post-test for students in Title I schools rece iving at least 10 hours of tutoring in math.

PAGE 71

71 Results of the ANOVA test of the means for Null Hypothesis 1 (Table 4-7) found a statistically significant difference between the math experimental group and the math control group on the FCAT developmental scale score gains (f = 11.161; p = .001). Thus, Null Hypothesis 1 stating that SES tutoring in math would have no statistically significant effect in FCAT developmental scale scores was rejected. Research Question 2: Do students who participated in Supplemental Educational Services exclusively in reading for 10 or more hours demonstrate higher developmen tal scale score gains, as determined by the FCAT, compared to students in a control group? The reading only experimental group was comp rised of 6,630 students who participated in the reading tutoring program. The control group mirrored the grade levels and number of students in the experimental gr oup. Table 4-5 details the number of students per grade level and district for the reading group. St udents received 106,600 hours of tutoring in reading provided by 73 SES providers. The minimum number of per st udent tutoring hours in r eading was 10 with the maximum tutoring hours of 53. The developmental scale score gain mean for the experimental group was 126.60 and the mean for the control group was 120.21 a mean score difference of 6.39 favoring the experimental group. Developmental scale score gains ranged from a low of -1250 to a high of 1498 for the experimental group. The co ntrol group developmental scale scores gains ranged from a low of -891 to a high of 1163. Null Hypothesis 2 : There is no statistically signi ficant difference between 2006 FCAT developmental scale scores as the pre-test a nd 2007 FCAT developmental scale scores as the post-test for students in Title I schools receiv ing at least 10 hours of tutoring in reading. Results of the ANOVA test of means (Table 4 8) for Null Hypothesis 2 found no statistically significant difference between the FCAT developmental scale score gains of the reading experimental group and the reading control group (f = 2.453; p = .117). As a result Null

PAGE 72

72 Hypothesis 2, stating that there is no statistically significant difference between the 2006 and 2007 FCAT developmental scale scores for st udents tutoring in reading was accepted. Research Question 3: Do students who participated in Supplemental Educational Services in both math and reading for 10 or more hours demonstrate higher deve lopmental scale score gains, as determined by the FCAT, compar ed to students in a control group? The reading and math experimental group was comprised of 1,084 students who received tutoring in both subjects. The control group for th is analysis also cont ained 1,084 students at the same grade levels. Table 4-6 summarizes the numbe r of students and the grade levels per district for this group. These students received 6,488 hours of reading tutoring and 6,505 hours of math tutoring from 38 SES providers. Individual stude nts received a minimum of 10 and a maximum of 36 hours of tutoring in reading. The same gr oup of students received a minimum of 10 hours and a maximum of 30 hours in math tutoring. Th e developmental scale score mean for the experimental reading group was 126.95 and the control reading group was 122.29, a mean scale score difference of 4.66 favoring th e experimental reading group. The mean developmental scale score gain for the experimental math group was 137.69 while the mean score gain for the control math group was 128.44. The experimental math group produced a higher mean score gain difference of 9.25. Developmental scale scor e gains ranged from a low in the reading experimental group of -956 to a high of 1369. The math experimental group had developmental scale score gains that varied from a lo w of -792 to a high of 1036. The control group developmental scale scores in reading ranged from -823 to a high of 1231 and in math ranged from a low of -816 to a high of 1120. Null Hypothesis 3 : There is no statistically signi ficant difference between 2006 FCAT developmental scale scores as the pre-test a nd 2007 FCAT developmental scale scores as the post-test for students in Title I schools receiv ing at least 10 hours of tutoring in math and reading.

PAGE 73

73 The ANOVA for Null Hypothesis 3 (Table 4 9) resulted in a finding of no statistically significant difference in mean scores between the experimental and control groups receiving tutoring in reading and math (reading: f = .212; p = .646; math: f = 1.347; p = .246). The finding of no statistically significant difference between the 2006 and 2007 FCAT developmental scale score gains between the experimental and c ontrol groups resulted in acceptance of Null Hypothesis 3. Research Question 4: Do students tutored by the thr ee major SES providers in each district (i.e., those that provided tutoring for the largest numbe r of students) for 10 or more hours demonstrate higher developmental scale score gains, as determined by the FCAT, compared to students in a control group? An ANOVA was used to assess the effectivene ss of individual SES provi ders that provided tutoring in the six districts incl uded in this study. The three SES providers that served the largest number of students in each distri ct were included in the analysis for a total of 18 SES providers. Results are reported by district in Tables 4-10 through 4-15. Of the 18 providers analyzed, th ree were found to have statistic ally significant results as tested by the ANOVA. These SES providers were (a) Advanced Learning, (b) Rocket Learning, and (c) Studentnest, Inc. In District 2 (Table 4-11), Advanced Learning tutored 88 students in math and 97 students in reading. Students received 1,857 hours of tu toring in reading and 1,782 in math. Reading students received a minimum of 10 hours of tu toring and a maximum of 29. A minimum of 10 hours and a maximum of 34 hours was provided to students in math tutoring. The mean developmental scale score for the math expe rimental group was 157.34 and the math control group developmental scale score mean was 84.64. A mean score difference of 72.70 resulted with the math experimental group having the higher mean score. The reading experimental group in the analysis of Advanced Learning had a mean developmen tal scale score of 158.94 and the

PAGE 74

74 control group had a mean score of 79.40 resul ting in a 79.54 difference. Developmental scale score gains in the experimental group ranged from a low in reading of -320 to a high of 483. In math the scores ranged from a low of -547 to a high of 1433. The control group developmental scale score gains ranged from a low in reading of -523 to a high of 445. In math the scores ranged from a low of -681 to a high of 588. In District 3 (Table 4-12), Ro cket Learning provided tutoring for 47 students in reading. This SES provider tutored students for a total of 1,120 hours. Students received a minimum of 10 hours of tutoring and a maximum of 34 hours. The mean developmental scale score for the experimental group was 189.02 and the control gr oup was 71.49, resulting in a difference in the mean developmental scale scores of 117.53. In th e experimental group developmental scale score gains for Rocket Learning students ranged fr om a low -342 to a high of 818. Developmental scale score gains for the control group ranged from a low -334 to a high of 357. The final SES provider that had statistically significant ANOVA results was Studentnest in District 4 (Table 4-13) who tu tored 155 students in Math for a total of 3,151 hours. A minimum of 10 hours of tutoring was provided to stude nts with a maximum of 27 hours offered per student. The control group had a mean develo pmental scale score of 96.14, and a mean of 151.13 was determined in the experimental group. Thus, a difference of 54.99 was found between the experimental group and the control group. The analys is of Studentnest developmental scale score gains in the experimental group resulted in a low of -543 to a high of 778. The control group developmental scale score gains ranged from a low of -427 to a high of 696. Null Hypothesis 4 : There is no statistically signi ficant difference between 2006 FCAT developmental scale scores as the pre-test a nd 2007 FCAT developmental scale scores as the post-test for students receiving at least 10 hours of tutoring from the three SES providers who tutored the largest number of students in each district.

PAGE 75

75 The ANOVA resulted in a finding of statistically significant difference for three of the SES providers analyzed, but the remaining 15 SES provi ders resulted in a find ing of no statistically significant difference between 2006 FCAT de velopmental scale scores and 2007 FCAT developmental scale scores. Tables 4-10 through 4-15 detail the statistic al findings for the 18 SES providers analyzed. Null Hypothesis 4, which states that tutoring provided by the three largest providers in each distri ct would have no statistically significant results, was rejected for the three providers who demonstrated statisti cally significant results: Advanced Learning, Rocket Learning, and Studentnest, Inc. The ANOVA results for the remaining 15 SES providers was retained for Null Hypothesis 4. The ANOVA analysis for Advanced Learning resulted in a finding of a statistically significant difference between the experimental gr oup receiving tutoring in reading and math and the matching control group on the developmental scale score gains on the FCAT reading (f = 4.688; p = .032) and math (f = 5.344; p = .022). This finding of a statistically significant difference between the 2006 and 2007 FCAT devel opmental scale score gains in reading and math resulted in rejection of Nu ll Hypothesis 4 for Advanced Learni ng in District 2 (Table 4-11). Results of the ANOVA (Table 4-12) for Rocket Learning in District 3 found a statistically significant difference between the mean scores of the reading experime ntal group and control group on FCAT developmental scale score gains (f = 7.193; p = .009). This finding resulted in rejection of Null Hypothesis 4 for Rocket Learning in District 3. The outcome of the ANOVA test of means (Table 4-13) for Studentnest in District 4 found a statistically significant difference between the math experimental group and the math control group on the FCAT developmental scale score gain s (f = 8.237; p = .004). This finding resulted in rejection of Null Hypothesis 4 for Studentnest in District 4.

PAGE 76

76 Summary The purpose of these analyses was to exam in e the effectiveness of the SES program in Florida urban school districts Ti tle I schools that were required to offer the SES program in the 2006-07 school year. The ANOVA test of means was used to analyze groups of tutored students that included students who were tutored in math only, reading only, and reading and math. Also analyzed were the three SES pr oviders who tutored the larges t number of students in each district. Of the 21 groups evaluate d, 4 groups resulted in a statistica lly significant difference in the mean scores of the experimental group and the control group. The analysis concluded that students tutored in math only, as well as st udents tutored by Advanced Learning, Rocket Learning, and Studentnest, Inc., significantly increased their academic achievement per the developmental scale scores on the 2007 FCAT. In Chapter 4 quantitative results of the st udy were presented. Chapter 5 will present a discussion of the findings, conclusions and reco mmendations, as well as implications, future research and summary.

PAGE 77

77 Table 4-1. SES Students by Grade Level Grade Level Number of Students 4 3,188 5 2,854 6 1,312 7 710 8 720 9 121 10 104 11 12 12 5 TOTAL 9,026 Table 4-2. SES Students by District Grade District District Distri ct District District District One Two Three Four Five Six 4 1,458 585 27 527 234 357 5 1,158 503 262 428 212 291 6 712 171 104 302 23 7 404 79 30 182 15 8 427 123 14 146 10 9 44 15 62 10 44 20 40 11 3 9 12 2 3 Total 4,247 1,501 437 1,699 494 648 N = 9,026

PAGE 78

78 Table 4-3. Active SES Providers by District District District Distri ct District District District One Two Three Four Five Six Providers 35 22 15 28 20 20 Table 4-4. Total Students: Math Only Group by District and Grade Grade District District Distri ct District District District One Two Three Four Five Six 4 88 217 41 32 86 5 112 200 17 48 39 85 6 65 76 98 15 7 51 25 1 76 5 8 49 50 1 57 1 9 4 5 11 10 2 5 3 Total 371 578 19 334 92 171 N = 1,565

PAGE 79

79 Table 4-5. Total Students: Reading Only Group by District and Grade Grade District District Distri ct District District District One Two Three Four Five Six 4 1,225 366 26 238 192 226 5 918 302 229 204 166 165 6 606 95 95 132 7 7 329 54 25 73 10 8 346 73 12 62 8 9 39 10 46 10 38 15 28 Total 3501 915 387 783 383 391 N = 6,360 Table 4-6. Total Students: Reading a nd Math Group by District and Grade Grade District District Distri ct District District District One Two Three Four Five Six 4 145 2 1 248 10 45 5 128 1 16 176 7 41 6 41 9 72 1 7 24 4 33 8 32 1 27 1 9 1 5 10 4 9 Total 375 3 31 570 19 86 N = 1,084

PAGE 80

80 Table 4-7. Analysis of Variance for Math Only Group Subject df 1 df 2 F p Math 1 3128 11.1161 .001* Statistically significant Table 4-8. Analysis of Variance for Reading Only Group Subject df 1 df 2 F p Reading 1 12,718 2.453 .117 Table 4-9. Analysis of Variance fo r Reading and Math Tutoring Group Subject df 1 df 2 F p Reading 1 2166 .212 .646 Math 1 2166 1.347 .246 Table 4-10. Analysis of Va riance for District 1 SES provider Subject df 1 df 2 F p Cool Kids Reading 1 414 .000 .984 Rocket Learning Reading 1 1536 .163 .687 Rocket Learning Math 1 118 1.053 .307 Education Station Reading 1 2507 .219 .640

PAGE 81

81 Table 4-11. Analysis of Va riance for District 2 SES provider Subject df 1 df 2 F p Academy of Success Reading 1 206 .167 .684 Academy of Success Math 1 206 .004 .947 Advanced Learning Reading 1 192 4.688 .032* Advanced Learning Math 1 174 5.344 .022* A+ Tutor U Reading 1 274 .242 .623 A+ Tutor U Math 1 224 2.131 .146 Statistically Significant Table 4-12. Analysis of Va riance for District 3 SES provider Subject df 1 df 2 F p Club Z! Reading 1 66 2.305 .134 Club Z! Math 1 107 3.656 .059 A+ Tutor U Reading 1 298 .256 .613 Rocket Learning Reading 1 92 7.193 .009* Statistically Significant Table 4-13. Analysis of Va riance for District 4 SES provider Subject df 1 df 2 F p AlphaBEST Reading 1 534 .001 .978 AlphaBEST Math 1 534 1.592 .208 Huntington Learning Reading 1 314 1.985 .160 Studentnest Math 1 308 8.237 .004* Statistically Significant Table 4-14. Analysis of Va riance for District 5 SES provider Subject df 1 df 2 F p A Wise Choice Reading 1 88 .644 .424 Huxtable Education Reading 1 116 .782 .378 A+ Tutor U Reading 1 164 .151 .698

PAGE 82

82 Table 4-15. Analysis of Va riance for District 6 SES provider Subject df 1 df 2 F p ACES Reading 1 112 .103 .748 ACES Math 1 138 .170 .680 Achievia Reading 1 84 .323 .571 PCTA Reading 1 108 1.285 .260 PCTA Reading 1 108 1.329 .251

PAGE 83

83 CHAPTER 5 DISCUSSION Poverty and inequality in Am ericas education system ar e barriers to opportunity and progress. This is apparent in the achieveme nt gaps between student groups with AfricanAmerican and Latino students continually lagging behind in academic achievement compared to other students (Kennedy, 2005). The fact that only 29% of all fourth grade students performed at or above the proficient level on the National A ssessment of Educational Progress in reading in 2000 clearly indicated a need for nationwide educ ation reformation and th e federal government enacted NCLB to implement this reform (U S. Department of Education, 2002; U. S. Department of Education, 2006). The highest priority of NCLB is to give low achieving students the opportunity and ability to meet high academic standards and reach pr oficiency on demanding state assessments. The Title I program, one of the largest programs f unded by NCLB, was designed to ensure that students living in poverty have an equal opportunity to obtain a high quality education and achieve the required academic growth to meet state standards and assessments. Title I provides federal funding to help students who are either behind academically or are at risk of falling behind academically. NCLB introduced the SES program to Title I schools to provide additional academic assistance (e.g., tutoring, remediatio n, and other educational interv entions). The purpose of this study was to assess the effectivene ss of the SES program in Florida urban school districts Title I schools that were required to offer the program in the 2006-07 school year. Nationwide, each Title I school that has not made AYP for three year s is required to provide SES to students from low-income families. This program is designed to increase the academic achievement of students in eligible Title I schools. NCLB defines SES as additional academic assistance such as tutoring,

PAGE 84

84 remediation, and other educationa l interventions, provided that th ese approaches are consistent with the content and instruction used by the school district an d are aligned with the states academic content standards (Kennedy, 2005). SES pr oviders are required to demonstrate that the tutoring provided increases student academic achie vement in order to maintain their status as approved and remain on the state-approved li st (U. S. Department of Education, 2005). Therefore, states are required to evaluate SES providers to determine their effect on increasing student achievement (Smole, 2004). While federal guidance places the responsibility of evaluating the effectiveness of the SES program on the state, Florida has not evaluated this program since it be gan in the 2004-05 school year. States are required to prohi bit providers from offering services to students if they fail to increase student achievement for two years, but Fl orida, as well as all ot her states, has neglected to enact this section of the law. Discussion of the Findings This study included 9,009 students f rom 6 large urban Florida school districts in grades 4 through 10 who were tutored by 86 state-approved SES providers. These SES students received a total of 113,148 hours of tutoring in reading and 31,128 hours of tutoring in math for a total of 144,276 hours of tutoring during the 2006-07 school y ear. Findings from this study demonstrated that, although there was some in crease in the academic achievement of students participating in this program, the increase was limited to a relati vely small number of students. Of the 21 tutoring groups analyzed, only 4 groups demonstrated sta tistically significant academic gains on FCAT developmental scale scores. These 4 groups include the math only tutoring group and 3 individual SES providers. Those providers are Advanced Learning, Rocket Learning, and Studentnest, Inc. The results of the st udy are summarized in Tables 5-1 and 5.

PAGE 85

85 The following research questions were addressed in the study. 1. Do students who participated in Supplementa l Educational Services exclusively in math for 10 or more hours demonstrate higher developmental scale score gains, as determined by the FCAT, compared to students in a control group? From an experimental group of 1,565 student s, an ANOVA test of means evaluated the developmental scale scores from the 2006 and 2007 FCAT to assess the academic gains of students tutored in math. The results from th e ANOVA were statistica lly significant with a significance level of .001. Of the three subject assessments (t utoring in math only, tutoring in reading only, and tutoring in r eading and math) this was the only analysis that resulted in a statistically significant differen ce in mean scores. Of the 9,009 st udents evaluated in this study, only 1,565 students received benefits from the math only tutoring. Null Hypothesis 1 was rejected for the math only tutoring group. 2. Do students who participated in Supplemental Educational Servi ces exclusively in reading for 10 or more hours demonstrate higher de velopmental scale score gains, as determined by the FCAT, compared to students in a control group? The number of students tutored in reading was considerably hi gher than the number tutored in math. This analysis contained 6,360 st udents who were tutored in reading only. The ANOVA test of mean scores was not statisti cally significant for th e reading group with a significance level of p = .117. Therefore, Null Hypothesis 2 was retained. The 106,600 hours of tutoring this group received, was not effective in increasing th eir academic achievement in reading. 3. Do students who participated in Supplemen tal Educational Services in both math and reading for 10 or more hours demonstrate higher de velopmental scale score gains, as determined by the FCAT, compared to students in a control group? This group of 1,084 students received tutoring in both math and reading. Reading tutoring totaled 6,488 hours and math tutoring totaled 6,505. The ANOVA test of mean scores resulted in

PAGE 86

86 a finding of no statistical difference between th e tutored experimental group and the control group with a significance level of p = .246 in math and .646 in reading. This group also failed to increase their academic achievement. Th erefore, Null Hypothesis 3 was retained. 4. Do students tutored by the three major SES providers in each distri ct (i.e., those that provided tutoring for the largest number of stude nts) for 10 or more hours demonstrate higher developmental scale score gains, as determined by the FCAT, compared to students in a control group? Each district was represented by three SES providers who provided tutoring for the largest number of students in that district. Of the 18 provide rs analyzed, 3 were found to have produced statistically significan t results per the ANOVA. As a result, the null hypothesis was rejected for these 3 providers. The ANOVA results from the remaining 15 providers resulted in a finding of no statistically signi ficant difference in the mean scores between the tutored experimental group and the control group. Therefor e, Null Hypothesis 4 was retained for these 15 SES providers. Conclusions and Recommendations Sunderm an & Kim (2004) defined the goal of SES as a program that was designed to increase the academic achievement of students in low-performing schools. The results of this study found minimal evidence to indicate that the SES tutoring program was producing significant academic growth for Title I students. In fact, the studys statistical evidence clearly demonstrates that the SES program had only a minimal impact on Title I students. More extensive results should be exp ected for a program that consum es 20% of districts Title I allocation. This analysis indicated that despite 9,009 students in grades 4-10 received a total of 144,276 hours of tutoring, major academic growth accrued for a relatively small number of students. Of the 6 districts assessed, only 3 of the 18 major SES provi ders had statistically

PAGE 87

87 significant results on increasing the academic ach ievement of students as measured by FCAT Developmental Scale Scores. The number of stude nts tutored by the three providers was small (n = 413) when compared to the total number of students (n = 9,009) who received tutoring in the 2006-07 school year. The only student group that was positively affected by the tutoring in math totaled 1,565 students. The students who demonstr ated increased academic achievement in at least one subject totaled 1,978 ( 22%), leaving 7,051 (78%) student s who received tutoring, but did not demonstrate increased academic achievement. Isolating the impact of the SES program on students who did make academic gains is difficult. Rickles & Barnhart (2007) stated that positive outcomes of students who participated in the SES program might be due to students who ar e more motivated and pers istent in nature or who have parents who are more involved in thei r childrens education th an those who did not participate. Burch (2007) noted th at that selection bias could also account for higher gain scores. For example, parents who enroll their children in SES tend to be better educated and take a more active role in their childrens education. He also stated that academic gains may have resulted from an academic program other than SES. Such se lection bias was not apparent from the results of this study. The cost of the SES program also must be c onsidered when assessing the effectiveness of the program. SES is funded by a 20% setaside that is deducted from the sc hool districts Title I funds. This setaside is taken from grant funds that in past years were distributed to Title I schools. Therefore, with the implementation of SES, school funding for Floridas Title I schools was decreased by 20%. The setaside encumber ed to fund SES and school choice in the 67 Florida school districts for 2007-08 amount ed to $115,140,533. The 20% setaside for Miami Dade County Public Schools alone amounted to $25,308,760, with an additional 21 Florida

PAGE 88

88 school districts encumbering at least $1 milli on each to fund the SES program (Florida Department of Education, 2007a). Full implementa tion of SES nationwide could total more than $2.5 billion of the annual Title I grant funds (Ascher, 2006; Cohen, 2006). Given the minimal number of Florida students in large urban schoo l districts who benefit from SES, it must be determined whether this is the most effective way to spend these funds. Students in Title I schools that have not made AYP can be challenging learners. In past years many research based, innovative programs ha ve been utilized with these students to increase their academic achievement. Despite th ese programs the achievement of these students has not significantly increased. The question must be asked whether it is f easible for after-school tutoring programs presented a few hours a week to increase the academic performance of these students. SES is expected to enable significant ac ademic strides in students who have continued to be unsuccessful. Perhaps the expectati ons of the SES program are inappropriate. The task of this analysis was to assess the effectiveness of the SES program in Florida urban district schools that were required to offer this program in the 2006-2007 school year. Through the literature review, data collection, and findings reported in this analysis, the following conclusions were developed: Conclusion 1: According to SES federal guidan ce, the state is charged with the responsibility to develop, implement, and publicly report the effectiveness of SES providers and is required to withdraw approval from providers that fail to increase the academic performance of students for two consecutive years (U. S. Department of E ducation, 2005). Hess & Finn (2004b) suggested that guidance be provided to states to help develop sound and rigorous evaluations, including planning and funding. The Northwest Regional E ducational Laboratory (2004) summarized the

PAGE 89

89 importance of evaluating SES programs by stating that it is not the lack of evaluation that is of the utmost concern; it is the importance of knowing if SES students are actually receiving services that increase their academic achievement. Florida implemented the SES program in 200405 in 11 urban districts that had Title I schools in their third year of non-adequate yearly progress. The number of SES districts increased every year until 2006-07, when SES was required in every school district in Florida. During the 2007-08 school year Florida appr oved 201 providers to provide SES (Florida Department of Education, n.d.). SES has been full y implemented in Florida, but there has been no evaluation of the effectivene ss of the program in the stat e. This study confirmed that relatively few providers were effective in raisin g the academic performance of students in large urban Florida school districts. E ffective providers need to be re tained, but ineffective providers need to be eliminated from the state-appr oved list. To meet the mandate of NCLB a comprehensive evaluation is necessary so that in effective providers can be removed from the list of state-approved providers. Conclusion 2: Reid (2004) indicated that one of the reasons for the lack of ev aluation is that states lack the capacity and funding to develop an effectiv e monitoring system and evaluation method. By the very nature of the NCLB sanction process, SE S enrollment continues to increase yearly. With increased enrollment, additional administrative and management burdens are created. Districts are forced to assume these res ponsibilities with no additional funding (Ascher, 2006). The Center on Education Policy released a 2006 study based on survey results from 50 states, concluding that NCLB created greater administrative burdens with inadequate funding. Re ntner, et al. (2006) concluded that districts and states lack funding and staff to manage the many demands of NCLB.

PAGE 90

90 These financial burdens are not part of the 20% setaside, but are additional funds taken from the Title I budget that woul d have been allocated to sch ools before NCLB was enacted. A portion of the required setaside should be used to ease the administra tive burden on districts created by SES. Using 4% of the 20% setaside w ould cover administrative costs such as printing, postage, parent outreach, monitoring, and personnel needs. Using this portion of the setaside for SES administrative costs would ul timately increase the financial support of all Title I schools and students if such costs were not taken from their allocation. Conclusion 3: Although there is research that identifies quality indicators (Table 2-1, p. 57) of effective tutoring (i.e., student-teacher ra tio, collaboration with the classroom teacher, tutor qualifications, tutor training, and length of tutori ng), NCLB does not require that they are used in SES tutoring programs. The law vaguely states that SES programs must be of high quality, research based, and designed to increase stud ent achievement (U. S. Department of Education, 2005). For SES to be effective in raising the academic achievement of students, established quality indicators of effective tutoring must be required of SES providers and be part of the evaluation of effectiveness. The literature review conducted for this analysis indicated that low student-teacher ratio is an important factor in effective tutoring. A 1:1 or 2:1 ratio is suggested for a successful tutoring program. Research indicated that 1: 1 tutoring is considered as the most effective form of tutoring (Baker et al., 2006; Elbaum et al., 2000; Juel 1996; Lauer, 2006; Wasik, 1998). Programs that provide 1:1 tutoring for at-risk students were proven to increase the academic achievement of students in reading. Currently, Florida allows SE S providers to choose their student group size. These choices include small group (not to exceed 5 students), large group (not to exceed 10

PAGE 91

91 students), and individual tutori ng. The allowed student grouping si ze for SES must be revised to provide 1:1 tutoring to in dividualize instruction. Another important aspect of successful tuto ring is collaboration between the childs classroom teacher and the tutor. Tutoring needs to be coordinated with classroom instruction to provide the student with additional instruction that parallels what is being taught in the classroom and that uses comparable materials. This enables the student to have multiple opportunities to work on difficult materials. As students master th e material during tutoring, they are more likely to perform better in class (Wasik, 1998). Currently, the curriculum used during tutoring is left to the discretion of the SES provider. Little is know n about what students are actually being taught during tutoring sessions beyond what is listed in the SES providers state application, on the providers website, or in marketing material s. Although NCLB encourages SES providers to align their curriculum with state st andards, it clearly prohibits stat es and districts from trying to influence a providers curriculum (Burch, 2007). A study conducted by Anderson & Laguarda (2005) noted a lack of coordination between the SES provider and the teacher. Many teachers in the study reported that they did not know which of their students were participating in the SES program and felt that collaboration with the tu tors would help their students. Collaboration between teachers and SES providers must be required so that tutoring will be meaningful to students. The next quality indicator that should be required is that of tutor qualifications. Experienced, certified teachers are the best tutors (Elbaum et al., 2000; Gordon, 2003; Gordon et al., 2004; Wasik, 1998). Teachers make effective tutors because they are specifically educated to teach. Tutoring enables them to individualize instruction, which th ey are often unable to do in a classroom setting. Their education and professi onal experience make teachers key in providing

PAGE 92

92 effective tutoring (Gordon et al., 2004). Currentl y, the Florida Department of Education requires that SES tutors meet the minimum standards for Title I paraprofessionals. These minimum standards include a high school diploma and tw o years of college, an associates (or higher) degree, or a passing score on a test required of paraprofessionals who are employed in Title I schools. The qualifications of tutors ought to be revised to prohibit anyon e from tutoring who is not a certified teacher. Teachers are specifically trained in differentiated instruction and can individualize instruction based on a students need s. Title I students have special academic needs that are most likely to respond to the expe rience and expertise of certified teachers. Tutors are most effective when they receive extensive training. Tutors with specialized training consistently produce highe r levels of student achievement than tutors with little or no training (Gordon et al., 2004).Tra ining must continue throughout the duration of the tutoring services and must include monitoring of tutors and students during tuto ring sessions, followed by immediate feedback (Wasik, 1998). Although traini ng programs which typi cally consist from 4 to 20 hours are provided by some SES providers, not all providers train their tutors (Ascher, 2006). Tutoring training is not re quired by NCLB and the Supplemental Educational Services Non-Regulatory Guidance does not address it (U. S. Department of Education, 2005). Curriculum implementation, behavior management, and individualized instru ction are just a few of many important tools n ecessary to deliver effective instruc tion in a tutoring environment. SES tutors would benefit from specific, ongoing tr aining to tutor SES students effectively. Tutoring delivered over the entire school year with the same tutor working with the same student was found to be most effective. Effective tutoring should be long term, intensive, and consistent (Baker et al., 2006; Gordon, 2003; Juel, 1996; Wasik, 1998). The SES program is designed to provide each student with a specific allo cation to fund tutorin g. When the allocation

PAGE 93

93 is depleted, services automati cally end for each student (U. S. Department of Education, 2005). A typical tutoring program provi des 30 hours of free tutoring (Hess & Finn, 2004a). It should also be noted that the frequency or intensity of service can vary b ecause NCLB has no set requirements (Kasmin & Farmer, 2006). If tutori ng is provided by a certificated teacher using research based curriculum in a low student to te acher ratio, it may not be effective because of the short length of time SES is offered. The U.S. De partment of Education currently allows SES providers to set their own hourly rate within the para meters established by each state. The hourly rate in Florida for the 2007-08 school year ranges from a low of $20 per hour to a high of $80 per hour, with an average being $60 an hour (F lorida Department of Education, n.d.). The average student alloca tion for the 2006-07 school year was $1,124 (Florida Department of Education, 2007d). Therefore, ba sed on the parameters set for SES providers hourly rates, a student could be tutored for 56 hours at $20 per hour or 14 hours at $80 per hour. Hourly rates need to be established by the Florida Departme nt of Education to allow students to receive consistent, long term tutoring serv ices. In addition, to enable tuto ring to be long term, the student allocation must be increased. If hourly rates ar e established by the Florida Department of Education and the student allocation is increased more tutoring hours would be provided to each student. Implications Per SES f ederal guidelines, it is the states re sponsibility to evaluate SES providers to ascertain whether they are effective in raising the academic achievement of students who receive tutoring. Although billions of federal education do llars are pumped into this program yearly, no state has completed a comprehensiv e evaluation of the effectiveness of SES. Research has shown that states are struggling with evaluation desi gn and funding for a compre hensive assessment of SES effectiveness (Minnici & Bartley, 2007; U. S. Government Accountability Office, 2006).

PAGE 94

94 SES evaluation is complicated and costly, and the U. S. Department of Education must take the lead in designing, as well as funding, an effective SES evaluation process. Current evaluations are typical ly conducted by large, urba n school distri cts concerned about the amount of money invested in a program with no proven results. There is considerable research to confirm the effectiveness of tutoring, but most evaluations completed on SES do not show positive results on increasing the academic ach ievement of students (Lauer et al., 2004; Moss et. al., 2001). Although SES has been implemented in Florida since the 2004-2005 school year, there has been no statewide evaluation of SES. The only avenue to remove ineffective SES pr oviders from the stat e-approved list is by state evaluation (U. S. Department of Educati on, 2005). Lack of state ev aluation forces school districts to enter into contracts year after year with SES providers whose effectiveness is not documented, and per the results of this study, ar e questionable, thereby wasting millions of federal dollars. More importantly, students who participate in SES are not provided the help they need to increase their academic skills. States must begin evalua ting SES providers; those who are effective should remain on the approved list and ineffective providers must be removed. If the overall evaluation results indicate that there are no effective SES providers, then the focus must shift to defining a program design that will increase student academic performance. A plethora of research exists that addresses the effectiveness of tutoring, and this information must be used to increase the effectiveness of this multibillion dollar program. To facilitate additional planning, future resear chers should be aware of the data collection problems experienced in this study. This study wa s designed to be a statewide assessment of SES, but the statewide data could not be obtained from the Florida Department of Education in a timely manner. The Florida Department of Educa tion puts data requests in a queue and the wait

PAGE 95

95 time is 18 months to have a data request reviewed Thus, the most current data is not available for use in such a study. Although Fl orida is considered by many to have the most advanced data system in the nation, essential data were not made available for this study. This roadblock required redesign of this study to include the seven largest urban sc hool districts in Florida. One of the districts initially include d in the study required payment of a substantial fee for processing the needed data. After much thought and convers ation, that district wa s excluded and the study was again redesigned to include si x urban Florida school districts. Timely receipt of data still produced complications in that it took eight months to receive data from two of the districts. Therefore, creating a timeline that allows ample time for redesign and data co llection is essential. Future Research The goal of the SES program to increase the academic achievement of Title I students is an important goal in todays education system. There are many problems facing educators in schools today, and new ways to educate student s effectively must be found. Decreasing the dropout rate, eliminating the achievement gap, and educating students to succeed in a global economy are worthy goals, and NCLB was implem ented in response to these needs. NCLB provisions are a starting poi nt, but school districts, regardless of their leve l of effectiveness, are currently unable to follow them. This study opens conversation on the effectiveness of SES, but future research is needed. The following recommendations address needs for future research regarding SES effectiveness. 1. A longitudinal study that follows SES students from year to year is needed to assess the cumulative effect of long term tutoring. Consis tency of results across multiple years would strengthen conclusions regard ing program effectiveness. 2. FCAT developmental scale scores are designed to track a students progress from year to year. Analyses that show statistically si gnificant results should be compared to the expected increase in scale scores from grade level to grade level to assess the actual academic achievement.

PAGE 96

96 3. In this study FCAT developmental scale scor es were used as the measurement tool. From an assessment perspective, this is a rather blunt instrument to use for this purpose. In future research to better understand and interpret the results, additional measures are needed to assess the effectiveness of SES tutoring. 4. Future evaluations should include systematic sampling to form a control group that groups students by pre-test FCAT developmental scale sc ores, in addition to gr ade levels, that are aligned with the experimental group. 5. Consideration must be gi ven to the confounders in this study. For example, SES is a parent-driven program in which a child can only receive tutoring if a parent completes and returns the tutoring application. Analysis shou ld be completed on the differences between parents who request services a nd parents who do not. This would help in determining if the differences in parental support are a contributing factor to the success or lack of success in advancing the academic achievement of SES students. 6. Another confounder to address in future res earch is the individual attributes of each SES school. Schools that have not made AYP for se veral years receive inte nse remediation with several new programs and strategies every ye ar. Adjustments in staff, curriculum, and instruction are ongoing in an attempt to provide an educational program that is effective in raising students academic achievement. If an SES school increases the academic achievement of its students, can this increase be entirely attributed to a tutoring program that lasts a few weeks? If not thought must be given to methods for isolating increased achievement as a result of SES or othe r educational programs and strategies. 7. Tutoring is an effective tool in raising th e academic performance of students (Moss et al., 2001; Lauer et al., 2004). Future research s hould focus on the implementation and fidelity of the SES program. Several quality indicators of successful tutoring (Table2-1) have been identified and future research should focus on the implementation of these indicators in the SES program. Summary The No Child Left Behin d Act was implemented in 2002 in an effort to decrease the achievement gap between student groups by pr oviding additional academic opportunities for Americas Title I students in low achieving sc hools. Supplemental Educational Services was included as one of these academ ic opportunities and was designed to provide students with tutoring. Although SES is specifically designed to increase the academic achievement of participating students through tutoring, there ha s been no evaluation of the success of this program (U. S. Government Accountability Office, 2006). Federal guidance places the

PAGE 97

97 responsibility of evaluating SES on the state, but there have been no evaluations completed by the Florida Department of E ducation. Therefore, the goal of this study was to assess the effectiveness of SES in increasing academic achievement for students This study focused on assessing the increased academic performance of students on the FCAT as measured by developmental scale scor es. The results found increased developmental scale scores for students who received tutoring in math, but there were no statistically significant findings on increased achievement for students w ho received tutoring in reading or in reading and math. This study also assessed 3 of the largest SES providers in each of the 6 districts that participated in this study. Of those 18 providers assessed, only 3 provided results that confirmed increased academic achievement. Although there was some increased academic achievement found in this study, the number of students benefiting from SES must be significantly increased to justify the time and money spent on implementing this tutoring program in Am ericas schools. At the time of this study, the reauthorization of NCLB is only months away. Therefore, this is the optimal time to make needed adjustments and revisions. Michael Kirs t (2004), a veteran policy analyst, found that it took more than 10 years and many legislative and administrative modifications before the initial Title I program in the 1965 Elementary and Secondary Education Act achieved its intent. Kirst stated that this is typical of aggressive new federal programs. The overall goal of Title I is to increase the academic achievement of students who live in poverty, and all programs funded through Title I must meet this goal. Therefore, SES must be evaluated and subsequent action must be taken to ensure its success, or the program needs to be abandoned and the funds returned to Title I schools where they can be used to meet the academic needs of students.

PAGE 98

98 Table 5. Summary of Findings for Hypotheses 1 through 3 Hypothesis Findings Status of Null Hypothesis___ Null Hypothesis 1 There is a relati onship between SES Rejected tutoring and increased FCAT developmental scale scores in math (p=.001) Null Hypothesis 2 There is no relationship between Retained SES tutoring and increased FCAT developmental scale scores in reading (p=.117) Null Hypothesis 3 There is no relationship between Retained SES tutoring and increased FCAT developmental scale scores in math and reading (Math p=.246; Reading p=.646) ______________________________________________________________________________

PAGE 99

99 Table 5. Summary of Fi ndings for Hypothesis 4* District SES Provider Subject p = Status of Null Hypothesis 4_ 1 Cool Kids Reading .984 Retained Rocket Learning Reading .687 Retained Rocket Learning Math .307 Retained Education Station Reading .640 Retained 2 Academy of Success Reading .684 Retained Academy of Success Math .947 Retained Advanced Learning Reading .032** Rejected Advanced Learning Math .022** Rejected A+ Tutor U Reading .632 Retained A+ Tutor U Math .146 Retained 3 Club Z! Reading .134 Retained Club Z! Math .059 Retained A+ Tutor U Reading .613 Retained Rocket Learning Reading .009** Rejected 4 AlphaBEST Reading .978 Retained AlphaBEST Math .208 Retained Huntington Learning Reading .160 Retained Studentnest Math .004** Rejected 5 A Wise Choice Reading .424 Retained Huxtable Education Reading .378 Retained A+ Tutor U Reading .698 Retained 6 ACES Reading .748 Retained ACES Math .680 Retained Achievia Reading .571 Retained PCTA Reading .260 Retained PCTA Math .251 Retained Hypothesis 4 states that there is no statistically significant difference between 2006 FCAT developmental scale scores as the pre-test and 2007 FCAT developmental scale scores as the post-test for students receiving at least 10 hours of tutoring from the three SES providers who tutored the largest number of students in each district. ** Statistically Significant

PAGE 100

100 APPENDIX A UNIVERSITY OF FLORIDA INSTITUTIONAL REVIEW BOARD

PAGE 101

101 APPENDIX B REQUEST FOR DISTRICT FCAT DATA July 20, 2007 XXXX County Public Schools Mr. XXX XXXX Manager Information Services XXX XXXXXXX XXXX, Florida XXXXX Dear Mr. XXXXX: Please find enclosed the required document s to gain permission from the XXXXXX School District to obtai n FCAT data for Title I students receiving Supplemental Educational Services (SES). I am currently pursuing a docto ral degree at the University of Florida and request permission from your distri ct to analyze the requested data to determine the effectiveness of NCLBs SES program for Florid a students. As well as being a UF doctoral student, I am an administrator for Duval County Public Schools in the Title I Office where my job responsibilities include coordinating the SES Progr am for our district. My dissertation will analyze FCAT scores for Title I students in the seven largest urba n Florida school districts, which includes: XXXXX XXXXX XXXXX XXXXX XXXXX XXXXX XXXXX This evaluation will be done by using FCAT test scores for two consecutive years (2006 and 2007) for students in urban Florida distri cts who have received SES services. FCAT developmental scale score gains will be used to determine academic growth. I am required by the University of Florida to submit my request to the University Institutional Review Board. I have enclosed a copy of this form and subsequent ap proval letter for your revi ew. I anticipate needing this data in September 2007, as I would then offici ally be a Ph.D. candidate at the university and would begin the data analysis of the requested FCAT scores. Dr. David Qu inn is my dissertation committee chair and will supervise the dissertation process for the University of Florida. I am submitting my request early in an effort to m eet your requests and requirements in a timely manner. I have discussed this study with Mr. XXX XXXXX, Director of Federal Programs of XXXXXX County Public Schools and will be happy to share my results with his office, as well as any other department in your school district The results of this study will be useful in

PAGE 102

102 enhancing and improving the SES Program in your dist rict, as well as other di stricts in Florida. I will also be submitting this study to the Florida Department of Education Bureau of Public School Options, which implements SES for the entire state. This request was also submitted to Ms. Teresa Miller, Director of Educational Policy, at Integrated Education Data Systems at FDOE in an effort to obtain all needed FCAT scores from her department, but was told that it would be at least an 18 month wait fo r this data. Due to my dissertation timeline and requirements, I cannot wait that length of time to begin the data analysis needed, so I am contacting each school district to gain permission to use this data. Enclosed you will find your districts requi red research form, a proposal narrative, a literature review, an example Excel spread sheet that lists the FCAT data needed, and the IRB form submitted to the University of Florida, as well as the s ubsequent approval letter. Please consider my request to use this student data in my research. If you have any questions please do not hesitate to contact me. I can be reached at XXX-XXX-XXXX or on my district cell at XXXXXX-XXXX. Thank you for your consideration. Sincerely, Myrna Allen Supervisor of Suppl emental Instruction Title I Office Duval County Public Schools 1701 Prudential Drive Jacksonville, Fl 32207 Office: 904-390-2123 Fax: 904-390-2634 allenm2@educationcentral.org cc: Dr. David Quinn Mr. XXXXXX Mr. XXXXXX

PAGE 103

103 APPENDIX C DISTRICT ONE INSTITUTIONAL REVIEW BOARD Superintendent of Schools Office of Program Evaluation The approval number for your study is 1382. This number should be used in all communications to clearly identify the study as approved by the Research Review Committee. The approval expires on June 30, 2008. During the approval period, the study must adhere to the design, procedures and instruments which were submitted. The compute r -generated data for the study will be provided by Mr. XXXXXX of the Office o f Program Evaluation of the XXXX. Contact him at XXX-XXX-XXXX to make the necessary arrangements. July 30, 2007 I am pleased to inform you that the Research Review Committee of the XXXXXX County Public Schools has approved your request to conduct t he study, "The Effectiveness of Supplemental Educational Services in Urban Florida School Districts." The approval is granted with the following conditions: 1. The anonymity and confidentiality of all subjects must be assured. 2. The compute r -generated data which are provided by theXXXXX will be eithe r aggregated or coded to ensure the subjects' anonymity. 3. The study is based on anonymous student records, so parent permission forms are not required. 4. The study will involve approximately 34,000 XXXXXX students. It should be emphasized that the approval of the Research Review Committee does not constitute an endorsement of the study. It is simply a permission to request the voluntary cooperation in the study of individuals associated with the XXXXX. It is your responsibility to ensure that appropriate procedures are followed in requesting an individual's cooperation, and that all aspects of the study are conducted in a professional manner. With regard to the latter, make certain that all documents and instruments distributed within the XXXXXXX as a part o f the study are carefully edited. Ms. Myrna Allen XXXXXXXXXX Jacksonville, FL XXXXX Dear Ms. Allen:

PAGE 104

104 If there are any changes in the study as it relates to the MDCPS, it may be necessary to resubmit your request to the committee. Failure to notify me of such a change may result in the cancellation of the approval. Sincerely, A PPROVAL NUMBER: 1382 A PPROVAL EXPIRES: 6-30-08 If you have any questions, please call me at XXX-XXX-XXXX. Finally, remember to forward an abstract of the study when it is complete. On behalf of the Research Review Committee, I want to wish you every success with your study. Chairperson Research Review Committee XXX:XX cc: Mr. XXXXXXXX

PAGE 105

105 APPENDIX D DISTRICT TWO INSTITUTIONAL REVIEW BOARD

PAGE 106

106 APPENDIX E DISTRICT THREE INSTIT UTIONAL RE VIEW BOARD RESPONSE OF XXXXXXX COUNTY PUBLIC SCHOOLS INSTITUTIONAL REVIEW BOARD SUBMISSION (CONTINUED ON NEXT PAGE) August 7, 2007 Your research project is approved but pleas e be aware that SES data are not readily accessible in the district. It would require substantial work on your part to retrieve the SES data. FCAT and demographic data could be provided. The contact person for Title 1 SES Services is XXX (XXX) XXX-XXX. Senior Director Accountability, Research, and Assessment Telephone: XXX-XXX-XXXX Fax: XXX-XXX-XXXX

PAGE 107

107

PAGE 108

108 APPENDIX F DISTRICT FOUR INSTITUTIONAL REVIEW BOARD November 9, 2007 Ms. Myra Allen Duval County Public Schools Title I Office 1701 Prudential Drive, Room 406 Jacksonville, FL 32207 Dear Ms. Allen: The Superintendent's Research Review Committee approved your request to conduct research concerning The Effectiveness of Supplemental Educ ation Services (SES) in Urban Florida School Districts, in the School District of XXXXXXX County. The purpose of this study is to analyze FCAT scores forTitle I students receiving Supplemental Educational Services (SES) and a comparison group t hat did not receive services, but have the same demographics. This report will assess whether the additional academ ic instruction giv en by SES providers increases the academic achievement of students in low-performing schools in the seven largest urban Florida school districts. To conduct the survey, you will Contact the Department of Supplemental Educational Services to identify a contact person; Request the contact person to provide 2006 and 2007 FCA T scores in Math/Reading, as well as, other needed data for all students who have received SES services and for a control group of students who were not enrolled in SES; Engage in no direct student contact; Receive only student data with the identity of all students deleted. As you conduct your research, please use the following guidelines: Submit to this office, a signed Affidavit of Good Moral Character for each researcher before they begin (A blank affidavit form is enclosed); Obtain permission from the principal or department administrator before beginning; In the case of student subjects, obtain written permission from the parent or guardian before proceeding; Provide a copy of all completed and signed parental/guardian consent forms to the principal or principal's designee;

PAGE 109

109 All collection activities involving students must occur in the presence of school staff members; If your research requires the use of additional resources in the future, you must first submit a written request to this office and then wait for a response before proceeding; One copy of the study results with an executive summary must be submitted to the Department of Research and Evaluation no later than one month after completion of the research; Your research activities at the school must not occur during the testing window of the Florida Comprehensive Assessment Test (FCAT). The FCAT testing window includes pre-test, administration, and post-test activities fr om February 1, 2008 through March 28, 2008. XXXXXXXXX,Director Research and Evaluation XX:XX Enclosure c: XXXX XXXX Page 2 Myra Allen The Effectiveness of Supplemental Educational Services in Urban Florida School Districts A ccording to our District's proced ures, participation is voluntary. Thank you for your interest in ou r school district.

PAGE 110

110 APPENDIX G DISTRICT FIVE INSTITUTIONAL REVIEW BOARD September 11, 2007 Dear Ms. Allen: Your request to conduct research in XXXX County Public Schools has been approved. This approval applies to your project in the form and content as submitted to this office for review. Any variations or modifications to the approved protocol must be cleared with this office prior to implementing such changes. Due to the fact that you are activ ely employed by XXXX County Public Schools, the data upon which your study is based has not been disguised in any way. However, no child or group of childre n is to be mentioned directly or made personally identifiable in the final product of your research. Upon completion of the study, it is customar y to forward a copy of the finished report to the Office of Instructiona l Research and Accountability, XXXXXX XXXXXXXXX, XXXXXXXX, Florida XXXXX. This office also shall be notified, in advance, of the publication of any reports/articles in which XXXX County is mentioned by name. If you have questions or concerns, pl ease don't hesitate to call me at XXXXXXX. General Director Instructional Research and Accountability

PAGE 111

111 APPENDIX H DISTRICT SIX INSTITUT IONAL REVIE W BOARD August 13, 2007 Ms. Myrna Allen Duval County Public Schools Title 1 Office 1701 Prudential Drive, Room 406 Jacksonville, Fl orida 32207 Dear Ms. Allen: I received your request to conduct re search in XXXXXXX County. Your study, "The Effectiveness of Supplemental Educational Se rvices in Urban Florida School Districts proposal number 070708-04 has been reviewed and the research has been approved. Your research request does not require sc hool; teacher or student's participation; therefore please contact our office to make arrangement to receive data you have requested. If there are ay questions or if additional information is needed, please contact ou r Research & Accountability Department at (XXX) XXX-XXXX. Once the research is completed please forw ard a copy of the results to my office. Sincerely, XXXXXXXXXX, Ph. D. Director, Program Evaluation XX:x

PAGE 112

112 LIST OF REFERENCES The 2006 Florida State Statutes (2006). Retrieved March 29, 2007 from http://flsenate.gov/ statutes/index.cfm?App_mode=Display_Sta tute&Search_String=&URL=CH1008/SEC33 1.HTM&TITLE=->2006->CH1008->Section%20331#1008.331. Anderson, L. M., & Laguarda, K. G. (2005). Case studies of Supplemental Services under the No Child Left Behind Act: Findings from 2003-04 Washington, DC: U. S. Department of Education, Office of Planning, Evaluation and Policy Development. Anderson, L.M., & Weiner, L. (2004). Early implementation of Supplemental Educational Services under the No Child Left Behind Act. Washington, DC: U. S. Department of Education, Office of the Under Secretary. Ascher, C. (2006). NCLB's Supplemental Educationa l Services: Is this what our students need? Phi Delta Kappan, 88 136-141. Baker, J. D., Rieg, S. A., & Clendaniel, T. ( 2006). An investigation of an after-school math tutoring program: University tutors + el ementary students = successful partnership. Education, 127 287-293. Beyond the pale: SES provisions emerge as 'loos e cannon' reform [Electronic (2006). Version]. American Teacher, 91, 7. Retrieved December 20, 2006 from http://www.aft.org/ pubs-reports/american_teacher/oct06/nclb.htm. Borja, R. R. (2006). Market for NCLB tutoring fa lls short of expectations [Electronic Version]. Education Week, 26, 5-13. Burch, P. (2007). Supplemental Education Services unde r NCLB: Emerging evidence and policy issues. Tempe, AZ: Arizona State University. Chicago Public Schools. (2005). SES tutoring programs: An evaluation of the second year part one of a two part report. Chicago: Office of Research, Evaluation and Accountability. Code of professional conduct and business et hics for Supplemental Educational Services provider. (2005). Retrieved December 20, 2006, from http:// www.educationindustry.org /tier.asp?downloadid=66. Cohen, J. H. (2006). 'Supplemental Services': Theory vs. practice. Education Week 25, 34-35. Cohen, P. A., Kulik, J. A., & Kulik, C. C. ( 1982). Educational outcomes of tutoring: A metaanalysis of findings. American Educational Research Journal, 19 237-248. Duval County Public Schools. (2007). Title I, Part A budget for 2007-08. Unpublished spreadsheet.

PAGE 113

113 Elbaum, B., Vaughn, S., Hughes, M. T., & Moody, S. W. (2000). How effective are one-to-one tutoring programs in reading for elementary students at risk for reading failure? A metaanalysis of the intervention research. Journal of Education Psychology 92 605-619. Florida Department of Education. (n.d.). Bureau of Public School Options approved SES providers. Retrieved May 10, 2007, from http://data.fldoe.org/ses/search. Florida Department of Education. (2006a). NCLB monitoring work papers. Tallahassee, FL: Office of Title I Programs and Academic Intervention Services. Florida Department of Education. (2006b). NCLB school choice leaders hip summit: Building on the school day Tallahassee, FL: Bureau of Public School Choice. Florida Department of Education. (2007a). 1st preliminary fiscal yea r 2007-08 Title I, Part A allocations distribution table. Tallahassee, FL: Office of Title I Programs and Academic Intervention Services. Florida Department of Education. (2007b). Assessment and accountability briefing book. Tallahassee, FL: Office of Accountab ility, Research, and Measurement. Florida Department of Education. (2007c). NCLB choice: Supplemental Educational Services update for 2006-2007 school year. Tallahassee, FL: Bureau of Public School Choice. Florida Department of Education. (2007d). SES per pupil allocation 2007-2008. Tallahassee, FL: Office of Title I Programs and Academic Intervention Services. Fusarelli, L. D. (2007). Restricted choi ces, limited options: Implementing choice and Supplemental Educational Services in No Child Left Behind, Educational Policy, 21 (1), 132-154. Gewertz, C. (2005). States wrestle with how to evaluate tutoring [Electronic Version]. Education Week 24, 5-5. Retrieved March 16, 2007. Gordon, E. E., & Gordon, E. H. (1990). Centuries of tutoring: A hist ory of alternative education in American and western Europe Lanham, MD: United Press of America. Gordon, E. E. (2003). Looking beyond the stereotype s: Ensuring the true potential of tutoring. Phi Delta Kappan, 84 456-459. Gordon, E. E., Morgan, R. R., Ponticell, J. A ., & O'Malley, C. J. (2004). Tutoring solutions for No Child Left Behind: Research, practice, and policy implications. NASSP Bulletin, 88 59-68. Grossman, K. N. (2007). $50 million for what? [Electronic Version]. Chicago Sun-Times Retrieved May 15, 2007 from http://www.suntimes.com/news/education/385515,CSTNWS-tutor15.article.

PAGE 114

114 Heistad, D. D. (2006). Analysis of 2005 Supplemental Educati on Services in Mi nneapolis Public Schools: An application of matc hed sample statistical design Minneapolis, MN: Minneapolis Public Schools. Hess, F. M., & Finn, C. E. (2004a). Inflating the life rafts of NCLB: Making public school Choice and Supplemental Services work for students in troubled schools. Phi Delta Kappan, 86, 34-40. Hess, F. M., & Finn, C. E. (2004b). Leaving no child behind? Options for kids in failing schools New York: Palgrave Macmillan. Juel, C. (1996). What makes literacy tutoring effective? Reading Research Quarterly, 31 268289. Kasmin, M. S., & Farmer, G. L. (2006). The prom ise of Supplemental E ducational Services: Is the policy failing? Children & Schools, 28 181-185. Kennedy, E. (2005). The No Child Left Behind Act. Human rights: Journal of the Section of Individual Rights & Responsibilities, 32 (4), 16-18. Kirst, Michael. (2004). To glimpse NCLB's future, look to the past. The Education Gadfly, 4. Retrieved December 12, 2007, http://www.edexcellence.net/foundation/ gadfly/issue.cfm? edition=&id=129#1610. Lauer, P. A. (2006). Out-of-school-time program s: A meta-analysis of effects for at-risk students. Review of Educational Research, 76 275-313. Lauer, P. A., Akiba, M., Wilkerson, S. B., Apthorp, H. S., Snow, D., & Martin-Glenn, M. (2004). The effectiveness of out-o f-school-time strategies in assisting low-achieving students in reading and mathema tics: A research synthesis Aurora, CO: Mid-continent Research for Education and Learning. Minnici, A., & Bartley, A. P. (2007). State implementation of Supplem ental Educational Services under the No Child Left Behind Act Retrieved May 15, 2007 from http://www.cepdc.org/_data/n_0001/resources /live/CEP-SES.pdf. Moss, M., Swartz, J., Obeidallah, D., Stewar G., & Green, D. (2001). AmeriCorps tutoring outcomes study. Retrieved January 5, 2007 from http://www.abtassociates.com/ reports/tutoring_0201.pdf. Northwest Regional Educational Laboratory. (2004). Evaluating Supplemental Educational Service providers: Issues and challenges. Retrieved January 12, 2007 from http://www.nwrel.org/planni ng/reports/esesp/esesp.pdf.

PAGE 115

115 Peterson, P. E., & West, M. R. (2003). No child left behind? The polit ics and practice of school accountability Washington, DC: Brookings Institution Press. Pines, S. (2006). Briefing by Mr. Steven Pines, Execu tive Director; Education Industry Association for the Senate Subcomm ittee on Education and Early Childhood Development. Retrieved January 12, 2007 from http ://www.educationindustry.org/EIA/ files/ccLibraryFiles/Filename /000000000125/EIA%20Testimony%20to%20 Senate%20HELP%20Comm%207-25-06.pdf. Potter, A., Ross, S. M., Paek, J., & McKay, D. (2006). Supplemental Educational Services in the state of Tennessee: 2005 2006 Memphis, TN: The University of Memphis, Center for Research in Educational Policy. Rees, N. S. (2006). MIA on SES [Electronic Version]. The Education Gadfly Retrieved January 12, 2007 from http://www.edexcellence.net/foundation/gadfly/issue.cfm?edition= &id=237#2775. Reid, K. S. (2004). Federal law spurs private companies to market tutoring [Electronic Version]. Education Week 24 1, 18-19. Retrieved January 5, 2007 from http://edweek/org//ew/ articles/2004/12/08/15tutor.h24.html. Rentner, D. S., Scott, C., Kober, N., Chudows ky, N., Chudowsky, V., Joft us, S., et al. (2006). From the capital to the classroom: Ye ar 4 of the No Child Left Behind Act Retrieved December 12, 2007 from http://www.cep-dc.or g/_data/global/nidocs/CEP-NCLB-Report4.pdf Rickles, J. H., & Barnhart, M. K. (2007). The impact of Supplemental Educational Services participation on student achievement: 2005-06 (No. Planning, Assessment and Research Division Publication No. 352) : Los Angeles: Los Angele s Unified School District, Program Evaluation and Research Branch. Sanderson, D. R. (2003). Setting up a successful afte r-school tutorial program: One district's journey. Reading Improvement, 40 (1), 13-20. Schwartzbeck, T. D. (2005). Understanding Supplemental Educational Services. School Administrator, 62(5), 12-13. Smole, D. P. (2004). Supplemental Educational Services for children from low-income families under ESEA Title I-A (No. RL31329). Washington, DC: The Library of Congress. Stullich, S., Eisner, E., Mc Crary, J., & Roney, C. (2006). National Assessment of Title I interim report: Volume I: Implementation of Title I (No. NCEE 2006-4001. Washington, DC: U. S. Department of Education, Institute of Education Sciences. Sunderman, G. L. (2006). Do Supplemental E ducational Services in crease opportunities for minority students? Phi Delta Kappan 88 117-122.

PAGE 116

116 Sunderman, G. L., & Kim, J. (2004). Increasing bureaucracy or increasing opportunities? School district experience with Supplemental Educational Services Cambridge, MA: The Civil Rights Project at Harvard University. U. S. Department of Education. (n.d.). Elementary and Secondary Education Part A Improving basic programs operated by lo cal educational agencies. Retrieved May 10, 2007 from http://www.ed.gov/policy/els ec/leg/esea02/pg2.html#ses. U. S. Department of Education. (1997). Evidence that tutoring works Washington, DC: Office of the Under Secretary Pla nning and Evaluation Service. U. S. Department of Education. (2002). No Child Left Behind: A desktop reference Washington, DC: Office of Elementary and Secondary Education. U. S. Department of Education Office. (2003). When schools stay open late: The national evaluation of the 21st-Century Co mmunity Learning Centers Program Washington, DC: Office of the Under Secretary. U. S. Department of Education. (2005). Supplemental Educational Services non-regulatory guidance. Retrieved April 8, 2007 from http://www.ed.gov/policy/ elsec/guid/suppsvcsguid.doc. U. S. Department of Education. (2006). Fiscal year 2007 budget summary February 6, 2006. Retrieved May 11, 2007, from http://www.ed.gov/about/overview/ budget/budget07/summary/edlite-section1.html. U. S. Department of Education. (2007). No Child Left Behind: Helping families by supporting and expanding school choice. Retrieved March 23, 2007 from http://www.ed.gov/ nclb/choice/schools/choicefacts.html. U. S. Government Accountability Office. (2006). No Child Left Behind Act: Education actions needed to improve local implementa tion and state evaluation of Supplemental Educational Services (GAO Publication No. 06-758) Washington, DC: Government Accountability Office. Vadasy, P. F., Jenkins, J. R., Antil, L. R., Wayne, S. K., & O'Connor, R. E. (1997). The effectiveness of one-to-one tutoring by commun ity tutors for at-risk beginning readers. Learning Disability Quarterly, 20 129-139. Viadero, D. (2007). Evidence thin on student ga ins from NCLB tutoring [Electronic Version]. Education Week. Retrieved June 12, 2007 from http://www.edweek.org/ew/ articles/2007/06/13/41ses.h26.html. Wasik, B. A. (1998). Using volunteers as reading tutors: Guidelines fo r successful practices. The Reading Teacher, 51 562-570.

PAGE 117

117 Zimmer, R., Gill, B., Razquin, P., Booker, & Lockwood, J. R. I. (2007). State and local implementation of the No Child Left Behind Act: Volume Title I choice, Supplemental Educational Services, and student achievement Washington, DC: Office of Planning, Evaluation and Policy Development.

PAGE 118

118 BIOGRAPHICAL SKETCH Myrna L. Allen graduated from Coventry High School in 1977 in Coventry, Ohio. After moving to Florida in 1988, she attended Florida Community College at Jacksonville and graduated with her associates degree in the spri ng of 1991. She then attended the University of Florida in the fall of that year and in 1993 she graduated Summ a Cum Laude with her Bachelor of Arts degree in education. Ms. Allen accepted her first teaching position at Switzerland Middle School in St. Johns County teaching middle school students with varyi ng exceptionalities for four years. During this time, Ms. Allen received her Master of Arts degr ee in educational leader ship and accepted her first administrative position as A ssistant Principal of Bannerman Learning Center, an alternative school for students in grades 6 through 12, in Clay County Public Schools. After working at Banner Learning Center for three years, she ac cepted a position in Duval County Public Schools as Assistant Principal of Southside Middle Schoo l where she was the administrator for the sixth grade. After several years at Sout hside Middle, Ms. Allen joined the district office and currently holds a position in the Title I Office as the dist ricts Supervisor of Supplemental Services. Included in Ms. Allens family is her son, Terry, daughter, Christine, son-in-law, Roger, and grandson, Braeden. She has resided in St Johns County since 1988 and, for leisure, she enjoys reading, gardening, cheering for the Florida Gators, and spending time with her family.


xml version 1.0 encoding UTF-8
REPORT xmlns http:www.fcla.edudlsmddaitss xmlns:xsi http:www.w3.org2001XMLSchema-instance xsi:schemaLocation http:www.fcla.edudlsmddaitssdaitssReport.xsd
INGEST IEID E20101111_AAAABS INGEST_TIME 2010-11-11T16:57:37Z PACKAGE UFE0021969_00001
AGREEMENT_INFO ACCOUNT UF PROJECT UFDC
FILES
FILE SIZE 2091 DFID F20101111_AABFYL ORIGIN DEPOSITOR PATH allen_m_Page_108.txt GLOBAL false PRESERVATION BIT MESSAGE_DIGEST ALGORITHM MD5
0b6f6032e17883ece78563016a09287d
SHA-1
181544a045037bc6d25860a23249899cebc2cf20
2631 F20101111_AABFBD allen_m_Page_107.txt
8bc3af4c8d6b0eace739bd1550364422
09d7ddb034ada6be414c918e7432161067cb6c05
19429 F20101111_AABGEF allen_m_Page_008.QC.jpg
d536211eabfc45d88d51a96e3bcf73fa
dffb3cada4f00b4ff81e5fb2964126c9ca2814df
6115 F20101111_AABGDR allen_m_Page_054thm.jpg
0fd479b85fab7d20feb055a3efe95d6a
afe09096ffacc83b23e1b3f3ed35195b32d8a58d
1295 F20101111_AABFYM allen_m_Page_109.txt
5ac9a420cb838f1029a023b45181d5e1
13e25a061b99f244698800dfd49a79df8265a242
2225 F20101111_AABFXX allen_m_Page_093.txt
ece2b32a675f18a9083cc01fbae1ea17
aeaf062becede77ecfd6e2444b59fd72528ad8b1
3948 F20101111_AABFBE allen_m_Page_006.txt
c1b84b400a322564286851f21f469475
5a8682e87641f60504c5b6bc24f0e6425e2f9be8
4813 F20101111_AABGEG allen_m_Page_008thm.jpg
9416d0289f52cc266b7e25af5f223313
535387701c7e4b2e9a4237ddb7ada078cef239d2
26720 F20101111_AABFZA allen_m_Page_083.QC.jpg
f05b61beabe0adfe8faf0d02a90eb627
7af39d08545786ec90265a1a4b897cd378ac4713
6566 F20101111_AABGDS allen_m_Page_086thm.jpg
5be6920ec45d63f9e4661fd1dee1d99b
b79faa094794ac290ad8428ecb5956aa2d513907
1356 F20101111_AABFYN allen_m_Page_110.txt
eb2d9a61c6b84e02e15dfde0af7038df
9fb0f6a2cbefad3e8bfeca1fd8f57b24ba4f4885
2184 F20101111_AABFXY allen_m_Page_094.txt
5b5a21e2bd29d2acf496e7309826fb16
7629f382d5903c1260c0e3c607471278b5d042b1
6626 F20101111_AABFBF allen_m_Page_017thm.jpg
e1bc867e05e0a882ec63b6bdd4ddca73
1f5929bd8c53c01b5293cd99912afe6b3c4ae4db
13549 F20101111_AABGEH allen_m_Page_010.QC.jpg
e89c0abe081c7556eacfebc91f5886da
bd567aa97f68488d3fd51d17a3e5159b7c779be6
28548 F20101111_AABFZB allen_m_Page_014.QC.jpg
d5bde1727e968d31c819979b910585fa
512a191331410bae6820f3671627fe80388cdbc6
28305 F20101111_AABGDT allen_m_Page_075.QC.jpg
9453171058a2197f366f8d4d0ea2fdfb
27f0108a446bf6dd21ee395f0f50ab36ea33c656
1028 F20101111_AABFYO allen_m_Page_111.txt
e80e57dbf378d338258d3af8add943ec
1782617aedd6d51a95948e10e4b008d112fcc2c4
2341 F20101111_AABFXZ allen_m_Page_095.txt
87947c3a812a045592ccc41f01af05b8
12324ea53740173d6ba0bf47dae10cab690af20e
25271604 F20101111_AABFBG allen_m_Page_034.tif
8e87427a3a3c9a136fbde65ebfeab22d
89b550005ae37ee6ed871780d6e1580376ccb136
24171 F20101111_AABGEI allen_m_Page_011.QC.jpg
559b1306613adb5328f6ac460410ac86
56ea8ef5285b226909d7ba15f0b8889126c0aab4
6740 F20101111_AABFZC allen_m_Page_087thm.jpg
70f54aba571a87ac79ee76d7dccd2c5e
2c5e6e052ede155d64a3ec15afb935842a4a3b37
4719 F20101111_AABGDU allen_m_Page_007thm.jpg
9dd07624a02aca8e8d6a3ed17c4fec70
92d0b46223ddecb9483bcdff922bc679a066f35b
2420 F20101111_AABFYP allen_m_Page_112.txt
8502cff9bdc29320a95ee75588d0813f
bea14f63b9c65c0100c7b78bca317fbe343c28c4
2140 F20101111_AABFBH allen_m_Page_030.txt
c377a6017313a773ee25a733122f65d8
8cfd780ed4e1d548083e6c61e5246cb10514fb1a
5821 F20101111_AABGEJ allen_m_Page_011thm.jpg
5325decd816aac584b4c939df38b0b38
403b0b067a8678dd7d6baab5c55bc4fe86faada3
30064 F20101111_AABFZD allen_m_Page_049.QC.jpg
1d30969dda5f175d3812ed967a5e3602
f3461193ba2fd5aef93614a01a8e47b5fa3951d8
6599 F20101111_AABGDV allen_m_Page_094thm.jpg
bbcf388dad11d9a488ed5ac179078271
2bbe70beaaa6e7603e034e278a6f7074f7f277d8
2623 F20101111_AABFYQ allen_m_Page_113.txt
7ded9e8b35061f5b4641731a6019d83d
a2feea9997d556798130b36efd6f307027ccf9b1
81844 F20101111_AABFBI allen_m_Page_009.jpg
27eaa5ebf70b43437baa52ae5c458720
cca577cf3653040e8a2b8bbd591dd987b022023c
2074 F20101111_AABGEK allen_m_Page_012thm.jpg
e89a1f4c85a122e2a3d41d3085154d55
520190fba5abe41ea2a542647f7245c3bd1f43e5
6627 F20101111_AABFZE allen_m_Page_062thm.jpg
1fa156c6fa8d371af10cea8265681649
2a224323da504cb2ae19cd671fa43a57a9f23ebe
2580 F20101111_AABGDW allen_m_Page_078thm.jpg
8fd4fd2e58bed75420b17a4508a2b2c8
1bdc7eafe1a5ca524a4f72cf014aeb6f34940fa3
2389 F20101111_AABFYR allen_m_Page_114.txt
63357c4554ab24e029e49d705a44b2d3
16aec0875aa03666b424edba4a487f1124aa7ac6
29397 F20101111_AABGFA allen_m_Page_037.QC.jpg
efaa21c9a2a9b39c0d318dd80aadbb77
6c9ccec8aa0a800dc2642316e743feaa88917364
F20101111_AABFBJ allen_m_Page_060.tif
06d47fb714c751e3cfd4042c72ad9260
edcc2360fdaccdebb48ccca2854f81c5240e2a34
27999 F20101111_AABGEL allen_m_Page_013.QC.jpg
cecd6320410e38a3b77937f1164fffe0
7da713e4779aebe1ba33f0f1a7139a9535fb025e
4439 F20101111_AABFZF allen_m_Page_117.QC.jpg
761fbed0d0f4c1974ce13888073ab129
1458288aaa8961c28debe779c5d1b9b5c3fa76b3
27258 F20101111_AABGDX allen_m_Page_088.QC.jpg
8d471417cde45dbac31da09a3f87bf22
13df7f68b7a13d8a4a5ce216d0be8cb326bc182c
2712 F20101111_AABFYS allen_m_Page_115.txt
11f86193c766c698190514154a99a989
04a4107d2ac659d33c532cabc64cecfe3f893af0
6830 F20101111_AABGFB allen_m_Page_037thm.jpg
3b55effb5ec29624a31bd9172173baec
4978c6ccd55eddff7633b8951a47ed51d3b2ced7
86665 F20101111_AABFBK allen_m_Page_090.jpg
2c57b2be5fac5b4fff16d765617543ff
de15552f51de52ea55979ef39b02fc3ae6bba0d1
29821 F20101111_AABGEM allen_m_Page_018.QC.jpg
d5ed80af1d650b1d956ca44deb446d14
365e830e38f4504e5c22926daa609017611b6c6c
27298 F20101111_AABFZG allen_m_Page_050.QC.jpg
c9023a4e01ae57ca99cb5ae507fb7eb8
4935d22f214bb5e74bacb9d02151551f1ecab4f5
6473 F20101111_AABGDY allen_m_Page_031thm.jpg
5f5cadf47ff8ee7634bea6ac9b7dbe2b
2b98ae98943192fe00ccb868e4f446f7d6e28ad7
1051951 F20101111_AABFCA allen_m_Page_091.jp2
7590d910ccfbfff247d36eefae061d85
93fb62e43860b6929135dd71d049947a8e78f361
28896 F20101111_AABGFC allen_m_Page_040.QC.jpg
6a5e793d07494425ff586824e716f47a
1300b1574fbe1ccb082e941b47c31304def2dd82
86727 F20101111_AABFBL allen_m_Page_089.jpg
5b0cc72e17361f2ec1a7bfd025113631
85477e19aece3654e4ce78807b3caf519d5641b1
26784 F20101111_AABGEN allen_m_Page_020.QC.jpg
15032bbae83a4460d0cae59b61649e4c
970f7a8ee9ddbcc9d3391e9887bbb4e2a71fe492
6663 F20101111_AABFZH allen_m_Page_046thm.jpg
32041b232fcc7642c591e215253c9a2d
7e0d5eebfc6ac741d5384f13f5596e9cf0417d54
19433 F20101111_AABGDZ allen_m_Page_007.QC.jpg
18a8bec744fcca90fa3fa05b052a34e8
9de19c7a5bd6675ccdc202a595277175755ab638
2577 F20101111_AABFYT allen_m_Page_116.txt
3cad71ac6c9d0e037d793e0736f4ee68
daaa75adea62fc7dabfccf9d9e2bb75ab1c16a7f
F20101111_AABFCB allen_m_Page_030.tif
26c8d7995b1d9905e240faf5a38d9731
c639e7644c495c5295f09309b9f8884f6f51fe7a
27836 F20101111_AABGFD allen_m_Page_042.QC.jpg
29f4aa53ace8a4e5976dac301c582d6c
6aa1a37eb226b2da76ed3d69ca778e10d2648f39
458 F20101111_AABFBM allen_m_Page_002thm.jpg
d322fbd1d14091c118b2708275d14e38
c6133454e339d3c8fd6f8cc27ce9ef43b2642773
28025 F20101111_AABGEO allen_m_Page_021.QC.jpg
e2ba5141055892edeca6f22ecd0ed02d
cb334a90e7ff0b0836d7f96c8ddd8ef0316d2079
2046 F20101111_AABFAY allen_m_Page_020.txt
da2e580df211e6a5a39e6e9845b8b6a8
ab34b178bc7dd77f75fc3e189053cc16ec2888a8
1066 F20101111_AABFZI allen_m_Page_003thm.jpg
129ac290a2cfabe76430d3db9133c4fc
e65a46c5633f8174e923fe72e3d690031dbe1f2e
329 F20101111_AABFYU allen_m_Page_117.txt
236d8fc1316d8e8d55dcd7b429836e3a
037ae97d8b5c2d66f0c5a3dfeb00e0415875967e
49302 F20101111_AABFBN allen_m_Page_099.pro
b3cf53f32fe8610be8f657c46397d7e2
ac64eddec7dc479ffae3d796f2200fd238b5edcc
6967 F20101111_AABGEP allen_m_Page_021thm.jpg
72f13663d6eb0a1681c4f8a5c7c29c87
b7f72d4ab5abac4cb9304d187c9d12d655cb6844
87393 F20101111_AABFAZ allen_m_Page_015.jpg
54d5d7605cfafbad8bb811e0a648acb8
23ac4ab7372eb5d5a7d3cd18ab790be3b853a3bb
27862 F20101111_AABFZJ allen_m_Page_046.QC.jpg
e2b1bcd76229e102d3c34ca90b202cfd
c79c1a6ea2e88be8ab87fd0a559ad912164bc823
1660 F20101111_AABFYV allen_m_Page_118.txt
a574515774d45b87d9bcf8070c44ea97
d17e4c2ba1f4f702d5c58e485a8061ee70fe1d2b
28951 F20101111_AABGFE allen_m_Page_043.QC.jpg
c10b9661bfe9c06de95969dd13922d75
4848b6f64c697137e919ca6c149d4fc76a2b1564
1051941 F20101111_AABFCC allen_m_Page_015.jp2
dc4123f265921fa69bfe5fb2edeec474
6867af5d785ea1fb8afdd9d42f3e5d2420baef1b
59306 F20101111_AABFBO allen_m_Page_112.pro
14fede295c3e6559db1029eef088f20c
6f78a5c22bf4c128f3b40e28ce73bd288deee670
27418 F20101111_AABGEQ allen_m_Page_022.QC.jpg
728559c2c06f18cbddb7b45ccc13016e
3376089ae21e52294a184aa8a5a45e436faaf3b7
27023 F20101111_AABFZK allen_m_Page_090.QC.jpg
7dcc9ae56e205914be497ff08ad138c0
2245ec2dddba76a030542fa82bce95816c9c1948
1718 F20101111_AABFYW allen_m_Page_001thm.jpg
ba2efa11dc32b790c32ade203b9fe51c
6238266e92d8e960000963aa8036eecccadb8e88
6780 F20101111_AABGFF allen_m_Page_043thm.jpg
5e8bed628c4285612424797def7430e7
26442a8940b07ecfe044fe538903f45c8e368367
87996 F20101111_AABFCD allen_m_Page_050.jpg
3b47dade8c1c25fb1fcf006b89843fb3
8e40fa2420e9f20c8c2ad2a888bbb3fd83b0dcfe
54416 F20101111_AABFBP allen_m_Page_031.pro
5f248b2d8f03d9270c7013467c7f2924
91e9bbbf020dedadcc9247edd459975e223055b2
6449 F20101111_AABGER allen_m_Page_022thm.jpg
844c1ccf7d788805459ab786f209efb7
43e512b07da3b59c8bafa0ae1f6047423d21cb8a
28780 F20101111_AABFZL allen_m_Page_032.QC.jpg
f56a591d6b53a1aa2ca390c15c4434fa
0320afddd841d90cec594462506379e7a9b52a40
28671 F20101111_AABFYX allen_m_Page_116.QC.jpg
cd570987ee34ddc3d8dd3905d92d7853
5ebb4fc4e49a881caaa4cd32cfb54b6da94bb7b5
6843 F20101111_AABGFG allen_m_Page_044thm.jpg
b0d3da3d205c35f205489b125e34c2a7
38c9326d087967919cd85c098ddea20b46e5580f
52127 F20101111_AABFCE allen_m_Page_088.pro
2f40f637b478da92aade7eba965c80e4
2d79fc39ecd1b23adb00900259b03f5e5ea16b06
F20101111_AABFBQ allen_m_Page_115.tif
1c1a2d5ce9566c1ed279132a11fe79ba
9e2be13f0294c3ab648f341c0182d001d730632f
6680 F20101111_AABGES allen_m_Page_023thm.jpg
34a39acdb4fb781b25f1edc36d0d2d36
820f88d538afddc290eaaab82bcf178b2dda2578
29882 F20101111_AABFZM allen_m_Page_044.QC.jpg
56f91d32fecf083f1d9845745923e15d
0f89ec147807ff1507584af50d67a9e8279c48fa
26414 F20101111_AABGFH allen_m_Page_045.QC.jpg
2d57b759c9355273c412336881ce73a3
037330ee5496f30fea828ccde2c3c885a92a0f6c
F20101111_AABFBR allen_m_Page_104.tif
d4dc5999e7dfcce0423aebcc13ed1e8e
c872ccf712398a1e1eceba66f97de6babf146c04
6050 F20101111_AABGET allen_m_Page_025thm.jpg
9c00151404362e0649788d1e6d022584
c019455164f073cd01193ff3871a7c97421ca7fc
6722 F20101111_AABFZN allen_m_Page_040thm.jpg
0ba2e126a8985ed69a0e43d9643fbef2
f328a5a94a78aeb95bc7a1e616060772309788ff
27756 F20101111_AABFYY allen_m_Page_052.QC.jpg
03c5d9a26d7460bedbde24a6968733b3
f89517436de20e7b3292a9359ea8b16f491e7ecd
11203 F20101111_AABFCF allen_m_Page_109.QC.jpg
0b633cc5a7cf9caa69b1511c8886f356
c9307eaf58374d4943752633f103eee86d8902be
6878 F20101111_AABGFI allen_m_Page_048thm.jpg
d2f8489d5787507cc6b8ab643f0feb49
e31d195cc4e91f0bf2c9772b30670271baf308db
2157 F20101111_AABFBS allen_m_Page_039.txt
f4965b3cdf26f546c27c29296200f2f7
28dcba5240753f962437e1645df9c9240972baac
1934 F20101111_AABGEU allen_m_Page_026thm.jpg
6c99658b086dc37041902801acd74ce1
17f08b6f714dcf2e68148604855d73dff1b534c8
6769 F20101111_AABFZO allen_m_Page_018thm.jpg
4e6dbc59e6622313860810900ffcd587
817ca0b014c089357f5b1da48f440eb7e668c2a6
5098 F20101111_AABFYZ allen_m_Page_118thm.jpg
c2d20d4b97786cdd983c425a5eccd67b
3a86e79e25746917e60232767770a0f08e7a435e
2189 F20101111_AABFCG allen_m_Page_032.txt
e5636b464def14d7150ae97e6d20f7b3
22fd3ba030a8f2cd8e163fb58cca8e9341a2dcc8
6504 F20101111_AABGFJ allen_m_Page_052thm.jpg
85fb8be516e1c83f1ed2d1ead138e668
b02ec744a7e9fda3e5c3bed1d833194dc7b960f3
10183 F20101111_AABFBT allen_m_Page_106.QC.jpg
cb6984a2e0a6f12ce9dcee00056918fa
032ec8bc15f72e5121894eb8ed9e29241288db74
6572 F20101111_AABGEV allen_m_Page_030thm.jpg
cf3128c902c96468375a9a3a9c50bf63
cb1d6095e373a2efa7c7b5dcf263249db4feb878
6222 F20101111_AABFZP allen_m_Page_039thm.jpg
debdbd61ac849bcdadef435b417d58d0
6c1a378247bfd4878933d61a1ddd6680f642f043
108292 F20101111_AABFCH allen_m_Page_116.jpg
34628e6a56a1ab04817788723c38f9f3
481550f451bf5fdc2b59384b973fdaae6e254275
6305 F20101111_AABGFK allen_m_Page_055thm.jpg
a4a48d349ddadd3aadd49d38deaaecf6
7ae113d62979b6464deacab4f8bc7245cd7aec02
6910 F20101111_AABFBU allen_m_Page_061thm.jpg
cba679a0d4a9feb773d20156e4bcd417
2d351f7558fc17ca75f6876e25fb7984325b6968
28328 F20101111_AABGEW allen_m_Page_031.QC.jpg
03daa297c003a92d7716b61f86176b17
a6ebb8c6699d692c87457f759da097d3b38a2420
17726 F20101111_AABFZQ allen_m_Page_102.QC.jpg
27da420138722017b6c87d37d0a00667
657c9f98bafe1c0d338e081ef55461a65091fbc7
1051980 F20101111_AABFCI allen_m_Page_057.jp2
35e9882a8a322db1641a57a2b6cd9804
d5b52fa3a4cdef793cce581c040bae840d2fc906
2696 F20101111_AABGGA allen_m_Page_077thm.jpg
7395420ff2425bf0cf330ebd22a631aa
2cac65de788589fd3810f7635ca6007adf03ebab
11443 F20101111_AABGFL allen_m_Page_056.QC.jpg
a220d2acced6351fd3c13cc678252a13
8a214c1b158c60d3771f0f6977e66121d552146f
1051926 F20101111_AABFBV allen_m_Page_074.jp2
1c58d3615ed761178a7266d39516ebce
022e35a23a79d853916651a0a725b31023819d5e
27033 F20101111_AABGEX allen_m_Page_033.QC.jpg
76d55175cd5e7f6d0c2d7d7a2b47b57c
0b91fdf483de9fbe9ab7374af55e8d0319459232
24455 F20101111_AABFZR allen_m_Page_058.QC.jpg
2c299e7cf906807babfd5a984b99eaad
a975728af9bfee625d6a38c153057760ceea0dad
28049 F20101111_AABFCJ allen_m_Page_087.QC.jpg
92cc9c87e0b39e7329c03ebb5931e491
264150ba562cef558dd85e461321a8746856b04b
9046 F20101111_AABGGB allen_m_Page_078.QC.jpg
2c4896d5d3d9f2580213182aaacd2e84
32ede56b2886dc52d62ed4ed2b3366969871fe00
6436 F20101111_AABGFM allen_m_Page_059thm.jpg
5590d0e7153b158639f059d890d7304d
1603330c0409a5dfede2cd2f5d61063e6dc664e5
652394 F20101111_AABFBW allen_m_Page_076.jp2
b054cc5c3df526f451724bd8a880381d
563885149491d4b08df419259962eac2ea4e2da9
6307 F20101111_AABGEY allen_m_Page_034thm.jpg
242f01f815514f3c3df722b19aa42339
a7f255b886306efad715f39abe50a4c441b2e75d
24936 F20101111_AABFZS allen_m_Page_054.QC.jpg
d514f12cf4bc208ce2b55385fc6a8e11
1bd7bef7184181a932926041e110d20ca1c29999
87804 F20101111_AABFCK allen_m_Page_022.jpg
365a908bf692aa669d54f8a6464233b4
15de4b4c21877f6579f830db90eb62786ad01bf3
1552 F20101111_AABGGC allen_m_Page_082thm.jpg
197a258767b29cb15961c7dd363697aa
2fc81d4b34366eee2e371c372eff07ed865ba923
29277 F20101111_AABGFN allen_m_Page_061.QC.jpg
3cc50bc0a4230841aa85933693c23d3e
e7228df60743ff721f3868a1762c8a698346ac45
93178 F20101111_AABFBX allen_m_Page_092.jpg
d9ad9733d44f24d5e1bbccdcd9f1a143
2b7b7f503af4727e9dd5e92acc2f4558ced44229
6775 F20101111_AABGEZ allen_m_Page_036thm.jpg
1cb962a3ab0bb3d8098d63b38f77680d
35dd4bb7459011bb2b365a512864aee390783ae7
3314 F20101111_AABFZT allen_m_Page_079thm.jpg
6ba5a3347d909feb9e598ebfe97d5013
92708b4ca02d6b4134fb48a88a2789f7d46dca18
6087 F20101111_AABFDA allen_m_Page_064thm.jpg
667f7efeceac0d77dc67ced994836f82
725d89b8c1b8978646367d083b1eef5f695183bd
23661 F20101111_AABFCL allen_m_Page_101.QC.jpg
63b90729ed59162e665889a97a58aedf
b66adab43b18420d5cdd50116808dbe1d5f40f39
6266 F20101111_AABGGD allen_m_Page_083thm.jpg
5ff0664598aafdca742e77dcaa88bae0
d1b7f214689ea1fcec0eda407b0b7332ad24b218
26947 F20101111_AABGFO allen_m_Page_063.QC.jpg
54e110af21c445343e62a5c9727ee10d
82cb11967a10f4fa91c6ccea1350cf83f51c68b0
2544 F20101111_AABFBY allen_m_Page_008.txt
54f14a4b03e81a032cbfed0f73ce7dbe
ed6fcbff088c0097013c023f0c6d8237fdcfae30
6728 F20101111_AABFZU allen_m_Page_091thm.jpg
7185f8df979b024d9b4e73a88cb0a9ae
d3503365742ee576c5c939128b120293cf9bf2b5
57123 F20101111_AABFDB allen_m_Page_059.pro
4eafe1128acd2f161f99e7ef0d8bcc00
a28810e49d8228f8279dbbb800ff02c799e8b10e
443912 F20101111_AABFCM allen_m_Page_065.jp2
b44ee2de3ad49d281885097f4f444b61
29d1e85b61449c601b3acc238ffdf311e8e85859
6569 F20101111_AABGGE allen_m_Page_090thm.jpg
79730b1a13e922ce8d1415471786d36e
6573c9b0de43990023dfad2716f9dc94df919412
6497 F20101111_AABGFP allen_m_Page_063thm.jpg
71f47b0d173660b0ea2ae38c38fa309a
410553c75a341c0ecf863823de9820ce54ef4acd
26874 F20101111_AABFBZ allen_m_Page_015.QC.jpg
db54e448e3bd4e4bba90954e6f3169f4
7e6e06490a3e918049707be3d623783b53f6a6ac
3179 F20101111_AABFZV allen_m_Page_003.QC.jpg
5f122017cec9adbf23d702a42a2c851d
d1943ed6b5e9f6cb2d5aa6e9200a2305b2825d03
26995 F20101111_AABFDC allen_m_Page_085.QC.jpg
517ea69292be1f2ba9316a69e332e743
10cfca0e74f2bc33cc2ba0a15bdd83f3607eefce
28084 F20101111_AABFCN allen_m_Page_112.QC.jpg
16f762eadb64a65a4c447613cdeab99f
7dc8b5bd47a208ecd592e403f0b8f02dd1fb6c94
24341 F20101111_AABGFQ allen_m_Page_064.QC.jpg
df837cdfc6f38d499a790c0a37360bf7
4047afdfae542d242214a38f98cdbcf98a8feb82
6788 F20101111_AABFZW allen_m_Page_092thm.jpg
b9331979c6a76a5eb3eab77ba0794cff
19c5a9120f0ebebf90352682b62e0fc1d2a51323
90082 F20101111_AABFCO allen_m_Page_031.jpg
cd3e5986cde95b28c125489ecc5558c4
c8ccbd1ae376b97bcda436e91e7f31a8e16faea3
29095 F20101111_AABGGF allen_m_Page_092.QC.jpg
794a3e391184ff94ed832a2f4e09237e
1a751f2666bda36addb3f07636c554da6543d2fb
2956 F20101111_AABGFR allen_m_Page_065thm.jpg
26185d63822bd295fbad7a25a463d723
acb7a2bc71c13002d550ce49ec38e5145cf8b24a
6645 F20101111_AABFZX allen_m_Page_029thm.jpg
4f50f51bdbca207d9b02bfe3f83289b8
b3c385868bc90d04a45551590976c418d9487d6a
83542 F20101111_AABFDD allen_m_Page_047.jpg
423a78721548080feb69013f4162d0a6
e5d0f5af0c7b40cc5ac37560ff9f5569331fb072
1968 F20101111_AABFCP allen_m_Page_064.txt
5a5d34c1752ea0c205554c92e1b5508f
48c7cd79d05f6fef5dd7763846ab1b9ea9937cba
28265 F20101111_AABGGG allen_m_Page_094.QC.jpg
c20ee07c9b5aed83eec1d3aeb7c075f3
5d4aa04c3be1f54d07f34ce58450a6b1b35d5155
8854 F20101111_AABGFS allen_m_Page_066.QC.jpg
2ff773e8079782c45d2bb9d64c6220ee
2d6a07b2958d9ad7d304866bdd7bf6a43392f5a2
3194 F20101111_AABFZY allen_m_Page_100thm.jpg
7d5b48b201f4632926aef0dce14335d2
815be9933d818351be428100e74dfeef59b5e72c
2162 F20101111_AABFDE allen_m_Page_028.txt
ff513e6d2faf78180cfd51c2ee588647
a28a289a97e2db4045b2ae274cda47aa0c86332c
24021 F20101111_AABFCQ allen_m_Page_002.jp2
86628150f0de4ac08ad447f6ce519493
b08cb028865123bfffca1719e6039854e5c9f86f
26875 F20101111_AABGGH allen_m_Page_097.QC.jpg
c9e5ae464e78a680cc4e1b7f00d5bcfc
59d0559d96f75878ace9845ef72304d4a8687a38
2984 F20101111_AABGFT allen_m_Page_066thm.jpg
6b6cd8466cb2831268e710b59e59c581
c44b4920840600d09b68712aac4243136b1ab900
82181 F20101111_AABFDF allen_m_Page_025.jpg
c321fb2e99a431d7dab96f8e7b7c09e4
827982ba2a793106a657935150622f916ec822d6
88803 F20101111_AABFCR allen_m_Page_013.jpg
105ae454cd379cd9d0014f829004a342
e1df8c9047092bab9d982e89523c99d93c480184
6582 F20101111_AABGGI allen_m_Page_097thm.jpg
fefd453264f54815ff8f99759e6e7fa1
290a4a24d6679cf4820ffe25e54dde0fee55eb63
6388 F20101111_AABGFU allen_m_Page_069thm.jpg
ef33ebaf92bf1816b9a343f6adf3113f
8e2ddf200eea24f05857da92b98bbdccd7550c1e
6655 F20101111_AABFZZ allen_m_Page_074thm.jpg
dfe9fb678aabe1eaa5d45cacb83f296e
669192b0157ab813ce84f960cfe53d31b01c1616
57086 F20101111_AABFDG allen_m_Page_092.pro
8a8246428dd3ed2cc24035c4128007a4
1943c2f6ed14a26c68e983e6d5f6c712d14cd7a7
5747 F20101111_AABFCS allen_m_Page_058thm.jpg
7312944937e246d9edace3204065c9e0
cf989b475494b0e378692d84789815b5e009856e
8717 F20101111_AABGGJ allen_m_Page_098.QC.jpg
446268c4842f524652471a2f5d7fad40
0a66897f5a61341e8bb02109bf322001fb4d7642
6648 F20101111_AABGFV allen_m_Page_071thm.jpg
f15a8435f35f5696165aa2d0122d0203
1e3a2f83ef16580125b280bda4b4ce58e14a8a11
F20101111_AABFDH allen_m_Page_074.tif
86ed376e8178f3148b063b0bce230e2d
59fd74f034e48af0366419f868d111f1ebebd3a9
92104 F20101111_AABFCT allen_m_Page_032.jpg
a632a2e26d59cb3fcbed3add02d939c4
70d0d2eeed1e86249abaf74a9f1a9a725a371756
5554 F20101111_AABGGK allen_m_Page_101thm.jpg
e1606ae575df625d68b3e0994a923c0b
b00eb61704286af1773d319a021d7c951bb8b931
28550 F20101111_AABGFW allen_m_Page_072.QC.jpg
393d92e571c6db5a6460e75ba00202fd
a99a9280cda8e2270b060c5cacc1c65b22ed9de2
63877 F20101111_AABFDI allen_m_Page_099.jpg
b09eecb1b135d4ea066d8bad2f5b3bb7
30f027c3856cc83874f93b3d9264c2908f311cc4
90524 F20101111_AABFCU allen_m_Page_087.jpg
0181afb4604bbf4d7c3848a0241fe2a0
096b119ba0e354f30b55900eb8a4c9232f67cc96
4181 F20101111_AABGGL allen_m_Page_102thm.jpg
8c4233f4480527071dfa0decb96a6744
cc69a6556f473f51e0882fd23a699dfb91b458b7
27677 F20101111_AABGFX allen_m_Page_073.QC.jpg
18cece76a130e6f0c2791465f4a492df
e0a71994552b56e4910ee2b71fb8839784fabc43
27194 F20101111_AABFDJ allen_m_Page_029.QC.jpg
b99988f36306224fc55b36855825d044
a0187e69c86daf817dfd6449b45745f9f67eb3d5
F20101111_AABFCV allen_m_Page_006.tif
dac566d3784e231d8de407b3a58a7073
34798dfd0e7ed2674438e4cc2f25cc822db0d86b
2353 F20101111_AABGGM allen_m_Page_104thm.jpg
a733683622c1a6b7b19fc2813e16e079
972db94190b7749f920fde177fb1385f71d09b75
3815 F20101111_AABGFY allen_m_Page_076thm.jpg
970fe22387bc5532987faff14a7d4adf
514fadb5872284d5c87c62455ee9148435f4a107
52422 F20101111_AABFDK allen_m_Page_084.pro
cdc7867af3bdae25372d880c34bc09e5
a6683ada352d14c0ec746537570416dd8dd564e3
192521 F20101111_AABFCW allen_m_Page_117.jp2
a0360d3ec5de18b784489936b3999099
423fd91c00ee38741017f5ed6882526056fffbdf
3627 F20101111_AABGGN allen_m_Page_105thm.jpg
2e1e992f357d6215b8d6d373683b7ff0
e4056a200a75a6d55ce240142bbe7d0a31407df2
9365 F20101111_AABGFZ allen_m_Page_077.QC.jpg
ab7459978df262be8676ce7366d1f291
580a81b2059ac2dcb1ec94574f58c0da700d373a
19435 F20101111_AABFEA allen_m_Page_108.QC.jpg
3cf761f9fb006c3973a8383773edc745
e8645be1a2ef0b55ba0a6d3a9b7d89c4953ee688
F20101111_AABFDL allen_m_Page_051.tif
0fa4b75f855831241e07cb7c12683b87
3aed13e6a692d639a5756c58225bc030c3403a51
739575 F20101111_AABFCX allen_m.pdf
82a673023db179d31e50a3c4505ea7d3
bb8c9759f40c8575d3976ede1c18be26b4b36bee
2532 F20101111_AABGGO allen_m_Page_106thm.jpg
4d23345c945bb4f953b1ae952b29acf1
7ec122828c4f6230f6036c323995a808a16b347a
F20101111_AABFEB allen_m_Page_081.tif
a90f8f176067af7b2dadc3515eb5dfa5
72033e2a50b2de345e6356b34deec39759e98491
2231 F20101111_AABFDM allen_m_Page_034.txt
931102608b7a40cc767fb1c763667d9f
12626db34e6746626eb2631f0561ca666f1657cb
26071 F20101111_AABFCY allen_m_Page_047.QC.jpg
106ec1b5cc43b5b3b932c8addeed9f0e
a44b09d6a3d2ef9573473eda3fed346b1d869bcc
21976 F20101111_AABGGP allen_m_Page_107.QC.jpg
57d5b34fbc13ca71216fb353f94791e6
149b3c90806fa5b874331f234637c913a029b03d
F20101111_AABFEC allen_m_Page_047.tif
53422b73ed47ab37716289e90730bf45
163a6fd8fa61984bdf3d185c1df28bd09289660c
26148 F20101111_AABFDN allen_m_Page_067.QC.jpg
b00e50e36a31f0cf5eaeb9d8449ade5d
356de401583c43018d380724efa5c074e0f12162
55955 F20101111_AABFCZ allen_m_Page_037.pro
1169da3d7b0228e8644347b44977ca90
a62f72858f554bd17e88b67bbce05067d56ecdaf
4605 F20101111_AABGGQ allen_m_Page_108thm.jpg
bc3023369f6623eeeb6297a43861ce07
887ef4b84966ce3291f7ac0e068fe5d69e20cb82
409986 F20101111_AABFED allen_m_Page_106.jp2
e44a61e9845cc7888ba9fd23b2de8430
423025dfa16eee7c7a92d96bc2222616f98ba883
53612 F20101111_AABFDO allen_m_Page_068.pro
b859bf214c56afa1e2dfcb94cbd3fe84
259c726085cce74eaea1df0958960195caf7fd51
2718 F20101111_AABGGR allen_m_Page_109thm.jpg
e137f33e724ad091fcd48aa4dd6c8972
0a397714aa18dc5060e98fc7ef7a0a6d328c96cb
F20101111_AABFDP allen_m_Page_054.tif
a1903cbb6bf61175ff85f3dbd604c6a0
d77e7f16aa5a56ada6d2d5d6edec8042ceccec07
2898 F20101111_AABGGS allen_m_Page_111thm.jpg
b74327729fb10e32575c49e6bf846fd1
beba4801e9af55c4fc1989a99e79e05fa8fc982f
6457 F20101111_AABFEE allen_m_Page_015thm.jpg
b2fb81254ad8c5a9664df85f21a914f2
7a0a462387e50444b60303d12f8a797990ca5bc9
90461 F20101111_AABFDQ allen_m_Page_014.jpg
356da73696137235cd647365832e78cb
6d697680e47a1335195154999240a6ca5c64e7dd
7025 F20101111_AABGGT allen_m_Page_113thm.jpg
7b4ce333566d63c9d878a386cf5de0fc
5d111fd8c5fa67f17fcfe3948a51df364e691f03
27414 F20101111_AABFEF allen_m_Page_039.QC.jpg
bcfb9e3924c731e490a63ba2e65e7939
347ba8a6fb89edd6ed1094a630fec2cd6ceff307
29117 F20101111_AABFDR allen_m_Page_023.QC.jpg
dd83e005ef5b534d8dac5a36e42a415e
6ce67fea2f7ebec270a953f7ebd98d95f3bc99f3
7111 F20101111_AABGGU allen_m_Page_115thm.jpg
229fb63df36d30c4cc76b2e8cee836d0
5a0a6448a28bdfb109b2ee4cac6cc34607816c84
6585 F20101111_AABFEG allen_m_Page_088thm.jpg
28650cda3764e203c37770791eaafb69
dd2c3da19b31233f4eeb7e60acb84c301d827c56
519146 F20101111_AABFDS allen_m_Page_109.jp2
df2092c76e2f88238d9f13b6e6015b3b
c94534ee679f29caef55fab6009d185050d8a190
6836 F20101111_AABGGV allen_m_Page_116thm.jpg
313952281fb106fef9f3710c45be919e
59a5020350df3ddacb227cb070cd3b20e124c1c5
27918 F20101111_AABFEH allen_m_Page_091.QC.jpg
db13b632b410dd20c626c43e511b2c97
187dda2b1340a65ce6a9588cbb73e90234cd83db
1045612 F20101111_AABFDT allen_m_Page_011.jp2
c3fbc4fb9e9f0deff149a9a508d7a999
b7a9f306919b92cb949a0cccacae6a851b8e668f
1225 F20101111_AABGGW allen_m_Page_117thm.jpg
cac7b6f3bb385ff975bab48bfc92bfc6
357f1ded7119ce58f4ff77a145187d9581e28be5
2271 F20101111_AABFEI allen_m_Page_048.txt
b0152534fdfdefbc8cbaad1d717929b5
14c00c355c9215d7b1142c658488885c487e72b7
89446 F20101111_AABFDU allen_m_Page_016.jpg
0b124b003de7d7aa642fa235217f7c80
549c15eb29cfea717fdd18f5302c07e5ecd0d813
F20101111_AABFEJ allen_m_Page_107.tif
ffff17e305509d6eb34ee458a54396a2
aee446d69cf11f7411dcf45c7c7dd847398df43a
55499 F20101111_AABFDV allen_m_Page_094.pro
5a5a5d9309201445126da8ebd4fd61ac
88e44264f446f806d2cf1d5e8cd95cefd8ad9a9a
2161 F20101111_AABFEK allen_m_Page_013.txt
97519dc2d104dab4abcd05ad961bacd5
f8c5b352ad218cbeb043df92ced784ec40259885
F20101111_AABFDW allen_m_Page_015.tif
8665c8ed80ad23dc44c2c0079d1de994
e7a69bd6716793b68044e7a2d02ff67d4be2b98d
2213 F20101111_AABFEL allen_m_Page_053.txt
d241e2bf3192ff20c07c97d9fdc38d3f
fe95d137a210eb06d7dd84692696b695644272a1
12913 F20101111_AABFDX allen_m_Page_080.QC.jpg
ca9f5bbdd18f09e51f400d735740b0a5
224860bbda5512eb9c6451f74fef1e86d4db2319
94174 F20101111_AABFFA allen_m_Page_006.jpg
864005dd89dfdf2888aa0bda6a3cd42b
dd49729c93a53b918840368a2560484171f99cee
F20101111_AABFEM allen_m_Page_057thm.jpg
12d9feff7546a872a8a3fb051f7521ca
446f2e4089c338d64e3249759b8f0f64f4d4d06c
3473 F20101111_AABFDY allen_m_Page_110thm.jpg
6e9d96b8db01f4a3aa0008fcb197d79a
ce02517e4f9df6e1900ba35a2454a8cf205b8f1c
82671 F20101111_AABFFB allen_m_Page_007.jpg
11bebcb2899619978dd21138b7b8f290
d1340724e0730b98be757adacab15a0475cff597
2863 F20101111_AABFEN allen_m_Page_056thm.jpg
eecb5c58394a25086d20092bbcd00c21
c77b428a8534778a0d93337b0ce9b5a20efa6c5b
8090 F20101111_AABFDZ allen_m_Page_026.QC.jpg
8d9cce1154df110a250dc0b246d2165b
c2f7f784e16a1468a90572bddfcecdac847c4f28
70423 F20101111_AABFFC allen_m_Page_008.jpg
ef214dda3384bde3ae228244f1b9f9d2
df187a053af6b56a90f510dee9b879ac20553440
40556 F20101111_AABFEO allen_m_Page_118.pro
e546d34d9a0bc6e941ff18578edc611d
76db6b18001a301704f2d41719ac02f3e9864336
52026 F20101111_AABFFD allen_m_Page_010.jpg
07c8336c6e7149ce00f9bca812c3bf8a
c93ed55c0c9bf83f4385b0924619282d43d7ff82
F20101111_AABFEP allen_m_Page_116.tif
ac24b2c786462ba3e5146c725b49b2f0
97cf501d30a91923c20d3be409f4725b98969895
79905 F20101111_AABFFE allen_m_Page_011.jpg
b101e74a31b7e50fb748539977427b0a
50cf09ff59e069cbce65a3e0bee5d8a048ffe694
F20101111_AABFEQ allen_m_Page_032.tif
d59a68172b01f9d34ae781c74b61e210
060cb586e7a29e7e59d0a3fbaf48d3eb5ab41bdf
1051973 F20101111_AABFER allen_m_Page_019.jp2
1153bd739d2a72d32069d32ec32483fe
828dde723559106237d5dca57b490abe6db63126
26929 F20101111_AABFFF allen_m_Page_012.jpg
54a82d20334a7b8eb14426cc7057bba4
dd6dd49c4b9e91bd69fcdecd8df256d7d4bd24ab
136518 F20101111_AABFES UFE0021969_00001.mets FULL
c964981d8841b045ec22c22ad579333b
2344b11c2e834e34f22c4fffd896c207298af9d4
89089 F20101111_AABFFG allen_m_Page_017.jpg
178cab51ac8be79dc1d5f13bd417989e
bc939e33bfe86b5c6c8a27af387611beb7e24760
92964 F20101111_AABFFH allen_m_Page_018.jpg
48f7ab234b3ce0b162c64f7bdbc62182
e4e65c8da97536bd165f6c3fbf1eecddfb496942
88385 F20101111_AABFFI allen_m_Page_019.jpg
6bb759ae2e640e752f228ed3eef4c43a
15d9ff2b9b4fe1427cc6eb0c90d5caeb504c490d
23437 F20101111_AABFEV allen_m_Page_001.jpg
65fb8b4769734589441c6b0695715190
1ad41251e4da8a9fa3fdee18c644ee0d2c4d4007
86113 F20101111_AABFFJ allen_m_Page_020.jpg
ab17179ddb93e29edcdf5598c55e8670
09762f9446c690dbf9fd0347f80b9ba4e0d66312
3522 F20101111_AABFEW allen_m_Page_002.jpg
3510e5edd9da95635f2ebfe0bb718146
6ea6aae238e8bab20b1d86581d7e58325f1f18bb
90634 F20101111_AABFFK allen_m_Page_021.jpg
e1b817efaa15b9bfce7b7969269ed56d
73cc8e1d16ed9f3c04964f3521fb20e2b79cd654
11743 F20101111_AABFEX allen_m_Page_003.jpg
d6eba410f24826deb97c56e21685eca1
d2cbed85d8fdfeab7dfcff93e454e7871b64f340
85129 F20101111_AABFGA allen_m_Page_041.jpg
52bba153e2726f7ad049804ba575bdf8
6bbfb15a9b9cf17789b0f540fa6d099b660bda83
95243 F20101111_AABFFL allen_m_Page_023.jpg
0cb83041567a4f8c2247e73afe7d5f2d
4d33efdf8cd42d68b62c72a3e6710a94b51a8394
88280 F20101111_AABFEY allen_m_Page_004.jpg
889a2ab0fcdbce32a69965f0240c1c11
f1aa2e09aa9307a9cb3d4006d1e0a906c2341592
88051 F20101111_AABFGB allen_m_Page_042.jpg
12c0af0c13676778e4725dcf369543d9
d223942d515c9fc11b73eb9cf70f8cab77e0dbd3
84647 F20101111_AABFFM allen_m_Page_024.jpg
687f42ef197ce46651f8a8bb32ed23c1
bf70fd7d68db8d70022f9fa45961233cedd0e3d7
19067 F20101111_AABFEZ allen_m_Page_005.jpg
1826a9a250f7ed89797a33e922612826
160cece3b9dafea33b7c54637380d9456bfae2ea
91279 F20101111_AABFGC allen_m_Page_043.jpg
4c078d7e9017a192b9cac69a3d37ce08
a58ee478f2562e7c1f55887ef03d0077994dad39
25076 F20101111_AABFFN allen_m_Page_026.jpg
5fe72587c0702a92f9889fed34c4ccfe
7dbfaf41901dee517afbac378f37243faf24eef4
93874 F20101111_AABFGD allen_m_Page_044.jpg
f3582cb47633520ceeb3bc5fc45cf781
659761781251d991f5e406604328aa0379051cca
83964 F20101111_AABFFO allen_m_Page_027.jpg
e06d7f3e7e7ef9f2b9f87d56eb915683
6f7465e9508dfd31f7881b88c8e0e972a5071f27
85112 F20101111_AABFGE allen_m_Page_045.jpg
f94195f773bc2c33aaaac7f18fefe76e
7dab3059694757e0fed4b3f4447fab06a09e419a
90671 F20101111_AABFFP allen_m_Page_028.jpg
938befd778463d8842341abf3405ff17
9522a168af4ce2a8074ce58ea5b1de6dbca13ca7
88416 F20101111_AABFGF allen_m_Page_046.jpg
e099378e32caa28ebd3bbfafb7e08727
39a37eca4a99b992e3d98635bfa7a1558e7f9a48
86426 F20101111_AABFFQ allen_m_Page_029.jpg
1377c7a4738bc8bd626c85ff081ad165
e63cb0ef3914411b62be59494d6802d8c137d59a
87747 F20101111_AABFFR allen_m_Page_030.jpg
1477a20418f7ed76e74f39b70a28849f
6322fc9992f1e17c40eb1993f6da6273fef74e83
94081 F20101111_AABFGG allen_m_Page_048.jpg
291609c20711e2d6c734d8da9b5b33d6
3f8539d7550ffc18a1e53c6da25b4f18ece5735c
90116 F20101111_AABFFS allen_m_Page_033.jpg
b88febde33df0bf15a739fd0e3a82544
cf727d03f35462486b1473387bff3e3739911de8
95727 F20101111_AABFGH allen_m_Page_049.jpg
97d3a9db70f59ed3ce13c8d5ea33b7e4
db54abd68537b6d10b4915ff9c13fc54354c7e2f
92035 F20101111_AABFFT allen_m_Page_034.jpg
7344150e7b56871cde71c7c13f037490
e3c4133b2437f0313df1dffeaf6086bfd296e262
93545 F20101111_AABFGI allen_m_Page_051.jpg
9fdb6ad8b8d930f23c4857a5f9c24c75
6fcf2761ba5081b6a840981968197025f6277644
110863 F20101111_AABFFU allen_m_Page_035.jpg
ea25efec4ec4236c8e75085153b697ee
282d2e14b963afd8c93ebb265793ca12d9128ef6
90095 F20101111_AABFGJ allen_m_Page_052.jpg
f0cad9ac5773371402f3022a0beee240
187c22681398cbc587712e1b76a578164e3b838f
92992 F20101111_AABFFV allen_m_Page_036.jpg
3b8a8085a633fce67b9a14aea831f316
094fc9dd571a72438fd3676042b2b15d50a2f60b
91175 F20101111_AABFGK allen_m_Page_053.jpg
8b22a9c73d7e8405b94eee52d1ee750b
463efb1eace79ff453135017c138f4bfdc490747
93853 F20101111_AABFFW allen_m_Page_037.jpg
e2317caee35b3ca1346c23bbec7d937a
8fe9478d8821ed8e8c83b083ac65c36c72d3157d
86851 F20101111_AABFHA allen_m_Page_069.jpg
7f5d1349260fda6bd31cc9605c80b085
8ce25a181439b3a299d7cbfcf6fff1b14fb72810
84622 F20101111_AABFGL allen_m_Page_054.jpg
5a56896fbc84257807c2e5a859378070
d346a9bebc1fd2c19ffb1e312afe8c01600f11d4
95673 F20101111_AABFFX allen_m_Page_038.jpg
efd707af0baba7c90fedf85c29fc54d8
c8ce3718cdca9f5fe95ec1ea8c662497bf6cd75f
87817 F20101111_AABFHB allen_m_Page_070.jpg
620775aa70292687e0254a133f499102
a53984d0be245f7b13bd41ae53cedd8783f3fbef
85619 F20101111_AABFGM allen_m_Page_055.jpg
431ee3234735d9e22f19778cf95b4c76
e31e356f9e247f9189ef59429540d5c00d66e256
87911 F20101111_AABFFY allen_m_Page_039.jpg
918384fe751e5398cfea0d7317ff3ef8
33823cc0f1fc81aba4358b14ea5108eed68c197c
91689 F20101111_AABFHC allen_m_Page_071.jpg
dd9264039121c9bac14cbf482f142b49
2dac5d29c63c4e20ea61f7a7a7dff88fcc64ea03
39153 F20101111_AABFGN allen_m_Page_056.jpg
daae98ee5c5346205e2dd238b0bf4a70
a9e6f7d674747a7be711a5a3ae98d0287e26e6f5
90467 F20101111_AABFFZ allen_m_Page_040.jpg
8b6e7267097961baeb6ab69028c76cad
d492d8870466d6143229d6cc44fa42adab4d0c05
91042 F20101111_AABFGO allen_m_Page_057.jpg
29bc387863feade19fa2df690a635c77
411937d9cf6094c7389de44ed894de091e19108e
95046 F20101111_AABFHD allen_m_Page_072.jpg
7c66f8d18fa427256faef46ddc4d279b
34b6186d796f56f33c6ae8ff1c4bc3c939c7ae89
76669 F20101111_AABFGP allen_m_Page_058.jpg
91ab15798b12da23d6fee6f3f968503f
edc6a5b63639355b491c2272394b9c44be4ddbc0
89827 F20101111_AABFHE allen_m_Page_073.jpg
f8e7688b5e09da618e6b0f22f4587945
149127fdcac75b9f223777982781c7744df74cc2
95205 F20101111_AABFGQ allen_m_Page_059.jpg
35c89f07d42b988682399995a7502bf2
536447ad98675d7a20493b9556c17d2dcb55b599
93785 F20101111_AABFHF allen_m_Page_074.jpg
4141c8f1f893230b1965068a95f14b39
124c31a72d3fd426a54a77aa97b4dbc18255b147
81111 F20101111_AABFGR allen_m_Page_060.jpg
fef78e4a9fb216e23075ac3df75cc027
f828a4e0ff6056c93d9867f7d72a93eef0a0510e
89504 F20101111_AABFHG allen_m_Page_075.jpg
345072eae978e5123ebb3265c1c9e879
2cf55edec43bb2d810d7cca7b13fd9abd2432647
91291 F20101111_AABFGS allen_m_Page_061.jpg
ae19f08de849578d0ea2505759445317
2a7f1abb8a593602338bc5584a73398b57f48117
88791 F20101111_AABFGT allen_m_Page_062.jpg
bdc531ba2ec54c20d10c4addc9369d68
94f1807473e659b9bcd655d152d476d4d233f7be
50597 F20101111_AABFHH allen_m_Page_076.jpg
a5840c1ce96e6577a9970393f3b3f5e4
ff56ed1c26fdc63cc2dafe8f34b6a3767eb75a0c
87314 F20101111_AABFGU allen_m_Page_063.jpg
f55f8d0e7a8ba5f853e8b1c08d3d8936
571656b4000b2106c9e7c6b14ad420edeac258c2
30439 F20101111_AABFHI allen_m_Page_077.jpg
9d7761a802adaf11638665f56d6a7ef6
cebcba9c6e5ef880a379870e4771b62c2ed820c2
80919 F20101111_AABFGV allen_m_Page_064.jpg
88d9ef2f348fc236edaa33d4b5a70275
80f8df93da2acefa0a1b48265ddd9f4b7a956245
28135 F20101111_AABFHJ allen_m_Page_078.jpg
c991197c4cc65fa3b082ca7d5effbc50
f36c4322601da98dc6027e49ef85dacb87e672aa
34709 F20101111_AABFGW allen_m_Page_065.jpg
161d059972c2855752256372c16e3174
0c3eb534e87ae2f9e93c012494d28b6e1759bfa6
35494 F20101111_AABFHK allen_m_Page_079.jpg
a305d8fe76d813f5c991f1ac34c4d1f9
45ab6379f8813596d8e04906d45bc07a854ca704
26799 F20101111_AABFGX allen_m_Page_066.jpg
5bc2d1ae155b6dc940b53008725cca64
0f90d2fc567f5be11ec1e6a0940e294305c61e2d
40102 F20101111_AABFIA allen_m_Page_100.jpg
7706fc5823643ce969b1358b93d34d96
3eb5f91158a530ae2fddcf5786310707ade975cd
39455 F20101111_AABFHL allen_m_Page_080.jpg
c43c7f3bfaec182b259c8949b6161596
982bf8458f6fd61de250590e38e8a1f7e67c8921
84005 F20101111_AABFGY allen_m_Page_067.jpg
938c642d64b3a79719f8515191ce2d47
e244cffd5a3e27659fdcc07539aa9488f45c67a6
94191 F20101111_AABFIB allen_m_Page_101.jpg
c81fc6d2a8778f7b67a731a1b11fd748
ca577f71c0797bd33516c7edbcdd012293bc505a
58424 F20101111_AABFHM allen_m_Page_081.jpg
346d5bb9888a62938e875af2a324c7b2
cf66930066f5f3e284a3425ca72d4fb6050cae1c
91437 F20101111_AABFGZ allen_m_Page_068.jpg
7a8a722b79b73f3c412dc65a9a5d1e4a
6beddd242327b9ec9c767fb0bb06322c93e229a7
66997 F20101111_AABFIC allen_m_Page_102.jpg
610deeae868893792191efdc3935695b
4f3384a3bb8e5d57c545dadfbf5295a22020e410
16095 F20101111_AABFHN allen_m_Page_082.jpg
72b2f5301b6a3ca81501f0669e4d3d0d
df45075d79acf9ed21135f3c15c502cfd5b76914
88534 F20101111_AABFID allen_m_Page_103.jpg
0f0c4b5dc7d8fa1c9695285b3ba88403
e005a347610180b5e1ad496b935eea5618ec58f9
85465 F20101111_AABFHO allen_m_Page_083.jpg
38606cb04b5fb8349937ea310d84de48
f9dce33dd6d7060584d5c33b88bc5250d062c688
30892 F20101111_AABFIE allen_m_Page_104.jpg
fe915f9dec5379d4450011d20f671493
808538fa28adb2b26f85d2876148f371b3682ebb
87151 F20101111_AABFHP allen_m_Page_084.jpg
b8aac9ecadb330de1f87dd76e9214193
24724207c8079be30f954f073bb1835e24e4cb8c
45564 F20101111_AABFIF allen_m_Page_105.jpg
b53909fb7f73c310bc41cdf1912ca1f7
e44ec8f2f744d860826d4bb5ecdfd640d686c78a
90449 F20101111_AABFHQ allen_m_Page_085.jpg
8fe9d9f07941389cb8af1082d05e7493
a3c74e4495e02fc9e6d80b7580ee6e606539ee8c
33754 F20101111_AABFIG allen_m_Page_106.jpg
b27870ae1fba17b93670b18da8216520
3e6a6d493b21eb77aa97454d98344cd2f50ef304
86209 F20101111_AABFHR allen_m_Page_086.jpg
6d2b9ab72d7ba52bd0dd9d8f81499bcc
a36df8c5eedb8390697cb85bb63411a464befa68
76422 F20101111_AABFIH allen_m_Page_107.jpg
fcc1f2cf0bacf05e46e9082baa17d63d
8e694baa45b29077b5249ea9e155deacd32a6cd8
87387 F20101111_AABFHS allen_m_Page_088.jpg
66be230ba33312b9a25a596f67b2bcc6
f19e32522fe5f1053684b6274451d2740e6e2a1a
88871 F20101111_AABFHT allen_m_Page_091.jpg
9f2243b65187605f889ce932f5f357bc
d4dad3f98e9d4a9878b3f2f8c18b53daa02a341c
70484 F20101111_AABFII allen_m_Page_108.jpg
13795fafe11596796c407db5c8542c07
53a3634548ca415f0824a8a6af3f49bdc3a3aaa9
90687 F20101111_AABFHU allen_m_Page_093.jpg
89034b805bf4b9fc527b9b809eca68c9
8f1df4ffaad1b051ad4f18ed68f46d146b34eb0d
40631 F20101111_AABFIJ allen_m_Page_109.jpg
3719c98afa242c8bbbfe233939efa4c7
31abed58ae68f9d13432c344a4cbe25cdb62805e
90259 F20101111_AABFHV allen_m_Page_094.jpg
1c86753512bee5e0fb610e6bd888b1c6
416fc7fd24b893e729c82eddcac75e3f80c45f63
53192 F20101111_AABFIK allen_m_Page_110.jpg
4c8b113c4a1545dd056122f49ca3d299
f0a7f09b65bc39b8fc0f1a6a069d86e49ac871d2
93905 F20101111_AABFHW allen_m_Page_095.jpg
e745b85adf1365e27f76a787fff20820
9c0566cb244761a1ee77335eeaa060c7a0c7eef7
677401 F20101111_AABFJA allen_m_Page_010.jp2
d8ef4e4ce101dc269118fe333c0f2afe
04af0c60ac8d6982337dd20223698438005913e0
38967 F20101111_AABFIL allen_m_Page_111.jpg
64e74dfb9887348e345db2f53497360c
906449dc7061f73eabe054869a28e5dfe75b66fe
115631 F20101111_AABFHX allen_m_Page_096.jpg
19fb12284c28aad5becab8e8cf426187
cca716458f3c83653ba2a7e4d16f9366b19438a6
334192 F20101111_AABFJB allen_m_Page_012.jp2
f381b1928457d9539ec488632fad1173
7f5f7c7cf20a523501dfbb3df385731f0a55983a
100909 F20101111_AABFIM allen_m_Page_112.jpg
bbdd8246737be42aa0bdccdc2ea9a1d8
8578d129d425343cc5533d5b6635b5237a768b22
86730 F20101111_AABFHY allen_m_Page_097.jpg
12926daad0e92c3ad71952dd203b7fa8
6006f0d912abedc7fd9c8e6d9f662ca14185bec4
1051981 F20101111_AABFJC allen_m_Page_013.jp2
ba42e55b02d345a32b1c6b5d01b3105b
07a3478125d0f44d9a920a1b212f9ac6134d4a0d
104443 F20101111_AABFIN allen_m_Page_113.jpg
d2320cbdbbd925c9a6fc7cc453888a3b
41497a9d31d294bf89b0438d7f38276ec25fa86c
29612 F20101111_AABFHZ allen_m_Page_098.jpg
8c40ef84493265900d63eb17e850853a
74de8b6be03fc7fb573153d7c1d243ddfd187873
1051945 F20101111_AABFJD allen_m_Page_014.jp2
580f4359cd68860acfc31884c607ecb8
ea33406418c26a4aa84fbfce596998eb8012084a
96841 F20101111_AABFIO allen_m_Page_114.jpg
29ee3623eef2aa6ef1d0e31788d64bc5
8af61b4ef72a62e76cde5c34b6d12cc184f4a277
1051968 F20101111_AABFJE allen_m_Page_016.jp2
54c7ea1891a1f424937ba389bb593cdf
0b46b37945f7cf0ac11785a935fbc4a0ee2d2e6e
111101 F20101111_AABFIP allen_m_Page_115.jpg
19cd17a8a31ecbdb358783a56f1d9edd
27b6086f164412fc4aab30693904d69b502eae5f
1051950 F20101111_AABFJF allen_m_Page_017.jp2
ee77746c479c2df70e010dfd3891223b
e9367e97e3eed8ba717b3b8ff01bced199aaf8e8
15800 F20101111_AABFIQ allen_m_Page_117.jpg
9c333d74fb2d1c9d150930fa761c1142
028be3330f4ffda6eed63a1b936c089b174fb211
F20101111_AABFJG allen_m_Page_018.jp2
e931435bc6463660ab6df0e5b8bd0684
8e31e8d2d4e146b2e9db7f619a7fd8a6435584ba
70382 F20101111_AABFIR allen_m_Page_118.jpg
bd5cf8e6784bec2129896424378fb48b
40246eabb3cfc5c6a57e09adcd3e398f4f869e1d
1051958 F20101111_AABFJH allen_m_Page_020.jp2
9d4b5b172aa7f217751d712e329fee21
6d19bbf048e52ded6eef765995986bc8ba3fc43a
260339 F20101111_AABFIS allen_m_Page_001.jp2
633f2f653537dff82ee1f963e70eb42d
595d5cf4066558ead987542e6e99f942d1ca2d67
1051916 F20101111_AABFJI allen_m_Page_021.jp2
594f2ae1d996af85307b3ac7fa9f6992
7afbb0191bc155107303dad4a4376437a6b35903
130303 F20101111_AABFIT allen_m_Page_003.jp2
19b42c3b02c9dde24f73232fe38668c6
d570368fb1e69a50f50e8f0865de037c7661900e
1051972 F20101111_AABFIU allen_m_Page_004.jp2
41ea3c6e39301052510b916187bf7722
6fc1ff89ef7e5a818d76eedb5f07f737e1dd2b77
1051975 F20101111_AABFJJ allen_m_Page_022.jp2
7ec66eb39b69272e40f57b2c55e20367
1fe17ce73c28bcd6de1d42e1e104540c350047ac
221884 F20101111_AABFIV allen_m_Page_005.jp2
4b72d9b4d58d619e335dfc637997d8f7
f77f151e1db8ceca94752e824083143c473d227d
1051953 F20101111_AABFJK allen_m_Page_023.jp2
a9866617069c63e36d64fc4d6bc2f624
8c74f656b74e4719a4b26a4d96981a0f8ea6c757
1051939 F20101111_AABFIW allen_m_Page_006.jp2
140c2cf23d329d9614cdda70f7dcea27
b2ee5dd0c80fb2c5165a5854ab40601b0dfdfb7e
1051982 F20101111_AABFJL allen_m_Page_024.jp2
2ee36b05e1a2f0bcfd332fd6188430d2
70151298e92b1c1b655341385d6d9a78321f002d
1051985 F20101111_AABFIX allen_m_Page_007.jp2
702f8087aecbbbb5fb8d54dfc2baf643
b21685d34268392ecc9404095289d7171c14af46
F20101111_AABFKA allen_m_Page_039.jp2
0444d085f1214c075c006d49ad1b2d4a
8104cb156f1516f2278c7136adb8d898372f9f09
1051978 F20101111_AABFJM allen_m_Page_025.jp2
b573018d0be8d7cf2c71fca16cb5b163
471966c769df3e3982ff9909cd7e3c7fc036b161
1051977 F20101111_AABFIY allen_m_Page_008.jp2
5f0c51991f033d2aadb3db9e74bc1bb4
dd2db150072184d0f71855942676e64e79574ed1
1051955 F20101111_AABFKB allen_m_Page_040.jp2
9df4917999dcbb048abd3b4276e2c7c9
a0c1e6c026a7e3ddf03cb86454b8541a8c9634e9
310823 F20101111_AABFJN allen_m_Page_026.jp2
2b7e94e85b6590aee62b7b8a68dbcc92
df62c842fba7bfdca04108f5a23dd0c284271588
1051957 F20101111_AABFIZ allen_m_Page_009.jp2
5f3d3d0adb80171a7554de977344a972
f7a697d76b717d6e45b69b55c73cb37f925c4348
1051881 F20101111_AABFKC allen_m_Page_041.jp2
e389761ba6cd7d12bccddaafe6366b35
2277738f5aa58eae1810e6980a65b954f70b22d7
F20101111_AABFJO allen_m_Page_027.jp2
cdc1be110a3f78e778afe8cb33b4ee45
4497344038d02126811995e5022a0b48f09db9d2
1051947 F20101111_AABFKD allen_m_Page_042.jp2
e887b919a147b68656fc94ce8983f9ed
8945a256ed4900972978e0a0b2ecb9346dca91e3
F20101111_AABFJP allen_m_Page_028.jp2
c2e50b0320c205ea0763f99748ed3223
f65854f5d512ae47567a1a38ef0aacefd591207f
F20101111_AABFKE allen_m_Page_043.jp2
f5582bdde7d784b1edbd0ffe7c1fff72
6ca5546efd33837b1d1f4589886ebe424b5a31c1
1051971 F20101111_AABFJQ allen_m_Page_029.jp2
543a3607c543d980aedeeb89b114d12f
512e36464d087ad1ffe032326cf5ec005c7ed8fe
1051963 F20101111_AABFKF allen_m_Page_044.jp2
d9a4ff0fd029e4fc1ed24882ad60cdc9
4185c60e1b10a8d7758aad68b40118d0d78d0514
F20101111_AABFJR allen_m_Page_030.jp2
7570b968ba816b1018050b01cf6038e4
ccebc4f41804ed13df47179e535bfce9e3a26714
1051936 F20101111_AABFKG allen_m_Page_045.jp2
0794dca3ce9491d0dcd83aa7ce6a8bd8
93eccbf2021437f16a398d8faeaa5433c4c104f0
1051956 F20101111_AABFJS allen_m_Page_031.jp2
14143f690ef72866fa242912005009ff
3ec73e4341dc4da524a78dc92b228a9505ceef39
1051949 F20101111_AABFKH allen_m_Page_046.jp2
d9870955e7c3e73dfc8accefdd8068b5
b1bc9eff1ea6f53a81c11fc2823ae9f0d7cd52be
1051986 F20101111_AABFJT allen_m_Page_032.jp2
2f78f7562c3fb35253fa4de6dfdaf262
e53c0fa773359d8953b84be116c314c497924b13
F20101111_AABFKI allen_m_Page_047.jp2
4664906071fcbb22f04c630587e0bd5a
049eb1d212a0d95b40f082259a8cbac6e9733bcf
1051979 F20101111_AABFJU allen_m_Page_033.jp2
68ddf4cd90174127aabf2d72f3e8e05d
3c18ae2691ef21c98e9334f7d5c2a36c3aca87bb
F20101111_AABFKJ allen_m_Page_048.jp2
136b304650049b998f9debc828353abd
86849f845b854eee6090b693ee270b90e5672b29
1051937 F20101111_AABFJV allen_m_Page_034.jp2
8372d4ea91dd5259c83fd231122fef2e
a753a5c4924e0762e7588ad03e0c1048d17317c5
F20101111_AABFJW allen_m_Page_035.jp2
8f1616b1d4605ef2f2a7c817e8e7bd2c
7ab7d16bc061fda73b56a5142a67ebb34b0897ba
1051965 F20101111_AABFKK allen_m_Page_049.jp2
ccb8d32910fa68dcfe5e843f4aa7d0cb
0f4aea85b9da15a57b095ffcf948c9431002c93d
1051962 F20101111_AABFJX allen_m_Page_036.jp2
b92ff2a35a273c8779d370164a119e88
c00be1622b041a34ab1374ce9de87f71bea3f4cb
1051959 F20101111_AABFLA allen_m_Page_067.jp2
7ed0bcf840656e9cbdcf344c67040738
da4815dfe24aa1feae5ca9642ffc327fd8e5481e
F20101111_AABFKL allen_m_Page_050.jp2
16c82bbabc99cf556f8793f3b906bb64
31b9e4382c56b3e0b84503c68af5f971f7839050
F20101111_AABFJY allen_m_Page_037.jp2
29325e2f583ec270ead963b61a2ce959
7807e7830243a43c5833e1ae0bf3a656a0491450
F20101111_AABFLB allen_m_Page_068.jp2
77aa9b1c3d3659ef587ed765f2ec7ac3
a91a09e00d5fae957e0d8dabac9a21c3eb2278e5
F20101111_AABFKM allen_m_Page_051.jp2
d49060469c51d9eea5cebce2c451a8a5
de9f8dd3f374a0f8ccff9ec2ddd9d264599cc36a
F20101111_AABFJZ allen_m_Page_038.jp2
f4f55cb1e67930fc59fe110a48299da0
2906f2e23ce7d96522f4670ad5fb510138c90870
1051983 F20101111_AABFLC allen_m_Page_069.jp2
3b853c09d6f70a06067174ac838b0f60
b46ed88fa22d9203cd78b385941c1ea227067c03
F20101111_AABFKN allen_m_Page_052.jp2
fd600f10c9cbaa5a1165aa4d014ef02f
28c113c57ce897c895d0f96bec4c6dbb0d384b00
1051976 F20101111_AABFLD allen_m_Page_070.jp2
56bfac5fa280b17a77a2af2d2acdd0b1
860389e034ccba7b5933fcc066447250eb137ff0
F20101111_AABFKO allen_m_Page_053.jp2
6d12c9ac924acda99fc331b8492d70fa
a95a1b621cd6444d2b245c7ba45c9788c9725c54
1051970 F20101111_AABFLE allen_m_Page_071.jp2
b122b5afd9eb37b4b4c16042750b1451
5d6fe562768ffa738a4af7b98659519333017de2
F20101111_AABFKP allen_m_Page_054.jp2
2331ab57bea80cc0b8f5f0a60870a538
be6e68ab1c9257b1d58bb84fd239f7028a46d642
1051946 F20101111_AABFLF allen_m_Page_072.jp2
b6f915fc277a675f369c8bd65ddf313f
2b1f8d40978cbf3601b5b11608b32ddf9e844bad
F20101111_AABFKQ allen_m_Page_055.jp2
d47ea311c79321b3a87944b4754b25bd
321fb0f8de57314d1dc2cc46a334d9114e8d9ce9
1051929 F20101111_AABFLG allen_m_Page_073.jp2
70bb6dafc35fccbdb7664b6a578b5df7
2b6dcbda78f26d5800b47ed7533338220ee01b7a
495991 F20101111_AABFKR allen_m_Page_056.jp2
3716bb133b722b98cd5445c2d6659800
c952d6362ac4a17f465148a75e986ab53155c9a4
F20101111_AABFLH allen_m_Page_075.jp2
6e5e8c117f4a8243ecdafc3746e77ccf
699fe4f157ac4e79bf7f7146cca2d1aca25823fe
1045795 F20101111_AABFKS allen_m_Page_058.jp2
7c38db3da50887bfb5375eccc86fea36
5baa911aa4e5ff2b8aefd74eece7474e93eaee56
317040 F20101111_AABFLI allen_m_Page_077.jp2
18c3b848f01156949c256a89ca7abb60
c4b0d4acbccbe1ce0fd93459dfdc109b1ec3643b
F20101111_AABFKT allen_m_Page_059.jp2
39b889e76eb2a1861f14e8d3ef62af3e
f2891bffaa851cd03efaa1d1921b496ef3af7cbd
298608 F20101111_AABFLJ allen_m_Page_078.jp2
9f8e6220b16a01433550c9854543c685
e80b0448195a74e65ba2d98d6b84b4e21bbfc4c8
1051940 F20101111_AABFKU allen_m_Page_060.jp2
27ac7d6817f176fa5b9e4d522c9c6173
bae205f996f65718c8b253a426a89459e0b8a5d1
398525 F20101111_AABFLK allen_m_Page_079.jp2
737b2999c8895bf568f0d196615e2888
2fa1eca0fcb40f4198047b88530af10cb538c8c8
1051967 F20101111_AABFKV allen_m_Page_061.jp2
9a2599c7dc7f2de98d6c77eaa5138acd
65a04a754059edf1167a098de7b34c843bc3c3b9
F20101111_AABFKW allen_m_Page_062.jp2
08ecc4abae51f3a31c338f38c8b24a36
1ca1f5780090d11cb536405adb2a8e49ed2675cf
429393 F20101111_AABFLL allen_m_Page_080.jp2
4b68691e433407d69c80570656bd3056
bf9426d8193ea2d2f61394df671de9695bbfcf8f
F20101111_AABFKX allen_m_Page_063.jp2
91f652739cc8192192e433ffc39de898
11e139e4bef62c8eb3124cb3bb12b7bea0daf907
F20101111_AABFMA allen_m_Page_096.jp2
0b491ba759f13a2c5ca235986c0e9f45
69d5205a5178a8d8a4167fac372456529ed62da1
695241 F20101111_AABFLM allen_m_Page_081.jp2
5bcadd12f31c6ff8430092d62c4ff429
4c3802e163156372483b569cf6d300d081e88ddb
F20101111_AABFKY allen_m_Page_064.jp2
b1f02dd525c3ff17d16b88a03cb23582
88183f83eaa892dd1318660d4a14faf8b4e22c38
F20101111_AABFMB allen_m_Page_097.jp2
60854d060868ce38543632ceb56dba31
bf52bcfe699718848bf1efa8fdd8cbd20840575b
166460 F20101111_AABFLN allen_m_Page_082.jp2
d5217154ed95823b1b1b0c39d782c139
01501727316db69267b0d5d3d3a2a17ee86ef515
327475 F20101111_AABFKZ allen_m_Page_066.jp2
9cd4bc81806fbd751afcced20dd2a28e
146f2399470459092dfae61b4aefb9cf4497e9f6
358633 F20101111_AABFMC allen_m_Page_098.jp2
bcf4e31e488433c8a2ac3e3f8c9ed40d
36818e172d9653e3e99a54053fc9b6aae5cd93d0
F20101111_AABFLO allen_m_Page_083.jp2
b0184ebd25b53840f33e710e5e98ab15
ff463482a30a139b44e7aeed758f7c9ec787b9a1
854229 F20101111_AABFMD allen_m_Page_099.jp2
f2c15cb335a7ab95bec655769e593464
63b850b8768d6304a9f8197fa95b6036e05dd893
F20101111_AABFLP allen_m_Page_084.jp2
73cf7ebdfe3607178c77d9a1052c7aaa
a4e69a295b5236ae7c73d9920f9e3106fcc8ff2c
598143 F20101111_AABFME allen_m_Page_100.jp2
21f9fa973c71281572328fb5ba32417b
6472dc7d7100dc5b1fcba6fade02cd9c0d23daa1
F20101111_AABFLQ allen_m_Page_085.jp2
d56a6501e08cf1ca9648d136321fd6b4
c89c3068b2231b9b75569ae4844cc73b4093a25e
F20101111_AABFMF allen_m_Page_101.jp2
373c523e697de40ffeb20a32ba1a4ff3
f226d239c8f6acacaee125091d2620e26fb89f55
F20101111_AABFLR allen_m_Page_086.jp2
b219142c97a12db04d775bfff43ede1b
90c569361eb20c74079a7000904e5dee8edae96f
893711 F20101111_AABFMG allen_m_Page_102.jp2
01ab36b0da9d19e1875dbf3ff5e3b967
4f15bfd0137deef5b5a5927fc332a60da269688e
F20101111_AABFLS allen_m_Page_087.jp2
3baa7991f1dbf8fc43dab6df4a4c5877
42b83d7ae2d40bfedd4907d7744d9399f599ba84
F20101111_AABFMH allen_m_Page_103.jp2
25ecb682501303bb6e62425a32c7fa01
3b7169724f61e20ab778dcda772388f60a0b30ad
F20101111_AABFLT allen_m_Page_088.jp2
b2aa2fb861a104d1ba1c9e16e13974b7
373856a393f49053703f33f4d895d8536fd6f339
349476 F20101111_AABFMI allen_m_Page_104.jp2
5d3325c033b64b2065c0a775ea869950
351e202fcd359f0091e1aed83463f7925bf727b0
1051924 F20101111_AABFLU allen_m_Page_089.jp2
b0c4f7754c98b87f9eda2d583de0471c
dba80f0bdd1d29065af5aa9a4423b808b6d4d0cd
744197 F20101111_AABFMJ allen_m_Page_105.jp2
b2a31c72ed8287f0b60ce9d46a8de6dc
6ef366715f98df5f35d3a73a474b72dbfb7f0cd8
F20101111_AABFLV allen_m_Page_090.jp2
f58ce20ba7c7f68b100babd63190765c
4924392dd92abeeaccf860eaa6cf630064f3588a
F20101111_AABFMK allen_m_Page_107.jp2
e4fcccd51d0068f00f7d7dfa9f2630e0
e5dbac0ff96d3f5dba7400906613e58c42fb79a0
F20101111_AABFLW allen_m_Page_092.jp2
84c299c8562f9856fd395ab07380a03d
b4b93ba77741cb4d5865fdf838591af580c6c829
946433 F20101111_AABFML allen_m_Page_108.jp2
db416c9495abaa359d706f2e5c9ca9a7
61b4b22102af24f32baf006e7b30875690f6730e
1051969 F20101111_AABFLX allen_m_Page_093.jp2
428c95e1e73c58c3ce66d7ce8e335aca
e31768dd6ad900f5d851a3d183f94588431a57b0
F20101111_AABFNA allen_m_Page_008.tif
356f29dcdb88c682519b24651a13c15a
57473b84239f7fb945c68965e8e5a9439c54cb84
F20101111_AABFLY allen_m_Page_094.jp2
48f766b2e812dd7d6769cf4c5692bed1
f7b75e3d9dbe006e4ca303f653297c4d07bb5d91
F20101111_AABFNB allen_m_Page_009.tif
9e33e4a0f75ac91f36796f3063066863
c118368ea456ce4b9f959218d3d9129c94a2dba2
701550 F20101111_AABFMM allen_m_Page_110.jp2
81d115cda8a7c88fedf58b2b57d7a422
a56a3c44b20eec343c548a5c622593aa5496fe00
F20101111_AABFLZ allen_m_Page_095.jp2
d214c1c419c30808b38f85f6f672d1ab
bcfafc4e520d01faa07892a6a8618b83aa333044
F20101111_AABFNC allen_m_Page_010.tif
5d0fac1000526aa887a66fda77d02c9d
599e761e51aa5d4ebcb7b50f5ac4586f29ad32be
497982 F20101111_AABFMN allen_m_Page_111.jp2
a5ef44b17523547395bd2ff811fb2973
0c3115f6f6d579d5ae1913088820e74e7944d581
F20101111_AABFND allen_m_Page_011.tif
2c2d5c27cd148b263a570f8e182c5beb
255108cf2051ee03b36ea8bac4c2677226b092cc
1051888 F20101111_AABFMO allen_m_Page_112.jp2
d5431f6cdabcd48f37a557e3dbf8727f
a56797d780d99f977023ee5f1cf2097cc5969848
F20101111_AABFNE allen_m_Page_012.tif
2ca7ef4da68ae89a30db78629e9db93a
837fedcc0876b938cbd82b46a1fb771ed0af9969
1051984 F20101111_AABFMP allen_m_Page_113.jp2
b462f415eec3d380f722af2747c6d520
9dcc932017e8e20ba210f6510b7eb8417d7dee9f
F20101111_AABFNF allen_m_Page_013.tif
15b85ebca0c6405c7be16d7a9231b62a
05d8e2896241490f64ca2d754bcdefce0ee1b031
F20101111_AABFMQ allen_m_Page_114.jp2
77b0047ecec6228a5ece14b725be6d17
955e84598f7503471f5b3b8e90a2c779429152d0
F20101111_AABFNG allen_m_Page_014.tif
e93ecfb81a068c37fee69d86f480c8e2
faf3c391c94e482842cfcca334694e4a59bb89e6
F20101111_AABFMR allen_m_Page_115.jp2
158d90b026de1d182ffb0ddc5dfb7b07
ef7bd4dba2902c287134125b7c9eae1dda639204
F20101111_AABFNH allen_m_Page_016.tif
09813c3137fe0873a877c165c4504ccf
e549713b3a4e7d4a4928b73ba4321f063ce83b6c
F20101111_AABFMS allen_m_Page_116.jp2
a9de0a43b73d57d3e4a489944eaba888
999ae7b46e8adae423fe7a7a0d41ac098d862fa5
F20101111_AABFNI allen_m_Page_017.tif
d5f505f7daa5aaa1e87e064dba1a93a4
c9b92dee98365a0a4b3145f0bdd1ce595dbb19ea
925781 F20101111_AABFMT allen_m_Page_118.jp2
0133845486c17d2b8a174c9f11118f1e
c9ca9a7069daa087c6ae95118da28fa885545eb1
F20101111_AABFNJ allen_m_Page_018.tif
9b4d6ea324898fa0e0d371867dd22bad
058a196acf0f9625c3d0ae75b11a95e44acb4909
F20101111_AABFMU allen_m_Page_001.tif
1f57f6cad4507a38b87d4546d1f9cda4
cd5b2e0bbe6e19476fdfe92ececde01fcb8ce3bc
F20101111_AABFNK allen_m_Page_019.tif
00747baa1f474d5b17c88e948aa23640
a193a82064ebb59374fab1471003acf41cfc27ce
F20101111_AABFMV allen_m_Page_002.tif
a10351291f8fc34f41079b28324678ba
eb4c41627c02051ff8340c2c27705b9981fc9159
F20101111_AABFNL allen_m_Page_020.tif
08fea67130cee270496eb54a3f54a162
eb509167284eadd7fbbb39b3d824b6777103ef62
F20101111_AABFMW allen_m_Page_003.tif
30fcd86ae6ab41e59321758f2d26b28f
838b351d89afe03c007635638b0e09879179365d
F20101111_AABFOA allen_m_Page_038.tif
592ad2c586dc7eba86258535b5011cdf
d32026a09e07d5972f970f24c0f00316be6665f2
F20101111_AABFNM allen_m_Page_021.tif
f94d1bf47c76b2c6925f475a0f37cba3
e42eccf8e0f6bf55ebe9e0f8c6e712088c36056c
F20101111_AABFMX allen_m_Page_004.tif
c77bc188870d5f522ab94356233b4e36
bbbe82aa27ecc8f59e4106923436b8d13e4804cd
F20101111_AABFOB allen_m_Page_039.tif
e9c19ac1492e58267d145f39a924f42b
878bdf3ef072527dcab44ad56d3d98048807555f
F20101111_AABFMY allen_m_Page_005.tif
3a11a05b184033b7ad04bd5f8950f97a
911fa293fa4a3ade127aa733d4d044f42e76f62b
F20101111_AABFOC allen_m_Page_040.tif
55fe7de892cf3b1dc585641089eec89f
8be687d403a85ee36938258304cd187997be1f55
F20101111_AABFNN allen_m_Page_022.tif
3a85b4c2a36b75afce129b4c9f872d13
aca32764e219e16e50bab9c100a8f47381452362
F20101111_AABFMZ allen_m_Page_007.tif
59c09e20b5c62da19f639e213de0a866
d1bd74821658c6bdf433c70167b91d0cff1e6093
F20101111_AABFOD allen_m_Page_041.tif
202e3debb61558508ba3a47cc7410dfa
4ea3c8e366bba0b8ebba56bc6f7578b17190ee27
8423998 F20101111_AABFNO allen_m_Page_023.tif
f59fc87d42714511308eab3b1eb3747e
9a9c7cee28a45151fe5ecf5d4e12e1bd2a50e0d8
F20101111_AABFOE allen_m_Page_042.tif
d02b7e2cac914b6170bb9c1898c8d35f
c3a602f1d3436f4e561103b4a8218662594286fc
F20101111_AABFNP allen_m_Page_024.tif
7fa317d7f18833632cd351f9b6981315
58008471e7644fef08886665a749c991447a44ef
F20101111_AABFOF allen_m_Page_043.tif
a7d473d32c4d784173a480bd170804a1
7d6a4adc978325821404c0dff90ddf58cbbf0622
F20101111_AABFNQ allen_m_Page_025.tif
700b03fbf839c75886ca25b2c0ad4404
52e72f635720cae414fba6cfe31322e1b90ef134
F20101111_AABFOG allen_m_Page_044.tif
1e96e5f5a0e645c628b368c0dbd347d3
33c7ff1bce5e03510ed3d48526ab88a27faf28c3
F20101111_AABFNR allen_m_Page_026.tif
24a191af60c1ee7bb4f5d6c000e934e6
36bf0cbb50f7ce83840e46db1005f1c7311be4fa
F20101111_AABFOH allen_m_Page_045.tif
b9376258c6a49537386c4a4194900c77
1791266603c103220bfa8996407b8cf84070e993
F20101111_AABFNS allen_m_Page_027.tif
aee4de5930947e2eef4fd20523253054
0349463f0b6d7cdeeaa16f0b7e42d292aec0cef7
F20101111_AABFOI allen_m_Page_046.tif
b267f5fd7cd3ed726543b3e05fe4d6f3
bb16d0214c86c796c8d08b7c871f49def45c0321
F20101111_AABFNT allen_m_Page_028.tif
4c1c6deb558a246166733e0063159c70
74db4c13cc0e320c3cfae355b1d702777391d29d
F20101111_AABFOJ allen_m_Page_048.tif
1ca7296c61f9154e095bc622d4720dc3
3002b56891660cf15ad6b188cf366855393050ee
F20101111_AABFNU allen_m_Page_029.tif
1b79bc04fb786d8405130f496287826c
ac780d7730e89deadf905c2486ea777f9d1b2223
F20101111_AABFOK allen_m_Page_049.tif
5d48dedb82d5317bd0f14a7c0e250a60
28056158b36fd58bb9c1891c905885eda7abd098
F20101111_AABFNV allen_m_Page_031.tif
d5d0eeb13f7f29b6f72b1f8b172cd6fb
a1c0bf054de99524982caa00526002a6c0cdf7dd
F20101111_AABFOL allen_m_Page_050.tif
111c1adfc08184e345682ee3114ea328
1605f7e9e30622a66665b0f1fd97440953a8e370
F20101111_AABFNW allen_m_Page_033.tif
e46fbcdfbef7aace1c354fc5bd655706
4f847983f09fd504baa644d941e44412fa056114
F20101111_AABFOM allen_m_Page_052.tif
91c76dd5cbab0050746e3123ab864f67
dd4581d9c69b7b94ab97f007074c755e62a0d67d
F20101111_AABFNX allen_m_Page_035.tif
2b8367c1e0a1a4e337e9f409147c23e2
1318d321bfb9cdc46835058b543d74aa5e008b98
F20101111_AABFPA allen_m_Page_069.tif
04ce4635bc1eb4c1ab9334af6685ddd4
c6c1a2cfcddf052e13c1c8953921aa8c515e875c
F20101111_AABFON allen_m_Page_053.tif
21d7ce4a4afa452941d3654f323b8f04
f37d12bc3288f4884c8b29b9ca88134e037b92da
F20101111_AABFNY allen_m_Page_036.tif
cd9483e659aac92791398f4f8a309716
a56fb4af9204376e2285a6deed62451e96f0d1f5
F20101111_AABFPB allen_m_Page_070.tif
b6ad734bec48d3c92310acf7c1403068
297326a402c873776c3911d5134ae240655e3e3c
F20101111_AABFNZ allen_m_Page_037.tif
6ad0a98e5d193881f0b3aaa86c50d8c4
6bbef2d83ba87ce4133fdf1bdc9219212c96d235
F20101111_AABFPC allen_m_Page_071.tif
ec05139dc295194b34111c1b8c26a569
ddd09b3430a84145b3904f8b1c02cb30f4f82be0
F20101111_AABFOO allen_m_Page_055.tif
6cb3bed7e5ea6bfe213f24913e86f809
9f27b9fa04be7303f5f1f634cc805a026bb0e116
F20101111_AABFPD allen_m_Page_072.tif
70e008572f18bd1a37ae849df61f3eb0
cc5b72175ecdf4af14f47abb5b02c68e410a550d
F20101111_AABFOP allen_m_Page_056.tif
5a140500dd831387a66530b365bbac52
261405463b71b06155a388a1ddd6460f49bcc94b
F20101111_AABFPE allen_m_Page_073.tif
e3498fcc2b906ffbcd59bfc8deaf9989
29d4aa7659aa88b1f035da23985dcf6da24d2b30
F20101111_AABFOQ allen_m_Page_057.tif
c7665e3ae069dc8aad02d4242ceedb8b
d67f76f90d508f85094bf6c4d3c1b3b302210dff
F20101111_AABFPF allen_m_Page_075.tif
ee291329c7a411aa75fe31610e2e6e59
b3d45cc5b9cff9a8fd91cb9ca50f0d9e3d12e1f1
F20101111_AABFOR allen_m_Page_058.tif
e29df0c45e990a18237911119a48da83
e2b6c6b438e605d2e62a22d76499b8e382154eaa
F20101111_AABFPG allen_m_Page_076.tif
b42a7b1247f3e8a40b7cd647087e914f
8b3d14eeff7a343ee01e6aa7427a905a4c3cc5f8
F20101111_AABFOS allen_m_Page_059.tif
c3b73f02c4f19672f6f21f2cd7a8b075
d821b06e44d44d19a470494dd8c98edae35fc95b
F20101111_AABFPH allen_m_Page_077.tif
ac8b07244f2bd93b1bd21c76f64cdd0f
5c945e80dc282d4567abaf603cf66e633bf107bf
F20101111_AABFOT allen_m_Page_061.tif
32c421c7d613622a07668068fa670eae
34070052531d9563df774c6f7038e4c0a50702cc
F20101111_AABFPI allen_m_Page_078.tif
0d6dac7b7c995c80e823fc531dfdccfc
1d6897c60787dea8b5ce3a58e45c59de7afa1894
F20101111_AABFOU allen_m_Page_062.tif
2b63f8f9571664feeb20afd9c4e496da
2b2316890c80bf2123b117a4950b94d6f1de630a
F20101111_AABFPJ allen_m_Page_079.tif
c1466b5ba879a7a56503ee7282ac4604
56180a2d5edfac710cc7519f6f4cd08de2d7ea3a
F20101111_AABFOV allen_m_Page_063.tif
fc4cc739c65f6afa74302db04947276f
50e3c00f477a4f6ca004ef0567527868a0940569
F20101111_AABFPK allen_m_Page_080.tif
53302fa4912dfd2b39beee29e223cb5a
aa108bd0c116d7a2bd54c343b7373cf30430ef7b
F20101111_AABFOW allen_m_Page_064.tif
2a269614fbc808a4e17242e2442d373a
9b05e76c8f51dbe1ea11ba4b45129a09f20550f1
F20101111_AABFPL allen_m_Page_082.tif
92121260681f6763950f09ca6f50415d
5ee01d99f358c9a2cdd7ed71b09edbc4a0111118
F20101111_AABFOX allen_m_Page_066.tif
872cc60fa16d6dc2cd347f30e33bdb0e
ed90ae63a05f3f69cd85aa140aed212e5e78a353
F20101111_AABFQA allen_m_Page_097.tif
8f36afb9013f5321fb31fc805fc81ab4
f4e075289c2b7f5d4146fabb3968d59d730520d1
F20101111_AABFPM allen_m_Page_083.tif
0c28e71fc1a01e46bf9e84581aee6577
831bda980aa41cf877626409eb2b5a32f4bc90f1
F20101111_AABFQB allen_m_Page_098.tif
ec4adcfdd176a894ada8dac765d1ad36
311cbe6848b0e386a17522a2fcbd3ddcca7cbefa
F20101111_AABFPN allen_m_Page_084.tif
a31cd86789f24b848ad0a36162a9a511
6ea35ff98870d19e59e0e9b22eac0169d8d08262
F20101111_AABFOY allen_m_Page_067.tif
bac1ad6be2a13fecd94db023018958fb
b61f86bac65784117f6497559ff8b525d309b448
F20101111_AABFQC allen_m_Page_099.tif
473ae8ea4170c95c5687a7b289a7a997
f1f989a926a2cf4167e28387ea661bfd06e9f901
F20101111_AABFPO allen_m_Page_085.tif
a0a75b249e3ac00fa00eed7e6ffc772e
9001f2c0b391c95141ff68889c5dea4c1fab91d2
F20101111_AABFOZ allen_m_Page_068.tif
6fc4440a8fa2dd461b9cb05300737445
f48de2b917581f1de3c49ef038d37bfccf175733
F20101111_AABFQD allen_m_Page_100.tif
dc44b4b6ca3b5a6a983789296eda20f5
a3b1c7ed77276a5e15595fc3bbbc72500bdf3695
F20101111_AABFQE allen_m_Page_101.tif
635deae527f1546f3957fbc247a9c679
2521fc7971ccf01b9c2cf55f58add54b438b4c76
F20101111_AABFPP allen_m_Page_086.tif
6bf1ec95cbc6a0cc9bb63b52c4adb909
592cbabec0bbda461a6a7e988054f950a5b3c366
F20101111_AABFQF allen_m_Page_102.tif
6b6554842cedda5e5bd7e67799b5401f
818937904b1f2f436ccf77a099780f363e089e80
F20101111_AABFPQ allen_m_Page_087.tif
7f534c302e450be02cad5e04132fcb86
068ba90987b64bafc84e5f2cb5f1bd8dc209deff
F20101111_AABFQG allen_m_Page_103.tif
12c7f9eee0d2ef7b02252f2127c10adc
2f7facd982f92d05c3585b1c920608b477529262
F20101111_AABFPR allen_m_Page_088.tif
ca263f67ca07f9738de7c5a325f63f1b
d537f392fd8737a4e5db132ef296e62f57382b33
F20101111_AABFQH allen_m_Page_105.tif
7a50ec9bd913fb3c858762e2de39b1a2
71a997fc0693036b3086168c0e8d916eb8602e1f
F20101111_AABFPS allen_m_Page_089.tif
8d0f9d890935e4c966d99d5e92731156
89992fc999f663980c610cd8968cfa39bcb21ac1
F20101111_AABFQI allen_m_Page_106.tif
b9afcafaf5ba193eb084d882a5efa15b
47d72fc8338f745ab2bde7c95b0d620f94318d1e
F20101111_AABFPT allen_m_Page_090.tif
709e1edf0d44c8cf3abbd2e3b78ba1a2
d633d0dbd3f015821f12c1a8686839832dbab434
F20101111_AABFQJ allen_m_Page_108.tif
724a10d3c0d7800d6dd5ba2539de6254
dc5f168e65a77e0a4c57fcab1a4290157094aebd
F20101111_AABFPU allen_m_Page_091.tif
83cea8abe1785ec2808484d243c9f6a7
6a9cb082678c9e54ccceb99473bd1055edcb7cfa
F20101111_AABFQK allen_m_Page_109.tif
8f5ffc7cfd7578348cf15f25433c05db
6890693456210f5f13975c34c96bd41ff34e4b50
F20101111_AABFPV allen_m_Page_092.tif
924f862e8db7db9858883dc84ad98434
98b1070cbba763c995ba175bb528287dc85350b7
F20101111_AABFQL allen_m_Page_110.tif
65cee33b7907f7452de2953905c7e1d6
8903056329c70727fcbc3ee443f1b38d1187f4b1
F20101111_AABFPW allen_m_Page_093.tif
726c0706b5314250324654253f9b17f9
6568bab5548cc754a56e2835cff47de550118ce0
F20101111_AABFQM allen_m_Page_111.tif
1aa2cf8383275fc76bd26d47ca0837e7
c92736b88c0b09e27aed2d3d2453560192a53121
F20101111_AABFPX allen_m_Page_094.tif
230f42b2a0a0019097dfeb283204190f
adfb8b191550d16957f84472cfb4f960bfccd6f3
51725 F20101111_AABFRA allen_m_Page_009.pro
92dd3aeafb973aae4417a8216695aad4
2823425e2abee2600a68e4f3171403cb789f49f8
F20101111_AABFQN allen_m_Page_112.tif
dff645510c01449e12b862be3738de71
10e5617e6cea082c25a31ccbbe5fe2b9298fad65
F20101111_AABFPY allen_m_Page_095.tif
5b6bd90aca46725c4040a09bba301c4b
3d39a441d59ba9cfd2f80cf41263463d37b30240
31648 F20101111_AABFRB allen_m_Page_010.pro
f6ae93e390ef33312abb504b41549296
c2f1775bc9fb787789f250b9103be71d33ef7384
F20101111_AABFQO allen_m_Page_113.tif
aedff230644351392c302aa70d953406
b492d527a07b6e6030d850dbce1ccf934126f52f
F20101111_AABFPZ allen_m_Page_096.tif
94ece4b83f71f2721ddd9c2c2e771fee
8820711f2a19e999583e121d370d56a18652f36f
45244 F20101111_AABFRC allen_m_Page_011.pro
49a941473e4b010a3feef37535442c51
ae42fb290bcc15129db4cf74f714dcd1abb82145
F20101111_AABFQP allen_m_Page_114.tif
106c33e9f22975c97b317f25bef9cb56
c6374219a1bfa74efe515ce3a5787a4c0b8f0dee
14661 F20101111_AABFRD allen_m_Page_012.pro
385f87eab13e43f3e2055063c0dd36a5
a1aa0c47d8e446860e3251f474a68dbf001cfa3e
53065 F20101111_AABFRE allen_m_Page_013.pro
fb896ef94f68f6b2752a9f3e4c0ea1ac
5b05fb964b6ef082ffacaa16e72531669969b4a2
F20101111_AABFQQ allen_m_Page_117.tif
6db94ef099021a254dd64ed14c25f422
3a872c2d277cb5c8195e0993f3f572574d26241f
54556 F20101111_AABFRF allen_m_Page_014.pro
0d7b5080dc0ffc145ebe19d838b331e1
b487cfb1cdd9b027fd515b84d360edeb97fc936c
F20101111_AABFQR allen_m_Page_118.tif
02d85178161593b34d5d45ad0244ff74
6bdce36b7f359f82bfa0aeed54a2d2b53098e3a3
50813 F20101111_AABFRG allen_m_Page_015.pro
77d0bb2e8864e657c1a877894ee28092
c411e90d9ff78e6ee967578766d83af3d409f470
8710 F20101111_AABFQS allen_m_Page_001.pro
9ac1bef7c284f781ded4efe73647312b
b8fb360eebfce45a62f65cd9a07d6399c07f0c81
54071 F20101111_AABFRH allen_m_Page_016.pro
2605544ddb0f9ef836357d3b098453e1
17619f10e45af13e611dc7e3a1c815a0091df61d
808 F20101111_AABFQT allen_m_Page_002.pro
5a80dc703feefb06363e447997850165
f496007d956fde09df7a9fbd7a01e2aa3c3dee92
53399 F20101111_AABFRI allen_m_Page_017.pro
1f09babc3526a2e9326a3aee3fae1795
9f5ab613623b38c142808b3a50c855dac7709643
5568 F20101111_AABFQU allen_m_Page_003.pro
e6beab200bb222111c39ce594908a4f5
1d3d399c1e67f83ea5738d82570ac6aeaf614e3c
55805 F20101111_AABFRJ allen_m_Page_018.pro
e01fbf290f4e4caa3bac6de69526268b
395f9bdaa42e93c2c16a2561c6101d0f49cbc395
53066 F20101111_AABFQV allen_m_Page_004.pro
3e566f9084353c82d1b67d110ffad16d
813e1c6dc987d8573ebb13aaa05d7777c32596ec
53521 F20101111_AABFRK allen_m_Page_019.pro
306a0d839606016ad906837a59216fca
f6108df3136bfaf3a9d35d6a488d52fedc3087fe
9446 F20101111_AABFQW allen_m_Page_005.pro
fe4bf52988c07fbecb71a51d74dd3256
6ad6443194ef1b8bf5fa594d89e43e6fc9c27443
51917 F20101111_AABFRL allen_m_Page_020.pro
7067e740493336a075cc7992f45d3f0a
6e55d36969e92fc5a3e553c51a75fdbf5a419449
96129 F20101111_AABFQX allen_m_Page_006.pro
2f96c5b84fb7083bccbaff056542e22c
df5b95b62d1e2a4fd09a0d6ff2f641a351b725c9
55666 F20101111_AABFSA allen_m_Page_036.pro
44ed0ebf048ed6f9d4514731a44e7c73
15d73e7102ebeaef610400fc4369607e978ad1c8
55290 F20101111_AABFRM allen_m_Page_021.pro
4b34f303aaee54d3f20639609137f9d2
69389981c5205285b5f59ba85de2dede46864d4c
74238 F20101111_AABFQY allen_m_Page_007.pro
4d7059940eb80982a7004940ec4b3547
0d44013ce34804fbf91e06c163ebff9cff12b2b5
57953 F20101111_AABFSB allen_m_Page_038.pro
de71bfb23651f55447acb559fd3924c7
9d3e0792d3b02bee805595af54078948748cb35e
52556 F20101111_AABFRN allen_m_Page_022.pro
189c7f7a421e87c32c14d17459e52546
6112388c5287ddfbccb829f2a66a5a95a90d0bba
62269 F20101111_AABFQZ allen_m_Page_008.pro
29394900b44106e1942f92cb99052033
77eca5e494a80c850d119f1cbf1301409e2f31e2
53139 F20101111_AABFSC allen_m_Page_039.pro
c677309732984e1e3fedfe2efdb77b47
9d1d0b48df590a657cca493db6ce98796bf8c3cc
57468 F20101111_AABFRO allen_m_Page_023.pro
c8657ea22c18e235e1e210754d8a8a71
1afe2411c7260f61d768508d4f2e6f1d65c9795f
54875 F20101111_AABFSD allen_m_Page_040.pro
353e259796de43745b013a5ae6531705
0530f7a7d09baa3957e6761fe8f31bc78479db6d
53035 F20101111_AABFRP allen_m_Page_024.pro
93c54b50777e90ecd03268287f50b4b4
4de0a7f20dd3d1f0645e4649d3842278fcc93005
51159 F20101111_AABFSE allen_m_Page_041.pro
924d3bc17926844acb311a3dd5b25b4b
d8271f93a5f8f2e5a4ea90cc4a3986b3b9c58649
48708 F20101111_AABFRQ allen_m_Page_025.pro
8c760999abf0a0907b2cceb7daa1922b
4caf391a35b08e81697127a4de4bbd87070bd526
54049 F20101111_AABFSF allen_m_Page_042.pro
da7c777811f1dd1fb5925675b068fe0a
ef12c58e6442dd44efd7dd8d06223817fef99c35
56089 F20101111_AABFSG allen_m_Page_043.pro
0a846b73790761ee5a9a89c812b5d045
5cf8ebb0632631f53b6d9fbbd828410b9f2d15f9
13673 F20101111_AABFRR allen_m_Page_026.pro
1d249121c0b1acf95e4945e7ba517d5f
bda57e554e5b5e048252dcaf80d35e5a3cacd229
56060 F20101111_AABFSH allen_m_Page_044.pro
c750bfaa2c626f3e98e2a656949a7f30
b5cc08a1ad6eb0f2bb37f73e9a895bcfcbe1da7c
50522 F20101111_AABFRS allen_m_Page_027.pro
3eb89b98192df2ac5f7edc6066c63e3a
67ca6e4a725fe13da50e78865f0f61907acac392
50502 F20101111_AABFSI allen_m_Page_045.pro
c2a4aec40ab9d24737804d75502988dc
bbf6fd5354d55715f560fd23b20b4ca60a426b45
55049 F20101111_AABFRT allen_m_Page_028.pro
cc6b4c06c474da74a587d4f4313b2ec7
eab0292807d1392992992ada2905c1aa5a684fa9
53237 F20101111_AABFSJ allen_m_Page_046.pro
8dce8e6ebe6500662f5d3f527dd3801f
aed61f371bc27583fb5dcb04e807563746eaa142
53197 F20101111_AABFRU allen_m_Page_029.pro
a5baac215ab9fca248f30763c7539971
786973bbc2bb69b6b5f5440214a8fd6d3361d5e7
49154 F20101111_AABFSK allen_m_Page_047.pro
f157f90dab5f7173328dfec735c28191
6153135102b64566c35cf9b482fe1c28a5ff4964
54355 F20101111_AABFRV allen_m_Page_030.pro
e749bfd581d6459f52936bb253c3555b
7de7f93a9bfff76bf1add32a535555f83e45f03d
57317 F20101111_AABFSL allen_m_Page_048.pro
ceff43a37e57828e74b7060067f2797d
4941228a98740e2ef101167878ac757df2d8b203
55931 F20101111_AABFRW allen_m_Page_032.pro
93af4b2ed0aa973a6ab92a64aa7f84fb
477803a9b0b46be32ecb352e79b5910dbad36535
48696 F20101111_AABFTA allen_m_Page_064.pro
06242dfb6276205565c369aa72c62922
7eecea5ec028f09f92bfa6151fd9c31db11c7016
57942 F20101111_AABFSM allen_m_Page_049.pro
5c7cb63d63db8abc736c0d7a6b9ba08c
ec8bdcd6bc9d35cdd15602a1ffbd4ea15aca2f15
55828 F20101111_AABFRX allen_m_Page_033.pro
0220493e6cf85d4c872e145ff2ab1fcd
0574ea28697ac8f7abd04478c40a707a97ba36fc
19506 F20101111_AABFTB allen_m_Page_065.pro
4d72c7372b9703a01836604613548a5f
a7dfeaa6eee826c09ab64908b7daa81756c83995
52729 F20101111_AABFSN allen_m_Page_050.pro
843eb08b9c5138155aab7a8da3d3ff3c
ac9ce40ca2009d4f53c3ddd1248fa534b9525fdb
54143 F20101111_AABFRY allen_m_Page_034.pro
59d531dfaca3221eb33d4f2fe0dd299f
3664a1eb52f5dd796ca6b25db8d2d63862e9294e
20678 F20101111_AABFTC allen_m_Page_066.pro
6b79f49df0ac9ea761e3529f623dee25
00dd26d2e6bfaa9404b9a99edb35afbdc212d015
55543 F20101111_AABFSO allen_m_Page_051.pro
f303bfd678f261100b33d365a168bea0
d3cf1588911ff182da43eb1a583781c2c93f3807
69098 F20101111_AABFRZ allen_m_Page_035.pro
98800845c040716f6ae7ad72e605c562
23160f40ee5239e3e86e828a97e27d88f1cbcbc9
50651 F20101111_AABFTD allen_m_Page_067.pro
7f8e08bbe5d674d95dc31831c76eb408
08efa5c55fdd59801046a4768b77fd73bd35a70b
54791 F20101111_AABFSP allen_m_Page_052.pro
788fd9032a4bda6073c184a36ee8a7f7
72888bb832453768c09a00eaa32734d7dde9e284
51825 F20101111_AABFTE allen_m_Page_069.pro
9ccaa68815db745fd2b469a093831c5f
f8d02d03ccfa52a93856145abba25db39014a6e7
54488 F20101111_AABFSQ allen_m_Page_053.pro
cc0b9c6233cacfb478bce591c9fb8eba
6587332c403b58893e6c061cef8c0d2eeea60eec
53360 F20101111_AABFTF allen_m_Page_070.pro
9b20597bdb8159d0c223095bdf178494
e89148963e8210d60d3ab193431d7d58c19c3c1a
53109 F20101111_AABFSR allen_m_Page_054.pro
f9f3812dabc87d9e8009c74832a534d9
a2e54829d21108a4feb4758b36c730004263c684
55192 F20101111_AABFTG allen_m_Page_071.pro
68d03c235edbd24ae03fdb9f864970ea
7df3f60d44cc456e53b848f90ad385bffcac41ea
56490 F20101111_AABFTH allen_m_Page_072.pro
04d8b24eecf137b64b6892b5212b815c
257941557731af7da7082a2a5a8a29596767a7a0
51760 F20101111_AABFSS allen_m_Page_055.pro
dce9403f9285b4c6a8e5262abc644056
922c745c0772b2e5dc46134e803d0758dc64f335
54836 F20101111_AABFTI allen_m_Page_073.pro
ddfeea689dc933c72b1b6e1df19d1fce
fcd71afb41072f2a1625e99a766ea94355db8c8b
22413 F20101111_AABFST allen_m_Page_056.pro
8fb662b90db0fe8fcec6e1b9d5ba4f69
1c2120efcb5d5305767d09f20166b4b562194233
56306 F20101111_AABFTJ allen_m_Page_074.pro
8cfe3a2145999a0d62afaf5ec401d6ba
21ac6c3fd3dd3762cd6bfbcf458cdb39eee326bc
54727 F20101111_AABFSU allen_m_Page_057.pro
c98660888f74471a900161b1aeaf639b
537e64c8fbe8563bd03b32c61a4c90955fd9c44c
53611 F20101111_AABFTK allen_m_Page_075.pro
713da1252da03ef85ddf09fc8322d93e
da56f062e1d02d77babeb0021431d9f9595b013a
46240 F20101111_AABFSV allen_m_Page_058.pro
33ed72ae88d9e082e275bfc919c3da5c
a87aa345007cf649ab802144b85fca19a3a67bb7
28796 F20101111_AABFTL allen_m_Page_076.pro
b9f3598042cfdf570ca2de7ef37e5af3
e122d657dd76d4fd0176757b2bf1f621662901dd
48345 F20101111_AABFSW allen_m_Page_060.pro
77a63e1c640352a9020e8073367cda28
4309bd3e099d83184cb3aa87a8da50d6c376deca
17073 F20101111_AABFTM allen_m_Page_077.pro
d61bc01c6d6b4106e18347be4d4c239b
4d0c3aafebeedd3fee4052f3a79f9e29c885f4de
55365 F20101111_AABFSX allen_m_Page_061.pro
f0e7162816d8dcf09ebabef094a83664
4cf35d4693e088a83d46eac6ba2fe3cf22b99e72
58358 F20101111_AABFUA allen_m_Page_095.pro
73190e725c122120468643ed9c6fa05e
cdd64f3a0e1faae5bf7bdf61e4a854a40c9eb6f0
12723 F20101111_AABFTN allen_m_Page_078.pro
efb45d70d01eb4dd168f6a3236bd2a1b
cd5eb28b38906e2ac0e36e8eac190bb3184e0684
53625 F20101111_AABFSY allen_m_Page_062.pro
e5e2b2b75bb949d2c005bc1bbacac467
53dacc261846d744cfb7168654a3ec40f289760c
73211 F20101111_AABFUB allen_m_Page_096.pro
65c08702199f16c55e0fb04a3a608186
b61be41361363f045fe80d21d51e05c1fdf40bab
24993 F20101111_AABFTO allen_m_Page_079.pro
db65ed69b12eadd75093edc5f3b84b62
ce4fe486bb59a811aec52c27cc37edbe1bd46340
52115 F20101111_AABFSZ allen_m_Page_063.pro
110c4d2cc338f4e7b40da9ed93e7e199
d6c2ff997408e58929cbb84bd04faabd6b5df4db
52314 F20101111_AABFUC allen_m_Page_097.pro
49cb732108c880a21cb94f2ba3f52792
311ad6f1f6709b75024f549969d9c8d696201c03
21215 F20101111_AABFTP allen_m_Page_080.pro
14d746673912c6417ad330dc3170524a
07104cc120dda027f464507677f652116f9a6fec
15595 F20101111_AABFUD allen_m_Page_098.pro
06e858b219fc9b4a401d83bdbf5e1d44
52408a40dd74c73f363cb6e2158957dd0cc5e092
35807 F20101111_AABFTQ allen_m_Page_081.pro
3c48a1ef34d3926ea0751ec5aa707e90
cf5ec59dc44108cc4f9182de814611dfcdf8d0ca
30682 F20101111_AABFUE allen_m_Page_100.pro
e73db66f4c91c255bc1b0cca5d7c9c43
38b53ea2de08dde1df99be93dc8e5b2cd7f0e11d
9704 F20101111_AABFTR allen_m_Page_082.pro
f965afb91d15e9c2467d4d0eb038ce4d
3b62e8a2535b775920aeaa561d164691c3c13b47
56217 F20101111_AABFUF allen_m_Page_101.pro
3c80bb5de953387c8a36e302540ea7bb
4061b5f77d62de65166185d3aef1865c49101988
51345 F20101111_AABFTS allen_m_Page_083.pro
412a013d643923e74ed778b0f5aa3b8c
2903d2c027eea47a386063dc70ece63957afa75f
11225 F20101111_AABGAA allen_m_Page_065.QC.jpg
35199f19a3378555124e887e697e9974
7a7165e57d492c1512e277ff0d80d61ddf6ce874
39209 F20101111_AABFUG allen_m_Page_102.pro
68d5731b0e3bc7af8c7d1c219680b170
7a5774864074bbed61f6f93966897ee4893e3a77
11632 F20101111_AABGAB allen_m_Page_100.QC.jpg
aa8cf29201c456a03202151c059ec5ff
1bfcab42d65f1a72ff5e39335473e65a15f78b0a
55806 F20101111_AABFUH allen_m_Page_103.pro
b3488fde4813712d3661fc7fdc790d11
c6edfa05dca84d6e15543b593a5d1bd7565921a6
55816 F20101111_AABFTT allen_m_Page_085.pro
362445b0f14aead45082c85395f66a40
480501b5ad863da5247c1293b3a3051086013c76
6590 F20101111_AABGAC allen_m_Page_114thm.jpg
4c41a04be93a61db2c67071c4706ce99
72f3927dbfbf8d4c9de2959e5d7b6d239d6830db
16484 F20101111_AABFUI allen_m_Page_104.pro
2464bbdb2f401e3f07fb5f5ad0a6726f
1925b08bda49e7e7bac8b8275cc7cdc5be63148e
52720 F20101111_AABFTU allen_m_Page_086.pro
1598cac212408e3867466379e3875335
0e483346210e9c322944f7f90ece2ac78b16e3ef
6936 F20101111_AABGAD allen_m_Page_038thm.jpg
969fee6bb2bb52d1b80f81739d1004f7
57ea28eb088ab0c6fa3403b6a308e7c261f1c1cf
50991 F20101111_AABFUJ allen_m_Page_105.pro
3ce8d3f6c98c6feec26bfda67659a3ed
92ec50a74b0246738b06766182efb983dff62902
27494 F20101111_AABGAE allen_m_Page_017.QC.jpg
a6f75e60cd7ab9505b032ba8fc190df7
0646e875409c79c3c7aa8d654af267d7a5279389
16078 F20101111_AABFUK allen_m_Page_106.pro
8654553935f5cbeb9087b26d818568c7
01b411ae20b09e9fa42761ba90204ed8e564e5c1
53902 F20101111_AABFTV allen_m_Page_087.pro
1438f68190c0d6272fead90f3536c93c
0dc7e051228885a2c7130c0bd954300684988179
6793 F20101111_AABGAF allen_m_Page_093thm.jpg
e9d32d342862445478882e74ee236a41
93de846e981a483832bf576c3b900135b8c7f849
59272 F20101111_AABFUL allen_m_Page_107.pro
bdaa7a9ff8f50a4c1d7ae226438e13f5
006a82b0ef21dca4362aead066d85ae445509a51
52207 F20101111_AABFTW allen_m_Page_089.pro
98c2424e34b235786a98f1fa3e30fc0b
be24bdb304352f72d5c29ff2ba53ea1d28c4e81d
24606 F20101111_AABGAG allen_m_Page_024.QC.jpg
17b1f62ad3bec4c41d72a4c9386fef18
51ac7350b419dfad59a540d71564c18beae45a41
2942 F20101111_AABFVA allen_m_Page_007.txt
45781912dd6615982def3a25ba141802
c1a6a9ce5dde0a85c34521c187448182c878ffdf
52811 F20101111_AABFUM allen_m_Page_108.pro
6ea8ddeff26da0e4214cfa2d3c44a427
929aa494ac98ab354e2cc00a2c2c011721d2f71c
53233 F20101111_AABFTX allen_m_Page_090.pro
f3d3060954776af0632bfe021a5d8c33
aa072d215758fb21d6575bb8bca4730d0bd367a8
4731 F20101111_AABGAH allen_m_Page_081thm.jpg
0d4dd361bc85acda7abf8b45ebd571ae
2f1f664ea37cb5122b1eabb8326704b14c171752
2055 F20101111_AABFVB allen_m_Page_009.txt
6476795f718a2dcf0ee702d5b2e50a28
0e9495fac905a7aeee3978dd889dab8ed88cab89
28841 F20101111_AABFUN allen_m_Page_109.pro
b0448999bf12ed2acdb4c9ef9d4c6889
13b45c33019137ab6e6881bebd97ff3bad957ab9
54770 F20101111_AABFTY allen_m_Page_091.pro
849615c11a63ca6de2043f3968b3ca07
753eb76f543e241903b00f177b15b72f45435ef0
27617 F20101111_AABGAI allen_m_Page_062.QC.jpg
7581f1d01e810baf3602ace1c653657a
76718998985c3d74d72c3c12e5b997c4d4f4df0a
2291 F20101111_AABFVC allen_m_Page_010.txt
444b5dadb8d0ddc286fedd77cdecb318
87716850d86506d6c87d6c40fb9bc978d20e91a9
30673 F20101111_AABFUO allen_m_Page_110.pro
bfcc428233efcca488cfdce923d55410
16bbafc4765f0c364224a9261759a1f8ac8d02cf
55988 F20101111_AABFTZ allen_m_Page_093.pro
4657baf5da41bb01cfa9a145a8a5cbb6
c6c8a9563e25596d7bbd76e3ff2ac1d57c83cc17
30817 F20101111_AABGAJ allen_m_Page_035.QC.jpg
5ef933fd08b13c3901b7c4feda7d2e55
1ee0c00703ae34175e99528bebc0395c2e0f1100
1985 F20101111_AABFVD allen_m_Page_011.txt
ed90b4c49d71246586ec924d70ba97f4
15df813b014cf2efe22d275e84db7313b753eb18
25890 F20101111_AABFUP allen_m_Page_111.pro
467b7905a64a712416e863e8c04ceb3c
2a16876975687ff6f3bc6d508fad6bd9d94135b4
27159 F20101111_AABGAK allen_m_Page_030.QC.jpg
e6ceed7e5f71e8a484e82c19854402f4
97cce848eb703edeb8d4b440b4eb5ad01436ad96
589 F20101111_AABFVE allen_m_Page_012.txt
05c32b13789e48ae2bd5808dbcfca511
e2faa384893c3929800fe0f049edf7f43662d190
64348 F20101111_AABFUQ allen_m_Page_113.pro
03ea2d6fc7050817774ae19507b724bb
f0dd3be0596a40f2f362753b6f5d7eb147a1794e
26571 F20101111_AABGAL allen_m_Page_089.QC.jpg
e53f2469303fb203b191ff7381c5e967
7bb49429bb0e5e548fee8b36cce0a310e7aeeb7c
2144 F20101111_AABFVF allen_m_Page_014.txt
9e3e0adc9779ebb88544b243181f0f9f
e2f3761b91e047560ac3e26368be94e63ce2d1c2
57962 F20101111_AABFUR allen_m_Page_114.pro
bc0051e96b56cad646350a7df12b9d5a
2cf68897df8a7ae1bdf37f0a2aeee413853f70f0
8871 F20101111_AABGAM allen_m_Page_012.QC.jpg
2d61150274e4dc1a092637bab14b6cef
1e01be49f573da7ae1e8941cad2feb6b4790eefa
2007 F20101111_AABFVG allen_m_Page_015.txt
c3d08c946f3feba5f64b59759cfed689
9bf3d106309b49097852438649e8fbfdc41b4784
66234 F20101111_AABFUS allen_m_Page_115.pro
1fb372a4a2a7efd405808b3e938a6545
1fe051a8be96af82ec67558ecdaa9c41ca040aa7
26102 F20101111_AABGBA allen_m_Page_055.QC.jpg
0227d680e95959aec542c99778b2cef1
0cc599c156911d7138d66eeae8fa552752a1ea82
29927 F20101111_AABGAN allen_m_Page_038.QC.jpg
aa7557926a8bf72b373b5abe994f8985
c1a1e38cec1a4e0dce5ddfea3e525ce57801c3e8
2136 F20101111_AABFVH allen_m_Page_016.txt
f326230e4fecc75192fb002abf6e4f75
8b0f6aa2e1a030efe5ef88c5238f54d96eb36852
62585 F20101111_AABFUT allen_m_Page_116.pro
eaf27486a05caf75979a6cfb1e9df354
8391486c38d72d549302495fd5cf1e3e804e27c5
4904 F20101111_AABGBB allen_m_Page_006thm.jpg
db97c22b1982dc3da47690c45d2a5c15
bc803303fc0512b8018c4ae1eec61e4b473ab4c7
6493 F20101111_AABGAO allen_m_Page_089thm.jpg
2afa1bfea2673b7d64f14a1ebc2ac336
547e5af8177fae3295c65f16ca040595c20a50bd
2102 F20101111_AABFVI allen_m_Page_017.txt
972e512008f8f4018f4f1f83b1b83917
10b84134b0035d8c48b993055b5eacb4ee7346ab
3219 F20101111_AABGBC allen_m_Page_010thm.jpg
91ff3ecc6bfb416888a8481c3dddd697
d0e9adccfcaecb0a6eb6a3935f49c4d90bcecafc
6336 F20101111_AABGAP allen_m_Page_084thm.jpg
5186a72ede852fe711bca080bfca64ff
27780b899b5dd149bd7a416111a9e95ecb155025
2214 F20101111_AABFVJ allen_m_Page_018.txt
9f7ec82ce0b3a41fab5e40cf7d971cd7
4cb866105745e68dc75edddd80ac61ec3fd6f78b
7845 F20101111_AABFUU allen_m_Page_117.pro
5fe06f6f6bbc5f4c189e5fa8ae06791a
d2a0ceb633a4cffcfb59764d4111e8851ca99679
6571 F20101111_AABGBD allen_m_Page_013thm.jpg
485d26a3025960d82ccecee8c7e98e06
ccd605f5e656bf62c37d806e00ed6265a81dbf6a
21791 F20101111_AABGAQ allen_m_Page_009.QC.jpg
c0ad8082f90907af1930e53fd93044fe
a4c70fdc9cdf638a8e1e96d101b9d79a38ac16a8
F20101111_AABFVK allen_m_Page_019.txt
3b25e1a0a34af053bc78980c5e35c148
a6f64d790dbfb5e5b52d1616822b2a2d4fffeada
500 F20101111_AABFUV allen_m_Page_001.txt
11ce6f2d4d9be854a29a204679a9b7ec
9c1318a37ecdfa3ed263423b43d4d528d8098ccc
6883 F20101111_AABGBE allen_m_Page_014thm.jpg
62f96c838d92409dc66d2a405c4dfc83
12e7842ff2ed5ea19508ce9732c22ca1beb5803d
8802 F20101111_AABGAR allen_m_Page_104.QC.jpg
1546f1476148165c7ec4409549866e84
6aec3fff6be3cdfc21a20c5231b19a791ac629e4
2178 F20101111_AABFVL allen_m_Page_021.txt
024abc7e0f9b049470a2c91632806392
49e7bd08dc6dadc8484cf0bda1a23fdbe10e3cab
91 F20101111_AABFUW allen_m_Page_002.txt
c304fafdfdcc61b672621f9e0750a83c
a6f76ba80136908d6cd1339c591252d306aa432f
26755 F20101111_AABGBF allen_m_Page_070.QC.jpg
05e8be3f66473dfec165126e06c73ee7
78f5d8c5168155ef7c93268567cc0de934776cf0
27423 F20101111_AABGAS allen_m_Page_084.QC.jpg
615149269d587bc0abc4ca197d65b216
a89f189d1db9c431222308f1d96b9f45c28f13c0
2106 F20101111_AABFVM allen_m_Page_022.txt
ced2f8a94905cac723ba34b1ea80d705
fdaee4e192e1e9d1075a2173ff85dbc43782f60a
305 F20101111_AABFUX allen_m_Page_003.txt
f52425d20a9dd32bc7eb94538ed6498e
53934e23e128cacf97fc8422a82e209126373221
6660 F20101111_AABGBG allen_m_Page_028thm.jpg
f05f9aceb9ad9b1a53567c299c440777
c5492b836a045b515a0e6f66d432d8909dd7311e
2015 F20101111_AABFWA allen_m_Page_041.txt
abcbadefe8eabfa613c039251d36feaf
aef40a746035726b8f57be3dc51b3afb75bb71ee
2298 F20101111_AABFVN allen_m_Page_023.txt
dfae4cf7382c472ca997a917656ec7ba
cf741738059ac3f1d9d5cf9ef1e3588f4092e025
2134 F20101111_AABFUY allen_m_Page_004.txt
ca78397b8ce0c8d826ea61cbd149c08c
059f1690a1c5ccd4939bf31245004c53f177d076
6044 F20101111_AABGBH allen_m_Page_027thm.jpg
9d6b12116fcb771bf33592745b1e1452
245ff7263fbac6bc45b7ce9a1b518d55c31a262e
2202 F20101111_AABFWB allen_m_Page_042.txt
f6dfc51debc7c893614c7ffc06704d9e
73cd29afb5ad327281b7d709ee6ea14636462f83
5787 F20101111_AABGAT allen_m_Page_103thm.jpg
b46ad7ea67dd0e98c1fcc62634bc6cab
26e595d93d147fd03b162912c463f8d87e2f95f5
2176 F20101111_AABFVO allen_m_Page_024.txt
fbae20004575d0494d0bef32a7261907
b790e3d5c8078863c7b2e5ac74489aca7f057b83
386 F20101111_AABFUZ allen_m_Page_005.txt
fc52deaefd4031c4ca20e2cef8b589c8
a201d7ed42b62d41b0fbac776003bc4fc51c389d
6546 F20101111_AABGBI allen_m_Page_085thm.jpg
d05ccc0b6a5fdeae5ce97c808d5c94b7
b7be5ca14ea32e79747098eb4f7557ec962772c2
2195 F20101111_AABFWC allen_m_Page_043.txt
d71852afb6311de9a86baf111cb07077
d6c594d618c11bff4be7ca89808888296a106b28
6481 F20101111_AABGAU allen_m_Page_020thm.jpg
fc44bc58a0edd62de3a6e919d973784e
1791968ef3efd8e5c4d131af2986feb23184b8c9
1977 F20101111_AABFVP allen_m_Page_025.txt
293998bc0046417f55d4b8e4c83263e2
0dad141f8c5e8121aa9ede933ed200bad484541c
6005 F20101111_AABGBJ allen_m_Page_068thm.jpg
33c0a5fa20c62cdb9af08a18fac10d25
d20ee84eb17e7e7c069e9bd07da54f8520f62670
2219 F20101111_AABFWD allen_m_Page_044.txt
9305133bee9a4ea10aebfcff84f149a1
fe3ec464d7babbcfdc3bd0df6236657b87087144
5856 F20101111_AABGAV allen_m_Page_107thm.jpg
77d7b92b6906d6942f8fead83f996411
d8781e605ba88b2031a4310c75ae720272eb390c
544 F20101111_AABFVQ allen_m_Page_026.txt
b51da6cb59dc8fc6479d7462ee7cd172
2e5c48b3cd4ece6afdb7fd7d572ff5fbebd2bf6e
6534 F20101111_AABGBK allen_m_Page_073thm.jpg
248b6aa8081e94c6533e4db20e9264a8
ffa8eb3e9094e742e109a74f2302c8428bceaa01
2017 F20101111_AABFWE allen_m_Page_045.txt
55a2febcbbf6cff456922d98f68b1bcf
c580d00f7be598b6e10903c440ffe2c30441b439
1713 F20101111_AABGAW allen_m_Page_005thm.jpg
f4219edb4a624e3a53424fc091e05877
1acd444c852bfeb21b67462897e6c35f149810fd
2100 F20101111_AABFVR allen_m_Page_027.txt
96e2988e22b9f1fc41cdba708a2c8f29
7d06b663d6b6fec91e0a059933af3dd6ceb36ab6
29489 F20101111_AABGCA allen_m_Page_051.QC.jpg
1258d00134b5fe47901fd4a23b9dc3d5
6019bf23e32d1f6d3fe78fd1e5625b94b76cbbe3
6197 F20101111_AABGBL allen_m_Page_070thm.jpg
8aa368eb1deeb59c33e9543adfda5522
6a4ceddb157025b374048cf58a446a8b2a97cd71
2101 F20101111_AABFWF allen_m_Page_046.txt
69e0a216eec3f9313c6a6e8b23df3019
b2bd33bb002e43d90b3b1c54e0f02f637c43577f
6237 F20101111_AABGAX allen_m_Page_060thm.jpg
02b2ab9748862232ba38c180fcaa3465
2fff4f8c6852b9988ac43c81d7eaaf81ea5b3360
2122 F20101111_AABFVS allen_m_Page_029.txt
380113ab2187d44cfb89453952b5744b
05436fc0aad1d2bc43503b234cb9fd46613d65da
6995 F20101111_AABGBM allen_m_Page_001.QC.jpg
7bb634c892cd3204bad40adf7d401fd4
f9f2e491619625c5ce4bfa111af6e0c653f550f7
1988 F20101111_AABFWG allen_m_Page_047.txt
4fc034058d07f3caaea7773cb6436d2d
5ced10258d3d284c3c4573957fb1d28600f3d5b3
29011 F20101111_AABGAY allen_m_Page_113.QC.jpg
3080b2f9770b2becc2b2850e408066d7
cbc4e3050c75b39bf0871ea52476096e6b0b5ef9
2160 F20101111_AABFVT allen_m_Page_031.txt
8dca864f11706bac00490007d1964c45
54e97519cc77e04821f7a98859bc423de93194b3
6575 F20101111_AABGCB allen_m_Page_019thm.jpg
c62e79c21398c6b10074d67426032348
da78b648965dfa88ad4e938887844b992de1ecd7
30268 F20101111_AABGBN allen_m_Page_096.QC.jpg
6bd5d3cf876f22d8395a29e0284f5ec1
31b39d4d279cbe81bfdca611fb45f03741b01603
2302 F20101111_AABFWH allen_m_Page_049.txt
39b458837b62b57dfff34c1010dbf78f
b016f91880eb95115b77f4df9a4fe8352b06c117
26246 F20101111_AABGAZ allen_m_Page_041.QC.jpg
c0422c8c54eef70c5eb9dcadfdaf7312
0576f31e2e3f70ff8ea026ad80ab372356915aa4
2320 F20101111_AABFVU allen_m_Page_033.txt
d066955203f7380c4722f7778257951a
8f89f2d1a64ca9f894f97f50ef3c0760276a55fc
6258 F20101111_AABGCC allen_m_Page_041thm.jpg
953641f245269feb95c4ea66c51ee3d4
2a742e7c68afd469f79fde0a0ffcc08b2bd4c194
14877 F20101111_AABGBO allen_m_Page_110.QC.jpg
a80d0a0ee2f56fb1cda14842a66dcf90
606d1af1b2643f10620bf51d0b204caad14eba09
2080 F20101111_AABFWI allen_m_Page_050.txt
c4b1e84c8535b807ad4831dc46f15a9a
87a1f219f66bc096a54585f34d6c46c7ebb1ac05
26530 F20101111_AABGCD allen_m_Page_025.QC.jpg
2e87890e8ec92c3eaa1bbecdd0e9d41e
e752b79295db53a3124c895cf0ed255fe112e764
25542 F20101111_AABGBP allen_m_Page_060.QC.jpg
0ad959b4320dea99f911b8b995a5d4ea
0500a87cfb249fa3da4828f328411da20b4be716
2206 F20101111_AABFWJ allen_m_Page_051.txt
156237e1c99050b097879177d235b0b1
82d774f801666d8cac00e4bd2cf902e1c328d8bd
2799 F20101111_AABFVV allen_m_Page_035.txt
c244e5f3a86c53042d7f3e0e35d21afe
b9070baf1c5c80db8b2308571ab9d9669c9f1188
11507 F20101111_AABGCE allen_m_Page_111.QC.jpg
876653024260ee29052fc719591cfd26
6df20221f6793a844f5ba30c1d6bee7611dae3d2
5123 F20101111_AABGBQ allen_m_Page_082.QC.jpg
701e3deedda576299617f907f214098d
40fd66c45c8a8bf252fdbf4cd8b0524c91c8f5ca
2151 F20101111_AABFWK allen_m_Page_052.txt
e5ce98439fda4754965b76005c996fc7
7bffebc0cafa40cb9bba449aab30a06287ee91d5
2218 F20101111_AABFVW allen_m_Page_036.txt
4a8149494b85800bfdbf220c626a4bcc
6b6987f18ef2b3e9df1a9f89900f78921dba3327
29925 F20101111_AABGCF allen_m_Page_115.QC.jpg
ba6e3baad30b3ac36f452b5598519081
4f7ef1e31ad6a82f733b45c733979e1abf8cbadd
6101 F20101111_AABGBR allen_m_Page_047thm.jpg
3bedab8203b55865c5fbb944e382bdb5
475bb627dd9f8ecba60d0feaa49d7c14a26418c2
2305 F20101111_AABFWL allen_m_Page_054.txt
7a8d7562e320d68eae5b000233088d45
b3b173224ba42273dc721b05ac5480e75ae5cc11
F20101111_AABFVX allen_m_Page_037.txt
0a8643f38d646f167ae562dee1cedb84
ddd7b408989c257ff4473cc29b183a5ee3435e1a
25403 F20101111_AABGCG allen_m_Page_027.QC.jpg
208101adb12dfb6a33fb57d7ef484263
5e38b94dbdff7ba4994ce48a691146e9a2bbc0bf
2108 F20101111_AABFXA allen_m_Page_070.txt
4e7f974cbc5e5a6dd22c4654b2445752
94ea172d77360811273b3c5f8d38cf7ec96fee97
29813 F20101111_AABGBS allen_m_Page_036.QC.jpg
3f37cf15c2c84a71e512179c6679c644
4e4c4ea82addbfbc776ea53976968a3987913a4e
2087 F20101111_AABFWM allen_m_Page_055.txt
bb429abe971e6e24bc22619e6ac8b22f
981a8c26cb23ab1c9c611de23eb9a0d0c914b7a6
2278 F20101111_AABFVY allen_m_Page_038.txt
a58f43a16a91cbb33df873c4fe194925
292f2aca3ca65340a208cd2eb75dbd8f10cc6f54
6592 F20101111_AABGCH allen_m_Page_016thm.jpg
bdcbafcab98b8c30cd7964cf6564bb5d
f2620e4aca8940d258db14a46fa8c81ac0ebdac6
2185 F20101111_AABFXB allen_m_Page_071.txt
fff4e5ebfc98469180cff633731d4a2b
b5cff4144139e689ab86f2f80b37622caa1716b8
27460 F20101111_AABGBT allen_m_Page_071.QC.jpg
fbf266affa209744fce323af1295299c
b93e8966577d0a10922624791cb656edea9ae0bd
910 F20101111_AABFWN allen_m_Page_056.txt
789b97cd006b26c10510de48b817579b
5d5206db94884344665077b5023521afdbbc619d
F20101111_AABFVZ allen_m_Page_040.txt
a1ab89393d1ee1782ef0bef280b5bc33
0bcbec85f3222430ffde234a8984065e3f24abc1
6584 F20101111_AABGCI allen_m_Page_050thm.jpg
ae5ee34725cae85f481c8c3a8372676b
a925c5f27042f875f9f43f0e21b9d619620d25e4
2220 F20101111_AABFXC allen_m_Page_072.txt
42d4bb42210370e62e2d5c217afbd876
8f9a9e27342c6624c8ef2572d4851d575e6f54cc
27542 F20101111_AABGBU allen_m_Page_019.QC.jpg
b17734febade0e377088b20c86e16cad
5c03602c949a29740de6911f3578afe21a33523e
2226 F20101111_AABFWO allen_m_Page_057.txt
f25886dbb646d3b97c2b7b95833d55d3
b0c338b272b2d3591404b9883141632f89092bc6
6611 F20101111_AABGCJ allen_m_Page_095thm.jpg
f6d438af951bb137b9e6162b58651020
b0ff821081f98d05e1a1b5a8756d61bf722fd25a
2169 F20101111_AABFXD allen_m_Page_073.txt
3d4a1e0e64844549aa566d46273c58e9
f43f3a85b89568ebf1bca3fa58b89a4247387d2e
28242 F20101111_AABGBV allen_m_Page_028.QC.jpg
a5b7b827da6f06dd22471a9b35cd7a01
2b1be1a690c39a7e63f1255e67543bf4113db8f1
1875 F20101111_AABFWP allen_m_Page_058.txt
99746609afc34deddf1c87a236b1e90d
ec30ad0a6d9dfe2d51168c12f0fc2a91ed20e437
27384 F20101111_AABGCK allen_m_Page_034.QC.jpg
9ec4ee8971a8a970095fd33e9487f7db
66531f7eadbff533d5bd4a26a6dead1c513f597b
2209 F20101111_AABFXE allen_m_Page_074.txt
dcbff769b22f6e3984e4b207d6b9021b
409a603a2248c32229e3e734ce34bc5794af2d69
F20101111_AABGBW allen_m_Page_004.QC.jpg
f827637e30013358813438f26f695910
74788719610c42bc79e5b1338d0a26d656f2a212
2348 F20101111_AABFWQ allen_m_Page_059.txt
cd5aeba6d4fd2471573c778acc5e7a81
e45720b4f113b47bfbc3c2893c704541758f7dc5
3493 F20101111_AABGCL allen_m_Page_080thm.jpg
b75ec71cf2f528df61cfba9dc3dc0715
43c5a94e7f0fc9be6bd2f04edc545cf7ad6339d6
2120 F20101111_AABFXF allen_m_Page_075.txt
051dfd90261d7625edf5fb6d51cd8762
31119c68d7f1128e790b5a9b9b8025afe3470c54
6437 F20101111_AABGBX allen_m_Page_042thm.jpg
cbe3c59c8b60387128a6e4c4ba49e904
5761f9119a617eacd5275a1050512a56c43572fc
1971 F20101111_AABFWR allen_m_Page_060.txt
9692616e8ff54bd550e159216fbb76dc
e3054b6957cd9ddb82e391723b96f5358d8f262d
2376 F20101111_AABGDA allen_m_Page_098thm.jpg
42b65845e53d118bb9aa4a1b9a7d0c2e
99076fc3908632d337d2ceb4438e5dfab66b523b
21580 F20101111_AABGCM allen_m_Page_118.QC.jpg
1e9b312186aca3fb26436fbcc3ba91f7
237565113fd7d1bd84b7a6b9e81f621f89086fb7
1197 F20101111_AABFXG allen_m_Page_076.txt
c89e8c4afcb44bdd2d5d81d58ebd4798
12ec8f5b322d4b710256ee1eeac6ca365e478289
6335 F20101111_AABGBY allen_m_Page_005.QC.jpg
7d5dbc006b6e73a4af8fd2602043717a
3e92132069f92636a25cc379bcd2e0ce95bae17f
F20101111_AABFWS allen_m_Page_061.txt
9e369419b02b8ad6f316220f3fb5f814
4a8dec7f9f33642ce30522f9c7d2304b63c2e543
28028 F20101111_AABGDB allen_m_Page_095.QC.jpg
7e2872ee2e357c9e92af99d2ada5ccc5
3ff7c1719eb61a95b10154172265c922fe143e15
6765 F20101111_AABGCN allen_m_Page_049thm.jpg
d71252a6ae58f85b4ed64e7c9d29d975
68bad0f8d808bd48b2824f085efff07eabde20ca
1116 F20101111_AABFXH allen_m_Page_077.txt
f2f0751b0e39f3573cafbc4c423f1587
3145e08dfea65dda367342e5b553a9ed33521898
15253 F20101111_AABGBZ allen_m_Page_076.QC.jpg
f539a1048ceffb25481868e8d45da43c
eb8facece294aafe217168992724dc7e9bbab574
F20101111_AABFWT allen_m_Page_062.txt
acfa410ab64248907f666d5472521fc5
7a96eba1b2ed55a347d7d746d6680c1dcead8be5
6175 F20101111_AABGCO allen_m_Page_067thm.jpg
a034e08713c12d89b4d3770225480010
370bbb0d3b432030eec8e97931458e7923054232
656 F20101111_AABFXI allen_m_Page_078.txt
0d9332bc1431a820bcc1018a2e073ea0
2a0667613f58dcc6e3828e03efa6b985d896569e
2069 F20101111_AABFWU allen_m_Page_063.txt
73e00b653d7d9ca82caf42ca8321ffea
31464b20e51668aab3f318344afa3a407fd8efb7
13324 F20101111_AABGDC allen_m_Page_105.QC.jpg
07d779390ee34bdfeb330ecc5e095ce9
79afa0b948f9324f87f8b5d41b0109a10ac5d4f7
5010 F20101111_AABGCP allen_m_Page_009thm.jpg
bc509cbd5237c7c8c3e0597316860a33
c03b6013fcbc7056c55ba364fb5fe8493ad1fee2
1348 F20101111_AABFXJ allen_m_Page_079.txt
2f2bceedb456e007166937ac78c454c8
434ae8221ea715b36d1ee78b89d8dd5437b91aa5
825 F20101111_AABFWV allen_m_Page_065.txt
05db88ef8ba2c94e7dbacc2aed5a69b3
570019e5a55de6eac39f4ebd6a49485bcb7cdd0c
6662 F20101111_AABGDD allen_m_Page_072thm.jpg
41d253004d5b09a2ec836494e51fc6a4
6b0404d8d82f304541841829915fa6afce41185c
6622 F20101111_AABGCQ allen_m_Page_112thm.jpg
07c7f7ead5bfec270812afc54f0c5fbe
03b8636c2fd4c178b02cfaf12c0e4dbfde363a39
1043 F20101111_AABFXK allen_m_Page_080.txt
54025fb4116145e2f622cad5fe833237
3b16ce4804b13596ced49369fe87f4592f120b9a
28053 F20101111_AABGDE allen_m_Page_016.QC.jpg
d8c7773f9417c5ae81543a7147f76c39
e905ec836826160291420709fbb74364cd687272
5993 F20101111_AABGCR allen_m_Page_024thm.jpg
37a1bc004d8be65e7695a79855f7c5e5
72e226b2b4461546ec542eb8c4b7b5d7e4533b3b
1774 F20101111_AABFXL allen_m_Page_081.txt
355d1ad7c87dc933bf6ba85e7cba2c4e
79ea7aa733f5748e7774991e4b340f733877263b
1201 F20101111_AABFWW allen_m_Page_066.txt
1afed7fb16973ebc94dae056f5e2f086
69bbb0579fec32dfec4c05fdbcfc72585a3aa04a
23477 F20101111_AABGDF allen_m_Page_103.QC.jpg
cc53f5b44421c1d0abce221a14ecc36e
c8417209209db2ceca22e4f107d1e3668c723280
3015 F20101111_AABFYA allen_m_Page_096.txt
b44df7bd82c2d0e8b5b86700cae3e8ba
8e060bc177aff30b3f6fe32e92ead0752ad7bf3d
29653 F20101111_AABGCS allen_m_Page_048.QC.jpg
821e584190d149321648513fcc425dc1
c7c9fcbb6e0c5c0f1bff941775099aaa664fd62c
505 F20101111_AABFXM allen_m_Page_082.txt
0f0ef340b0c199d3905bee58a0bbd265
d2d6e63dfff3149186e622ed9125d2dffa30296a
2079 F20101111_AABFWX allen_m_Page_067.txt
2291b4c7069612ab9bb7e816ac3500bf
2fde34454ff228cd34461f71f9090afba96a9a05
27329 F20101111_AABGDG allen_m_Page_069.QC.jpg
a06e057d923630fbf407630ab59d5651
10051a41739c6d02a57b1ce97a60bf395b0a3652
2067 F20101111_AABFYB allen_m_Page_097.txt
073083b1d4527a5548b2d777fb5b4192
23c99871117d2fe9779a1fc627bfc589c8eaa350
F20101111_AABGCT allen_m_Page_033thm.jpg
8e2a12f621ec28ed3146fce1fb159671
acda6ecc0e3e89b06732293466814c33beb0a95b
F20101111_AABFXN allen_m_Page_083.txt
101e0735d29c9a649f20e715ed055729
d2bfe5340e63d89130dae1b65c7d7e5d99fde878
2235 F20101111_AABFWY allen_m_Page_068.txt
b21746c80ed56746e983352a7d932ba8
eafbd53ca429c10029570d735d8105fd508fde1a
28668 F20101111_AABGDH allen_m_Page_074.QC.jpg
7ea6245f775312b96a482c76249865d2
30a147c0825c3a49e6c7ac861331d220cd7daa31
651 F20101111_AABFYC allen_m_Page_098.txt
ca265e1ee100d2abf582dc6288485989
8e7d38c8a8dd404f01c87058409d6b9a03b1405f
1030 F20101111_AABGCU allen_m_Page_002.QC.jpg
2f08f986d8d2aed7ccd2d2c499de6ceb
dda5d1da06bf9983480bb67171b75e595fd38131
2094 F20101111_AABFXO allen_m_Page_084.txt
4743c1e19c315bf5b2c7915a82a3fb23
b78a8fd3461daed86adac493b8e27f0d2d25e3d7
F20101111_AABFWZ allen_m_Page_069.txt
800ed35d5d608fcdb4c14455650cb224
52e30c1dc4ca2a35abb44081d04494babe08cdb6
17499 F20101111_AABGDI allen_m_Page_081.QC.jpg
38848a0b06f4746588fe4ef09093c59a
aad7ad5063afc6d9090169b0e4165483ab4ee8bb
2831 F20101111_AABFYD allen_m_Page_099.txt
eda5fdb49279680cd8bb1c56250c1f4b
cd58b86206a1294df00284147098ec109b6e3d75
29510 F20101111_AABGCV allen_m_Page_093.QC.jpg
167f27f9f9559ff73681c21f37855b13
cf7ac30438e14880db4498c21c3cd446c859ee97
F20101111_AABFXP allen_m_Page_085.txt
8561559fab86c3d46fd559fff436a6ae
85c6829b8150ac3cf1d3b1b8992feda930b7f40a
6397 F20101111_AABGDJ allen_m_Page_053thm.jpg
500ee2ecbae4e46ed5e349ffc834bd44
961000f282393464650275cad510e1ec1b7b1c0c
1253 F20101111_AABFYE allen_m_Page_100.txt
6b4ea193990da7bd3714b29c8178222a
81bd24d780983720ffafb0b9ac83cbbe954c43e8
6798 F20101111_AABGCW allen_m_Page_051thm.jpg
7198e64f33e34db00f4f2e17719bb3d2
a726a0bab475f2d9db5226bf6df81120ce51f988
2119 F20101111_AABFXQ allen_m_Page_086.txt
35a3043527c2254f0277f30accdc06d9
7b994864a450deac7b8ef296e5794d8bac221716
6669 F20101111_AABGDK allen_m_Page_075thm.jpg
98c033a65626e416315c273120f4d433
fa0ef9b7dcb17a57bf8135fdd57395dc9c3a6564
2345 F20101111_AABFYF allen_m_Page_101.txt
31d0d9434520156d4f36ee19f7c3c3ec
12947a214756804ca21ec7ac9cfe000a3187aba4
4841 F20101111_AABGCX allen_m_Page_099thm.jpg
868cc3a5023884307b27fa287d7c0c60
f7d9e3dc69892c02c86b1e32195faaf35087fdbe
2118 F20101111_AABFXR allen_m_Page_087.txt
bfa99db3ddba8d7d9cf45f17696f5231
d99c77002778ae7efb996cf17ceb2ebadb408bd3
27703 F20101111_AABGEA allen_m_Page_059.QC.jpg
284383dee5d15f4eb36ed0c8bee2fbd5
1b2cb485c73c0ae453cabe89179655fa705e434a
6338 F20101111_AABGDL allen_m_Page_045thm.jpg
efe36de70b0dbd640c2cdb703412e0e7
37d2f8610f88e35fc22be330cb7d018518eac3f0
1563 F20101111_AABFYG allen_m_Page_102.txt
0ac7a45fe29f1b8af48109653235c256
2a2bf5a0a2fd544d9311343c8e7170e3c44e1dd0
11119 F20101111_AABGCY allen_m_Page_079.QC.jpg
f96ec04a548a59b819536cddb59c214d
9892c7424912e8ec47403994160ca7367157dc34
F20101111_AABFXS allen_m_Page_088.txt
d2655966b4c90e73445b4287e1e8c272
172817d2333d6f72d3199ee1538523e6dc98a6a5
25914 F20101111_AABGEB allen_m_Page_068.QC.jpg
ac8f7fd5a67324e1031a04e18667d1eb
531444060dee94629e65625fcc695176c877a2e0
27199 F20101111_AABGDM allen_m_Page_053.QC.jpg
5c049ebe30e6776e6240bdd21348b20f
6d79e24aede9e2b64f2cdf4381fdd3dde18671df
2181 F20101111_AABFYH allen_m_Page_103.txt
843559a7b661105c8ca05aaaded7db0d
51fa4f75ab5d467db494f30569b9ff5046fc3637
6928 F20101111_AABGCZ allen_m_Page_096thm.jpg
5c1a6eab092c08de390382a1d5788f29
9ba48e049b766faf55a3895377d2c5ebabd6bfed
2051 F20101111_AABFXT allen_m_Page_089.txt
a81521961ef5bab1ac91ec6864df3f91
89078f592c8e5718a48213e342061204af804efc
F20101111_AABFBA allen_m_Page_065.tif
9cce4e928a35addc159836a4513343ff
dbb45ff9438c4d176625ce0fdfbe4b36bc8732fc
176568 F20101111_AABGEC UFE0021969_00001.xml
22851ba978e19801104d9002d0bf1379
d6ded496f8fc15b8bcd27001980306082d840f2a
18055 F20101111_AABGDN allen_m_Page_099.QC.jpg
9e18ec4caf7af48f5dc72043f3e2dab2
014d3a818bf7154b7a62bb9e0e1b41c4b7626f85
640 F20101111_AABFYI allen_m_Page_104.txt
11468821bab5d28ba7f64ee7935e99a1
c76d437f0e4893a908d39168e8fccb53c41cea30
2097 F20101111_AABFXU allen_m_Page_090.txt
4d558393a9236d53268209f3434c78f2
0547445c3adb32dd51c265cadf279cf38f4ef437
27966 F20101111_AABGDO allen_m_Page_057.QC.jpg
df1fb447ee2defb3fbd58b6b628cfbf8
7d37aa664b0d5b690d1ad82e41960f8d3a3a3d35
2124 F20101111_AABFYJ allen_m_Page_105.txt
0355d821788835294824cb4dcadaf979
dcd37b3977aafee60c27d9532ee8b087d25fd4dc
2149 F20101111_AABFXV allen_m_Page_091.txt
26ad54a0b6b41bf35cffee0babac5688
d3788d0d246d82dfb30b9bac974a1446e5ff04ec
6685 F20101111_AABFBB allen_m_Page_035thm.jpg
dcd5a69e23ce1ecd435f697fda68cc57
1a935c77c46ac85d0bbc2783975ffbb2d10ecf73
6419 F20101111_AABGED allen_m_Page_004thm.jpg
765fcf06649d249a8ad6fd0d6854e745
08dd8a7b1079882b0aeb9cd919a37daba2cac2b0
6879 F20101111_AABGDP allen_m_Page_032thm.jpg
639b67dc300b760a39f1bf3aa177594e
8cb01a1b51c3e5a5b07b0be6e33c2fb0ed70aa6a
708 F20101111_AABFYK allen_m_Page_106.txt
6a42fd34773d3f00858bccafab5ba551
272f36e7b7c9dace77ba5555a4484f7b3f86f5a7
2239 F20101111_AABFXW allen_m_Page_092.txt
466a1320dca1c86107adea718733f822
669cf5ac37394a1f093176896dcaa7151369e394
26461 F20101111_AABFBC allen_m_Page_114.QC.jpg
b9959825b099938101f784df4cb6eb22
17407d630c147026f9d5e6846af8767941343fa2
20270 F20101111_AABGEE allen_m_Page_006.QC.jpg
f6562014ed7a807376d9819cdbf422aa
25e2389a1e0ae3295aac20e6a04ae1bde07a9854
27302 F20101111_AABGDQ allen_m_Page_086.QC.jpg
cd5f9fa9e2592de45126fe9217fd882e
aba53642fe258f46caa5fe16758b9baefdb765ef