A meta-evaluation of esea Title VII bilingual education project evaluations in the state of Florida, fiscal year 1984-85

MISSING IMAGE

Material Information

Title:
A meta-evaluation of esea Title VII bilingual education project evaluations in the state of Florida, fiscal year 1984-85
Physical Description:
viii, 144 leaves : ill. ; 28 cm.
Language:
English
Creator:
Fry, Ella Bessie Staes
Publication Date:

Subjects

Subjects / Keywords:
Education, Bilingual -- Evaluation -- Florida   ( lcsh )
Curriculum and Instruction thesis Ph. D
Dissertations, Academic -- Curriculum and Instruction -- UF
Genre:
bibliography   ( marcgt )
non-fiction   ( marcgt )

Notes

Thesis:
Thesis (Ph. D.)--University of Florida, 1986.
Bibliography:
Bibliography: leaves 133-142.
Statement of Responsibility:
by Ella Bessie Staes Fry.
General Note:
Typescript.
General Note:
Vita.

Record Information

Source Institution:
University of Florida
Rights Management:
All applicable rights reserved by the source institution and holding location.
Resource Identifier:
aleph - 000924332
oclc - 15911164
notis - AEN4949
System ID:
AA00002158:00001

Full Text


















A META-EVALUATION


OF ESEA


TITLE


VII BILINGUAL


EDUCATION


PROJECT


EVALUATIONS


IN THE STATE


OF FLORIDA,


FISCAL


ELLA


YEAR


BESSIE


1984-85


STATES


A DISSERTATION


PRESENTED


TO THE


GRADUATE


SCHOOL


OF THE UNIVERSITY


FULFILLMENT OF


OF FLORIDA


THE REQUIREMENTS


IN PARTIAL


FOR THE DEGREE


DOCTOR


OF PHILOSOPHY


UNIVERSITY


OF FLORIDA


1986



















This dissertation is dedicated to

the loving memory of


my father,


Sidney Matthew Staes


, 1889-1978,


who always wanted me to be a teacher;

and


my husband, Norman James Fry


who shared with me his


Ph.D


love of the


1940-1978;


langua


and cultures of the world;

and to

the celebration of life with


my mother,


Ella Be


ssie Staes


who always believed I should be well educated and
prayed that she would live to see me reach my goal;


my son "Matt,"


James Matthew Staes Fry,


who has spent his
patiently making all


childhood on campus
the sacrifices necessary


for me to succeed.


















ACKNOWLEDGMENTS


I am deeply grateful

School, and Dr. Michael Y

College of Education, for


to Dr.


. Nunner


Madelyn Lockhart, Dean of the Graduate

y, Assistant Dean of Graduate Studies,


their unwavering belief in me.


Without


their support and the support of my committee


closure on this research project.


I might not have reached


would like to express my


appreciation to Dr.


Clemens L.


Hallman,


chairperson of my committee,


who wrote the original


Title VII Teacher


Trainer Fellowship proposal


which brought me to the College of Education and supported my studies


in bilingual-multicultural


education.


also wish to express my


appreciation to Dr.


Sandra B.


Damico,


who taught me qualitative research


skills and whose encouragement and assistance has made the completion


of this work possible.


My gratitude also goes to Dr.


Robert Wright,


who cared,


and because he cared he taught me not to bleed when I swim


with sharks.


I am also most grateful


to Dr.


Phillip A.


Clark,


challenged the students in his education leadership class to

seek a position of leadership in a professional organization.


sponsored my nomination to Phi Delta Kappa,


of serving in many capacities,


join and

He


where I had the privilege


including chapter president and delegate.












am indebted


to Dr. John


R. Stiefel


has taught


me to be


confident


and without


whose


support


encouragement


might


have


finished.


gratitude


goes


to Ms


Leila


Cantara,


typist


nurturer,


whose


support


assistance


was invaluable.


Many


loving


friends


provided


valued


assistance


at the


various


stages


in the


research


and provided


daily


doses


of friendship


and encouragement.


special


appreciation


goes


to Phillip


Simmons,


has been


there


when


needed


most,


and who spent


four weeks


this


summer


entertaining my


son and his


surfboard while


wrote.


Ms. Susan


Nuttall


Ms. Chari


Campbell


and Mrs


Mary


Ellen


Page


have


given


invaluable

keeping me


assistance


in caffeine


in revising

. but most


the original

importantly,


draft,


being


proofreading,


friends.


family


has made


many personal


sacrifices


during


my graduate


career.


son,


Matt,


and my


mother,


Ella


Staes,


have


patiently


waited


for me


to finish.


Thank


you.






















TABLE


ACKNOWLEDGMENTS


OF CONTENTS




S S S S S S S S S S S S S S S S S


ABSTRACT


CHAPTERS


INTRODUCTION


1


Statement of
Significance
Scope of the
Limitations


Definition
Overview o


REVIEW


Purpose .
of the Study
Study


of Terms
Chapters


OF RELATED


* S S S S S S a 5 5 5
* S S S S S S S S S S S S S
* S S S S S S S S S S S


S S S* S S 0 .S 7


and V


LITERATURE


S . 5 1


S S S S S S . 13


Historic


Historical
Historical


Sele


action


Standards,


Context
Context:
Context:


for Evaluation
Evaluation a


Bilingual


s a Profe


Meta-Evaluation


of Meta-Evaluation


Guidelines


Meta-Evaluation


Instruments,


Criteria


Pitfalls


Techniques


Education


ssion


. 0 0 5 17
.. .19
* S S S S 21
. . . 2


and Devices


23


Discrepancy
Application
Education


Analysis


Techniques


of Meta-Evaluation


Theory


to Bilingual


Evaluation


Research


Versus


Program


Evaluation


. . 35


METHODOLOGY


Conceptualization
Sample


Data


Collection


of Meta


-Evaluation


.


Procedures .
procedures *


* S S S S S S S
* S S S a S S S


Instrumentation


. . 43


46


&n I vC1c c


n P


46


n'a-f-













Program Characteristics


Process
Summary
Question 2


Variables
of Results
Evaluatio


for Question 1
n Models, Designs


and Reports


Evaluation


ReDorts


Evaluation Models


Evaluati
Summary


stion


on Designs
of Results


for Qu


Professional


estion


Standards


for Evaluation


Utility


Feasibilit
Propriety
Accuracy S
Summary of


Standards


Standards


Standards
standards
Results


* 0 S *
* S S S


for Ques


tion


Question 4:
Proposal


Discrepancies


Evaluation


Between


Proposal


and Report


Plans


Discrepancies


Between


Plans


and Reports


Summary
Conclusions


of Results


for Ques


tion


4 .


SUMMARY


, DISCUSSION,


IMPLICATIONS


RECOMMENDATI ONS


the Study


Rationale
Purpose


Summary


SCUSSi


of Methodology
on of Findings


Question
Question
Question
Question
Implications
Implications


Recommendations
Recommendations


1 .


for the
for the


.* . S S S S S S S
* S S S S S S S S S *
.* S S S S S U S S S S S *
* . S S S S S S S S S S
* S S S S S S S S S S S S S S
* S S S S S S S S S S S S S S S


Field of Evaluation
Field of Bilingual
. S S S S S S S


for Further


Education


earch


APPENDICES


STANDARDS


FOR EVALUATIONS


OF EDUCATIONAL


PROGRAMS


INSTRUMENTS


REFERENCES


BIOGRAPHICAL SKETCH


* S S S S S S




* S S S S S fe


* S S S S S a * S

* S SS S S S S S S U


Overview



















Abstract


of Dissertation


Presented


to the Graduate


School


of the University
the Requirements


of Florida


for the


in Partial


Degree


of Do


Fulfillment


ctor


of Philo


sophy


A META-EVALUATION OF


ESEA


TITLE


VII BILINGUAL


EDUCATION


PROJECT EVALUATIONS


FISCAL


Ella


IN THE STATE


YEAR


Bessi


OF FLORIDA,


1984-


Staes


December,


1986


Chairman:


Maj or


Clemens


Department:


L. Hallman
Instruction


and Curriculum


The meta-evaluator


investigated


the current


status


of evaluation


in the field


of ESEA


Title


VII funded


bilingual


education


proj ects


in the State


of Florida


for fiscal


year


1984


-85.


A total


12 final


evaluation


reports


and 11 corresponding


application


proposal


evaluation


plans


were


analyzed.


The meta-evaluation


addressed


the quality


the final


evaluation


reports


not the quality


of the projects.


Four


meta-evaluation


instruments


were


employed


in the data


gathering


stage.


reports


were


analyzed


in relation


to the 30 Joint


Committee


Standards


for Educational


Evaluation,


while


the plans


were


analyzed


relation


to the U.S. Department


of Education


regulations


for fiscal


___














were not


adequately


addressed


in the sample


reports.


In addition,


procedures


could


not be described


as motivated


a thorough


understanding


application


a consistent


model


conceptualization.


The evaluation


design,


which


provided


the plan


and organization


the collection


of evaluation


data,


was the weakest


area


found


the meta-evaluation.


No confidence


can be


placed


the reported


results


of the assessment


the Joint


Committee's


of achievement.


30 professional


The sample


standards


did not


for good


meet


evaluation


practices

proposals


and only

provided


59.4%

more


of the applicable


descriptive


federal


information


regulations.


on contextual


, program,


process


variables


and included


the proposed budget,


were


weakest in formulating

comparison procedures


evaluation

to estimate


questions


what


evaluation


the performance o


designs,

f participants


would have


been


in the absence


of the project,


data


analysis


procedures,


and data


collection


methods.


These


same


areas


were


also


found


to be


inadequate


in the final


evaluation


reports.


is recommended


that


Bilingual


Education Act


(Title


VII of the Elementary


and Secondary


Education Act

administrators


as amended

and propos


in 1968) 1

al writing


ocal

team


education

s obtain


agency

technic


project


al assistance


from a


qualified


evaluator


in writing


the application


proposal


plan.


It is also


recommended


that


the writers


include


specific


plans


implementation


evaluation


and evaluation


utilization


in the proposals

















CHAPTER I
INTRODUCTION



What is the current status of evaluation in the field of


bilingual education?


Is it healthy and advancing methodologically


or is bilingual education evaluation biased?


According to Keith


Baker


U.S.


Department of Education,


Office of Planning,


Budget,


and Evaluation


(OPBE),


"the cannons of science have been sacrificed


on a wide scale to ideological needs and political


expediency in


bilingual research


.'proof'


of the effectiveness of bilingual


instruction that vanishes into a cloud of methodological


chicanery"


(Baker,


1984,


Secretary of Education,


William J.


Bennett,


a September


1985,


speech before the Association For A Better New


York,


called bilingual education itself a failure.


"After


years


of federal involvement and after $1.7 billions of federal funds, we

have no evidence that the children whom we sought to help--that the


children who deserve our help have benefited"


(Bennett,


1985,


2C).


The evidence Bennett and a host of other critics across the years

have been looking for should have come from nearly two decades of

bilingual education evaluations and basic research in related fields.


Can this


lack of evidence be due to the quality of programs or the













Statement


of Purpose


The

Florida


purpose


Title


of this

(i.e.,


study was

the 1968


to conduct

amendment t


a meta-evaluation


o the Elementary


Secondary

bilingual


Education

education


Act of 1965)


proj ect


local


evaluations


education

. fiscal


agency


year


(LEA)


1984-85.


Meta-evaluation


assesses


the merit


and worth


of the final


evaluation


report.


focus


of this study was


an examination


and assessment


the importance of


evaluation


objectives,


the appropriateness


evaluation models


and designs


the adequacy


of implementation


of the


designs,


the technical


adequacy


of data


analvsi


and the quality


importance


of evaluation


results.


The researcher performed


an indepth


analysis


using


criteria


related


to evaluation


model


characteristics,


evaluation


sign


characteristics


data


collection


procedures,


testing


instruments,


data


analysis


techniques,


and general


evaluation


methodologies,


as well


as discrepancies


between


proposed


evaluation


plans


and summative


evaluation


reports


in order


answer


the following


four


questions:


Which


context


process


variables


are addressed


Title


VII project


summat ive


evaluation


reports


within


the State


of Florida?


What


are the characteristics


of Florida


Title


VII evaluation


models,


designs,


reports


terms


of information


coverage,


content,


and procedures













there


major


discrepancies


between


LEA evaluation


proposal s


and summative


evaluation


reports


What


is the nature


these


discrepancies?


Significance


of the Study


If evaluations


evaluations


considerations,


questions,


are to provide


themselves


must


they must


accurate


proper
e sound.
focused


in their portray


guidance


Among


other


on the right
Is, free from


bias,


understandable


, and fair


to those whose


work


is under


examination


(Stuffl ebeam


Shinkfield


1985


183)


This


meta-evaluation


of Title


VII of the Elementary


and Secondary


Education


Act (ESEA)


the knowledge


base


proposals


of bilingual


and final e

education.


evaluation reports

Tallmadge, Lam,


adds

and


Camarena


(1985a)


warned


that


all relevant


chara


cteristics


students,


settings


, and


treatments


part
do s


must


of any bilingu


o would


relationships


pooled
and/or


across


run the


would


carefully do
al education
risk that e


be obscur


different


settings.


types


cumented as
program. F
educationally


whenever
students


an integral
failure to
significant


data


were


treatments,


xii)


Tallmadge


et al. questioned


the effectiveness


of the variations


educational


treatments


which


under


name


of bilingual


education


for the


various


types


students


in the multiplicity


settings.


Evaluators


cannot


answer


that


question


until


they


have


documented


the range of


treatments,


students,


settings.


Instruments


used


in this


meta-evaluation


research


proj ect


provided


for the













implemented rather than treatment


as intended in the proposal.


Tallmadge et al.


(1985a)


indicated that "the actual


treatment,


unfortunately, may bear


little resemblance to what was intended


and may


consequently,


have a very


low construct validity relative


to what the study set out to evaluate"


xiii).


This meta-evaluation also


serves


as an aide to designers of


future bilingual programs and the planners of any national data


reporting systems.


Fuentes'(1986)


contended that "a need thus


exists for empirically grounded information that can guide planners


in designing the most effective programs.


For example,


educators


need information on the effectiveness of ongoing practices and on the


need for improvement of practices"1


This meta-evaluation


research satisfies Fuentes


(1986)


requirement and fits into Sanders'


(1981)


first stage for the theoretical design of a national system


for monitoring federally-funded bilingual programs.


Sanders'


(1981)


conceptual analysis stage is composed of case study data and survey

data from Title VII projects "to discover the extent to which certain

project characteristics exist and to identify further distinctions


that must be incorporated into the reporting system"''


two is the pilot stage.


tested,


Stage


Once the pilot reporting program is field


then stage three involves a full-scale implementation of the


design and continuous monitoring of operation for quality control.













students


would


provide


for the United


States


Department


of Education,


Office


of Bilingual


Education


and Minority


Language


Affairs


(OBEMLA)


a much


more


accurate


regional


and national


picture


of the


current


status


of LEA evaluation


practices


than


is currently


available


from


past


studies


and would


contribute


greatly


to OBEMLA's


current


efforts


at systematizing


bilingual


education


program


evaluation.


As further


indication


of the willingness


of the federal


government


improve


the quality


of bilingual


education


evaluations,


the 1984


bilingual


reauthorization


bill


provided


for the establishment


two bilingual


education


evaluation


assistance


centers.


At the


present


time,


only


one center


is currently


operation,


located


at Georgetown


University.


Dr. Michael


O'Malley,


Director


of the Title


Evaluation A


assistance


Center,


funded


OBEMLA,


expressed


great


interest


in this


meta-evaluation


of Florida


Title


VII project


evaluations


as the results,


in his opinion,


would


provide


timely


contextual


information


an evaluation


system


useful


in the designing


for bilingual


education


and implementation


O'Malley


personal


communication,


1986).


The results


of this


meta-evaluation


study


provide


a description


current e

the Miami


Evaluation

Bilingual


practices

Education


within

South


Florida,


East


data


Service


deemed

Center


valuable

(BESES),


the bilingual


consultants


in the Florida


Department


of Education,













Scope of


Study


This


study was


confined


to federally


funded


Title


bilingual


project


proposals


and summative evaluations


within


State


of Florida.


Locally


funded bilingual


proj ects


are not required


to evaluate


effectiveness


due to the absence


of bilingual


education


legislation


within


the State


of Florida.


The study was


further


confined


to the fiscal


year


1984-85,


since


final


evaluation


reports


were


not required


to be


submitted


to OBEMLA


until


the December


following


the close of


each


grant


(fiscal)


year


, making


1984-85


mos t


recent


available


data.


Limitations


Although


final


evaluation


reports


come


under


the "government


the sunshine"


regulations


within


the State


of Florida,


making


them


theoretically


accessible


to the public,


the documents


used


in this


study were


limited


to those


proposals


and final


evaluation


reports


voluntarily


Title


submitted by


VII projects


districts


in the fiscal


which


year


received


1984-85.


federal


Also,


funds


Florida does


have


legi


slation


which


authorizes


or funds


bilingual


education


programs.


Limit nations


must


be placed


on the generalizability


of the reported


results


to other


states,


especially


states


which


do have


bilingual


education


legislation.


The evaluator,


unlike


the researcher in


or her laboratory,


must













degree


of implementation


must


be placed


on the generalizability


of the


reported


results.


Definition


of Terms


Definitions


of the following


terms


are presented


in order


clarify


the meaning


of the study


and to avoid


needless


ambiguity.


Contextually relevant


terms


are presented


together


rather


than


alphabetical


order.


Where


applicable,


the author


or source


for the


definition


is identified.


Bilingual


education.


Bilingual


education


provides


special


programs


for non-


and limited-English


proficient


language


minority


students


which


have


the dual


objectives


of developing


proficiency


English


language


skills


while


preventing


the student


from


falling


behind


English


proficient


peers


in content


areas


(Tallmadge,


Lam, &


Camarena,


1985b,


vii).


Title


VII.


The 1968


Bilingual


Education


Act authorized


bilingual


education


amending


Title


VII of the Elementary


and Secondary


Education


Act of 1965.


Title


VII,


therefore,


is used


to refer


to bilingual


education


projects.


LEP.


by predetermine


Limited

d cutoff


English

score


Proficient (

on a language


LEP),


is a category


assessment


test.


defined

This


category may


also


include


non-English-speaker


(NES)


and limited-English


speaker


(LES).


S, -


1 ^-


I


1 1


1 1. ~n


I


rrn (1













where


impact
subject


a language


on their
to such


other
level


than


English


English


regulations


has had


language


a significant


proficiency,


as the Commissioner


determines


to be
diffi


necessary;


culty


the English
opportunity


language


, by reason


speaking,
language
to learn


reading,
to deny


succ


instruction


thereof


writing


such


essfully i
s English.


have


sufficient


or understanding


individuals


classrooms


(United


where


States


Department


of Edu


cation,


1983,


ESL.


English


as a Second


Language


(ESL)


a component


of the


bilingual


program


that


teaches


English


speakers


of other


languages


in an


intensified manner


, taking


into


consideration


the challenges


each


student


within


context


of their


primary


language


and their


English


language


skills


(Martin,


1981,


categories.


system


cate


gorizing


students


language


proficiency


, the Lau categories


grew


out of the Lau v. Nichols


Supreme


Court


Decis


and subsequent


Lau Remedies.


The Lau categories


exclusively monolingual


speaker


a language


other than


English,


predominantly


speaker


a language


other


than


English


although


speaks


some


English,


equally


speaker


a language


other


than


English


and English,


predominantly


speaker


of English


but speaks


some


the other


language


as well


and (E)


exclusively monolingual


speaker of


English.


Students


are classified


Lau-A


and Lau-B


are considered


be limited


English


proficient


(LEP).


LEA.


A local


education


agency


(LEA)


a school


district.


OBEMLA.


OBEMLA


is an acronym


for the Office


of Bilingual


are













Evaluation.


Evaluation


is the assessment


of worth


and merit


(Lincoln


Guba,


1984)


a process


of delineating,


obtaining,


providing


useful


information


judging


decision


alternatives


(Stufflebeam,


determine

evaluand,


1978)


the value


such


"a type


(merit


as a treatme


of di


and/or worth)

nt, program,


sciplined


some


facility


inquiry


undertaken


entity

performance,


the


and the


like


in order


to improve


or refine


the evaluanl


(formative


evaluation)


or to


assess


its impact


summativee


evaluation)"


(Lincoln


Guba,


1984,


p. 9).


Meta-evaluation.


Meta-evaluation


is defined


as the evaluation


of evaluations


an evaluation,


(Scriven,


1969)


including


the assessment


all levels


of the worth


of evaluation


above


and merit

the primary


evaluation


(Stufflebeam,


1978);


the pro


cess


of delineating,


obtaining,


and using des

practicality,


;criptive


ethics


and judgmental

and technical


information


adequac


about


the utility,


an evaluation


in order


to guide


public


evaluation


its strengths


(formative


and weaknesses


meta-evaluation)


summativee


and to report


meta-evaluation)


(Stufflebeam,


1981a).


Primary


evaluation.


primary


evaluation


is the


assessment


of worth


merit


some


entity, such


as an ESEA


Title


education


project,


and is an evaluation


that


is the subject


meta-evaluation


(Stufflebeam,


1974a).














Summative


meta-evaluation.


Summative meta-evaluation


a study


that


judges


the worth


merit


of completed


evaluations


Stufflebeam


, 1981a)


such


as the meta-evaluation


of the sample


12 Florida


ESEA


Title


VII bilingual


education


final


evaluation


reports


for the fiscal


year


1984-85.


Standards.


Widely


shared


principles


for the measure


worth


and merit


an evaluation


defines


the word


standards.


Guidelines.


Guidelines


are procedural


suggestions


intended


to help


evaluators


meet


evaluation


standards


(Stufflebeam,


1981a).


Pitfalls.


Mistakes


that


are commonly


made


by persons


are unaware


of the importance


a given


principle of


sound


evaluation


are pitfalls


(Stuffl ebeam,


1981a).


QUEMAC.


QUEMAC is


a series


questions


which


when


answered,


reveals


the logical


structure


of evaluation


studies


(question,


event


or object,


method,


answer,


concept)


(Gowin


Millman,


1978).


Discrepancy


Evaluation Model.


The Discrepancy


Evaluation


Model


(DEM)


, developed


Provus


(1972)


is a comparison


between


present


state


of affairs


with


"what


should be;"


a matter


of making


judgments


discrepancy


about


the worth


between


standard


or adequacy


an object


and performance


measure


based


upon


(Brannon,


1985,


standard.


standard


a list


, description


, or


representation


qualities


or characteristics


the object


should













Impact


evaluation.


Impact


evaluation,


typically


quantitative


in approach,


is used


accomplishments


assess


the effect


a social

s of the


program

program


s overall


performance,


on participant


outcome.


Impact


evaluation


asks


whether


program


is a success


or failure.


As used


the General


Accounting


Office


(GAO)


, impact


evaluation


provides


Congress


with


an assessment


of the performance of


a federally-


funded


social


program,


whether


a social


program


is achieving


legislative


ective,


and how the


program


is affecting


the intended


beneficiaries.


Research.


For the


purposes


of this


study,


research


is defined


as a type


of disciplined


inquiry undertaken


to resolve


some


problem


order


to achieve


understanding


or to facilitate


action--problem


resolution


or amelioration


(Lincoln


Guba,


1984).


Overview


of Chapters


III, IV,


and V


An introduction


to the meta-evaluation


was provided


in Chapter


purpose,


significance,


scope,


and limitations


of the study were


identified and


relevant


terms


were


defined.


Literature


related


the historical


projects,


context


evaluation


for the evaluation


as a profession,


of bilingual


and meta-evaluati


education

on is reviewed


in Chapter


along


with


applicable


literature


on the selection


meta-evaluation


criteria


appropriate


standards
*


the application


meta-evaluation


theory


to the


field of bilingual


education


, and













and summarized


in the order


of the four main


research


questions.


In Chapter


findings


the final


are discussed,


chapter


of the report,


implications


are


the meta-evaluation


identified,


and recommendations


are offered.


















CHAPTER


REVIEW


OF RELATED


LITERATURE


This


review


of related


literature


places


the fields


of bilingual


education


, evaluation,


and the development


of meta-evaluation


historical

legislative


context.

history


Brief


introductory


of bilingual


educati


segments i

on and the


introduce t

emergence


evaluation


as a profession,


followed


a discussion


of the development


of the theory


of meta-evaluation.


A brief


discussion


of studies


which


have


applied


meta-evaluation methodology


to bilingual


education


program


evaluation


follows.


Questions


of the appropriateness


of using


local


program


evaluation


data


for the


purpose


reporting


research


on the effectiveness


of Title


VII bilingual


education


programs


Congress


are discussed.


The chapter


concludes


with


a discussion


of the distinction


between


evaluation


research


program


evaluation.


Historical


Context


for Evaluation


Bilingual


Education


The federal


government


first


became


involved


with


the problems


of non-native


Rights


speakers


Act of 1964.


Education Act of


1968


of English


This


with


was closely


otherwise


known


the passage of


followed


as the Title


the Civil


he Bilingual

VII amendment













supplemental


funding


for school


districts


to establish


programs


meet


the special


educational


needs


of large


numbers


of children


limited

United


English

States.


speaking

Neither


ability

the Civil


from


Rights


income


families


nor the ESEA


in the

Title


amendment


specified what


actions


needed


to be taken


to assure


language


minority


students


equal


educational


opportunities.


January


of 1974


the U


.S. Supreme Court


affirmed


that


school


districts


would

1964


be compelled


to provide


under


special


the Titl

language


VI of the Civil


programs


Rights


for children who


Act of

speak


little


or no English


to provide


them with


an equal


educational


opportunity.


It was


this


supreme


court


decision


which


spurred most


state


and local


educational


agencies


to design


and implement


bilingual


educational


programs.


Since


the Bilingual


Education Act


of 1968


not specifically


require


program


evaluations,


little was


known


about


success


programs


or student


achievement


for nearly


years.


Development


Associates


(1973),


under


contract


to the U


Office of Education,


submitted


the first


part


a process


evaluation


of Title


programs


in December of


1973.


emphasis


of that


evaluation was


on the


extent


of adherence

relationship


to guidelines

between such


individual


adherence


Title

success


VII project

based on


and the


subjective


ratings


evaluation


team


leaders.


The question


of assessment


- rr .. - - S


m -


(


__ ___ _Y__













Bilingual


Education Act


of 1974


removed


criterion


income


and added


, among


other


things


the requirement


that


U.S.


Commissioner


of Education


and the National


Advisory


Council


for Bilingual


Education


would


report


to the U.S. Congress


on the


state


of education


in the nation,


including


an evaluation


of Title


activities.


According


to the United


States


Commission


on Civil


Rights


(197


nature


of evaluation


was still


not clear


support


was limited


for the overall


program.


was not until


1976


that


specific


"Title


VII Regulations"


program


evaluations


were


published


in the Federal


Register.


Tallmadge,


Lam,


Camarena


(1985a)


commented


that


guidance


provided


for completing


the required


evaluation


was minimal.


They


posited


that


three fa

priority


ctors--lack


of evaluation


and low funding


levels


expertise at

and technical


the local le

difficulties


v


el, low

inherent


in conducting


not surprising

The U.S.


bilingual


outcome of

Department


program


evalautions--"combined


basically


useless


of Education,


Office


to produce


viii).


of Bilingual


Education


and Minority


Language


Affairs


(OBEMLA)


, is currently


in the


process


exploring


ways


to improve


the quality


of bilingual


education


evaluations.


A three-year


contract


was granted


to SRA Technologies


in 1985


develop

that wo


and field


uld


test


incorporate


an evaluation system

methdologically sound


for bilingual


designs


education


and procedures.


1 A- t .3 r .. 2 p n a


- 1


I I


data"


- -


I)


I


1













Historical


Context:


Evaluation


as a Profession


The 1964


federal


mandate


for evaluation of


ESEA


Title


federally


funded


compensatory


education


programs


caught


educators


and academicians


alike


unprepared.


College professors


from


several


disciplines


including


psychology


and education


responded


to the needs


of the local


meet

UCLA


school


new federal


Center


retreads


districts


evaluation


for the Study


from other


fields


evaluation


requirements.


of Evaluation,

, typically re


consultants


James


reminisced


:search


Ponham


"most


and measuremel


to help


them


(1981),

us were

nt, since


was thought


in those days


that


educational


evaluation


was


little


more


than


warmed


over


research


with


applied


focus"


The growing need


consultants


with


special


technical


training


in evaluation


methodology


gave


rise


to training


institutes


professional


organizations


which


provided much


needed


attention


the skills


of the field


the development


of theory.


the late


1970s


nascent


and early


1980s


profession


establishment

journals, and


with


evaluation had


universities


societies,


at lea


of evaluation


the development


developed


providing


into


a separate


degree programs


the publication

st two sets of


, the


of evaluation


professional


standards


by which


the profession


could


begin


self-regulation


improve


the quality


its procedures


and products.


When


asked


of 1980


to comment


on the


status


of educational


evaluation,


Nick


- r S. - -S* ---1--- A :n


-- w


n-_-_l,_ _


rr


LT.1l ^ A


1<













navigating


through
banks
shoals
our ba
current


each new turn


h the changing
of legislation
of special in
lance quickly,


ts


the ESEA


or merely


economic


trying


currents


and judicial


terest


groups


having been
legislation


control
We h


g to keep upright
the encroaching
1. and the


av


launched


e had to gain
in the swift


the mid-sixti


es.


(Smith,


1981c, p.


With t

profession,


:his


growing


theory


self-consciousness


and methodology


worked


emerging


together


into


to provi


an independent

de a way


assure


quality


evaluation


services,


to guard


against


or deal


with


malpractice


or services


not in the public


interest,


to provide


direction


understanding


for improvement


of the profession


of the evaluation


enterprise"


to promote


(Stufflebeam


increased


Shinkfield,


1985,


Historical


Context:


Meta-Evaluation


Although


Scriven


(1969)


is given


credit


for introducing


concept


of meta-evaluation and


defining


as "the


evaluation


evaluation,


published an


issue of


" it was


article


Journal


Pedro


entitled


. Orata


of Education


introduced


"Evaluating

1 Research,


Evaluation"

in which he


the concept


in the May,


critiqued


Orata


1940


Tyler


and Wrightstone'


new evaluation


practices


in the 30 schools


the Eight


Year


Study,


versus


traditional


testing


and measurement


practices


of the day.


Orata


recommended


"for


the evaluation


practices


to be evaluated


and subsequently modified


so as to provide,


in fact


as well


as theory,


new tests


for new needs"


653)


-~ .- V


1


I t T .I -


1.~





Ir~l













as "the methodological


assessment


of the role


of evaluation"


practically


as "concern with


the evaluation


of specific


evaluation


performances"


Scriven


(1976)


applied


the underlying


concept


in a brie


review


on the


assessment


a specific


educational


product--the


social


science


curricula.


Scriven


(1969)


meta-evaluation

nor did he work


was Daniel


not offer


or discuss th

out a logical


Stufflebeam who


an elaborate


e theoretic

structure


expanded


conceptualization


aspects


of explicit


the theory


of meta-evaluation,


methodology.


a detailed


model.


"Good


evaluation


requires


that


evaluation


efforts


themselves


be evaluated


. t is necessary


to check


evaluations


for problems


such


as bias,


technical


error


administrative


difficulties,


and misuse"


(Stufflebeam,


1974a,


"If all evaluations


are


potentially


faulty


, then,


theoretically,


each


one


requires


scrutiny


assistance


from meta-evaluation"


(Stufflebeam,


1981a,


153).


Stuffl ebeam


(1974a),


a 106-page monograph,


addressed


conceptual


and practical


development


of meta-evaluation.


Part


the monograph


detailed


a rationale-justification


for meta-evaluation,


suggested what


criteria


should be


used


to guide


the development


methodology,


and outlined


six classes


of problem areas


that


jeopardize


evaluation


which


need


to be addressed


by meta-evaluation


and presented


a logical


structure


for designing


meta-evaluation


studies.


Part


* ..


I *


1


I













Cook


and Gruder


(1978)


described


four


attempts


at the


employment


of meta- evaluation


theory


and methods


and developed a


classification


of seven meta-evaluation


research


models


for improvement


of technical


quality


and relevance


of empirical


summative


evaluations.


Cook


and Gruder


conceived


of meta-evaluation


as the


genus


and unifying


theory under which


three


species


of research


traditions


functioned:


the application


of formal


standards


a collection


of evaluation


studies


to judge


technical


adequacy;


the empirical


reevaluation


raw data


assess


the validity


for the purpose of


answering


original


research


question


with


better


statistical


techniques


, or


answering new


questions


with


data--secondary analysis


and research


on research.


Gene Glass


(Glass,


McGaw,


Smith,


1981)


added


additional


species


of research


tradition,


meta-analysis,


one approach


to research


integration which


Glass


defined


as the statistical


analysis


of the quantitative


summary


findings


many


individual


empirical


studies.


This


meta- evaluate ion


research


project


employed


Stufflebeam'


(1978)


fifth meta-evaluation


esign,


a retroactive assessment


evaluation


studies--their


goals,


designs


, implementation,


and results


combined


into


a simple


summary


case


study


(see


Figure


Selection


of Meta-Evaluation


Criteria


Stufflebeam


stated


that


"good


evaluation


requires


that


evaluation


one


.














META-EVALUATION


Generic


term--Cook


Gruder


(1978)


META-


EVALUATION

I
The application
of formal


standards
collection
evaluation


studies


to a


judge
technical
adequacy


(Cook
Gruder


, 1978).


SECONDARY
ANALYSIS


The empirical
re-evaluation


raw data


assess


validity


purpose


answering
original
research
question
better


with


J
META-


ANALYSIS


The statistical


analysis


of the


quantitative
summary
findings of
many
individual


empirical
studies


(Glass
1981).


I
RESEARCH O0
RESEARCH

I
Empirical
research o
completed
evaluation


research
(Cook &
Gruder,
1978).


et al.,


statistical


techniques
answering
questions


data


Gruder,


or
new
with


(Cook
1978).


Retroactive


goals,


assessment


designs,


of evaluation


implementation,


studies,


and results


their


combined


into


a simple


summary


case


study


(Stufflebeam,


1978)













major


concern


to the meta-evaluator


is the selection


appropriate


criteria a


for judging


the meta-evaluation.


Scriven


(1969)


proposed


technical


adequacy


Stufflebeam


(1978)


added


utility


cost


effectiveness.


Phi Delta


Kappa Study Committee


listed


11 specific


criteria


for judging


technical


adequacy,


utility,


cost


1983).


effectiveness


The four


criter:


(Stufflebeam, 1

ia of technical


974a;


Stufflebeam 6


adequacy


are (a)


Madau


internal


validity,


seven


external


standards


validity


of utility


reliability,


are (a)


relevance,


and (d)


objectivity.


importance,


scope,


credibility


timeliness


, (f)


pervasiveness


(dissemination


to intend


audiences)


and (g)


cost


effectiveness.


Standards,


Guidelines


and Pitfalls


The meta-evaluator must


deal


with


question


of which


guidelines


and pitfalls


are most


influential


meeting


given standards


of sound


evaluation.


Stufflebeam


defining


set of


standard:


dependent


evaluation


(1981a)

s one i


variables


studies.


explained


that,


.s essentially
regarding the


defining


guidelines


developing
outcomes o


and pitfalls


one is essentially


developing


a set of independent


variables
defined,


for
such


asses
sets


sing evaluation
of independent


operations.
and dependent


Once
variables


would


be a


hypotheses
meeting the


pitfalls,


good
about


resource


for deriving


how an evaluation'


procedural


influences


guidelines


the evaluation's


and testing
performance,


avoiding
overall


common


satisfaction


of the standards.


Such


guidelines


and standards


constitute


a basis


of meta-evaluation


for the
studies


actual


design


and conduct













17 members


to act as the Joint


Committee on


Standards


for Educational


Evaluation


(Stufflebeam,


Stufflebeam affirmed d


that


1981b).

"one c


As director of


characteristicc


proj ect,


a profession


is the


maintenance

(p. 40).


of high


The Joint


standards

Committee


for achievement


developed


and ethical


and published


conduct"


30 standards


good


practice


presented


in four


groups


that


correspond


to the four


main


concerns


about


evaluation--its


utility


feasibility,


propriety


and accuracy


1981).


(Joint


Together,


Committee


on Standards


the 30 standards


for Educational


are a working


philosophy


Evaluation,

of evaluation,


defining


principles


that


should


guide


govern


evaluation


efforts,


including meta-evaluation


efforts,


and offer practical


suggestions


for observing


these


principles.


Stufflebeam


(1981b)


recommended


use of the standards


for all stages


of evaluation


including


meta-


evaluation.


good


"Now


practice,


that


it will


the field ha

be important


s articulated

to ascertain


30 standards

the quality


evaluations


in relation


to each


of the standards"t


43).


The Joint


Committee


not the


only


source of


standards.


Government


Accounting


Office


(GAO)


standards


were


designed


to guide


in the judgment


impact


evaluations.


Finding


that


the Joint


Committee'


evaluation


standards


were


best


applicable


to education


and the GAO's


standards


were


only


applicable


to impact


evaluations,


the Evaluation


Research


Society


(now


called


the American


Evaluation


- -A


I


II


--m a













Evaluation


Research


Society


Standards


for Program


Evaluation


adopted


the Evaluation


Research


Council


(Evaluation


Research


Society,


1984).


The meta-evaluator must


have


access


to accurate


information


numerous


testing


instruments


used


in all of


the evaluations


she encounters.


The Mental


Measurement


Yearbooks


(Buros,


1972;


Mitchell,


1985)


might


well


be called


the standard


reference


tests


in use.


Campbell


and Stanley'


(1963)


Experimental


Quasi-Experimental


Designs


for Research


is still


considered


the standard


for questions


concerning


the evaluation


of alternative


experimental


designs.


Meta-Evaluation


Instruments


, Techniques


and Devices


literature


is surprisingly


full


checklists


for organizing


meta-evaluation.


Scriven'


s (1974,


1985)


Evaluation


Checklist


a synthesis


of his


contemplation


of meta-evaluation.


He has


at times


referred


to his checklist


as the multimodel


of evaluation,


reflecting


his view


that


evaluation


involves


multiple


dimensions,


should


employ


multiple


perspectives


, involves


multiple


levels


of measurement,


must


employ multiple methods.


There are


18 items


in the Key


Evaluation


Checklist,


the 18th


referring


to meta-evaluation.


The evaluation must


be evaluated,


preferably prior


implementation,


External


evaluation


final


dissemination


is desirable,


first


of report.
the primary


evaluator should apply
the evaluation itself.


the Key
Results


Evaluation Checklist to
of the meta-evaluation


should


be used


- Y. -


formatively
a~~ -t a~ a: *.


a.


n~ ar n.r p --


iso be incorporated
Sr-nvn t-4- ,rn^n' *rF.I t1


f^


h













Stufflebeam's


(1974a)


monograph,


Meta-Evaluation,


is organized


like


a giant


106-page


outline,


the whole


of which


could


be reduced


a brief


checklist


for doing


a meta-evaluation.


If not


the whole


document,


then


the discussion


of "Problems


that Jeopardize


Evaluation"


could


certainly


be used


as such.


There


is one checklist


in the monograph,


"An Administrative Checklist


for Reviewing


Evaluation


Plans


" designed


for formative meta-evaluation


but quite


usable


as a supplement


a summative meta-evaluation.


Millman


s (1981)


"The


Checklist"


focuses


on the merit


or worth


of the evaluator


s assessment


of the program or product


and the merit


or worth


of the evaluation


itself.


Questions


are directed


to three


component s


of the


program


or product


and the evaluation:


preconditions


or preliminaries


to the


conduct


of the


program


development


of the product


and to the evaluation


, (b)


the effects


the program or product


the evaluation,


and (


the utility


of the


program


or product


and the evaluation.


There are a


number


of forms


a meta-evaluation


can take.


Smith


(1981b)


suggested


the following


as possible


topics


of inquiry


a meta-evaluation:


design,


management,


instruments,


data,


results,


impact,


personnel,


purpose,


setting,


reporting,


use of formal


criticism,

and McNeill


or any

(1979)


combination of t

, in considering


he above.


Stevenson,


meta-evaluation


Longabaugh,


practices


in the


-Cn*-rb ni1 ,-rnc i ,ra


r ~


,- mn'n 1


ri 1 i1 i nn


-4 A- I rA j^ I~ n f.* /^


I~U: A*


_


fCj


r~













evaluation


can be classified


"to determine


extent


to which


evaluators


and clients


intended


the study


to fulfill


particular


purposes,


and the


extent


to which


their


expectations


were met"


(Stevens et

educational


al., 1979,

evaluation


46).


with


To date,


the question


little


has been


overt


done


covert


intentions


for decision


making.


Philosophical


analysis


was proposed


Gowin


(1981)


added


QUEMAC Value Appraisal


to the list


of forms


which


meta-evaluation


take.


QUEMAC,


a construct


for detecting


the logical


structure


an evaluation,


is an acronym


questions,


events


or objects,


methods,


answers,


concepts


(Gowin,


1981


300).


The meta-


evaluator,


following


the QUEMAC guide,


answers


six questions


about


the evaluation


study


answering


these questions


or she is able


uncover


embedded


value


concepts.


Gowin


advised


that


"value


theory


can help


in evaluation


appraisal,


and when applied


critically


evaluation


practice,


should


show


us how we


treat


value questions


also


to exhibit what


our values


are"


(Gowin


Millman,


1978


Smith


nature of


(1981a)


suggested


an evaluation


while


that


the meta-evaluator


simultaneously


assessing


illuminate


its quality.


He recommended the


use of the techniques


of formal


criticism--


description,


analysis,


interpretation,


and evaluation.


nature


an evaluation


be illuminated


just


as adequately


through













Discrepancy


Analysis


Techniques


Tallmadge,


Lam,


and Camarena


(1985a)


have


observed


that


treatment


as implemented,


not the treatment


as intended,


that


is evaluated. The

resemblance to what


actual


treatment,


was intended


unfortunately,


may,


consequently


bear

have


little

very


low construct


validity


relative


to what


the study


set out to evaluate"


xiii).


Sanders


experiences
the actual


witl


(1981)

h local


objectives


continued

projects


along

have


this

shown,


consequences


line


however,


a project


servingn,

that


tend


to drift
evolves.


away
What


from stated
is written


objectives
and what i


as the
done


project
differ and


the design


for monitoring


Title


VII bilingual


education


proj ects


should


consider


the latter.


approach


to analyzing


the discrepancies


between


what


proposed and


what


is done


as chronicled


in the final


evaluation


report


to apply


an adaptation


of the Discrepancy


Evaluation Model


(DEM)


proposed by


places


Provus


the proposal


(1972).


A modification


s evaluation plan


as th


of Provus'

e standard


Stage


representation


of how


the evaluation


should


final


evaluation


report


represents


as complete


against


concluded


about


ed and


yields


that


the worth


the performance


reported.


discrepancy


discrepancy

or adequacy


measure


Comparison


actual


of the congruence


information


analysis


is a matter


an object


based


Steinmet z

of making


upon


evaluation


of (


(198


judgments


information


which


is the result


a comparison


of (S)


against


The model













discrepancy


to the judgment


of the evaluator who must


answer


following


three questions:


Is the information


on each


program


element


complete?


Is the information


reliable


and valid?


Are the discrepancies


uncovered


ones


which


will


significantly


diminish


the program's


chances


of success?


Application


of Meta-Evaluation


Theory


to Bilingual


Education


Currently,


soundness


Still


the best


and technical


methodology


adequacy


a relatively new procedure,


available

evaluations


each


for assaying


is meta-evaluation.


meta-evaluation


that


completed


adds


to and extends


what


is known


about


meta


-evaluation


theory


and methodology.


Gowin and Millman


(1978)


considered


this


"fertile


area


for research


in evaluation"'


Stufflebeam


(1981a)


asserted


that


meta-evaluation


can play


a vital


role


in advancing


evaluation


as a profession


by providing


quality


assurance,


self-


regulation


renewal


and accountability.


"It is strongly


recommended


that


educational


evaluators


make


meta-evaluation


a matter


common


practice


in their work"


158).


Lois-Ellen


Datt a


(1981a)


of the National


Institute


of Education,


provided


evidence


for the need


for meta-evaluation when


she said


that


"the effectiveness


evaluation


has been


questioned


since


least


1969 when


Guba,


followed


by Wholey


et al.


(1970)


and Weiss













impact


on programs,


practices


or policy"


125).


Looking


specifically


reviews


of the literature on


the effectiveness


bilingual


education


evaluations


(Alkin


, Kosecoff,


Fitz-Gibbon,


Seligman,


Dulay


1974;


Burt,


Baca,


1979a,


1984;

1979b;


Baker

English,


de Kanter,


1983;


1981


Martin,


Burry


1981,


1979


1982a,


1982b;


Okata,


1983


Zappert


Cruz,


1977)


the overwhelming


consensus


indicated


the continued


presence of


serious


technical-methodological


flaws,


which


seem


uncorrected


with


time


and without


having


deleterious


effect


on the funding


continuation


grants.


Keith


Baker


(1984)


Office


of Planning,


Budget,


and Evaluation,


Department


of Education,


was one of the more


outspoken


critics


of bilingual


education


evaluation


practices.


a paper presented


at the American


Educational


Research Association


(AERA)


convention,


Baker


(1984)


offered


the following


indictment.


have


come


the research


literature


after


in bilingual


extensive
education


study


that


this


discipline


cannons


is pervaded
scientific


by the
research


subordination o
to ideological


political
selected


needs.
examples


their methodologic


is that


This


briefly


education


the extensive methodological


reviews
research


The thesis


problems


some
and


explored
found in


this
toward
that


literature ar
d conclusions


t


hey


cannot


incompetence.
the observed


been


flaws


widespread


to ensure
education


e so severe
supporting
e accounted


in th


abandonment


the dominance of


minority


and so strongly
the bilingual i
for as merely e


explanation t
is literature


hat


proper


a particular


language


children.


biased
deology
examples


can account


there


is that
research


ideology
(p. 1)


of
for
has


to the conclusion


paper


of bilingual
al faults.


The only


procedures


in the


<














thorough


evaluation of


the most


recent


bilingual


evaluation


reports


hence


the need


for this


meta-evaluation.


Stufflebeam and


Shinkfield


(1985)


declared


that


"evaluators


have


a professional


obligation


ensure


that


their proposed


or completed


evaluations


are subjected


to competent


"people


evaluation"


and agencies


321).


throughout


Stufflebeam


educational


(1981a)


evaluation


exhorted,


need


sustain and


increase


their


efforts


to develop


and use meta-evaluation"


158).


One of the first


meta-evaluations


of Title


VII project


evaluations


was done


Alkin,


Kosekoff,


Fitz-Gibbon,


and Selgiman


(1974),


at the


UCLA


Center


for Study


of Evaluation


(CSE)


The authors


reported


findings


on a nationwide


to determine


impact


sample of


of evaluations


42 LEA Title


upon


VII bilingual


decision-making


projects


at the


local


and federal


levels.


raters


could


not make


any judgments


about


the effectiveness


of the


proj ects


based


on the final


evaluation


reports.


Alkin


et al.


reported


that


they


felt


confident


that,


many

found


cases,

useful


reports.


Wh


formative e

by project

en asked di


valuation was


director


rectly


conducted


evaluators


was not evident


the majority


proj ect


in evaluation

directors


indicated


that


evaluation had


been


influential


in decision making,


particularly


on those


decisions


concerning


modifications


for the


following


school


year.


Although


refunding


decisions


were made


before













and comprehensive


evaluation


reports,


which


tended


to favorably


impress


federal


monitors,


and thus


the best


chance


of influencing


federal


decision


makers.


The next


year,


Campeau,


Roberts


Bowers


Austin


and Roberts


(1975)


examined


but judged


Cruz


that


175 final


only


looked


evaluation


of them merite


at 108


proj


reports

d site


ect evaluations


from bilingual

visits. Zappert


and 76 research


programs

and


studies


to judge


performance.


67 research


the effectiveness


With


proj ects


of bilingual


the rejection


for "serious


education


of 105 project


methodological


on student


evaluations


weaknesses"


the researchers


found


a significant


positive


effect


(58%)


or a


nonsignificant


effect


(41%)


on student performance,


based


on 3


project


evaluations


research


studies.


authors


advised


that


with


requirement


demonstrates


of learning


the positive


two languages,


advantages


a nonsignificant


of bilingual


finding


education.


rese


arch


demonstrates


that


bilingual


education


bilingualism
development,
and social s


improves,
reading a


studies


and self-image.


that


bilingual


or does


writi


achievement


In addition


education


not impede,
ng abilities
, cognitive


there


programs


is emp


improve


oral


language


, mathematics
functioning,
irical evidence
school


attendance.


Possibly the most


publicized


and controversial


study


to date


appeared


on the national


scene


in three


volumes


during


1977-1978.


Known


as the "AIR


Report


" authors


Danoff,


Coles,


McLaughlin,













Planning,

funded bi


Budget,


.lingual


and Evaluation,


education programs.


assess

Even


the

before


status


of federally-


the first


volume


was out,


staff


members


Arias


, Delgado


DeProcel,


and Irzarry


quit,


claiming


inadequate


design


and methodology


(Willig,


1984,


The results


states


from


indicted


this


evaluation


bilingual


education


of 38 Title


, influenced


VII projects


legi


in eight


station,


infuriated


critics.


Non-Title


VII Hispanics


outperformed


Title


Hispanics


in English


proficiency,


while


reverse


was found


true


for mathematics


for the interim report


but corrected


to equal


the final


report.


Two-thirds


of Title


VII classroom


space was


taken


up with


non-limited-English-speaking


students


failed


to exit


when


proficient.

proficiency


Finally,

in Spanish


Title

than


VII Hispanic

the non-Title


students


evidenced


VII control


group.


higher

Critics


such


as Lopez


and Cervantes


(1978)


Cardenas


(1977)


Gray


(1977),


O'Malley


(1978)


and general


and Willig


findings.


These


(198


challenged


findings,


however


the AIR Report

. stimulated c


s design


changes


in the Educational


Amendments


of 1978


, limiting


implementation


any project


to five


years,


and broadening


the definition of English


proficiency


for LEP students,


setting


an enrollment


requirement


or more


LEP classified


students.


Lyon,


Dosher,


McGranahan,


and Williams


(1978)


looked at


final


evaluation


report s


assess


whether


they met


minimum


criteria


/













reports,


62% did


not describe


the program well


enough


so that


objectives


were


clear;


90% did


not address


questions


of validity


and reliability


54% were no

and 72% did


of data


t comprehensive


not show


sources;


enough


congruence


the data


to answer


between


collection


sources


the evaluation


information


questions;


provided and


conclusions.


Troike'


s (1978)


monograph made


a strong


case


for federal


funding


to develop


an adequate


research


base


for bilingual


education


while


noting


that,


although


evaluations


should


provide


evidence


for program


results,


"the


vast


majority


them


(program


evaluations)


worthless


for this


purpose"


. 3).


He then


cited an


earlier


meta-evaluation


met minimal


of 150 final


criteria


evaluation


for acceptability.


reports,

Troike


only

then


of which


proceeded


discuss


12 unpublished


evaluation


studies


and made


the following


pronouncement:


"Despite


the lack


of research


and the inadequacy


evaluation

possible t


reports,


o say with


enough


evidence


confidence


that


now accumulated


quality


bilingual


to make

programs


meet


the goal


of providing


equal


educational


opportunity


for students


from


non-English-speaking


backgrounds"


Troike


has been


criticized


Baker


(1984)


for incorrect


data


, questionable


interpretation


of data,


and citing


a project


evaluation


that


one else


can find.


"A good


example


of the


misuse


of research


in the


a -


are


* ^ 1 i m -


- -


~


A













studies,


and 3


bilingual


program


evaluations


met minimum


research


design


standards.


However,


from


those


12 studies,


66 findings


were


generated,

education,

effects.


with


41% show

Bilingual


showing positive

ng no effects, a

education was co


effects


nd only

nsidered


as a result


showing


successful


of bilingual


negative


Dulay


Burt


because


it either


improved


or did


not hinder


academic


achievement


in school.


Martin


(1981)


completed


a meta-evaluation


of the final


evaluations


of 43 out


California.


a total


The results


128 Title


this


VII projects

meta-evaluatio


in the State


study provided a


description


of evaluation


design


characteristics,


data


collection


procedures


, testing


instruments


in use,


data


analysis


techniques,


and general


evaluation methodologies.


Another


author


(Willig,


1984)


used


a specialized


form of


meta-evaluation,


called meta-analy


sis,


where


statistics


from


the sample of


evaluations


were


aggregated


reported


a metric


called


effect


size.


Willig


conducted


reanalysis


of the 23 studies


analy


zed by


Baker


and de


Kanter


(1981)


under

U.S.


the auspic

Department


of the Office


of Education.


of Planning,


Baker


Budget,


and de Kanter use


and Evaluation,

d the traditional


vot e


score


method


for aggregation


of data


while


Willig


calculated


effect


sizes.


Baker


and de


Kant er'


study,


which


received a


good


deal


of media


attention


concluded


that


case


for bilingual













progress.


Baker


de Kanter's


rese


arch


was criticized


Littlejohn


(1981)


, Rotberg


(1983)


Seidner


(1982),


Yates


and Ortiz


(1983)


and Willig


(1982)


on methodological


grounds


and possible


bias


selection


of studies.


Willig'


results


indicated


that,


contrary


to Baker


differences


de Kanter'


favoring


results


bilingual


there


students


were


over


small


to moderate


comparison


students


for reading

in English,


in English,

and total a


language


Achievement


skills


in English,


in English,


when


mathematics

the effects


the other


variables


were


held


constant.


Willig


attributed


differences


in results


to meta-analysis'


greater


power


to detect


true


differences


between


groups


due to the accumulation


of estimated


effect


sizes


over


students


and to make


possible


statistical


controls


for variables


in the final


analysis


(Willig


1984


116).


Finally,


synthesis


the last


of Title


study


to be considered was


VII evaluations


and research


also


reports


a meta-analytic


from


years


1977-1981


but reported


in 1983


James


English,


Office


Planning,


Budget,


and Evaluation,


U.S. Department


of Education.


English


had major


difficulties


locating


documents


, only


finding


approximately


60%.


Of the


reports


available,


only


of those


prior


to 1980

English

internal


and 25% from

recommended

support for


1980-1981


that


proj


were


the Departm

ect evaluate


deemed qualified

ent of Education

ion and provide


to be included.


increase


technical













improving a
adequately
research de
significant


academic
assess


signs,
positive


achievement
d because of


of students


a poor


all factors


effect


are not


evaluation,


which


of bilingual


obscure
programs


Boruch,


Cordray,


Pion


(1981)


agreed,


"the


evidence


presented


the majority


of evaluations


is often


insufficient


for judging


effects


of the


projects


or programs


on children"


Finally,


Baker


and P


elavin


(1984)


contributed


the fitting


summary


to this


review

reviews


of bilingual


agreed:


evaluation

overall q


studies


quality


stating,


of bilingual


one point


evaluation


research


[emphasis


added]


is very poor"


Evaluation


Research


Versus


Program


Evaluation


In the previously


stated


remark


Baker


and Pelavin


(1984


the underlining


of the word


research


did not


appear


in the ori


final


quotation,


added


for emphasis.


federal


government


many


of the


previously


cited


researchers


have


not made


a distinction


between


evaluation


research


and program evaluation.


Stufflebeam


(1978)


stated


that


"only


chance


renders


an information


system adequate


when


the decision


maker


does


specify


the expected


decision


situations


to be served


term"


123).


Sanders


(1981)


continued,


"information


collected


a particular purpose


level


have


little


or no


utility


at other


levels


a national


system of


proj ects"


Willig


(1984)


added,


- 1 S


one


* .


~


~













be put


in the position


of attempting


to answer a question


that


is truly


impossible


answer


in non-research


settings,


i.e.


what


are the effects


this


program on


participants


without


such


compared
a program


to what
? (p.


the effects


would


What


we have


seen


is the


post


expectation


of the federal


government


and,


many


researchers


in the field


as well


that


research


data


proving


the effectiveness


of bilingual


programs


should


be available


collecting


and synthesizing


any quantity


of LEA program evaluations


originally prepared


to meet


the information


needs


of the local


project.


The previously mentioned


reviews


have


illustrated


that


such


a plan


puts


an undue


expectancy


on a system


that


is not adequately


prepared


to handle


, nor provided with


the technical


assistance


accomplish


Lincoln and


Guba


(1984)


distinguished


program


evaluation


from


evaluation


research by purpose,


products,


outcomes,


and audiences.


De George


(1983)


saw evaluation


a device


intended


to inform


the manager and hi


s/her


staff
have


to what


degree


they


have


set out to do and how they


accomplished
can improve


what


they


Because


it is the


account
funding


vehicle


for their
agencies


whereby


that


stewardship


sponsors


will


same


manager


the basis


and staff
on which


decide whether to


continue or


is more


than


terminate


research.


program,
. 6)


clearly


evaluation


Lipsey


Crosse,


Dunkle,


Pollard,


and Stobart


(1985)


after wrestling


with


question


of program


evaluation


and the experimental


paradigm


came


to the following


conclu


sion:













theoretical
and program
approaches


policy m
specific


propositions
evaluation,


to provide


akers


programs


very nature,


to routine


embodied


which


prompt,


program
with wh
program


application


will


in social
use less


useful


administrators


ich


they


research


under


are


programs,
restrictive


information


about


concerned.


may not


circumstances


be amenable
where


answers
extensive


are needed quickly


e


programmatic


or where


research


support


is not feasible.


And,


nature,


program evaluation may not


be capable


answering


the ultimate


causal


question,


"Does


this


social


intervention


produce


the intended


ects?"


The challenge


to the evaluation


profe


sslon


is to know


the diff


erence


between


these


approaches


and their


domains


of applicability.


Several


solutions


have


been


proposed


to solve


the dilemma


of the


need


for research


to judge


the effectiveness


of bilingual


programs


report


information


congress


needs


and program


of the local


evaluation


bilingual


to provide


education


data


project.


to meet


Willig


(1984)


proposed


establishment


of federally-funded


district


research


centers


to conduct


research


in bilingual


education,


where


students


teachers


could


be randomly


assigned


and where


threats


to validity


reliability


could


be controlled.


These


centers


would


operate


independently


of the LEA projects


where


program


evaluations


would


seek


to meet


the information


needs


the local


decision


makers.


Millman


(1981)


proposed


the distinction between


a federal


reporting


system and

summative


need


an evaluation


information


information.


system.


about


proj ects


The data


would


reporting

to address


be comparable


system would

a specific f


across


provide


federal


projects













particular project.


Sanders


(1981)


suggested


that


experience


gained


from


Title


program monitoring


be transferred


to the


monitoring


of bilingual


accomplishments.


The preceding


discussion


is timely


because


the federal


government


is currently


considering


alternative


procedures


and materials


the evaluation


of bilingual


programs.


In July,


1985,


SRA Technologies,


awarded a


contract


to design


develop


a new system


the evaluation


of bilingual


programs.


Field


testing


is expected


begin


in 1986-1987.


The question


of research


versus


program


evaluation


important


to this


meta-evaluation


study.


Although


it is critical


to the


development


a new


system


to have an


indepth


understanding


of the


current


status


of LEA


program


evaluations


the major


focus


of this


meta-evaluation


was on program


evaluation


for the local


decision


makers.


was



















CHAPTER


METHODOLOGY


The purpose of


this


study was


to conduct


a meta-evaluation


Florida


Title


VII LEA bilingual


education


project


evaluations,


fiscal


year


1984-


This


study


examined


evaluation


model


design


characteristics,


instruments,


data


data


analysis


collection


techniques,


procedures,

and general


testing

evaluation


methodologies,


as well


as discrepancies


between


proposed


evaluation


plans


summative


evaluation


reports


in order


to answer


following


four questions:


Which


context


and process


variables


are addressed


Title


VII project


summative


evaluation


reports


within


the State of


Florida?


What


are the characteristics


of Florida


Title


VII evaluation


models,


designs,


reports


in terms


of information


coverage,


content,


and procedures?


Do Florida


Title


VII evaluation


reports


meet


current


accepted


professional


standards


of good


evaluation


practices,


detailed


r. I -


the Joint

rl-A -l- _--


Committee


_As


on Standards


- --- 1 j- C- ~ -


for Educational


- i' r>-l -*-: -


n, wflflnf .l













Are there major discrepancies between LEA proposals and


summative evaluation reports?


What is the nature of these


discrepancies?


Using Stufflebeam's


(1981a)


terminology,


summative meta-evaluation


may be classified as a retroactive assessment of evaluation studies,


their goals,


designs,


implementation,


and results combined into a


simple summary


case


study.


Stufflebeam and Shinkfield


(1985)


further explained that "meta-evaluation invokes the accepted standards


of the profession and


assesses


and tries to assure that they are


met"


35).


The aim of meta-evaluation


according to Stufflebeam


and Shinkfield


, is "to assure quality evaluation services,


to guard


against or deal with malpractice or services not in the public


interest,


to provide direction for improvement of the profession,


and to promote increased understanding of the evaluation enterprise"

(p. 34).

Conceptualization of Meta-Evaluation


Daniel Stufflebeam (1974a,


1974b)


was not only the first researcher


to define the basic concepts of meta-evaluation, but even today his


conceptualization remains the standard.


Stufflebeam posited that the


conceptualization of meta-evaluation must be consistent with an


evaluator's conceptualization of evaluation.


The following eight


premises for the conceptualization of evaluation and meta-evaluation













Evaluation is the assessment of merit; thus, meta-
evaluation means assessing the merit of evaluation
efforts.
Evaluation serves decision making and accountabili


formative meta


information
meta-evalua
for account
Evaluations
and results
importance
of evaluati


of the
and the


Eva
inf


d


luati
ormat
a-eva
k and
util
luati
affe
a-eva
are


esi
qua
on
ion
lua
sh
iza
on
cte


-eval


for dec
ion ret
ability
should
thus,
f evalu


on d<
gns,
lity
shou
and
tion
would
tion
shou
d by


luation
interest


shou
form
rs (
ity)
eta-
f th
acti
invo
o be
. an


Evaluation
(generally
and outside
accountabil
formative m
judgments o
evaluation
Evaluation
questions t
information
making and
must decline
to be addre
information
appropriate
Evaluation
cost effect


same criteria.


C


O

d


pro-
makin
vely
t eva
goal


active
g, wh
provi
luati
s, de


valuation sh
objectives,
e adequacy o
ical adequac


and importance of eva


Id prov
approp
should
provide
of eva
id serve
the pr
should
ted in
Id be c
active e
general
SHenc
evaluate
e over
vities.
'lves th
address
id using
)untabil
the spe
L: colle


provides
summative
information
work.


gns,
would
the a
f imp
y of
luati


im
as
PP
le
da
on


ide descriptive and ju
rate recommendations;
describe and judge ev
e recommendations for
luations.
e all persons who are
ogram being evaluated;
serve evaluators and
the work.
conducted by both inside
valuation for decision
ly summative evaluatic
e, evaluators should c
ion and they should ot
11 merit of their comic


process


ie
s
r


ity.
Scific
ct, o


nd apply the
cision-making
t be technical


of


obtainin
informant
Hence,
meta-ev
organize,
obtained
and acc
lly adeq


, and meta-evaluation


(Stufflebeam,


1974,


plementation,
sess the
ropriateness
mentation
ta analysis,
results.
dgmental
thus,
aluation
improvement

involved in
hence,
all persons


ters
making),
mn for
conductt
tainn external
)leted


delineating the
g the needed


ion in de
the meta-
aluation
and anal
I informant
:ountabili
luate, use
must sat
pp. 70-7


cislon
evaluator
questions
yze needed
ion to the
ty tasks.
ful, and
isfy the
1)


Sample


mF,. -. '-'Ti


ty;


I


r I .~. -I nll


'Itln rr


(


r


L


r

!

l













the Bilingual


Consultants


Office,


Florida


Department


of Education,


Tallahassee,


Florida,


new


or continuing


Title


VII basic


demonstration


grants


were


funded


in the fiscal


year


1984-85.


The sample


consisted


of Title


VII projects


which


voluntarily


contributed


either


or both


application


proposals


and final


evaluation


reports.


Data


Collection


Procedures


The data


was confined


to application


proposals


which


included


a proposed


evaluation


plan


and final


evaluation


reports


from


the sample.


Most


Title


VII proposals


are kept


on file


in the Bilingual


Consultants


Office,


Florida


Department


of Education.


Those


proposals


that


were


on file


were


mailed


directly


to the researcher


the project


managers.


Proposals


are considered


sensitive


documents


districts


because


Title


VII Bilingual


Education


Basic


Grants


are awarded


on a competitive


basis,


discouraging


intra-district


collaboration


on proposals.


district


refused


to submit


a proposal,


citing


past


negative


experiences


with


proposal


material


being


used


other


districts


without


permission.


Therefore,


out of 12 proposals


were


included


in the data


set.


The final


evaluation


reports,


written


project


evaluator,


were


requested


directly


from


the project


managers,


with


12 out of


total


of 14 final


evaluation


reports


received


the researcher.


district


sent


a formal


letter


requesting


not to be included


in the


0 *0I


a


I __


1













the meta-evaluation


if and when


they


receive


them.


One arrived


, leaving


the second


of the


two to be omitted


from


the data


set.


It must


be noted


that


federal


regulations


require


final


evaluation


reports


to be submitted


to OBEMLA


in Washington,


not later


than


90 days


after


the termination


of the


grant,


placing


the deadline


in December,


1985.


Instrumentation


one instrument


currently


exists


which


synthesizes


all of the


criteria


advocated


in the multitude


of models,


checklist


s, standards,


guidelines,


pitfalls,


and recommendations


found


in the literature


for conducting meta-evaluations.


Indeed,


if all of the elements


of the


various


aforementioned


documents


were


synthesized


into one


meta-evaluation


instrument


it would be


unwieldly


and ineffective


for any meta-evaluation.


However,


one


set of instruments


developed


Martin


field-test


(1981)

ed, an


id


has the advantage of being spe

emoloved in a meta-evaluation


I


cifically

of 47 Tit


designed,

le VII project


final


evaluations


for the


year


1979-80.


instruments


represent


careful


synthesis


of the following


11 documents:


Evaluation


Checklist


(Scriven,


1974)


Illustrative


An Admini


List


strator'


of Pitfalls


Checklist


in Evaluation

for Reviewing


Works


(unknown)


Evaluation


Plans


(Stufflebeam,


1974a)


L


- --












Checklist


for Judging


the Adequacy


an Evaluation


Design


(Sanders


Nafziger


, 1976)


Table of


Contents


for a Final


Evaluation


Report


(Stake,


1983)


Suggested


Title


VII Evaluation


Design


Specifications


(Stake,


1983)


Suggested


Title


VII Evaluation


Design


Specifications


(Office


of Program


Evaluation


and Research


and Bilingual-Bicultural


Education Section,


California


State


Department


of Education,


October,


1977)


Checklist


for Planning


and Managin


the Evaluation


Title


VII Projects


(OPBE,


Department


Education)


Checklist


for Planning


the Evaluation


of Multi-funded


Programs


(Office


of Program


Evaluation


and Research,


California State


Department


of Education).


Since


their


development


no other


researcher


employed Martin's


(1981)


instruments


in a Title


VII meta-evaluation.


present


study


used


all four


of Martin'


instruments


for initial


data


gathering


purposes.


Instrument


the "CES Meta-Evaluation Checklist"


(Martin,


1981),


contains


153 items


and provides


an indepth


analysis


of criteria


related


to evaluation model


characteristics


information


coverage,


general


and specific


design


characteristics


data


collection


techniques,














Instrument


"Program


Design


Data Sheet"


(Martin,


1981),


contains


items


pertaining


to such


program characteristics


as size,


number of


schools,


grade


levels,


and languages


served,


staffing,


evaluation


resources


, program


emphasis,


program


type,


instructional


approaches


student


achievement,


staff


development,


community


involvement,


curriculum


evaluation,


and materials


in use.


Instrument


"Supplemental


Data


Sheet


#1" (Martin


, 1981)


modeled after


Alkin


, Kosecoff,


Fitz-Gibbon,


and Seligman' s


(1974)


study rating


overall


quality


of various


data


collection


and analysis


techniques


in Title


VII evaluation


reports.


This


a 51-item


checklist


rating


items


on a 1-through-3


scale.


items


are related


to the general


organization


and presentation


of evaluation


findings,


overall


proposal


coverage,


overall


evaluation


design


coverage,


assessment


pertaining


techniques,


to validity,


data analysis


objectivity,


, and


report


credibility,


characteristics


and utility


Instrument


"Supplemental


Data


Sheet


(Martin,


1981)


reserved


for extra narrative


comments


regarding model


characteristics,


design


characteristics,


and special


notes


on data


analysis


techniques.


There


is also


space provided


for comments


on exemplary practices


areas


in need


of improvement.


present


researcher


designed


an instrument


based


on the 30


standards


for evaluations


of educational


projects,


developed













Data Analysis


Meta-evaluation procedures


as outlined


Stufflebeam


(1974a)


were


used


in the analysis


of data.


retroactive meta-evaluation


of goals

Title VI


designs,


I bilingual


implementation


education


proj ect


results of

evaluations


the 1


were


984-85 Florida

accomplished


through


evaluation


a careful


application


instruments


(Martin,


of the four previously


1981)


described


the instrument


meta-


based


the standards


developed by


the Joint


Committee


on Standards


Educational


Evaluation.


Data


collection


ranged


from a


simple


check


on a checklist


type


instrument


indicating whether


specific


items


were


or were


addre


ssed in each


describing


proposal


and rating


and final


the quality


report


of the


copious


primary


notes


evaluator


s report


terms


of such


factors


as comprehensiveness


coverage,


timeliness,


and technical


adequacy.


Analysis


of the meta-evaluation


data


employed


descriptive


statistics


to characterize


various


aspects


of the sample


of evaluation


designs


reports.


Frequency


counts,


means


, modes,


percentages


were


calculated where


appropriate


to identify


most


common


characteristics


in the designs


reports.


Averages


were


used


determine


extent


to which


various


context


process


variables


were


developed and


objectives


implemented.


Discrepancy an


alysis













The results


of the


various


analyses


which


make


the meta-


evaluation

A listing


were


employed


of variables


answer


the four main


characteristics


were


research

required


questions.


answer


questions


and 2.


In question


, meta- evaluator


judgment


required


to determine


if the sample of


final


evaluation


reports


or did not meet


each


of the 30 standards


of good


evaluation practices.


A combined


sample


score


of 80%


on all relevant


variables


or criteria


was used


as the cut-off


point.


A score of


or higher was


required


to assess


that


the standard


was met.


description


of the


nature


the discrepancies


and the fina

The results


between


evaluation


of questions


application


reports


proposal


was required


evaluation


to answer


are presented


plans


question


in Chapter


was



















CHAPTER I
RESULTS


purpose


of this


study was


to conduct


a meta-evaluation


Florida


Title


VII LEA bilingual


proj ect


evaluations,


fiscal


year


1984-85.


This


chapter presents


the meta-evaluation


findings


to the


following


four questions:


Which


context


process


variables


are addressed


Title


VII project


summative


evaluation


reports


within


the State of


Florida?


What


are the characteristics


of Florida


Title


VII evaluation


models,


designs


reports


in terms


of information


coverage,


content,


and procedures


Do Florida


Title


VII evaluation


reports


meet


the current


accepted professional


standards


of good


evaluation


practices,


detailed by


(1981)


the Joint


Standards


Committee


for Evaluation


on Standards

s of Educatio


for Educational

nal Programs, P


Evaluation


roj ects,


and Materials?


there


maj or


discrepancies


between


LEA proposals


summative


evaluation


reports?


What


is the nature


of these


discrepancies?













Question


Context


and Process


Variables


Final


evaluation


reports


should


include


sufficient


descriptive


information


that


they


can stand


on their


own merit


without


reference


to proposals


or other


an evaluation


in which

being ev


a final


to give


the project


aluated.


evaluation


documentation.


the reader


functions


Sufficient


report


Context


variables


a clear picture


the unique


contextual


so that


are included


of the


features


variables


reader may


should


setting


of the project

d be addressed


judge


the effect


of the contextual


conditions


on the project


and under what


similar


conditions

variables


such


the findings


include


as ethnic


may be appli


geographic


composition


cable.


location,


and migration,


Examples


community


context


characteristics


a description


of the district,


and a description


of how


project


fits


into


the local


school


sites.


Program c

of the project


characteristics


such


provide


as the number


a picture

students


of the unique


receiving


features


treatment


the number of


sites


the ethnicity


of students


program


type,


design,


and instructional


approaches


program


emphasis;


staffing.


Process


variables


should


describe


the implementation


of the project


goals,


management


development


of the project,


materials,


progress


timeline

records,


for activities

and community


s, staff

involvement.


The first


meta-evaluation


question


specifically


asked


which


context


process


variables were


addressed


in Title


VII project













12 final


reports


were


analyzed


the results


are presented


the order


context


variables,


program character


stics


process


variables.


Context


Variables


Geographic


location.


Only


two projects


representing


16.7% of


the sample


narrative.


cover,


indication


discussed


(N=12)


A total


but in five


the location


of 10 reports


of these


of the location


reports


of the project.


of the project


listed


in the


county


cover was


An additional


name on


the only


reports


five


mentioned


country


name


in the narrative,


two of these


reports


(16.7%)


did not


identify


the name


of the


county


on the


cover.


Description


of district.


Three


final


reports


25%)


included


brief


count


description


of the district.


of limited-English


proficient


Five


(LEP)


reports


students


included


for the whole


district


while


these


five plus


an additional


three


reports,


making


a total


eight


discussed


the project


student


population


such


factors


as ethnicity


and grade,


ethni


city


and school,


school


and grade,


exceptionality


and school,


ethnicity,


school


grade.


Seven


reports


(58.3%)


discussed


socioeconomic


factors


of the student


population


such


as parents'educational


level,


parents'


occupations,


and percent


of students


receiving


free


and reduced


lunch.


Available


resources.


Six reports


listed


district


L


~F













Community


characteristics.


most


frequently mentioned


community

(83.3%) 1


characteristic


isting


was ethnic


the following


composition,


combinations:


with


Hispanic


10 reports

in general,


Puerto

Korean,


Rumanian,


Rican, Haitian,

Chinese, Greek


and Black.


Asian in general

, Tagalog, Hebrew


One report


, Vietnames

, European


characterized


Thai


in general,


a community


, Laotian,


French,


as a long


time


area


Anglo


of the


reports


crops


community with


county.

7%), ma


and in


recent


Migration


linly


immigration


was mentioned


stimulated


one instance


a milita


of Puerto


as a factor


agricultural

rv base and


Ricans


workers


a refugee


one


five

following

resettlement


center.


One report


calculated


length


of time


in the U.S.


another


report


(8.3%)


calculated


number


of days


in school


nationality


and migrant/non-migrant


status.


Four


reports


(33.3%)


mentioned


socioeconomic


factors


connected


with


occupations


and unemployment


within


the community.


Local


school


context.


Not a great


deal


of information


provided


on the Title


VII project


within


the local


school


site.


Nine


reports


(75%)


did address


site


selection,


how or why


sites


were


selected


to house


the bilingual


program.


Seven


reports


provided


information


that


Title


VII projects


were


housed


sites


that


already


had bilingual


programs


established.


Seven


reports


58.3%


identified


individual


school


sites


name


, either


in the body


the report


was


J


I _I _













Program Characteristics


Project


type.


Two projects


(16.7%)


were


identified


as demonstration


projects,


while


the remaining


projects


(83.3%)


were


either


identified


or assumed


to be basic


as new projects


projects.


completing


their


Three


first


projects


year


were


receiving


identified


their


first


evaluation.


Six projects


(50%)


were


identified


as functioning


under


continuation


grants,


with


five


of the six (41.6%)


identified


as having


completed


their


second


year,


however,


there was


confusion


in one


report which


identified


current


year


as the second


year


one point


and as the fourth


year when


attainment


of objectives


presented.


sixth


report


not specify


year


of the


continuation


of time


grant.


the project


Three


proj ects


had been


(25%)


operation,


identify


a serious


the length


oversight


in that


there


are differing


expectations


for information


coverage


such


degree


of implementation


for first


and final


year


evaluations.


Size


project.


Seven


of the 12


reports


(58.3%)


presented


unclear and confu


sing


picture


of either


or both


the number


of students


served


nine


or the number


reports,


project


a total


sites.


students


With


were


data


served


available


under


from only


Title


grants


in those


nine


sites,


averaging


site.


average


number


is misleading


however


, since


three of the


proj ects


were


very


large


(844


581,


and 377)


and the remainder were


between


50 and 162


was













Five


projects


(41%)


provided


services


exclusively


to elementary


schools,


two projects


provided


services


to elementary


middle


schools,


three


proj ects


(25%)


provided


services


to elementary,


middle,


and senior


high


schools,


one project


report


did not


identify


number


services


sites


to staff.


or levels,


one project


Of the 10 project


reports


only provided


which


training


recorded


number


of sites


served


was reported


that


51 schools


housed


Title


VII project


serve


ices.


Nine of


the 10


project


reports


recorded


the number


sites


at each


level


with


sites


(49%)


allocated


elementary


school


sites


allocated


to middle,


sites


allocated


One of the 7


high


to high

school


schools,

sites was


and 7


in


sites


a private


13.7%) unidentified.

religious school


setting.


Languages.


LEP Lau category


A and B


students


whose


dominant


language


was Spanish


received


treatment


in 75%


proj ects


with


four projects


exclusively


serving


Hispanic


students,


three


proj


ects


25%)


serving


a majority


of Hispanic


a minority


Haitian-Creole


and others,


students,


" and


one


one project


project


(8.3%)


(8.3%)


serving


serving "Spanish,


multiple


black,


languages


including


Spanish,


Haitian-Creole,


and Vietnamese.


The remaining


consisted


one Greek/Anglo


project,


one proj


serving


mostly


Asian


languages,


one


project


which


not identify


languages.


. ~ ~ ~ n --


n.


q~~ .


I I


rl 1













music,


instruction as


reason


physical


their


the majority o


education,


comprehension

f reports dis


and increasing


of English


tinguished


amounts


increases.


students


content


For this

grade


level,


providing


one report


services


.3%)


to exceptional


a project


students


that


specialized


did identify


students


age groupings.


It is


uncommon,


however,


for a LEP student


spend more


than


one year


a certain


grade


level


or more


than


year


in the bilingual


presented


Three


student


reports


project.


achievement


mentioned


Only

data b


transiency


one report


y number


(8.3%)

years


as a factor but


onl


, however,

in the project.

y one project


(8.3%)


presented


average


number


of days


of attendance by


language


group


and occupation.


retention,


falsely


Finally,


labeling


only


one


as drop-out


report


rate,


addressed


with no


rate


reports


addressing


the drop-out


rate.


Program


type.


Seven


projects


(58. 3%)


were


described


in the final


evaluation


reports


as transitional


bilingual


projects,


one project


was described


not labeled


as a maintenance program,


referred


as an intensive


English


program,


one project


was not labeled,


and the distinction


not apply


to two


projects


There


no uniformly


applied


and accepted


definition


of what


constitutes


a bilingual


program


either


at the federal


level


or in the


* -~ a a


one


was


one


I. .


I


*


I*













practice


as there


are programs


in this


state,


which


contributes


the difficulty


of identifying


and classifying


a program using


nationally


recognized


labels


such


as full-bilingual,


partial-bilingual,


or intensive


English.


terms


of the instructional


approaches


described


in the final


evaluations,


five


projects


might


labeled as


partial


bilingual,


while


three


projects


25%)


might


labeled


as full-bilingual


programs


with


one (8.3


being


identified


as intensive


English.


of instruction


content


However,


subject


if the label


matter,


applies


the three


to the language


full-bilingual


two of the partial-bilingual


programs


presented


content


material


in the student


s first


language


while


providing


English


a second

bilingual


language

programs


(ESL)


25%)


instruction.


provide


content


three


additional


instruction


partial-


in English


the first


language


the classroom


full-bilingual


as necessary.


structure


programs


in which


were


Finally,


program


self-contained


if the label


is delivered,


bilingual


applies


three


classrooms


except


art,


music,


and physical


education.


Two of these bilingual


classrooms


were


staffed


with


bilingual


teachers


one bilingual


classroom,


in what


was described


not labeled


as a maintenance


program,


was


staffed with


bilingual


aides


and a monolingual


English


teacher who


alternated


content


instruction


in first


language


and ESL


with


second


language


instruction


to native


English


speakers













language


arts


the remainder


of the day


to assignment


in a mainstreamed


classroom with


a full-time


bilingual


aide


(8.3%)


with


pull-out


to assignment


in a mainstreamed


classroom with


an itinerant


bilingual


aide


(25%)


as needed


with


pull-out


ESL.


Middle


seni or


high


school


programs


were


described


as providing


a much


more


standard


program


from


one to two hours


ESL with


tutorial


help


in content


classes


as necessary


from bilingual


aides.


The final


evaluations


for five


projects


mentioned


use of


individualized


instruction


but specific


bilingual


or ESL


teaching methodologies


were


not addressed


evaluation


report.


Five


projects


did address


the question


amount


of ESL


instruction


time.


Entry


and exit


criteria.


a student


qualifies


to receive


treatment


in the Title


VII project


and what


proof


and procedures


required


for moving


a student


from


proj


ect to mainstream English


classes


was addressed


in seven


final


evaluation


reports


(58.3%).


Three


of the


seven


project


reports


25%)


simply noted


that


such


criteria


were


in existence


but four


reports


actually


described


specific


criteria.


Program


emphasis.


proj ects


existed


to provide


training


staff


in bilingual


centers


to parents


of LEP students.


remaining


10 projects


emphasized


the following


goals


in descending


__ -


r..~~~~~1 --IIIS 1C


* S rr *
-. .. '-* 4- -fl nf 4p- an


itr a1 nnm On 9


are


-4. Cf


C


#


f~ IE:_


I I 1













Staffing.


Six project


reports


(50%)


made


some


reference


staff


qualifications.


Of the 12 project


reports,


10 (83.3%)


made


some


attempt


to identify


staff


positions


but the data


were


unclear


as to the total


number of


staff


positions,


those


paid


by Title


and those paid


the district.


was


also


unclear


as to which


staff


members


were


bilingual


and which


were monolingual


English.


Most


reports


not address


the qualifications


or amount


of inservice


received


by mainstream


classroom


teachers


with


and without


bilingual


aides.


The qualifications


of instructional


bilingual


aides was


addressed.


Most


evaluation


reports


not address


the question


whether


project


director was


a trained


bilingual


educator


or a


district

Process


administrator.

Variables


Implementation


evaluation.


Certifying whether


a project


has been


implemented as proposed


and documenting


when


each


aspect


of the


project


was set in place


to determine


whether


as fully


or not


functioning


the project


enables


is valuable.


the evaluator


Evaluation


not be valid


or reliable


a project


has incurred


major


delays


obtaining


and training


staff,


providing


materials,


assigning


students


to all sites, s

administration,


;tandardiz

or other


ing


curriculum,


similar


problems.


teaching methodologies,


Determining


test


a project


has been


sufficiently


implemented


to warrant


a full


evaluation


should


* n a


"I 1' -


4


r r


n r


/*













documentation


as to how the project


is actually


functioning


in all


sites.


Evaluation


based


on the


assumption


that


a project


functioning


according


to proposal


specifications


no more


reliable


than


a public


relations


document.


Two final


evaluation


reports


(16.7%)


out of the sample


of 12


reports


part


completed


of the final


a thorough


report.


implementation


One of the


evaluation


two projects


was


as an integral


identified


being


that


a first-year project


information


while


but consisted


the other


entirely


report


a process


not specify


evaluation


conducted

questions


an internal

student ach


evaluator who


ievement.


Four other


t address

projects


product

out of the


total


of 14 awarded


in Florida


in 1984-85


were


in their


first


grant


year.


grant


administrator


did not


submit


a final


report


to the


meta-evaluator,


one report


was


submitted


but did not


identify


project


as being


in its first


year


and two other


submitted


reports


identify


the project


as being


a first


year


grant


but did


not provide


an implementation


assessment.


Although

it alone can


an activity timeline

not be considered an


one step


implementation


in the right

n evaluation.


direction,

Four


reports


one of which


identified


the project


as being


in its


first


year,


present


in chart


form


the project'


main


objectives,


proposed


activities


and a timeline


consisting


of dates


of planned













Staff


development.


The inservice


training


of Title


bilingual


teachers,


bilingual


instructional


aides,


and mainstream


classroom


teachers


was addressed


nine


reports


(75%)


with


project


designed


specifically


to install


an instructional


management


system and


to provide


inservice


training


to locally


funded


bilingual


program


staff


and regular


classroom


teachers


with


students.


Seven


of the nine


projects


which


addressed


staff


development


tabulated


the number


of inservice


presentations, which


ranged


from


to 58,


while


five


projects


provided


a listing


of program


types


and topics.


Five


projects


presented


participants'


assessment


of the value


of the staff


development


activity


while


two projects


(16.7%)


addressed


the staff'


application


of acquired


knowled


to the classroom


situation.


proj ects


(16.7%)


also


reported


providing


training


to parents


and the


community.


Four projects


provided


college


and university


courses


for bilingual


staff


in one project


provided


university


courses


to parents


with


the aim of preparing


a well-trained


pool


applicants


for future


aide


positions.


Materials.


Acquisition


of instructional


materials


was addressed


10 projects


but there


was little


agreement


on topics


covered.


project


which


did an implementation


evaluation


documented


that


materials


arrived


in good


time


to be used


throughout


the project


one













report


discussed


the development


and evaluation


computer


software.


Seven


reports


(58.3%)


addressed


materials


development,


adaptation,


and translation,


while


three


references


(25%)


were made


to ordering


published materials.


reports


were


not specific with


regard


the language


mention


of the materials


materials


discussed


in Spanish


although


Greek


three


and several


projects


other


(25%)


districts


mentioned


using


regular


district-adopted


English


texts.


Curriculum.


development,


revision,


or adaptation


of curriculum


was discussed


in five


reports


(41.7%).


One project


requested


funds


for a three


summer


sequence of


curriculum


development


and evaluation


a committee


VII funds.


of teachers


Another project


who receive

presented


a stipend


evidence of


paid


out of Title


the revision


of the


ESL curriculum


for use


in computer


assisted


instruction.


Record


keeping.


Various


systems


accumulating


and storing


accurate


project


records


were


discussed


by eight


project


reports


(66.7%).


plans


received


Instruments


continuums


Title


such


were


VII funds


as various


identified


to install


types


pupil


in all eight


a management


progression


reports.


information


district


service


for keeping


student


data


which


provided


an individual


student


"map"


of language


skills


mastered.


Funds


also


covered


the inservice


training


for bilingual


staff


and mainstream


classroom


teachers


to ensure


appropriate


installation


and use of


system.


Other districts


* -_ __1 .. A ,_ -


-- -- A -- J -


~nrr. *


I


,--,A


r 1












least


one parent meeting


the parents


and seek


advertised


and the community


support


at lar


for the project.


in a local

ge of the


Once


newspaper

proposed a


the grant


to inform


application


is awarded,


parent


advisory


council


(PAC)


is required.


Provisions


are made


that


non-English-speaking


contribute


to the meetings


expectancy


was that


parents

Since


some


will b

these


aspect


e able


to understand


are Title


parent


VII requirements,


participation would


mentioned


in all of the final


evaluation


reports,


however,


various


aspects


of this


topic


were


addressed


only


seven


proj ect


reports


.3%).


Five


project


reports


mentioned


PAC meetings


with


several


reports


appending


bilingual


invitations


to meetings


least


focus


one attendance


one project


"sign-in"


and three


sheet.


others


Parent


(totaling


education


33.3%)


was


the main


discussed


provision


some


form


of parent


education


although


one of the three


was not able


to provide


the parent


education


services


it had


proposed.


parent


education project


included


as a portion


of its curriculum


the preparation


of non-English-speaking


parents


to become


involved


in community

to the school


and school

Another


activities


and to provide


report mentioned


a needs


volunteer

assessment


services

and a third


report


indicated


parent-community members


who had


taken


college


courses


through


project


sponsorship


as previously mentioned


under


staff


development.


Summary


of Results


for Question













the State

adequate


of Florida?"

repres entat ion,


Implied

so that


in this


question


question


is the notion


might


be rephrased


"which


context


process


variables


are most


frequently


represented


in the Title


State


of Florida?


project

" The


summative


ethnic


evaluation


composition


reports


of the


within


community was


most


frequently mentioned


context


variable


followed


in descending


order


site


selection


in the local


school


context,


project


student


population


in the description


of the distri


district


contributions


in the available


resources,


and geographic


location.


languages


represented


in the student


population


were


most


frequently


reported


program


characteristic,


followed


descending


order


identification


program


emphasis,


project


type,


entry


and exit


criteria,


staffing,


program


type,


size


program,


student


characteristics.


Materials


were


the most


frequently mentioned


process


variable,


followed


in descending


order


staff


development,


record


keeping,


curriculum, and


implementation


evaluation.


Question


Evaluation Models,


Designs,


and Reports


Although


some


dictionaries


attribute


synonym


status


to "model"


and "design,


" they


are considered


as two quite


separate


entities


the specialized


field


of evaluation.


It would not


a gross


exaggeration


that


there


are as many


evaluation


models


as there


are theorists


in the field.


label


model


identifies


a philosophically


- .


--~~~~~~~~~~~ ~ ~ ~ ~ ~ ~ ~ ----I. -a. --- S--a -* ,--~-


,,,~


C


I













formed


"a basis


against


which


assess


the quality


of different


kinds


of instructional


sequences"


23).


Question


attempted


to discern whether


the procedures


used


in evaluating


Title


bilingual


pro grams


in Florida


were motivated


identifiable model


conceptualizations


or eclectic


or random


actions.


An evaluation


design


identifies


the "conditions


and schedule


under which


Scriven,


measures


Stufflebeam,


are taken


1983,


or the data


122).


coll


Seldom are


ected"


bilingual


(Madaus,


evaluators,


working


in the


context


of local


school


districts,


able


to impose


experimental


esign'


requisite


of randomization


to sample


and control


group


selection.


This


second


question


attempted


to identify


describe


the actual


conditions


under which


evaluation


data


were


collected.


Question


asked


"what


are the characteristics


of Florida


Title


VII evaluation models,


designs,


reports


in terms


of information


coverage,


content,


were examined


and procedures?"


for such


information


evaluation


as authorship,


reports


information


themselves


coverage,


procedures,


form


and style


of presentation.


Data


from


the 12 final


evaluation


reports


which


were


submitted


to the meta-evaluation


were


analyzed


and the results


are presented


in the order


of evaluation


reports,


evaluation models,


and evaluation


designs.


Evaluation


Reports













to the hiring


an external


evaluator


identified her


as "she,


while


name


of another


evaluator


appear


in a meeting


agenda


placed


in the appendix.


From


nine


reports


(75%)


which


did identify


the evaluator,


the following


information


was available:


five


reports


(41.7%)


were


completed


Ph.D.


three


reports


identified


evaluator'


a team


company name


two Ph. D


and addre


two (16.


one


report


of the nine


(8.3%)


Title


was


completed


VII districts


which


identified


name


their


evaluator


hired


the same external


evaluator,

the evaluate


and, finally,

or for the tw


one named


to districts


evaluator was


which


also


not submit


entified as

final


evaluations


to the meta-evaluation.


Three


Title


VII projects


(25%)


out of the total


sample of


12 used


internal


evaluators,


8 projects


hired


report


3%).


external

Only t


evaluators


wo reports


, and

(16.7%


no information

) indicated th


appeared


e point


in 1

entry


of the evaluator to


the project,


March


and April,


well


after


projects


been


placed


in operation.


The final


evaluation


report


submission


date was


provided


seven


reports


(58.3%).


Final


evaluation


reports.


An informative


introduction


to the


final


evaluation


report


giving


such


information


as number


of students,


staff,


curriculum,


type of


treatment,


etc.


appeared


in eight


reports


(66.7%)


while


some


information


such


as purpose


and curriculum was


provided


in one


introduction


8.3%).


final


reports


were prefaced,


I














Five


reports


clearly


stated


purpose


of the evaluation,


(8.3%)


alluded


to a


purpose e,


while


six reports


(50%)


address


purpose.


The obj


ectives


of the evaluation


were


specified


two of the


reports


which


formalized


a purpose


statement.


Information


coverage was


fairly


comprehensive


in 10 (83.3%)


the 12 final


evaluation


reports


with


most


reports


presenting


data


a broad


range of


program objectives


and activities.


One evaluation


report (

provided


8.3%)


described


according


five different


to LEP student


instructional


characteristics


components


the majority


of evaluation

instructional


information

components.


presented

Another


pertained

evaluation


to only

report


one

(8.3


of the five

1 addressed


only process

of the produce


variables.

t evaluation


response


the district


an inquiry

volunteered


regarding


that


a copy


no further


evaluation


had been


done


for the


year


1984-85.


following


topics


covered


in Title


VII evaluations


are presented


in descending


order


of frequency:

development,


cognitive


parent


achievement--prespecified


participation/community


objectives,


involvement


staff


activities


in relation

materials d


to goals


development,


nmd objectives,

management, s


affective


;taff


prespecified


performance


goals,


and attitudes,


and curriculum


development.


Seven

evaluator


of the 12 final

recommendations.


evaluation


while


reports


reports (4


(58.3%)


1.7%)


closed with


provided


one













providing


an abstract


or separate


summary


of results.


mentioned


in two reports


(16.7%)


that


the parent


advisory


council


would


review


the final


evaluation


report.


Evaluation Models


An objective-ba


sed evaluation


model


was evident


from


of the 12


final


reports


a goal-based


model


was evident


in 1


report


3%).


Results


were


presented


in these


reports


following


the statement


of the


objective


statement


or the goal.


of the evaluation


An additional


question


two reports


followed


presented


the results.


report


presented


both


the objective


or goal


and the evaluation


question.


Specific


evaluation


questions


were not


stated


in 10 reports


(83.3%).


The process-product


formative- summative


model


model


was in evidence


was mentioned


in five


reports


in two reports


7%).


7%),


while


seven


reports


(58.3%)


mentioned


final


or summative


only.


Implementation


evaluation


was


mentioned


in three


reports


(25%),


one objective-based/


final,


one question-based/process-product,


and the goal-based/process-


product


report.


Evaluation


Designs


Control


groups.


Randomization,


the requisite of


a true


experimental


design,


was not accomplished


in any


of the 12 projects,


although


addressed


two reports.


A control


group was


mentioned


in one


report


as having


been selected


randomly from


those


LEPs


whose


parents


was


was













training.


The evaluator


treated


this


as one of Campbell


and Stanley'


(1963)


true


experimental


designs--a


pretest-posttest


control


group


design.


evaluator'


treatment


does


not qualify


because


there


was no randomization


however,


appears


to qualify


as one of


Campbell


and Stanley


s (1963)


quasi-experimental


designs--the


nonequivalent


control


group


design.


Had the evaluator


used


non-


equivalent


control


group


design,


he/she


would


have


had to deal


with


the problems


of self-selection


as a factor


in the control


target


group


formation


regression


to the mean,


since


both


groups


were


selected


originally


for lack


prog


ress


and difficulty


in passing


the Statewid


Student


Assessment


Test


SSAT)


A second


report


mentioned


that


the school


sites


to be visited were


selected


at random


but in


letter


to the district


principles


included


an appendix


to the


report


was stated


that


schools


were


selected


from


those


not visited


previous


year.


Comparison


groups.


mos t


frequently used


comparison


was the


norm


referenced


design.


Eight


projects


relied


either


in total


or partially on


norm


referenced


design.


The national


norms


supplied


test


publishers


was the rule,


but two projects


(16.7%)


reported


use of locally


established


norms.


project


reported


a comparison


between


design,


LEP target


all students


LEP non-target


in Lau categories


students.


through


Unfortunately


were


for the


identified as


.3 I *-. I r -~--! - -: 4- Sr C... ~-


Cr T rl%


,cl, n


/ ~ Lr t I-IBr t i1


*


I


*V *


rl _A


I ru; rr













Another


serious


error


for the design


occurred


because


all LEP


students,


target


and non-target,


received


treatment


although


target


student s


received


"primary


attention.


A longitudinal


comparison


was discussed


in three


reports


but employed


only


one project


which


compared


1983-84


project


scores


with


1984-85


scores


on the


Comprehensive


Test


of Basic


Skills.


Unfortunately,


the evaluator


questioned


the reliability


of the 1983-84


scores,


positing


too much


assistance


from


the aides.


A third


proj ect


compared


scores


those


students


who were


in their


second


year


of proj ect


instruction


with


those


students


were


in their


first


year


of project


instruction


for each


grade


level.


Comparison


of student


progress


with


predetermined


mastery


level


was employed


nine


proj ects


75%)


using


proj ect


developed


and published


criterion-referenced


tests.


Data


collection:


Tests.


Although


an evaluator may


collect many


types


of data,


both


quantitative


and qualitative,


from


numerous


sources,


the most


frequently


relied


upon


data


were


student


achievement


test


results


collected


on a pretest-posttest


basis


from


target


students


using


a norm referenced


model.


Since


the State


of Florida


legislation


establishing


or regulating


bilingual


education


and since


evaluators


are usually not


hired


until


a project


has been


funded and


operation,


it is quite


frequently


the district


personnel


write


Title


VII proposals


who make


critical


deci


sions


regarding


- I -a- -I - -a -J -.*. fl n


r "
-n* in- j- ^/


.- an- c?


* T j _


IlnrC+


t


__


1













There


does


appear


to be much


agreement


in the sample of


reports


regarding


achievement


test


instruments


used.


Three


projects


25%),


all resident


in the


same


school


district,


administered


California Achievement


noted


use


Test


of the Stanford


two final


Achievement


evaluation


Test,


reports


two (16


(16.7%)


reported


results


each


from


employed


the Comprehensive


the following


Test


tests:


of Basic

Brigance


Skills,


Diagnostic


one proj ect

Inventory of


Basic

System


Skills,

Tests,


Diagnostic


IDEA


Science


Reading


Proficiency


Research


Survey,


Test


, Language


Associates


Stanford


Early


Assessment


Achievement

Achievement


Management


Series,

Test, a


Stanford


Lnd the


Wide Range


Achievement


Test.


There


is agreement


on one test,


Statewide Student


Achievement


Test


(SSAT).


All Florida


students


have


been


in English-speaking


schools


for two


years


and who


are in


the 3rd, 5th,


and llth


grades


are required


pass


the Statewide


Student


Achievement


Although


there


Test


is wide


entrance


variation


to the next


grade.


in instruments


used


procedures


bilingual


for screening


proj ects


are


students


fairly


entrance


standard.


into


Students


Title


and parents


screened


on enrollment


regarding


the language of


the home


and the


languages


of the student.


When


a language


other


than


English


listed,


the student


usually undergoes


an individual


oral


interview


followed


a test


of language


dominance


and quite


frequently


a test


a- -


are


- /- -i. *


* 1' 1 r */













Crane


Oral


Dominance


Test


Bilingual

Broward C


Syntax Measure


countyy Oral


Assessment


Dade


County


Aural


Comprehension


Dade County


Oral


Comprehension


Dade County Comprehensive


Test


Dade County


Elementary


Language


Development


Dade County


Test


of Language


Development


(Receptive)


Dade


County


Secondary


Placement


Test,


ESL 7-12


Language

Language


Assessment

Assessment


Scales

Battery


Language

Language


Assessment

Assessment


Management

Management


System

System


Diagnostic

Placement


Tests

Indicators


Brigance


Diagnostic


Inventory


of Basic


Skills


Woodcock


Language


Proficiency


Battery


IDEA


Proficiency


Test


Most


pro ects


used


district-developed


pupil


progression


plans,


continuums


, or published


language


management


programs


to record


student


accomplishment


of specific


skills.


The Broward


County


Pupil


Progression


Plan


was used


three


proj ects


and each


of the following


were


used


at least


one project:


Highlands


County Minimal


Skills


Screening


Instrument,


Language Assessment


Management


System,


TflFA


nrl1


T.ano'irna Mainae Pment


Prn-ram-













(Intermediate


and Senior)


and the Dade


County


Hispanic/Anglo


American


Cultural


Attitude


Inventory


Test.


districts


used


the Bilingual


Vocational


Oral


Proficiency Test


with


older


students


received


vocational


counseling.


Bilingual


exceptional


students


were


tested


with


following


instruments:


Leitner


International


Performance,


Stanford


Binet


Spanish/English


Bender Gestalt


Spanish/Engli


System


of Multicultural


Pluralistic Assessment.


Five


proj ects


developed


their


own criterion


referenced


tests


(CRTs)


to meet


special


project


needs.


They were


the following:


Student Academic/Behavior


Assessment


Inventory


State


Assessment


Tutoring


Skills


Parenting


Skills


School/Community/Cultural


Bilingual

Computer


Relations


Education

Literacy


Criterion


Reference


Test


Language


Proficiency


(proposed


to be validated by the


Dade County


Aural


Comprehension


test


but validation


not accomplished)


District produced Math


Criterion


Referenced


Test


Transitional

Transitional


Bilingual

Bilingual


Listening

Speaking


Skills

Skills


County Math


Greek


System


Proficiency


Program--Greek/English

Tests


n S.. C. n *n -


-- m i -


I


AA.


-ll_ 1 -- r -













Most


tests


were


administered


on a pretest-posttest


basis,


although


four projects


(33.3%)


reported


at least


one posttest


only


test


administration.


Once


passed


the "short"


fall-spring


first


year,


most f

showed


inal


evaluation


a marked


reports


preference


which a

spring


dressed


spring


testing


schedules


testing with


only


one project


8.3%


reporting


fall


spring


testing.


Data


collection:


Questionnaires.


Although


tests


accounted for


the bulk


of data


results


presented


in the sample of


12 Title


evaluation


reports,


75% (9)


of the final


reports


presented


results


from at


presented


least


one


results


questionnaire.


questionnaires


Eight


of the nine


to staff,


both


reports


bilingual


mainstream,


while


only


three


reports


(25%)


each


presented


results


from


students


or administrators


and only


two reports


presented


results


from


questionnaires


to parents.


Data


collection:


Documentation.


A multitude


of information


about


Title


VII projects


can be gleaned


from


various


inclusions


to the


appendices


the final


evaluation


reports.


Most


of the material


included


to support


claims


of meetings


held


, parent


involvement,


such


things


as staff


workshop


titles


and dates


newspaper


clippings


public


cizing


student activities.


Unfortunately,


project


evaluators


in general


not make


reference


in their


comments


to the interesting


information available


at their fingertips


nor to other


appropriate


^. .-:". 1..1-.1.-. ^.... ^ .n a n a--- ^ n


was


Life, 1- < *


^*1*^ *


,,,,:1,, f


nHnm n~


nn~


*k













rate which


was wrongly


described


on the following


page


as the


drop-out


rate.


One evaluation


a first


year


grant


recipient,


which


included


an implementation


evaluation


as part


of the final


report


made


excellent


use of documents.


The evaluator


examined


team


teacher-aide


lesson


coordination


sheets


and individual


aide'


lesson


plan


books


for evidence


along


with


aide


team


teacher


questionnaires


to support


the project'


claim


that


there


was coordination


of teaching


effort


in content


classes


between


the Title


VII project


and the


mainstream


classroom


teachers.


Other


documents


mentioned


in that


particular


evaluation


report


were


travel


vouchers,


the district'


audio-visual


catalog,


purchase


orders,


invoices


for consultants


and for bilingual


psychoeducational


evaluations.


No reports


looked


at suspensions,


Data


disciplinary


collection:


records,


Interviews


or drop-out


observation.


rate.


Four reports


presented


data


from


face-to-face


interviews


with


mentioning


phone


interviewing


as well.


Administrators


were


subjects


face-to-face


interviews


in all four


evaluations


while


staff


in general


provided


interview


data


in two reports


(16.7%).


No evaluation


members.


reported


Direct


interviewing


observation


students,


of the actual


parents,


functioning


or community

of the Title


VII project


in its various


classroom


sites


was reported


in only


projects


with


evaluators


not reporting personal


observation


_ -


one


.













Data


analysis.


None


of the final


evaluations


reported


sophisticated


statistical


analyses.


majority


of pretest-posttest


data


were


analyzed


for central


tendency


and significance.


Seven


projects


reported pretest-posttest


means


and six (50%)


tested


the significance


of the difference


between


means


using


a form


t-test.


One of


seven


projects


used


an analysis


which


calculated


the median


observed


and expected


scores


grade


level


on their


district'


square.


pupil


Four


progression


proj ects


plan,


testing


reported


significance


gains


with


by percentage


of students


increasing


"percent


posttest


increase.


score


over pretest


" One of the


projects


score


added


labeling


an additional


the column


column


labeling


it the percent


of significant


increase


when


the t-test


significant.


Most


reports


sent


data


on variability.


project


level


second


presented mean


for a


project


score,


project-developed


presented


standard


criterion


separate


deviation,


referenced


analyses


range


language


1983-84


per grade


test.


and 1984-85,


identifying posttest


grade


equivalent


mean


scores


and standard


deviations


grade


level


and subtest


for program,


non-program


former program students


on the Comprehensive


Test


of Basic


Skills


(CTBS).


because of


It was


small


reported


sample


that


size.


significance was


last


screened


of the three


visually


projects


which


addressed


variability presented


scale


score means,


standard


S
-1-. - -


-a-1n-rn ant\ cti t


was


c? 1-^


HFMJ~k CF- 4f





~ 41 1


rr n*














County


Aural


Comprehension


Test


using


Spearman'


(rho)


Rank


Order


Correlation.


Finally


projects


(50%)


calculated


data


using


percentages.


Of the six


proj ects


one reported


percentages


mean


ratings,


exclusively


range,


and three


as their only


tool


projects


for stati


used


stical


percentages


analysis.


Summary


of Results


for Question


Question


asked,


"what


are the characteristics


of Florida


Title


VII evaluation models


, designs,


reports,


in terms


of information


coverage,


content,


procedures?"''


The majority


reports


organized


presentation


of evaluation


results


statement


proj ect


objectives


or by


instruments.


When


a model


could


be identified,


process-product


model


was used most


frequently,


followed


summative- format ive


or just


summative


model.


most


frequently used


design


was the


comparison


group


norm


referenced


Since


design,


no project


followed


successfully


comparison

employed ra


a criterion


ndomization,


reference.


no true


experimental


designs


were


used,


although


one project


design


qualified


as a quasi-experimental


non-equivalent


control


group


design.


Although


there


was wide


variation


in the


testing


instruments


that


were


employed,


mos t


instruments,


normed


and criterion


referenced,


appeared


to have


poor validity


and reliability


for the LEP student


population


curriculum


of the Title


VII bilingual


projects.


Questionnaires


were


- -


- a -








J


tn


I-


1


~L ,, L-L













Authorship of


the final


evaluation


reports


was not consistently


indicated,


and the evaluators


not provide


information


to establish


their


credibility.


The reports


were


uneven


in quality,


evidencing


weaknesses


in description


context,


program


characteristics,


process


variables


statement


of evaluation


questions,


employment


consistent


model


, employment


of adequate


data


collection


and analysis


methods,


and presentation


of insightful


evaluator


recommendations.


Question


Professional


Standards


for Evaluation


Meta-evaluation,


according


to Stufflebeam


(1974a)


should


address


the merit


of evaluation


efforts


to provide


information


accountability


past


evaluation


work.


Meta-evaluation


should


assess


the importance of


evaluation


objectives,


appropriateness


evaluation


designs,


the adequacy


of data


analysis


and the quality


and importance


of evaluation


results.


maj or


concern


to the


meta-evaluator


is the selection


appropriate


criteria


for judging


the meta-evaluation.


The Joint


Committee


on Standards


for Educational


Evaluation,


headed


Daniel


Stufflebeam and


sponsored


12 professional


organizations,


published


30 standards


for good


evaluation


practices


(Joint C

together


committee

form th


on Standards


e basis


, 1981).


for a working


These 30 standards


philosophy


taken


of evaluation,


defining


principles


that


should


guide


govern


evaluation


- A- -A-- -, -


rr -


n1f lI1l


an a nrrnr nt A


tt In.-,


S-1,n4-


r L.._J 11- -__














important


to ascertain


the quality


of evaluations


in relation


each


of the standards"


listing


of the 30 standards


be found


in Appendix


explicit


intent


of this


third


question


to ascertain


quality


of 1984-85


Florida


Title


VII final


evaluation


reports


relation


to each


professional


standard.


Question


asked,


Florida


Title


VII final


evaluation


reports


meet


the current


accepted


professional

the Joint Co


standards


mmittee


of good


Standards


evaluation pra

for Educational


ctices,


as detail


Evaluation


(1981),


Standards


for Evaluations


of Educational


Programs,


Projects,


Materials?"


This


analysis


follows


the published


order


grouping


of the 30 standards:


Utility


Standards


, Al-A8;


Feasibility Standards,


B1-B3;


Propriety


Standards,


C1-C8:


and Accuracy


Standards,


D1-D11.


Utility


Standards


following


eight


standards


were


designed


to meet


the practical


information


needs


of given


audiences.


Al--Audience


identification.


This


standard


was designed


ensure


that


various


individuals


groups


are involved


or affected


the evaluation


are identified


so their needs


will


represented


in the evaluation


and so they will


receive


feedback


from


the evaluation


results.


sample


of 12 evaluation


reports


meet


this standard.


was mentioned


question


only


two reports


-n~.n .. l *I 1


- r .t -


can


___~r___ __


t -_ _


1


~I


L.


-A I













from


the evaluation


report.


In the remaining


10 reports


the question


of audience was


not addressed.


A2--Evaluator


credibility.


It is


the responsibility


of the


evaluator


to provide


the client


and the reader with


a statement


his or her qualifications.


Such


documentation


should


establi


integrity


substantive


competence


knowledge,


in terms


experience


training,


which


technical


qualify


skills


him or


perform


credibility


the evaluation.


in the final


standard


report


of establishing


was not met.


evaluator


was reported


question


, three


reports


(25%)


not identify the


name


of the


evaluator,


nine


reports


not identify


a title,


company


name,


or address.


None of the reports


provided


a statement


regarding


the qualifications


of the evaluator


beyond


notation


a Ph.D.


after


the evaluator


s name


in five of the


reports.


evaluator


did,


however,


provide


a statement


certifying professional


and ethical


treatment


of data


and interpretation


results.


Standard


was not


met by


the sample


of 12 final


evaluation


reports.


A3--Information


scope


and selection.


This


standard


addresses


comprehensiveness


and selectivity


of the data


collected


answer


evaluation


questions


and meet


the information


needs


of the


various


identified


audiences.


In order


to meet


this


standard


unless


a goal


free


evaluation


model


is specified,


it is


a requisite


that













random


broad


collection


spectrum or


of information


appropriate


test


the data


results


are that


no matter


are collected.


Given


this


premise,


must


be judged


that


the sample


of 12 final


evaluatio

coverage


not meet


was judged


the standard,


as fairly


although


comprehensive


information


in question


Data


reported


in question


confirmed


that


audiences


and their


information


needs


were


not identified,


purpose


for the evaluation


was not


stated


in 50% of the


not formulated


in 83.


reports,

3% of the


and specific

reports, al


evaluation


though


questions


66.7% of


were


the reports


substituted


project


objectives


for evaluation


questions.


A4--Valuational


interpretation.


This


standard


involved


the value


interpretations


made


on the significance


of the information


collected


in the evaluation,


identifying


who made


these


value


interpretations


and on what


a Title


basis


the interpretations


VII project,


for example,


were


made.


it would


In the perspective


a serious


error


assume


that


the objectives


of the bilingual


teachers


and staff


were


the only values


that


needed


to be taken


into


account,


without


considering


the value


judgments


parents,


administrators,


mainstream


teachers


either


have


LEP students


a portion


each


day,


or who


receive


students


when


they


exit


program.


Unless


a decision


to the


contrary


is made


with


the client,


is usually


the evaluator


takes


all of the competing


values


into


account,


offers


alternative


- r ...-


1


* J


1. m


r"














able


to discern


that


was the evaluator


who made


the value


judgments


in all 12 of the final


reports.


However,


only


seven


(58.3%)


of the evaluators


provided


recommendations


based


on the value


interpretations


they


placed


on the data


collected,


only


one (8.3%)


the evaluators


and only


suggested


two (16.


alternative


of the evaluators


bases


for interpreting


issued


specific


findings,


recommendations


to the projects


they


evaluated.


one instance


the evaluator


persisted


in making


a value


judgment


when


attrition


reduced


treatment


group


from 41


subjects


completing


the first


cycle


to 10 of


those


same


subjects


completing


cycle


two.


Data


were


collected


on the


total


group


and a valuation


interpretation


was attempted


even


though


75% of the subjects


not received


the full


treatment


in cycle


two.


Finally,


at least


of the final


reports


were


based


on the value


judgments


of the


project


staff,


without


consideration


of the values


other


relevant


audiences


such


as the administration,


mainstream


teachers,


or parents.


For these


reasons


it must


be judged


that


standard


was not met.


AS--Report


clarity.


The final


evaluation


report


should


describe


project,


procedures


context


findings,


in which


it functioned,


and recommendations.


purposes


report


should


written


a concise


narrative


style,


organized


consistently,


with


appropriately


captioned


and understandable


tables


and charts.













and tables,


program,

purpose,


good


contextual


documentation

and evaluation


information,


of implementation,


questions.


thorough description


statement


Unfortunately,


of evaluation

the individual


reports

report


were

with


all lacking

the most pro


in some


fessional 1


critical


looking


elements.


charts,


For example,


graphs,


and tables


also


the most


questionable


data.


A thorough


presentation


contextual


variables


program


description


was discussed


under


question


Four


report s


(25%)


exhibited


multiple


problems,


from


careless


errors


to questionably


inflated


numbers


of subjects


actually


receiving


treatment


to the basing


a case


success


on a test


reported


as significant


at the


.05 level


in the summary when


narrative


and table


presenting


the data


pages


earlier


clearly


showed


Clarity


that


the difference was


was not


met by


"not


the sample


significant.


of Title


Standard AS,


VII final


Report


evaluation


reports.


A6--Report


dissemination.


A negotiated


dissemination


plan


specifying


editorial


control,


audiences


for the


evaluation


results,


and formats


for presentation


of results,


and rights


to release


results


was recommended


the Joint


Committee.


was also


the position


the Joint


Committee


that


special


effort


be made


the client


evaluator


to inform all


"right-to-know''


audiences


and that,


in the


cases


of withholding


of evaluation


results


or misrepresentation


1














audiences


entitled


are defined


to be informed


about


Joint


Committee


the results


as those


of the evaluation


the following


reasons:


they


are the client;


they


commissioned


evaluation;


they


bear


legal


responsibility


for the


project;


they


funded


the project


through


taxes


or gifts


time


or money;


they


supplied


data;


they


are stakeholders


such


as staff,


parents,


students,


and representatives


of the


mass


media.


Although


the State


of Florida


has enacted


"government


in the


sunshine


laws"


which


validate


statute


the public's


right-to-know,


this


standard


was frequently


both


passively


and actively


ignored


Florida


Title


VII project


evaluations.


The collection


of final


evaluation


reports


for the meta-evaluation


a perfect


example


both


passive


and active


ignoring


of the standard


and the law.


sample


of 12 out


of 14 final


evaluation


reports


were


ultimately


collected


over


a five-month


period.


One project


reported


that


they


had not received


a copy


of the final


report


by' May


1986


five


months


after


report


was due in the U.S.


Department


of Education


Office


of Bilingual


Education


and Minority


Language


Affairs


(OBEMLA).


The 14th


project


finally


requested


in writing


not to be included


the sample.


No final


Two projects


evaluation


(16.7%)


reports


mentioned


identified


sending


right-to-know


the reports


audiences.


to OBEMLA













Joint


Dissemination


Review


Panel.


Standard A6,


Report Dissemination,


was not met by


the population


of 14 Title


VII final


evaluation


reports.


A7--Report


timeliness.


Timeliness


is usually


defined as


providing


delivery of


the final


report


so that


the information


be used


in planning


and decision making.


The Title


VII fiscal


year


for 1984-85


extended


from October


1984


through


September


1985.


Although


OBEMLA


require


final


evaluation


reports


to be


submitted


until


December,


1985,


those


reports


that


were


submitted


to the projects


in September


and October


afforded


most


opportunity


for being


used


for planning


and decision making


next


grant


year


especially


since


the school


year


in Florida


began


in August,


1985.


seven


reports


which


included


a submission


date were


received


prior


to the December OBEMLA


deadline


and five


of the


seven


reports


arrived


a timely manner


for planning


and decision making


purposes


one in April


(process


evaluation


only),


one in August,


one in


September,


two in October.


Two reports


arrived


in early


November.


Five


reports


(41.7%)


of the sample


of 12 reports


received


meta-evaluation


not indicate


a submission


date.


One of


the five


reports


which


did not provide


a submission


dat e


had not


been


received


project


by May


the meta-evaluator


, 1986,


in time


was submitted


to be received


by May


by the


30th.


project


of the two


can













report


by May


1986,


and the project


which


requested


in writing


not to be included


in the sample,


all employed


same


evaluator.


Since data


were


available


on the timeliness


of only


58.3%


of the


proj ect


final


evaluation


reports


the standard


was


not met,


even


though


five of


the seven dated


reports


did arrive


in time


planning


and decision making.


A8-- Evaluation


impact.


Impact


means


the utilization


of evaluation


results


for decision making


and program planning;


the influence


evaluation


results


exert


on the project


that


was evaluated.


In most


cases


evaluation


results


that


are submitted without


recommendations


the evaluator


have


little


impact


on program


planning


and decision


making


Five


reports


include


evaluator recommendations.


None of


the final


evaluation


reports


addressed


plans


for the evaluator


to act as change agent


or specific


steps


for evaluation


utilization.


Standard A8,


Evaluation


Impact,


was


met by


sample.


Feasibility Standards


The following


three


standards were


designed


to ensure


that


evaluation


process


would


be realistic,


practical,


diplomatic,


cost


effective.


Bl--Practical


procedures


The


activities


involved


in the


evaluation need


to be realistic


for the context


of the evaluation


setting


and should


be carefully planned


to avoid


disruption


to normal


-- ---













functioning


at which


evaluator was


hired,


it is impossible


most


cases


to determine who


was responsible


for data


gathering


procedures,


which


are potentially the


most


disruptive


evaluation


activity.


Field


notes,


final


evaluation


reports,


are the


evaluator'


usual


repository


detailing


procedures


and commenting


on the


execution


of each


step


of the evaluation.


Since


field


notes


were


not part


of the meta-evaluation


no additional


data


were


specifically


collected


to address


this


standard,


no judgment


can be


rendered.


B2--Political


viability.


The early


identification


of official


and informal


power


structures


and special


interest


groups


gives


evaluator


the option


to make


constructive


use of political


forces


avoid


political


conflict,


attempts


to bias,


or misapply results.


identifying


the right-to-know


audiences


and giving


them


opportunity


express


positions


concerns


regarding


the evaluation


the evaluator can


assurance


demonstrate


an impartial


integrity


evaluation.


and give


all stakeholders


As was mentioned


in A6,


report


listed


right-to-know


audiences


as such,


although


administrators,


teachers

83.3% of


and parents


the projects


were

and


asked

were i


to respond


interviewed


to questionnaires


in 16.7% of the projects.


The standard


was not met.


B3--Cost


effectiveness.


Evaluations


should


as economical













in the evaluation


effort.


This


standard


addresses


the situation


where a

budget.


in evaluator negotiates


In the Title


with


VII evaluations


a client


for an evaluation


in Florida,


the evaluator


was offered


a fixed


sum for his or her


time


expertise


which


predetermined


in the


project


proposal


budget.


The district


absorbed


such


costs


as the purchase


of testing


instruments,


test


administration


by district

facilities


personnel,


such


test


as district


scoring,

mail. S


xeroxing,


;ince


and the


the evaluation


use of district

is a requirement


of the grant


and the


expense


of the evaluator


is an expected


grant


budget


item,


the question


cost


effectiveness


was not addressed


projects.


This


standard


was not applicable


to the


Florida


1984-85


Title


VII project


final


evaluation.


Propriety


Standards


The next


eight


standards


deal


with


the legal


ethical


conduct


of evaluations


respect


for the rights


of human


subjects


and those


are affected


the results.


Cl--Formal


obligation.


This


standard


addressed


the written


agreement


between


client


and evaluator


and recommended


that


a copy


of the evaluation


contract


be appende


to the final


evaluation


report.


No mention


of contractual


agreements


was-


made


in or appended


of the final


evaluation


reports.


It is clear


that


this standard


not met,


but it


is not clear


whether


this


standard


is applicable


was


was














be unavoidable,


but if disclosed


openly


and honestly,


should


not compromise


the objectivity


or credibility


of the report.


such


disclosures


were


made


in the sample


of 12 final


evaluation


reports.


An objective


tone


seemed


to characterize


the majority


evaluation


reports


(75%)


with


no evidence


of evaluator


hostility


bilingual


education


and only


limited


evidence


a loss


of objectivity


25%)


in becoming


an apologist


for the project


(see


explanation


C7).


There


is always


potential


for a loss


of objectivity


in internal


evaluation


but the final


reports


submitted


strict


personnel


were written


a professionally


obj ective


tone.


Judgment


reserved


on this


standard.


C3--Full


and frank


disclosure.


A candid


and honest presentation


of all relevant


evaluation


findings


including


the evaluator'


judgments,


limitations


of the evaluation,


and implications


for the finding


recommendations


are requirements


of this


standard.


was


mentioned


in C2,


a majority of


the evaluations were written


an objective


tone,


but only


Seven


them


four reports


evaluation


discussed


reports


implications


addressed


limitations


and constraints.


provided recommendations


for the findings


none


and recommendations.


standard was


not met in the sample of


Title


VII final


evaluation


reports.


C4--Public


s right


to know.


Right-to-know


audiences


were


defined


the Joint


Committee


as those


are entitled


to be informed


about


*- r. i a


was


A.L 1


1


1


n


nn


1,,


* /













or gifts


of time


or money;


they


supplied


data;


they


are stakeholders


such


as staff


, parents,


students,


and representatives


of the mass


media.


No evaluation


report


identified


right-to-know


audiences.


The standard


was not met.


C5--Rights


of human


subjects.


rights


and welfare


of human


subjects must


be protected.


These


rights


are derived


from


the law


ethics,

rights


common


sense,


subject


courtesy.


the evaluator


Violation


to legal


of legal


prosecution


and ethical


or professional


sanctions

of human


Four


subjects.


final


evaluation


In three


reports


reports


(25%


parent


violated

sign-in


the rights

sheets


were


included


in the appendix


as proof


PAC activity.


The sign-in


sheets


contained


such


personal


information


as name


(signature),


school


name,


address,


phone


number.


One of


these


three


reports


included


copies


of grade


level


summary profile


sheets,


listing


name


of the


school


, aide,


grade,


year,


names


of all LEP students,


date


entrance


into


program,


pretest-posttest


raw scores


for language


proficiency


tests,


and pretest


stanines


for the achievement


test.


A fourth


final


report


included


a copy


of the bilingual


follow-up


form


for each


student


which


had been


placed


a mainstream


classroom.


This


form


listed


the student's


name


and the


mainstream


teacher's


comments


regarding


identifying


academic


defi


and social


ciencies


progress


assessing


as well


degree


as a narrative


of deficiency.


- .1 -I- -- -. ~ -4- ,~ -S ~


nfl r 1n


LAA4-


*


I


- __ -- -


t













and identifying


descriptions


been


white


out before


copying,


although


a summary


of the


relevant


information


would have


been


even better.


This


standard


was


not met.


C6--Human


interactions.


This


standard


addresses


the evaluator'


respect


for human


dignity and


worth


in interpersonal


transactions


with


those


who participate


in the evaluation.


Of particular


importance


a Title


effort


VII bilingual


to understand


evaluation


the social


is that


the evaluator make


cultural


values


every


of the participants


and the appropriate protocol


for school


visits


and staff


interviews.


Only


one piece of


data


appropriate


to this


standard


was included


the sample

informing

advising t


of final


them o

hem of


evaluation


f the evaluation'


the date


reports


a letter


purpose,


the evaluator would


to the schools


introducing


the evaluator,


on campus,


indicating who would


be observed


and/or


interviewed.


This


standard


could


be judged


as no further


data


were


collected


in the meta-


evaluation


for Standard


Human


Interactions.


C7--Balanced


reporting.


A balanced


evaluation


report


provides


fair


assessment of both


strengths


and weaknesses


a project.


report


should


be complete;


where


data


omissions


occur,


they


should


identified.


Weaknesses


should


not be deleted


to avoid


embarrassment.


This


standard


is especially relevant


to Title


VII project


reports


where


there


strong


competition


funding


and uncertainty


about


S-i


-L *


/ 4













anxiety rather

a feeling that


than

OBEML


providing

A would D


direction


refer


a public


improvement.

relations dc


There


cument


listing project

be aggregated n


accomplishments


nationally


and student


and presented


achievement


to Congress


that


could


at refunding


time,


rather


than an


assessment


merit


and worth,


strengths


and weaknesses,


with


recommendations


for proj


improvement.


In the sample


of 12


final


reports,


evaluations


(50%)


achieved


a fair measure of balance


in addressing


strengths


and weaknesses,


while


50% did


not address


weakness es.


A balanced


report


also


has to do with


the completeness


of the data


collection.


One project


with


five


components


presented


evaluative


data


on one component,


descriptive


data


on a second,


completely ignored


rest,


except


to combine


achievement


data


all five


components


together,


although


they


served


totally


different


student


populations


and the


treatment


was designed


for the specific


student


population


and therefore


differed.


No project


which


served


more

group


than


one language


although


this


group


practice


separated


achievement


presenting


data


data by

language


language

group i


recommended


by OBEMLA.


Finally


in order


for a report


to be balanced,


first-hand knowledge


of the project


sites


as they


actually


functioned


is a requisite.


Only


two evaluators


reported


personal


observation


data,


and only


four


evaluators


reported


face-to-face


interviews,


indicating


first


hand


knowledge


of the project


- .. --.-. --. 1 1 ,t


a 4- an n A


Ti, a e + en -ln -r A


.a .* en+ atmo


1.













evaluator,

expenditure


addresses


the maintenance


for the evaluation.


This


of accurate accounting


standard


records


is not applicable


to Florida


Title


VII projects


because


the evaluator


is paid


for his


or her time


and expertise


but the school


district maintains


control


of the evaluation


Accuracy


budget.


Standards


These


11 standards


were


designed


ensure


that


technically


adequate


information


is provided


for the assessment


of merit


and worth


project.


Dl--Object


identification.


The project


should


be described


enough


detail


that


unique


features


can be identified


and associated


with program


program


effects.


characteristics


Discrepancies


should be


between


noted


proposed


and special


and actual


assistance


students s

evaluation


should


results


be described.


related


Reported


to program


for question


characteristics,


were


project


meta-

type,


size


of project,


languages,


student


characteristics,


program


characteristics


exit


entry


criteria,


and program


emphasis.


Although


many


of the evaluations


an excellent


in describing


characteristic or another,

D2--Context analysis.


taken


as a whole,


Sufficient


the standard


contextual


variables


was not met.


should


addressed


a final


evaluation


report


so that


the reader


judge


the effect


of the contextual


conditions on


the project


and under what


one














community


characteristics,


and local


school


context.


Although


many


projects


did describe


certain


elements


context,


the standard


was not met.


D3--Described


purposes


and procedures.


The evaluation


purposes


and procedures


should


be sufficiently


described


so they


can be


identified,


replicated,


and/or


asses


sed.


The evaluation


purposes


identify

indicate


obj ectives


how the


data


intended


were


uses


gathered,


of results

organized,


while the procedures

analyzed, and reported.


Question


reported


the meta-evaluation


results


related


to the


description


of evaluation


purposes


procedures.


reports


did not address


the purpose of


evaluation


certain


procedures


proj ect


were addressed


reported


in about


specifically


same


on evaluation


number


procedures


reports


in the final


evaluation


report.


Procedures


have


been


addressed


a separate


report


of field notes


not included


in the meta-evaluation.


standard was


not met.


D4--Defensible


information.


Documentation


of the


sources


information


is addressed


in this


standard


so their adequacy may


assessed.


Sampling


procedures,


attrition


, copies


of proj ect-developed


instruments,


observation


and interview


schedules


notes,


lists


standardized


tests,


rationale


for selection


of instruments


use with


the sample


are all considered


in this


standard.


two reports