Citation |

- Permanent Link:
- https://ufdc.ufl.edu/UF00098341/00001
## Material Information- Title:
- An empirical examination of analysis of covariance with and without Porter's adjustment for a fallible covariate
- Creator:
- McLean, James Edwin, 1945-
- Publication Date:
- 1974
- Copyright Date:
- 1974
- Language:
- English
- Physical Description:
- x 59 leaves. : ; 28 cm.
## Subjects- Subjects / Keywords:
- Analysis of variance ( jstor )
Covariance ( jstor ) Educational research ( jstor ) Factorial design ( jstor ) Psychometrics ( jstor ) Sample mean ( jstor ) Sample size ( jstor ) Sampling distributions ( jstor ) Statistical discrepancies ( jstor ) Statistics ( jstor ) Ability -- Testing ( lcsh ) Dissertations, Academic -- Foundations of Education -- UF ( lcsh ) Foundations of Education thesis Ph. D ( lcsh ) Mathematical statistics ( lcsh ) - Genre:
- bibliography ( marcgt )
non-fiction ( marcgt )
## Notes- Thesis:
- Thesis -- University of Florida.
- Bibliography:
- Bibliography: leaves 56-58.
- General Note:
- Typescript.
- General Note:
- Vita.
## Record Information- Source Institution:
- University of Florida
- Holding Location:
- University of Florida
- Rights Management:
- Copyright [name of dissertation author]. Permission granted to the University of Florida to digitize, archive and distribute this item for non-profit research and educational purposes. Any reuse of this item in excess of fair use or other copyright exemptions requires permission of the copyright holder.
- Resource Identifier:
- 022828509 ( AlephBibNum )
14101091 ( OCLC ) ADB0705 ( NOTIS )
## UFDC Membership |

Downloads |

## This item has the following downloads:
empiricalexamina00mcle ( .pdf )
empiricalexamina00mcle_Page_71.txt empiricalexamina00mcle_Page_66.txt empiricalexamina00mcle_Page_04.txt empiricalexamina00mcle_Page_23.txt empiricalexamina00mcle_Page_59.txt empiricalexamina00mcle_Page_65.txt empiricalexamina00mcle_Page_05.txt empiricalexamina00mcle_Page_31.txt empiricalexamina00mcle_Page_40.txt empiricalexamina00mcle_Page_33.txt empiricalexamina00mcle_Page_22.txt empiricalexamina00mcle_Page_08.txt empiricalexamina00mcle_Page_50.txt empiricalexamina00mcle_Page_57.txt empiricalexamina00mcle_Page_32.txt empiricalexamina00mcle_Page_20.txt empiricalexamina00mcle_Page_13.txt empiricalexamina00mcle_Page_51.txt empiricalexamina00mcle_Page_07.txt empiricalexamina00mcle_Page_67.txt empiricalexamina00mcle_Page_42.txt empiricalexamina00mcle_Page_01.txt empiricalexamina00mcle_Page_11.txt empiricalexamina00mcle_Page_53.txt empiricalexamina00mcle_Page_35.txt empiricalexamina00mcle_Page_61.txt empiricalexamina00mcle_Page_64.txt empiricalexamina00mcle_pdf.txt empiricalexamina00mcle_Page_55.txt empiricalexamina00mcle_Page_39.txt empiricalexamina00mcle_Page_19.txt empiricalexamina00mcle_Page_15.txt empiricalexamina00mcle_Page_12.txt empiricalexamina00mcle_Page_36.txt empiricalexamina00mcle_Page_56.txt empiricalexamina00mcle_Page_03.txt empiricalexamina00mcle_Page_68.txt empiricalexamina00mcle_Page_43.txt empiricalexamina00mcle_Page_72.txt empiricalexamina00mcle_Page_45.txt empiricalexamina00mcle_Page_34.txt empiricalexamina00mcle_Page_54.txt empiricalexamina00mcle_Page_60.txt empiricalexamina00mcle_Page_44.txt empiricalexamina00mcle_Page_16.txt empiricalexamina00mcle_Page_41.txt empiricalexamina00mcle_Page_38.txt empiricalexamina00mcle_Page_27.txt empiricalexamina00mcle_Page_49.txt empiricalexamina00mcle_Page_58.txt empiricalexamina00mcle_Page_63.txt empiricalexamina00mcle_Page_37.txt empiricalexamina00mcle_Page_69.txt empiricalexamina00mcle_Page_06.txt empiricalexamina00mcle_Page_47.txt empiricalexamina00mcle_Page_26.txt empiricalexamina00mcle_Page_52.txt empiricalexamina00mcle_Page_14.txt empiricalexamina00mcle_Page_10.txt empiricalexamina00mcle_Page_25.txt E5SREUA0R_7BH8NS_xml.txt empiricalexamina00mcle_Page_18.txt empiricalexamina00mcle_Page_62.txt empiricalexamina00mcle_Page_28.txt empiricalexamina00mcle_Page_29.txt empiricalexamina00mcle_Page_21.txt empiricalexamina00mcle_Page_17.txt empiricalexamina00mcle_Page_48.txt empiricalexamina00mcle_Page_46.txt empiricalexamina00mcle_Page_30.txt empiricalexamina00mcle_Page_24.txt empiricalexamina00mcle_Page_09.txt empiricalexamina00mcle_Page_70.txt |

Full Text |

AN EMPIRICAL EXAMINATION OF ANALYSIS OF COVARIANCE WITH AND WITHOUT PORTER'S ADJUSTMENT FOR A FALLIBLE COVARIATE By JAMES EDWIN McLEAN A DISSERTATION PRESENTED TO THE GRADUATE COUNCIL OF THE UNIVERSITY OF FLORIDA IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY UNIVERSITY OF FLORIDA 1974 ( DEDICATION I dedicate this study to the late Dr. Charles M. Bridges, Jr., my teacher, advisor and friend. ACKNOWLEDGMENTS I wish to express my grateful appreciation to my committee members for their guidance in this study. Dr. William B. Ware, my chairman, suggested the topic and was a major influence in its development. Dr. James T. McClave was a constant source of assistance, both theoretically and editorially. Drs. Vynce Hines, William Mendenhall, and P. V. Rao all provided direction along with many helpful suggestions. This study could not have been completed without the spirit of cooperation which I encountered between members of the two departments involved. The late Dr. Charles M. Bridges, Jr. first encouraged me to enroll in the program and served as my chairman until his death. His counseling and guidance were a major reason for my successful completion of graduate study. The searching questions of my fellow students led to several worthwhile modifications and I wish to express my appreciation to them. I also wish to thank my wife, Sharon, who for the last five years, has held two jobs (wife and medical technologist), so that I might further my education. She has also been most understanding about the many hours spent away from home. TABLE OF CONTENTS Page ACKNOWLEDGMENTS . . . . . . . . . . . iii LIST OF TABLES . . . . . . . . ... .... .. . vi ABSTRACT . . . . . . . . ... . . . . . .viii CHAPTER I. INTRODUCTION . .. . . . . . . . . . . 1 The Problem of Comparing Groups of Differing Abilities . . 1 Methods of Comparing Groups of Differing Abilities . . . 3 Limitations Used in this Study . . . . . . . . 4 Procedures . . . . . . . ... . . . . 5 Relevance of the Study . . . . . . . . . 7 Organization of the Dissertation . . . . . . . 7 II. REVIEW OF THE RELATED LITERATURE . . . . . . . 8 Historical Review of the Problem . . . . . . . 8 Analysis of Covariance . . . .. . . . . . . 10 Analysis of Covariance with Porter's Adjustment for a Fallible Covariate . . . . . . . . 11 Summary. . . . . . . . . . . . . .. 14 III. PROCEDURES . . . . . . . . ... .... . 16 The Model. ......................... 16 Selecting the Reliabilities. ............... . 17 The Regression of Y on X . . . . . . . ... 18 Selecting Means. . . . . . . . . . . .. 20 Generation of Random Normal Deviates with Specified Means and Variances. . . . . . . . . ... 26 Analysis for Comparing the Selected Methods. . . . ... 26 IV. RESULTS. . . . . . . . . ... ....... 29 V. DISCUSSION . . . . . . . . ... ... .. . 41 Comparison of the Two Methods of Analysis. . . . . ... 41 Factors that Affect Alpha and Beta . . . . . ... 42 Page Predicting Alpha and Power .. ............. 43 A Direction for Future Research. . . . . . . . 46 VI. SUMMARY . . . . . . . . . . .. 47 APPENDIX . .................... ...... 49 Fortran Program Used to Perform Analysis . . . . 50 BIBLIOGRAPHY . . . . . . . . . .......56 BIOGRAPHICAL SKETCH.................. .......59 v LIST OF TABLES Table Page 1. CONDITIONS UNDER WHICH COMPARISONS WERE MADE WHEN THE RELIABILITIES FOR BOTH GROUPS WERE EOUAL. .... .24 2. CONDITIONS UNDER WHICH COMPARISONS WERE MADE WHEN THE RELIABILITIES FOR BOTH GROUPS WERE UNEQUAL. ... 25 3. FACTORIAL DESIGN FOR STANDARD ANALYSIS OF COVARIANCE WHEN MEAN GAIN OF GAIN GROUP WAS ZERO AND CRITERION VARIABLES ARE MONTE CARLO GENERATED ALPHA VALUES. . 28 4. FRACTION OF SIGNIFICANT F'S AND NUMBER OF TIMES F EXCEEDS F WHERE THERE WAS NO GAIN IN EITHER ROUP AND THE PRETEST RELIABILITIES WERE EQUAL. . . .. 32 5. FRACTION OF SIGNIFICANT F'S AND NUMBER OF TIMES F EXCEEDS F WHERE THERE WAS NO GAIN IN EITHER -ROUP AND THE PRETEST RELIABILITIES WERE NOT EQUAL. .... .33 6. FRACTION OF SIGNIFICANT F'S AND NUMBER OF TIMES Fs EXCEEDS F WHERE THERE WAS GAIN IN THE GAIN GROUP AND THE PRETEST RELIABILITIES WERE EQUAL. . . .. 34 7. FRACTION OF SIGNIFICANT F'S AND NUMBER OF TIMES F EXCEEDS Fp WHERE THERE WAS NO GAIN IN THE GAIN GROUP AND THE PRETEST RELIABILITIES WERE NOT EQUAL . . . . . . . . . . . 35 8. ANOVA SUMMARY TABLE USING MONTE CARLO GENERATED ALPHAS THE STANDARD ANALYSIS OF COVARIANCE AS THE CRITERION VARIABLES . . . . . . . .... ... . 36 9. ANOVA SUMMARY TABLE USING MONTE CARLO GENERATED ALPHAS FROM ANALYSIS OF COVARIANCE WITH PORTER'S ADJUSTMENT AS THE CRITERION VARIABLES. . . . . . . ... 37 10. ANOVA SUMMARY TABLE USING MONTE CARLO GENERATED POWERS FROM THE STANDARD ANALYSIS OF COVARIANCE AS THE CRITERION VARIABLES . . . . . . . .... 38 11. ANOVA SUMMARY TABLE USING MONTE CARLO GENERATED POWERS FROM ANALYSIS OF COVARIANCE WITH PORTER'S ADJUSTMENT AS THE CRITERION VARIABLES. . ... . . . .. 39 Table Page 12. COEFFICIENTS FOR THE LINEAR CONTRASTS USED WHEN RELIA- BILITY WAS SIGNIFICANT IN THE ANALYSIS OF VARIANCE. . 40 13. NINTY-FIVE PERCENT CONFIDENCE INTERVALS FOR THE EXPECTED VALUES OF ALPHAS UNDER SPECIFIED CONDITIONS . . .. 44 14. NINTY-FIVE PERCENT CONFIDENCE INTERVALS FOR THE EXPECTED VALUES OF POWERS UNDER SPECIFIED CONDITIONS . . .. 45 Abstract of Dissertation Presented to the Graduate Council of the University of Florida in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy AN EMPIRICAL EXAMINATION OF ANALYSIS OF COVARIANCE WITH AND WITHOUT PORTER'S ADJUSTMENT FOR A FALLIBLE COVARIATE by James Edwin McLean August, 1974 Chairman: William B. Ware Major Department: Foundations of Education The purpose of this study was to determine if analysis of covariance or analysis of covariance with Porter's true covariate substitution ad- justment or neither is appropriate for analyzing pretest-posttest educa- tional experiments with less than perfectly reliable measures. Monte Carlo techniques were employed to generate two thousand sam- ples for each of forty-eight sets of conditions. These conditions in- cluded six combinations of reliability, two levels of sample size, two levels of gain, and the equality or inequality of pretest means. Each sample was analyzed by both of the tested methods. The proportion of re- jections for each method of analysis and the number of times the F statistic for standard analysis of covariance exceeded that for analysis of covariance with Porter's adjustment were recorded. viii The hypothesis, "The sampling distributions of the test statistics from analysis of covariance with and without Porter's adjustments are the same," was rejected at the .01 level of significance using the sign test for each of the forty-eight sets of conditions. To gain further insight into how the two methods of analysis dif- fered, four factorial experiments were conducted using either the com- puter generated alphas or powers as the criterion variables. The first factorial experiment examined analysis of covariance when the mean gain was zero in both groups, thus, the computer generated alphas were used as criterion variables. The second experiment examined analysis of co- variance with Porter's adjustment when the mean gain was zero in each group. Again, computer generated alphas were used as criterion variables. The third and fourth factorial experiments examined standard analysis of covariance and analysis of covariance with Porter's adjustment when the mean gain in only one group was positive. The four factorial experiments included three factors. These were reliability at six levels, sample size at two levels, and the equality of pretest means at two levels pretestt means equal and pretest means not equal). In each experiment, it was found that unequal pretest means combined with low reliability were significant sources of variation. Additional study showed that unequal pretest (covariate) means com- bined with low reliability produced misleading results. More specifi- cally, when either method of analysis produced a significant F statis- tic and there was a mean gain of zero in both groups but the pretest mean of one group was less than that of the other, the adjusted post- test means indicated that the group with the higher pretest mean had the larger gain. Likewise, when there was a positive mean gain in the group with the lower pretest mean, the adjusted posttest means indicated the group with a mean gain of zero had the larger gain. The results of this study combined with the results of others concerned with analysis of covariance where the covariates are fallible point to the inadequacy of the technique as it now exists. These results should be kept in mind if the technique is to be used. x CHAPTER I INTRODUCTION Regardless of the theory or hypothesis being investigated, educa- tional researchers must, at some point, relate it to learning. One common element of most definitions of learning is that a change must take place in the learner (DeCecco, 1968, p. 243; Hilgard, 1956, p. 3; Hill, 1971, p. 1-2). Thus most investigations in education are concerned with identifying, measuring, and comparing these changes. In the past decade, analysis of covariance has become a standard procedure for comparing groups with different levels of ability. The basic derivation of this procedure requires the assumption that the co- variable does not contain errors of measurement (Cochran, 1957). In recent years, several researchers have recognized that this assumption is violated when.the covariate is a mental test score (Lord, 1960; Porter, 1967; Campbell and Erlebacher, 1970). Lord (1960) and Porter (1967) have proposed adjustments to the analysis of covariance procedure to use when the covariable is fallible, that is, when measurement error is present in the covariable. The focus of this study is to determine if analysis of covariance and analysis of covariance with Porter's adjustment are different for a fallible covariate and if either is appropriate under varying conditions of reliability, sample size, gain, and pretest mean. The Problem of Comparing-Groups of Differing Abilities The problem of comparing groups based on change manifests itself prominently when the variables are mental measurements. Most physical science measurements are made with the aid of some type of physical instrument. If an engineer is interested in the weight gain of a metal before and after a galvanizing process, instruments are available to measure the weight of the metal to within at least one microgram. On the other hand, if an educator is interested in the change in a student's I.Q. before and after taking part in an experimental program, an error of 5 points or more would not be uncommon. Based on the Wechsler Intelligence Scale for Children with a standard error of measurement of 5.0 (Cronbach, 1970, p. 222), an error of 5.0 I.Q. points or more would occur in approximately 32 percent of the measurements, assuming a normal distribution. If the actual change in I.Q. were near zero for an individual, it is very likely that the error of measurement would exceed the change itself. The problem is compounded further as the groups being compared often are not of the same ability level. Many of the recent programs featuring innovative teaching practices for compensatory education are available only for the most needy and the comparison group is then sampled from the general population of students (Campbell and Erlebacher, 1970). Programs such as Head Start and Follow Through have probably been the victim of tragically misleading analyses, such as in the Westinghouse/ Ohio University study (Campbell and Erlebacher, 1970). Personal contact with the evaluation of Follow Through projects has emphasized the magnitude of the problem. Warnings about the inappropriate uses of existing modes of analysis have come from several sources (Lord, 1960, 1967, 1969; Campbell and.Erlebacher, 1970). Remedies have been offered (Lord, 1960; Porter, 1967), but at this time there is no conclusive evidence to indicate these remedies are appropriate. Studies concerning the robustness of the analyses to violated assumptions have also been in conflict (Peckham, 1970). Methods of Comparing Groups of Differing Abilities Two general approaches are available to compare groups with dif- fering abilities. The first is to compare the average change in one group with the average change in the other group by means of a t test or analysis of variance. The second is to use analysis of covariance, where the pretest score is the covariate. There are at least three methods of computing change scores to be considered if the first approach is used. One method is to obtain raw difference scores by simply subtracting each pretest score from its corresponding posttest score. Another method is Lord's true gain (Lord, 1956) which was further developed by McNemar (1958). Basically, this method requires the use of a regression equation to estimate the "true" gain or change from pretest to posttest. A third method of measuring gain, the method of residual gain, was proposed by Manning and DuBois (1962). This method involves regressing the posttest scores on the pretest scores and obtaining a predicted posttest score for each subject. The indicator of change is the observed posttest score minus the pre- dicted posttest score. This method is.mathematically equivalent to the standard analysis of covariance (Neel, 1970, p. 30-31), the second approach. The second general approach is the standard analysis of covariance procedure found in many statistical texts (e.g., Kirk, 1968; Winer, 1971). One of the reported functions of a covariate is to adjust the treatment means of the dependent variable for differences in the values of corres- ponding independent variables (Hicks, 1965). Logically, it seems that by using the pretest scores of groups differing in ability as a covariate, the treatment means can be statistically adjusted for the differences in pretest means. A deficiency of this method is the fallibility of the covariate. A fallible variable is one which is not measured without error or with perfect reliability (Lord, 1960). Lord (1960), recognizing this deficiency in the use of analysis of covariance, derived an adjust- ment to compensate for the fact that the pretest scores were fallible. Porter (1967) modified Lord's procedure for more than two groups and empirically investigated the sampling distribution of Lord's statistic. Porter's solution has been suggested as an alternative to the standard -> analysis of covariance when the covariate is fallible. Limitations Used in this Study This study was designed to examine the characteristics of the two covariance methods within the framework of simulated situations which may occur in the analysis of a compensatory education project. Realis- tically, several limitations were imposed. Even with these limitations, the two methods of analysis were investigated under forty-eight combi- nations of differing reliability, sample size, mean gain, and pretest means. The study was limited to two groups, henceforth called the gain and no-gain groups. Investigating more than two groups would not only in- crease the number of possibilities appreciably but would not be in line with the objective stated previously, that is, comparing a compensatory education group with a "comparison" group. The levels of the reliabil- ities being investigated are .also limited to the range .50 to .90. Sel- dom do mental measurement instruments have reliabilities above .90 and instruments with reliabilities below .50 are generally considered in- adequate. The levels of.sample size were 10 and 100 subjects per group. These quantities are representative of small and large sample sizes en- countered in educational experiments. Furthermore, the true score and true gain variances will be fixed a priori with the true gain variance always four percent of the true score variance. These figures are empirically based on Follow Through achievement data in keeping with the aforementioned objective. Procedures The study compared the two methods of analysis under forty-eight sets of conditions. The two methods were analysis of covariance and analysis of covariance with Porter's adjustment for a fallible covariate. Three levels of reliability and two levels of sample size were used. The three levels of reliability will include situations where the re- liability of the protests are assumed equal and situations where they differ to more closely simulate compensatory education project evalua- tion. Each of these combinations of conditions was repeated for four separate cases. Case I is where there is no-gain in either group and both groups have equal pretest means. Case II is where there is gain in only one group (the gain group) and both groups have equal pretest means. Cases III and IV are with and without gain where the pretest means are different. For each of the forty-eight sets of conditions, two thousand samples were computer generated and analyzed using both methods of analysis and a .05 level of significance. Two thousand samples provide empirical estimates of the fraction of type I and type II errors to within .01 of their true values with ninety-five percent confidence. The samples were generated from an assumed normal population. The generation technique is one proposed by Box and Muller (1958) and modified by Marsaglia and Bray (1964). This procedure generates random normal deviates with a mean and variance of 0 and 1 respectively. These normal variables are then transformed to attain the specified means and variances. The IBM 370/165 computer of the North Florida Regional Data Center was used for the generation and analyses. Using these analyses, two research questions were examined: 1. Is there any difference between the sampling distributions of the test statistics of standard analysis of covariance and analysis of covariance with Porter's adjustment? 2. What factors affect each type of analysis? The first research question can be restated in terms of a null hypothesis for each set of conditions: The sampling distributions of the test statistics from analysis of covariance with and without Porter's adjustment are the same for each set of conditions. This hypothesis was tested for each reliability, sample size, gain, and pretest mean combination. The second research question was examined in four factorial experi- ments. Each factorial experiment included the factors of reliability at six levels, sample size at two levels, and equality of pretest means at two levels. The first experiment consisted of examining standard analy- sis of covariance with computer generated alpha values as the criterion variables. The second examined analysis of covariance with Porter's cor- rection also using computer generated alpha values as the criterion var- iables. The third and fourth factorial experiments examined analysis of covariance with and without Porter's adjustment respectively, using computer generated powers as criterion variables. Relevance of the Study This study is designed to compare two recommended methods of ana- lyzing educational experiments under known simulated conditions which are presumably realistic. The results should indicate whether either of the methods is appropriate for its recommended use, thus giving edu- cational researchers some direction when faced with the choice. If both methods were shown to be appropriate, the study could determine which one is superior. The study should also indicate if either method of analysis is appropriate for a set of restricted conditions, e.g. for only certain levels of reliability. Organization of the Dissertation A statement of the problem, a description of possible solutions, and an overview of the procedure of this study has been included in Chapter I. A comprehensive review of the related literature is provided in Chapter II. This review includes a historical overview of the pro- blem and a detailed description of the methods being compared. A description of the procedures followed for this study is contained in Chapter III. The data and analyses are presented in Chapter IV and a discussion of the results is provided in Chapter V. A summary of the study is provided in Chapter VI. CHAPTER II REVIEW OF THE RELATED LITERATURE Comparing groups of differing abilities has been a problem for some time. The literature has included discussions of possible solutions for at least the past three decades. Many of these discussions have been in conflict with one another, and there is still no general agreement. The evaluation of federal compensatory education projects in the last decade has intensified the debate and the need for a theoretically sound and practically useful solution. Some of the solutions proposed over the years are matching subjects, gain scores, and analysis of covariance. A discussion of these solutions and why they may not be considered satisfactory is provided in this chapter. The major properties of analysis of covariance are also considered. Because Porter's adjustment for a fallible covariable in analysis of covariance is not readily avail- able in the literature, its derivation is provided in this chapter. Historical Review of the Problem A paper by Thorndike (1942) examined in detail the fallacies of comparing groups of differing abilities by matching subjects. The crux of his argument was that the regression effects were systematically dif- ferent "whenever matched groups are drawn from populations which differ with regard to the characteristics being studied," (p. 85). During the late 1950's and early 1960's the literature was inundated with papers on how to or how not to measure gain or change. One of the leaders during this period was Lord (1956, 1958, 1959, 1963). His proposal for estimating the "true" gain was originally set forth in 1956 with further developments in 1958 and 1959. McNemar (1958) extended his results for the case of unequal variances among the groups. Garside (1956) also proposed a method for estimating gain scores and Manning and DuBois (1962) presented their derivation of residual.gain scores. The debate over what techniques, if any, should be used for measuring gain or change was at its peak in the early 1970's. Cronbach and Furby (1970) concluded that one should generally rephrase his questions about gain in other ways. Marks and Martin (1973) underscored the Cronbach and Furby (1970) conclusion. O'Connor (1972) reviewed developments of gain scores in terms of classical test theory. Neel (1970) employed Monte Carlo techniques to compare four identified methods for measuring gain. The compared methods were raw difference, Lord's true gain, residual gain, and analysis of covariance. Under equivalent conditions, he found that Lord's true gain tended to produce a greater significance level than the user would intend, that is, a higher fraction of type I errors than alpha. The analysis of covariance method has been widely recommended by a number of authorities in the field (Thorndike, 1942; Campbell and Stanley, 1963; O'Connor, 1972). Campbell and Erlebacher (1970) point out that the Westinghouse/Ohio University Study was evaluated, possibly incorrectly, using the analysis of covariance. Lord (1960, 1967, 1969) has sounded a warning about its use that has been echoed by others (Werts and Linn, 1970; Campbell and Erlebacher, 1970; Winer, 1971). The warning stressed that the analysis of covariance requires the assumption that the covariate" is measured without error. Lord (1960) has proposed an adjustment which was later generalized by Porter (1967). The adjustment is based on the substitution of the true score estimate for the observed value of the covariate. Analysis of Covariance The analysis of covariance procedure was originally introduced by Sir Ronald A. Fisher (1932, 1935). According to Fisher (1946, p. 281), the analysis of covariance "combines the advantages and reconciles the requirements of two widely applicable procedures known as regression and analysis of variance." The procedure is well documented in contemporary texts (Snedecor and Cochran, 1967, p. 419-446; Kirk, 1968, p. 455-489; Winer, 1971, p. 752-812). Analysis of covariance is a popular technique in both the physical and social sciences. Among the principal uses of analysis of covariance pointed out by Cochran (1957), p. 264) is "to remove the effects of disturbing variables in observational studies." It was thought that by using the pretest score as the covariate and comparing two groups with analysis of covariance, the effects of different pretest scores could be eliminated. Lord (1960) pointed out that this was not necessarily the case when the covariate was fallible. The assumptions necessary for the analysis of covariance are the same as those for analysis of variance with the addition of the following: 1. The covariates are measured without error. 2. The regression coefficient is constant across all treatment groups (Peckham, 1970). Violation of the first assumption induces a bias in the analysis of covariance because of "the presence of 'error' and 'uniqueness' in the covariate, i.e. variance not shared by the dependent variable. If the proportion of such variance can be correctly estimated, it can be corrected for," (Campbell and Erlebacher, 1970, p. 199). This position was upheld by Glass et al. (1972). The basic algebraic derivation of this correction was presented by Lord (1960) and expanded by Porter (1967). Analysis of Covariance with Porter's Adjustment for a Fallible Covariate Lord's (1960) derivation of the analysis of covariance adjustment was limited to two treatment groups and is slightly more difficult than is Porter's (1967) procedure. Thus, Porter's procedure will be derived here and used in the analysis. Porter's adjustment is based on the sub- stitution of the estimated true value for the covariate. Let X denote a fallible variable (e.g. a pretest score), r an esti- mate of the reliability of X, and Ti the true value of the variable Xi. Let Ti denote the estimated true score of Xi. The definition of Ti for each individual is (1) Ti = X + r(Xi-X), or Ti = rXi + Y(l-r). Porter (1967) derived the mean and .variance of Ti as follows. By definition, the mean of all Ti's is: (2) T = Ti N = z[rXi+X(l-r)] N by substitution. Thus T = rX + NX-NrX , N where N denotes the sample size and it is understood that i is summed over the values 1, 2, ---, N. Also, by definition, the variance of T is 2 2 (3) S T = z(Ti-X) T N-1 = r2S2, and (4) Sy = z(Ti-X)(Yi-) , N-1 = rSxy, where Y is the dependent variable. Based on these results, Porter (1967) derived an analysis of covar- iance procedure replacing the fallible covariate, Xi, with the estimated true score, Ti. The analysis of covariance requires the computation of analysis of variance sums of squares for the dependent variable, the co- variable, and on the cross-products of the dependent variable and the co- variable. Porter (1967) showed that the use of estimated true scores for the covariable did not affect the analysis of variance of the dependent variable, Y. The changes found by Porter (1967) in the other two cases are as follows: For the analysis of variance of the estimated true scores, T, (5) SSW = E(Tij-T.j)2 where SS denotes the within groups sum of squares, (6) SSB. = nz(T.j-t..)2 = n(7. j-X..)2 where SSB, denotes the between groups sum of squares, (7) SST. = z( ij-f..)2 = r2zEX +(l-r)E( Xij)2 -(ZEij)2, where SST^ denotes the total sum of squares, n denotes the number of (T,Y) pairs per treatment group, and N denotes the total number of (T,Y) pairs. In a similar manner, the cross-products sums of squares are: (8) SSwjY = rcz(Xij-X.j) (Yij-Y.j), (9) SSBef = nz(X.j-X..)(Y.j-Y..), (10) SSTi = rzXi-jYij+(l-r)z (Exij)(EYij-_(xXY'ij)2 (10) TY 1 - Thus, the adjusted sums of squares are (rSS )2 (11) SSW' = SSW XY r2SSWX (SS )2 = SS (SSWXY 2 Y SS- W X (rSSy + SSB )2 (12) SS'T = SST (rSXY + SSB r2SSW + SSB SS'B = SS'T SS'W . Note that the adjusted within groups sum of squares remains un- changed by the substitution of T for X but the substitution does alter the adjusted total sum of squares and consequently, the adjusted between groups sum of squares. A question arises concerning what value to use as the estimate of reliability in the formulae. A solution to this problem was proposed by Campbell and Erlebacher (1970). In a pretest-posttest situation one may find it reasonable to make two assumptions that would generate appropriate common- factor coefficients. First, if one has only the pretest-posttest correlations, one may assume that the correlation in the experi- mental group was unaffected by the treatment. (We need a survey of experience in true experiments to check on this.) Second, one may assume that the common-factor coefficient is the same for both pretest and posttest. Under these assumptions, the pretest- posttest correlation coefficient itself becomes the relevant common-factor coefficient for the pretest or covariate, the "reliability" to be used in Lord's and Porter's formulas. (p. 200) This recommendation will be followed in this study, thus the corre- lation between the pretest scores and posttest scores will be used as the reliability estimate in. the formulae. Summary As noted, analysis of covariance and analysis of covariance with Porter's adjustment have been recommended by several authors. Campbell and Erlebacher (1970) computer generated data for two overlapping groups with no true treatment effect and concluded that the analysis of covariance method was inappropriate and that analysis of covariance with Porter's adjustment should only be undertaken with great tenta- tiveness. Porter (1967) computer generated data to compare the F sampling distribution with the theoretical F distribution using his adjustment. He concluded that samples of 20 or larger were needed to have a useful approximation to the theoretical F distribution. He also 15 found that the estimation degenerated further when the reliability was less than .7. No study was found which compared both techniques under similar conditions for both gain and no-gain groups. CHAPTER III PROCEDURES The analysis of covariance and the analysis of covariance with Porter's adjustment were compared under forty-eight sets of conditions on the basis of computer generated data. Six combinations of reliability, two sample sizes, two levels of gain, and two different sets of pretest means were used. Both equal and unequal reliabilities were used in the comparisons. A random sample of two thousand observations was gen- erated under each combination of conditions and analyzed by analysis of covariance with and without Porter's adjustment. The sampling distributions of the two analyses were statistically compared with a sign test for each of the forth-eight sets of conditions. The computer generated alpha values and powers were then used as the criterion variables in four factorial experiments. Subsequent a poste- riori analyses were performed where warranted. The Model A standard model was used to represent the pretest and posttest scores of one subject. The model follows the traditional measurement approach as found in Gulliksen (1950) or Lord and Novick (1968) and extended to gain score theory by O'Connor (1972). (14) X = T + El and (15) Y = T + G + E2 where X = observed pretest score, Y = observed posttest score, T = true pretest score G = true gain, E1= random measurement error in pretest score, E2= random measurement error in posttest score. The following properties about E1 and E2 are assumed to exist: The errors El and E2 i) have zero means in the group tested, ii) have the same variances for both groups, iii) are independent of each other and of the true parts of each test. It is further assumed that T and G are independent (across subjects) and that all components follow a normal probability distribution. These assumptions parallel Lord's' (1956) stated and implied assumptions. Selecting the Reliabilities Reliability is related by definition to the variances of the ob- served scores, the true scores, and error. This section shows how that relationship can be used to establish desired reliabilities. The basic definition of reliability (Helmstadter, 1964, p. 62) is given by equation (16) where the symbol, pXX, denotes reliability. (16) pXX = X'2E 2 aX Then, ~ (17) 02 = o2(1-p ) E X XX Choosing the variance of X a prior to be 100, the variance of El can be found in the following manner as the reliability of X assumes different values: (18) o2 = 100(1-p ) El xx The independence of T and El in (14) implies (19) 02 02 + 2 X T El Combining (18) and (19) and solving for o2 yields (20) o2 = 100 T XX The posttest variances can be selected in a similar manner. Based on the assumptions, (21) 02 -,2 + G2 + 2 Y T G E The variance of the gain scores is chosen in the manner prescribed in Chapter I. Then, combining (16) and (21), (22) 2 = 2 o G 02T oG PYY where pyy denotes the established reliability of the posttest scores. Thus it can be seen that the effect of selecting specified reliabilities can be obtained by selecting the variances of El, E2, and T in accor- dance with (18), (20), and (22) respectively. The Regression of Y on X Recall that one of the assumptions necessary for analysis of co- variance (Cochran, 1957) is that the regression slopes of the dependent variable on the covariate must be equal for each treatment group. Let BY.X denote the regression slope for one treatment group. Then, (23) B-X = PXY OY (Ferguson, 1971, p. 113), oX implies (24) BY-X = XY 2 2 T T E The covariance between X and Y is equal to the variance of T since it has been assumed that all the components of the pretest and posttest are independent except T-with itself. Therefore 2 (25) By.X = T , 2 2 OT +OE1 and (26) e-X = PXX by equation (16). Thus, the slope, Y on X, of any group is equal to the reliability of its pretest. If the reliabilities of the protests were the same for both groups, the assumptions concerning slopes would be satisfied. From equation (16), it can be seen that the reliability of a test is dependent to a certain extent upon the variability'of the sample for which it is given. In equation (16), o2 is in both the numerator and denominator of the fraction with o2 being subtracted from the numerator. As o~ increases both the numerator and denominator increase by the same amount assuming oE is not changed thus, pXX increases. Two groups of equal ability would likely produce similar variances, hence similar reliabilities. However, the technique is often used for comparing compensatory programs with a comparison group sampled from the general population of untreated children in the same community such as with the Westinghouse/Ohio University study (Campbell and Erlebacher, 1970). These comparison children tend to be higher in ability than the treatment group. The resulting differences in variation tend to produce different reli- abilities. In order to simulate such a situation this study used dif- ferent reliabilities for the gain and the no-gain groups in addition to the comparisons when the reliabilities are equal. The reliabilities for the gain group are decreased by fifteen percent which roughly approxi- mates the reduced variation empirically observed in Project Follow Through achievement data. Selecting Means The true score mean was set a priori at 100 when both the gain and no-gain groups have equal means. The situation in which the gain group has a lower mean was also analyzed. In this case, the true score mean of the gain group was set at 80. These values have been empirically chosen based on Project Follow Through data. Based on the assumptions, (27) E(X) = E(T+E1) = PT' hence the mean of the observed pretest scores is equal to the mean of the true scores. Also (28) E(Y) = E(T+G+E2) = "T+"G. Thus, the mean of the observed posttest scores is equal to the sum of the means of the true scores and the gain scores. Clearly in the case of the no-gain group and in both groups where no gain was used, the mean gain was zero. The selected value of pG for the gain situation was based on power considerations, that is, PG was chosen such that the power of an F test for analysis of covariance was .50. A linear model representation of the analysis of covariance for two groups is (29) Y = B0 +BX+B2W+E where X is the pretest score (covariate), Y is the posttest score, and W is a dummy variable designating group membership (W=l if gain group, 0 if no-gain group). Testing the hypothesis that 62 is equal to 0 in equation (29) is equivalent to the F test for treatments in the analysis of covariance procedure. It can be shown that 02 in equation (29) is equivalent to the mean gain, pG. The mean of the posttest for the gain group is (30) E(YG) = BO+B60 XG+02 where YG is a posttest score for the gain group and pXG is the mean pre- test score for the gain group. Likewise, the mean of the posttest for the no-gain group is (31) E(YNG) = B0"1PXNG where YNG is a pretest score for the no-gain group, pXNG is the mean pre- test score for the no gain group. But it has been assumed that pXG and PXNG are equal. Furthermore, the gain group has mean gain, uG and the no-gain group has mean gain zero, thus (32) "G = E(YG)-E(YNG) = B2 Hence, choosing the value of B2 that yields a power of .50 is equivalent to choosing a value of PG to produce a power of .50 in the analysis of covariance procedure. This power can be obtained from the following probability state- ment: (33) Pr[t*>tj = .50 where t* is a noncentral t statistic. This expression can be approxi- mated by the substitution of az statistic for t*. (34) Pr~2-B2 > 2 = .50 . Thus a value of B2 can be chosen such that the substituted z statistic is equal to t 025 with the appropriate degrees of freedom. This value of 82 is the value of the average gain, 1G, such that the power of analysis of covariance is .50 under the condition of perfect reliability. In order to find this value of PG, a numerical expression for 02 is needed. B2 The variance of #2, 2, can be approximated in the following manner. Karmel and Polasek (1970, p. 245) state that o2 is 12 (35) 02 = G2 z(X-Y)2 2 2 2 [z(X-X) 2[z(w-iq) 2-[z(X-)(W-T)]2 Dividing both the numerator and denominator by N2 and substituting population variances for sample variances, o? is approximately equal 2 to the following: 2 2 (36) o : Y oX 82 N 2 2 2 0X 0W -(oXW The quantity, OXW, the covariance between X and W has been assumed equal to zero thus, in equation (36), the o2's divide out. Hence x (37) o1 = 1 B2 N 2 OW 2 Under the conditions assumed for the model, the variance of Y, GY, 2 is equal to 4. The variance of W, OW can be computed to be .25. Thus, 2 2 by substitution, 2 equals .80 when N is equal to 20 and 2 equals .08 when N is equal to 200. Hence, for N = 20, vG can be found by the following expression: (38) PG20 =t.025,172 = 2.11 vl17 = 1.88. Likewise, for N = 200, "G can be found by solving equation (39). (39) G200 = t.025,19702 = 1.97v08 = .56. The approximated values of pG were tested using Monte Carlo generated variables and found to indeed produce a power of .50. The approximated values of vG under each set of conditions are shown in Tables 1 and 2. TABLE 1 CONDITIONS UNDER WHICH COMPARISONS WERE MADE WHEN THE RELIABILITIES FOR BOTH GROUPS WERE EQUAL GROUP GAIN GROUP PRETEST PRETEST SAMPLE MEAN MEAN FOR MEAN FOR RELIABILITY SIZE GAIN GAIN GROUP NO-GAIN GROUP .90 10 0 100 100 .90 10 0 80 100 .90 100 0 100 100 .90 100 0 80 100 .70 10 0 100 100 .70 10 0 80 100 .70 100 0 100 100 .70 100 0 80 100 .50 10 0 100 100 .50 10 0 80 100 .50 100 0 100 100 .50 100 0 80 100 .90 10 1.88 100 100 .90 10 1.88 80 100 .90 100 .56 100 100 .90 100. .56 80 100 .70 10 1.88 100 100 .70 10 1.88 80 100 .70 100 .56 100 100 .70 100 .56 80 100 .50 10 1.88 100 100 .50 10 1.88 80 100 .50 100 .56 100 100 .50 100 .56 80 100 TABLE 2 CONDITIONS UNDER WHICH COMPARISONS WERE MADE WHEN THE RELIABILITIES FOR BOTH GROUPS WERE UNEQUAL RELIABILITY RELIABILITY GROUP GAIN GROUP PRETEST PRETEST FOR GAIN FOR NO-GAIN SAMPLE MEAN MEAN FOR MEAN FOR GROUP GROUP SIZE GAIN GAIN GROUP NO-GAIN GROUP .76 .90 10 0 100 100 .76 .90 10 0 80 100 .76 .90 100 0 100 100 .76 .90 100 0 80 100 .60 .70 10 0 100 100 .60 .70 10 0 80 100 .60 .70 100 0 100 100 .60 .70 100 0 80 100 .42 .50 10 0 100 100 .42 .50 10 0 80 100 .42. .50 100 0 100 100 .42 .50 100 0 80 100 .76 .90 10 1.88 100 100 .76 .90 10 1.88 80 100 .76 .90 100 .56 100 100 .76 .90 100 .56 80 100 .60 .70 10 1.88 100 100 .60 .70 10 1.88 80 100 .60 .70 100 .56 100 100 .60 .70 100 .56 80 100 .42 .50 10 1.88 100 100 .42 .50 10 1.88 80 100 .42 .50 100 .56 100 100 .42 .50 100 .56 80 100 Generation of Random Normal Deviates with Specified Means and Variances The study required the use of computer generated normally distrib- uted random variables with specified means and variances. Two thousand sets of variables were generated for each of the forty-eight sets of conditions. Muller (1959) identified and compared six methods of gen- erating normal deviates on the computer. A method described by Box and Muller (1958) was judged most attractive from a mathematical standpoint. According to Muller (1959, p. 379), "Mathematically this approach has the attractive advantage that the transformation for going from uniform deviates to normal deviates is exact." This method was endorsed by Marsaglia and Bray (1964). They modified the algorithm to reduce central processing computer time without altering its accuracy. The method first requires the generation of two independent uniform random variables, U1 and U2, over the interval (-1, 1). The variables Z1 = U [-2 In(U +U2) / 2 + 1/2 and Z2 = U2[-2 ln(U+U2) / (U2 + U)]1/2 will be two independent random variables from the same normal distrib- ution with mean zero and unit variance. The variables were then trans- formed to have the desired means and variances. Analysis for Comparing the Selected Methods Each of the two thousand sets of generated data for the forty-eight sets of conditions was analyzed by analysis of covariance and analysis of covariance with Porter's adjustment. The number of times the F statistic from analysis of covariance exceeded the F statistic from analysis of covariance with Porter's adjustment was noted along with the proportion of rejections by each method of analysis. A sign test (Siegel, 1956, p. 63-67) was then performed for each set of conditions to test the null hypotheses in Chapter I, that is, "the sampling distributions of the test statistics from analysis of covariance with and without Porter's adjustment will be the same for each set of conditions." These tests were run at the .01 level of significance. The fraction of rejections noted alphass and powers) was then used as the dependent variable in four factorial experiments to gain further insight into what factors affected each method of analysis being studied. Each factorial experiment had three factors. These were reliability which included six combinations of reliability used in the study, sample size, which included n = 10 and n = 100 subjects per group, and the equality of pretest means, which included a level where both pretest means were equal and a level where they differed. The layout of the factorial experiments is illustrated in Table 3. The factorial experiment illus- trated is the situation for which there was no gain in either group and the standard analysis of covariance was used. Thus, the dependent variables are the alpha values generated by the computer. When significant main effects or interactions occurred, the appropriate a posteriori analytical procedures to locate the sources of the variation were followed. 0 0 CM 0 Sa- 0 O- 0 in o S O Lt < 00 CD O o : c. o. W>< 00 UJ CC J02 Z U) 0 0 C )0- 0 C 0 C'.n M Z C Z -t 00 St4 J r 0U m < C o C 0 < =0 0 F- CD _J N U0-2 U)n 00 CS) co oll O 0 0. c1 ) coN- 2 0 N Ct Co C- =D 0Z 02 02 2m U) 02 02 U) U) co 02o 02U 0LI (f) rV) a0- H : H- Z= :r-Z 0 CM0 (n 4 C : CL m CHAPTER IV RESULTS The number of times the F statistic from the standard analysis of covariance exceeded the F statistic from analysis of covariance with Porter's adjustment is listed for each set of conditions in Tables 4, 5, 6, and 7. These quantities were used as test statistics for the sign tests used to test the hypothesis,"There is no difference between the sampling distributions of the test statistics from analysis of covariance with and without Porter's adjustment." This hypothesis was tested for each of the forty-eight sets of conditions at the .01 level of signifi- cance. Siegel (1956) stated that a large sample test statistic for the sign test is (40) z = x-.5N .5VN where N is equal to the number of pairs of observations, x is equal to the number of times the first measurement of the pair exceeds the second measurement of the pair, and z is the standard normal variate. For a level of .01, the null hypotheses would be rejected when z was less than -2.33 or greater than 2.33. This is equivalent to rejecting the null hypotheses when x was less than 949 or greater than 1051 and N equals 2000. Thus, an inspection of the last column of Tables 4, 5, 6, and 7 reveals that the null hypothesis was rejected for each of the forty- eight sets of combinations. The analysis of variance summary tables are presented in Tables 8, ~ 9, 10, and 11. The analysis of variance summary table for the case when the Monte Carlo generated alpha values from the standard analysis of covariance were used as the criterion variables is presented in Table 8. The analysis of variance summary table for the case when the Monte Carlo generated alpha values from the analysis of covariance with Porter's ad- justment were used as the criterion variables is presented in Table 9. The analysis of variance summary tables for the cases when Monte Carlo generated powers were used as the criterion variables for standard analysis of covariance and analysis of covariance with Porter's adjust- ment are presented in Tables 10 and 11, respectively. Each analysis of variance table includes analyses of the simple effects where they are warranted. Scheffe's S method for testing linear contrasts is included in Table 8. These linear contrasts are defined in Table 12. Each F statistic which exceeds the critical value at the .05 level is denoted by an asterisk. A result of particular interest is applicable to analysis of co- variance both with and without Porter's adjustment. That is, when a spuriously high fraction of significant F statistics occurred when there was a mean gain of zero in both groups and the pretest means differed, the adjusted posttest means indicated that the gain was in favor of the group having the largest pretest mean. This result was more pronounced for lower reliabilities. When the gain group had a positive gain, the no-gain group had a mean gain of zero, and the no-gain group also had a larger pretest mean, a similar situation occurred. In this situation, when a significant F statistic occurred, the adjusted posttest means usually indicated the no-gain group had recorded the larger gain. Again, these results be- came more pronounced as the reliability of the scores were reduced. LU mci SLUJ 2: LU JU ci c-- r-i -- -- LL.L OLU LU >F-- : cl LU LU LU CD -ci H- LU H - L- LU : oi 2: C 0- t 2I LL- LU ci ci LL- mC L- -D ( L U i Li- 2f: H- = 0- oI S LU00 t >- c) 2 S C) D C c =I L) - : U -: I CC O 3C - UC V) C L U i LU 0 H- Qt z 2: U2:C) cI l-i 0. CD CEL a- LU =0 0- V) 2 CD H- Ll LU 0 LL = LU ci -J 0- >- LU -LJ L LU L 0C 0 0 CO 0 CD0 0 c0 C O c0 cO C 00 C) CD CD CD S c ci ci 0 O 9 O i i 9 o o o o o o o- o o o o Sc o c r r r r L L L Ln t. Ji- p- ci t c) %i 03 rz In CM i C\j C ci c00 m C oo c' c0 cri cm (1) to co r- r-. c0 cO C0 ci ci N c1 i co = C ) ci i rN. 10 (o ci 'C- o' c LO ki Lo c' n ci tO Ln c0 c0 CD CD C Ci Ci C Ln CD cD co c0 ci C0 Co co c C CD o .0 -r'C 'Ci- ci 'C- c) in C C- ci i ci cin c cij c ci. ci ci c ci CO CO CO CO CO CO CD C0 CD C0 C0 C0 O O O O O :R 0 0 0 0 0 0 0 0 000 C C)O O O C) C 0O co C C4 -c -r- 4-1 in s- 0 LO cu ,E o- 4-1 i.n 4-i- -0 io 2: c 10 .Q V3 >- LU H- O~ C- C 1-1 0 CD< CDi CD cizv ci - -I 0 -u u- ar Q- C D LU O O 33 N- CJ m() ( C'J 00 r- cz C CD CO CD MO ZC Lo 0') CO C C LU LU 2: (C WH IO LU LU 2: C C Z w-l C ( LU LU LU m = :0 H H ( U- LU U-f 00 I SLU 00 LL- 7 LU LU - CO E LU 2 2 4 H e CD C) LU -o U-| LL (A C C 0 LU C3 LU - C2: z (-C 0-1 LU C- Co LU La CO I- LU H- Z) U- >- L C IH- L--I -z O 3 - U-H- C CQ SLU D >- 2: C) Z- Z: (A C): CQ : CD1 0 cl 0 L cn C UH-LL O 1 00L L-i-:c C) iO F = 2f: Z l- U- C C C Y ( D (D SlLL 0 H- LL Z LU U 3- 0 U0 A C C LU D LC I- Z 0i L1- CC 0CC C0 0 0 0 0 0 0 0 0 e 0 0 0 0 0 C C 0 o o o 0 0 0 0 0 0 0 0 0 0 0 C O C CO o O0 CO o CO o o 0 ~ oo 000o O O 0 0 0 0 - 0 0 0 0 0 0 -0 0 0 0 o 0 S- o C C C C C C C C C C C C oa oN o oN -o rCO rCO rO CO Lo 0 W w kD cD O O C) <\I C0 O r-^ ^. -- -- D ^ ^ ^ ^ ^ It r "- co ( 00 "\ ". O C'. 0) 00 LO r- Lo ko LO C) r- LO mD N-I r C'. C) C C') C C) CD o CD C C ko cQ Ln O oCt C) m tD cr ir C '^ O m O UU O C CO m O C O C C C C') C (- rC C4 41 S.- lu u >4-' Ln 4 cu C4 (U 0 i-: fu' 4-, U (U (UO C04- Da -(U (UU (Uu, rr0 ClI -I 0 LU Q: LU a3 I- - LU C LU LU CM V Wu SLU a SIA LU I LU LU - LU Co LU mZ r C, - LU F- u- LU 0- LL a- CD CD LU SQa ct C C - a C F- a3 g) LQl- LL.Z U-| U-1 LU c wU - C C of Co F- 2; LU F- t 0 0 CD C C) LU a- 70 L o, - LU cF I-- vi ; v-> C Z A LU e 0 >- 0 2 n -J u- 1-1 I S1C I u LU LU- .1c a 0 Z >- D <-u C C- u- w 0 CD 21- LL; o F- a C CD F-i - F- w; :D 0 Y- LL4 z V) LU CD u- w C 0 0 LU U- 0 SC) CD LL u- LU C LlD a; AO 0 0 0 0 o o o o o oo o CO O O O O O O O O o C C o 0C C Co CD 0C0 C L LO CO 0 0 O 0 C C C D 00C C n Co o0 0 0 - C C C LC O 0 O O C C O O O CC O O O N- O CO O O . . o . *o * C C C C C C C C o o ooo o oo 34 S- co i CM C O CM 0- Zl Co Ln C C D Co om Co OD C7i oj ^ r m o o: m m C NC Cn I- 0C 0 0 -C CC 0 0- CO SM o r c C- O O C O C *- C 'a S- a; 0 L (U 4- +J 4-) a +3 *- a)- C-^ +3 c C C ui LU CC 3: F- 0- LU LU 0I LO - LU 0U 3Z LL.]CJ LU LU C) (- ) w U -1 LU UJ C) VId LC -t C.) L 2i LI-L (A Q; U-_ LU3 LLl _j LU _I =33 Z - C) Q -: L! Z Z (/) If) u- 0; U O Z L Zt U-l-i 0v D- WmI HL LU IO LL - =1 U- F- >< 0 Z C (n LLl 0- ~I LUl C,) U -D C w-C) - 0 I- - ;I- -< QLU UZ UI- I (A w I- F -3) V) O) Z I- >- LI- - u C- .. U- lc L-- L= = D wZ O I- Ll ZL LU w I a 0 U- u C) O LU C D L U -j 0 0 C) co S-1 0 CQ 1C1 a O O lo O O 0 0 0 0 0 0 0 0 cz r- r. o. or ID L9 m o 0 0 o 0 o 0 o 0 o Co 0 00 L0 I0 C C o o 5n 5n w w Lo w O O O O C l CM C'_ CJ ~~~zd- rN - o -d LO M a C o t oN rr- o o o o O O CO 0 0 0 0 0 0 0 C O 0 0 0CCD C C C C 0 o C o 0 o D 0 o 0 0 0 o o o c o o 0 o M N. to o CN r- .o r 0o o co w mo fln cE co O o C C CO O OO O0 ( C0 r- i- O - 0 0 C9- O 0; 0 >C 4-) s- ow '4- LO 4 C r- 0 0 -c I-.- c0* 4-, (U.1 > cO co L0 C 09 Lo CD C:) C) o1 O O TABLE 8 ANOVA SUMMARY TABLE USING MONTE CARLO GENERATED ALPHAS FROM THE STANDARD ANALYSIS OF COVARIANCE AS THE CRITERION VARIABLES SUM OF DEGREES OF MEAN SOURCE SQUARES FREEDOM SQUARE F Pretest Mean 1.59960 1 1.59960 (544.08) PM at R1 .07049 1 .07049 23.98* PM at R2 .33582 1 .33582 114.22* PM at R3 .40513 1 .40513 137.80* PM at R4 .16687 1 .16687 56.76* PM at R5 .33466 1 .33466 113.83* PM at R .42510 1 .42510 144.59* PM at SS10 .11525 1 .11525 39.20* PM at SS100 2.10000 1 2.10000 714.29* Reliability .10996 5 .02199 (7.48) R at PM1 .00008 5 .00001 <1.00 R at PM2 .22278 5 .04455 15.15* 1 .00018 1 .00018 <1.00 '2 .10306 1 .10306 35.05* '3 .73933 1 .73933 251.47* '4 .04743 1 .04743 16.13* Sample Size .59977 1 .59977 (204.00) SS at PM1 .00005 1 .00005 <1.00 SS at PM2 1.21540 1 1.21540 413.40* PM x R .11291 5 .02258' 7.68* PM x SS .61568 1 .61568 209.41* R x SS .01470 5 .00294 1.00 Residual .01468 5 .00294 Total 3.06730 23 *Sample statistic greater than critical value at .05 level. TABLE 9 ANOVA SUMMARY TABLE USING MONTE CARLO GENERATED ALPHAS FROM ANALYSIS OF COVARIANCE WITH PORTER'S ADJUSTMENT AS THE CRITERION VARIABLES SUM OF DEGREES OF MEAN SOURCE SQUARES FREEDOM SQUARE F Pretest Mean .45844 1 .45844 (25.08) PM at SS1o .00832 1 .00832 <1.00 PM at SS100 .75050 1 .75050 41.06* Reliability .17552 5 .03510 1.92 Sample Size .33018 1 .33018 (18.06) SS at PM1 .00035 1 .00035 <1.00 SS at PM2 .63021 1 .63021 34.48* PM x R .15727 5 .03145 1.72 PM x SS .30038 1 .30038 16.43* R x SS .10189 5 .02038 1.11 Residual .09139 5 .01828 Total 1.61507 23 *Sample statistic greater than critical value at .05 level. I TABLE 10 ANOVA SUMMARY TABLE USING MONTE CARLO GENERATED POWERS FROM THE STANDARD ANALYSIS OF COVARIANCE AS THE CRITERION VARIABLES SUM OF DEGREES OF MEAN SOURCE SQUARES FREEDOM SQUARE F Pretest Mean .97768 1 .97768 (85.54) PM at SS10 .01620 1 .01620 1.42 PM at SS100 1.61334 1 1.61334 141.15* Reliability .12431 5 .02486 2.17 Sample Size .65076 1 .65076 (56.93) SS at PM1 .00001 1 .00001 <1.00 SS at PM2 1.30482 1 1.30482 114.16* PM x R .19910 5 .03982 3.48 PM x SS .65406 1 .65406 57.22* R x SS .05665 5 .01133 <1.00 Residual .05714 5 .01143 Total 2.71970 23 *Sample statistic greater than critical value at .05 level. TABLE 11 ANOVA SUMMARY TABLE USING MONTE CARLO GENERATED POWERS FROM ANALYSIS OF COVARIANCE WITH PORTER'S ADJUSTMENT AS THE CRITERION VARIABLES SOURCE Pretest Mean PM at SS10 PM at SS100 Reliability Sample Size SS at PM1 SS at PM2 PM x R PM x SS R x SS Residual Total SUM OF SQUARES DEGREES OF FREEDOM .15714 .00157 .36018 .14179 .23285 .00046 .43701 .16990 .20461 .13367 .12132 MEAN SQUARE .15714 .00157 .36018 .02836 .23285 .00046 .43701 .03398 .20461 .02673 .02426 1.16127 ,*Sample statistic greater than critical value at .05 level. (6.48) <1.00 14.85* 1.17 (9.60) <1.00 18.01* 1.40 8.43* 1.10 TABLE 12 COEFFICIENTS FOR THE LINEAR CONTRASTS USED WHEN RELIABILITY WAS SIGNIFICANT IN THE ANALYSIS OF VARIANCE LEVELS OF RELIABILITY RELIABILITY OF GAIN GROUP 90 70 50 76 50 42 RELIABILITY OF NO GAIN GROUP 90 70 50 90 70 50 1 1 1 1 1 1 1 3 3 3 -3 1 1 0 1 1 0 2 2 7 CONTRAST 1 0 1 1 0 1 0 1 1 0 1 1 24 2 -2 2 - CHAPTER V DISCUSSION In general, the results of this study support the positions of Lord (1967, 1969), Campbell and Erlebacher (1970), and O'Connor (1972) with respect to their warnings about the implications of analysis of covariance using unreliable test scores. The study shows that even when there is no gain in either group, a much higher fraction of rejections occur than would be expected. The fraction of rejections is even more extreme when the reliability is .70 or lower and the pretest means differ. In addition, Porter's adjustment seems to offer little improve- ment. Comparison of the Two Methods of Analysis The rejection of all forty-eight null hypotheses concerning the equality of the sampling distributions for the two methods of analysis shows that there is a difference in the results obtained from analysis - of covariance and analysis of covariance with Porter's adjustment. A closer examination shows that these differences are more extreme when the pretest means of the two groups differ and the reliabilities are low. Further study shows that although the two methods of analysis are different, neither method does an adequate job of modeling reality, that is, both methods tend to produce erroneous proportions of type I and II errors when pretest means differ and the reliabilities are low. When the pretest means are equal, the power of the tests seem to be directly re- lated in a positive manner to the reliabilities when other variables are held constant. 41 Possibly the most far reaching results were a function of reliability. The data indicated that incorrect decisions about which group had the larger gain could be made using either method of analysis when the pre- test means of the two groups differed and the reliabilities were low. When the pretest mean of'the gain group was less than that of the no-gain group, the adjusted posttest means showed that the no-gain group was superior both when the mean gain was zero in both groups and when the mean gain was positive only in the gain group. This possibility was pointed out by Lord (1967) and Campbell and Erlebacher (1970). Factors that Affect Alpha and Beta The factorial experiments using the computer generated alphas and betas allow one to infer which factors affect the levels of type I and type II errors. Using Monte Carlo generated alphas from the standard analysis of covariance in a factorial experiment indicated that an inter- action between pretest means and reliability and an interaction between sample size and pretest means affected the alphas significantly. Further analyses (simple effects) showed that the pretest mean factor was signif- icant at every level of reliability. Both reliability and sample size were significant when the pretest means differed. The tests of linear contrasts showed that there was no significant difference between equal reliabilities and unequal reliabilities when the pretest means differed, but there were significant differences among the levels of reliability when the pretest means differed. These results indicate that a difference in reliabilities between groups, thus a difference in slopes, has no effect, however, the level of reliability does. The other three factorial experiments indicated that interactions between pretest means and sample size were the major contributors to the differing levels of alpha and power. In all three experiments (See Tables 10, 11, and 12) the sample size was significant when the pretest means differed. Reliability seems to have a somewhat moderate effect in these cases. Predicting Alpha and Power Using the results of this study, a regression equation can be set up to predict the experimental probability of a type I error or the experimental probability of rejecting a false null hypotheses when the established probabilities are .05 and .50 respectively. The basic regression equation is (41) Y = Bo+BIRG+12RN+B3S+B4M+B5RGRN+B6RGS +B7RGM+g8RNS+9BRNM+B10SM+E where Y = the predicted alpha or power, RG = the reliability of the gain group scores, RN = the reliability of the no gain group scores, S = sample size M = 1 if the pretest means are equal, 0 otherwise, and Bi = the regression coefficient, i. Tables 14 and 15 provide confidence intervals for the expected values of alpha and power respectively for each set of conditions noted. TABLE 13 NINETY-FIVE PERCENT CONFIDENCE INTERVALS FOR THE EXPECTED VALUES OF ALPHAS UNDER SPECIFIED CONDITIONS RELIABILITY OF GAIN GROUP SCORES RELIABILITY OF NO GAIN GROUP SCORES GROUP SAMPLE SIZE EQUALITY OF PRETEST MEANS* LOWER CONFIDENCE LIMIT OF ALPHA S.000 .000 .000 .556 .017 .108 .000 .780 .000 .297 .000 .932 .000 .043 .036 .691 .003 .228 .014 .879 .000 .307 .000 .967 UPPER CONFIDENCE LIMIT OF ALPHA .137 .137 .100 .741 .162 .325 .141 .945 .125 .451 .120 1.000 .136 .223 .143 .871 .136 .361 .146 1.000 .097 .472 .117 1.000 *If 1, the pretest means are not equal. are equal and if 0, the pretest means TABLE 14 NINETY-FIVE PERCENT CONFIDENCE INTERVALS FOR THE EXPECTED VALUES OF POWER UNDER SPECIFIED CONDITIONS RELIABILITY OF GAIN GROUP SCORES RELIABILITY OF NO GAIN GROUP SCORES GROUP SAMPLE SIZE EQUALITY OF PRETEST MEANS* LOWER CONFIDENCE LIMIT OF POWER .000 .000 .000 .351 .019 .048 .000 .677 .000 .155 .000 .863 .000 .000 .000 .523 .000 .097 .010 .782 .000 .156 .000 .909 UPPER CONFIDENCE LIMIT OF POWER .300 .115 .191 .666 .266 .295 .236 .924 .176 .417 .224 1.000 .265 .199 .235 .829 .210 .323 .236 1.000 .130 .438 1.000 *If 1, the pretest means are not equal. are equal and if 0, the pretest means The predicted value of alpha or power can be found by using the data found in Tables 4, 5, 6, and 7 to fit regression equation (41) to obtain the following estimated regression parameters: (42) alpha = .228 + .395RG + .332RN + .008 S .661M -1.09 RGRN .004 RGS + .619 RGM -.003 RNS + .195 RNM .007 SM and (43) power = .045 + .536RG + .404 RN + .010 S .700 M -1.228 RGRN .006 RGS + .846 RGM + .002 RNS + .218 RNM .007 SM. A confidence interval for the predicted values of alpha and power could be obtained in the usual manner. A Direction for Further Research The results of this study combined with other research (Lord, 1967; Campbell and O'Connor, 1972), indicates a need for further study and the development of a robust method for comparing groups of differing ability when the scores are not perfectly reliable. Any new technique which is proposed should first be investigated under known conditions, either analytically, or by Monte Carlo techniques. This would insure that another inappropriate method is not used for the evaluation of compensa- tory education projects. CHAPTER VI SUMMARY This study was designed to determine if either analysis of covar- iance or analysis of covariance with Porter's adjustment is an appropri- ate analytical procedure for evaluating educational pretest-posttest experiments. In particular, these methods were compared with respect to their use in the analysis of compensatory education projects where the groups may differ in ability. The study was carried out by computer generating 2000 sets of normal data under forty-eight sets of predetermined conditions of reli- ability, sample size, gain, and equality of pretest means. Each of the 2000 sets of data for each set of conditions was then analyzed using both standard analysis of covariance and analysis of covariance with Porter's adjustment. A sign test was used to compare the two methods of analysis under each of the forty-eight sets of conditions. It was concluded that - the two methods of analysis yielded different results. Factors affecting the two methods of analysis were then studied separately using the computer generated alphas and powers as criterion variables in four factorial experiments. The factors included reliability at six levels, sample size at two levels, and the equality of pretest means at two levels. From these experiments, it was concluded that pre- test means interacting with sample size and sometimes with reliability were significant factors. More specifically, sample size was statistically significant in each case where the pretest means differed. Also, pretest means were significant at every level of reliability for the computer generated alphas produced by the standard analysis of covariance. It was also learned that when the pretest means differ, both standard analysis of covariance and analysis of covariance with Porter's adjustment produced erroneous results with respect to which group if either had a gain. When both groups had a mean gain of zero and the pretest means differed, significant results usually indicated that the group with the larger pretest mean had the gain. This would correspond to the control group sampled from the general population being credited with the gain in a compensatory education experiment. When there was a gain in only one group and the pretest mean was lower in that group, the analyses still indicated that the other group had the gain. The results of this study point to the recommendations that analysis of covariance with or without Porter's adjustment should be approached V with caution when the reliabilities are below .90 and the pretest means (covariate means) are likely to be different for the groups. APPENDIX C FCRTRAN PRCGRAV WHICH PERFORMED DATA GENERATION AND ANALYSIS C C DIVENSICN E1G (1002),E2GN(100),GAINGN(1CO),XX(2CO),YY(200), *X(100),EING(100),E2NC(100),GAINNC(100),TG(100),TN(100) REAL .VSB,PSE,MSBP,MSEP NGRCUF=O C C INPUT CF GRCUP PARAMETERS WIEREO C RELGN REPRESENTS TFE RELIABILITY OF THE GAIN GROUP DATA C RELNG REPRESENTS THE RELIABILITY OF THE NO GAIN GROUP C NPERG .REPRESENTS ThE NUMBER OF OBSERVATIONS PER GROUP C GBAR REPRESENTS ThE MEAN GAIN FOR GAIN GROUP C PRVNG REPRESENTS THE GAIN GROUP PRETEST MEAN C PRVNNG REPRESENTS THE NO GAIN GROUP PRETEST MEAN C ISEEC REPRESENTS THE SEED FOR RANDOM NUMBER GENERATOR C READ (5,99) ISEEO,NGP,NSPL 99 FORMAT (I ,2X,215) 100 READ (5,101) RELGN,RELNG,NPERG,GOAR, NGPRMNPRMNNG 101 FORMAT (2F3.2,I3,F3.2,2F3.0) GGRCUP=NCGICUP+1 NSAVF=O NF10=0 NF100=0 NFP10=0 NFP100=0 C C HEADER CARC FOR NEW SET OF PARAMETERS C WRITE (6,102) 102 FORMAT ('1',T43,'F STATISTICS BASED ON THE FOLLOWING PARME' *,'TERS') WRITE (6,103) 103 FORMAT ('O',T22,'RELGN ',2X,'RELNG ',2X,'NPERG ',2X,' GOAR' *,' ',2X,'PRMNG ',2X,'PRMNNG') WRITE (6,104) RELGN,RELNG,NPERG,GBAR,PRMNG,PRMNNG 104 FORMAT (1X,T22,2F8.5,I5,3X,3F8.3) WRITE (6,105) 105 FORMAT ('O',T45,'SAMPLE NUMBER',5X,'STANDARD F',5X,'PORTER' I,'S F') C C COMPUTE VARIANCE COMPONENTS C VARELG=IGO*(1-RELGN) VARE1N=100I (1-RELNG) VARTG=100IRELGN VART=1003*RELNG VARGNG=.044VARTG VARGKN=.04*VARTN VARE2G=(VARTG+VARGNG)/RELGN-VARTG-VARGNG VARE2N=(VARTN*VARCNN)/RELNG-VARTN-VARGNN C 106 CONTINUE NSAMP;NSAVP+1 C C GENERATING CF CATA FOR GAIN GROUP C C C TRUE SCORES C CALL RANGEN(NPERG,ISEEC,X) 00 300 I=1,NPERG TG(I)=X(I)*SQRT(VARTCI+PRMNG 300 CCNTIKUE C C PRETEST ERRCR SCORES C CALL RANGEh(NPERG,1SEED,X) CC 301 I=1,NPERG EIGN(I)=X(I)*SCRT(VAREIG) 301 CCNTINUL C C POSTTEST ERROR SCORES C CALL RANGEN(NPERGISEED,.X) DC 302 I=1,NPERG E2G ( I)=X(I)*SCRT(VARE2G) 302 CCNTIKUE C C GAIN SCORES C CALL RANGEN(NPERG,ISEEC,X) CC 303 I=1,NPERG GAINGN(II=XII)*SQRT(VARGNG)+GBAR 303 CCNTINUt C C PRETEST ANC PUSTTEST SCORES C CO 304 I=1,NPERG XX(I)=TGII)+EI GN(I) YY(I)=TC(I)+GAINGN( I)E2GN(I) 304 CCATINUE C C C GENERATION CF CATA FOR NO GAIN GROUP C C TRUE SCORES C CALL RANGEN(NPERG,ISEED,X) OC 400 I=1,NPERG 52 TMI(I) X(I) SQORT(VARTN)+PRMNNG 400 CCKTIhUE C C PRETEST ERRCR SCORES C CALL RANGE (NFPERG,ISEEC,X) OG 401 I=1,NPERG EING(I)=X(I)*SQRT(VAREI1N) 401 CCNTINUE C C POSTTEST ERROR SCORES C CALL RANGEK(NPERG,ISEEO,X) CC 402 I=1,NPERG E2NG(I)=X(II)SQRT(VARE2N) 402 CCNTIrUt C C GAIN SCORES C CALL RANGEN(NPERG,ISEED,X) CC 403 I=1,NPERG GAINNG(I)=X(I)*SQRT(VARGNN) 403 CCNTINUt C C PRETEST ANC POSTTEST SCORES C DO 404 I=1,NPERG L=NPERG+I XX(L)=T {(I)+EI NG(I) YY(L)=TMN( )+GAINNG(I)+E2NG(I) 404 CONTINUE C C C STANCARC ANALYSIS OF COVARIANCE COMPUTATIONS C N=2*NPERG C C INITIALIZATICN C SUPXG=0.0 SUPX2G=O.C SUVYG=0.0 SUMY2G=0.O SUMXYG=0.0 SUVXN=0.0 SUPX2N=0.0 SUVYN=O.O SUVY2N=O.C SUVXYN=O.C C C GRCUP SUMS AND SUMS OF SQUARES C CC 600 I=1,NPERG C SUVXC=SUPXG+XX(II SUrX2G=SUPX2(;+XX(I)*XX(I) SUPYG=SUfYG+YY(1) SUMY2G=SUPY2G+YY(I)*YY(I) SUVXYG=SUNXYG+XX(I)*YY(I) K=NPERG+I SUMXN=SUPXN+XX(K) SUPX2N=SUIX2N+XX(K) XX(K) SUMYN=SUMYN+YY(K) SUVY2N=SUPY2N+YY(K)*YY(K) SUXXYN=SUUXYN+XX(KI*YY(K) 600 CCNTINUC C C TOTAL SUMS AND SUMS OF SQUARES C TSU X=SUPXG+SUMXN TSUPX2=SUFX2G+SUMX2N TSUPY=SUMYG+SUMYN TSUVY2=SUrY2G+SUMY2N TSUVXY=SUPXYG+SUMXYN C C CCMPUTE TCTAL SUMS OF SQUARES C CFX=TSUMX*TSUPX/N CFY=TSUMY*TSUMY/N CFXY=TSUMX*TSUPY/N TXX=TSUPX2-CFX TYY=TSUVY2-CFY TXY=TSUMXY-CFXY C C COMPUTE BETWEEN GROUPS SUMS OF SQUARES C BXX=(SUX'G*SUMXG+SUMXN*SUMXN)/NPERG-CFX BYY=(SUMYG*SUMYG+SUMYN*SUMYN)/NPERG-CFY BXY=(SUMXCG*SUVYG+SUMXN*SUMYN)/NPERG-CFXY C C COMPUTE ERPCR SUMS OF SQUARES C EXX=TXX-PXX EYY=TYY-BYY EXY=TXY-e.XY C C CCMPUTE ADJUSTED SUMS OF SQUARES C TYYACJ=TYY-TXY*TXY/TXX iYYACJ=EYY-EXY*EXY/EXX BYYACJ=EYY-(TXY*TXY)/TXX+(EXY*EXY)/EXX C CCVPUTE ACJUSTEC KEAN SQUARES C VSB=BYYAEJ/1.0 VSE=EYYA[J/IN-3) C C CCVPUTE F STATISTIC C .F=VSB/MSE C C C ANALYSIS .CF COVARIANCE WITH PORTER'S ADJUSTMENT C C COMPUTE CORRELATION BETWEEN X AND Y C CENCP=TXX*TYY RXY=TXY/SCRT(CENOP) C C COMPUTE SUPS OF SQUARES WITH PORTER'S ADJUSTMENTS C EPCRT=EYYACJ TFCRT=TYY-((RXY*EXY+GXY)*2)/(IRXY*RXY*EXX)+BXX) BFCRT=TPCRT-EPGRT C C COMPUTE MEAN SQUARES C SBEP=EPCRT/1.G PSEP=EPCRT/(N-3) C C COMPUTE F STATISTIC WITH PORTER'S ADJUSTMENT C FPORTMPSEP/MSEP C IF (F.GT.4.45) NF10O=F10+1 IF (F.GT.3.89) NF100=NF1CO+1 IF (FFORT.GT.4.45) NFP10=NFP10+1 IF (FFCRT.GT.3.89) NFP100=NFP100+1 C WRITE CUT RESULTS C WR-ITE (6,800) NSAMP,F,FPORT 80 FORPAT (LX,T50,I4,T64,F8.3,T77,F8.3) C IF (NSAMP.LT.NSPL) GC TO 106 C WRITE (6,801) 801 FORMAT ('C',T10,'NF10',.T20,'NF10 ',T30,'NFP10',T40,'NFP100' a) WRITE (6,802) NF10,NF100,NFP10,NFP100 802 FORMAT (IX,T10,4(14,6X)) IF (GNRCUP.LT.NGP) GC TO 100 STCP ENh C C SUBRCLTINE RANGEN(M, IR,.X) CIPENSICN X(M) l=1 1 CALL RANCU(IR,JR,R1) IR=JR CALL RANCUIIR,JR,R2) IR=JR 1=2.0*(IR1-.5) R2=2.CI*R2-.5) S2=R1IRl+R2*R2 IF(S2.GT.1.0) GO TO 1 Y=SCRT(-2.O*(ALOG(S2)/S2)) X(I)=RisY X ( I ) = I1 Y IFII.EC.Y) GO TO 2 X(I+1)=R2*Y IF(I+I.EC.M) GO TO 2 1=1+2 GC TC I 2 RETURN END C SUBRCUTINE RANCU( IX, IY,.YFL) IY=IXn65539 IF(IY)5,6,6 5 IY=IY+2147483647t+ 6 YFL=IY YFL=YFL*.4656613E-9 RETURN ENC BIBLIOGRAPHY Box, G. E. P. and Muller, Mervin E. "A Note on the Generation of Random Normal Deviates," Annals of Mathematical Statistics, 29 (June, 1958), 610-611. Campbell, Donald T. and Erlebacher, Albert. "How Regression Artifacts in Quasi-Experimental Evaluations can Mistakenly Make Compen- satory Education Look Harmful," in Disadvantaged Child, Vol. 3, Ed. Hellmuth, Jerome. New York: Brunner/Marzel, Inc., 1970. Campbell, Donald T. and Stanley, Julian C. Experimental and Quasi- Experimental Designs for Research. Chicago: Rand McNally and Company, 1963. Cochran, William G. "Analysis of Covariance: Its Nature and Uses," Biometrics, 13 (September, 1957), 261-281. Cronbach, Lee J. Essentials of Psychological Testing, Third Edition. New York: Harper and Row, 1970. Cronbach, Lee J. and Furby, Lita. "How We Should Measure 'Change' or Should We?," Psychological Bulletin, 74 (January, 1970), 68-80. DeCecco, John P. The Psychology of Learning and Instruction: Educa- tional Psychology. Englewood Cliffs, N. J.: Prentice Hall, Inc., 1968. Ferguson, George A. Statistical Analysis in Psychology and Education, Third Edition. New York: McGraw-Hill Book Company, 1971. Fisher, Ronald A. Statistical Methods for Research Workers, Fourth Edition. London: Oliver and Boyd Ltd., 1932. Fisher, Ronald A. Statistical Methods for Research Workers, Tenth Edition. London: Oliver and Boyd Ltd., 1946. Fisher, Ronald A. The Design of Experiments. London: Oliver and Boyd Ltd., 1935. Garside, R. F. "The Regression of Gains Upon Initial Scores," Psychometrika, 21 (March, 1956), 67-77. Glass, Gene V., Peckham, Percy D., and Sanders, James R. "Consequences of Failure to Meet Assumptions Underlying the Fixed Effects Analysis of Variance and Covariance," Review of Educational Research, 42 (Summer, 1972), 237-288. Gulliksen, Harold. Theory of Mental Tests. New York: John Wiley and Sons, Inc., 1950. Helmstadter, G. C. Principles of Psychological Measurement. New York: Appleton-Century-Crofts, 1964. Hicks, Charles R. "The Analysis of Covariance," Industrial Quality Control, 20 (December, 1965), 282-287. Hilgard, Ernest R. Theories of Learning. New York: Appleton-Century- Crofts, Inc., 1956. Hill, Winfred F. Learning: A Survey of Psychological Interpretations, Revised Edition. Scranton: Chandler Publishing Company, 1971. Karmel, P. H. and Polasek, M. Applied Statistics for Economists, Third Edition. Bath, Great Britain: Pitman Publishing, 1970. Kirk, Roger E. Experimental Design: Procedures for the Behavioral Sciences. Belmont California: Wadsworth Publishing Company, Inc., 1968. Lord, Frederic M. "A Paradox in the Interpretation of Group Comparisons," Psychological Bulletin, 68 (1967), 304-305. Lord, Frederic M. "Elementary Models for Measuring Change," in Problems in Measuring Change. Ed. Chester W. Harris. Madison, Wisconsin: The University of Wisconsin Press, 1963. Lord, Frederic M. "Further Problems in the Measurement of Growth," Educational and Psychological Measurement, XVIII (1958), 437-451. Lord, Frederic M. "Large-Sample Covariance Analysis When the Control Variable is Fallible," Journal of the American Statistical Association, 55 (1960), 307-321. Lord, Frederic M. "Statistical Adjustments When Comparing Preexisting Groups," Psychological Bulletin, 72 (1969), 336-337. Lord, Frederic M. "Statistical Inferences about True Scores," Psycho- metrika, 24 (March, 1959), 1-17. Lord, Frederic M. "The Measurement of Growth," Educational and Psycho- logical Measurement, XVI (1956), 421-437. Lord, Frederic M. and Novick, Melvin R. Statistical Theories of Mental Test Scores. Reading, Massachusetts: Addison-Wesley Publishing Company, 1968. Manning, Winton H. and DuBois, Philip H. "Correlational Methods in Research on Human Learning," Perceptional and Motor Skills, 15 (1962), 287-321. ~~ ~~ ___ Marks, Edmond and Martin, Charles G. "Further Comments Relating to the Measurement of Change," American Educational Research Journal, 10 (Summer, 1973), 179-191. Marsaglia, G. and Bray, T. A. "A Convenient Method for Generating Normal Variables," SIAM Review, 6 (July, 1964), 260-264. McNemar, Quinn. "On Growth Measurement," Educational and Psychological Measurement, XVIII (1958), 47-55. Muller, Mervin E. "A Comparison of Methods for Generating Normal Deviates on Digital Computers," Association for Computing Machinery Journal, 6 (1959), 376-383. Neel, John Howard. A Comparative Analysis of Some Measures of Change. Dissertation, University of Florida (1970). O'Connor, Edward F., Jr. "Extending Classical Test Theory to the Measurement of Change," Review of Educational Research, 42 (Winter, 1972), 73-97. Peckham, Percy D. "The Robustness of the Analysis of Covariance to Heterogeneous Regression Slopes," Paper read at the Annual Meeting of the American Educational Research Association, Minneapolis, Minnesota, (March 5, 1970). Porter, Andrew Colvin. The Effects of Using Fallible Variables in the Analysis of Covariance. Dissertation, University of Wisconsin, University Microfilms, Ann Arbor, Michigan: (1967). Siegel, Sidney. Nonparametric Statistics for the Behavioral Sciences. New York: McGraw-Hill Book Company, 1956. Snedecor, George W. and Cochran, William G. Statistical Methods, Sixth Edition. Ames, Iowa: The Iowa State University Press, 1967. Thorndike, R. L. "Regression Fallacies in the Matched Groups Experiment," Psychometrika, 7 (June, 1942), 85-102. Werts, Charles E. and Linn, Robert L. "A General Linear Model for Studying Growth," Psychological Bulletin, 73 (1970), 17-22. Winer, B. J. Statistical Principles in Experimental Design, Second Edition. New York: McGraw-Hill Book Company, 1971. BIOGRAPHICAL SKETCH James Edwin McLean was born January 29, 1945, at Greensboro, North Carolina. He grew up in Orlando, Florida and graduated from Edgewater High School in June, 1963. He graduated from Orlando Junior College in January, 1966, and entered the United States Marine Corps Reserve. In December, 1968, he received the degree, Bachelor of Science, with a major in mathematics education from the University of Florida. In January, 1969, he enrolled in the Department of Statistics at the University and received the degree, Master of Statistics, in June, 1971. During this period, he worked as a graduate assistant in that department where he taught elementary statistics and probability. He accepted a teaching assistantship in the College of Education at the University of Florida in September, 1971. He currently holds that position part-time along with the position of research associate for a Project Follow Through evaluation grant. James Edwin McLean is a member of the American Educational Research Association, National Council on Measurement in Education, the American Statistical Association, and Phi Delta Kappa. He is married to the former Sharon Elizabeth Robb and they have no children. I certify that I have read this study and that in my opinion it conforms to acceptable standards of scholarly presentation and is fully adequate, in scope and quality, as a dissertation for the degree of Doctor of Philosophy. William B. Ware, Chairman Associate Professor of Education I certify that I have read this study and that in my opinion it conforms to acceptable standards of scholarly presentation and is fully adequate, in scope and quality, as a dissertation for the degree of Doctor of Philosophy. Vyjce A. Hines Professor of Education I certify that I have read this study and that in my opinion it conforms to acceptable standards of scholarly presentation and is fully adequate, in scope and quality, as a dissertation for the degree of Doctor of Philosophy. il liam ndenhall Profes-oor of Statistics I certify that I have read this study and that in my opinion it conforms to acceptable standards of scholarly presentation and is fully adequate, in scope and quality, as a dissertation for the degree of Doctor of Philosophy. P. V. Rao Professor of Statistics I certify that I have read this study and that in my opinion it conforms to acceptable standards of scholarly presentation and is fully adequate, in scope and quality, as a dissertation for the degree of Doctor of Philosophy. Jam T. McClave Assis ant Professor of Statistics This dissertation was submitted to the Dean of the College of Education and to the Graduate Council, and was accepted as partial fulfillment of the requirements for the degree of Doctor of Philosophy. August, 1974 Dean, College of Education Dean, Graduate School |

Full Text |

PAGE 1 AN EMPIRICAL EXAMINATION OF ANALYSIS OF COVARIANCE WITH AND WITHOUT PORTER'S ADJUSTMENT FOR A FALLIBLE COVARIATE By JAMES EDWIN McLEAN A DISSERTATION PRESENTED TO THE GRADUATE COUNCIL OF THE UNIVERSITY OF FLORIDA IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY UNIVERSITY OF FLORIDA 1974 PAGE 2 iS/Â«S'.Z?;f/^0R,0A 3 1262 08552 7488 PAGE 3 DEDICATION I dedicate this study to the late Dr. Charles M. Bridges, Jr., my teacher, advisor and friend. PAGE 4 ACKNOWLEDGMENTS I wish to express my grateful appreciation to my committee members for their guidance in this study. Dr. William B. Ware, my chairman, suggested the topic and was a major influence in its development. Dr. James T. McClave-was a constant source of assistance, both theoretically and editorially. Drs. Vynce Hines, William Mendenhall, and P. V. Rao all provided direction along with many helpful suggestions. This study could not have been completed without the spirit of cooperation which I encountered between members of the two departments involved. The late Dr. Charles M. Bridges, Jr. first encouraged me to enroll in the program and served as my chairman until his death. His counseling and guidance were a major reason for my successful completion of graduate study. The searching questions of my fellow students led to several worthwhile modifications and I wish to express my appreciation to them. I also wish to thank my wife, Sharon, who for the last five years, has held two jobs (wife and medical technologist), so that I might further my education. She has also been most understanding about the many hours spent away from home. m PAGE 5 TABLE OF CONTENTS Page ACKNOWLEDGMENTS iii LIST OF TABLES . vi ABSTRACT . .viii CHAPTER I. INTRODUCTION 1 The Problem of Comparing Groups of Differing Abilities .... 1 Methods of Comparing Groups of Differing Abilities ...... 3 Limitations Used in this Study 4 Procedures . . 5 Relevance of the Study 7 Organization of the Dissertation .... 7 II. REVIEW OF THE RELATED LITERATURE 8 Historical Review of the Problem 8 Analysis of Covariance 10 Analysis of Covariance with Porter's Adjustment for a Fallible Covariate . . . . . . . . . . ... . . . .11 Summary. ............ 14 III. PROCEDURES .16 The Model 16 Selecting the Reliabilities. . . . . . . . . . . . 17 The Regression of Y on X .18 Selecting Means .20 Generation of Random Normal Deviates with Specified Means and Variances 26 Analysis for Comparing the Selected Methods .26 IV. RESULTS 29 V. DISCUSSION 41 Comparison of the Two Methods of Analysis. . 41 Factors that Affect Alpha and Beta 42 TV PAGE 6 Page Predicting Alpha and Power . ;.......... 43 A Direction for Future Research. 46 VI. SUMMARY. . . . 47 APPENDIX ... : . . . . . . ... . ... ..... .49 Fortran Program Used to Perform Analysis 50 BIBLIOGRAPHY 56 BIOGRAPHICAL SKETCH. . 59 PAGE 7 LIST OF TABLES Table Page 1. CONDITIONS UNDER WHICH COMPARISONS WERE MADE WHEN THE RELIABILITIES FOR BOTH GROUPS WERE EOUAL. ..... 24 2. CONDITIONS UNDER WHICH COMPARISONS WERE MADE WHEN THE RELIABILITIES FOR BOTH GROUPS WERE UNEQUAL. . . . .25 3. FACTORIAL DESIGN FOR STANDARD ANALYSIS OF COVARIANCE WHEN MEAN GAIN OF GAIN GROUP WAS ZERO AND CRITERION VARIABLES ARE MONTE CARLO GENERATED ALPHA VALUES. ... 28 4. FRACTION OF SIGNIFICANT F'S AND NUMBER OF TIMES F^ EXCEEDS F WHERE THERE WAS NO GAIN IN EITHER GROUP AND THE PRETEST RELIABILITIES WERE EQUAL . 32 5. FRACTION OF SIGNIFICANT F'S AND NUMBER OF TIMES F^ EXCEEDS fo WHERE THERE WAS NO GAIN IN EITHER GROUP AND THE PRETEST RELIABILITIES WERE NOT EQUAL 33 6. FRACTION OF SIGNIFICANT F'S AND NUMBER OF TIMES Fg EXCEEDS L, WHERE THERE WAS GAIN IN THE GAIN GROUP AND THE PRETEST. RELIABILITIES WERE EQUAL. ....... 34 7. FRACTION OF SIGNIFICANT I'S AND NUMBER OF TIMES fs EXCEEDS Fp WHERE THERE WAS NO GAIN IN THE GAIN GROUP AND THE PRETEST RELIABILITIES WERE NOT EQUAL , 35 8. ANOVA SUMMARY TABLE USING MONTE CARLO GENERATED ALPHAS THE STANDARD ANALYSIS OF COVARIANCE AS THE CRITERION VARIABLES 36 9. ANOVA SUMMARY TABLE USING MONTE CARLO GENERATED ALPHAS FROM ANALYSIS OF COVARIANCE WITH PORTER'S ADJUSTMENT AS THE CRITERION VARIABLES. ....... . 37 10. ANOVA SUMMARY TABLE USING MONTE CARLO GENERATED POWERS FROM THE STANDARD ANALYSIS OF COVARIANCE AS THE CRITERION VARIABLES ..... 38 11. ANOVA SUMMARY TABLE USING MONTE CARLO GENERATED POWERS FROM ANALYSIS OF COVARIANCE WITH PORTER'S ADJUSTMENT AS THE CRITERION VARIABLES . .... . . .39, VI PAGE 8 Table Page 12. COEFFICIENTS FOR THE LINEAR CONTRASTS USED WHEN RELIABILITY WAS SIGNIFICANT IN THE ANALYSIS OF VARIANCE. . . 40 13. NINTY-FIVE PERCENT CONFIDENCE INTERVALS FOR THE' EXPECTED VALUES OF ALPHAS UNDER SPECIFIED CONDITIONS 44 14. NINTY-FIVE PERCENT CONFIDENCE INTERVALS FOR THE EXPECTED VALUES OF POWERS UNDER SPECIFIED CONDITIONS . . . . . .45 vn PAGE 9 Abstract of Dissertation Presented to the Graduate Council of the University of Florida in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy AN EMPIRICAL EXAMINATION OF ANALYSIS OF COVARIANCE WITH AND WITHOUT PORTER'S ADJUSTfCNT FOR A FALLIBLE COVARIATE by James Edwin McLean August, 1974 Chairman: William B. Ware Major Department: Foundations of Education The purpose of this study was to determine if analysis of covariance or analysis of covariance with Porter's true covariate substitution adjustment or neither is appropriate for analyzing pretest-posttest educational experiments with less than perfectly reliable measures. Monte Carlo techniques were employed to generate two thousand samples for each of forty-eight sets of conditions. These conditions included six combinations of reliability, two levels of sample size, two levels pf gain, and the equality or inequality of pretest means. Each sample was analyzed by both of the tested methods. The proportion of rejections for each method of analysis and the number of times the Â£ statistic for standard analysis of covariance exceeded that for analysis of covariance with Porter's adjustment were recorded. viii PAGE 10 The hypothesis, "The sampling distributions of the test statistics from analysis of covariance with and without Porter's adjustments are the same," was rejected at the .01 level of significance using the sign test for each of the forty-eight sets of conditions. To gain further insight into how the two methods of analysis differed, four factorial experiments were conducted using either the computer generated alphas or powers as the criterion variables. The first factorial experiment examined analysis of covariance when the mean gain was zero in both groups, thus, the computer generated alphas were used as criterion variables. The second experiment examined analysis of covariance with Porter's adjustment when the mean gain was zero in each group. Again, computer generated alphas were used as criterion variables, The third and fourth factorial experiments examined standard analysis of covariance and analysis of covariance with Porter's adjustment when the mean gain in only one group was positive. The four factorial experiments included three factors. These were reliability at six levels, sample size at two levels, and the equality of pretest means at two levels (pretest means equal and pretest means not equal). In each experiment, it was found that unequal pretest means combined with low reliability were significant sources of variation. Additional study showed that unequal pretest (covariate) means combined with low reliability produced misleading results. More specifically, when either method of analysis produced a significant Â£ statistic and there was a mean gain of zero in both groups but the pretest mean of one group was less than that of the other, the adjusted posttest means indicated that the group with the higher pretest mean had the larger gain. Likewise, when there' was a positive mean gain in the IX PAGE 11 group with the lower pretest mean, the adjusted posttest means indicated the group with a mean gain of zero had the larger gain. The results of this study combined with the results of others concerned with analysis of covariance where the covariates are fallible point to the inadequacy of the technique as it now exists. These results should be kept in mind if the technique is to be used. PAGE 12 CHAPTER I INTRODUCTION . Regardless of the theory or hypothesis being investigated, educational researchers must, at some point, relate it to learning. One common element of most definitions of learning is that a change must take place in the learner (DeCecco, 1958, p. 243; Hilgard, 1956, p. 3; Hill, 1971, p. 1-2). Thus most investigations in education are concerned with identifying, measuring, and comparing these changes. In the past decade, analysis of covariance has become a standard procedure for comparing groups with different levels of ability. The basic derivation of this procedure requires the assumption that the covariable does not contain errors of measurement (Cochran, 1957). In recent years, several researchers have recognized that this assumption is violated when. the qovari'ate is a mental test score (Lord, 1960; Porter, 1967; Campbell and Erlebacher, 1970). Lord (I960) and Porter (1967) have proposed adjustments to the analysis of covariance procedure to use when the covariable is fallible, that is, when measurement error is present in the covariable. The focus of this study is to determine if analysis of covariance and analysis of covariance with Porter's adjustment are different for a fallible covariate and if either is appropriate under varying conditions of reliability, sample size, gain, and pretest mean. The Problem of Comparing Groups of Differing Abilities The problem of comparing groups based on change manifests itself prominently when the variables are mental measurements. Most physical PAGE 13 science measurements are made with the aid of some type of physical instrument. If an engineer is interested in the weight gain of a metal before and after a galvanizing process, instruments are available to measure the weight of the metal to within at least one microgram. On the other hand, if an educator is interested in the change in a student's I.Q. before and after taking part in an experimental program, an error of 5 points or more would not be uncommon. Based on the Wechsler Intelligence Scale for Children with a standard error of measurement of 5.0 (Cronbach, 1970, p. 222), an error of 5.0 I.Q. points or more would occur in approximately 32 percent of the measurements, assuming a normal distribution. If the actual change in I.Q. were near zero for an individual, it is srery likely that the error of measurement would exceed the change itself. The problem is compounded further as the groups being compared often are not of the same ability level , Many of the recent programs featuring innovative teaching practices for compensatory education are available only for the most need^^ and the comparison group is then sampled from the general population of students (Campbell and Erlebacher, 1970). Programs such as Head Start and Follow Through have probably been the victim of tragically misleading analyses, such as in the Westinghouse/ Ohio University study (Campbell and Erlebacher, 1970). Personal contact with the evaluation of Follow Through projects has emphasized the magnitude of the problem. Warnings about the inappropriate uses of existing modes of analysis have come from several sources (Lord, 1960, 1967, 1969; Campbell and Erlebacher, 1970). Remedies have been offered (Lord, 1960; Porter, 1967), but at this time there is no conclusive evidence to indicate these remedies are PAGE 14 appropriate. Studies concerning the robustness of the analyses to violated assumptions have also been in conflict (Peckham, 1970). Methods of Comparing Groups of Differing Abilities Jvio general approaches are available to compare groups with differing abilities. The first is to compare the average change in one group with the average change in the other group by means of a _t test or analysis of variance. The second is to use analysis of covariance, where the pretest score is the covariate. There are at least three methods of computing change scores to be considered if the first approach is used. One method is to obtain raw difference scores by simply subtracting each pretest score from its corresponding posttest score. Another method is Lord's true gain (Lord, 1956) which was further developed by McNemar (1958). Basically, this method requires the use of a regression equation to estimate the "true" gain or change from pretest to posttest. A third method of measuring gain, the method of residual gain , was proposed by Manning and DuBois (1962). This method involves regressing the posttest scores on the pretest scores and obtaining a predicted posttest score for each subject. The indicator of change is the observed posttest score minus the predicted posttest score. This method is mathematically equivalent to the standard analysis of covariance (Neel , 1970, p. 30-31), the second approach. The second general approach is the standard analysis of covariance procedure found in many statistical texts (e.g.. Kirk, 1968; Winer, 1971). One of the reported functions of a covariate is to adjust the treatment means of the dependent variable for differences in the values of corresponding independent variables (Hicks, 1965). Logically, it seems that by PAGE 15 using the pretest scores of groups differing in ability as a covariate, the treatment means can be statistically adjusted for the differences in pretest means, A deficiency of this method is the fallibility of the covariate. A fallible variable is one which is not measured without error or with perfect reliability (Lord, 1960). Lord (1960), recognizing this deficiency in the use of analysis of covariance, derived an adjustment to compensate for the fact that the pretest scores v^ere fallible. Porter (1967) modified Lord's procedure for more than two groups and empirically investigated the sampling distribution of Lord's statistic. Porter's solution has been suggested as an alternative to the standard *-Â•' analysis of covariance when the covariate is fallible. Limitations Used in this Study This study was designed to examine the characteristics of the two covariance methods within the framework of simulated situations which may occur in the analysis of a compensatory education project. Realistically, several limitations were imposed. Even with these limitations, the two methods of analysis were investigated under forty-eight combinations of differing reliability, sample size, mean gain, and pretest means. The study was limited to two groups, henceforth called the gain and no-gain groups. Investigating more than two groups would not only increase the number of possibilities appreciably but would not be in line with the objective stated previously, that is, comparing a compensatory education group with a "comparison" group. The levels of the reliabilities being investigated are also limited to the range .50 to .90. Seldom do mental measurement instruments have reliabilities above .90 and instruments with reliabilities below .50 are generally considered inadequate. PAGE 16 The levels of sample size were 10 and 100 subjects per group. These quantities are representative of small and large sample sizes encountered in educational experiments. Furthermore, the true score and true gain variances will be fixed a priori with the true gain variance always four percent of the true score variance. These figures are empirically based on Follow Through achievement data in keeping with the aforementioned objective. Procedures The study compared the two methods of analysis under forty-eight sets of conditions. The two methods were analysis of covariance and analysis of covariance with Porter's adjustment for a fallible covariate. Three levels of reliability and two levels of sample size were used. The three levels of reliability will include situations where the reliability of the pretests are assumed equal and situations where they differ to more closely simulate compensatory education project evaluation. Each of these combinations of conditions was repeated for four separate cases. Case I is where there is no-gain in either group and '^ both groups have equal pretest means. Case II is where there is gain in only one group (the gain group) and both groups have equal pretest means. Cases III and IV are with and without gain where the pretest means are different. For each of the forty-eight sets of conditions, two thousand samples were computer generated and analyzed using both methods of analysis and a .05 level of significance. Two thousand samples provide empirical estimates of the fraction of type I and type II errors to within .01 of their true values with ninety-five percent confidence. The samples were PAGE 17 generated from an assumed normal population. The generation technique is one proposed by Box and Muller (1958) and modified by Marsaglia and Bray (1964). This procedure generates random normal deviates with a mean and variance of and 1 respectively. These normal variables are then transformed to attain the specified means and variances. The IBM 370/165 computer of the North Florida Regional Data Center was used for the generation and analyses. Using these analyses, two research questions were examined: 1. Is there any difference between the sampling distributions of the test statistics of standard analysis of covariance and analysis of covariance with Porter's adjustment? 2. What factors affect each type of analysis? The first research question can be restated in terms of a null hypothesis for each set of conditions: The sampling distributions of the test statistics from analysis of covariance with and without Porter's adjustment are the same for each set of conditions. This hypothesis was tested for each reliability, sample size, gain, and pretest mean combination. The second research question was examined in four factorial experiments. Each factorial experiment included the factors of reliability at six levels, sample size at two levels, and equality of pretest means at two levels^ The first experiment consisted of examining standard analysis of covariance with computer generated alpha values as the criterion variables. The second examined analysis of covariance with Porter's correction also using computer generated alpha values as the criterion variables. The third and fourth factorial experiments examined analysis of PAGE 18 covariance with and without Porter's adjustment respectively, using computer generated powers as criterion variables. Relevance of the Study This study is designed to compare two recommended methods of analyzing educational experiments under known simulated conditions which are presumably realistic. The results should indicate whether either of the methods is appropriate for its recommended use, thus giving educational researchers some direction when faced with the choice. If both methods were shown to be appropriate, the study could determine which one is superior. The study should also indicate if either method of analysis is appropriate for a set of restricted conditions, e.g. for only certain levels of reliability. Organization of the Dissertation A statement of the problem, a description of possible solutions, and an overview of the procedure of this study has been included in Chapter I. A comprehensive review of the related literature is provided in Chapter II. This review includes a historical overview of the problem and a detailed description of the methods being compared. A description of the procedures followed for this study is contained in Chapter III. The data and analyses are presented in Chapter IV and a discussion of the results is provided in Chapter V. A summary of the study is provided in Chapter VI. PAGE 19 CHAPTER II REVIEW OF THE RELATED LITERATURE Comparing groups of differing abilities has been a problem for some time. The literature has included discussions of possible solutions for at least the past three decades. Many of these discussions have been in conflict with one another, and there is still no general agreement. The evaluation of federal compensatory education projects in the last decade has intensified the debate and the need for a theoretically sound and practically useful solution. Some of the solutions proposed over the years are matching subjects, gain scores, and analysis of covariance. A discussion of these solutions and why they may not be considered satisfactory is provided in this chapter. The major properties of analysis of covariance are also considered. Because Porter's adjustment for a fallible covariable in analysis of covariance is not readily available in the literature, its derivation is provided in this chapter. Historical Review of the Problem A paper by Thorndike (1942) examined in detail the fallacies of comparing groups of differing abilities by matching subjects. The crux of his argument was that the regression effects were systematically different "whenever matched groups are drawn from populations which differ with regard to the characteristics being studied," (p. 85). During the late 1950 's and early 1960's the literature was inundated with papers on how to or how not to measure gain or change. One of the leaders during this period was Lord (1956, 1958, 1959, 1963). His PAGE 20 proposal for estimating the "true" gain was originally set forth in 1956 with further developments in 1958 and 1959. McNemar (1958) extended his results for the case of unequal variances among the groups. Garside (1956) also proposed a method for estimating gain scores and Manning and DuBois (1962) presented their derivation of residual .gain scores. The debate over what techniques, if any, should be used for measuring gain or change was at its peak in the early 1970's. Cronbach and Furby (1970) concluded that one should generally rephrase his questions about gain in other ways. Marks and Martin (1973) underscored the Cronbach and Furby (1970) conclusion. O'Connor (197.2) reviewed developments of gain scores in terms of classical test theory. Neel (1970) employed Monte Carlo techniques to compare four identified methods for measuring gain. The compared methods were raw difference. Lord's true gain, residual gain, and analysis of covariance. Under equivalent conditions, he found that Lord's true gain tended to produce a greater significance level than the user would intend, that is, a higher fraction of type I errors than alpha. The analysis of covariance method has been widely recommended by a number of authorities in the field (Thorndike, 1942; Campbell and Stanley, 1963; O'Connor, 1972). Campbell and Erlebacher (1970) point out that the Westinghouse/Ohio University Study was evaluated, possibly incorrectly, using the analysis of covariance. Lord (1960, 1967, 1969) has sounded a warning about its use that has been echoed by others (Werts and Linn, 1970; Campbell and Erlebacher, 1970; Winer, 1971). The warning stressed that the analysis of covariance requires the assumption that the covariate ''''^ is measured without error. Lord (1960) has proposed an adjustment which PAGE 21 10 was later generalized by Porter (1967). The adjustment is based on the substitution of the true score estimate for the observed value of the covariate. Analysis of Co variance The analysis of covariance procedure was originally introduced by Sir Ronald A. Fisher (1932, 1935). According to Fisher (1946, p. 281), the analysis of covariance "combines the advantages and reconciles the requirements of two widely applicable procedures known as regression and ^ analysis of variance." The procedure is well documented in contemporary texts (Snedecor and Cochran, 1967, p. 419-446; Kirk, 1968, p. 455-489; : Winer, 1971, p. 752-812). Analysis of covariance is a popular technique in both the physical and social sciences. Among the principal uses of analysis of covariance pointed out by Cochran (1957), p. 264) is "to remove the effects of disturbing variables in observational studies." It was thought that by using the pretest score as the covariate and comparing two groups with analysis of covariance, the effects of different pretest scores could be eliminated. Lord (1960) pointed out that this was not necessarily the case when the covariate was fallible. The assumptions necessary for the analysis of covariance are the same as those for analysis of variance with the addition of the following: / 1. The covariates are measured without error. / 2. The regression coefficient is constant across all treatment groups (Peckham, 1970). Violation of the first assumption induces a bias in the analysis of covariance because of "the presence of 'error' and 'uniqueness' in PAGE 22 11 the covari ate, i .e. variance not shared, by the dependent variable. If the proportion of such variance can be correctly estimated, it can be corrected for," (Campbell and Erlebacher, 1970, p. 199). This position was upheld by Glass et al . (1972). The basic algebraic derivation of this correction was presented by Lord (1960) and expanded by Porter (1967] Analysis of Covariance with Porter's Adjustment for a Fallible Covariate Lord's (1960) derivation of the analysis of covariance adjustment was limited to two treatment groups and is slightly more difficult than is Porter's (1967) procedure. Thus, Porter's procedure will be derived here and used in the analysis. Porter's adjustment is based on the substitution of the estimated true value for the covariate. Let X denote a fallible variable (e.g. a pretest score) , r an estimate of the reliability of X, and T-j the true value of the variable X^. Let T-j denote the estimated true score of X-j . The definition of T-j for each individual is (1) Ti = X + r(Xi-X), or T^. = rX^+ X(l-r), Porter (1967) derived the mean and variance of T-j as follows. By definition, the mean of all T-j's is: (2) T= ^Ti^ IT' = E[rX^+)r(l-r)] ii by substitution. Thus T = rX + NX-NrX , N = X , PAGE 23 12 where N denotes the sample size and it is understood that i is summed over the values 1, 2, ---, N. Also, by definition, the variance of t is (3) S^ = E(fi-X)^ ^ N-1 = r2s2 X' and (4) Sj^ = z(fi-)()(Yi-Y) , N-1 where Y is the dependent variable. Based on these results. Porter (1967) derived an analysis of covariance procedure replacing the fallible covariate, Xj , with the estimated true score, T^-. The analysis of covariance requires the computation of analysis of variance sums of squares for the dependent variable, the covariable, and on the cross-products of the dependent variable and the covariable. Porter (1967) showed that the use of estimated true scores for the covariable did not affect the analysis of variance of the dependent variable, Y. The changes found by Porter (1967) in the other two cases are as follows: For the analysis of variance of the estimated true scores, f, (5) SS^^ = lE(fij-f.j)2 = r2j:z(Xij-X.j)^, PAGE 24 13 where SSj,, denotes the within groups sum of squares. (6) SSn. = nE(t.-i-f..)' = nE(X..-X..)^ where SSg^ denotes the between groups sum of squares, (7) SSy. = ZE(Ti-j-T..)' T = r2zzX?.+(l-r)E n N where SSj^ denotes the total sum of squares, n denotes the number of (T,Y) pairs per treatment group, and N denotes the total number of (t,Y) pairs. In a similar manner, the cross-products sums of squares are: (8) (9) (10) SSwfv = ^^^(Xij-X.j) (Yij-Y.j), SSBfY ^ "^(X.j-X..)(Y.j-Y..)^ SSt. = rEZX,-iY,--i+(l-r)E (i:Xij)(zY,-j) Â— J^ Thus, the adjusted sums of squares are (rSS )2 SSu' = SSi^ ^XY , Y ~2^7 " (11) SS,, X ^XY SS,, (rSSw + SSn )2 (12) SS't = SSt XY XY ^^^\ \ SSb^ (13) SS ' = SS' SS ' PAGE 25 14 Note that the adjusted v/ithin groups sum of squares remains unchanged by the substitution of T for X but the substitution does alter the adjusted total sum of squai'es and consequently, the adjusted between groups sum of squares. A question arises concerning what value to use as the estimate of reliability in the formulae. A solution to this problem was proposed by Campbell and Erlebacher (1970). In a pretest-posttest situation one may find it reasonable to make two assumptions that would generate appropriate commonfactor coefficients. First, if one has only the pretest-posttest correlations, one may assume that the correlation in the experimental group was unaffected by the treatment. (We need a survey of experience in true experiments to check on this.) Second, one may assume that the common-factor coefficient is the same for both pretest and posttest. Under these assumptions, the pretestposttest correlation coefficient itself becomes the relevant common-factor coefficient for the pretest or covariate, the "reliability" to be used in Lord's and Porter's formulas, (p. 200) This recommendation will be followed in this study, thus the correlation between the pretest scores and posttest scores will be used as the reliability estimate in, the formulae. S umma ry As noted, analysis of covariance and analysis of covariance with Porter's adjustment have been recommended by several authors. Campbell and Erlebacher (1970) computer generated data for two overlapping groups with no true treatment effect and concluded that the analysis of covariance method was inappropriate and that analysis of covariance with Porter's adjustment should only be undertaken with great tentativeness. Porter (1967) computer generated data to compare the _F sampling distribution with the theoretical Â£ distribution using his adjustment. He concluded that samples of 20 or larger were needed to have a useful approximation to the theoretical F distribution. He also PAGE 26 15 found that the estimation degenerated further when the reliability was less than .7. No study was found which compared both techniques under similar conditions for both gain and no-gain groups. PAGE 27 CHAPTER III PROCEDURES The analysis of covariance and the analysis of covariance with Porter's adjustment were compared under forty-eight sets of conditions on the basis of computer generated data. Six combinations of reliability, two sample sizes, two levels of gain, and two different sets of pretest means were used. Both equal and unequal reliabilities were used in the comparisons. A random sample of two thousand observations was generated under each combination of conditions and analyzed by analysis of covariance with and without Porter's adjustment. The sampling distributions of the two analyses were statistically compared with a sign test for each of the forth-eight sets of conditions. The computer generated alpha values and powers were then used as the criterion variables in four factorial experiments. Subsequent a posteriori analyses were performed where warranted. The Model A standard model was used to represent the pretest and posttest scores of one subject. The model follows the traditional measurement approach as found in Gulliksen (1950) or Lord and Novick (1968) and extended to gain score theory. by O'Connor (1972). (14) X = T + E^ and (15) Y = T + G + E2 16 PAGE 28 17 where . X = observed pretest score, Y = observed posttest score, T = true pretest score G = true gain, Â£]= random measurement error in pretest score, Â£2= random measurement error in posttest score. The following properties about E-j and E2 are assumed to exist: The errors E^ and Eo i) have zero means in the group tested, ii) have the same variances for both groups, iii) are independent of each other and of the true parts of each test. It is further assumed that T and G are independent (across subjects) and that all components follow a normal probability distribution. These assumptions parallel Lord's' (1956) stated and implied assumptions. Selecting the Reliabilities Reliability is related by definition to the variances of the observed scores, the true scores, and error. This section shows how that relationship can be used to establish desired reliabilities. The basic definition of reliability (Helmstadter, 1964, p. 62) is given by equation (16) where the symbol, p^^, denotes reliability. (16) p^x = 4~4 2 Â°X . Then, PAGE 29 ;i7) a2 = o2(l_p ) 1 XX Choosing the variance of X a priori to be 100, the variance of E-\ can be found in the following manner as the reliability of X assumes different values: (18) -1 100(1 -p ) XX The independence of T and E-] in (14) implies (19) "2 ^x = ^T ' i. Combining (18) and (19) and solving for a^ yields (20) 1 = ^Â°SxThe posttest variances can be selected in a similar manner. Based on the assumptions. (21) a^ + a ? + a2 The variance of the gain scores is chosen in the manner prescribed in Chapter I. Then, combining (16) and (21), ;22l 4. + 2 'T P|YY where pyy denotes the established reliability of the posttest scores. Thus it can be seen that the effect of selecting specified reliabilities can be obtained by selecting the variances of E-] , E2, and T in accordance with (18), (20) , and (22) respectively. The Regression of Y on X Recall that one of the assumptions necessary for analysis of covariance (Cochran, 1957) is that the regression slopes of the dependent PAGE 30 19 variable on the covariate must be equal for each treatment group. Let gy-x denote the regression slope for one treatment group. Then, (Ferguson, 1971, p. 113), (23) PAGE 31 20 compensatory programs with a comparison group sampled from the general population of untreated children in the same community such as with the Westinghouse/Ohio University study (Campbell and Erlebacher, 1970). These comparison children tend to be higher in ability than the treatment group. The resulting differences in variation tend to produce different reliabilities. In order to simulate such a situation this study used different reliabilities for the gain and the no-gain groups in addition to the comparisons when the reliabilities are equal. The reliabilities for the gain group are decreased by fifteen percent which roughly approximates the reduced variation empirically observed in Project Follow Through achievement data. Selecting Heans The true score mean was set a priori at 100 when both the gain and no-gain groups have equal means. The situation in which the gain group has a lov;er mean was also analyzed. In this case, the true score mean of the gain group was set at 80. These values have been empirically chosen based on Project Follow Through data. Based on the assumptions, (27) E(X) = E(T+Et) = y^, hence the mean of the observed pretest scores is equal to the mean of the true scores. Also (28) E(Y) = E(T+G+E2) = y-^+yg. Thus, the mean of the observed posttest scores is equal to the sum of the means of the true scores and the gain scores. Clearly in the case of the no-gain group and in both groups where no gain was used, the mean gain was zero. The selected value of yQ for the gain situation was based on power considerations, that is, y^ was PAGE 32 21 chosen such that the power of an Â£ test for analysis of covariance was .50. A linear model representation of the analysis of covariance for two groups is (29) Y = eg +BiX+e2W+e where X is the pretest score (covari ate), Y is the posttest score, and W is a dummy variable designating group membership (W=l if gain group, if no-gain group). Testing the hypothesis that 62 ""s equal to in equation (29) is equivalent to the F test for treatments in the analysis of covariance procedure. It can be shown that B2 i" equation (29) is equivalent to the mean gain, pg. The mean of the posttest for the gain group is (30) E(Yg) = 30+^1^X3+^2 where Yq is a posttest score for the gain group and p^ "is the mean preG test score for the gain group. Likewise, the mean of the posttest for the nogain group is (31) E(Yng) = Bo+6iyXf^e where Y|^q is a pretest score for the no-gain group, pv is the mean preNG test score for the no gain group. But it has been assumed that p^ and PXfjQ are equal. Furthermore, the gain group has mean gain, pg and the no-gain group has mean gain zero, thus (32) pg = E(Yg)-E(YNg) = B2 ' Hence, choosing the value of 62 'that yields a power of .50 is equivalent to choosing a value of pg to produce a power of .50 in the analysis of, covariance procedure. PAGE 33 22 This power can be obtained from the following probability statement: (33) Pr[t*>tJ = .50 where t_* is a noncentral t_ statistic. This expression can be approxi' mated by the substitution of a z, statistic for t*. (34) Pr ^2-H > t = .50 Thus a value of 60 can be chosen such that the substituted z_ statistic is equal to t Q25 with the appropriate degrees of freedom. This value of &2 ""s the value of the average gain, \ir, such that the power of analysis of covariance is .50 under the condition of perfect reliability. In order to find this value of yg, a numerical expression for or is needed. The variance of 63 > of Â» can be approximated in the following manner. Karmel and Polasek (1970, p. 245) state that a? is 62 (35) 2 2 J= ar; 32 Y e(x-x) v^2 [e(X-X)2][e(W-W)^]-[i:(X-X)(W-W)]^ Dividing both the numerator and denominator by N^ and substituti ng population variances for sample variances, a? is approximately equal 62 , . . to the following: (36) 32, 2 2 ^Y Â°X N 2 2/ .2 Ov o,, -{a^J 'X "W XW' The quantity, oy^^, the covariance between X and W has been assumed equal to zero thus, in equation (36), the Ow's divide out. Hence PAGE 34 23 (37) a? = a? 2Under the conditions assumed for the model, the variance of Y, '^Y, 2 is equal to 4. The variance of W, Â°W can be computed to be .25. Thus, 2 2 by substitution, oequals .80 when N is equal to 20 and ^g equals .08 1^2 2 when N is equal to 200. Hence, for N = 20, Mq can be found by the following expression: (38) yg = t ^^^ ,^a;= 2.1] /rm = 1.88. 20 ~-U^o,l/ ^2 Likewise, for N = 200, yg can be found by solving equation (39). (39) yg =t 025 igyaa = 1. 97/708 = .56. 200 -u^o^iy/ ^2 The approximated values of yg were tested using Monte Carlo generated variables and found to indeed produce a power of .50. The approximated values of yg under each set of conditions are shov/n in Tables 1 and 2. PAGE 35 24 TABLE 1 CONDITIONS UNDER WHICH COMPARISONS WERE MADE WHEN THE RELIABILITIES FOR' BOTH GROUPS WERE EQUAL PAGE 36 25 TABLE 2 CONDITIONS UNDER WHICH COMPARISONS WERE MADE WHEN THE RELIABILITIES FOR BOTH GROUPS WERE UNEQUAL RELIABILITY PAGE 37 26 Generation of Random Normal Deviates with Specified Means and Variances The study required the use of computer generated normally distributed random variables with specified means and variances. Two thousand sets of variables were generated for each of the forty-eight sets of conditions. Muller (1959) identified and compared six methods of generating normal deviates on the computer. A method described by Box and Muller (1958) was judged most attractive from a mathematical standpoint. According to Muller (1959, p. 379), "Mathematically this approach has the attractive advantage that the transformation for going from uniform deviates to normal deviates is exact." This method was endorsed by Marsaglia and Bray (1964). They modified the algorithm to reduce central processing computer time without altering its accuracy. The method first requires the generation of two independent uniform random variables, U^ and U2, over the interval (-1, 1). The variables and Z^ = U^[-2 ln(U^+U^) / {U^ + ul)2^^^ I2 = U2[-2 In(U^ui) / (U? + \ilW^^ will be two independent random variables from the same normal distribution with mean zero and unit variance. The variables were then transformed to have the desired means and variances. Analysis for Comparing the Selected Methods Each of the two thousand sets of generated data for the forty-eight sets of conditions was analyzed by analysis of covariance and analysis of covariance with Porter's adjustment. The number of times the F statistic from analysis of covariance exceeded the Â£ statistic from analysis of covariance with Porter's adjustment was noted along with the proportion of rejections by each method of analysis. PAGE 38 27 A sign test (Siegel, 1956, p. 63-67) was then performed for each set of conditions to test the null hypotheses in Chapter I, that is, "the sampling distributions of the test statistics from analysis of covariance with and without Porter's adjustment will be the same for each set of conditions." These tests were run at the .01 level of significance. The fraction of rejections noted (alphas and powers) was then used as the dependent variable in four factorial experiments to gain further insight into what factors affected each method of analysis being studied. Each factorial experiment had three factors. These were reliability which included six combinations of reliability used in the study, sample size, which included n =10 and n = 100 subjects per group, and the equality of pretest means, which included a level where both pretest means v/ere equal and a level where they differed. The layout of the factorial experiments is illustrated in Table 3. The factorial experiment illustrated is the situation for which there was no gain in either group and the standard analysis of covariance was used. Thus, the dependent variables are the alpha values generated by the computer. When significant main effects or interactions occurred, the appropriate a posteriori analytical procedures to locate the sources of the variation were followed. PAGE 39 28 LU O <: CD o O rO < 1Â— I in LU _J IÂ— <; eC LU CO O LU CC Q tJ3 i-i cC O 2: rvi PAGE 40 CHAPTER IV RESULTS The. number of times the Â£ statistic from the standard analysis of covariance exceeded the F statistic from analysis of covariance with Porter's adjustment is listed for each set of conditions in Tables 4, 5, 6, and 7. These quantities were used as test statistics for the sign tests used. to test the hypothesis, "There is no difference between the sampling distributions of the test statistics from analysis of covariance with and without Porter's adjustment." This hypothesis was tested for each of the forty-eight sets of conditions at the .01 level of significance. Siegel (1956) stated that a large sample test statistic for the sign test is (40) z = X-.5N 757R~ where N is equal to the number of pairs of observations, x is equal to the number of times the first measurement of the pair exceeds the second measurement of the pair, and z_ is the standard normal variate. For a level of .01, the null hypotheses would be rejected when z was less than -2.33 or greater than 2.33. This is equivalent to rejecting the null hypotheses when x was less than 949 or greater than 1051 and N equals 2000. Thus, an inspection of the last column of Tables 4, 5, 6, and 7 reveals that the null hypothesis was rejected for each of the fortyeight sets of combinations. The analysis of variance summary tables avq presented in Tables 8, 29 PAGE 41 30 9, 10, and 11. The analysis of variance summary table for the case when the Monte Carlo generated alpha values from the standard analysis of covariance were used as the criterion variables is presented in Table 8. The analysis of variance summary table for the case when the Monte Carlo generated alpha values from the analysis of covariance with Porter's adjustment were used as the criterion variables is presented in Table 9. The analysis of variance summary tables for the cases when Monte Carlo generated powers were used as the criterion variables for standard analysis of covariance and analysis of covariance with Porter's adjustment are presented in Tables 10 and 11, respectively. Each analysis of variance table includes analyses of the simple effects where they are warranted. Scheffe's S method for testing linear contrasts is included in Table 8. These linear contrasts are defined in Table 12. Each F statistic which exceeds the critical value at the .05 level is denoted by an asterisk. A result of particular interest is applicable to analysis of covariance both with and without Porter's adjustment. That is, when a spuriously high fraction of significant Â£ statistics occurred when there was a mean gain of zero in both groups and the pretest means differed, the adjusted posttest means indicated that the gain was in favor of the group having the largest pretest mean. This result was more pronounced for lower reliabilities. When the gain group had a positive gain, the no-gain group had a mean gain of zero, and the no-gain group also had a larger pretest mean, a similar situation occurred. In this situation, when a significant Â£ statistic occurred, the adjusted posttest means usually indicated the PAGE 42 31 no-gain group had recorded the larger gain. Again, these results became more pronounced as the reliability of the scores were reduced. PAGE 43 32 CO Q Q (yO LU Q; LU : 2: <: Q ct: -I O LU LU IÂ— z: uj tÂ— cC o d: I u. IÂ— X o o t/) LU Q. 1Â— Â—I PAGE 44 33 CO Q Q t/) ljj q: uj q: 2: cc Q o; UJ 1Â— I Q UJ LU CQ IÂ— :s LU IÂ— S < O Di ZD U(Â— X O ^ o 1/1 UJ o. UJ _l PAGE 45 34 t/) a Q 1/1 LU O^ UJ QC ^ PAGE 46 35 U_| LL.\ 00 Q Q (Â•) LU Di LJ q: s: ct Q cc: LU >Â— I Q LU LU -CQ IÂ— 2: LU fÂ— 2: ca: o q; ZD UIÂ— X O 2: O CyO UJ CL. ,0 hC/) c: _j PAGE 47 36 TABLE 8 ANOVA SUMMARY TABLE USING MONTE CARLO GENERATED ALPHAS FROM THE STANDARD ANALYSIS OF COVARIANCE AS THE CRITERION VARIABLES PAGE 48 37 TABLE 9 ANOVA SUMMARY TABLE USING MONTE CARLO GENERATED ALPHAS FROM ANALYSIS OF COVARIANCE WITH PORTER'S ADJUSTMENT AS THE CRITERION VARIABLES SOURCE PAGE 49 38 TABLE 10 ANOVA SUMMARY TABLE USING MONTE CARLO GENERATED POWERS FROM THE STANDARD ANALYSIS OF COVARIANCE AS THE CRITERION VARIABLES SOURCE PAGE 50 39 TABLE 11 ANOVA SUMMARY TABLE USING MONTE CARLO GENERATED POWERS FROM ANALYSIS OF COVARIANCE WITH PORTER'S ADJUSTMENT AS THE CRITERION VARIABLES SOURCE PAGE 51 40 TABLE 12 COEFFICIENTS FOR THE LINEAR CONTRASTS USED WHEN RELIABILITY WAS SIGNIFICANT IN THE ANALYSIS OF VARIANCE LEVELS OF RELIABILITY RELIABILITY OF GAIN GROUP RELIABILITY OF NO GAIN GROUP 90 PAGE 52 CHAPTER V DISCUSSION In general, the results of this study support the positions of Lord (1967, 1969), Campbell and Erlebacher (1970), and O'Connor (1972) : with respect to their warnings about the implications of analysis of covariance using unreliable test scores. The study shoves that even when there is no gain in either group, a much higher fraction of rejections occur than would be expected. The fraction of rejections is even more extreme when the reliability is .70 or lower and the pretest means differ. In addition. Porter's adjustment seems to offer little improvement. Comparison of the Two Methods of Analysis The rejection of all forty-eight null hypotheses concerning the equality of the sampling distributions for the two methods of analysis shows that there is a difference in the results obtained from analysis '-'^ of covariance and analysis of covariance with Porter's adjustment. A closer examination shows that these differences are more extreme when the pretest means of the two groups differ and the reliabilities are '' low. Further study shows that although the two methods of analysis are different, neither method does an adequate job of modeling reality, that is, both methods tend to produce erroneous proportions of type I and II errors v/hen pretest means differ and the reliabilities are low. When the pretest means are equal, the power of the tests seem to be directly related in a positive manner to the reliabilities when other variables are held constant. 41 PAGE 53 42 Possibly the most far reaching results were a function of re Ij Ability-. The data indicated that incorrect decisions about which group had the larger gain could be made using either method of analysis when the pretest means of the two groups differed and the reliabilities were low. When the pretest mean of the gain group was| less than that of the no-gain group, the adjusted posttest means stiowed that the no-gain group was superior both when the mean gain was zero in both groups and when the mean gain was positive only in the gain group. This possibility was pointed out by Lord (1967) land Campbell and Erlebacher (1970). -Â• I '.-' Factors that Affect Alpha and Beta The factorial experiments using the computer generated alphas and betas allow one to infer which factors affect the levels of type I and type II errors. Using Monte Carlo generated alphas from the standard analysis of covariance in a factorial experiment indicated that an interaction between pretest means and reliability and an interaction between sample size and pretest means affected the alphas significantly. Further analyses (simple effects) showed that the pretest mean factor was significant at every level of reliability. Both reliability and sample size were significant when the pretest means differed. The tests of linear contrasts showed that there was no significant difference between equal reliabilities and unequal reliabilities when the pretest means differed, but there were significant differences among the levels of reliability when the pretest means differed. These results indicate that a difference in reliabilities between groups, thus a difference in slopes, has no effect, however, the level of reliability does. The other three factorial experiments indicated that interactions PAGE 54 43 between pretest means and sample size were the major contributors to the differing levels of alpha and power. In all three experiments (See Tables 10, 11, and 12) the sample size was significant when the pretest means differed. Reliability seems to have a somewhat moderate effect in these cases. Predicting Alpha and Power Using the results of this study, a regression equation can be set up to predict the experimental probability of a type I error or the experimental probability of rejecting a false null hypotheses when the established probabilities are .05 and .50 respectively. The basic regression equation is (41) Y = eo+3TRG+32RN+B3S+34M+B5RQRf^+36RGS +e7RQM+B3RfjS+3gR[^M+6iQSM+G where Â• Y = the predicted alpha or power, Rq = the reliability of the gain group scores, R^ = the reliability of the no gain group scores, S = sample size M = 1 if the pretest means are equal, otherwise, and 3i = the regression coefficient, i. Tables 14 and 15 provide confidence intervals for the expected values of alpha and power respectively for each set of conditions noted. PAGE 55 44 TABLE 13 NINETY-FIVE PERCENT CONFIDENCE INTERVALS FOR THE EXPECTED VALUES OF ALPHAS UNDER SPECIFIED CONDITIONS PAGE 56 45 TABLE 14 NINETY-FIVE PERCENT CONFIDENCE INTERVALS FOR THE EXPECTED VALUES OF POWER UNDER SPECIFIED CONDITIONS PAGE 57 46 The predicted value of alpha or power can be found by using the data found in Tables 4, 5, 6, and 7 to fit regression equation (41) to obtain the following estimated regression parameters: (42) alpha = .228 + .395Rq + .332R,^ + .008 S .661M -1.09 RqR,^ .004 RqS + .619 RgM -.003 R|^jS + .195 R|^M .007 SM and (43) power = .045 + .536Rg + .404 R|^ + .010 S .700 M -1.228 RqR,^ .006 RqS + .846 RgM + .002 RfjS + .218 RfjM .007 SM. A confidence interval for the predicted values of alpha and power could be obtained in the usual manner. A Direction for Further Research The results of this study combined with other research (Lord, 1967; Campbell and O'Connor, 1972), indicates a need for further study and the development of a robust method for comparing groups of differing ability when the scores are not perfectly reliable. Any new technique which is proposed should first be investigated under known conditions, either analytically, or by Monte Carlo techniques. This would insure that another inappropriate method is not used for the evaluation of compensatory education projects. PAGE 58 CHAPTER VI SUMMARY This study was designed to determine if either analysis of covar^. iance or analysis of covariance with Porter's adjustment is an appropriate analytical procedure for evaluating educational pretest-posttest experiments. In particular, these methods: were compared with respect to their use in the analysis of compensatory education projects where the groups may differ in ability. The study was carried out by computer generating 2000 sets of normal data under forty-eight sets of predetermined conditions of reliability, sample size, gain, and equality of pretest means. Each of the 2000 sets of data for each set of conditions was then analyzed using both standard analysis of covariance and analysis of covariance with Porter's adjustment. A sign test was used to compare the two methods of analysis under each of the forty-eight sets of conditions. It was concluded that -^ the two methods of analysis yielded different results. Factors affecting the two methods of analysis were then studied separately using the computer generated alphas and powers as criterion variables in four factorial experiments. The factors included reliability at six levels, sample size at two levels, and the equality of pretest means at two levels. From these experiments, it was concluded that pre"^ test means interacting with sample size and sometimes with reliability were significant factors. More specifically, sample size was statistically significant in each case where the pretest means differed. Also, pretest means were significant at ^M^r^ level of reliability for the computer 47 PAGE 59 48 generated alphas produced by the standard analysis of covariance. It was also learned that when the pretest means differ, both standard analysis of covariance and analysis of covariance with Portier's adjustment produced erroneous results with respect to which group if either had a gain. When both groups had a mean gain of zero and the pretest means differed, significant results usually indicated that the group with the larger pretest mean had the gain. This would correspond to the control group sampled from the general population being credited with the gain in a compensatory education experiment. When there was a gain in only one group and the pretest mean was lower in that group, the analyses still indicated that the other group had the gain. The results of this study point to the recommendations that analysis of covariance with or without Porter's adjustment should be approached ^ with caution when the reliabilities are below .90 and the pretest means (covariate means) are likely to be different for the groups. PAGE 60 APPENDIX PAGE 61 50 FCRTRAN PRCGRAN WHICH PERFORMED DATA GFNERATIOiN AND ANALYSIS DIh'ENSICN EIGN(IOO), E2GN( 100) ,GAINGN{ ICO) tXX( 2G0 ) , YY ( 200 ) , Â»X(100) .ElNGdOO), E2NG( 100 ),GAINNC( I0a),TG(100) ,TN(I00) REAL .NS[3,NSE,MSBP,MSeP C c c c c c c c c c 99 100 101 C c c 102 103 104 105 C C C . NGRCUP=0 INPUT CF GRCUP PARAMETERS WHEREO RELGN REPRESENTS THE RELIABILITY OF THE GAIN GROUP DATA RELNG REPRESENTS T HE R EL I AB IL I T Y OF THE NO GAIN GROUP NPERG REPRESENTS THE NUMBER OF OBSERVATIONS PER GROUP GBAR REPRESENTS THE MEAN GAIN FOR GAIN GROUP PR^NG REPRESENTS THE GAIN GROUP PRETEST MEAN FR^NNG REPRESENTS THE NO GAIN GROUP PRETEST MEAN ISEEC REPRESENTS THE SEED FOR RANDOM NUMBER GENERATOR READ (5,99) I S EED , NGP , NS PL FORMAT ( IH,2X,2I5) READ 15,101) RELGN, RELNG, NPERG, GDAR,PRMNG,PRMNNG FORMAT {2F3.2, 13, F3.2,2F3.0) NGRCUP = NGl-iGUP + l NSAh'F = NF10=0 NF 100=0 NFP10=0 NFP100=0 HEADER CARD FOR NEW SET OF PARAMETERS WRITE (6,102) FORMAT (Â• l',T43, ' F STATISTICS BASED ON THE FOLLOWING PARME' Â• ,Â«TERSM WRITE (6,103) FORMAT ( Â•0',T22, 'RELGN Â•,2X, 'RELNG Â•,2X, 'NPERG ',2X,' GBAR' Â»,' 'i^X.'PRMNG Â• ,2X, 'PRMNNG' ) WRITE (6,104) RELGN, RELNG, NPERG, GBAR, PRMNGfPRMNNG FORMAT (1X,T22,2F8.5, I5,3X,3F8.3) WRITE (6,105) FORMAT ( '0' ,T45, "SAMPLE NUMB ER' , 5X ,' STANDARD F ', 5X ,' PORTER ' Â», 'S F') COMPUTE VARIANCE COMPONENTS VARE1G=100Â»{1-RELGN) VARE1N=100Â«(1-RELNG) VARTG=100Â«RELGN VARTK=100"RELNG VARGNG=.04Â»VAf?TG VARGNN=.04Â«VARTN PAGE 62 51 C 106 C C C C c c 300 c c c 301 C c c 302 C C C 303 C C C 304 C C C C C C VARE2G=(V/iRTG+VARGNG)/RELGN-VARTG-yARGNG VARE2N=( VARTN+VARGNN ) /RELNG-V ARTN-VARGNN CCNTINUc NSAMP=;NSA^P + 1 GENERATICN CF CATA FOR GAIN GROUP TRUE SCORES CALL RAKGENCNPERG, ISEECX) DO 3C0 I=1,NPERG TG(I )=X( I ) Â•SQRTIVARTGI+PRMNG CCNTINUE PRETEST ERROR SCORES CALL RANGEN(NPERG,ISEED,X) CO 301 I=1,NPERG eiGN(I)=X(I)Â»SeRT(VARElG) CCNTINUE POSTTEST ERROR SCORES CALL RANGEN(NPERG,.IS'EEn,,X) DC 302 I^l.NPERG E2GN( I)=XI I)Â»SCRT(VARE2G) CONTINUE GAIN SCORES CALL RANGEMNPERG, ISEEO.X ) CO 303 I = 1,NPERG GAINGNi I )=X( I )Â«SQRT{ VARGNG)+GBAR CONTINUE PRETEST ANC POSTTEST SCORES [JO 30 A I = 1,NPERG XXd )=TG{ I)+E1GN( I ) YY( I l=TG{ I)+GAINGN( I ) + E2GN( I ) CCNTINUE GENERATION OF CATA FOR NO GAIN GROUP TRUE SCORES CALL RANGEN(NPERG,ISEED,X) 00 400 I=1,NPERG PAGE 63 52 400 r. c c 401 c c c 402 C C c 403 C C C 404 C C C C C C C TN( I )nX( I )Â«SQRT(VARTN)+PRMNNG CCMINUC PRETEST ERRCR SCORES CALL PANGENCMPERG, ISEECX) DC .401 I = 1,NPERG . E1NG( I) = Xn )Â»SGRTIVAPE1N) CCNTINUE POSTTEST ERROR SCORES CALL PANGEN(NPERG, ISEED.X ) CC 402 I=1,NPERG E2NG( I)=X(I )Â»SCRT(VARE2N) CCNTINUt GAIN SCORES CALL RANGEN(NPERG, ISEED.X) DC 403 1=1,NPERG GAINNG( I )=X( I )Â«SQRT{ VARGNN) CCNTlKUt: PRETEST ANC POSTTEST SCORES DO 404 I=1,NPERG L=NPERG+I XX(L)=TN( I)+E1NG( I ) YY{L)=TN( I )+GAINNG{ I ) + E2NG( I ) CCNTINUE STANCARC ANALYSIS OF COVARIANCE COMPUTATIONS N=2*NPERG IMTIALIZATICN SUNXG=0.0 SUNX2G=0.C SU^YG=0.0 SUMY2G=0.0 SUNXYG=O.C SUNXN=a.O 50^X2^=0.0 SUN'YN=0.0 SUyY2N=0.C SUNXYN=0.C GROUP SUMS AND SUMS OF SQUARES PAGE 64 53 600 C C C DC 6G0 I=1,NPERG sur'XG=sur'XG+xx{i ) SUNX2G = SUNX2GtXX( I )Â»XX( I ) . SUr'YG = SUI^YG+YY( I ) SUNY2G = SUNY2G + YY( I )Â»YY( I ) . SU^'XYG = SU^XYG + XX( I )Â»YY( I ) K=NPtRG+I SUr'XN^SUNXN+XXCK ) SU^X2N = SUr'X2N + XX(K)Â»XX(K) SUNYN = SUNYN + YY (K ) SU^Y2^ = SU^Y2^J+YY(K)Â»YY{K) SUr'XYN=SUNXYrj+XX (K)Â»YY(K ) GCNTINUC TOTAL SUMS AND SUMS OF SQUARES TSUVX=SUNXG+SUMXN TSUMX2=SU^X2G+SUMX2N TSUI^Y = SU^'YG + SUMY^I TSUNY2=SUNY2G+SUMY2N TSU^XY=SUNXYG+SUMXYN COMPUTE TOTAL SUMS OF SQUARES CFX=TSUMXÂ«TSU^X/N CFY=TSUMYÂ«TSU^Y/N CFXY = TSUMXÂ«TSU^'Y/N TXX=TSUMX2-CFX TYY=TSUKY2-CFY TXY=TSU^XY-CFXY COMPUTE BETWEEN GROUPS SUMS OF SQUARES BXX= (SUMX'G*SUMXG4-SUMXNÂ»SUMXN )/NPERG-CFX BYY=(SUMYGÂ«SUMYG+SUMYN*SUMYN)/NPERG-CFY BXY= (SUMXGÂ»SUMYG+SUMXNÂ»SUMYN )/NPERG-CFXY COMPUTE ERROR SUMS OF SQUARES Exx=Txx-exx EYY=TYY-eYY EXY = TXY-e.XY COMPUTE ACJUSTED SUMS OF SQUARES TYYACJ=TYY-TXYÂ«TXY/TXX Â£YYACJ=EYY-EXYÂ«EXY/EXX BYYACJ=LiYY-{TXYÂ»TXY)/TXX + (EXYÂ«EXY)/EXX PAGE 65 54 C G 80C C C 801 802 CCNPUTE ACJUSTEC MEAN SQUARES ^Se=BYYACJ/l.C ySE = EYYA[:j/ (M-3) CC^'PUTE F STATISTIC . F = N'SB/MSE ANALYSIS CF COVARIANCE WITH PORTER'S ADJUSTMENT CCPPUTE CCRRELATION BETWEEN X AND Y CENCN=TXXÂ«TYY RXY=TXY/SGRT( CENOy) COMPUTE SUNS OF SQUARES WITH PORTER'S ADJUSTMENTS EPCRT=EYYACJ TPCRT=TYY-l {RXYÂ»EXY+eXY)Â»Â«2)/( (RXY*RXYÂ»EXX)+BXX) BPCRT=TPCRT-EPCRT COMPUTE MEAN SQUARES NSEP=EPCRT/I.O MSEP=EPCRT/{N-3) COMPUTE F STATISTIC WITH PORTER'S ADJUSTMENT FPORT-MSBP/MSEP IF (F.GT.A.^5) NFlO=NFin+l IF (F.GT.3.89J NF I00 = KF 1G0+, 1 IF (FFORT.GT.^.45) N FP 10 = NFP lC + 1 IF (FFCRT.GT.3.89) N FP IOO=NFP 1C0+ 1 WRITE CUT RESULTS WRITE 16,800) NSAMP, F,FPORT FORMAT (IX,T50, I4,T6A, F8.3,T77,F8.3) IF (NSAMP. LT. NSPL ) GC TO 106 WRITE (6,801) FORMAT ( 'C ,T10, Â•NF10.Â«,.T20, 'NF10CÂ«,T30, 'NFPIO' ,T40, 'NFPIOO' Â•) WRITE (6,802) NF 10 , N F 100 , NFP IC ,NFP 100 FORMAT ( 1X,T10,4( H,6X ) ) IF (KGRCUP.LT.NGP) GC TO 100 STOP END PAGE 66 55 SUDRCL'TINE RANGEN ( K, I R ,.X ) CINENSICN X(M) 1 = 1 CALL RANCU( IRtJR.Rl) 1R= JR CALL PANCU( IR, JR,R2) 1R = JR Rl = 2.GÂ» PAGE 67 BIBLIOGRAPHY Box, G. E. P. and Muller, Mervin E. "A Note on the Generation of Random Normal Deviates," Annals of Mathematical Statistics , 29 (June, 1958), 610-611. ' ' Campbell, Donald T. and Erlebacher, Albert, "How Regression Artifacts in Quasi -Experimental Evaluations can Mistakenly Make Compensatory Education Look Harmful," in Disadvantaged Child , Vol. 3, Ed. Hellmuth, Jerome. New York: Brunner/Marzel , Inc., 1970. Campbell, Donald T. and Stanley, Julian C. Experimental an d Quasi Experimental Designs for Research . Chicago: Rand McNally and Company, 1963. Cochran, William G. "Analysis of Covariance: Its Nature and Uses," Biometrics , 13 (September, 1957), 261-281. Cronbach, Lee J. Essentials of Psychological Testing , Third Edition . New York: Harper and Row, 1970. Cronbach, Lee J. and Furby, Lita. "How We Should Measure 'Change' or Should We?," Psychologic al Bulletin, 74 (January, 1970), 68-80. DeCecco, John P. The Psychology of Learning and Instruction: Educational Psychology . Englev;ood Cliffs, N. J.: Prentice Hall, Inc., 1968. Ferguson, George A. Statistical Analysis in Psychology and Education, Third Edition . New York: McGraw-Hill Book Company, 1971. Fisher, Ronald A. Statistical Methods for Research Workers, Fourth Edition. London: Oliver and Boyd Ltd., 1932. Fisher, Ronald A. Statistical Methods for Research Workers, Tenth Edition . London: Oliver and Boyd Ltd., 1946. Fisher, Ronald A. The Design of Experiments . London: Oliver and Boyd Ltd., 1935. Garside, R. F. "The Regression of Gains Upon Initial Scores," Psychometri ka , 21 (March, 1956), 67-77. Glass, Gene V. , Peckham, Percy D. , and Sanders, James R. "Consequences of Failure to Meet Assumptions Underlying the Fixed Effects Analysis of Variance and Covariance," Revjew of Educational Research , 42 (Summer, 1972), 237-288. 56 PAGE 68 57 Gulliksen, Harold. Theory of Mental Tests . New York: John Wiley and Sons, Inc., 1950. Helmstadter, G. C. Principles of Psychological Measurement . Nev; York: Appleton-Century-Crofts, 1964. Hicks, Charles R. "The Analysis of Covariance," Industrial Quality Control , 20 (December, 1965), 282-287. Hilgard, Ernest R. Theories of Learning . New York: Appleton-CenturyCrofts, Inc. , 1956. Hill, Winfred F. Learning: A Survey of Psychological Interpretations , Revised Edition . Scranton: Chandler Publishing Company, 1971. Karmel , P. H. and Polasek, M. Applied Statistics for Economists, Third Edition . Bath, Great Britain: Pitman Publishing, 1970. ~ Kirk, Roger E. Experimental Design : Procedures for the Behavioral Sciences . Belmont California: Wadsworth Publishing Company, Inc., 1968. Lord, Frederic M. "A Paradox in the Interpretation of Group Comparisons," Psychological Bulletin , 68 (1967), 304-305. Lord, Frederic M. "Elementary Models for Measuring Change," in Problems JnMeasuring Change . Ed. Chester W. Harris. Madison, Wisconsin: The University of Wisconsin Press, 1963. Lord, Frederic M. "Further Problems in the Measurement of Growth," Educational and Psychological Measurement . XVIII (1958), 437-451. Lord, Frederic M. "Large-Sample Covariance Analysis When the Control Variable. is Fallible," Journal of the American Statistical Association , 55 (1960), 307-321. Lord, Frederic M. "Statistical Adjustments When Comparing Preexisting Groups," Psychological Bulletin , 72 (1969), 336-337. Lord, Frederic M. "Statistical Inferences about True Scores ," Psycho metrika . 24 (March, 1959), 1-17. Lord, Frederic M. "The Measurement of Growth," Educational and Psychological Measurement , XVI (1956), 421-437. Lord, Frederic M. and Novick, Melvin R. Statistic al The ories of Mental Test Scores . Reading, Massachusetts: Addi son-Wesley Publishing Company, 1968 . Manning, Winton H. and DuBois, Philip H. "Correlational Methods in Research on Human Learning," Perceptional and Motor Skills . 15 (1962), 287-321. PAGE 69 58 Marks, Edmond and Martin, Charles G. "Further Comments Relating to the Measurement of Change," American Educational Research Journal , 10 (Summer, 1973), 179-19T: ' ~ Marsaglia, G. and Bray, T. A. "A Convenient Method for Generating Normal Variables," SIAM Review . 6 (July, 1964) .' 260-264. McNemar, Quinn. "On Growth Measurement," Educational and Psychological Measurement , XVIII (1958), 47-55. Muller, Mervin E. "A Comparison of Methods for Generating Normal Deviates on Digital Computers," Association for Computing . Machinery Journal , 6 (1959), 376-383. Neel , John Howard. A Comparative Analysis of Some Measures of Change . Dissertation, University of Florida (1970). O'Connor, Edward F., Jr. "Extending Classical Test Theory to the Measurement of Change," Review of Educational Research , 42 (Winter, 1972), 73-97. Peckham, Percy D. "The Robustness of the Analysis of Covariance to Heterogeneous Regression Slopes," Paper read at the Annual Meeting of the American Educational Research Association, Minneapolis, Minnesota, (March 5, 1970). Porter, Andrew Colvin. The Effects of Using Fallible Variables in the ' Analysis of Covariance . Dissertation, University of Wisconsin, University Microfilms, Ann Arbor, Michigan:. (1967) . Siegel , Sidney. Nonparametric Statistics for the Behavioral Sciences . New York: McGraw-Hill Book Company, 1956. Snedecor, George W. and Cochran, William G. Statistical Methods, Sixth Edition . Ames, Iowa: The Iowa State University Press, 1967. Thorndike, R., L. "Regression Fallacies in the Matched Groups Experiment," Psychdmetrika , 7 (June, 1942), 85-102. Werts, Charles E. and Linn, Robert L. "A General Linear Model for Studying Growth," Psychological Bulletin , 73 (1970), 17-22. Winer, B. J. Statistical Principles in Experimental Design, Second Edition . New York: McGraw-Hill Book Company, 1971. PAGE 70 BIOGRAPHICAL SKETCH James Edwin McLean was born January 29, 1945, at Greensboro, North Carolina. He grew up in Orlando, Florida and graduated from Edgewater High School in June, 1963. He graduated from Orlando Junior College in, January, 1966, and entered the United States Marine Corps Reserve. In December, 1968, he received the degree. Bachelor of Science, with a major in mathematics education from the University of Florida. In January, 1969, he enrolled in the Department of Statistics at the University and received the degree. Master of Statistics, in June, 1971. During this period, he worked as a graduate assistant in that department where he taught elementary statistics and probability. . He accepted a teaching assistantship in the College of Education at the University of Florida in September, 1971. He currently holds that position part-time along with the position of research associate for a Project Follow Through evaluation grant. James Edwin McLean is a member of the American Educational Research Association, National Council on Measurement in Education, the American Statistical Association, and Phi Delta Kappa. He is married to the former Sharon Elizabeth Robb and they have no children. 59 PAGE 71 I certify that I have read this study and that in my opinion it conforms to acceptable standards of scholarly presentation and is fully adequate, in scope and quality, as a dissertation for the degree of Doctor of Philosophy. William B. Ware, Chairman Associate Professor of Education I certify that I have read this study and that in my opinion it conforms to acceptable standards of scholarly presentation and is fully adequate, in scope and quality, as a dissertation for the degree of Doctor of Philosophy. A/''>'L^-(Â—VyXce A. Nines Professor of Education I certify that I have read this study and that in my opinion it conforms to acceptable standards of scholarly presentation and is fully adequate, in scope and quality, as a dissertation for the degree of Doctor of Philosophy. [Â• ^Ji 1 1 laflrt'lftndenhal 1 i ~~ Profes~sorJ of Statistics I certify that I have read this study and that in my opinion it conforms to acceptable standards of scholarly presentation and is fully adequate, in scope and quality, as a dissertation for the degree of Doctor of Philosophy. ^ Â— y V ii p. V. Rao Professor of Statistics I certify that I have read this study and that in my opinion it conforms to acceptable standards of scholarly presentation and is fully adequate, in scope and quality, as a dissertation for the degree of Doctor of Philosophy. n. ^m-^W Jame^T. McClave Assistant Professor of Statistics PAGE 72 This dissertation was submitted to the Dean of the College of Education and to the Graduate Council, and was accepted as partial fulfillment of the requirements for the degree of Doctor of Philosophy. August, 1974 Dean, Graduate School xml version 1.0 encoding UTF-8 REPORT xmlns http:www.fcla.edudlsmddaitss xmlns:xsi http:www.w3.org2001XMLSchema-instance xsi:schemaLocation http:www.fcla.edudlsmddaitssdaitssReport.xsd INGEST IEID E5SREUA0R_7BH8NS INGEST_TIME 2017-07-17T20:29:48Z PACKAGE UF00098341_00001 AGREEMENT_INFO ACCOUNT UF PROJECT UFDC FILES |