UFDC Home  Search all Groups  UF Institutional Repository  UF Institutional Repository  UF Theses & Dissertations   Help 
Material Information
Thesis/Dissertation Information
Subjects
Notes
Record Information

Full Text 
PAGE 1 1 SAMPLE SIZE IN EXPLORATORY FACTOR ANALYSIS WITH ORDINAL DATA By RONG JIN A DISSERTATION PRESENTED TO THE GRADUATE SCHOOL OF THE UNIVERSITY OF FLORIDA IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY UNIVERSITY OF FLORIDA 2012 PAGE 2 2 2012 Rong Jin PAGE 3 3 To my Parents Caiyun Li and Yonghe Jin PAGE 4 4 ACKNOWLEDGMENTS First of all, I would like to thank God for leading me through the doctorial study with peace and rejoice. I am thankful to Dr. James Algina, Dr. David Miller, and Dr. Walter Leite for giving me the opportunity to study in Research and Evaluation Methodology Program, and for supporting me during the entire doctoral study. I especially thank Dr. James Algina, my committee chair and advisor, for his insightful mentoring, patient guidance, and for being a n academic role model. I thank Dr. David Miller for his instrumental advice which helped my confidence growing in the entire process. I am grateful to Dr. Walter L eite for his constant encouragement and systematic training in the simulation research which made me strong academically. I also want to thank Dr. Lei Zhang for being my very supportive external committee member I wish to thank Dr. John M. Ferron, Chair o f Department of Educational Measurement and Research at University of South Florida, for his detailed technical explanation related to my research. I want to express my gratitude to the Office of Graduate Studies Alumni Fellowship Committee for the financial support that has made these four years of study possible. I also wish to thank Dr. Maureen Conroy, professor at Department of Special Education, for giving me t he opportunity to work at Best i n Class Project for the last two semesters, and fo r letting me obtain valuable research experience in the special education field I also appreciate Dr. Jann MacInnes for the sincere friendship and constant encouragement Last but not the least: I am thankful for the endless love and solid support by my husband, daughter, and my parents. Without them, it would not have been possible PAGE 5 5 TABLE OF CONTENTS page ACKNOWLEDGMENTS ................................ ................................ ................................ .. 4 LIST OF TABLES ................................ ................................ ................................ ............ 7 LIST OF ABBREVIATIONS ................................ ................................ ............................. 9 ABSTRACT ................................ ................................ ................................ ................... 10 CHAPTER 1 INTRODUCTION AND LITERATURE REVIEW ................................ ..................... 12 Background ................................ ................................ ................................ ............. 12 Modeling Ordinal Data in FA and IRT ................................ ................................ ..... 14 Finding from Past Studies ................................ ................................ ....................... 16 Variables Influence Factor Recovery in CFA with Ordinal Data ....................... 16 Effect of sample size and its interaction with estimation method ............... 17 Effect of factor loadings ................................ ................................ ............. 18 Effect of overdetermination of factors and its interaction with estimation method ................................ ................................ ................................ .... 18 Effect of number of response categories in item response and its interaction with estimation methods ................................ ........................ 19 Effect of degree of non normality of ordinal data ................................ ....... 19 Effect of estimation method ................................ ................................ ........ 21 Effect of interfactor correlation and its interactions with estimation method ................................ ................................ ................................ .... 23 Effect of number of factors ................................ ................................ ......... 23 Effect of model size and its interaction with estimation method ................. 23 Variables that Influence Factor Recovery in EFA with Continuous Data .......... 24 Effects of sample size and its interaction with factor loading ..................... 26 Effect of factor loading ................................ ................................ ............... 26 Effects of number of factors, number of items, and overdetermination of factors and their interaction with factor loading ................................ ....... 27 Effect of rotation methods ................................ ................................ .......... 27 Ranks of the effects ................................ ................................ ................... 28 Variables that Influence Factor Recovery in EFA with Ordinal Data ................. 28 Summary ................................ ................................ ................................ ................ 30 Research Questions ................................ ................................ ............................... 31 2 DESIGN O F MONTE CARLO SIMULATION STUDY ................................ ............. 42 Data Generation ................................ ................................ ................................ ..... 42 Estimation Methods ................................ ................................ ................................ 44 Number of Factors ................................ ................................ ................................ .. 44 PAGE 6 6 Number of Items per Factor ................................ ................................ .................... 44 Factor Loading ................................ ................................ ................................ ........ 45 Sample Size ................................ ................................ ................................ ............ 46 Selecting the Number of Factors and Rotation ................................ ....................... 47 Interfactor Correlation ................................ ................................ ............................. 47 Number of Categories in Item Response ................................ ................................ 48 Number of Replications ................................ ................................ ........................... 48 Factor Recovery Indices ................................ ................................ ......................... 48 3 RESULTS ................................ ................................ ................................ ............... 52 Convergence Rates and Improper Sol ution Rates ................................ .................. 52 Repeated Measures ANOVA ................................ ................................ .................. 53 Congruence Coefficient ................................ ................................ ................ 55 Ro ot Mean Squared Deviation g ................................ ................................ ...... 58 General Pattern Accuracy ................................ ................................ ................ 61 Total Pattern Accuracy ................................ ................................ ..................... 63 Per El ement Accuracy ................................ ................................ ...................... 64 4 CONCLUSIONS AND DISCUSSION ................................ ................................ ...... 81 Brief Summary ................................ ................................ ................................ ........ 81 Research Question 1 ................................ ................................ ........................ 81 Research Question 2 ................................ ................................ ........................ 83 Research Question 3 ................................ ................................ ........................ 84 Comparison of Findings with Previous Studies ................................ ....................... 86 DWLS vs. ULS ................................ ................................ ................................ 86 Main Effects and Interactions ................................ ................................ ........... 87 Sample Size ................................ ................................ ................................ ..... 89 Future Research Questions ................................ ................................ .................... 91 Conclusions ................................ ................................ ................................ ............ 92 LIST OF REFERENCES ................................ ................................ ............................... 99 BIOGRAPHIC AL SKETCH ................................ ................................ .......................... 104 PAGE 7 7 LIST OF TABLES Table page 1 1 Overview of previous studies ................................ ................................ .............. 32 1 2 Analysis type and data type in the previous studies ................................ ........... 33 1 3 Simulation conditions in the previous studies part 1 ................................ ........... 34 1 4 Simulation conditions in the previous studies part 2 ................................ ........... 35 1 5 Estimation methods in the previous studies ................................ ....................... 36 1 6 Number of factors in the previous studies ................................ .......................... 37 1 7 Number of items per factor in the previous studies ................................ ............. 38 1 8 Factor loadings in the previous studies ................................ .............................. 39 1 9 Sample size in the p revious studies ................................ ................................ ... 40 1 1 0 Number of replications in the previous studies ................................ ................... 41 3 1 Congruence coefficient ................................ ................................ ....................... 66 3 2 Summary ANOVA table results for congruence coefficient ................................ 67 3 3 Mean congruence coefficient by NF and PFL ................................ ..................... 68 3 4 Mean congruence coefficient by NI and PFL ................................ ...................... 68 3 5 Mean congruence coefficient by PFL and SS ................................ ..................... 68 3 6 Root mean squared deviation ................................ ................................ ............. 69 3 7 Summary ANOVA table results for root mean squared deviation ....................... 70 3 8 Mean root mean squared deviation by NF and PFL ................................ ........... 71 3 9 Mean root mean squared deviation by NI ................................ ........................... 71 3 10 Mean root mean squared deviation by SS ................................ .......................... 71 3 11 General pattern accuracy ................................ ................................ ................... 72 3 12 Summary ANOVA table results for general pattern accuracy ............................. 73 3 13 Mean general pattern accuracy by NF and PFL ................................ ................. 74 PAGE 8 8 3 14 Mean general pattern accuracy by PFL an d SS ................................ ................. 74 3 15 Total pattern accuracy ................................ ................................ ........................ 75 3 16 Summary ANOVA table results for total pattern accuracy ................................ .. 76 3 17 Mean total pattern accuracy by NF and NI ................................ ......................... 77 3 18 Per element accuracy ................................ ................................ ......................... 78 3 19 Summary ANOVA table results for per element accuracy ................................ .. 79 3 20 Mean per element accuracy by NF and PFL ................................ ...................... 80 3 21 Mean per elemen t accuracy by SS ................................ ................................ ..... 80 4 1 Sample size recommendation for a good and an excellent congruence coefficient ................................ ................................ ................................ ........... 95 4 2 Mean congruence coefficient by PFL, NI, and SS ................................ .............. 96 4 3 Mean root mean squared deviation by PFL, NI, and SS ................................ ..... 96 4 4 Mean general pattern accuracy by PFL, NI and SS ................................ ........... 97 4 5 Mean total pattern accuracy by PFL, NI, and SS ................................ ................ 97 4 6 Mean per element accuracy by PFL, NI, and SS ................................ ................ 98 PAGE 9 9 L IST OF ABBREVIATIONS CFA Confirmatory Factor Analysis DWLS Diagonally Weighted Least Squares EFA Exploratory Factor Analysis ML Maximum Likelihood RMSE Root Mean Square Error ULS Unweighted Least Squares WLS Weighted Least Squares PAGE 10 10 Abstract of Dissertation Presented to the Graduate School of the University of Florida in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy SAMPLE SIZE IN EXPLORATORY FACTOR ANALYSIS WITH ORDINAL DATA By Rong Jin August 2012 Chair: J ames Algina Major: Research and Evaluation Methodology The effect of sample size in e xploratory factor analysis with ordinal data was explored by a simulation study which investigated factor recovery under different settings of estimation methods, number of factors, number of items per factor, magnitude of primary loadings, and sample size. F actor recovery was measured by congruence coefficient, root mean squared deviation, general pattern accuracy, total pattern accuracy, and per element accuracy. The re sults suggested the factor recovery was very similar for diagonally weighted least squares and unweighted least squares across all of the simulation conditions B etter factor recovery was associated with a smaller number of factors, a larger number of items per factor, higher primary loadings, and a larger sample size. The number of factors, number of items per factor, and sample size had a small effect on factor recovery when the primary loadings were high and became important determ inants when the primary loadings were medium. S mall sample size was sufficient for a good congruence coefficient as long as the primary factor loadings were high However, larger sample size was required when the factor loadings were medium and the sample PAGE 11 11 size requirement was influenced by the number of factors and the number of items per factor Sample size recommendations were made based on the findings in the present study. A sample size of 200 resulted in a good congruence coefficient for all of the c onditions with 2 factors and for all of the conditions with 4 and 6 factors when the primary loadings were large However, a larger sample size was needed with medium primary factor loadings : for four factors a sample size 200 was sufficient for 14 items per factor and a sample size 600 was sufficient for 7 items per factor. For 6 factors, no sample sizes in the range of 200 to 800 produced a good congruence coefficient. PAGE 12 12 CHAPTER 1 INTRODUCTION AND LIT ERATURE REVIEW Background Item s used on tests and questionnaires in social science typically have ordered categorical response formats, and factor analysis (FA) and the item response the ory (IRT) are often used for modeling data collected by using this type of item response. In factor analysis, appropriate sample size is always an important issue in order to obtain acceptable factor recovery. Several early studies provided recommendation s about minimum sample size in factor analysis: 200 observations by Guilford (1954), 200 500 by Comrey and Lee (1973), 50 to 200 by Gorsuch (19 83 ), and 200 250 by Cattell (1978). Other researchers suggested sample size based on the ratio between sample size and number of items: 3:1 to 6:1 by Cattell (1978), at least 10:1 by Everitt (1975), and 20:1 by Hair, Anderson, Tatham, and Black (1979). However, Arrindell and Van der Ende (1985) found out the rules for sample size, whether in terms of the minimum sample size or minimum ratio of sample size to number of items, could not be generalized to different assessments when principal component analysis was used. The Fear Survey Schedule (FSS III) assessment contained 76 items and the Marks and Matthew Fear Questionnaire (FQ) had 20 items. Arrindell and Van der Ende reported that the minimum ratio of sample size to number of items was 1.3:1 for FSS III and was 3.9:1 for FQ, and th e respective minimum sizes were 100 and 78 for the two assessments. MacCallum, Widaman, Zhang, and Hong (1999) claimed that absolute rules of minimum sample size or minimum ratios were not appropriate because the factor analysis solution can be improved by increasing sample size, larger factor loading s (or communality), or stronger PAGE 13 13 overdetermination of facto rs (number of items per factor) and the necessary sample size was influenced by the magnitude of factor loadings (or communality) and the degree of ove rdetermination of the factors. Conclusions in MacCallum et al. (1999) were drawn from exploratory factor analysis (EFA) with continuous data. Similar results were also found in other simulation studies in EFA with continuous data ( d e Win ter, Dodou, & Wieringa, 2009; Guadagnoli & Velicer, 1988; Hogarty Hines, Kromrey, Ferron, & Mumford 2005; MacCallum et al., 1999; MacCallum, Widaman, Preacher, & Hong 2001 ). The variables influencing factor recovery in CFA with ordinal data have been thoroughly inve stigated. These variables include sample size, magnitude of factor loadings, overdetermination of factors, number of categories, degree of skewness of the ordinal data, number of factors, estimation methods, interfactor correlation, and their interactions ( Beauducel & Herzberg, 2006; Boulet, 1996; DiStefano, 2002; Flora & Curran, 2004; Forero & M aydeu Olivares, 2009; Foreo, Maydeu Olivares, & G allardo Pujol 2009; Gagn & Hancock, 2006; Muthn, du Toit, & Spisic, 1997; Nye and Drasgow 2011; Oranje, 2003; P arry & McArdle, 1991 ; Rhemtulla, Brosseau Liard, and Savalei 2010; Rigdon & Ferguson, 1991; Wirth & Edwards, 2007; Yang Wallentin, Jreskog, & Luo, 2010 ). However, similar research in EFA for ordinal data is very limited and only a few have been found (Sa ss, 2010). Therefore, the present study examined how factor recovery quality of EFA with ordinal data was affected by variables significantly influencing the quality of CFA with ordinal data and EFA with continuous data in the previous studies. In the pres ent study, the impact of number of factors, PAGE 14 14 number of items per factor magnitude of primary factor loading sample size estimat or and their interactions were investigated Modeling Ordinal Data in FA and IRT The FA model implies a hypothesis that the observed covariance matrix for a set of measured variables is equal to the covariance matrix implied by a hypothesized model. This relationship can be stated as ( 1 1) where represents the population covariance matrix of a set of observed variables and represents the population covariance matrix as a function of a vector of model parameters. In factor analysis with continuous observed variables the model implied cov ariance matrix is related to the model parameters described as following ( 1 2) where is the matrix of factor loadings, is a matrix of variances and covariance among latent factors, and is the variance and covariance matrix of meas urement errors. The assumption s for factor analysis specify that the measurement errors and latent factors are uncorrelated and measurement errors are uncorrelated. Therefore is a diagonal matrix. Maximum likelihood (ML) is the most popular method to estimate the model parameters with assumptions that the o bserved variables are continuous and follow a multivariate normal distribution. However, many data in educational and psychological research are ordinal or do not follow a multivariate normal distrib ution. Theoretical and empirical research shows that ML does not perform well especially when the number of observed categories for ordinal items is five or fewer. The alternative approaches for PAGE 15 15 analyzing ordinal data include least square methods using the polychoric correlation matrix. A polychoric correlation is the correlation between the two latent continuous variables, each of which underlies an observed ordinal variable. The latent continuous variables are assumed to be bivariate normal with zero means and unit variances. The three least squares methods are Weighted Least Squares ( WLS; Muthn, 1978, 1984 ), Diagonal Weighted Least Squares ( DWLS; Muthn et al. 1997 ), and Unweighted Least Squares (U LS; Muthn, 1993 ). Application of WLS, DWLS, and UL S involves a two step estimation procedure. First, polychoric correlations and their asymptotic covariance matrix are estimated. Secondly, the model parameters and are estimated by minimizing the fit function ( 1 3) where is a vector of sample polychoric correlations, is the positive definite weight matrix, and is the model implied vector of population elements in The different choices of weight matrix lead to three different least squares methods. W LS: ( 1 4) DWLS: ( 1 5) ULS: ( 1 6) The main differences between these weight matrices are that the weight matrices for DWLS and ULS are diagonal, whereas the weight matrix for WLS is the inverse of the full matrix In addition to the three least squares methods, ML applied to polychoric correlation (ML_PC) is another option in FA to analyze ordinal data. PAGE 16 16 Modeling ordinal data in IRT framework does not rely on the correlations between observed variables but generally on the observed raw data. Marginal Maximum Likelihood via the Expectation Maximization algorithm (MM L/EM; Bock & Aitkin, 1981; Bock, Gibbons, & Muraki 1988) is one of the most common methods. In the present study, the estimation methods suitable for ordinal data, such as ML_PC, WLS, DWLS, ULS, and MML/EM, are of our interest. The following literature review focuses on the studies related to these estimators. Finding from Past Studies The literature review focuses on three areas: CFA with ordinal data, EFA wi th continuous data, and EFA with ordinal data. It summarizes findings about the variables that affect factor recovery in FA. The summary is organized in terms of the analysis type (EFA vs. CFA) and data type (continuous vs. ordinal). Details about simulati on conditions in the literature I reviewed are presented in Tables 1 1 to 1 10. Variables Influence Factor Recovery in CFA with Ordinal Data The variables that influence factor recovery in CFA with ordinal data include estimation method, sample size, magn itude of factor loading s overdetermination of factors, number of categories, number of factors, skewness of the ordinal data, and interactions of estimation method with sample size, number of categories, model complexity, and magnitude of interfactor cor relations. Factor recovery is measured by multiple indices of loading estimation accuracy, such as relative bias, relative bias of standard errors, root mean square error (RMSE), the discrepancy index, and the absolute difference between loading estimates and population loadings PAGE 17 17 Effect of sample size and its interaction with estimation method Several studies found that increasing sample size usually improved factor loading estimation in WLS, ULS, DWLS, ML_PC, and MML/EM. For ML_PC, a larger sample size dec reased relative bias ( Oranje, 2003 ; Rigdon & Ferguson, 1991; Yang Wallentin et al. 2010 ), RMSE (Oranje, 2003 ; Yang Wallentin et al., 2010 ), absolute difference between loading estimates and population loadings (Oranje, 2003) and RMSE of the standard errors of loading estimates ( Yang Wallentin et al., 2010 ) For MML/EM, relative bias ( Forero & Maydeu Olivares, 2009 ) and RMSE were improved by a larger sample size ( Boulet, 1996; Forero & Maydeu Olivares, 2009). For WLS, increasing sample size improved th e relative bias ( DiStefano, 2003; Flora & Curran, 2004 ; Oranje, 2003; Rigdon & Ferguson, 1991; Yang Wallentin et al. 2010 ), the RMSE ( Yang Wallentin et al. 2010 ), the absolute difference between loading estimates and population loadings (Oranje, 2003), t he relative bias of the standard errors of loading estimates (DiStefano, 2003; Flora & Curran, 2004) and the RMSE of the standard errors of loading estimates ( Yang Wallentin et al. 2010 ) For DWLS, larger sample size reduced relative bias ( Flora & Curran 2004 ; Foreo et al., 2009; Muthn et al. 1997; Orangje, 2003; Rigdon & Ferguson, 1991; Yang Wallentin et al. 2010 ), RMSE ( Yang Wallentin et al. 2010 ), relative bias of the standard error s of loading estimates ( Flora & Curran, 2004 ; Muthn et al. 1997), RMSE of the standard errors of loading estimates ( Yang Wallentin et al. 2010 ), standard errors of loading estimates (Beauducel & Herzberg, 2006), and absolute difference between estimated loadings and population loadings (Oranje, 2003). For ULS, in creasing the sample size yielded decreased relative bias ( Forero & Maydeu Olivares, 2009 ; Foreo et al., 2009; Rigdon & Ferguson, 1991; Yang Wallentin et al. 2010 ), RMSE ( Boulet, 1996; Forero & Maydeu Olivares, 2009 ; PAGE 18 18 Yang Wallentin et al. 2010 ), RMSE of standard errors of loading estimates ( Yang Wallentin et al. 2010 ), and discrepancy index of factor loading estimates (Parry & McArdle,1991). However, sample size had a stronger effect on standard errors than on means of loading estimates in DWLS (Beauduce l & Herzberg, 2006) and had stronger effect on relative bias of standard errors than on relative bias of loading estimates ( Muthn et al. 1997). The sample size required for accurate estimation using WLS was larger than for DWLS (Wirth & Edwards, 2007), MML/EM (Wirth & Edwards, 2007), and ULS (Parry & McArdle, 1999). Increasing sample size improve d the convergence rate in MML/EM (Forero & Maydeu Olivares, 2009 ), ML_PC (Rigdon & Ferguson, 1991), WLS (Parry & M cArdle; 1991), DWLS ( Foreo et al. 2009), and ULS (Forero & Maydeu Olivares, 2009 ; Foreo et al. 2009 ). Also the improper solution rate in ML_PC, ULS, DWLS, and WLS were reduced by using larger sample sizes (Rigdon & Ferguson, 1991). The percentage of the improper factor loading estimates of MML decli ne d with increasing sample size (Boule, 1996). Effect of factor loading s Research results suggest ed that larger factor loadings yield ed better factor loading estimation in MML/EM (Forero & Maydeu Olivares, 2009), ULS ( Forero & Maydeu Olivares, 2009 ; Parr y & McArdle, 1991 ), and WLS (Parry & McArdle, 1991). Effect of overdetermination of factors and its interaction with estimation method Stronger overdetermination of factors was associated with more accurate factor loading estimation in MML/EM ( Boulet 1996; Forero & Maydeu Olivares, 2009), ULS ( Boulet, 1996; Forero & Maydeu Olivares, 2009), ML_PC (Oranje, 2003), and DWLS (Oranje, 2003). Forero and Maydeu Olivares (2009) and Foreo et al. (2009) found that PAGE 19 19 s tronger overdetermination improve d the converg ence rate in MML/EM DWLS, and ULS. Boulet (1996) indicated the percentage of the improper loading estimates in MML/EM and the actual number of the improper loading estimates in ULS w as decreased substantially by stronger overdetermination Boulet (1996) f ound that the effect of overdetermination was not different in MML/EM and ULS. Effect of number of response categories in item response and its interaction with estimation methods The effect of number of categories in the item response format on factor loa ding estimation varies among estimation methods. Increasing the number of categories improve d estimation accuracy in MML/EM and ULS (Forero & Maydeu Olivares, 2009), but ha d little effect in WLS (Flora & Curran, 2004) and DWLS (Beauducel & Herzberg, 2006). Flora and Curran (2004) indicated that overall improper solution rates in WLS were greater with two response categories than with five According to Yang Wallentin et al. (2010), nonconvergence of ML_PC, WLS, DWLS, and ULS seemed to decrease with increasi ng number of categories but this was not true for the relative bias and RMSE of loading estimates and its standard errors. Rhemtulla et al. ( 2010) found that both convergence failures and improper solutions of ULS occurred most frequently with 2 categories and declined as the number of categories increased. ULS had largely accurate estimates with 2 to 4 categories and remained accurate with 5 7 categories. Effect of degree of non normality of ordinal data Ordinal data usually are non normal but some previous studies generated ordinal data that were approximate ly normally distributed ( Beauducel & Herzberg, 2006; DiStefano, 2002; Flora & Curran, 2004; Forero & Maydeu Olivares, 2009). Three PAGE 20 20 different methods were used to manipulate the degree of non normality of ordinal data: using t hresholds on normally distributed continuous variables ( DiStefano, 2002 ; Rigdon & Ferguson, 1991); categorizing cont inuous data with zero to moderate non normality by the same set of thresholds (Flora & Curran, 2004); and controlling the percentage of endorsement on the items ( Forero & Maydeu Olivares, 2009 ; Parry & McArdle, 1991). Many researchers found that that compa red to ordinal data that were approximately normally distributed, more severely non normal ordinal data introduce d more bias in loading estimates and their standard errors in MML/EM (Forero & Maydeu Olivares 2009), ML_PC (Rigdon & Ferguson, 1991), WLS (D iStefano, 2002; Flora & Curran, 2004 ; Parry & McArdle, 1991; Rigdon & Ferguson, 1991), ULS (Forero & Maydeu Olivares, 2009; Foreo et al. 2009; Parry & McArdle, 1991; Rigdon & Ferguson, 1991), and DWLS ( Flora & Curran, 2004 ; Foreo et al. 2009; Muthn et a l. 1997 ; Rigdon & Ferguson, 1991 ). The RMSE of loading estimates in MML and ULS increased with the skewness of the latent distribution (Boulet, 1996). However, Flora and Curran (2004) indicated that standard errors in WLS and DWLS were more biased as func tion of increasing model size and decreasing sample size but were not systematically affected by the non normality of the continuous variables. The contradiction reflects the varying amount of non normality in these studies. The ordinal data in Flora and C urran (2004) had skewness and kurtosis that did not substantially deviate from the skew and kurtosis of normal data: the maximum skewness was 0.49 and maximum kurtosis was 3.39. However, the non normality was more severe in the other studies, such as skewness of 2.3 and kurtosis of 5.5 in DiStefano (2002), high skewness in Rigdon and Ferguson (1991), 10 % of endorsement in dichotomous items PAGE 21 21 in Forero and Maydeu Olivares (2009), 80% or 90% endorsement in dichotomous items in Parry and McArdle (1991). Increased non normality of ordinal data reduce d the convergence rate in MML/EM and ULS (Forero & Maydeu Oli vares, 2009). Highly skewed ordinal response data cause d both nonconvergent and improper solutions to occur commonly in ML_PC, WLS, ULS, DWLS (Rigdon & Ferguson, 1991). However, Flora and Curran (2004), who generated the ordinal variables by categorizing n on normally distributed continuous data, suggested that the variation in convergence rate in WLS was not systematically associated with the degree of nonnormality of the continuous data. This is likely due to the fact that the ordinal variables were not st rongly nonnormal in Flora and Curran. Effect of estimation method Several studies c ompared different estimation methods under different simulation conditions and some results relevant to this dissertation have been found. Forero and Maydeu Olivares ( 2009 ) and Foreo et al. (2009) investigated the performance of MML/EM vs. ULS and the performance of DWLS vs. ULS with the same simulation design. Estimation by ULS and MML/EM was comparable with ULS yielding slightly more accurate loading estimates and MML/EM slightly more accurate standard errors (Forero & Maydeu Olivares, 2009). ULS outperformed DWLS slightly in loading estimation accuracy in terms of relative bias of loading estimates and their standard errors ( Foreo et al. 2009). N either method generated a dequate loading estimates with relative bias less than 10% under conditions with small number of items per factor, binary items, low factor loading around 0.4, skewness larger than 1.5, and small sample size around 200. The estimation accuracy of DWLS was close to that of MML/EM (Wirth & Edwards, 2007). The loading estimates in DWLS were close to those from ML_PC in PAGE 22 22 Oranje (2003), when 60% of the items had a 2 category response format, 20% ha d a 3 category response format and 20% had a 5 category response f ormat. Rigdon and Ferguson (1991) compared the performa nce of WLS, DWLS, ULS, and ML_PC. For each population loading, mean square error (MSE) of the estimates from each estimation method over all sample sizes were calculated and were ranked from smallest t o largest. The sums of ranks over eight loadings were ordered from smallest to largest: WLS, DWLS, ML_PC and ULS. However, the relative performance of ML_PC was improved to match that of the WLS with increasing sample size so that with a sample size of 500 the methods were ranked as ML_PC, WLS, DWLS and ULS. When the latent distribution was normal, the loading estimates from ULS were more accurate and more stable than those from MML/EM (Boulet, 1996). Yang Wallentin et al. (2010) compared the performance of WLS, DWLS, ULS, and ML_PC. The results indicated that the estimation accuracy of ULS, DWLS, and ML_PC were better than WLS in terms of the relative bias and RMSE of the loading estimates and their standard errors. In gen eral, the performance of ULS, DWLS, and ML_PC were similar over all conditions. ULS loading estimates ha d significantly better relative bias than those from DWLS under majority cases and ha d better relative bias of loading estimates than those from ML_PC u nder some cases. Forero and Maydeu Olivares (2009) found convergence rates for MML/EM and ULS were satisfactory (98.3% for MML/EM and 96.5% for ULS). Both methods had unacceptable convergence rates (i.e., less than 80% on average) when the number of items per factor was 3, skewness was greater than 1.5, and sample size was 200. Foreo et al. (2009) found that DWLS slightly but generally outperformed ULS in convergence rates: the average convergence rates across the 324 simulation PAGE 23 23 conditions were 97.4% for DW LS and 96.4% for ULS. Flora and Curran (2004) indicated that unlike the WLS, the likelihood of an improper solution or convergence difficulty was near zero with DWLS, even with a complex model and small sample size 100. In Rigdon and Ferguson (1991), the n umber of improper solutions produced by ML_PC (18.9%) was larger than the sum of improper solutions produced by ULS, DWLS and WLS (1.6%). Flora and Curran (2004), Oranje (2003), and Parry and AcArdle (1991) agreed that large models and small sample sizes o ften led to convergence problems in WLS. According to Rhemtulla et al. (2010), ML produced more convergence problems than ULS but ULS generated a greater number of improper solutions. Maydeu Olivares (2001) suggested that when modeling categorical data the performance of DWLS and ULS was similar Effect of interfactor correlation and its interactions with estimation method Forero and Maydeu Olivares (2009) indicated that interfactor correlation had different effects on factor loading estimation in MML/EM and ULS. Accuracy of loading estimates and standard errors in ULS were unaffected by the magnitude of the interfactor correlations (0.2 and 0.6) whereas estimation accuracy was reduced in MML/EM as the magnitude of the correlations decreased. Effect of nu mber of factors Beauducel and Herzberg (2006) studied the effect of number of factors (four levels: 1, 2, 4, and 8) and concluded that it had extremely small effect on accuracy of factor loading and standard error estimation in DWLS. Effect of model size and its interaction with estimation method The model size or the complexity of the model includes the number of factors and the number of items. Previous studies indicated that performance of WLS was reduced PAGE 24 24 significantly by a large model size. Flora and Curran (2004) investigated WLS and DWLS in models with four different levels of complexity (1 factor with 5 items, 1 factor with 10 items, two correlated factors with 5 items per factor, 2 correlated factors with 10 items per factor). The model size had a strong effect on WLS but almost no effect on DWLS. The increased model size not only caused more biased factor loading estimates in WLS but also prevented converged solutions in the most complex model with a sample size of 100. Oranje (2003) also showed th at in models with more factors (1 vs. 3), stronger overdetermination (5, 10, or 15 items per factor) produced less accurate estimates or stopped WLS from yielding loading estimates because the number of parameters was too large for the sample size. Wirth a nd Edwards (2007) compared the estimated factor loadings obtained from WLS and DWLS under two models (a simple model with one factor and 10 items and a complex model with four factors and 40 items) and a sample size of 1000. Loading estimates obtained from DWLS were always close to the population value regardless of model size while estimates obtained from WLS were affected by the model size: loading estimates for the complex model showed more discrepancy from the population values than did those for the si mple model. Yang Wallentin et al. (2010) found out that the relative bias and RMSE of loading estimates from ML_PC, WLS, DWLS, and ULS increased from a model with 2 factors and 6 items to a more complex model with 4 factors and 16 items. Variables t hat In fluence Factor Recovery in EFA with Continuous Data In EFA, factor recovery concerns how closely the estimated factor loading matrix approximates the population matrix or the extent to which all items have estimated high loadings on the correct primary fac tor and low cross loadings on the secondary factors as shown in the population factor loading matrix. The accuracy indices for the loading PAGE 25 25 matrix include congruence coefficient (indicated as ), mean factor score correlation coefficient (indicated as mea n FSC), g 2 V and pattern accuracy. The congruence coefficient measures the congruence between the sample and population factor loadings for one factor. A larger indicates more accurate recovery of the population factor loadings in the sample solution. Mean FSC is the mean correlation between two sets of factor score estimates, which are obtained by using factor loading matrices from sample and population loading matrices. The g 2 index is interpreted as the average squared difference between comparable loadings of the sample and population factor loading matrix and g is the square root of g 2 The index V assesses variability of factor loa ding matrix over replications within each simulated condition. Pattern accuracy measures the extent of agreement between the sample and population factor loading matrix and it includes five indices: loading sensitivity, specificity, general pattern accurac y, total pattern accuracy, and per element accuracy. Factor loading sensitivity is the average proportion of items loading meaningfully (loading is larger than 0.3) on at least one factor both in the sample and in the population. Similarly, factor loading specificity is the average proportion of items that do not load meaningfully on any factor in both the sample solution and the population solution. G eneral pattern accuracy is the proportion of replications in which all items load on the same factor in bot h the sample and population. T otal pattern accuracy is the proportion of replications in which the factors on which all items load are in agreement with the population matrices. Per element accuracy is the proportion of items that load correctly in each sample matrix, averaged across replications. PAGE 26 26 Effects of sample size and its interaction with factor loading Increasing the sample size improve d factor recovery as measured by a variety of indices: bias and RMSE of factor loading estimates (Hogarty et al., 2005), mean FSC ( de Winter et al., 2009), g 2 (Guadagnoli & Velicer,1998), g (MacCallum et al., 2001 Velicer & Fava, 1998 ), standard deviation of g ( Velicer & Fava, 1998), V (MacCallum et al., 1999), ( de Winter et al., 2009 ; Hogarty et al., 2005 ; Mac Callum et al., 1999; MacCallum et al., 2001), and pattern accuracy (Hogarty et al., 2005). Some researchers (Guadagnoli & Velicer, 1988; Hogarty et al., 2005 ; MacCallum et al., 1999; MacCallum et al., 2001; Velicer & Fava, 1998 ) found that the effect of s ample size interacted with the size of the factor loading or the communality: sample size had a strong effect when factor loading (or communality, component saturation) was small but a weak effect when factor loading was high. MacCallum et al. (1999) and M acCallum et al. (2001) concluded that sample size had little effect when factor loadings were high (0.6 0.8) and factors were strongly determined (10 or 20 items measuring 3 factors). Similarly, d e Winter et al. ( 2009) suggested that the effect of sample s ize was negligible when factor loadings were high (0.8 0.9), number of factors small (1 4), and number of items high (12 96). These researchers found that under the preceding conditions, a relatively small sample size was required for an accurate factor re covery ( d e Winter et al., 2009 ; MacCallum et al., 1999; MacCallum et al., 2001). Effect of factor loading The magnitude of factor loading (or communality, component saturation) is an important determinant o f factor recovery. Generally speaking, larger factor loadings or communalities improve d the EFA solution quality (Guadagnoli & Velicer, 1988; Hogarty PAGE 27 27 et al., 2005 ; MacCallum et al., 1999; MacCallum et al., 2001; Velicer & Fava, 1998 ). MacCallum et al. (1999 ) and MacCallum et al. (2001) suggested that good to excellent factor recovery was achieved as long as the communality was high and that the influence of communality was stronger as factors became more poorly determined. D e Winter et al. (2009) showed that higher factor loading s were very helpful in achieving satisfactory factor recovery so that a smaller sample size would be sufficient. Effects of number of factors, number of items, and overdetermination of factors and their interaction with factor loadin g A larger number of items or fewer factors improve d factor recovery but the effects were evident only when factor loading was low ( d e Winter et al., 2009 ; Guadagnoli & Velicer, 1988; Velicer & Fava, 1998 ). According to papers in EFA with continuous data, variation of overdetermination can be achieved in three ways: varying the number of items with number of factors fixed ( Hogarty et al., 2005 ; Velicer & Fava, 1998 ); varying the number of factors with the number of items fixed ( Hogarty et al., 2005 ; MacCallum et al., 2001); varying both the number of items and number of factors (MacCallum et al., 1999). Stronger overdetermination improve d factor recovery ( Hogarty et al., 2005 ; MacCallum et al., 1999; MacCallum et al., 2001) and overdetermination ha d l ittle effect on factor recovery when communality was high but beca me important determinant when communality was low (MacCallum et al., 1999; MacCallum et al., 2001). Effect of rotation methods Rotation is used to minimize the complexity of the factor load ing matrix and yield a simple or interpretable pattern. The row or column in the loading matrix is regarded as being simple if it has few nonzero elements and complex if it has many. Sass and PAGE 28 28 Schmitt (2010) compared four rotation methods under two factor s tructures: (a) perfect simple structure with each item having a large loading on only one factor and no cross loadings in the population loading matrix and (b) approximate simple structure with each item having large loading on one factor and small cross l oadings on secondary factors. Direct Quartimin and Geomin rotation methods were found suitable for perfect simple structure because their goals were to generate a rotated loading matrix with only large primary loadings and no cross loadings. Crawford Fergu son (CF) Equamax and CF Facparsim were appropriate for approximate simple structure because their rotated loading matrix were allowed to have a complex structure with small cross loadings. Ranks of the effects Some researcher s ( Hogarty et al., 2005 ; MacC allum et al., 1999; MacCallum et al., 2001; Velicer & Fava, 1998 ) conducted repeated ANOVA to measure the size of the effects that influence factor recovery. The effect size was measured by omega square the proportion of variance accounted for in the population by each effect The common finding was that the strongest effects were factor loading (or communality), sample size, and overdetermination of factors. For most factor recovery indices, either factor loading or sample size was the strongest effec t and the ir rank depend ed on the recovery index. Variables t hat Influence Factor Recovery in EFA with Ordinal Data Sass (2010) is the only research I found that evaluated EFA with ordinal data. The simulation conditions included factor structures (two lev els: perfect simple structure and approximate simple structure), interfactor correlations (six levels: 0, 0.10, 0.20, 0.30, 0.4, and 0.8), rotation methods (two levels: varimax and promax), sample size (six levels: 100, 200, 300, 400, 500, and 1000). For p erfect simple structure, 15 items had large loadings ranging from 0.40 to 0.82 on each of the two factors and there were no PAGE 29 29 cross loadings. The approximate simple structure was obtained from perfect simple structure by adding small cross loadings ranging f rom 0.02 to 0.28 on each of the factor. NOHARM (Fraser, 1988; Fraser & McDonald, 1988), an approximation of ULS, was used to analyze the dichotomous response data. The bias and RMSE of factor loading estimates improved with increasing sample size. The per formance of rotation procedures depended on the factor structures and interfactor correlations. The performance (including bias, RMSE of loading estimates, inter factor correlation estimates) of promax was acceptable with interfactor correlation less than 0.4 and perfect simple structure data, whereas varimax only succeeded under orthogonal factors. Neither rotation method performed well with larger interfactor correlations (0.8) or approximate simple structure data. Increasing interfactor correlation had a tendency to increase the bias and RMSE of loading estimates and its effect had some variation related to sample size, rotation method, and magnitude of loading. For zero and small cross loadings, when the interfactor correlation was in range of 0.0 to 0.4 increasing the correlation increased bias and RMSE of loadings obtained from varimax rotation regardless of the sample size. With promax rotation the correlation increased bias and RMSE of loadings only with a small sample size (100) but had trivial effe ct on bias and RMSE with larger sample sizes (300 1000). When the interfactor correlation was increased from 0.4 to 0.8, bias and RMSE of zero and small cross loadings were increased dramatically regardless of rotation methods and sample size. For large lo adings, interfactor correlation had little effect on bias and RMSE when correlation was in range of 0.0 to 0.4 but increased them substantially when correlation was increase d from 0.4 to 0.8. PAGE 30 30 Summary Although the research es in the literature review come f rom CFA with ordinal data and EFA with continuous and ordinal data, there are some common findings on the variable s that influence factor recovery. All papers agreed that increasing sample size improved factor recovery. All papers except the one in EFA wit h ordinal data (Sass, 2010) recognized that larger factor loading (or communality) and stronger overdetermination of factors were advantageous to solution accuracy. Due to the differences in data (continuous vs. ordinal) and factor analysis (CFA vs. EFA), the papers in literature review showed different research emphas e s The EFA papers in the literature review studied the determinants of factor recovery quality and none compared different estimation methods. The estimation method was fixed at approximate ULS (implemented by NOHARM) for ordinal data or selected from maximum likelihood and principal component analysis for continuous data. However, the performance of estimation methods commonly used in ordinal data have been studied in researches on CFA. The accuracy of factor analysis in CFA is a local measurement which focuses on the estimation of individual factor loading whereas the accuracy in EFA is more a global assessment which emphasizes the recovery of overall factor loading matrix. Although the effe cts of sample size, magnitude of factor loading, and overdetermination of factors have been found across studies, research in EFA had the additional following conclusions. The effects of sample size and overdetermination of factors interacted with the magn itude of factor loadings: the effects were weak when factor loadings were large but were strong when factor loadings were small. The most influential variables were factor loading, sample size, and overdetermination of factors. PAGE 31 31 For most factor recovery ind ices, either factor loading (or communality) or sample size had the strongest effect in terms of omega square and their ranks depended on the recovery index. A relatively small sample size was required for accurate factor recovery as long as primary factor loadings were large and factors were strongly determined. Research Questions Compared to the studies in CFA with ordinal data, two areas of research have not been sufficiently investigated for EFA with ordinal data. First, the performance of estimation me thods appropriate for the ordinal data has been investigated thoroughly in CFA but not in EFA. Second, the variables that influence the factor recovery have been investigated in CFA with ordinal data and EFA with continuous data bu t not in EFA with ordinal data. Considering that WLS requires a much large sample size than DWLS and ULS require and that sample size, magnitude of factor loading, and overdetermination are the most important variables that influence the factor recovery, the present study has thre e main objectives : To investigate, with ordinal data, the effects of sample size, magnitude of primary factor loading s number of factors, and number of items per factor on factor recovery by DWLS and ULS in EFA. To study the influence of magnitude of primary factor loading s, number of factors, and number of items per factor on the effect of sample size on factor recovery in DWLS and ULS. To recommend sample size that is necessary to produce acceptable solution for factor analysis. PAGE 32 32 Table 1 1. Overvie w of previous studies Paper Number Author(s) Year 1 Beauducel and Herzberg 2006 2 Boulet 1996 3 DiStefano 2002 4 Flora and Curran 2004 5 Foreo, Maydeu Olivares, and Gallardo Pujol 2009 6 Forero and Maydeu Olivares 2009 7 Maydeu Olivares 2001 8 Muthn, du Toit, and Spisic 1997 9 Nye and Drasgow 2011 10 Oranje 2003 11 Parry and AcArdle 1991 12 Rhemtulla, Brosseau Liard, and Savalei 2010 13 Rigdon and Ferguson 1991 14 Wirth and Edwards 2007 15 Yang Wallentin, Jreskog, and Luo 2010 16 de Winter, Dodou and Wieringa 2009 17 Guadagnoli and Velicer 1988 18 Hogarty, Hines, Kromery, Ferron, and Mumford 2005 19 MacCallum, Widaman, Preacher, and Hong 2001 20 MacCallum, Widaman, Zhang, and Hong 1999 21 Sass 2010 22 Sass and Schmitt 2010 23 Velicer and Fava 1998 PAGE 33 33 Table 1 2 Analysis type and data type in the previous studies Paper Number CFA EFA Ordinal Data Continuous Data 1 O O 2 O O 3 O O 4 O O 5 O O 6 O O 7 O O 8 O O 9 O O 10 O O 11 O O 12 O O 13 O O 14 O O 15 O O 16 O O 17 O O 18 O O 19 O O 20 O O 21 O O 22 O O 23 O O PAGE 34 34 Table 1 3 Simulation conditions in the previous studies part 1 Paper Number Number of factors Number of items per factor Factor loading Sample size N umber of items 1 O O O 2 O O 3 O O O 4 O O 5 O O O O 6 O O O O 7 O O 8 O 9 O 10 O O O 11 O O 12 O O 13 O O 14 O O 15 O O 16 O O O 17 O O O O 18 O O O O 19 O O O 20 O O O 21 O O 22 O 23 O O O PAGE 35 35 Table 1 4 Simulation conditions in the previous studies part 2 Paper Number N umber of categories M odel misspecification N on normality Interfactor correlation R otation 1 O O 2 O 3 O 4 O O 5 O O 6 O O 7 8 O 9 O O 10 O 11 O 12 O O 13 O O 14 O 15 O O 16 17 18 19 O 20 21 O O 22 O O 23 PAGE 36 36 Table 1 5. Estimation methods in the previous studies Paper Number Estimators 1 DWLS, ML 2 MML/EM, ULS 3 ML, WLS 4 DWLS, WLS 5 DWLS, ULS 6 MML/EM, ULS 7 DWLS, ULS, WLS 8 DWLS 9 DWLS, WLS 10 DWLS, ML_PC,ULS, WLS 11 ULS 12 ML, ULS 13 DWLS, ML_PC, ULS, WLS 14 DWLS, MML/EM, WLS 15 DWLS, ML_PC, ULS, WLS 16 N/A 17 N/A 18 N/A 19 ML 20 ML 21 ULS 22 N/A 23 ML Note: The estimation methods investigated in the literature review include DWLS, ML, ML_PC, WLS, and ULS. N/A means either the estimation method is not clarified in the research or it is not of our interest. PAGE 37 37 Table 1 6. Number of factors in the previous studies Paper Number Number of factors 1 1, 2, 4, 8 2 1 3 Model 1: 2 Model 2: 3 4 1, 2 5 1, 3 6 1, 3 7 N/A 8 3 9 2 10 1, 3 11 1 12 2 13 2 14 Model 1: 1 Model 2: 4 15 Model 1: 2 Model 2: 4 16 1, 2, 3, 4, 8 17 3, 6, 9 18 3, 5, 7 19 3, 7 20 3, 7 21 2 22 2 23 6 PAGE 38 38 Table 1 7. Number of items per factor in the previous studies Paper Number Number of items per factors 1 5 2 15, 30, 45, 60 3 Model 1: {4,8} Model 2: {4, 4, 8} 4 5, 10 5 3, 7, 9, 14, 21, 42 6 3, 7, 9, 14, 21, 42 7 N/A 8 4 9 {7, 8} 10 5, 10, 15 11 8 12 5, 10 13 4 14 10 15 Model 1: 3 Model 2: 4 16 3, 4, 6, 8, 12, 16, 24, 32, 48 17 4, 6, 8, 12, 16, 18, 24 18 10:3, 20:3, 30:3, 15:5, 20:5, 30:5, 20:7, 30:7, 40:7 19 20:3, 20:7 20 10:3, 20:3, 20:7 21 15 22 15 23 3, 4, 5 Note: If the number of items is fixed in the study, their values are listed inside braces. The commas inside the braces separate the number of items for each factor. The commas in the descriptions without braces separate the different levels of manipulated number of items per factor. For example, {4, 8} means the number of items per factor is fixed and one factor has 4 items and another one has 8 items. PAGE 39 39 Table 1 8 Factor loadings in the previous studies Paper Number Factor loading 1 model 1 (orthogonal latent factors):0.5 model 2 (oblique latent factors): 0.55 2 discrimination parameter: {(0.5, 1.0, 1.5)} 3 model1: {(0.7, 0.6), (0.5, 0.4, 0.3)},{(0.5, 0.4, 0.3),(0.7, 0.6)} model 2: {(0.7, 0.6), (0.7, 0.6), (0.5, 0.4, 0.3)},{(0.5, 0.4), (0.5, 0.4), (0.7, 0.6)} 4 0.7 5 0.4, 0.6, 0.8 6 0.4, 0.6, 0.8 7 N/A 8 {(0.95),(0.85), (1.3)} 9 {(0.4 to 0.7), (0.05 to 0.2)} 10 0.8 11 0.45, 0.7, 0.9 12 {0.3, 0.4, 0.5, 0.6, 0.7} 13 {0.9, 0.8, 0.8, 0.7}, {0.7, 0.6, 0.6, 0.5} 14 Model 1: {(0.6, 0.7, 0.8)} Model 2: {0.6, 0.7, 0.8} 15 model 1: {(0.9, 0.8, 0.7), (0.6, 0.7, 0.8)} model 2: {(0.4, 0.5, 0.6, 0.7), (0.8, 0.7, 0.6, 0.6), (0.6, 0.7, 0.8, 0.9), (0.8, 0.7, 0.5,0.3)} 16 0.2, 0.4, 0.6, 0.8, 0.9 17 0.4, 0.6, 0.8 18 {0.6,0.7,0.8}, {0.2, 0.3, 0.4, 0.5, 0.6,0.7,0.8}, {0.2, 0.3, 0.4} 19 {0.6 to 0.8}, {0.2 to 0.8}, {0.2 to 0.4} 20 {0.6,0.7,0.8}, {0.2, 0.3, 0.4, 0.5, 0.6,0.7,0.8}, {0.2, 0.3, 0.4} 21 0.40 to 0.82 for primary loadings and 0 for crossloadings 0.40 to 0.82 for primary loadings and 0.02 to 0.28 for crossloadings 22 0.40 to 0.82 for primary loadings and 0 for crossloadings 0.40 to 0.82 for primary loadings and 0.02 to 0.28 for crossloadings 0.40 to 0.82 for primary loadings and 0.20 to 0.48 for c rossloadings 0.40 to 0.82 on one factor and 0.02 to 0.28 on another factor 23 0.4, 0.6, 0.8 Note: If all of the factor loadings are not equal, their values are listed inside braces. The commas between braces separate the different levels of loadings. The parenthesis inside the braces contains the loadings for one factor. If the loadings from diff erent factors have the same settings, the parentheses are omitted. The commas between parentheses separate the loadings for different factors. The commas inside parentheses separate the possible loading magnitudes for one factor. For example, {(0.7, 0.6), (0.5, 0.4, 0.3)}, {(0.5, 0.4, 0.3), (0.7, 0.6)} means the factor loadings have two levels: 0.7 or 0.6 for factor 1 and 0.5, 0.4, or 0.3 for factor 2; 0.5, 0.4, or 0.3 for factor 1 and 0.7 or 0.6 for factor 2. When all of the factor loadings are equal, thei r values are listed without any braces or parenthesis and the commas separate their different levels. For example, 0.4, 0.6, 0.8 means all items have equal loadings and they have three levels: 0.4, 0.6, and 0.8. For example, {0.6, 0.7, 0.8}, {0.2, 0.3, 0.4 0.5, 0.6, 0.7, 0.8}, {0.2, 0.3, 0.4} means the factor loadings have three levels: 0.6, 0.7, or 0.8 for all items; 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, or 0.8 for all items; 0.2, 0.3, or 0.4 for all items. PAGE 40 40 Table 1 9. Sample size in the previous studies Paper N umber Sample size 1 250, 500, 750, 1000 2 study 1: sample size is selected based on the ratio between sample size and number of items and the ratios are 16.67, 33.33, and 66.67. for 15 items case, sample sizes are 150, 500, 1000 for 30 items case, sample sizes are 500, 1000, 2000 for 45 items case, sample sizes are 750, 1500, 3000 for 60 items case, sample sizes are 100, 2000, 4000 study 2: sample sizes are 250, 500, 1000, 10000 3 350, 700 4 100, 200, 500, 1000 5 200, 500, 2000 6 200, 500, 2000 7 100, 300 8 200, 400, 800, 1600 9 400, 800, 1600 10 200, 500, 1000, 2000 11 50, 100, 200 12 100, 150, 350, 600 13 100, 300, 500 14 300, 1000 15 100, 200, 400, 800, 1600 16 N/A 17 50, 100, 150, 200, 300, 500, 1000 18 3 p 5 p 10 p 20 p 40 p p is the number of items and equals to 10, 20, and 30 for 3 factors case, 15, 20, and 30 for 5 factors case, and 20, 30, and 40 for 7 factors case. 19 60, 100, 200, 400 20 60, 100, 200, 400 21 100, 300, 500, 1000 22 300 23 50, 100, 150, 200, 400, 800 Note: N/A means sample size is not a studied effect in this research and the minimum sample size is calculated. PAGE 41 41 Table 1 10. Number of replications in the previous studies Paper Number Number of replications 1 500 2 100 3 100 4 500 5 1000 6 1000 7 1000 8 500 9 200 10 500 11 30 12 1000 13 300 14 0 15 2000 16 5000 17 5 18 10000 19 100 20 100 21 1000 22 1000 23 10 PAGE 42 42 CHAPTER 2 DESIGN OF MONTE CARLO SIMULATION STUDY A Monte Carlo study was implemented to investigate the effects of number of factors, number of items per factor magnitude of primary factor loading s and sample size on the factor recovery quality in DWLS and ULS when ordinal data were modeled by EFA. Da ta Generation The data were generated using SAS/IML 9.3 and included two steps: producing latent continuous variables and then categorizing the variables into ordinal observed variables. With the assumption that there is a latent variable underlying an ob served ordinal variable, the latent variable was generated according to its relationship with common factors described by the common factor model ( 2 1 ) where is the latent variable corresponding to the observed variable = 1,2 is the total number of observed variables, is a vector representing the common factors, is the total number of common factors, is a vector of factor loading s and is a measurement error. Both common factors an d measurement errors are assumed to be normally distributed so the underlying continuous variables are also normally distributed. Common factors and measurement errors are assumed to be uncorrelated. The observed ordinal variable was obtained by categoriz ing the corresponding latent variable w ith a set of thresholds that made the distribution of the ordinal variable approximate ly normal ( Beauducel &Herzberg 2006) : PAGE 43 43 ( 2 2 ) where{ v 1 c } are the set of the thresholds and c is the total number of categories in the observed variables. The thresholds were set at 1.53 ; 0.49 ; 0.49 ; and 1.53 so that the scores on the ordinal items were approximately normally distributed ( Beauducel & Herzberg 2006). The data were generated by using the following steps. 1. Scores were generated on uncorrelated variables that followed standard normal distributions. Let denote the vector of these scores. 2. Scores were generated on correlated factors by using (2 3) where is a Cholesky factor of a correlation matrix in which all correlation coefficients are 0.3. 3. The vector of factor loadings, for observed variable was generated. The primary factor loading was randomly selected from a uniform distribution with range 0.6 to 0.8 in the large primary factor loading condition and 0.4 to 0.6 in the medium primary loading condition. The secondary factor loadings were randomly selected from a uniform distribution with range 0.3 to 0.3. If the communality ( was equal to or larger than 0.9 the secondary factor loadings were selected again. This process was repeated until the communality was less than 0.90. The repeated sample of secondary factor loadings was only necessary when the number of factors was 4 of 6. The factor loading vectors were stacked to form the factor loading matrix 4. Scores were generated on uncorrelated variables that followed standard normal distributions. Let denote the vector of these scores. 5. Score on an item specific latent variable w as generated by using (2 4) w here is the diagonal element in matrix of and is the element in vector PAGE 44 44 6. O bserved variable s were obtained by categorizing using the thresholds: 1.53, 0.49, 0.49, and 1.53. Estimation Methods As mentioned earlier, five estimation methods were investigated in the literature on factor analysis of ordinal data. However, the appropriate methods for modeling ordinal data are WLS, DWLS, ULS, and MML/EM. The MML/EM demands heavy computation because of the quadrature based integration over all of the dimensions of the latent factors ( Wirth & Edwards, 2007 ). Among the three least squares methods, WLS is criticized because the weight matrix is often nonpositive definite and a large sample size is required to produce stable estimates of the s ampling c ovariance matrix for the polychoric correlations (Flora & Curran, 2004 ; Nye & Drasgow, 2011 ). DWLS and ULS require smaller sample size than WLS for accurate estimation ( Parry & McArdle, 1999 ; Wirth & Edwards, 2007 ). Therefore, DWLS and ULS were selected as the estimation methods in the present study. Number of Factor s The numbers of factors in this simulation study were selected based on the literature review in Chapter 1. As seen in Table 1 6, 2 factors and 4 factors were quite common in the previous stud ies. In order to investigate the effect of number of factors, 6 factors were selected as well. Therefore, the number of factors in this study had three levels: 2, 4, and 6. Number of Items p er Factor From Table 1 7, the number s of items per factor in the previous studies cover ed a wide ra nge (3 to 48). However, DiStefano (2002) surveyed issues of Educational and Psychological Measurement and Psychological Bulletin between 1992 and 1999 and PAGE 45 45 found out that two factor models with 12 to 16 items were common model in psychological research. A review of psychological instruments suggest ed that the number of items per factor was frequently in the range 6 to 17. These instruments were Counseling Center Assessment of Psychological Symptoms 62 (CCAPS 62; Loc ke, Buzolitz, Lei, Boswell, McAleavey, et al. 2011), Mindful Attention Awareness Scale (Brown & Ryan, 2003), A Big Five Adolescent Personality Scale (APSI; Lounsbury, Tatum, Gibson, Park, Sundstrom, et al., 2003), Multiculturally Sensitive Mental Health S cale (MSMHS; Chao & Green, 2011), t he revised Self report P sychopathy S cale (SRP III; Mahmut, Menictas, Stevenson, & Homewood 2011) Severity Indices of Personality Problems (SIPP; Verheul, Andrea, Berghout, Dolan, Busschbach, et al., havior Questionnaire (CBQ; Rothbart, Ahadi, Hershey, & Fisher 2001), The Fear of Hypoglycemia (FH 15; Anarte Ortiz, Caballero, Ruiz de Adana, Rondn, Carreira, et al., 2011) scale Hypomanic Personality Scale (HPS; Schalet Durbin, & Revelle 2011), and Elemental Psychopathy Assessment Scale (EPA; Lynam, Gaughan, Miller, Miller, Mullins Sweatt, et al., 2011). Therefore, two levels of the number of items per factor were selected : 7 and 14. Factor Loading In factor analysis with two or more fact ors, the ideal result is perfect simple structure: each item measures only one factor. That is, each item has a high loading only on one factor (i.e., the primary factor) and zero loadings on the other factors (i.e., the secondary factors). This ideal is o ften unrealistic in practice, as items frequently assess latent traits other than the intended trait (Stout Habing, Douglas, Kim, Roussos, & Zhang 1996). Quite often, an item has a medium or high loading on the primary factor and small loadings on the se condary factors. According to Sass and Schmitt PAGE 46 46 (2010) and Sass (2010), approximate simple structure occur s when all loadings on secondary factors (i.e., the cross loadings) ha ve absolute values less than 0.3. In the present study, approximate simple struct ure was of interest because it is very typical in the practice. In reality, the magnitudes of factor loadings on the primary factor always vary across the items. Therefore, in this study, the factor loadings on the primary factor were randomly selected fro m predetermined range s as were the loadings on the secondary factors. Two types of factor loading matrices were studied. In the approximate simple structure matrix with high loadings, each primary factor loading was randomly selected from a uniform distrib ution that range d from 0.6 to 0.8 and each cross loading was selected from a uniform distribution that range d from .30 from .30. In the approximate structure with medium loadings on primary factors, each primary factor loading was randomly selected from a uniform distribution that range d from 0.4 to 0.6 and each cross loading was selected from a uniform distribution that ranges from .30 to .30. Sample Size Muthn et al. (1997) evaluated DWLS in a model with 3 factors and 12 items and they found the estimation accuracy was acceptable even for a sample size of 200. In addition, previous EFA review studies (Fabrigar Wegener, MacCallum, & Strahan, 1999; Ford MacCallum, & Tait 1986; Russell, 2002) indicated that sample size of roughl y 300 was the most common sample size in published research using EFA. The model in the present study is larger than the one in Muthn et al. (1997). Therefore, 200 was selected as the smallest sample size and 400, 600, and 800 were the other levels of sam ple size. PAGE 47 47 Selecting the Number of Factors and Rotation Once factors have been extracted from a correlation matrix in EFA, one has to decide how many factors to retain as being meaningful or important. In order to concentrate on the effects of our interest s, number of factors, number of items per factor, magnitude of primary factor loadings, and sample size, the number of factors retained in EFA was predetermined and equal ed to its population value. For any given solution in EFA with two or more factors, t here are an infinite number of alternative orientations of the factors in multidimensional space that will explain the data equally well. Therefore EFA models with more than one factor do not have a unique solution and single solution needs to be selected from the infinite number of equally fitting solutions. Thurstone (1947) suggested rotating the factors in multidimensional space to achieve the solution with the best possible simple structure. In this study, CF Equamax rotation method was used because it was recommended for factor loading matrices with approximate simple structure by Sass and Schmitt (2010) The rotation criterion of CF Equamax is to minimize the complexity function described below ( 2 5 ) where is a factor loading matrix with element in the row and column, is the number of common factors, and is the number of items. Interfactor Correlation The correlations between common factor s (or latent traits ) w ere set to 0.3, which was the value commonly used in the previous studies in Table 1 4. A single value was selected in order to have a manageable simulation. However, it should be noted that PAGE 48 48 Sass (2010) provided evidence that the correlation affects the accuracy of estimated of factor loadings in EFA with ordinal data Number of Categories i n I tem Response All observed variables in this study ha d res ponses with five categories, such as neither agre e or disagree As note d earlier the observed ordinal variables were approximately normally distributed Number of Replication s In Table 1 10, 1000 replications per condition were very common in the previous studies and it wa s also selected as the number of replications in this study. Factor Recovery Indices In this study, the overall congruence between sample and population factor loading matrix was of interest. Therefor e, various indices related to factor recovery were calculated. The quality of the sample solution was evaluated by three types of indices: pattern accuracy, the congruence (indicated as ), and root mean squared deviation (indicated as g ). Pattern accuracy was measured by following indices (Hogarty et al. 2005): General pattern accuracy is the proportion of replications in which all observed variables have the same load population loading matrices on at least one factor Total pattern accuracy is the proportion of replications in which all observed population loading matrices on all factors. Per element accuracy is the proportion of the elements in the sample factor sample and population loading matrices. The proportion is averaged across replications. PAGE 49 49 An example is used to explain the pattern accuracy. Suppose is a population factor loading matrix and is a sample factor loading matrix, a nd T his sample matrix could be counted as a solution that meets the criterion for general pattern accuracy because all of the sample and population solution although some variables (1 and 6) have loading T his sample matrix could not be c ounted as a solution that meets the criterion for total pattern accuracy because the sample loading for the 6 th variable on the first factor and the 1 st variable on the second factor are not in agreement with the corresponding population loadings P er elem ent accuracy is the proportion of the 12 elements in the matrix that load the same way in the population and sample solutions. In this example, per element accuracy is 10/12=0.8333 because 1 0 out of the 12 elements loaded on the same way 0.30) in the sample and population matrix The coefficient of congruence, was calculated for each factor to assess the similarity between the sample and population factor loading matrices. According to Abdi PAGE 50 50 (2010), the coefficient of congruence was first suggested by Burt (1948) and named by Tucker (1951). This measure of congruence is given by ( 2 6 ) where is the true population factor loading for observed variable on factor and is the estimated fa ctor loading for observed variable on factor To obtain the degree of congruence across all factors in a given solution, the average value of is computed as below: ( 2 7 ) Higher value of indicates more accurate recovery of the population factors in the sample solution. Guidelines suggested for interpreting the value of is (MacCallum et al., 1991): 0.98 to 1.00 = excellent, 0.92 to 0.98 = good, 0.82 to 0.92 = borderline, 0.68 to 0.82 = poor, and below 0.68 = terrible. I t is known that the order of the factors in a sample solution may not be the same as the order of factors in the population solution To address this issue was calculated for every possible permutation of the sample fa ctor loading matrix and the permutation that resulted in the maximum value of was selected. According to Ferron (2012), this procedure was used by Hograty et al (2005). In the remaining part of the dissertation, the congruence coefficient always refers to index To further assess the degree of factor recovery, root me an squared deviation (MacCallum et al., 2001) was computed as an index of the discrepancy between the sample and population factor loading matrices: PAGE 51 51 ( 2 8 ) In E quation 2 6 is the population factor loading matrix, is the estimated sample factor loading matrix, is the number of observed variables, and is the number of factors. The lower value of g indicated the better the sample solution. PAGE 52 52 CHAPTE R 3 RESULTS A simulation study was conducted to examine the effects of five factors on the sample solution quality of EFA models with ordinal data. These factors were: The number of factors (NF), with three levels: 2, 4, and 6. The number of items per fac tor (NI), with two levels: 7 and 14. The magnitude of the primary factor loadings (PFL), with two levels: high (randomly selected in the range of 0.6 to 0.8) and medium (randomly selected in the range of 0.4 to 0.6). The sample size (SS), with four levels: 200, 400, 600, and 800. The estimators (E), with two levels: DWLS and ULS. The factors NF, NI, PFL, and SS were between subject factors and E was a within subjects factor. Therefore, 96 total conditions or 48 conditions per estimator were investigated and 1000 replications for each condition were obtained. The results are presented in two sections. First, the convergence rate and improper solutions rates for all simulation conditions are reported. Second, the effects of the five factors and their interactions on the studied statistics (congruence coefficient, root mean squared deviation, general pattern accuracy, total pattern accuracy, and per elem ent accuracy) are reported. Convergence Rates and Improper Solution Rates The convergence rate is the proportion of replications per simulation condition that converge within the specified number of iterations. The default number of iterations in M plus (i. e., 1000) was used as the maximum number of iteration s Among the 96 conditions investigated, convergence failures were not found. Thus, the convergence rate was 100% for each condition and was not affected by the number of factors, PAGE 53 53 number of items per fac tors, magnitude of primary loadings, sample size, and estimation methods. A solution is defined as improper if it contains estimated parameter values that are statistically unacceptable. In this study, the only improper estimate that occurred was a negat ive residual variance. Therefore, the improper solution rate is the proportion of replications per simulation condition which converge but ha d negative residual variances. Among the 96 conditions, improper solutions were only found in conditions with 6 fac tors, 14 items per factor, and medium primary factor loadings: one for DWLS at sample size of 400, one for ULS at sample size of 200, and one for ULS at sample size of 800. Each of these three conditions had an improper solution rate of 0.1% because only 1 out of 1000 replications had negative residual variances in the solution. Repeated Measures ANOVA The congruence coefficient, root mean standard deviation, general pattern accuracy, total pattern accuracy, and per element accuracy obtained in each conditi on were presented in Table s 3 1, 3 6, 3 11, 3 15, and 3 18, respectively. In order to investigate the influence of the simulation design factors and their interactions on the statistics of interest, a series of repeated measures five way ANOVAs were conduc ted. In each ANOVA, the dependent variable was one of the factor recover y indices and the independent variables were the five simulation design factors. In addition to the main effects, the interactions of these factors were examined. With 96 conditions an d 1000 replications per condition, the total number of observations in each ANOVA was 96000. The large sample size precluded the use of the significance levels to determine meaningful effects because with a large sample size small effects are likely to be detected as statistically significant. An effect size measure is a solution PAGE 54 54 to such a problem. According to Olejnik and Algina (2003), an effect size measure is a standardized index, estimates a parameter that is independent of sample size, and quantifies the magnitude of the difference between conditions or the relationship between exploratory and response variables. There are two categories of effect size measures: the standardized mean difference and the measures of association or the proportions of vari ance explained. In the present study, partial omega square was used as the effect size In the context of a simulation study partial omega square estimates the proportion of the sum of variance due to an effect and the variance due to replicati ons that is accounted for by the effect itself. The general formula for estimating partial omega square is (Olejnik & Algina, 2003) ( 3 1 ) where is the sum of squares for the effect for which the effect size is being estimated, is its degrees of freedom, is the corresponding mean square error, and is the total number of observations. Whe n applied to the simulation design with four between subjects factors and one within subjects factor, is defined as ( 3 2 ) The ANOVA results for congruence coefficient, root mean squared deviation, general pattern accuracy, total pattern accuracy, and per element accuracy are presented in Table s 3 2, 3 7, 3 12, 3 16, and 3 19, respectively. In eac h table, the sum of squares, degree of freedom, mean squared error, F statistics, p value, and partial omega square of all effects are listed. The effects to be interpreted are those that are significant and ha ve a partial omega square that is equal to or larger than 0.05. To PAGE 55 55 interpret the effect of a main factor on a studied outcome, the marginal means of the statistics were calculated by collapsing over all the other factors Similarly, interactions were interpreted by cal c ul a ting collaps ed over the facto rs that were not involved in the interaction I f both a main effect of a factor and interaction of that factor with one or more of the other factors were significant and has met the rule, the effect of the interaction on the studied outcome was interpreted The collapsed means for the congruence coefficient are presented in Tables 3 3 to 3 5, for root mean squared deviation in Tables 3 8 to 3 10, for general pattern accuracy in Tables 3 13 to 3 14, for total pattern accuracy in Table 3 17, and for per eleme nt accuracy in Tables 3 20 to 3 21. It should be noted that readers interested in effects for which partial omega square was smaller than .05 can use results in Tables 3 1, 3 6, 3 11, 3 15, and 3 18 to construct appropriate tables of collapsed means. Cong ruence Coefficient The congruence coefficient assesses the similarity between the sample and population factor loading matrices. Higher values of indicate more accurate recovery of the population factors in the sample solution. The criterion from Mac Callum et al. (1999) is used to define the quality of the congruence. The value is considered excellent if it is in the range of 0.98 to 1.00, good if in the range of 0.92 to 0.98 borderline if in the range of 0.82 to 0.92 poor if in the range of 0.68 to 0.82 and terrible when it is smaller than 0.68. Table 3 1 presents mean under all conditions. Across all conditions, for DWLS and ULS were almost identical with a maximum difference of 0.001 in some conditions. Because the congruence coefficient s from the two estimation methods showed almost no difference, the following description is based on DWLS and PAGE 56 56 can be generalized to ULS. In 38 out of 48 conditions, the congruence coefficients were considered good or excellent with magnitude larger than 0. 92. These conditions included all conditions with 2 factors, all conditions with 4 factors except those with 7 items per factor, medium loadings, and sample sizes of 200 or 400, and all conditions with 6 factors and high primary loadings. The congruence co efficients from 8 out of 48 conditions were in the range of 0.8 35 to 0.9 1 1 and belong to the borderline category. All of them were in conditions with medium loadings: 4 factors with 7 items per factor and sample sizes of 200 or 400; 6 factors with 7 items per factor and sample sizes of 600 or 800; and 6 factors with 14 items per factor and all sample sizes. Poor congruence coefficients in the range of 0.732 and 0.809 were generated in 2 conditions, which ha d 6 factors, 7 items per factor, medium loadings, and sample size of 200 or 400. No terrible congruence coefficients were obtained in this study. Excellent congruence coefficients occurred in 16 out of 48 conditions, all of which involved 2 or 4 factors. Fo r 2 factors excellent congruence coefficients were found in all conditions with high primary factor loading s regardless of the number of items per factor or the sample sizes, in one condition with medium primary loadings and sample size of 800 and 7 items per factor, and in conditions with medium primary loadings and sample size of 400 to 800 and 14 items per factor. For 4 factor s excellent congruence occurred only in conditions with high primary loading s : those with a sample size of 800 and 7 items per fa ctor and sample sizes of 400 to 800 and 14 items per factor. The ANOVA result s for congruence coefficient are reported in Table 3 2. All of the between subject factors were significant with p value smaller than 0.0001. Almost all of the within subject effe cts were non significant except the E x NF x NI x SS and the E x PAGE 57 57 NF x NI x PFL x SS interactions with the p values of 0.02. The significant effects with partial omega squares equal to or larger than 0.05 were the NF, NI, PFL, and SS main effects and the NF x PFL, NI x PFL, and PFL x SS interactions but only the two way interactions are interpreted. The partial omega square was 0.30 for NF x PFL, 0.05 for NI x PFL, and 0.05 for PFL x SS. M ean values of congruence coefficient s by number of factors and magnitude of the primary factor loading are reported in Table 3 3. The results indicated that congruence increased when the magnitude of the primary factor loadings increased and effect of the magnitude of the primary factor loadings increased as the number of factors increased. Congruence decreased as the number of factors increased and the magnitude of the decline was larger when the primary factor loadings were smaller. Based on T able 3 3 mean values of c ongruence coefficients were good or excellent except when with 6 facto rs and medium primary loadings. Mean values of c ongruence coefficients with 2 or 4 factors and with the high primary loadings were considered excellent while those from 2 or 4 factors with the medium loadings were classified as good. M ean values of congruence coefficients from 6 factors with high primary loadings were still good but those with medium primary loadings were in the borderline category. M ean values of congruence coefficient s by number of items per fac tor and magnitude of the primary factor loading are presented in Table 3 4. C ongruence increased when the magnitude of primary loadings increased and the effect of the magnitude of primary loadings w as smaller when the number of items per factor was larger Congruence increased as number of items per factor was larger and the amount of the increase was smaller when the primary factor loadings were larger. According to PAGE 58 58 T able 3 4, m ean values of congruence coefficients for 7 items per factor were classified as borderline and good for the medium and high primary loadings, respectively. In addition m ean values of congruence coefficients for both medium and high primary loadings cases were larger in conditions with 14 items pe r factor and were classified as good and excellent. M ean values of congruence coefficient s by the magnitude of the primary factor loading and sample size are reported in Table 3 5. C ongruence increased when the sample size was larger and the effect of sam ple size was smaller when the primary loadings were larger. Congruence increased as the magnitude of primary loadings increased and the amount of the increase was smaller when the sample size w as larger. In T able 3 5 m ean values of congruence coefficient s for the medium primary loadings were in the borderline or good categories and a ll mean values of congruence coefficients were in the good or excellent category when the primary loadings were high. R oot M ean S quared D eviation g The root mean squared deviation represents the average difference between estimated and population factor loading in a sample solution. Therefore, smaller values of g indicate more accurate recovery of the population factors in the sample solution. Mean values of r oot mean squa red deviations for all conditions are reported in Table 3 6. Because no published criterion was found in previous studies, a guideline for g is proposed in the present study. Considering a population loading with a magnitude of 0.6, estimate s in the range of 0.55 to 0.65 would be considered as very accurate, 0.5 to 0.7 as accurate, 0.45 to 0. 75 as borderline, 0.4 to 0.8 as bad, and beyond the range of 0.4 to 0.8 as terrible. Accordingly, the proposed criterion suggests a g less than 0.05 is PAGE 59 59 classified as ex cellent, 0.05 0.1 as good, 0.1 0.15 as borderline, 0.15 0.20 as poor, larger than 0.20 as terrible. Across all conditions, g for DWLS and ULS were almost identical with a maximum difference of 0.001. Because g showed almost no difference between ULS and DW L S the following description is based on DWLS and can be generalized to ULS. Excellent root mean squared deviation was observed in only 2 conditions, which had 2 factors, 14 items per factor, high primary loadings, and sample sizes of 600 or 800. Good dev iations were found in 32 conditions. For 2 factors, all conditions except the condition with medium loadings, 7 items per factor, and a sample size of 200 had mean root mean squared deviations in the good category For 4 factors, all conditions with 7 item s per factor and high loadings, and all conditions with 14 items per factor except the one with medium loadings and sample size 200 had mean root mean squared deviations in the good range. For 6 factors, mean root mean squared deviation in the good range o ccurred only with high primary factor loadings regardless of number of items per factor. Borderline mean root mean squared deviations were found in 12 conditions with medium primary factor loadings : 2 factors, 7 items per factor, and a sample size of 200 ; 4 factors, 7 items per factor, and all four sample sizes; 4 factors, 14 items per factor, and a sample size of 200; 6 factors, 7 items per factor, and a sample size of 600 or 800 ; 6 factors, 14 items per factor, and all four sample sizes Similar to the results for congruence coefficients, two poor root mean squared deviations were generated in the conditions with 6 factors, 7 items per factor, medium loadings, and sample size of 200 or 400. The ANOVA result s for root mean squared deviation are presented in Table 3 7. All of the between subject factors were significant with p value smaller than 0.0001. PAGE 60 60 Almost all of the within subject factors were not significant except the E PFL E SS, and E PFL SS interactions with the p value s in the 0.00 to 0. 01 range The effects that were significant and had partial omega square equal to or larger than 0.05 were NF, NI, PFL, SS, and the NF PFL interaction Due to the interactions, the effects that were interpreted are NI with partial omega square equal to 0 .13, SS with partial omega square equal to 0.18, and NF PFL with partial omega square equal to 0.14. Mean values of the root mean squared deviation by the number of factors and magnitude of the primary factor loading are reported in Table 3 8. T he root mean squared deviation dropped when the magnitude of the primary factor loadings increased and the amount of the decline increased as the number of factors increased. Root mean squared deviation increased as the number of factors increased and the magnitud e of the increase was larger when the primary factor loadings were smaller. According to T able 3 8 m ean values of root mean squared deviation were classified as good when there were 2 factors or when there were 4 or 6 factors and high pr imary loadings an d me an values were classified as borderline when there were 4 or 6 factors and medium primary factor loadings. Mean values of the root mean squared deviation by levels of number of items per factor are reported in Table 3 9. Root mean squared deviation de creased as the number of items per factor increased As indicated by T able 3 9 th e mean root mean squared deviations were in the good category for both levels of numbers of items per factor Mean values of the root mean squared deviation by level s of sam ple size s are reported in Table 3 10. R oot mean squared deviation declines as the sample size was PAGE 61 61 increased and the magnitude of the decline was smaller as the sample size became larger. Except when sample size was 200 the m ean root mean squared deviations in T able 3 10 w as in the good range When the sample size was 200, mean root mean squared deviation in the table was in the borderline range General Pattern Accuracy General pattern accuracy evaluates the accuracy of the sample factor loadings in reference to the 0.3 rule of thumb and it is the proportion of replications in which all variables load on the same factor in the sample as they do in the population ignoring all other loadings for the items. Table 3 11 contains values of general pattern accuracy for each condition. According to Hogarty et al. (2005) g eneral pattern accuracy at least 0 .8 was considered as borderline, and larger than 0.92 was consider as good. Therefore, these cutoff values were used in the present study. Because the pattern accuracy from two estimation methods showed small difference, the following description is based on DWLS and can be applied to ULS. Good values of general pattern accuracy w ere found in all conditions with high loadings and were in the range of 0.993 to 1.0. Most were perfect. In the medium primary loadings conditions, value s in the good range w ere found with 2 factor s, 7 items per factor and a sample size 800 and with 14 items per factor and a sample size 600 or 800. For the rest of conditions with medium loadings, general pattern accuracy was in the 0. 402 to 0.903 range for 2 factors, the 0.0 53 to 0.684 range for 4 factors, and the 0.000 to 0.0 73 range for 6 factors. The ANOVA result s for general pattern accuracy are presented in Table 3 12. Similarly to the results reported earlier all of the between subject factors were significant with p value s smaller than 0.0001 and the majority within sub ject factors were not significant. Five significant effects had partial omega square equal to or larger PAGE 62 62 than 0.05 but only the NF PFL interaction with a partial omega squared of 0.32 and the PFL SS interaction with a partial omega squared of 0.07 are interpreted because the other three important effects were NF, PFL, and SS. Mean general pattern accuracy by the number of factors and primary factor loading is reported in Table 3 13. G eneral pattern accuracy increased when the magnitude of the primary f actor loadings increased and the amount of the growth increased as the number of factors increased. General pattern accuracy decreased as the number of factors increased and the magnitude of the decline was larger when the primary factor loadings were smal ler In Table 3 13 t he g eneral pattern accuracy for high loadings conditions w as perfect when there were 2 or 4 factors and were very close to perfect when there were 6 factors. However, general pattern accuracy in the conditions with medium primary loadings was almost un acceptable (0.037 and 0.338) except when there were 2 factors (0.824). Mean values for general pattern accuracy by the primary factor loadings and sample size are reported in Table 3 14. G eneral pattern accuracy increased as the samp le size increased and the effect of sample size declined markedly when the magnitude of primary loadings was larger. Based on the results in table 3 14, w hen the primary factor loadings were large, general pattern accuracy was almost unaffected by sample s ize because the index was close to perfect at sample size 200 and had little room to increase with increasing sample size. General pattern accuracy increased as the magnitude of primary loadings increased and the amount of the increase was smaller when the sample size s were larger. When the primary factor loadings were PAGE 63 63 large general pattern accuracy was almost perfect but when the primary factor loadings were medium it was unacceptable (0.196 to 0.541). Total Pattern Accuracy Total pattern accuracy is anot her index to measure the accuracy of the sample factor loadings in reference to the 0.3 rule of thumb and it is the proportion of samples that agree with the population factor solution on all items and all factors. A larger value of total pattern accuracy represents a better sample solution. Table 3 15 contains values for total pattern accuracy under all the simulated conditions. In Hogarty et al. (2005 ) total pattern accuracy was in the range of 0.0 to 0.60 and they were considered as low. In this study, t he range of the t otal pattern accuracy was 0.019 to 0.470 for 2 factors, and 0.0 to 0.01 8 for 4 factors For 6 factors all values of total pattern accuracy was zero Accordingly, all values of total pattern accuracy were classified as low. Because the tota l pattern accuracy from two estimation methods showed a small difference, the following description is based on DWLS and can be generalized to ULS. The ANOVA result s for the total pattern accuracy are reported in Table 3 16. Almost all of the between subject factors were significant with p value smaller than 0.0001 and the majority of within subject effects were not significant. Two significant effects had partial omega square equal to or larger than 0.05 but o nly the of NF NI interactions with effect size of 0.05 is interpreted. Mean total pattern accuracy by the number of factors and number of items per factor is summarized in Table 3 17. The results showed that total pattern accuracy decreased when the num ber of factors increased and the amount of the decline decreased as the number of items per factor was larger. Total pattern accuracy decreased as the number of items per factor was increased and the magnitude of the PAGE 64 64 decrease was smaller when the number of factors was larger. When the number of items per factor, or the number of factors, or both, increased, it was more difficult for every loading in a sample solution to have the same pattern (its absolute value is either larger than 0.3 or smaller than 0.3 ) as in the population solution. Therefore the total pattern accuracy decreased dramatically to zero when the re are more factor loadings to estimate : 4 factors and 14 items per factor; 6 factors and 7 items per factor; and 6 factors and 14 items per factor This was shown in either Table 3 1 5 which contained cell means, or in Table 3 1 7 which contained collapsed means Per Element Accuracy Per element accuracy is t he proportion of elements in the loading matrices having the same pattern of the loadings in the sample solution and the population solution. Larger value s of per element accuracy indicate a better sample solution. Per element accuracy is a less conservative measure of factor recovery than general or total pattern accura cy is. Mean values of per element accuracy are reported in Table 3 18 for all simulated conditions. Criteria to interpret per element accuracy were not presented in Hogarty et al. (2005). Because per element accuracy is less conservative than general patt ern accuracy a cutoff which is more stringent than that for the general pattern accuracy is proposed in this study: larger than 0.95 is considered as good and larger than 0.85 as borderline. Because the per element accuracy from two estimation methods sho wed almost no difference, the following description is based on DWLS and can be applied to ULS. All mean values of the per element accuracy were larger than 0.95 when there were 2 factors and the sample size was at least 400. When the sample size was 200, PAGE 65 65 m e an values in the good range were obtained when the primary factor loadings were large. For 4 factors and 6 factors cases, mean per element accuracy was good in the high primary loadings conditions and between borderline and good in the medium primary lo adings conditions. The ANOVA result s for per element accuracy are presented in Table 3 19. All of the between subject factors were significant with p value s smaller than 0.0001 and the majority within subject factors were not significant. Four significant effects had partial omega square s equal to or larger than 0.05 but only NF PFL with effect size of 0.12 and SS with effect size of 0.11 are interpreted Mean p er element accuracy by number of factors and primary factor loading is reported in Table 3 20. P er element accuracy increased as the magnitude of the primary factor loadings increased and the effect increased as the number of factors increased. Per element accuracy decreased as the number of factors increased and the magnitude of the decline was lar ger when the primary factor loadings were smaller Mean per element accuracy was borderline with medium primary loadings and 4 factors or 6 factors and good with high primary factor loadings According to T able 3 20 m ean per element accuracy was in the go od category when there were 2 factors conditions. Mean per element accuracy by sample size is reported in Table 3 21. P er element accuracy was increased as the sample size increased As shown in T able 3 21 m ean per element accuracy for sample size 200 and 400 was in the range of 0.922 to 0.943 and conside red between borderline and good and m ean per element accuracy for sample size s 600 and 800 was 0.951 and considered as good. PAGE 66 66 Table 3 1. C ongruence c oefficient NF I N PFL SS DWLS ULS 2 7 Medium 200 0.958 0.958 400 0.978 0.978 600 0.978 0.978 800 0.984 0.984 High 200 0.991 0.991 400 0.993 0.993 600 0.994 0.994 800 0.994 0.994 14 Medium 200 0.970 0.970 400 0.983 0.983 600 0.988 0.988 800 0.989 0.989 High 200 0.991 0.991 400 0.995 0.995 600 0.995 0.995 800 0.996 0.996 4 7 Medium 200 0.850 0.850 400 0.911 0.911 600 0.939 0.939 800 0.924 0.923 High 200 0.972 0.972 400 0.978 0.978 600 0.979 0.979 800 0.982 0.982 14 Medium 200 0.928 0.928 400 0.948 0.948 600 0.965 0.965 800 0.955 0.954 High 200 0.977 0.977 400 0.984 0.984 600 0.986 0.986 800 0.986 0.986 6 7 Medium 200 0.732 0.732 400 0.809 0.810 600 0.856 0.856 800 0.835 0.835 High 200 0.956 0.956 400 0.965 0.965 600 0.967 0.967 800 0.970 0.970 14 Medium 200 0.843 0.845 400 0.873 0.872 600 0.876 0.876 800 0.889 0.889 High 200 0.966 0.966 400 0.975 0.974 600 0.977 0.977 800 0.978 0.978 PAGE 67 67 Table 3 2 Summary ANOVA table results for congruence coefficient df SS MS F p Between Subject Factor NF 2 108.68 54.34 20294.00 <.0001 0.45 NI 1 11.16 11.16 4167.24 <.0001 0.08 PFL 1 104.55 104.55 39043.00 <.0001 0.44 SS 3 14.07 4.69 1752.10 <.0001 0.10 NFNI 2 3.99 1.99 744.14 <.0001 0.03 NFPFL 2 55.11 27.55 10289.70 <.0001 0.30 NFSS 6 2.80 0.47 174.11 <.0001 0.02 NIPFL 1 6.35 6.35 2371.28 <.0001 0.05 NISS 3 1.97 0.66 245.22 <.0001 0.01 PFLSS 3 7.57 2.52 942.10 < .0001 0.05 NFNIPFL 2 2.28 1.14 425.57 <.0001 0.02 NFNISS 6 1.14 0.19 71.04 <.0001 0.01 NFPFLSS 6 1.67 0.28 103.64 <.0001 0.01 NIPFLSS 3 2.00 0.67 248.56 <.0001 0.02 NFNIPFLSS 6 1.04 0.17 64.83 <.0001 0.01 Within Subject Factor E 1 0.00 0.00 0.11 0.74 0.00 ENF 2 0.00 0.00 0.76 0.47 0.00 ENI 1 0.00 0.00 0.38 0.54 0.00 EPFL 1 0.00 0.00 0.00 0.97 0.00 ESS 3 0.00 0.00 0.45 0.72 0.00 ENFNI 2 0.00 0.00 0.25 0.78 0.00 ENFPFL 2 0.00 0.00 0.79 0.46 0.00 ENFSS 6 0.00 0.00 0.72 0.63 0.00 ENIPFL 1 0.00 0.00 0.36 0.55 0.00 ENISS 3 0.00 0.00 2.21 0.09 0.00 EPFLSS 3 0.00 0.00 0.58 0.63 0.00 ENFNIPFL 2 0.00 0.00 0.23 0.79 0.00 ENFNISS 6 0.00 0.00 2.51 0.02 0.00 ENFPFLSS 6 0.00 0.00 0.74 0.62 0.00 ENIPFLSS 3 0.00 0.00 2.26 0.08 0.00 ENFNIPFLSS 6 0.00 0.00 2.48 0.02 0.00 Note: NF indicates the number of factors, NI the number of items per factor, PFL the magnitude of primary loadings, SS the sample size, and E the estimation method. PAGE 68 68 Table 3 3. Mean c ongruence c oefficient by NF and PFL NF PFL Mean 2 Medium 0.979 High 0.993 4 Medium 0.927 High 0.981 6 Medium 0.839 High 0.969 Note: NF indicates the number of factors and PFL the magnitude of the primary factor loadings. Table 3 4. Mean congruence coefficient by NI and PFL NI PFL Mean 7 Medium 0.896 High 0.978 14 Medium 0.934 High 0.984 Note: NI indicates the number of items per factor and PFL the magnitude of the primary factor loadings Table 3 5. Mean congruence coefficient by PFL and SS PFL SS Mean Medium 200 0.880 400 0.917 600 0.934 800 0.929 High 200 0.976 400 0.982 600 0.983 800 0.984 Note: PFL indicates the magnitude of the primary factor loadings and SS the sample size. PAGE 69 69 Table 3 6 Root m ean s quared d eviation NF NI PFL SS DWLS ULS 2 7 Medium 200 0.107 0.107 400 0.080 0.080 600 0.075 0.075 800 0.067 0.067 High 200 0.072 0.072 400 0.061 0.061 600 0.057 0.057 800 0.055 0.055 14 Medium 200 0.091 0.090 400 0.070 0.070 600 0.059 0.059 800 0.056 0.056 High 200 0.068 0.068 400 0.054 0.054 600 0.050 0.050 800 0.047 0.047 4 7 Medium 200 0.153 0.152 400 0.120 0.120 600 0.101 0.101 800 0.108 0.109 High 200 0.090 0.090 400 0.079 0.079 600 0.076 0.076 800 0.073 0.073 14 Medium 200 0.110 0.110 400 0.092 0.092 600 0.078 0.078 800 0.084 0.085 High 200 0.080 0.080 400 0.069 0.069 600 0.065 0.065 800 0.063 0.063 6 7 Medium 200 0.182 0.182 400 0.154 0.154 600 0.135 0.135 800 0.143 0.143 High 200 0.096 0.096 400 0.086 0.086 600 0.083 0.083 800 0.081 0.081 14 Medium 200 0.141 0.140 400 0.125 0.125 600 0.122 0.123 800 0.116 0.116 High 200 0.084 0.084 400 0.075 0.075 600 0.071 0.071 800 0.069 0.069 PAGE 70 70 Table 3 7. Summary ANOVA table results for root mean squared deviation df SS MS F p Between Subject Factor NF 2 30.19 15.09 15992.10 <.0001 0.40 NI 1 6.59 6.59 6985.48 <.0001 0.13 PFL 1 31.10 31.10 32955.50 <.0001 0.40 SS 3 10.24 3.41 3614.82 <.0001 0.18 NFNI 2 0.50 0.25 264.44 <.0001 0.01 NFPFL 2 7.22 3.61 3825.51 <.0001 0.14 NFSS 6 0.13 0.02 22.44 <.0001 0.00 NIPFL 1 1.18 1.18 1244.96 <.0001 0.03 NISS 3 0.18 0.06 65.30 <.0001 0.00 PFLSS 3 1.50 0.50 528.34 <.0001 0.03 NFNIPFL 2 0.16 0.08 83.53 <.0001 0.00 NFNISS 6 0.14 0.02 25.49 <.0001 0.00 NFPFLSS 6 0.10 0.02 18.06 <.0001 0.00 NIPFLSS 3 0.27 0.09 94.87 <.0001 0.01 NFNIPFLSS 6 0.12 0.02 20.47 <.0001 0.00 Within Subject Factor E 1 0.00 0.00 0.15 0.70 0.00 ENF 2 0.00 0.00 1.22 0.29 0.00 ENI 1 0.00 0.00 0.07 0.79 0.00 EPFL 1 0.00 0.00 7.67 0.01 0.00 ESS 3 0.00 0.00 4.10 0.01 0.00 ENFNI 2 0.00 0.00 0.28 0.76 0.00 ENFPFL 2 0.00 0.00 1.14 0.32 0.00 ENFSS 6 0.00 0.00 0.79 0.58 0.00 ENIPFL 1 0.00 0.00 0.23 0.63 0.00 ENISS 3 0.00 0.00 1.96 0.12 0.00 EPFLSS 3 0.00 0.00 4.42 0.00 0.00 ENFNIPFL 2 0.00 0.00 0.22 0.80 0.00 ENFNISS 6 0.00 0.00 1.46 0.19 0.00 ENFPFLSS 6 0.00 0.00 0.95 0.46 0.00 ENIPFLSS 3 0.00 0.00 1.88 0.13 0.00 ENFNIPFLSS 6 0.00 0.00 1.55 0.16 0.00 Note: NF indicates the number of factors, NI the number of items per factor, PFL the magnitude of primary loadings, SS the sample size, and E the estimation method. PAGE 71 71 Table 3 8. Mean r oot mean squared deviation by NF and PFL NF PFL Mean 2 Medium 0.076 High 0.058 4 Medium 0.106 High 0.074 6 Medium 0.140 High 0.081 Note: NF indicates the number of factors and PFL the magnitude of the primary factor loadings. Table 3 9. Mean r oot mean squared deviation by NI NI Mean 7 0.097 14 0.081 Note: NI indicates the number of items per factor. Table 3 10. Mean r oot mean squared deviation by SS SS Mean 200 0.106 400 0.089 600 0.081 800 0.080 Note: SS indicates the sample size. PAGE 72 72 Table 3 11 General p attern a ccuracy NF NI PFL SS DWLS ULS 2 7 Medium 200 0.597 0.598 400 0.896 0.897 600 0.903 0.897 800 0.956 0.958 High 200 1.000 1.000 400 1.000 1.000 600 1.000 1.000 800 1.000 1.000 14 Medium 200 0.402 0.398 400 0.891 0.884 600 0.979 0.979 800 0.977 0.976 High 200 1.000 1.000 400 1.000 1.000 600 1.000 1.000 800 1.000 1.000 4 7 Medium 200 0.053 0.045 400 0.321 0.328 600 0.590 0.593 800 0.437 0.430 High 200 1.000 1.000 400 1.000 1.000 600 1.000 1.000 800 1.000 1.000 14 Medium 200 0.131 0.130 400 0.319 0.308 600 0.684 0.674 800 0.548 0.544 High 200 1.000 1.000 400 1.000 1.000 600 1.000 1.000 800 1.000 1.000 6 7 Medium 200 0.000 0.000 400 0.015 0.016 600 0.049 0.047 800 0.053 0.054 High 200 0.993 0.993 400 1.000 1.000 600 1.000 1.000 800 1.000 1.000 14 Medium 200 0.001 0.001 400 0.029 0.026 600 0.046 0.046 800 0.073 0.066 High 200 0.999 0.999 400 1.000 1.000 600 1.000 1.000 800 1.000 1.000 PAGE 73 73 Table 3 12 Summary ANOVA table results for general pattern accuracy df SS MS F p Between Subject Factor NF 2 2523.58 1261.79 11749.00 <.0001 0.32 NI 1 1.59 1.59 14.83 0.0001 0.00 PFL 1 8247.95 8247.95 76800.00 <.0001 0.61 SS 3 434.73 144.91 1349.32 <.0001 0.08 NFNI 2 9.15 4.57 42.59 <.0001 0.00 NFPFL 2 2511.39 1255.70 11692.30 <.0001 0.32 NFSS 6 188.69 31.45 292.83 <.0001 0.03 NIPFL 1 1.40 1.40 13.06 0.0003 0.00 NISS 3 8.56 2.85 26.57 <.0001 0.00 PFLSS 3 427.78 142.59 1327.76 <.0001 0.07 NFNIPFL 2 9.26 4.63 43.10 <.0001 0.00 NFNISS 6 17.32 2.89 26.88 <.0001 0.00 NFPFLSS 6 194.61 32.43 302.02 <.0001 0.04 NIPFLSS 3 9.21 3.07 28.58 <.0001 0.00 NFNIPFLSS 6 16.74 2.79 25.98 <.0001 0.00 Within Subject Factor E 1 0.03 0.03 9.93 0.00 0.00 ENF 2 0.01 0.00 1.22 0.29 0.00 ENI 1 0.02 0.02 5.52 0.02 0.00 EPFL 1 0.03 0.03 9.93 0.00 0.00 ESS 3 0.00 0.00 0.06 0.98 0.00 ENFNI 2 0.00 0.00 0.40 0.67 0.00 ENFPFL 2 0.01 0.00 1.22 0.29 0.00 ENFSS 6 0.01 0.00 0.49 0.82 0.00 ENIPFL 1 0.02 0.02 5.52 0.02 0.00 ENISS 3 0.02 0.01 2.51 0.06 0.00 EPFLSS 3 0.00 0.00 0.06 0.98 0.00 ENFNIPFL 2 0.00 0.00 0.40 0.67 0.00 ENFNISS 6 0.05 0.01 2.74 0.01 0.00 ENFPFLSS 6 0.01 0.00 0.49 0.82 0.00 ENIPFLSS 3 0.02 0.01 2.51 0.06 0.00 ENFNIPFLSS 6 0.05 0.01 2.74 0.01 0.00 Note: NF indicates the number of factors, NI the number of items per factor, PFL the magnitude of primary loadings, SS the sample size, and E the estimation method. PAGE 74 74 Table 3 13. Mean g eneral pattern accuracy by NF and PFL NF PFL Mean 2 Medium 0.824 High 1.000 4 Medium 0.383 High 1.000 6 Medium 0.037 High 0.999 Note: NF indicates the number of factors and PFL the magnitude of the primary factor loadings. Table 3 14. Mean g eneral pattern accuracy by PFL and SS PFL SS Mean Medium 200 0.196 400 0.411 600 0.541 800 0.506 High 200 0.999 400 1.000 600 1.000 800 1.000 Note: PFL indicates the magnitude of the primary factor loadings and SS indicates the sample size. PAGE 75 75 Table 3 15 Total p attern a ccuracy NF NI PFL SS DWLS ULS 2 7 Medium 200 0.143 0.155 400 0.320 0.325 600 0.322 0.329 800 0.390 0.385 High 200 0.316 0.324 400 0.379 0.381 600 0.440 0.446 800 0.470 0.461 14 Medium 200 0.019 0.020 400 0.086 0.089 600 0.149 0.151 800 0.147 0.144 High 200 0.081 0.087 400 0.125 0.125 600 0.183 0.183 800 0.178 0.175 4 7 Medium 200 0.001 0.000 400 0.000 0.000 600 0.003 0.003 800 0.001 0.002 High 200 0.008 0.006 400 0.011 0.010 600 0.007 0.010 800 0.015 0.018 14 Medium 200 0.000 0.000 400 0.000 0.000 600 0.000 0.000 800 0.000 0.000 High 200 0.000 0.000 400 0.000 0.000 600 0.000 0.000 800 0.000 0.000 6 7 Medium 200 0.000 0.000 400 0.000 0.000 600 0.000 0.000 800 0.000 0.000 High 200 0.000 0.000 400 0.000 0.000 600 0.000 0.000 800 0.000 0.000 14 Medium 200 0.000 0.000 400 0.000 0.000 600 0.000 0.000 800 0.000 0.000 High 200 0.000 0.000 400 0.000 0.000 600 0.000 0.000 800 0.000 0.000 PAGE 76 76 Table 3 16 Summary ANOVA table results for total pattern accuracy df SS MS F p Between Subject Factor NF 2 1165.93 582.97 5436.15 <.0001 0.18 NI 1 145.63 145.63 1357.96 <.0001 0.03 PFL 1 16.41 16.41 152.99 <.0001 0.00 SS 3 37.18 12.39 115.57 <.0001 0.01 NFNI 2 269.62 134.81 1257.09 <.0001 0.05 NFPFL 2 27.28 13.64 127.20 <.0001 0.01 NFSS 6 71.34 11.89 110.88 <.0001 0.01 NIPFL 1 3.64 3.64 33.93 <.0001 0.00 NISS 3 2.85 0.95 8.85 <.0001 0.00 PFLSS 3 1.83 0.61 5.68 0.0007 0.00 NFNIPFL 2 4.86 2.43 22.65 <.0001 0.00 NFNISS 6 4.88 0.81 7.58 <.0001 0.00 NFPFLSS 6 4.20 0.70 6.52 <.0001 0.00 NIPFLSS 3 0.64 0.21 1.98 0.1153 0.00 NFNIPFLSS 6 1.62 0.27 2.51 0.0196 0.00 Within Subject Factor E 1 0.01 0.01 5.22 0.02 0.00 ENF 2 0.02 0.01 3.99 0.02 0.00 ENI 1 0.01 0.01 2.25 0.13 0.00 EPFL 1 0.00 0.00 0.35 0.56 0.00 ESS 3 0.04 0.01 5.29 0.00 0.00 ENFNI 2 0.01 0.00 1.49 0.23 0.00 ENFPFL 2 0.00 0.00 0.81 0.45 0.00 ENFSS 6 0.12 0.02 7.84 <.0001 0.00 ENIPFL 1 0.00 0.00 0.35 0.56 0.00 ENISS 3 0.01 0.00 1.07 0.36 0.00 EPFLSS 3 0.00 0.00 0.19 0.91 0.00 ENFNIPFL 2 0.00 0.00 0.81 0.45 0.00 ENFNISS 6 0.03 0.00 2.04 0.06 0.00 ENFPFLSS 6 0.00 0.00 0.24 0.96 0.00 ENIPFLSS 3 0.00 0.00 0.57 0.63 0.00 ENFNIPFLSS 6 0.01 0.00 0.35 0.91 0.00 Note: NF indicates the number of factors, NI the number of items per factor, PFL the magnitude of primary loadings, SS the sample size, and E the estimation method. PAGE 77 77 Table 3 17. Mean t otal pattern accuracy by NF and NI NF NI Mean 2 7 0.349 14 0.121 4 7 0.006 14 0 6 7 0 14 0 Note: NF indicates the number of factors and NI the number of items per factor. PAGE 78 78 Table 3 18 Per e lement a ccuracy NF NI PFL SS DWLS ULS 2 7 Medium 200 0.922 0.923 400 0.959 0.959 600 0.956 0.956 800 0.966 0.965 High 200 0.963 0.964 400 0.969 0.969 600 0.973 0.973 800 0.975 0.975 14 Medium 200 0.929 0.930 400 0.957 0.957 600 0.966 0.966 800 0.967 0.967 High 200 0.958 0.958 400 0.968 0.968 600 0.971 0.971 800 0.972 0.972 4 7 Medium 200 0.858 0.859 400 0.910 0.910 600 0.933 0.933 800 0.921 0.921 High 200 0.952 0.952 400 0.962 0.962 600 0.964 0.964 800 0.968 0.968 14 Medium 200 0.907 0.908 400 0.929 0.929 600 0.946 0.946 800 0.939 0.939 High 200 0.948 0.949 400 0.961 0.961 600 0.964 0.964 800 0.965 0.965 6 7 Medium 200 0.836 0.836 400 0.876 0.876 600 0.902 0.902 800 0.892 0.892 High 200 0.953 0.954 400 0.964 0.964 600 0.966 0.966 800 0.969 0.969 14 Medium 200 0.878 0.880 400 0.900 0.900 600 0.903 0.903 800 0.912 0.912 High 200 0.953 0.953 400 0.964 0.965 600 0.967 0.967 800 0.968 0.969 PAGE 79 79 Table 3 19 Summary ANOVA table results for per element accuracy df SS MS F p Between Subject Factor NF 2 20.40 10.20 4618.97 <.0001 0.16 NI 1 1.42 1.42 642.68 <.0001 0.01 PFL 1 48.14 48.14 21801.80 <.0001 0.31 SS 3 13.84 4.61 2089.87 <.0001 0.11 NFNI 2 0.62 0.31 139.27 <.0001 0.01 NFPFL 2 14.50 7.25 3282.28 <.0001 0.12 NFSS 6 0.31 0.05 23.23 <.0001 0.00 NIPFL 1 2.07 2.07 935.76 <.0001 0.02 NISS 3 0.44 0.15 65.85 < .0001 0.00 PFLSS 3 3.90 1.30 588.44 <.0001 0.03 NFNIPFL 2 0.40 0.20 89.59 <.0001 0.00 NFNISS 6 0.36 0.06 27.03 <.0001 0.00 NFPFLSS 6 0.25 0.04 18.63 <.0001 0.00 NIPFLSS 3 0.65 0.22 98.57 <.0001 0.01 NFNIPFLSS 6 0.34 0.06 25.88 <.0001 0.00 Within Subject Factor E 1 0.00 0.00 42.13 <.0001 0.00 ENF 2 0.00 0.00 0.10 0.90 0.00 ENI 1 0.00 0.00 7.06 0.01 0.00 EPFL 1 0.00 0.00 1.17 0.28 0.00 ESS 3 0.00 0.00 8.29 <.0001 0.00 ENFNI 2 0.00 0.00 0.14 0.87 0.00 ENFPFL 2 0.00 0.00 0.01 0.99 0.00 ENFSS 6 0.00 0.00 0.36 0.91 0.00 ENIPFL 1 0.00 0.00 2.02 0.15 0.00 ENISS 3 0.00 0.00 4.18 0.01 0.00 EPFLSS 3 0.00 0.00 1.47 0.22 0.00 ENFNIPFL 2 0.00 0.00 0.56 0.57 0.00 ENFNISS 6 0.00 0.00 1.18 0.31 0.00 ENFPFLSS 6 0.00 0.00 0.12 0.99 0.00 ENIPFLSS 3 0.00 0.00 0.55 0.65 0.00 ENFNIPFLSS 6 0.00 0.00 2.40 0.03 0.00 Note: NF indicates the number of factors, NI the number of items per factor, PFL the magnitude of primary loadings, SS the sample size, and E the estimation method. PAGE 80 80 Table 3 20. Mean p er element accuracy by NF and PFL NF PFL Mean 2 Medium 0.953 High 0.969 4 Medium 0.918 High 0.960 6 Medium 0.887 High 0.963 Note: NF indicates the number of factors and PFL indicates the magnitude of the primary factor loadings. Table 3 21. Mean p er element accuracy by SS SS Mean 200 0.922 400 0.943 600 0.951 800 0.951 Note: SS indicates the sample size. PAGE 81 81 CHAPTER 4 CONCLUSIONS AND DISCUSSION The main purpose of this study was to investigate the effect of sample size, number of factors, number of items per factor magnitude of the primary factor loadings, and estimation procedure on accuracy of factor recovery in EFA with ordinal data. To address this purpose, two steps were conducted. First, the factor loadings were estimated by using DWLS or ULS under all combin ations of numbers of factors (2, 4, 6), number of items (7, 14), and magnitude of primary factor loadings (medium, high), and sample size (200, 400, 600, 800). Second, the effects of the five factors (estimation procedure, number of factors, number of item s per factor, primary factor loading s and sample sizes) and their interactions on factor recovery indices were examined by using a 3 (Number of Factors) x 2 (Number of Items) x 2 (magnitude of the Primary Factor Loading) x 4 (sample Size) x 2 (Estimation Method) ANOVA with repeated measures on the estimation method factor. Brief Summary Research Question 1 do sample size, magnitude of primary factor loading, number of factors, and numb er of items per factor influence the factor recovery by DWLS and ULS in EFA conclusions drawn from the findings of the study are summarized below. Increasing sample size improve d most of the studi ed factor recovery indices. The congruence coefficient, general pattern accuracy, root mean squared deviation, and per element accuracy all improved as the sample size increased However, sample size interacted with the magnitude of factor loadings for the first two indices and the effec t of PAGE 82 82 sample size wa s stronger when the primary factor loadings were medium T h er e was a declining effect of sample size The largest improvement tended to occur as the sample size increased from 200 to 400; there was little improvement in either congruence coefficient or general pattern accuracy as the sample size increased from 600 to 800. M ost of the factor recovery indices improved when th e primary factor loading was in the range .6 to .8 rather than in the range .4 to .6 The congruence coefficient, ge neral pattern accuracy, root mean squared deviation, and per element accuracy ha d better values when the primary factor loadings were large But the magnitude of the primary factor loading s typically interacted with number of factors, number of items per f actor or sample size and th e interactions that emerged varied with the factor recovery index. The magnitude of the primary factor loading had a stronger effect on the congruence coefficient, general pattern accuracy, root mean squared deviation, and per e lement accuracy when the number of factor s was larger ; ha d a more evident effect on congruence coefficient and general pattern accuracy when the sample size was smaller ; and more strongly affect ed congruence coefficient when the number of items per factor was smaller. The mean values for the congruence coefficient, root mean squared deviation, and total pattern accuracy usually improved with more items per factor. But number of items per factor interacted with either the number of factors or the magnitude of the primary factor loadings. Number of items per factor had a more noticeable effect on the total pattern accuracy when the number of factors was smaller. In particular the effect vanished when the number of factor was 6. Number of items per factor also ha d stronger effect on the congruence coefficient when the magnitude of the primary factor loadings was lower PAGE 83 83 In addition, number of factors affected all of the factor recovery indices M ean values of the congruence coefficient, general patt ern accuracy, root mean squared deviation, per element accuracy and total pattern accuracy tended to get worse as t he number of factors increased But the number of factors interacted with other factors such as magnitude of factor loadings or the number of items per factor and th e interactions that emerged depended on the factor recovery index The number of factors had a stronger effect on the congruence coefficient, general pattern accuracy, root mean squared deviation, and per element accuracy when th e primary factor loadings were lower and a more evident effect on total pattern accuracy when the number of factors per item was smaller. Research Question 2 The second research question was To what extent is the effect of sample size on factory recove ry in DWLS and ULS influence d by the magnitude of primary factor loading s, number of factors, and number of items per factor factors on the effect of sample size on factor recovery is reflected by the interaction of sample size wit h the other factors. Partial omega squared was equal to or larger than .05 for the interaction of sample size and magnitude of the primary factor loading when the recovery indices were congruence coefficient and general pattern accuracy. For all recovery i ndices, partial omega squared was less than .05 for the interaction of sample size with number of items per factor and number of factors. Therefore the following summary is for the effect of sample size on the congruence coefficient and general pattern acc uracy. T he effect of sample size wa s stronger when the primary factor loadings were in the .4 to .6 range and the interaction was more apparent for general PAGE 84 84 pattern accuracy. But larger sample size did not always guarantee a better factor recovery index and sometimes it resulted in little improvement in an index When the primary factor loadings were high, the congruence coefficient improved slightly with increasing sample size but general pattern accuracy did not improve for sample sizes larger than 400 bec ause it reached the maximum value of 1.0 at sample size of 400. When the primary factor loadings were medium, sample size in the range of 200 to 600 had an evident positive effect on the congruence coefficient and general pattern accuracy but sample size in the range of 600 to 800 did not. Research Question 3 The third research question was What sample size is necessary to produce acceptable five statistics, only the congruence coefficient has a criterion published and cited in the references ( Hogarty et al., 2005 ; MacCallum et al., 1999; MacCallum et al., 2001). The congruence coefficient reflects overall recovery of factor loading matrix by measuring the similarity between factor loading matrices from sample and population. Therefore, the sample size recommendations are based on the congruence coefficient. Result in Table 3 1 provides a basis for guidelines for the sample s ize under differen t conditions and the recommended sample size is summarized in Table 4 1. The congruence coefficient s for DWLS and ULS under the same condition are almost identical, so the following sample size suggestions apply to both estimators unless the estimation met hod is specifically mentioned. As presented in Table 4 1, a sample size of 200 resulted in good congruence coefficients (i.e., 0.92 to 0.98 ) for all of the conditions with 2 factors and for all of the high primary loading conditions when there were 4 or 6 factors. To achieve an excellent PAGE 85 85 congruence coefficient (i.e., 0.98 to 1.00 ) in conditions with high primary factor loadings the following sample sizes were sufficient: a sample size 200 when there were 2 factors regardless of the number of items per fact or; a sample size 400 when there were 4 factors and 14 items per factor; and a sample size 800 when there were 4 factors and 7 items per factor. For 6 factors with high primary loadings, sample sizes in the range of 200 to 800 did not result in an excellen t congruence coefficient. To obtain a good congruence coefficient in conditions with medium primary factor loadings the following sample sizes were sufficient: a sample size 200 when there were 2 factors; a sample size of 200 when there were 4 factors and 14 items per factor ; and a sample size 600 for 4 factors and 7 items per factor. For 6 factors with medium primary loadings, no sample sizes in the range of 200 to 800 induced a good congruence and larger sample size might be helpful. When the primary fac tor loadings were medium, excellent congruence occurred only with 2 factors case: a sample size 800 was sufficient for 7 items per factor and sample size 400 for 14 items per factor. When the primary factor loadings were medium, excellent congruence coeffi cients did not occur for 4 factors or 6 factors cases with the sample sizes in the range of 200 to 800. A sample size larger than 800 would be required when the primary factor loadings are medium and there are 4 or more factors The sample size recommendations in Table 4 1 were based on the congruence coefficient. If the root mean squared deviation were used as criterion, more conservative sample size recommendations would likely be needed, which was suggested by Table 3 6. PAGE 86 86 Applie d researchers who do not have any information about the likely size of the primary factor loadings may want to use the recommendation based on medium primary factor loadings, as these recommendations are more conservative, and take into account the number of items and expected number of factors for their study. Comparison of Findings with Previous Studies DWLS vs. ULS Several previous studies in CFA with ordinal data compared the performance of DWLS and ULS ( Foreo et al. 2009; Rigdon & Ferguson 1991; Yang Wallentin et al., 2010) under different simulation conditions. Yang Wallentin et al. (2010) found that the overall performance of DWLS and ULS were close but ULS had significantly better relative bias of loading estimates than DWLS in the majority of cases. Foreo et al. (2009) indicated that DWLS generally slightly outperformed ULS in terms of convergence rates but ULS outperformed DWLS slightly in loading estimation accuracy as measured by relative bias of loading estimates and their standard errors However, Rigdon and Ferguson (1991) concluded that DWLS had relative better performance than ULS in regard to the MSE of loading estimates. In the present study, the performances of DWLS and ULS in EFA with ordinal data were very close in terms of five s tatistics in Tables 3 1, 3 6, 3 11, 3 15, and 3 18, and the effect of estimation method was trivial with a partial omega equal to .00 for each of the statistics. The congruence coefficient, r oot m ean s quared d eviation and per element accuracy were almost identical for two estimation methods across all simulation conditions. However, slight differences between estimation methods occurred for general pattern accuracy and total pattern accuracy. The overall general pattern accuracy of DWLS was slightly better than that of ULS. Among 48 simulation conditions, PAGE 87 87 DWLS had better general pattern accuracy in 13 conditions with a difference of at least 0.005 in 7 out of 13 conditions, whereas ULS had better general pattern accuracy in 7 conditions with a difference of at least 0.005 in 1 out of 7 conditions. In contrast, the overall total pattern accuracy of ULS was better than that for DWLS. ULS had better total pattern accuracy in 13 conditions with a difference of at least 0.005 in 6 out of 13 conditions, and DWLS h ad better total pattern accuracy in 6 conditions with a difference of at least 0.005 in 2 of 6 conditions. DWLS and ULS ha d very similar performance in terms of convergence rate and improper solution rate : both ha d perfect convergence rates across all simulation conditions, and DWLS and ULS generate d 1 and 2 improper solutions, respectively, among 48000 replications of this study. Main Effects and Interactions Although the simulation conditions and studied statistic s are different from study to study, the findings in the present study about the effects of magnitude of primary factor loading, sample size, number of factors, and number of items per factor are generally consistent with results summarized in the literatu re review. Better factor recovery is associated with higher fa ctor loading, larger sample size, and more items per factor. The additional finding from present study is factor recovery was also improved by a smaller number of factors. The effect of number o f factor was mentioned in Beauducel and Herzberg (2006) but with extremely small influence In contrast with the effects of number of items per factor reported in the previous studies, in the current study better total pattern accuracy was observed with a smaller number of items per factor when the number of factors was 2. Total pattern accuracy requires agreement in terms of the 0.3 rule between corresponding elements in the sample and population factor loading matrix. As the number of items per factor inc reases with a fixed number of factors, a PAGE 88 88 larger number of factor loadings make it more difficult to achieve agreement between the sample and population loadings. Therefore, total pattern accuracy is worse when there are more items per factor. The conclusi ons about the effects of magnitude of prima ry factor loading, sample size, number of factors, and number of items per factor are compared with conclusions in Hogarty et al. (2005), MacCallum et al. (1999), MacCallum et al. (2001), and Velicer and Fava (199 8) These later studies also used ANOVA to study the effects of factors. However, MacCallum et al. (1999), MacCallum et al. (2001), and Velicer and Fava (1998) did not include number of factors, whereas number of factors was included in the present study a nd in Hogarty et al. (2005) The present and comparison studies all reported that sa mple size and number of items per factor ha d small effect s on congruence coefficient and root mean squared deviation when the primary factor loadings were large and stronge r effects when the loadings were medium. In the present study and in Hogarty et al. (2005) the effect of number of factors was weak when the primary factor loadings were high and strong when the factor loadings were medium. Consistent with results in Hogarty et al. (2005), the present study also indicated that the variables influencing factor recovery were dependent on the specific factor recovery index. For g eneral pattern accuracy, the important interaction s w ere magnitude of primary factor loading b y sample size and number of factors by magnitude of primary loadings. For total pattern accuracy, the only important interaction was between number of factors and number of items per factor: number of items per factor ha d a more pronounced effect when t he number of factors was smaller and became PAGE 89 89 negligible when the nu mber of factors was large. The PFL x SS interaction on general pattern accuracy was found in both the present study and Hogarty et al. (2005). However, t he NF x N I interaction on total pattern accuracy that was found in the present study was not found in Hogarty et al. (2005). For the per element accuracy, the interaction between primary factor loadings and number of factors was reported in the present study but not in Hogarty et al. (2005), whi le the interaction of magnitude of the primary factor loadings and sample size was reported in Hogarty et al. (2005) but not in the present study. Sample Size The influence of number of factors, number of items per factor and magnitude of factor loadings on the effect of sample size will be discussed by comparing the results in the current study with results in Hogarty et al. (2005), MacCallum et al. ( 1999 ), and MacCallum et al. ( 2001 ) In the current study and the previous studies ANOVA was conducted. Res ults for the congruence coefficient are discussed because it was included in all four studies and has a published criterion. Results for magnitude of the primary factor loading, number of items per factor, and sample size are discussed because these factor s were manipulated in all four studies. However the number of factors was investigated only in the current study and Hogarty et al. (2005). According to MacCallum et al. (1999) and MacCallum et al. (2001), because sample size had little effect when the loadings were high (communality was 0.6 0.8) and factors were strongly determined (the ratio between number of items and number of factors was 10:3 or 20:3), a good congruence coefficient can be achieved even with fairly small samples (less than 100). In o rder to see if the same conclusion can be drawn from the present study, the average congruence coefficient by factor loading, PAGE 90 90 number of items per factor, and sample size are reported in Table 4 2 The results in Table 4 2 are consistent with the conclusion s in MacCallum et al. (1999) and MacCallum et al. (2001): W hen the primary factor loadings were high (0.6 0.8) and the factors were strongly determined (7 or 14 items per factor), the smallest sample size in this study (200) was sufficient for a good congr uence coefficient (0.973 0.978) regardless of the number of factors. The same collapsed tables for root mean squared deviation, general pattern accuracy, total pattern accuracy, and per element accuracy are reported i n Tables 4 3 to 4 6 respectively. The smallest sample size in this study (200) was sufficient for good recovery indices for root mean squared deviation (0.077 0.086), for general pattern accuracy (0.998 1.0), and per element accuracy (0.953 0.956). Another conclusion about sample size from the previous studies ( Hogarty et al., 2005 ; MacCallum et al., 1999; MacCallum et al., 2001) is that w hen communalities are low or cover a wide range, good factor congruence coefficients require a larger sample size and more items per factor The results in Ta ble s 4 2 to 4 6 validate this finding: the effect of sample size and number of items per factor on congruence coefficient, general pattern accuracy, and per element accuracy, are more evident in the medium loadings condition than in high loadings condition The present study showed that the number of factors, number of items per factor, and sample size had a small effect on factor recovery when the primary loadings were high and became important determinants when the primary loadings were medium. These tre nds affect the sample size that is sufficient for obtaining good congruence coefficients. Table 4 1 shows sample size recommendations, from among the sample sizes investigated in the study, for obtaining good and excellent congruence PAGE 91 91 coefficients. These re commendations vary by number of factors, number of items per factor, and magnitude of primary factor loading. A small sample size was sufficient provided the primary factor loadings were high: a sample size of 200 resulted in a n excellent congruence coeffi cient for all of the conditions with 2 factors and a good congruence coefficient for all of the conditions with 4 and 6 factors. In addition when the primary factor loadings were medium and the number of factors was 2 a sample size of 200 resulted in a good congruence coefficient. However for 4 or 6 factors a larger sample size was required when the factor loadings were medium and the required sample size was influenced by the number of items per factor and number of factors. For four factors a sample size 200 was sufficient for 14 items per factor and a sample size 600 was sufficient for 7 items per factor. For 6 factors, no sample sizes in the range of 200 to 800 produced a good congruence coefficient. Future Research Questions As is true in all studi es only a limited number of questions could be investigated and future studies can broaden the scope of investigation. First, the performance of the standard errors of the parameter estimates was not investigated because of the time demands of calculating standard errors in M plus when the EFA model is applied to ordinal data. For example, when the number of factors was 6 and the number of items per factor was 14 running time per replication would have been approximately 10 minutes. Although conditions with a smaller number of factors and a smaller number of items per factor would not have been so computationally demanding, the standard errors of the parameter estimates were not included in the present study Future simulation work with a smaller number of re plications per conditions, (e.g., 500 ) can be conducted to detect the important determinants o f accurate standard errors of PAGE 92 92 parameter estimates. Similarly, good ness of fit tests and indices were not included because they are also computationally demanding. Therefore, another research possibility is to assess how model fit indices are affected by variables in the present design. As shown in the literature review, the influence of additional variables on factor recovery has been investigated in CFA with ordin al data but not in EFA with ordinal data. These effects include the number of item response categories and the distribution of the responses across the categories Sass (2010) investigated the effect of interfactor correlation in EFA with ordinal data, but used factor recovery ind ices ( bias and RMSE of factor loading estimate) that focused on the estimation accuracy of individual factor loading not the overall accuracy of factor loading matrix. In the current study, the se three variables were not investi gated In the future study, the influence of these effects on factor recovery (congruence coefficient, root mean squared deviation, and pattern accuracy) could be studied by manipulating their settings: varying the number of response categories ( e.g., from 2 to 7), the distribution of ordinal data ( e.g., from approximate normal to severely skewed), or the co rrelation among factors (from low to high ). In the present study the model wa s always correctly specified and this is not always true in the reality. T herefore, the effect of model misspecification on factor recovery might be of interest in future research. There are several ways to misspecify a model. For example, the number of factors in the model may not be equal to the number of factors in the population, or residuals may be uncorrelated in the sample, but correlated in the population. Conclusions Based on the findings presented, there are four conclusions can be set forth PAGE 93 93 First, the factor recovery is very similar for DWLS and ULS across all o f the simulation conditions and very small differences were reported between them in terms of five factor recovery indices. Second, better factor recovery is associated with higher primary factor loadings, a smaller number of factors, a larger number of i tems per factor, and a larger sample size. However, there are interactions among these variables The magnitude of primary factor loading s typically interacts with number of factors, sample size, and the number of items per factor. The magnitude of the pri mary f actor loadings has a stronger effect when the number of factors is larger, the sample size is smaller, or the number of items per factor is smaller The effect of sample size is stronger when the primary factor loadings are smaller The effect of num ber of items per factor is more pronounced when the number of factors is smaller, or the magnitudes of factor loadings are lower. In addition, the number of factor has a stronger effect when the factor loadings are lower, or the number of items per factor is smaller These effects (main and interactions) are factor recovery index dependent and do not occur for each of the indices. Third, sample size had little effect when the loadings were high and the number of items per factor was large. Therefore, a go od congruence coefficient can be achieved even with fairly small samples under these conditions. When factor loadings are lower, good factor congruence coefficients occurred with a larger sample size and a larger number of items per factor. Finally, a sample size of 200 resulted in a good congruence coefficient for all of the conditions with 2 factors and for all of the conditions with 4 and 6 factors when the primary loadings were large However, with medium primary factor loadings a larger PAGE 94 94 sample size was needed and it depended on the other effects ( the number of factors and the number of items per factor) F or four factors a sample size 200 wa s sufficient for 14 items per factor and a sample size 600 wa s sufficient for 7 items per factor. For 6 factor s, no sample sizes in the range of 200 to 800 produced a good congruence coefficient. PAGE 95 95 Table 4 1. Sample size recommendation for a good and an excellent congruence coefficient NF NI PFL Sample size for a g ood mean congruence Sample size for an e xcellent mean congruence 2 7 Medium 200 800 High 200 200 14 Medium 200 400 High 200 200 4 7 Medium 600 N/A High 200 800 14 Medium 200 N/A High 200 400 6 7 Medium N/A N/A High 200 N/A 14 Medium N/A N/A High 200 N/A Note: NF indicates the number of factors, NI the number of items per factor, and PFL the primary factor loadings. A good congruence coefficient is in the range of 0.92 to 0.98 and an excellent congruence is in the range of 0.98 to 1.00. The N/A in the sample siz e means that no sample size s in the range of 200 to 800 produce the good or excellent congruence coefficient. PAGE 96 96 Table 4 2 Mean c ongruence c oefficient by PFL NI and SS PFL NI SS Mean Medium 7 200 0.847 400 0.900 600 0.924 800 0.914 14 200 0.914 400 0.934 600 0.943 800 0.944 High 7 200 0.973 400 0.979 600 0.980 800 0.982 14 200 0.978 400 0.984 600 0.986 800 0.986 Note: PFL indicates the primary factor loadings, NI the number of items per factor, and SS the sample size. Table 4 3 Mean r oot mean squared deviation by PFL NI and SS PFL NI SS Mean Medium 7 200 0.147 400 0.118 600 0.104 800 0.106 14 200 0.114 400 0.096 600 0.087 800 0.085 High 7 200 0.086 400 0.076 600 0.072 800 0.070 14 200 0.077 400 0.066 600 0.062 800 0.060 Note: PFL indicates the primary factor loadings, NI the number of items per factor, and SS the sample size. PAGE 97 97 Table 4 4 Mean g eneral pattern accuracy by PFL NI and SS PFL NI SS Mean Medium 7 200 0.216 400 0.412 600 0.513 800 0.481 14 200 0.177 400 0.410 600 0.568 800 0.531 High 7 200 0.998 400 1.000 600 1.000 800 1.000 14 200 1.000 400 1.000 600 1.000 800 1.000 Note: PFL indicates the primary factor loadings, NI the number of items per factor, and SS the sample size. Table 4 5 Mean t otal pattern accuracy by PFL NI and SS PFL NI SS Mean Medium 7 200 0.050 400 0.108 600 0.110 800 0.130 14 200 0.007 400 0.029 600 0.050 800 0.049 High 7 200 0.109 400 0.130 600 0.151 800 0.161 14 200 0.028 400 0.042 600 0.061 800 0.059 Note: PFL indicates the primary factor loadings, NI the number of items per factor, and SS the sample size. PAGE 98 98 Table 4 6 Mean p er element accuracy by PFL NI and SS PFL NI SS Mean Medium 7 200 0.872 400 0.915 600 0.930 800 0.926 14 200 0.905 400 0.929 600 0.938 800 0.939 High 7 200 0.956 400 0.965 600 0.968 800 0.971 14 200 0.953 400 0.965 600 0.967 800 0.968 Note: PFL indicates the primary factor loadings, NI the number of items per factor, and SS the sample size. PAGE 99 99 L IST OF REFERENCES Abdi, H. (2010). Congruence: Congruence coefficient, R V coefficient, and Mantel coefficient. In Neil Salkind (Ed.), Encyclopedia of Research Design Thousand Oaks, CA: Sage. Anarte Ortiz, M. T., Caballero, F. F., Ruiz de Adana, M. S., Rondn, R. M., Carreira, M., Domnguez Lpez M., Machado, A., Gonzalo Marn, M., Tapia, M. J., Valds, S., Gonzlez Romero, S., & Soriguer, F. C. (2011). Development of a New Fear of Hypoglycemia Scale: FH 15. Psychological Assessment 23, 398 405. Arrindell, W. A., & Van der Ende, J. (1985). An e mpirical test of the utility of the observations to variables ratio in factor and components analysis. Applied Psychological Measurement 9, 165 178. Beauducel, A., & Herzberg, P. Y. (2006). On the performance of maximum likelihood versus means and varianc e adjusted weighted least squares estimation in CFA. Structural Equation Modeling, 13, 186 203. Bock, R. D., & Aitkin, M. (1981). Marginal maximum likelihood estimation of item parameters: an application of the EM algorithm. Psychometrika 46, 443 459. Bo ck, R. D., Gibbons, R., & Muraki, E. (1988). Full information item factor analysis. Applied Psychological Measurement 12, 261 280. Bolt, D. (2005). Limited and full information estimation of item response theory models. In A. Maydeu Olivares & J. J. McAr dle (Eds.), Contemporary psychometrics (pp. 27 71). Mahwah, NJ: Lawrence Erlbaum Associates, Inc. Boulet, J. R. (1996). The effect of nonnormal ability distribution on IRT parameter estimation using full information methods Unpublished doctoral dissertati on University of Ottawa, Ottawa, Canada. Burt, C. (1948). Factor analysis and canonical correlations. British Journal of Psychology, Statistical Section 1, 95 106. Browne, M. W. (1984). Asymptotically distribution free method for the analysis of covariance structures. British Journal of Mathematical and Statistical Psychology 37, 62 83. Brown, K. W., & Ryan, R. M. (2003). The benefits of being present: Mindfulness and its role in psychological well being. Journal of Personality and Social Psychol ogy 84, 822 848. Cattell, R. B. (1978). The scientific use of factor analysis in behavioral and life sciences New York: Plenum. PAGE 100 100 Chao, R. C. & Green, K. E. (2011). Multiculturally sensitive mental health scale (MSMHS): development, factor analysis, re liability, and validity. Psychological assessment 23, 876 887. Comrey, A. L. & Lee, H. B. (1973). A first course in factor analysis New York: Academic Press. pp. 217. D e Winter, J.C. F., Dodou, D. & Wieringa, P. A. (2009). Exploratory factor analysis with small sample sizes Multivariate Behavioral Research 44, 147 181. DiStefano, C. (2002). The impact of categorization with confirmatory factor analysis. Structural Equation Modeling, 9, 327 346. Everitt, B. S. (1975). Multivariate analysis: The need f or data, and other problems. British Journal of Psychiatry 126, 237 240. Fabrigar, L. R., Wegener, D. T., MacCallum, R. C., & Strahan, E. J. (1999). Evaluating the use of exploratory factor analysis in psychological research. Psychological Methods 4, 272 299. Ferron J. M. (2012). Personal communication. Flora, D., & Curran, P. (2004). An empirical evaluation of alternative methods of estimation for confirmatory factor analysis with ordinal data. Psychological Methods, 9, 466 491. Forero, C. G., & Maydeu Olivares, A. (2009). Estimation of IRT graded response models: limited versus full information methods. Psychological Methods 14, 275 299. Forero, C. G., Maydeu Olivares, A., & Gallardo Pujol, D. (2009). Factor analysis with ordinal indicators: a M onte C arlo study comparing DWLS and ULS estimation. Structural equation modeling 16(4), 625 641. Ford, J. K., MacCallum, R. C., & Tait, M. (1986). The application of exploratory factor analysis in applied psychology: A critical review and analysis. Personnel P sychology 39, 291 314. Fraser, C. (1988 ). NOHARM: An IBM PC computer program for fitting both unidimensional and multidimensional normal ogive models of latent trait theory Armidale, Australia: University of New England. Fraser, C., & McDonald, R. P. (19 88). NOHARM: Least squares item factor analysis. Multivariate Behavioral Research 23, 267 269. Gagn, P., & Hancock, G. H. (2006). Measurement model quality, sample size, and solution propriety in confirmatory factor analysis. Multivariate Behavioral Rese arch, 41, 65 83. PAGE 101 101 Gorsuch, R. L. (1983). Factor analysis (2 nd E d.). Hillsdale, NJ: Lawrence Erlbaum. Guadagonli, E., & Velicer, W. F. (1988). Relation of sample size to the stability of component pattern. Psychological Bulletin 103, 265 275. Guilford, J. P. (1954). Psychometric methods (2 nd E d.). New York: McGraw Hill. pp. 533. Hair, J. F. J., Anderson, R. E., Tatham, R. L., & Black,W. C. (1998). Multivariate data analysis (4 th E d.). Saddle River, NJ: Prentice Hall. pp 97. Hogarty, K. Y., Hines, C. V., Kr omrey, J. D., Ferron, J. M., & Mumford, K. R. (2005). The quality of fac tor solutions in exploratory factor analysis: the influence of sample size, communality, and overdetermination Educational and Psychological Measurement 65, 202 226. Holgado Tello, F.P., Chacn Moscoso, S., Barbero Garca, I., & Vila Abad, E. (2010). Polychoric versus Pearson correlations in exploratory and confirmatory factor analysis of ordinal variables. Quality and Quantity 44(1), 153 166. Locke, B. D., Buzolitz, J. S., Lei, P. W., Boswell, J. F., McAleavey, A. A., Sevig, T. D., Dowis, J. D., & Ha yes, J. A. (2011). Development of the counseling center assessment of psychological symptoms 62 (CCAPS 62). Journal of Counseling Psychology 58, 97 109. Lounsbury, J. W., Tatum, H., Gibson, L. W., Park, S., Sundstrom, E. D., Hamrick, F. L., & Wilburn, D. (2003). The development of a Big Five adolescent personality inventory. Journal of Psychoeducational Assessment 21, 111 133. Lynam, D. R., Gaughan, E. T., Miller, J. D., Miller, D. J., Mullins Sweatt, S., & Widiger, T. A. (2011). Assessing the basic traits associated with psychopathy: Development and validation of the Elemental Psychopathy Assessment. Psychological Assessment 23, 108 124. MacCallum, R. C., Widaman, K. F., Preach er, K. J., & Hong, S. (2001). Sample size in factor analysis: the role of model error. Multivariate Behavioral Research 36, 611 637. MacCallum R. C., Widaman, K. F., Zhang, S., & Hong, S. (1999). Sample size in factor analysis. Psychological Methods 4, 8 4 99. Mahmut, K. M., Menictas, C., Stevenson, R. J., & Homewood J. (2011). Validating the factor structure of the self report psychopathy scale in a community sample. Psychological Assessment 23, 670 678. Maydeu Olivares, A. (2001). Limited information e stimation and testing of Thurstonian models for paired comparison data under multiple judgment sampling. Psychometrika 66, 209 227 PAGE 102 102 Muthn, B. (1978). Contributions to factor analysis of dichotomous variables. Psychometrika, 43, 551 560. Muthn, B. (1984). A general structural equation model with dichotomous, ordered categorical and continuous latent variable indicators. Psychometrika, 49, 115 132. Muthn, B. (1993). Goodness of fit with categorical and other nonnormal variables. In K. A. Bollen & J. S. Lon g (Eds.), Testing structural equation models (pp. 205 234). Newbury Park, CA: Sage. Muthn, B., du Toit, S. H. C., & Spisic, D. (1997). Robust inference using weighted least squares and quadratic estimating equations in latent variable modeling with catego rical and continuous outcomes Unpublished manuscript. Nye, C. D. & Drasgow, F. (2011). Assessing goodness of fit: simple rules of thumb simply do not work. Organizational Research Methods 14(3), 548 570. Olejnik, S., & Algina, J. (2003). Generalized Eta and Omega squared statistics: Measures of effect size for some common research designs. Psychological Methods 8, 434 447. Oranje, A. (2003, April). Comparison of estimation methods in factor analysis with categorized variables: Applications to NAEP data Paper presented at the annual meeting of the American Educational Research Association, Chicago, IL. Parry, C. D. H., & McArdle, J. J. (1991). An applied comparison of methods for least squares factor analysis of dichotomous variables. Applied Psychologica l Measurement, 15, 35 46. Rhemtulla, M., Brosseau Liard, P., & Savalei, V. (2010). How many categories is enough to treat data as continuous? A comparison of robust continuous and categorical SEM estimation methods under a range of non ideal situations. Un iversity of British Columbia, February 8, 2010. Rigdon, E. E., & Ferguson, C. E. (1991). The performance of the polychoric correlation coefficient and selected fitting functions in confirmatory factor analysis with ordinal data. Journal of Marketing Resea rch, 28, 491 497. Rothbart, M. K., Ahadi, S. A., Hershey, K. L., & Fisher, P. (2001). Investigations of temperament at 3 Child Development 72, 1394 1408. Russell, D. W. (2002). In search of underlying dimensions: The use (and abuse) of factor analysis in PSPB Personality and Social Psychology Bulletin 28, 1629 1646 Sass, D. A. (2010). Factor loading estimation error and stability using exploratory factor analysis Educational and Psychological Measurement 70, 557 577. PAGE 103 103 Sass, D. A., & Schmitt, T. A. (2010). A comparative investigation of rotation criteria within explorat ory factor analysis Multivariate Behavioral Research 45, 73 103. Schalet, B. D., Durbin, C. E., & Revelle, W. (2011). Multidimensional structure of the hypomanic personality scale. Psychological assessment 23, 504 522. Stout, W., Habing, B., Douglas, J., Kim, H. R., Roussos, L., & Zhang, J. (1996). Conditional covariance based nonparametric multidimensionality assessment. Applied Psychological Measurement 20 331 354. Thurstone, L. L. (1947). Multiple factor analysis Chicago: University of Chicago Press. Tucker, L. R. (1951). A method for the synthesis of factor analysis studies (Personnel Research Section Report No. 984). Washington: Department of the Army. Verheul, R., Andrea, H., Berghout, C. C., Dolan, C., Busschbach, J. J. V., Van der Kroft, P. J. A., & Fonagy, P. (2008). Severity Indices of Personality Problems (SIPP 118): development, factor structure, reliability, and validity. Psychological Assessment 20, 23 34. Wirth, R. J., & Edwards, M. C. (2007). Item factor analysis: Current approaches and future direc tions. Psychological Methods, 12, 58 79. Yang Ordinal Variables With Misspecified Models. Structural Equation Modeling 17(3), 392 423. PAGE 104 104 BIOGRAPHICAL SKETCH Rong Jin was born in Lanzhou, China. She graduated from Beijing University of After being awarded the waiver of graduate school entrance examination, she entered the graduate school of Beijing University of Chemical Engineering in 1997 and received a master degree in control theory and engineering in 2000. In 2004, she graduated from Southern University and A & M College at Baton Rouge with a master degree in t elecommunications and computer network engineering. She began doctoral studies in Research and Evaluation Methodology at University of Florida in the spring of 2008. 