<%BANNER%>

Additive Models for Multitrait-Multimethod Data Assuming a Multiplicative Trait-Method Relationship

Permanent Link: http://ufdc.ufl.edu/UFE0043017/00001

Material Information

Title: Additive Models for Multitrait-Multimethod Data Assuming a Multiplicative Trait-Method Relationship A Simulation Study
Physical Description: 1 online resource (52 p.)
Language: english
Creator: ZHANG,LIDONG
Publisher: University of Florida
Place of Publication: Gainesville, Fla.
Publication Date: 2011

Subjects

Subjects / Keywords: Human Development and Organizational Studies in Education -- Dissertations, Academic -- UF
Genre: Research and Evaluation Methodology thesis, M.A.E.
bibliography   ( marcgt )
theses   ( marcgt )
government publication (state, provincial, terriorial, dependent)   ( marcgt )
born-digital   ( sobekcm )
Electronic Thesis or Dissertation

Notes

Abstract: Confirmatory factor analysis (CFA) is widely used for analyzing multitrait multimethod (MTMM) data. But there is no consensus about whether the multiplicative or the additive trait-method effect of its parameterization, most appropriately represents the underlying structure of MTMM data. The popularity of the two additive CFA models, the correlated traits-correlated methods (CT-CM) model and the correlated traits-correlated uniqueness (CT-CU) model, calls for investigation of their performance to characterize MTMM data that conforms to a multiplicative trait-method effect model. A Simulation study was used to compare the model fit and parameter estimates of the CT-CM and CT-CU models for MTMM data with that of a multiplicative CFA model, assuming a multiplicative trait-method relationship. The results showed that CT-CU works well in most conditions and is quite robust to MTMM data with multiplicative underlying trait-method effect when the matrix size is 3T3M.
General Note: In the series University of Florida Digital Collections.
General Note: Includes vita.
Bibliography: Includes bibliographical references.
Source of Description: Description based on online resource; title from PDF title page.
Source of Description: This bibliographic record is available under the Creative Commons CC0 public domain dedication. The University of Florida Libraries, as creator of this bibliographic record, has waived all rights to it worldwide under copyright law, including all related and neighboring rights, to the extent allowed by law.
Statement of Responsibility: by LIDONG ZHANG.
Thesis: Thesis (M.A.E.)--University of Florida, 2011.
Local: Adviser: Miller, M David.
Electronic Access: RESTRICTED TO UF STUDENTS, STAFF, FACULTY, AND ON-CAMPUS USE UNTIL 2012-04-30

Record Information

Source Institution: UFRGP
Rights Management: Applicable rights reserved.
Classification: lcc - LD1780 2011
System ID: UFE0043017:00001

Permanent Link: http://ufdc.ufl.edu/UFE0043017/00001

Material Information

Title: Additive Models for Multitrait-Multimethod Data Assuming a Multiplicative Trait-Method Relationship A Simulation Study
Physical Description: 1 online resource (52 p.)
Language: english
Creator: ZHANG,LIDONG
Publisher: University of Florida
Place of Publication: Gainesville, Fla.
Publication Date: 2011

Subjects

Subjects / Keywords: Human Development and Organizational Studies in Education -- Dissertations, Academic -- UF
Genre: Research and Evaluation Methodology thesis, M.A.E.
bibliography   ( marcgt )
theses   ( marcgt )
government publication (state, provincial, terriorial, dependent)   ( marcgt )
born-digital   ( sobekcm )
Electronic Thesis or Dissertation

Notes

Abstract: Confirmatory factor analysis (CFA) is widely used for analyzing multitrait multimethod (MTMM) data. But there is no consensus about whether the multiplicative or the additive trait-method effect of its parameterization, most appropriately represents the underlying structure of MTMM data. The popularity of the two additive CFA models, the correlated traits-correlated methods (CT-CM) model and the correlated traits-correlated uniqueness (CT-CU) model, calls for investigation of their performance to characterize MTMM data that conforms to a multiplicative trait-method effect model. A Simulation study was used to compare the model fit and parameter estimates of the CT-CM and CT-CU models for MTMM data with that of a multiplicative CFA model, assuming a multiplicative trait-method relationship. The results showed that CT-CU works well in most conditions and is quite robust to MTMM data with multiplicative underlying trait-method effect when the matrix size is 3T3M.
General Note: In the series University of Florida Digital Collections.
General Note: Includes vita.
Bibliography: Includes bibliographical references.
Source of Description: Description based on online resource; title from PDF title page.
Source of Description: This bibliographic record is available under the Creative Commons CC0 public domain dedication. The University of Florida Libraries, as creator of this bibliographic record, has waived all rights to it worldwide under copyright law, including all related and neighboring rights, to the extent allowed by law.
Statement of Responsibility: by LIDONG ZHANG.
Thesis: Thesis (M.A.E.)--University of Florida, 2011.
Local: Adviser: Miller, M David.
Electronic Access: RESTRICTED TO UF STUDENTS, STAFF, FACULTY, AND ON-CAMPUS USE UNTIL 2012-04-30

Record Information

Source Institution: UFRGP
Rights Management: Applicable rights reserved.
Classification: lcc - LD1780 2011
System ID: UFE0043017:00001


This item has the following downloads:


Full Text

PAGE 1

1 ADDITIVE MODELS FOR MULTITRAIT MULT I METHOD DAT A ASSUMING A MULTIPLICATIVE TRA I T METHOD RELATIONSHIP: A SIMULATION STUDY By LIDONG ZHANG A THESIS PRESENTED TO THE GRADUATE SCHOOL OF THE UNIVERSITY OF FLORIDA IN PARTIAL FULFIL LMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF ARTS IN EDUCATION UNIVERSITY OF FLORIDA 2011

PAGE 2

2 2011 Lidong Zhang

PAGE 3

3 To Ruowen and Yalan for endless love and encouragement

PAGE 4

4 ACKNOWLEDGMENTS I would like to express my sincere gratitude to my advisor Dr. Miller, who emphasized the importance of multitrait multimethod matrix in his measurement class and inspired my interest to explore more about this topic. I would also like to thank Dr. Miller for supporting and guiding me ever since I enter ed this program. My sincere gratitude also goes to Dr. Algina and Dr. Leite for their great help for me to turn the idea into real research. I would also like to thank Jin Rong for important technical support and warm encouragement during the research proc ess. Thank s to my whole family for lending me moral support all the time.

PAGE 5

5 TABLE OF CONTENTS page ACKNOWLEDGMENTS ................................ ................................ ................................ ............... 4 LIST OF TABLES ................................ ................................ ................................ ........................... 7 LIST OF ABBREVIATIONS ................................ ................................ ................................ .......... 8 ABSTRACT ................................ ................................ ................................ ................................ ..... 9 CHAPTER 1 INTRODUCTION ................................ ................................ ................................ .................. 10 1.1 Purpose of the Study ................................ ................................ ................................ ......... 11 1.2 Research Questions ................................ ................................ ................................ ........... 11 2 LITERAT U R E REVIEW ................................ ................................ ................................ ....... 12 2.1 The CT CM Model for MTMM Data ................................ ................................ ............... 12 2.2 The CT CU Model for MTMM Data ................................ ................................ ............... 13 2. 3 The Composite Direct Product Model ................................ ................................ .............. 14 2.4 Performance of the three models for MTMM Data ................................ .......................... 15 3 METHOD ................................ ................................ ................................ ............................... 20 3.1 Factors Manipulated ................................ ................................ ................................ ......... 20 3.1.1 Sample Size ................................ ................................ ................................ ............ 20 3.1.2 Matrix Size ................................ ................................ ................................ ............. 21 3.1.3 Trait Factor Correlation and Method Factor Correlation ................................ ....... 21 3.1.4 Other Matrix Characteristics ................................ ................................ .................. 21 3.2 Data Generation and Analysis ................................ ................................ .......................... 22 4 RESULTS ................................ ................................ ................................ ............................... 25 4.1 Convergence and Proper Solutions ................................ ................................ ................... 25 4.2 Model Fit ................................ ................................ ................................ .......................... 26 4.3 Parameter Estimates ................................ ................................ ................................ .......... 26 5 DISCUSSION AND CONCLUSION ................................ ................................ .................... 43 5.1 Limitations and Suggestions for Future Research ................................ ............................ 47 5.2 Concluding Remarks ................................ ................................ ................................ ........ 48 LIST OF REFERENCES ................................ ................................ ................................ ............... 49

PAGE 6

6 BIOGRAPHICAL SKETCH ................................ ................................ ................................ ......... 52

PAGE 7

7 LIST OF TABLES Table page 3 1 P opulation parameter values for current simulation study ................................ ................ 24 4 1 Convergence rates and proper solution rates for the CDP model ................................ ...... 32 4 2 Convergence rates and proper solution rates for CT CU and CT CM models .................. 33 4 3 Range of goodness of fit indices ................................ ................................ ........................ 34 4 4 Chi sq uare statistics, degrees of freedom, and p values for three models in 4T3M case ................................ ................................ ................................ ................................ ..... 35 4 5 Chi square statistics, degrees of freedom, and p values for three models in 4T3M case ................................ ................................ ................................ ................................ ..... 36 4 7 Summary of relative bias of trait factor correlation estimates for CT CU model ............. 39 4 8 Summary of relative bias of trait factor correlation e stimates for CT CM model ............. 40 4 9 Summary of relative bias of method factor correlation estimates for CDP model ............ 41 4 10 Summary of relative bias of method factor correlation estimates for CT CM model ...... 42

PAGE 8

8 LIST OF ABBREVIATION S ANOVA Anal yses of variance CDP Composite direct product CFA Confirmatory factor analysis CFI Comparative fit index C.R Converg ence rate CT CM Correlated traits correlated methods CT C(M 1) Correlated traits correlated methods minus one CT CU Correlated traits correlated uniqueness DP Direct product 4T3M Four traits by three methods M Method MTMM Multitrait multimethod MTMO Multi trait multioccasion MTMR Multitrait multirater N Sample size RMSEA Root Mean Squared Error of Approximation S.R Proper solution rate SRMR Standardized root mean squared residual T Trait TLI Tucker Lewis Index TMU Trait method unit 3T3M Three traits by thre e methods

PAGE 9

9 Abstract of Thesis Presented to the Graduate School of the University of Florida in Partial Fulfillment of the Requirements for the Degree of Master of Arts in Education ADDITIVE MODELS FOR MULTITRAIT MULTIMETHOD DATA ASSUMING A MULTIPLICATIVE TRAIT METHOD RELATIONSHIP: A SIMULATION STUDY By Lidong Zhang Ma y 2011 Chair: David Miller Major: Research and Evaluation M ethodology Confirmatory factor analysis (CFA) is widely used for analyzing multitrait multimethod (MTMM) data. But there is no consensus about whether the multiplicative or the additive trait method effect of its parameterization, most appropriately represents the underlying structure of MTMM data. The popularity of the two additive CFA models, the correlated traits c orrelated methods (CT CM) model and the correlated traits correlated uniqueness (CT CU) model, calls for investigation of their performance to characterize MTMM data that conforms to a multiplicative trait method effect model. A Simulation study was used t o compare the model fit and parameter estimates of the CT CM and CT CU models for MTMM data with that of a multiplicative CFA model, assuming a multiplicative trait method relationship. The results showed that CT CU works well in most conditions and is qui te robust to MTMM data with multiplicative underlying trait method effect when the matrix size is 3T3M.

PAGE 10

10 CHAPTER 1 INTRODUCTION Since Campbell and Fiske (1959) proposed using the multitrait multimethod (MTMM) matrix as a framework to establish convergent and discriminant validity, this methodology has had a far reaching impact on the process of evaluating construct validity of educational and psychological measures. In the MTMM framework, convergent validity is the degree to which multiple attempts to meas ure the same trait are in agreement. Discriminant validity is the degree to which measures of different constructs are distinct. Four descriptive criteria to evaluate the construct validity were offered by Campbell and Fiske. During the past decades, diffe rent quantitative methods have been developed to analyze MTMM data, and confirmatory factor analysis (CFA; Kenny, 1976) has emerged as the most generally applied approach, and it is most amework of the MTMM matrix (Kenny & Kashy, 1992). Among the variants of CFA model for MTMM data, there are two classes of approaches depending on the assumption of the underlying trait method effect on measured variables: one class assumes additive and lin ear trait method effects, the other assumes multiplicative trait method effects. The correlated traits correlated methods model (CT CM, Widaman, 1985), and the correlated traits correlated uniqueness model (CT CU, Marsh, 1989) are two widely applied additi ve CFA approaches to analyze MTMM data, while the composite direct product (CDP, Browne, 1984) is the multiplicative model for MTMM data. Research has been conducted to compare the CT CM, CT CU and CDP approaches in terms of convergence, admissibility, and overall fit. However, there is no agreement on which CFA model is the most appropriate representation of the underlying structure of MTMM data (Conway, Lievens, Scullen & Lance, 2004). Some researchers found that whenever the multiplicative model fits the data, a certain type of additive model also fits the data (Kumar & Dillon, 1992; Becker & Cote, 1994;

PAGE 11

11 Hern ndez & Gonz lez Rom 2002). Thus, they suspected that when a multiplicative model fits the data adequately some appropriate additive model can also be found that fits the data adequately, and recommended further research on this phenomenon. 1.1 Purpose of the Study Based on the findings of previous research, the current research aims to use a Monte Carlo simulatio n study to investigate whether the two additive CFA models provide appropriate model fit and parameter estimation for MTMM data that conform to a multiplicative trait method effect model. The performance of the two additive models for multiplicative MTMM d ata will be compared with that of the multiplicative model. 1.2 Research Questions The research questions addressed in this study are: Do the two additive CFA models have adequate fit for the MTMM data that conform to a multiplicative trait method effect model? Do the two additive CFA models have acceptable parameter estimates (i.e., trait factor correlations and method factor correlations) for the MTMM data that conform to a multiplicative trait method effect model?

PAGE 12

12 CHAPTER 2 LITERAT U R E REVIEW The liter ature review chapter first presents the theoretical framework of the three CFA models for analyzing the MTMM data. Some of their advantages and disadvantages for analyzing the MTMM data are briefly introduced. The contradictory findings from previous studi es regarding the performance of the three models for the MTMM data are summarized. Finally, the need of a study on the additive CFA models for analyzing MTMM data conforming to a multiplicative underlying structure is specified. 2.1 The CT CM Model for MTM M Data Among the various parameterizations of the CFA model, CT CM is identified as the most popular approach for MTMM data (Tom s, Hontangas & Oliver, 2000). Within the CT CM model, there are at least three traits and three methods. Trait fact ors and method factors are separated. Correlations among the trait factors and correlations among the method factors are allowed, but method factors are usually assumed to be independent of trait factors (Widaman, 1985); moreover, for each trait method uni t, there is only one indicator, and uniquenesses are assumed to be uncorrelated among themselves. Based on the rationale of this model, relatively high loadings on trait factors would indicate convergent validity, high loadings on the method factors would indicate common method effects, and moderate correlations among the trait factors would suggest discriminant validity. The model equation for CT CM could be expressed as follows: (2 1) indicates the ij th trait method unit, an observed variable representing the Trait measured by the Method; is the factor loading of on Trait and is the factor

PAGE 13

13 loading on Method ; the is the residual consisting of systematic effects of a particular measure and nonsystematic effects of random measurement error on the (Lance, Noble & Scullen, 2002). 2.2 The CT CU Model for MTMM Data Despite its popularity, the CT CM model suffers from frequent nonconvergence and improper solutions, and it may be empi rically under identified under some conditions (Kenny and Kashy, 1992). To address these issues, Marsh (1989) proposed the correlated trait correlated uniqueness (CT CU) model for MT features measurement er ror correlations among indicators based on the same method instead of separate method factors. The CT CU model equation is (2 2) still represents the factor loadings on the Trait However, is a combination of several specific effects (i.e., where is a systematic residual unique to a particular measure, refers to random error, and refers to the unmeasured systematic method effect on ). The advantage of the CT CU model over CT CM model is that it yields fewer non convergent and im proper solutions. However, it also has some weaknesses. The undifferentiated effects in make the method effects confounded with other systematic errors. In addition, the method factor correlations are set to zero in CT CU model, which potentially might cause bias in the estimation of trait factor loadings and trait factor correlations (Lance et al., 2002; Conway et al., 2004). Regardless of the strength and weakness of the CT CM, CT CU models, they all assume linear and additive T rait and Method effects on that is, the magnitude of method effects

PAGE 14

14 after examining a large number of MTMM matric es, pointed out that the method effects were multiplicative rather than additive. If so, the Trait and Method interaction will determine the common variance in measures rather than the sum of them. Consequently, the linear and addi tive models would be inappropriate for data of this nature. 2.3 The Composite Direct Product Model product model for MTMM data to implement the multiplicative structure o bserved by Campbell direct product of a covariance matrix of methods and a covariance matrix of traits. Browne osing a composite direct product model (CDP) for MTMM data; this model also expresses the covariance matrix of observed measurements as the direct product of a covariance matrix of methods and a covariance matrix of traits, and allows for measurement error s and different scales of measurement among the observed variables. The mathematical representation of the direct product model is (2 3) is the tm tm diagonal matrix of true score standard deviations (scale constants, some of which are set equal to unity to achieve identification), with typical diagonal elements is th e diagonal matrix of uniqueness indicating the amount of unique variances, with diagonal elements and are the Method corre lation matrix and the Trait correlation matrix respectively, whose elements are particular multiplicative components of true or common score

PAGE 15

15 correlations; ct. As in the CFA model, the CDP model assumes that the traits factors are uncorrelated with methods factors. Additional constraints could be imposed on the CDP model. One type of constrained CDP model restricts such that: (2 4) where and are defined as before, while the and are diagonal constituent matrices of with diagonal elements and respectively. The effect of imposing this type of constrained CDP model is that trait communality indice s have the same rank order for each method (Browne, 1992). Browne (1992) has developed a computer program, MUTMUM, to estimate the parameters in the CDP model. 2.4 Performance of the three models for MTMM Data Research findings on the performance of the ab ove three models for analyzing MTMM data are inconsistent. Several simulation studies (Marsh & Bailey, 1991; Tom s, Hontangas, & Oliver, 2000; Co nway, Lievens Scullen & Lance 2004) compared the performance of the CT CU and the CT CM models wi th additive MTMM data. In Marsh & Bailey (1991), they simulated data with an underlying structure of both the CT CU and the CT CM models. For data simulated with CT CU structure, the CT CU model out performed the CT CM model in terms of number of proper so lutions and accuracy of parameter estimates. For data simulated with a CT CM structure, the CT CU model also yielded more proper solutions and more precise parameter estimates considering only proper solutions. They pointed out that when the matrix size an d sample size were small, the CT CM model had high ill definition rates, while the CT CM model performed better when the MTMM matrix size and sample size were larger. Tom s, Hontangas, & Oliver (2000) simulated MTMM data with a CT CM structure, and varied the number of

PAGE 16

16 indicators per trait method combination. They found that in the case of single indicator per trait method combination, CT CU performed very well for this design, and performed consistently better than the CT CM model. The CT CM m odel only worked well and better than the CT CU model for designs with more than two indicators per trait method combination Co nway, Lievens Scullen & Lance 2004 ) simulated data with a CT CM model structure, they detected the bias with CT CU model when population method factor correlations and method factor loadings were large. They concluded that when both method loadings and method correlations were large, CT CU model tended to produce biased estimates in trait factor loadings and correlations. Their a nalyses with CT CM also showed that with small sample size and matrix size (e.g., sample size = 125, 3T3M) the CT CM had very low proper solution rate (6%), while it worked better for larger matrix size and sample size. Some researchers compared CT CM and CDP model. Goffin & Jackson (1992) reported that for four multitrait multirater performance appraisal data, which conformed to the identification criteria of at least three traits and three methods, CDP model was superior to CT CM model for all four multit rait multirater matrices in terms of convergence to proper solutions and fit indices. The CT CM model failed to converge to proper solutions or was empirically under identified for all four matrices. Coovert, Craiger & Teachout (1997) also asserted that CD P model provided a better representation of the data than the CTCM model. Bagozzi & Yi (1990) examined eleven MTMM matrices on affect and perceptions at work, the CT CM model fit nine of the eleven MTMM data and CDP model fit one of the remaining two MTMM data that the CT (1977) study. Thus, they concluded that either additive or multiplicative method effects could be used to explain MTMM data, but not both of them. They su ggested using simulation study to

PAGE 17

17 compare the performance of the two models over a range of relevant factors and detected the consequences of model misusing. Bagozzi & Yi (1991) applied the CDP and CT CM models to four MTMM data sets on consumer behavior, they found that CDP model fit two of the MTMM data and the CT CM model fit none. Thus, they contended that methods often have multiplicative effects which supported the CDP model. Bagozzi, Yi & Phillips (1991) inspected four MTMM matrices in organizational research and found that CT CM model fit two of the data sets while CDP model fit one of the other two data sets. There is no data set that both of the models fit, then they concluded that the CDP model could address the inability of CT CM model to detect the interactions between the traits and methods. They also indicated that whenever the CDP model fits the data, the method effects are multiplicative, and if CT CM model fits the data, the method effects are additive. After a conceptual and analytical exam ination of the additive models (CT CM and its variations), multiplicative models (DP and CDP models), and covariance components analysis method, Kumar & Dillon (1992) concluded that it is possible that both the CT CM and CDP models can fit the same MTMM da ta set, which is contrary to the conclusi ons of Bagozzi & Yi (1990) and Bagozzi, Yi & Phillips (1991). More recent studies have compared the CT CM, CT CU and CDP models altogether. Byrne & Goffin (1993) applied the three models to one MTMM covariance matri x, the CT CM and CT CU models had better goodness of fit indices than CDP models. Focusing only on the degree to which method or trait effects are prevalent, all three models provided quite similar conclusions. Nevertheless, the three models yielded very d ifferent results regarding individual parameter estimates, evidence of discriminant validity and depth of information provided. The authors suggested using a larger number of MTMM matrices to investigate this issue. Becker & Cote (1994) looked at seventeen MTMM matrices, and the CT CU model turned out to be more

PAGE 18

18 effective in general. If the CT CM model has adequate fit for the data, the CT CU model and the CDP model also fit the same data well. Moreover, whenever the CDP model provides a good fit and proper solution, the CT CU model also yields a proper solution. However, when the CDP model fit the data, the CT CM model did not. They argued that whenever a multiplicative model fits the MTMM data some appropriate additive model can also be found fit the same data. Conway (1996) examined 20 multitrait multirater (MTMR) performance appraisal matrices and CU model appeared to be the most effective model of the three models for these MTMR data, it g enerated appropriate solutions for 16 of the 20 MTMR data. While CT CM was appropriated for only one matrix and CDP model was appropriate for three of these matrices. Whenever the CT CM or CDP model produced a proper solution, so did the CT CU model. They also investigated the extent to which the three models produce similar results and found that the three models yielded similar parameter estimates considering only proper solutions. Hern ndez & Gonz lez Rom (2002) inves tigated whether empirical multitrait multioccasion (MTMO) data conform more closely to multiplicative models than to additive models. Twenty one MTMO matrices were analyzed, CDP and CT CU models converged to proper solutions with satisfactory goodness of f it indices much frequent than CT CM model. CT CU model turned out to be the best additive model in terms of both convergence and goodness of fit. CT CU and CDP models usually fit the same MTMO data, and general conclusions about construct validity are simi lar across the two models. However, number of traits and occasions will have substantive impact on conclusions about the relevance of each trait and occasion. Therefore, they suggest further simulation study to explore the effect of number of traits and oc casions and other factors such as sample size and correlations in obtaining conclusions across models.

PAGE 19

19 Since the underlying trait and method relationship for the MTMM data (i.e., additive or multiplicative) is unknown, given the popularity of the CT CM an d CT CU models, the current study explored their ability for analyzing MTMM data assuming a multiplicative relationship between trait and method factors.

PAGE 20

20 CHAPTER 3 METHOD To test the relative robustness of the two additive models (i.e., CT CM and CT CU ) we simulated MTMM data with an underlying multiplicative trait method effect. Then, we fit the CDP, CT CM, CT CU models to the simulated data. We compared these models with respect to fit indices, trait factor correlations, and method factor correlations (except for the CT CU model). The results of CDP model were used as a baseline to evaluate how well the additive models characterize the multiplicative MTMM data. 3.1 Factors Manipulated We chose population values for the parameters of the simulated data based on the descriptive statistics from review of published MTMM analyses performed by Co nway, Lievens Scullen & Lance (2004). We manipulated the following conditions: (1) sample size, with three levels: 125, 261, 500; (2) matrix size, either 3 traits an d 3 methods or 4 traits and 3 methods; (3) Trait factor correlation, with three levels: .28, .44, and .62; (4) Method factor correlation, with three levels: .16, .31, and .49. The manipulated factors resulted in a simulation design with 54 unique condition s. 3.1.1 Sample Size We selected three levels for the sample size: 125, 261, and 500. The two smaller sample sizes we selected, (i.e., 125, and 261) correspond to the median and 75 th percentile values from Co nway, Lievens Scullen & Lance (2004) review. We also added a larger sample size of 500, which has been commonly included in MTMM simulation studies (Marsh & Bailey, 1991; Tom s Hontangas & Oliver 2000; Conway, Lievens Scullen & Lance 2004).

PAGE 21

21 3.1.2 Matrix Size The matrix size has two l evels, 3 traits by 3 methods (3T3M), and 4 traits by 3 methods (4T3M). We chose to simulate MTMM matrices with three methods because this was the median number of methods identified in Co nway, Lievens Scullen & Lance (2004) review. We chose to simulate MTMM matrices with three and four traits because these corresponded to the 25 th percentile and median number of traits in Co nway, Lievens Scullen & Lance (2004) review. 3.1.3 Trait Factor Correlation and Method Factor Correlation We chose three levels for trait factor correlation: .28, .44, and .62; we also chose three levels for method factor correlation: .16, .31, and .49. Three levels of trait factor correlations and three levels of method factor correlations were manipulated to represent high, mediu m and low levels of correlations. In Co nway, Lievens Scullen & Lance (2004) review, the values for the trait factor correlation were calculated by averaging across all values within a study (i.e., the mean trait factor correlation for a study), a simila r approach was used for the method factor correlation. In current study, we reviewed the values for trait factor correlations and method factor correlations that appeared in empirical studies adopting the CDP model, and selected those values which average to the 25 th percentile, median, and 75 th percentile values in Co nway, Lievens Scullen & Lance (2004) review of trait factor correlations and method factor correlations. 3.1.4 Other Matrix Characteristics After we performed the manipulations described above, we held constant other matrix characteristics, (i.e., and ). According to Browne (1984), is the diagonal matrix of true score standard deviations, with diagonal elements, is the diagonal matrix of uniqueness, with diagonal elements, represent ratios of unique score

PAGE 22

22 variances to common score variances. Therefore, the relationship between and is as follows: = (3 5) If the CDP model is fitted to a correlation matrix, then the diagonal element, of is equal to the communality index, = = (3 6) The communality index is the correlation coefficient between the observed score and the common score communality index is the reliability (or the communality interpreted in the sense of factor analysis). We decided to use a reliability value of 0.65, which is common in empirical studi es. Thus, all diagonal elements of were fixed to the square root of 0.65 in the current simulation. The parameter values for our simulation were summarized in Table 3 1. 3.2 Data Generation and Analysis First, the population MTMM equation (2 3). For each condition, the matrix equation (2 3) of the CDP model with the parameters filled in were calculated with the software R 2.10.1 (R Development Core Team, 2010) to get the popu lation MTMM correlation matrix. Then, the obtained population MTMM correlation matrix was used in an R function to generate multivariate normally distributed random numbers representing the scores of all individuals on each measurement. We generated and an alyzed an initial set of 1,000 datasets for each condition to calculate convergence rates and proper solution rates, then simulated and analyzed additional datasets to obtain 1,000 converged analyses for each condition.

PAGE 23

23 Mplus 6.0 (Muth n & Muth n, 2010) was used to fit the CT CM, CT CU and CDP models to each generated data set with maximum likelihood estimation. The three models were compared with respect to model fit, convergence rates, frequency of improper solutions, relative bias in the parameter estimates of the trait factor correlations and method factor correlations (except for the CT CU model). The formula to calculate the relative bias is (Hoogland & Boomsma, 1998): B ( = (3 7) where B ( is the relative bias of each parameter estimate in each replication r of each condition c given the generating population parameter If the absolute value of the rel ative bias B ( <0.05, then the parameter estimate is considered to be acceptable for that condition ( Hoogland & Boomsma, 1998). In current study, model fit of all three models was examined through chi square test statistics, p va lue, degree of freedom, and goodness of fit indices. The fit indices explored were Comparative Fit Index (CFI), Tucker Lewis Index (TLI), Standardized Root Mean Squared Residual (SRMR), and Root Mean Squared Error of Approximation (RMSEA). These indices ar e widely used in applied research settings to judge the model data fit. We sued the cut off criteria Analyses of variance (ANOVA) were emp loyed to detect the effect of the conditions on the magnitude of the relative bias of the parameter estimates. The dependent variable of these ANOVAs was the relative bias estimates for all conditions with each of the three MTMM model, and the independent variables were trait factor correlation, method factor correlation, sample

PAGE 24

24 size, matrix size. Main effects and two way interactions were examined. We used the eta squared (Cohen, 1973) as a measure of the effect size of each condition. The same 1000 data s ets were applied by each of the three models to compute their respective convergence rate, and based on the converged results of the 1000 replications, the number or proper solutions for each model was documented. The parameter estimates and model fit indi ces were obtained by averaging across 1000 converged replications for each of the three models. Table 3 1. Population parameter values for current simulation study Sample Size 125, 261, 500 Matrix Size 3T3M, 4T3M Method Factor Correlation (Three Methods) Low = 0.16, (0.09, 0.213, 0.20) Medium = 0.31, (0.23, 0.198, 0.505) High = 0.49, (0.395, 0.508, 0.59) Trait Factor Correlation (Three Traits) Low = 0.28, (0.256, 0.286, 0.32) Medium = 0.44, (0.42, 0.449, 0.461) High = 0.62, (0.685, 0.664, 0.513) Trait Factor Correlation (Four Traits) Low = 0.28, (0.256, 0.286, 0.34, 0.513, 0.173, 0.461) Medium = 0.44, (0.32, 0.616, 0.449, 0.386, 0.36, 0.513) High = 0.62, (0.72, 0.607, 0.34, 0.733, 0.685, 0.66) True Score Standard Deviation Unique Variance =

PAGE 25

25 CHAPTER 4 RESULTS 4.1 Convergence and Proper Solutions The same 1000 datasets for each condition were analyzed with the three models to calculate the convergence rate and number of proper solutions. The convergence rates and n umber of proper solutions were lower with sample size 125 than with 261 and 500 for all three models. Unexpectedly, although the data were simulated assuming a multiplicative trait and method effect, the convergence rates for analyses with the CDP model we re not uniformly high over all conditions. The convergence rate was comparatively higher for the CDP model under conditions of larger sample size, however, under either low trait factor correlation or low method factor correlation, or both of them, the con vergence rates for the CDP models were comparatively low. Table 4 1 presents the proper solution rates and convergence rates for the CDP model under all conditions. With 3T3M matrices, the lowest convergence rate was 60.1%, which occurred when both trait f actor correlation and method factor correlation were low, and the sample size was small. As trait factor correlation and method factor correlations increased, the convergence rates for CDP model increased accordingly. All converged analyses with the CDP mo del were proper solutions. Table 4 1 shows that the proper solution rates for the CDP model were all 100%, and were the highest among the three models. Table 4 2 shows the proper solution and convergence rates for both the CT CU model and the CT CM model. The pattern of convergence rates across all conditions for the CT CU model was heavily influenced by the magnitude of method factor correlations. Under conditions with low method factor correlation, the convergence and proper solution rates for analyses wi th the CT CU model were relative low, especially when the sample size is small. For instance, in the 3T3M case, when the sample size was 125 and both trait factor and method factor correlations

PAGE 26

26 were low, the convergence rate was .366 and the proper solutio n rate was .42. However, the convergence rate increased to .748 and the proper solution rate increased to .57 in the condition with sample size of 125, low trait factor correlation and medium method factor correlation. Moreover, the convergence rate became .986 and the proper solution was up to .84 with the condition of low trait factor correlation and high method factor correlation. Similar trend was found with the 4T3M matrix size. From Table s 4 1 and 4 2 we can see that among the three models, the analy ses with the CT CM model had the lowest convergence rates and proper solution rates across all combination of conditions. This model produced more inadmissible solutions, which contained negative residual variances and/or inaccurate standard errors due to non positive definite first derivative product matrices. Like the CDP model, both the low trait factor correlation and low method factor correlation conditions, tend to produce comparatively lower convergence rate and proper solution rate for CTCM model. 4.2 Model Fit We examined the chi square statistics, degree of freedom, p values, and goodness of fit of all three models based on 1,000 converged solutions for each of them. We obtained the striking result that the three models met the criteria for close fi t based on the fit indices ( Table 4 3) examined across all combination of conditions. The p values showed that the chi square statistics were non significant for all analys es across the three models ( Table 4 4, Table 4 5) which indicated that all three m odels had exact fit across all combination of conditions. 4.3 Parameter Estimates T here are two types of parameter estimates in current study: estimates of trait factor correlations and estimates of method factor correlations. The CDP, CT CU, and CT CM all

PAGE 27

27 produced estimates of trait factor correlations, but only the CDP and CT CM had estimates of method factor correlations. To detect the accuracy of the parameter estimates, the CDP, CT CU and CT CM models were compared with regard to the relative bias of t he trait factor correlation estimates for various conditions. The CDP and CT CM models were compared with regard to the relative bias of the method factor correlation estimates for various conditions. ANOVAs were conducted for the relative bias of the two types of parameter estimates to investigate which factors had an effect on the relative bias. We conducted separate ANOVAs for the 3T3M and 4T3M to explore which factors had more influence on the relative bias of the trait factor correlation estimates with CDP models. For these ANOVAs, the dependent variable was the relative bias of the trait factor correlation estimates. The independent variables were trait factor correlation, method factor correlation, and sample size. We also combined across the trait fa ctor correlation estimates (e.g., T21, T31, T32) six levels for the 4T3M model, to check whether the relative bias was different between trait factor correlations Only main effects and two way interactions were examined. Because there were no main effects of the traits factor or interactions involving it, we then combined the two r but with a matrix size factor, with two levels (i.e., 3T3M and 4T3M). The rank order of the for these factors or interactions that had a substantial effect on the relative bias of trait factor correlations were: method factor correlation ( = 0.29), method factor correlation sample size ( 0.12), sample size ( 0.11), trait factor correlation ( 0.10), trait factor correlation method factor correlation interaction ( 0.09), matri x size ( 0.036). Conditions with low level of population method factor correlation tended to produce more biased trait factor

PAGE 28

28 correlation estimates, especially when the population trait factor correlation was simultaneously low; conditions wit h medium or high level of population method factor correlation turned out to yield more accurate trait factor correlation estimates. Larger sample size or matrix size allowed more accurate trait factor correlation estimates than smaller sample size or matr ix size. Therefore, the largest bias in trait factor correlation estimates emerged when the population method and trait factor correlations were both low, with matrix size 3T3M and sample size 125. Medium or higher level of method and trait factor correlat ion tended to produce more accurate trait factor correlation estimates. Table 4 6 presents the summary of relative biases of the trait factor correlation estimates for CDP model. We conducted separate and overall ANOVAs for the 3T3M and 4T3M to e xplore which factors had more influence on the relative bias of the trait factor correlation estimates with CT CU model, with ANOVA models identical to those used with CDP model. We found that the ve bias, so we only report the results collapsing across traits. For the overall ANOVA, the rank order of the for the factors or interactions that had a substantial effect on the estimates accuracy of the CT CU model were: method factor correla tion ( = 0.15), matrix size ( = 0.073), method correlation matrix size interaction ( = 0.07), sample size matrix size interaction ( = 0.05). Population method factor correlations had the largest effect on the quality of trait factor correlation estimates with CT CU model. In particular, conditions with low level of population method factor correlation returned more biased trait factor correlation estimates, conditions with medium or high level of populatio n method factor correlation allowed more accurate trait factor correlation estimates. Matrix size was another major effect on trait factor correlation estimates with CT CU model. From what we observed, the trait factor correlation estimates with 3T3M CT CU model were

PAGE 29

29 more accurate, and there were fewer unacceptable relative biases than with the 4T3M CT CU model. Furthermore, the CT CU model outperformed CDP model in terms of the accuracy of trait factor correlation estimates when the matrix size was 3T3M. T he separate ANOVA tests for 3T3M and 4T3M CT CU models indicated that the method factor correlation sample size interaction had the largest effect on the accuracy of trait factor correlation estimates in the 3T3M case, while the method factor correlation had the largest effect in the 4T3M case. Nevertheless, the CT CU model produced less accurate trait factor correlation estimates with the larger 4T3M model. Mor eover, although sample size matrix size interaction had an of 0.05 in the overall ANOVA test for CT CU model, it did not correspond to interpretable differences in relative bias. Table 4 7 presents the summary of relative biases of the trait factor correlation estimates for CT CU model. Separate ANOVAs were used for the CT explore which factors had more influence on the relative bias of the trait factor correlation estimates. The ANOVA for 4T3M mat rix showed that the relative bias was different between trait factor correlations, therefore, we did not conducted an overall ANOVA for CT CM model. In contrast to the good fit indices we showed in Table 4 3, which indicated that the CT CM model fit the da ta well, all the relat ive biases in analyses with CTCM mode l had unacceptable large values. ANOVA results for 3T3M CT CM model showed that trait factor correlation ( =0.527), sample size ( = 0.179), trait factor correlation sample size interaction ( 0.094) had relatively larger .The most striking effect associated with trait factor correlation and sample size: from Table 4 8 we can see that very large negative biases occurred when the trait factor correlati on was low or medium. Unlike the 3T3M CDP and CTCU models,

PAGE 30

30 larger sample size in the 3T3M CTCM model did not correspond to more accurate estimation of the trait factor correlation; on the contrary, other conditions being equal, larger sample sizes were ass ociated with larger relative bias of trait factor correlation estimates. We conducted separate ANOVAs for each trait factor combination in the 4T3M matrix, and the results showed that the majority of these trait factor correlations were heavily influenced by the interaction between the trait factor correlation and method factor correlation. Table 4 8 presents the summary of relative biases of the trait factor correlation estimates for CT CM model. We conducted separate ANOVAs for the 3T3M and 4T3M to explor e which factors had more influence on the relative bias of the method factor correlation estimates with CDP models. For these ANOVAs, the dependent variable was the relative bias of the method factor correlation estimates. The independent variables were tr ait factor correlation, method factor correlation, and sample size. We also combined across the method factor correlation estimates (e.g., M21, M31, model and the 4T3 M model, to check whether the relative bias was different between method factor correlations. Only main effects and two way interactions were examined. Because we x matrix size factor which had two levels (i.e., 3T3M and 4T3M). The results showed that trait factor correlation ( =0.192), trait factor correlation met hod factor correlation interaction ( = 0.127), method factor correlation ( = 0.088), matrix size ( 0.076), method factor correlation sample size interaction ( 0.075), sample size ( 0.069), trai t factor correlation sample size interaction ( 0.068) had relatively larger We found that lower levels of trait factor and method factor correlations combined with smaller sample size tended to

PAGE 31

31 correspond to positively biased e stimates of the method factor correlation by CDP model. Matrix size also played an important role on the accuracy of method factor correlation estimates by CDP model. There is a tendency that larger matrix size condition tended to produce more accurate met hod factor correlation estimates by CDP model. As we observed in Table 4 9, the proportion of unacceptable relative bias in method factor correlation estimates by 4T3M CDP model was less than that by the 3T3M CDP model (6% vs. 20%). We used the same ANOVAs described above for the CT CM model to detect which factors affected the relative bias of the method factor correlation estimates. All the relative biases of the method factor correlation estimates with CT CM model had unacceptable negative values. The re sults of the overall ANOVA showed that method factor correlation ( =0.334), matrix size ( = 0.188), method factor correlation matrix size interaction ( 0.129) had relatively larger Across the two matrix size, low method factor correlation tended to return larger relative bias in method factor correlation estimates with the CTCM model, nevertheless, when the matrix size increased from 3T3M to 4T3M, the magnitude of the relative bias in method factor correl ation estimates decreased. Table 4 10 presented the summary of relative bias of method factor correlation estimates for CT CM model.

PAGE 32

32 Table 4 1. Convergence rates and proper solution rates for the CDP model 3T3M 4T3M T M N C.R S.R C.R S.R low low 12 5 0.601 1.000 0.670 1.000 261 0.761 1.000 0.843 1.000 500 0.814 1.000 0.909 1.000 medium 125 0.778 1.000 0.868 1.000 261 0.845 1.000 0.945 1.000 500 0.918 1.000 0.985 1.000 high 125 0.844 1.000 0.927 1.000 261 0.903 1.000 0.986 1.000 500 0.967 1.000 0.999 1.000 medium low 125 0.745 1.000 0.745 1.000 261 0.852 1.000 0.851 1.000 500 0.898 1.000 0.922 1.000 medium 125 0.860 1.000 0.884 1.000 261 0.928 1.000 0.950 1.000 500 0.978 1.000 0.992 1.000 high 125 0.909 1.000 0. 944 1.000 261 0.980 1.000 0.996 1.000 500 0.996 1.000 0.999 1.000 high low 125 0.813 1.000 0.831 1.000 261 0.914 1.000 0.951 1.000 500 0.970 1.000 0.988 1.000 medium 125 0.940 1.000 0.963 1.000 261 0.988 1.000 0.998 1.000 500 0.999 1.0 00 1.000 1.000 high 125 0.974 1.000 0.994 1.000 261 1.000 1.000 1.000 1.000 500 1.000 1.000 1.000 1.000 Note. T = trait factor correlations; M = method factor correlations; N = sample size; C.R = convergence rate; S.R = proper solution rate;

PAGE 33

33 Table 4 2. Convergence rates and proper solution rates for CT CU and CT CM models 3T3M 4T3M CT CU CT CM CT CU CT CM T M N C.R S.R C.R S.R C.R S.R C.R S.R low low 125 0.366 0.415 0.212 0.325 0.393 0.387 0.346 0.48 261 0.577 0.633 0.366 0.43 7 0.605 0.474 0.499 0.451 500 0.820 0.728 0.499 0.545 0.852 0.697 0.671 0.525 medium 125 0.748 0.568 0.317 0.322 0.809 0.460 0.519 0.443 261 0.962 0.785 0.476 0.464 0.978 0.780 0.633 0.480 500 0.996 0.953 0.652 0.569 1.000 0.974 0.752 0.451 hi gh 125 0.986 0.845 0.401 0.329 0.994 0.813 0.643 0.525 261 1.000 0.983 0.525 0.497 1.000 0.981 0.637 0.443 500 1.000 1.000 0.633 0.660 1.000 0.999 0.695 0.442 medium low 125 0.445 0.400 0.278 0.371 0.419 0.315 0.358 0.626 261 0.670 0.616 0.389 0. 501 0.655 0.489 0.406 0.479 500 0.895 0.798 0.538 0.615 0.879 0.745 0.554 0.590 medium 125 0.849 0.598 0.369 0.407 0.856 0.528 0.508 0.705 261 0.987 0.858 0.496 0.474 0.990 0.839 0.571 0.517 500 1.000 0.989 0.662 0.532 1.000 0.985 0.669 0.485 high 125 0.996 0.893 0.476 0.403 0.997 0.837 0.668 0.538 261 1.000 0.988 0.603 0.498 1.000 0.984 0.665 0.516 500 1.000 1.000 0.688 0.705 1.000 1.000 0.683 0.464 high low 125 0.516 0.328 0.328 0.366 0.481 0.208 0.433 0.577 261 0.762 0.642 0.442 0. 439 0.746 0.383 0.516 0.494 500 0.941 0.809 0.567 0.596 0.943 0.636 0.64 0.568 medium 125 0.908 0.617 0.411 0.387 0.910 0.474 0.573 0.723 261 0.991 0.885 0.508 0.484 0.997 0.839 0.650 0.400 500 1.000 0.988 0.641 0.580 1.000 0.985 0.722 0.339 h igh 125 0.996 0.906 0.534 0.401 1.000 0.844 0.658 0.303 261 1.000 0.992 0.582 0.545 1.000 0.992 0.722 0.407 500 1.000 0.989 0.662 0.532 1.000 1.000 0.773 0.520 Note. T = trait factor correlations; M = method factor correlations; N = sample size; C. R = convergence rate; S.R = proper solution rate;

PAGE 34

34 Table 4 3. Range of goodness of fit indices 3T3M Averaged Averaged Averaged R Averaged CDP (0.963 0.997) (0.994 1.025) (0.008 0.023) (0.024 0.056) CT CU (0.985 0.998) (0.996 1.190) (0.00 8 0.023) (0.018 0.041) CT CM (0.998 0.999) (1.008 1.351) (0.000 0.000) (0.014 0.029) 4T3M CDP (0.970 0.997) (0.992 1.005) (0.009 0.016) (0.028 0.064) CT CU (0.986 0.998) (0.991 1.087) (0.006 0.019) (0.020 0.049) CT CM (0.982 0.998) (0.951 1.085) (0.004 0.036) (0.022 0.045) Note a. Comparative Fit Index; b. Tucker Lewis Index ; c. Root Mean Square Error of Approximation; d. Standardized Root Mean Square Residual;

PAGE 35

35 Table 4 4. Chi square statistics, degrees of f reedom, and p values for three models in 4T3M case 3T3M CTCM (df=12) CTCU (df=15) CDP (df=25) N Chi square p Chi square p Chi square p low low 125 6.862 0.824 11.891 0.669 23.884 0.546 261 7.625 0.778 13.016 0.608 23.880 0.542 500 8.504 0.720 13.831 0.563 24.137 0.534 medium 125 7.407 0.788 13.802 0.557 24.241 0.532 261 8.392 0.728 14.709 0.516 24.515 0.521 500 8.807 0.700 14.982 0.502 24.881 0.505 high 125 8.002 0.752 15.227 0.489 25.061 0.499 261 8.311 0.732 15.120 0 .492 24.653 0.516 500 9.248 0.672 14.823 0.509 25.107 0.497 medium low 125 7.233 0.801 12.105 0.658 23.970 0.540 261 7.781 0.766 12.992 0.607 24.352 0.525 500 8.754 0.705 14.183 0.546 24.557 0.520 medium 125 7.999 0.752 14.200 0.541 25.171 0.49 4 261 8.066 0.749 14.880 0.504 25.035 0.504 500 8.952 0.688 14.632 0.517 25.170 0.490 high 125 8.412 0.725 15.136 0.495 24.852 0.503 261 8.720 0.705 15.431 0.481 25.121 0.495 500 9.471 0.657 15.000 0.502 25.396 0.487 high low 125 7.418 0.791 12.065 0.660 23.856 0.547 261 7.612 0.776 13.058 0.603 24.417 0.527 500 8.288 0.734 14.191 0.544 24.636 0.517 medium 125 8.039 0.749 14.587 0.519 25.558 0.481 261 8.422 0.724 14.981 0.500 24.996 0.499 500 8.878 0.695 14.994 0.500 25.514 0.47 5 high 125 8.144 0.744 15.610 0.472 26.349 0.446 261 8.808 0.698 14.881 0.506 25.732 0.472 500 9.126 0.681 14.885 0.507 25.043 0.502 Note df= degrees of freedom p = p value of chi square statistics

PAGE 36

36 Table 4 5. Chi square statistics, degre es of freedom, and p values for three models in 4T3M case 4T3M CTCM (df=33) CTCU (df=30) CDP (df=51) N Chi square p Chi square p Chi square p low low 125 28.945 0.646 25.442 0.671 51.297 0.495 261 35.167 0.422 26.441 0.634 50.682 0.511 500 45.474 0.171 28.399 0.562 50.845 0.508 medium 125 30.407 0.593 29.146 0.531 52.388 0.462 261 35.247 0.424 29.761 0.510 51.893 0.477 500 43.482 0.212 29.652 0.515 50.926 0.503 high 125 30.034 0.601 31.201 0.458 52.433 0.460 261 33.808 0. 474 30.515 0.484 52.358 0.467 500 40.076 0.283 30.157 0.493 51.634 0.484 medium low 125 25.916 0.754 25.447 0.673 51.924 0.477 261 26.997 0.715 26.836 0.617 50.811 0.508 500 29.309 0.631 28.189 0.571 50.894 0.501 medium 125 27.676 0.691 29.152 0.528 53.362 0.434 261 28.785 0.653 29.818 0.508 51.415 0.486 500 30.605 0.585 30.187 0.493 50.732 0.506 high 125 28.291 0.673 30.549 0.484 52.045 0.466 261 29.184 0.637 30.687 0.478 51.772 0.476 500 30.940 0.574 30.227 0.495 51.945 0.475 hi gh low 125 32.173 0.533 25.589 0.666 51.297 0.490 261 40.931 0.259 27.378 0.598 51.260 0.492 500 56.435 0.050 29.063 0.533 51.090 0.498 medium 125 32.156 0.529 29.496 0.516 53.457 0.432 261 40.080 0.280 30.114 0.496 51.675 0.482 500 52.285 0. 088 29.450 0.521 51.588 0.480 high 125 30.520 0.587 30.757 0.474 53.394 0.434 261 36.039 0.398 30.715 0.476 52.552 0.457 500 45.118 0.181 30.441 0.485 51.602 0.481 Note df= degrees of freedom p = p value of chi square statistics.

PAGE 37

37 Table 4 6. Summary of relative bias of trait factor correlation estimates for CDP model 3T3M 4T3M M N T21 T31 T32 T21 T31 T32 T41 T42 T43 low low 125 0.345 0.327 0.361 0.186 0.177 0.128 0.168 0.164 0.133 261 0.186 0.170 0.171 0.098 0.081 0.048 0.096 0.070 0.053 500 0.090 0.089 0.093 0.005 0.015 0.009 0.031 0.018 0.016 medium 125 0.103 0.114 0.112 0.053 0.056 0.024 0.044 0.030 0.043 261 0.034 0.030 0.020 0.008 0.002 0.009 0.010 0.028 0.006 500 0.019 0.019 0.022 0.010 0.008 0.003 0.003 0.004 0.001 high 125 0.023 0.034 0.017 0.013 0.003 0.002 0.011 0.000 0.002 261 0.004 0.007 0.003 0.001 0.010 0.001 0.001 0.012 0.001 500 0.002 0.001 0.004 0.001 0.000 0.001 0.002 0.006 0.004 medium low 125 0.175 0.179 0.189 0.120 0.081 0.102 0.105 0.139 0.084 261 0.076 0.058 0.058 0.035 0.026 0.023 0.035 0.030 0.020 500 0.024 0.027 0.028 0.009 0.005 0.012 0.002 0.016 0.005 medium 125 0.017 0.008 0.012 0.028 0.022 0.021 0.032 0.023 0.027 261 0.010 0.017 0.015 0.018 0.004 0.00 8 0.007 0.005 0.002 500 0.013 0.014 0.009 0.006 0.004 0.004 0.009 0.004 0.003 high 125 0.005 0.002 0.005 0.016 0.010 0.010 0.008 0.009 0.010 261 0.002 0.001 0.005 0.003 0.002 0.001 0.001 0.001 0.000 500 0.001 0.004 0.003 0.009 0.002 0 .008 0.004 0.002 0.004 high low 125 0.077 0.064 0.077 0.039 0.048 0.028 0.075 0.043 0.037 261 0.032 0.030 0.040 0.008 0.016 0.017 0.008 0.009 0.015 500 0.006 0.001 0.006 0.005 0.001 0.002 0.009 0.005 0.005 medium 125 0.011 0.015 0.014 0.007 0.004 0.005 0.011 0.007 0.013

PAGE 38

38 Table 4 6. Continued 261 0.006 0.002 0.001 0.001 0.002 0.004 0.002 0.000 0.005 500 0.004 0.001 0.007 0.002 0.002 0.004 0.001 0.004 0.003 high 125 0.004 0.002 0.007 0.002 0.002 0.003 0.003 0.003 0.001 261 0.002 0.005 0.002 0.003 0.001 0.000 0.003 0.004 0.001 500 0.001 0.001 0.001 0.002 0.001 0.001 0.001 0.003 0.002 Note. T = trait factor correlations; M = method factor correlations; N = sample size; T21 = Trait factor correlation between trait one and trait two; T31 = Trait factor correlation between trait one and trait three; T32 = Trait factor correlation between trait two and trait three; T41 = Trait factor correlation between trait one and trait four; T42 = Trait factor correlation between trait two and trait four; T43 = Trait factor correlation between trait three and trait four;

PAGE 39

39 Table 4 7. Summary of relative bias of trait factor correlation estimates for CT CU model 3T3M 4T3M T M N T21 T31 T32 T21 T31 T32 T41 T42 T43 low low 125 0.053 0.064 0.053 0.128 0.091 0.081 0.191 0.081 0.136 261 0.049 0.066 0.004 0.057 0.158 0.107 0.006 0.270 0.085 500 0.047 0.051 0.049 0.069 0.086 0.051 0.068 0.118 0.06 0 medium 125 0.033 0.011 0.059 0.118 0.137 0.076 0.116 0.102 0.052 261 0.023 0.051 0.047 0.081 0.072 0.036 0.046 0.074 0.053 500 0.039 0.015 0.028 0.009 0.016 0.000 0.019 0.001 0.006 high 125 0.027 0.023 0.036 0.036 0.009 0.018 0.032 0.031 0.013 261 0.008 0.000 0.004 0.012 0.004 0.011 0.014 0.017 0.003 500 0.002 0.001 0.006 0.002 0.011 0.005 0.013 0.006 0.007 medium low 125 0.153 0.006 0.118 0.014 0.099 0.104 0.150 0.063 0.120 261 0.0 75 0.047 0.041 0.114 0.040 0.088 0.115 0.155 0.032 500 0.054 0.055 0.041 0.008 0.041 0.059 0.057 0.016 0.061 medium 125 0.055 0.067 0.042 0.100 0.051 0.121 0.094 0.100 0.071 261 0.025 0.021 0.015 0.026 0.022 0.051 0 .018 0.045 0.028 500 0.007 0.016 0.013 0.019 0.005 0.013 0.006 0.007 0.003 high 125 0.035 0.021 0.015 0.024 0.017 0.053 0.023 0.041 0.023 261 0.002 0.005 0.005 0.001 0.005 0.006 0.008 0.009 0.000 500 0.003 0.006 0.0 02 0.021 0.004 0.007 0.002 0.006 0.003 high low 125 0.079 0.016 0.072 0.040 0.032 0.058 0.126 0.038 0.045 261 0.032 0.029 0.055 0.039 0.031 0.006 0.166 0.030 0.061 500 0.027 0.018 0.038 0.022 0.039 0.021 0.089 0.031 0.0 42 medium 125 0.030 0.027 0.032 0.042 0.048 0.019 0.119 0.043 0.035 261 0.003 0.002 0.006 0.003 0.011 0.000 0.012 0.010 0.021 500 0.006 0.001 0.009 0.004 0.005 0.005 0.015 0.004 0.002 high 125 0.013 0.019 0.013 0.015 0.023 0.004 0.025 0.013 0.001 261 0.002 0.005 0.003 0.003 0.002 0.001 0.015 0.003 0.005 500 0.001 0.003 0.002 0.000 0.001 0.002 0.003 0.001 0.001 Note. T = trait factor correlations; M = method factor correlations; N = sample size ; T21 = Trait factor correlation between trait one and trait two; T31 = Trait factor correlation between trait one and trait three; T32 = Trait factor correlation between trait two and trait three; T41 = Trait factor correlation bet ween trait one and trait four; T42 = Trait factor correlation between trait two and trait four; T43 = Trait factor correlation between trait three and trait four;

PAGE 40

40 Table 4 8. Summary of relative bias of trait factor correlation est imates for CT CM model 3T3M 4T3M T M N T21 T31 T32 T21 T31 T32 T41 T42 T43 low low 125 1.325 0.670 0.523 0.220 0.873 0.178 0.123 1.881 0.338 261 0.900 1.094 0.705 0.278 1.255 0.232 0.282 1.549 0.457 500 1.729 2. 180 1.698 0.269 1.059 0.215 0.103 1.376 0.489 medium 125 1.033 0.946 0.955 0.484 0.955 0.364 0.371 1.537 0.659 261 1.015 1.047 0.905 0.509 0.973 0.404 0.259 1.426 0.663 500 1.429 1.864 1.273 0.378 1.131 0.629 0.181 1 .274 0.782 high 125 0.900 1.069 0.744 0.446 0.749 0.338 0.252 1.096 0.361 261 1.490 1.079 1.482 0.322 0.825 0.320 0.173 0.907 0.535 500 1.546 1.487 2.368 0.325 0.739 0.372 0.133 0.999 0.614 medium low 125 0.480 0.493 0.444 0.688 0.107 0.594 0.594 0.310 0.572 261 0.589 0.675 0.535 0.744 0.394 0.699 0.660 0.332 0.677 500 1.264 1.417 1.091 0.619 0.403 0.696 0.636 0.304 0.647 medium 125 0.604 0.687 0.730 0.786 0.465 0.645 0.821 0.48 2 0.652 261 0.830 0.850 0.682 0.864 0.717 0.651 0.739 0.455 0.735 500 0.881 0.932 1.000 0.710 0.733 0.717 0.671 0.381 0.727 high 125 0.573 0.666 0.545 0.676 0.489 0.503 0.561 0.435 0.538 261 0.718 0.924 0.797 0.87 3 0.783 0.702 0.795 0.425 0.637 500 0.627 0.952 0.990 0.665 0.682 0.640 0.601 0.407 0.992 high low 125 0.217 0.276 0.486 0.269 0.323 0.495 1.751 0.340 0.194 261 0.274 0.360 0.647 0.060 0.341 0.478 1.794 0.319 0.072 500 0.479 0.460 0.588 0.133 0.348 0.746 1.667 0.399 0.128 medium 125 0.501 0.398 0.724 0.289 0.530 0.537 1.264 0.415 0.340 261 0.397 0.473 0.571 0.237 0.429 0.569 1.195 0.434 0.334 500 0.432 0.495 0.576 0.240 0.398 0.552 1.102 0.307 0.273 high 125 0.395 0.368 0.563 0.338 0.425 0.494 1.075 0.378 0.337 261 0.672 0.394 0.521 0.263 0.379 0.519 0.938 0.316 0.265 500 0.469 0.437 0.508 0.259 0.380 0.511 1.045 0.368 0.267 Note. T = t rait factor correlations; M = method factor correlations; N = sample size; T21 = Trait factor correlation between trait one and trait two; T31 = Trait factor correlation between trait one and trait three; T32 = Trait factor correlat ion between trait two and trait three; T41 = Trait factor correlation between trait one and trait four; T42 = Trait factor correlation between trait two and trait four; T43 = Trait factor correlation between trait three and trait four ;

PAGE 41

41 Table 4 9. Summary of relative bias of method factor correlation estimates for CDP model 3T3M 4T3M T M N M21 M31 M32 M21 M31 M32 low low 125 0.333 0.273 0.306 0.114 0.100 0.097 261 0.140 0.125 0.139 0.046 0.033 0.026 500 0.095 0.065 0 .054 0.028 0.010 0.011 medium 125 0.105 0.157 0.082 0.035 0.024 0.014 261 0.037 0.043 0.018 0.004 0.007 0.003 500 0.041 0.028 0.017 0.008 0.002 0.002 high 125 0.046 0.055 0.030 0.000 0.003 0.010 261 0.018 0.014 0.006 0.001 0.003 0. 001 500 0.012 0.000 0.004 0.002 0.003 0.001 medium low 125 0.085 0.123 0.087 0.059 0.060 0.044 261 0.038 0.012 0.015 0.009 0.026 0.010 500 0.015 0.008 0.003 0.006 0.009 0.016 medium 125 0.023 0.046 0.001 0.007 0.017 0.001 261 0.008 0.0 21 0.006 0.009 0.014 0.002 500 0.027 0.013 0.006 0.001 0.006 0.000 high 125 0.015 0.006 0.003 0.001 0.006 0.008 261 0.003 0.001 0.006 0.000 0.004 0.002 500 0.003 0.004 0.000 0.001 0.002 0.001 high low 125 0.084 0.009 0.027 0.004 0. 039 0.027 261 0.052 0.023 0.045 0.025 0.031 0.046 500 0.020 0.014 0.011 0.002 0.016 0.019 medium 125 0.008 0.003 0.007 0.003 0.000 0.014 261 0.009 0.002 0.005 0.004 0.002 0.001 500 0.006 0.004 0.001 0.016 0.003 0.005 high 125 0.001 0.007 0.010 0.013 0.010 0.015 261 0.002 0.003 0.001 0.001 0.002 0.000 500 0.002 0.003 0.002 0.000 0.002 0.000 Note T = trait factor correlations; M = method factor correlations; N = sample size; M21 = Method factor correlation between method one and method two; M31 = Method factor correlation between method one and method three; M23 = Method factor correlation between method two and method three;

PAGE 42

42 Table 4 10. Summary of relative bias of method fa ctor correlation estimates for CT CM model 3T3M 4T3M T M N M12 M13 M23 M21 M31 M32 low low 125 2.237 0.897 1.128 1.128 0.775 0.908 261 3.232 1.842 3.237 1.533 0.787 0.859 500 3.666 4.111 3.009 0.790 0.872 0.850 medium 125 0.511 0.665 0.593 0.578 1.000 0.513 261 0.745 1.330 0.936 0.389 0.663 0.507 500 1.010 0.695 1.135 0.652 0.506 2.139 high 125 0.572 0.548 0.407 0.620 0.502 0.464 261 0.499 0.762 0.642 0.587 0.593 0.666 500 0.614 1.031 0.846 0.503 0.406 0.384 medium low 125 2.768 1.391 1.879 1.007 0.606 0.578 261 2.944 1.203 1.535 1.434 0.503 0.504 500 3.388 3.408 3.411 1.099 1.658 1.753 medium 125 0.748 1.255 0.642 0.462 0.545 0. 394 261 0.838 0.893 2.586 0.441 0.508 0.435 500 0.919 0.711 1.369 0.516 0.678 0.810 high 125 0.517 0.738 0.409 0.507 0.465 0.396 261 0.761 0.620 0.901 0.402 0.504 0.487 500 0.836 0.734 0.921 0.475 0.528 0.823 hi gh low 125 2.041 1.088 1.608 0.763 0.449 0.441 261 2.662 1.318 1.920 0.647 0.654 0.676 500 2.802 2.208 1.855 0.971 0.646 0.846 medium 125 0.490 1.031 0.531 0.418 0.479 0.334 261 0.990 1.028 0.568 0.384 0.441 0.328 500 1.108 0.950 0.910 0.479 0.440 0.387 high 125 1.005 0.478 0.775 0.497 0.457 0.347 261 0.532 0.550 0.572 0.732 0.370 0.341 500 0.612 0.686 0.830 0.372 0.299 0.307 Note T = trait factor correlations; M = method f actor correlations; N = sample size; M21 = Method factor correlation between method one and method two; M31 = Method factor correlation between method one and method three; M23 = Method factor correlation between method two and method t hree;

PAGE 43

43 CHAPTER 5 DISCUSSION AND CONCL USION review of the ranges of correlations that were presented in published studies using the MTMM validation approach. Thus, the co nditions simulated in our study are reasonable representations of the realistic settings of MTMM research. We used the CDP model as a baseline to compare the performance of the CT CU and CT CM models, but found that the CDP model did not perform well in a ll the conditions simulated. more likely to offer adequate fit to data than the additive models. When we analyzed the simulated data with CT CU model, we found tha t if the population method factor correlation is low, the convergence rate and proper solution rate were relatively lower than when the method correlation is of medium or high level. For the low trait factor correlation and low method factor correlation co ndition, the CT CU model had worse convergence rates and proper solution rates than those of CDP model under the same condition. As for the accuracy of trait factor correlation estimates by CT CU model, our analyses showed that population method factor cor relation had the largest effect on it. Specifically, conditions with low level of population method factor correlation yielded more biased trait factor correlation estimates. This finding contradicts with the logical expectation for CT CU models which assu mes that method factor correlations are zero, therefore, the trait factor correlation estimates should be less biased when the population method factor correlation is rather low. CM model and CT CU model and their study showed that when the method correlation was as high as .49, only trivial biases in trait factor correlation estimates were observed for CT CU model; another simulation study which

PAGE 44

44 simulated data using the CT CM model reached a similar con clusion (Tom s, Hontangas, & (2004) simulation study which simulated data using CT CM model, they contended that a large majority of CT CU solutions ex hibiting both admissible parameter estimates and good model fit statistics came from low method correlations conditions. The low method factor correlation condition was commonly regarded as the situation in which the CT CU model is most likely to produce l east biased trait factor correlation estimates. A remarkable finding in our study is that when the matrix size is 3T3M, the CT CU model outperformed the CDP model in terms of accuracy of trait factor correlation estimates. The proportion of biased estimate s was lower with the CT CU model and for the biased estimates under the same conditions (e.g., low trait factor correlation and low method factor correlation), the magnitude of the bias was lower than with the CDP model. Matrix size exhibited a significant effect on the CT CU model, when the population method factor correlation was low: larger matrix size tended to produce more improper solutions for CT CU model and more biased estimates on the trait factor correlation compared with those of the CDP model. Consequently, the parameter estimates were more accurate with the 4T3M CDP model than with the 4T3M CT CU model. Consistent with other MTMM simulation studies (Marsh & Bailey, 1991; Tom s, Hontangas, & Oliver, 2000; Conway et. al. 2004), as sam ple size increased, the frequency of proper solutions and accuracy of trait factor correlation estimates with the CT CU model went up accordingly. Among the three models, the CT CM has the poorest convergence rate and proper solution rate, but they can be improved with larger sample sizes and matrix sizes. The parameter estimates by CT CM model were seriously biased compared with those with the CDP and CT CU models. We suspect that the reason is mostly likely due to the large number of improper

PAGE 45

45 solutions p roduced by the CTCM model. Marsh & Bailey (1991) mentioned that if these improper solutions were considered, then parameter estimates by CT CM model, in particular, trait factor correlation estimates, are much less accurate than those based on CT CU model. Therefore, generalizations based on the results of CT CM model should be made cautiously. In our study, we took into account of all the converged solutions of CT CM model, i.e., both proper and improper solutions, because we wanted to examine the model pe rformance considering the whole sampling distribution of parameter estimates rather than a certain range of the distribution. Furthermore, bias is defined with respect to the entire distribution of parameter estimates, so it would be inadequate to eliminat e improper solutions in the calculation of bias. We then did supplementary analyses using 1000 proper solutions for the three models, however, the result was very similar to the previous analyses. Trait factor correlation estimates from CT CM model were s till seriously biased, in contrast, estimates from CT CU model were much more accurate. This result was in line with Marsh and Bailey (1991) and Conway et al. (2004) findings considering only proper solutions. Conway et al. (2004) also conducted further si mulation using sample size 500 and matrix size 7 trait 4 method, the results by CT CM model showed that the proper solution rate for 100 data sets was 99% and the trait factor correlation estimates were unbiased. Therefore, they indicated that CT CM mode l could perform well only when the study design went beyond the typical sample size and matrix size, this is sizes and sample sizes examined in our study are t ypical in real MTMM research setting, it is not surprising to see the poor performance of CT CM model here. them. For the 3T3M CT CM model, the accuracy of the tra it factor correlation estimates was

PAGE 46

46 greatly affected by population trait factor correlations. More specifically, conditions with low trait factor correlation and larger sample sizes resulted in more biased estimates. In the 4T3M CT CM case, various combina tions of inter trait correlation also had an impact on the accuracy of trait factor correlation estimates. This makes the interpretation of the trait factor correlation estimates from CT CM model untrustworthy. The method factor correlations estimated with the CT CM model were mainly influenced by population method factor correlation, with low method factor correlation conditions leading to more biased estimates. However, increasing the matrix size will reduce the bias on method factor correlation estimates On the whole, if the underlying structure of the MTMM data has multiplicative trait and method effects and the method factor correlations are low, then the convergence rates and proper solution rates are low with either the CT CU model or CT CM model, bu t higher with the CDP model. However, if the matrix size is 3T3M, the CT CU model works better than the CDP model in terms of accuracy of the trait factor correlation estimates. If the matrix size is 4T3M, the CDP model returned more accurate trait factor correlation estimates than the CT CU model. Under conditions of medium or high method factor correlation, the convergence rates and proper solution rates are relatively lower for the CT CM model, but higher with the CT CU and CDP models. With both matrix s izes, the CT CU model tended to produce unbiased estimates in trait factor correlations. The CT CM model yielded more inadmissible solutions for all conditions compared with the CT CU or CDP models, hence more biased parameter estimates. Our simulation stu dy and the three previous simulation studies (Marsh & Bailey, 1991; Tom s, Hontangas, & Oliver, 2000; Co nway, Lievens Scullen & Lance 2004) all showed that the CT CU model had better convergence and proper solution rates than the CT CM model, and the

PAGE 47

47 parameter estimates yielded from CT CU model tended to be more precise than those from CT CM model, especially when the matrix size was comparatively small (e.g., 3T3M, 4T3M). 5.1 Limitations and Suggestions for Future Research There are some lim itations in our simulation study: we assumed that all the measures in our simulated data had equal reliability, and we utilized a constant value for it (0.65). In a real research setting, it is possible that the reliability for different measurements could vary. Moreover, only two additive MTMM models were examined in our study, the performance of other additive models (e.g. correlated traits correlated methods minus one CT C(M 1), Eid, 2000, 2003) needs to explore as well. We studied the case of single ind icator per trait method combination, but the case of multiple indicators per trait method combination was not categorical variables. Coenders & Saris (2000) proposed tha definitions of additive and multiplicative method effects could be formulated by putting constraints in the parameters of the CT CU model. They demonstrated that the CT CU models with additive constraints are equivalents o f the constrained versions of the CT CM models. Likewise, with multiplicative constraints, the CT CDP model for MTMM data with 3 methods. Following Coenders & Sari Corten et al. (2002) examined the fit of the two types of constrained CT CU model to 87 MTMM matrices consisting of categorical variables. Their results showed that additive models had better fit t han the multiplicative models. Also, the performance of the CDP model for MTMM data with an underlying additive trait method effect was not examined in our study, which should be addressed in the future research.

PAGE 48

48 5.2 C oncluding Remarks Previous simulation studies using the additive model as the underlying structure proposed that the CT CU mode l is more appropriate when method factor correlation were low or close to zero ( Co nway, Lievens Scullen & Lance 2004 ). However, our study suggested conditions with low method factor correlations will yield more biased estimates in trait factor correlatio ns with both the CT CU model and CDP model. Compared with the CDP model, the CT CU model produces more accurate estimation of trait factor correlations when the matrix size was smaller (i.e., 3T3M). However, if the matrix size is 4T3M, CDP model has better performance. Our final conclusion is that, if the underlying structure of the MTMM data consists of a multiplicative trait method relationship, the CT CU model works well in most conditions. Considering the good performance of CT CU model for additive MTM M data reported in other simulation studies, CT CU model is quite robust to MTMM data with either additive or multiplicative underlying trait method effect when the matrix size is 3T3M. The current study also found that all the three models had adequate f it for the MTMM data assuming a multiplicative trait method effect on the measured variables, but not all of them yielded unbiased parameter estimates for the trait factor correlation and method factor correlation for all combination of conditions. In part icular, the CT CM model produced no acceptable relative bias of the trait factor correlation or the method factor correlation for all combination of conditions. In 3T3M case, if the population method factor correlation is of medium or high level, the CDP m odel produced similar accurate parameter estimates of the trait factor correlation as those from CT CU model under the same conditions.

PAGE 49

49 LIST OF REFERENCES Bagozzi, R. P., & Yi, Y. (1990). Assessing method variance in multitrait multimethod matrices:The c ase of self reported affect and perceptions at work. Journal of Applied Psychology, 75, 547 560. Bagozzi, R.P., & Yi, Y. (1991). Multitrati multimethod matrices in consumer research. Journal of Consumer Research, 17 426 439. Bagozzi, R. P., Yi, Y. & Phill ips, L. W. (1991). Assessing construct validity in organizational research. Administrative Science Quarterly, 36 421 458. Becker, T. E., & Cote, J. A. (1994). Additive and multiplicative effects in applied psychological research: An empirical assessment o f three models. Journal of Management, 20, 625 641. Browne, M. (1984). The decomposition of multitrait multimethod matrices. British Journal of Mathematical and Statistical Psychology 37 1 21. Browne, M. (1992). uide The Ohio State Univer sity. Byrne, B. M., & Goffin, R.D. (1993). Modeling MTMM data from additive and multiplicativ e covariance sturctures: A n audit of construct validity concordance. Multivariate Behavioral Research, 28 67 96. Campbell, D. T., & Fiske, D. W. (1959). Convergen t and discriminant validation by the multitrait multimethod matrix. Psychological Bulletin 56 81 105. trait multimethod matrices: Multiplicative rather additive? Multivariate Behavioral Research 2 409 426. Coenders, G., & Saris, W. E. (2000). Testing nested additive, multiplicative and general multitrait multimethod models. Structural Equation Modeling, 7 219 250. Cohen, J. (1973). Eta squared and partial eta squared in fixed factor a nova designs. Educational and Psychological Measurement 33, 107 112. Conway, J. M. (1996). Analysis and design of multitrait multirater performance appraisal studies. Journal of Management, 22, 139 162. Conway, J. M., Lievens, F., Scullen, S. E., & Lance, C.E. (2004). Bias in the c orrelated uniqueness model for MTMM d ata. Structural Equation Modeling 11 535 559. Coovert, M. D., & Craiger, J. P., & Teachout, M. S (1997). Effectiveness of the direct p roduct versus confirmatory factor m odel fo r feflecting the structure of m ultimethod multirater job performance d ata. Journal of Applied Psychology 82 271 280.

PAGE 50

50 Corten, I. W., Saris, W. E., Coenders, G., Van der Veld, W., Aalberts, C. E., & Kornelis, C. (2002). Fit of different models for m ultitr ait multimeth od e xperiments. Structural Equation Modeling 9 213 232. Eid, M. (2000). A multitrait multimethod model with minimal assumptions. Psychometrika 65 241 261. Eid, M., Lischetzke, T., Nussbeck, F.W., & Trierweiler, L. I. (2003). Separating t r ait effects f rom trait specific method effects in multitrait multimethod models: A m ultiple i ndicator CT C ( M 1) m odel. Psychological Methods 8 38 60. Goffin, R. D., & Jackson, D. (1992). Analysis of multitrait multirat er performance appraisal data: C omposite dire ct product method versus confirmatory factor analysis. Multivariate Behavioral Research 27 363 385. Hern ndez, A., & Gonz lez Rom V. (2002). Analysis of the multitrait multioccasion data: Additive versus multiplicati ve models. Multivariate Behavioral Research, 37 59 87. Kenny, D. A. (1976). An empirical application of confirmatory factor analysis to the multitrait multimethod matrix. Journal of Experimental Social Psychology 65 507 516. Kenny, D. A., & Kashy, D.A. (1992). Analysis of the multitrait multimehtod matrix by confirm atory factor analysis. Psychological Bulletin 112 165 172. Kumar, A., & Dillon, W. R. (1992). An integrative look at the use of additive and multiplicative covariance structure models in th e analysis of MTMM data. Journal of Marketing Research, 29 51 64. Lance, C. E., Noble, C. L., & Scullen, S. E. (2002). A critique of the correlated trait correlated method and correlated uniqueness models for multitrait multimethod data. Psychological Me thods, 7 228 244. Lehman, D. R. (1988). An alternative procedure for assessin g convergent and discriminant validity. Applied Psychological Measurement 12 411 423. Lievens, F., & Conway, J. M. (2001). Dimension and exercise variance in assessment center scores: A large scale evaluation of multitrait multimehtod studies. Journal of Applied Psychology 86 1202 1222. Marsh, H. W. (1989). Confirmatory factor analysis of multitrait multimehtod data: Many problems and a few solutions. Applied Psychological M easurement 13 335 361. Marsh, H. W., & Bailey, M. (1991). Confirmatory factor analysis of multitrait multimehtod data: A comparison of alternative models. Applied Psychological Measurement 15 47 70. Muthn & Muthn. (2010). Mplus (Version 6.0). Los Ang eles, CA: Muthn & Muthn.

PAGE 51

51 R Development Core Team. (2010). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing Swain, A. J. (1975). Analysis of parametric structures for variance matrices Unpu blished doctoral dissertation, University of Adelaide, Australia. Tom s. J. M., Hontangas. P. M., & Oliver. A. (2000). Linear confirmatory factor models to evaluate multitrait multimethod matrices: The effects of number of indicators an d correlation among methods. Multivariate Behavioral Research 35 469 499.

PAGE 52

52 BIOGRAPHICAL SKETCH Zhang Lidong is an international student from China. She obtained her Bachelor of Arts in English Education at Henan Normal University in China in 1999. She completed her Master of Arts in Linguistics and Applied Linguistics at Shanghai Jiaotong University in 2005. In the spring of 2011, she received her Master of Arts in Education from the program of Research and Evaluation Methodology at the University of Florida. Currently, she is a full time Ph.D student in Resea r ch and Evaluation Methodology at University of Florida.