UFDC Home  myUFDC Home  Help 



Full Text  
EVALUATING ADJUSTMENTS TO THE MEAN SQUARED ERROR DUE TO ESTIMATING VARIANCE PARAMETERS IN LINEAR MIXED MODELS By JAMIE MCCLAVE BALDWIN A DISSERTATION PRESENTED TO THE GRADUATE SCHOOL OF THE UNIVERSITY OF FLORIDA IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY UNIVERSITY OF FLORIDA 2007 2007 Jamie McClave Baldwin To my husband, Ian, and to my children, Caryss, Zachary, and "Baby" Baldwin ACKNOWLEDGMENTS First and most importantly, I thank my husband, Ian Baldwin, for providing endless patience over the last 10 years while I have pursued this degree. I would not have been able to endure this process without his unwavering support and constant belief in me, even when I couldn't believe in myself. I thank him for maintaining both the house and my sanity when times were hard and for making me smile when days were too long to manage. I express my eternal love and gratitude to my children, Caryss, Zach, and Baby Number 3, for their infinite supply of laughter and hugs. They have inspired me to push beyond any limitation and live life to the fullest. They are the lights of my life and the most beautiful souls I have ever encountered and I give thanks for them everyday. My family is a daily demonstration of God's Grace; there is nothing I could ever do to deserve their unconditional love and support. I thank my parents, Jim and Mary Jay McClave, for their hard work, love, patience, and constant support. To my father, I am so grateful to have such a wonderful role model, both professionally and personally. I am forever indebted to him for continual editing and mathematical advice. I am so fortunate to have the opportunity to work with the smartest and bravest man that I know, not to mention the best consultant in the industry. To my mom, I am so thankful for her wise parenting advice, too many hours of childcare to even count, and girl time when it was most needed. I literally could not have finished this monstrous endeavor without her emergency child care services. I also thank them for all of the generosity they have shown me and my family in so many ways. I would also like to thank my brother, Will McClave, his wife, Amber, and my awesome nephews, Jake and Luke. They have been a constant support for my family and I look forward to the next chapter of our lives together. I am so blessed to have such a wonderful, supportive family that lives so close by. It is a rarity in this world which I truly appreciate. To the best friends anyone could ask for, I am thankful for their friendship, love, and encouragement through this entire process. I thank Holly Holly for always helping me see the bright side, laughing at my stories, listening to my endless complaints, and allowing me to learn so much about motherhood and balance from her wonderful example. I thank Heather Bristol for being our pioneer through the statistics program. She has shown me what true bravery is and taught me how to deal with the most unexpected of situations with grace and strength. I thank Dr. Patches Johnson for encouraging my continuation when I thought I would go no further, and for being my mentor in so many ways. I thank her for being a fantastic and fun traveling companion and for her contagious laugh. I thank Meghan Medlock Abraham, my college roommate, for a friendship that picks up right where it was left off no matter how much time has passed and for having a mouth big enough to eat her own fist. I thank Brooke Bloomberg, my "oldest" and dearest friend, for her sisterly bond and for beating me to the finish line; it was all the motivation I needed! It is amazing that we have both accomplished our childhood ambitions. We rock!! I am grateful for other friends that have helped me along the way with their support and laughter. My sincerest appreciation goes to Andy and Tracy Bachmann, Eric and Ann Bloomberg Beshore, Nicole Provost and Kathy Carroll, Debbie Hagan, Lisa Jamba, Ron and Cindy Marks, Larry and Sandy Reimer, and Yvette and Scott Silvey. I consider each one of them family and I am so lucky to have them in my life. Finally, I thank Dr. Ramon Littell for his patience and guidance during this seemingly endless project. Without his loyal support and enduring belief, this dissertation could not have been completed. This degree would not have been possible without his wealth of knowledge, generous spirit, and encouragement. TABLE OF CONTENTS page A CK N O W LED G M EN T S ................................................................. ........... ............. ..... L IST O F T A B L E S ...................................................................................................... . 8 LIST OF FIGURES .................................. .. .. .... ..... ................. .9 A B S T R A C T ............ ................... ............................................................ 12 CHAPTER 1 INTRODUCTION AND BACKGROUND ........................................ ....................... 14 1.1 Introduction to M ixed M odels ............................................. ............................. 14 1.2 T he L near M ixed M odel ................................ .. ........... ................................. ...... 15 1.3 Best Linear Unbiased Predictors (BLUPs) When 0 is Known: Mixed Model E q u atio n s ........................................................................ .. 1 5 1.4 Empirical Best Linear Unbiased Estimation (EBLUP)...............................................16 1.5 TwoStage Estim action and Unbiasedness ............................ .....................17 2 L IT E R A TU R E R E V IE W ........................................................................ ... ...................... 19 2.1 Precision of EBLU Ps ............................. ...... ................... ... .......... ...... .....19 2.1.1 Mean Squared Error of Prediction (MSEP) of EBLUPs...............................19 2.1.2 KackarHarville Approximation to MSEP.............. .... .................22 2.1.3 Estim eating C ( ) .................... .................. ........................ ............... 24 2.1.4 Estim eating M ( ) .................... ......................................................... 25 2.1.5 Estimators of M (0) Based on the KackarHarville Approximation ..............26 2.2 Hypothesis Testing for Single Dimension Linear Combinations ..............................27 2.3 Hypothesis Testing Methods for Multiple Dimension Linear Combinations ..............29 2.3.1 KenwardRoger Multiple Dimension Hypothesis Testing Method ...............29 2.3.2 FaiCornelius Multiple Dimension Hypothesis Testing Methods ....................35 2.4 Negative Variance Component Estimates ............... ............................................. 38 3 ESTIMATES OF MSEP FOR MULTIPLE DIMENSION LINEAR COMBINATIONS ....42 3.1 Reconciling KenwardRoger (1997) and FaiCornelius (1996) Definitions of the Correction Term Approximation...................... ......... .......................... 42 3.1.1 Linear Combinations of Fixed Effects................ ............................................42 3.1.2 Linear Combinations of Fixed and Random Effects.......................................44 3.2 M multiple Dimension Bias Correction ................... ... ....................................... 49 3.2.1 Bias Correction Estimator for Multiple Dimension Linear Combinations of F ix ed E effects ................................................... ................ 4 9 3.2.2 Bias Correction Estimator for Multiple Dimension Linear Combinations of Fixed and R andom E effects ........................................................... .................. 51 3.2.3 Definiteness Properties of the Estimator for MSEP for Multiple Dimension Linear Combinations of Fixed and Random Effects ...................59 3.3 Transform Invariance of M SEP Estim ator......................................... .....................61 4 BALANCED ONEWAY RANDOM EFFECTS MODEL.............. ............... 63 4.1 KackarHarville Approximation Based on REML Estimation Method......................64 4.2 KackarHarville Approximation Based on ANOVA Estimation Methods .................75 4.3 Bias Correction Term for the Balanced OneWay Random Effects Model..................84 4.3.1 Bias Correction Term Approximation under 0 Parameterization .....................84 4.3.2 Expected Value of BIAS" (0) with REML Covariance Parameter E stim action ................................. ........................................... ........ 87 4.3.3 Expected Value of BIAS" (0) with ANOVA Covariance Parameter E stim action .................................... .... ... ... ....... .. ...... ............... 92 4.3.4 Impact of Negative Values of Bias Correction Term Approximation..............95 4.4 Performance of Overall Estimators for the MSEP....................................................98 4.5 Comparison of Prediction Interval Methods .......... .................. ..... ...........101 5 THE EFFECT OF NEGATIVE VARIANCE COMPONENTS ON EBLUP ......................137 5.1 Negative Variance Component Estimates and the Balanced One Way Random Effects M odel ...................... ............................... ......... 137 5.2 Considering the Variance Parameter as a Covariance ............................................. 141 5.3 BLUP derivation for Random Effects Model with Correlated Errors ......................143 6 CON CLU SION AND FUTURE W ORK .................................................. .....................148 APPENDIX DETAILS FOR MONTE CARLO SIMULATION STUDY ON PREDICTION IN T E R V A L S ............................................................................ 152 L IST O F R EFE R EN C E S ..................................................................... .......................... 159 B IO G R A PH IC A L SK E T C H ......................................................................... ... ..................... 162 LIST OF TABLES Table page 21 Summary of developments for MSEP estimation and hypothesis testing ....................40 41 Accuracy of KackarHarville estimator (417) for CR (0) ............................................71 42 Relative bias (% ) of estimators of correction term .................................... ..................84 43 Relative bias (%) of MSEP estimators under REML and ANOVA estimation ..............100 44 Summary of prediction interval procedures.................. ................... ...................102 45 Effect of prediction interval procedures for k = 6, n = 6, !2 = 1 and ^2 = .5 ................108 51 Generated data that produces negative variance component estimate .............................138 52 REML variance parameter estimates...............................................138 53 Solution for random effects with REML variance parameter estimates ..........................138 54 REML NOBOUND variance parameter estimates ........... .............. ...............139 55 Solution for random effects with REML NOBOUND variance parameter estimates.....139 56 LSM EAN S from PRO C GLM ............................................... .............................. 139 57 Data generated from distribution in M odel (51).................................. ............... 145 58 Covariance parameter estimates under Model (51).................... ............................. 146 59 Solution for random effects under M odel (51)................................... ............... 146 510 LSM EAN S from PROC GLM ............................................... .............................. 146 511 Variance parameter estimates under random effects model ........................................147 512 Solution for random effects under random effects model ............................................147 LIST OF FIGURES Figure page 41 Equation (413) as a function of v for several values of k, holding n = 6. Equation (414) holds when functions cross zero line. .......................................................109 42 Accuracy of approximations tr [A (0) B (0)] and tr [A (0) (0)] for the correction term, CR (0), under REML variance component estimation .....................110 43 Accuracy of approximation tr [A(0)BR (0) and estimator E tr A(0)BR (O)] for the correction term, CR (0), under REML variance component estimation..................111 44 Accuracy of approximation tr [A(0)I1 (0) and estimator E{tr A( )I1 (o)]) for the correction term, CR (0), under REML variance component estimation............... 112 45 Comparison of accuracy of estimators for the correction term, CR (0), under REML variance com ponent estim ation......... ............................ ............... ...............1.13 46 Comparison of accuracy of approximations tr [A () 11 (0)], tr [A(0) B (0)], and tr [A(0)BA (0)] for the correction term, CA (0), under ANOVA variance com ponent estim action. ...... ..... ......... ..................... ............114 47 Accuracy of approximation tr A (0)BA (0) and estimator E tr A(0)BA (O)] for the correction term, CA (), under ANOVA variance component estimation .............115 48 Accuracy of approximation tr[A(0)B* (0)1 and estimator E tr A(0)B* (O)]) for the correction term, CA (), under ANOVA variance component estimation .............116 49 Accuracy of approximation tr [A (0)11(0)] and estimator E tr A (o)1 ()]} for the correction term, CA (), under ANOVA variance component estimation .............117 410 Comparison of accuracy of estimators for the correction term, CA (), under ANOVA variance component estimation. ............. .................................................... 118 411 Accuracy of estimators for the correction term under REML, 0, and ANOVA, 0, variance com ponent estim ation ......................................................... ....................... 119 412 Relative bias of estimators for the correction term under REML, 0, or ANOVA, 0, variance com ponent estim ation ....................................................................................... 120 413 Bias produced by estimating M, (0) with M, (). ................................ ........... 121 414 Accuracy of approximation trA (0) I (0)1 and estimator E tr A(o)I1 (o)]} for the true bias correction term, M, (0) E[M ()1, under REM L variance component estimation. .................................... .....................122 415 Accuracy of approximation tr[A(0)l1 (0)] and estimator E tr A(O)I 1)l) for the true bias correction term, M, (0) E M (O), under REML variance component estim ation............... .... ........................... .............................................123 416 Comparing the accuracy of E tr[A(O)I1 (6)]} and E{tr A (0)1(o)l for the true bias correction term, M (0) E[M ( )] ................................. ..................124 417 Bias produced by estimating M, (0) with M (). .................................................125 418 Accuracy of approximation tr A(0)I1 (0) and estimator E tr A(6)I1 (e)1} for true bias correction term, M, (0) E[M, (O), under ANOVA variance component estimation. ..........................................................126 419 Accuracy of approximation, tr[A(0)I 1 (0) and estimator E{tr A( )I'1 ()l} for true bias correction term, M, (0) EMI (0)], under ANOVA variance component e stim a tio n ................................................................................................................1 2 7 420 Comparing the accuracy of E ltr[A( ()I ( )1 and E{tr A(o) (9)1} for the true bias correction term, M (0) E M ( )] ............................................................128 421 Comparison of estimators of the MSEP for the EBLUP of a utilizing REML variance com ponent estim ation....................................................... ........................ 129 422 Comparison of estimators of the MSEP for the EBLUP of a utilizing ANOVA variance com ponent estim ation............................................................. ..................... 130 423 Relative bias of possible estimators for MSEP of EBLUP for .............. ...............131 424 Simulated coverage rates of prediction interval procedures for t = a, with k = 3...........132 425 Simulated coverage rates of prediction interval procedures for t = a, with k= 6...........133 426 Simulated coverage rates of prediction interval procedures for t = a, with k = 15.........134 427 Simulated coverage rates of prediction interval procedures for t = a, with k = 30.........135 428 Simulated coverage rates for the KenwardRoger prediction interval procedure............136 Abstract of Dissertation Presented to the Graduate School of the University of Florida in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy EVALUATING ADJUSTMENTS TO THE MEAN SQUARED ERROR DUE TO ESTIMATING VARIANCE PARAMETERS IN LINEAR MIXED MODELS By Jamie McClave Baldwin December 2007 Chair: Ramon Littell Major: Statistics Several methods are commonly used to handle the propagation of variance stemming from estimating unknown variance components in Linear Mixed Models (LMM). The accuracy and consequences of using these methods, however, have not been thoroughly investigated. Empirical Best Linear Unbiased Predictors (EBLUP) for analyzing LMMs are widely used, yet the best way to evaluate the precision of the EBLUP is not generally understood. Many developments in the estimation of the Mean Squared Error of Prediction (MSEP) of the EBLUP, and the use of these estimates for hypothesis testing, have occurred during the last two decades. This dissertation begins with a thorough review of these developments. Existing methodologies for evaluating the precision of the EBLUP are generalized to apply to multiple dimension linear combinations of fixed and random effects, and the definiteness properties of these methodologies are examined. The methods for evaluating the MSEP of the EBLUP are examined thoroughly for the balanced oneway random effects model to assess the accuracy of the methods, the effect of the parameterization of the model, and the impact of variance parameter estimation techniques on the components of the MSEP estimators. The impact of negative variance component estimates on EBLUP methodologies is examined, and an alternative solution to account for negative variance estimates is developed for the oneway model. CHAPTER 1 INTRODUCTION AND BACKGROUND 1.1 Introduction to Mixed Models In 1984, Kackar and Harville introduced a new method to handle the propagation of variance stemming from estimating unknown variance parameters in linear mixed models. In the nearly quarter century since this development, there have been many adjustments, improvements and new techniques resulting from their idea. Linear mixed models and the procedures used to analyze them are full of complexities and nuances that can confuse even the most experienced analyst. For example, complex covariance structures coupled with unbalanced data and variance parameter estimates on or near the boundary of the parameter space can produce confounding results with many mixed models procedures. This paper begins to unwind and isolate these complexities using simple cases to explore the best options available and determine when procedures for estimating the realized values of the random effects and evaluating the variability of these estimates succeed and fail. Mixed models methodologies have developed over time to accommodate situations where prediction of the realized values of the random effects is required. These situations arise, for example, in agriculture, education, and clinical trials. A typical agriculture example is the need to assess the genetic contribution of a particular animal for breeding, which is considered to be drawn from a population of animals. An example from education is the desire to assess the effect of a particular teacher on the test scores of his or her pupils. A clinical trial with multiple clinics and varying patient dropout rates gives another arena where mixed models are frequently used to analyze data. Many of these analyses involve complex covariance structures and almost all analyses involve unknown covariance parameters. It is important not only to develop accurate methods for evaluating the realized value of the random effect, but also to evaluate the associated mean squared error to determine the precision of the estimate. We begin by describing the framework of the model. 1.2 The Linear Mixed Model The general form of the linear mixed model is y= Xp +Zu+e (11) where X is a known (n xp) design matrix for the fixed effects, P is a (p x 1) vector of fixed effect parameters, Z is a known (n x q) design matrix for the random effects, u is a (q x 1) vector of random effects, and e is an (n x 1) vector containing the random error terms. We will assume that E(u)= 0, V(u)= G, and E(e) =0, V(e)= R, cov(u,e)= 0. From the model in Equation (11), it follows that E(y) = Xp and V(y)= V = ZGZ'+ R. The matrices G and R are taken to have known structures but depend on an unknown dispersion parameter, 0 e 1l, for some known parameter space 2 Multivariate normal distributions on u and e are often assumed and many of the results are derived under normality assumptions; the validity of these results under other distributional assumptions is questionable. For simplicity, all matrices are assumed to be of full rank. 1.3 Best Linear Unbiased Predictors (BLUPs) When 0 is Known: Mixed Model Equations Best Linear Unbiased Predictors (BLUPs) originated from the desire to estimate the realized values of the random effects along with the fixed effects, or predictable linear combinations of both types of effects. This method was originally developed by Henderson (1950) for estimating genetic merits in animal breeding. Henderson approached the problem using maximum likelihood methodology, although he later acknowledged that his method cannot be called "maximum likelihood" since the "function being maximized is not a likelihood" (Robinson 1991, p.18). Assuming that u and e are normally distributed, Henderson maximized the joint density of y and u with respect to P and u. This method results in Henderson's "mixed models equations" (MMEs): X'R1Xp + X'RZii = X'Rly (12) Z'R1Xo+(Z'R1Z+ G)ii =Z'Ry Note here that G and R are assumed fully known. The solutions to these equations, P = X'V'X) X'V1y (13) ii = GZ'V1 (Y X), are the Best Linear Unbiased Estimate (BLUE) and Best Linear Unbiased Predictor (BLUP), respectively: they are linear in the data, unbiased, and achieve minimum variance or mean squared error (MSE), respectively, among all linear, unbiased estimators or predictors (see, for example, Harville 1990 and Robinson 1991). Note that the solution for p in Equation (13) gives the GLS estimate. Robinson summarized several other justifications for the BLUP estimates, including a Bayesian method with a flat prior on P and a N(0,G) prior on u yielding the same result as the MMEs, as well as justifications that do not require normality. A linear combination of fixed and random effects is predictable when the fixed effect portion is estimable. That is, for vectors X and 6, X'P + 6'u is predictable if 'p1 = a'Xp for some a (Littell, Milliken, Stroup, Wolfinger, and Schabenberger 2006, p. 211). When estimation of a predictable linear combination of fixed and random effects is desired, X'p + 6'ii gives the BLUP for the predictable function. 1.4 Empirical Best Linear Unbiased Estimation (EBLUP) The estimates in Equation (13) assume G and R (and thus V) are fully known; this assumption, however, is rarely satisfied. A common alternative is to assume G and R are known up to a dispersion parameter 0 = (0,...,0 ,), thus G =G(0) and R= R(0). The BLUP solutions depend on G and R, and therefore 0, which is typically unknown. The standard way of resolving this issue is to employ "twostage estimation;" that is, first estimate the dispersion parameter 0 by, say, 0 then use these estimates in the equations for the BLUP of the predictable function, yielding X'p + 6'ui, where P and u are the solutions to Equations (12) with 0 estimating 0: G=G(0) p=(XiX)1 X'V Y R=R( ) = GZ'Vjy Xp) These predictors are often referred to as Empirical BLUPs (EBLUP), although they are no longer linear functions of the data, they are not necessarily unbiased, and they no longer necessarily have minimum MSE. There are many ways to estimate 0, including ANOVA, MINQUE, MIVQUE, maximum likelihood (ML), and restricted maximum likelihood (REML). The REML methods tend to be favored from a theoretical perspective (Searle, Casella, and McCulloch, 1992). However, from a data analytic perspective it is unclear which method is most advantageous. The technicalities of these methods will not be reviewed here. For an overview of the technical details of these estimation methods see, for example, Khuri and Sahai (1985) and Robinson (1987). We will review the advantages and disadvantages of the methods as necessary in the context of evaluating the precision of the EBLUP methodology. 1.5 TwoStage Estimation and Unbiasedness The consequences of twostage estimation on the estimates of the linear combination, 'p + 6'u, are discussed in Kackar and Harville (1981). They show that when the dispersion parameter estimates are translation invariant, even functions of the data, and the data are symmetrically distributed, the twostage estimation procedure resulting in the EBLUPs provides unbiased predictors of linear combinations of effects; i.e., E [' + 6'ii = 1.'. Khatri and Shah (1981) provide similar results. Kackar and Harville (1981) show that these conditions are satisfied by assuming normality of the random effects and error and using REML, ML, ANOVA, MINQUE or MIVQUE variance component estimation methods. Note that this result can be extended to the broader class of location equivariant estimators, as shown in Kackar and Harville (1984), citing a theorem by Wolfe (1973). In evaluating the precision of EBLUPs, we must account for sampling variability contributed by the estimation of 0 When this added variability is ignored, the true precision is often grossly underestimated, as demonstrated by Kackar and Harville (1984), Harville and Jeske (1992), and Tuchscherer, Herrendorfer, and Tuchscherer (1998). Goals of this dissertation include clarifying and comparing existing methods of estimating the precision of the EBLUP, assessing the accuracy of these methods and evaluating the methods of hypothesis testing which utilize the various methods of estimating the precision. After reviewing the related literature in Chapter 2, we generalize results on evaluating the precision of the EBLUP to include multiple dimensional cases and random effects in Chapter 3. A case study of the balanced oneway random effects model in Chapter 4 sheds light on the problems with existing methods and suggests possibilities for improvement. Chapter 5 tackles the issue of negative variance component estimates and the impact on EBLUP methodology. Conclusions and future work are summarized in Chapter 6. CHAPTER 2 LITERATURE REVIEW 2.1 Precision of EBLUPs 2.1.1 Mean Squared Error of Prediction (MSEP) of EBLUPs Let t = X'p + 6'u and let f = f (0) = X'p + 6'i denote the estimate of t when 0 (and hence V) is known. When 0 is unknown, denote the estimate of t, the EBLUP, by S= f () = x,' + 6'f. The prediction error for the EBLUP is t and the mean squared error of prediction (MSEP) is M(0)= E[ t]2. By decomposing the prediction error into four components, each assuming a different, successive state of knowledge, Harville (1985) shows how each component adds to the variability. The first state of knowledge supposes full knowledge of the joint distribution of (y, t). The unbiased minimum MSE predictor of t in this case is clearly E[t I y]= x'p + 6'E [u I y] (Harville 1985, sec. 2.1). This is known as the Best Predictor. The second state of knowledge supposes the first and second moments are known, i.e., P and 0 are known but the functional form of the joint distribution of (y, t) is unknown. In this case the predictor with minimum MSE among all linear predictors is z = 'P + 'GZ'(ZGZ'+ R)1 (y Xp) = /+6'GZ'(ZGZ'+ R)y y where 7 = 'p 6'GZ'(ZGZ'+ R)I X (Harville 1985, sec. 2.2). This is known as the Best Linear Predictor. The third state of knowledge assumes only the variances and covariances are known, i.e., o is known but p and the functional form of the joint distribution of (y, t) are unknown. Since rq depends on P we must replace P with an estimate. Choosing p, the BLUE for P, results in f = B (y)+6'GZ'(ZGZ'+ R) y where (22) B (y) = k'p 6'GZ'(ZGZ' + R)1 Xp (Harville 1985, sec. 2.3). This is the Best Linear Unbiased Predictor (BLUP). The fourth state of knowledge represents the typical situation in data analysis: all moments and the form of the joint distribution of (y, t) are unknown. That is, now 0 is also unknown. Because the BLUP in Equation (22) depends on 0 we cannot directly use it without further adjustment. Let z be maximal invariant, that is, let z = L'y where L is an n x (n p) matrix with rank(L) = n p and L'X = 0. A function g(y) is translation invariant if and only if g(y) = h(L'y). Thus let 0(z) = 0 be an arbitrary translation invariant estimate of 0. Then iB (Y)= IB (y;W)dP(o; z), where P(o; z) is an arbitrary probability distribution on 0, gives a feasible estimate oft. The Empirical Best Linear Unbiased Predictor, EBLUP, corresponds to TB (y; ), the value of ZB (y) when P(*; z) is the degenerate probability distribution assigning all mass to 0 (Harville 1985, sec. 2.4). Note that ,B (y) is translation invariant, thus preserving the unbiasedness of the predictor. With this notation, we now have the four components of the prediction error: t =Et(t y) t+r E(ty)+ Bq+ B(y;) TB. (23) (1) (2) (3) (4) Each successive component is associated with moving from one state of knowledge to the next. This decomposition allows analysis of the role of each component in the MSEP. Component (1) of Equation (23) is uncorrelated with all other components by the following result (Harville 1985, p. 135): cov(h(y), E(t y) t)) = E [h(y) (E(t I y) t)] =E {E[h(y)(E(t y) t)] y (24) =0. Components (2) and (3) of Equation (23) are also uncorrelated since cov(y,r E(t I y)) = 0 and r, is linear in y (Harville 1985, p. 135). Thus the Mean Squared Error ofPrediction (MSEP), M(0), can be written as the following six components: M (0) =E[ t]2 =E[E(t y) t]2 +E[rE(t y)]2 +E[B, +]2 +E [,y; ) f, (0) (25) (1) (2) (3) (4) +E[ E(t Iy))(i, (y; ,) (0)) + E[(77)(B (Y; ( )B ())]. (5) (6) In the development of estimators for M(0), it is often assumed that the data follow a normal distribution. When the joint distributions of (z,r t) and (z, B (y)) are multivariate normal, as is the case when u and e are normally distributed, then components (5) and (6) of Equation (25) are 0. However, without normality, there is no guarantee that these crossproduct terms disappear. The remainder of this chapter will review the developments over the last two decades for evaluating the MSEP of the EBLUP and conducting tests of hypotheses. Table 21 contains a summary of the developments reviewed herein and stands as a helpful reference for these developments and the notation used throughout. 2.1.2 KackarHarville Approximation to MSEP Kackar and Harville (1984) and Harville and Jeske (1992) assume normality in developing an approximation and estimator for M () Thus, only components (1) (4) of Equation (25) need to be assessed for the approximation. Components (1), (2), and (3) of Equation (25) comprise the MSEP when 0 is known. This is referred to as the naive MSEP and will be denoted by M (0) = E [, t]2 In the mixed model case with V(u) = G and V(e) = R, this is M, (0) ='G6  'GZ'V ZG, + y' v'GZViX) (X'V'X)1 XV'ZG5). When normality is assumed, the crossproduct terms are zero, and the only contribution to the MSEP of estimating 0 is component (4) of Equation (25). We will refer to this component as the correction term and denote it by C(0). M1 (0) is often used as an approximation for the MSEP. When the data are normally distributed, this can seriously underestimate the MSEP since it ignores the contribution of estimating 0 captured by C(0), as shown by Kackar and Harville (1984), Harville and Jeske (1992), and Tuchscherer et al. (1998). Peixoto and Harville (1986) expand on results in Kackar and Harville (1984) to derive exact, closedform expressions of the MSEP for certain predictors for the special case of the balanced oneway random effects model. We use similar methods in Chapter 4 to enable comparisons of approximations and estimators to the exact MSEP in the balanced oneway random effects model. Assuming normality, Kackar and Harville (1984) provide an approximation to C(0) using Taylor series methods. Using a Taylor series expansion of i = f () about 0 yields () = (0)+d(y; 0)' (00), where d(y;0)= Of(y;0)/a0. This leads to the approximation, []2 = [i()i(0)] 2 [d(y;0)'( )]2. (26) Note that when f is linear in 0 this Taylorseries approximation is exact. The validity of the Taylorseries approximation is contingent on two conditions: First, the function being expanded must have derivatives of all orders defined in a neighborhood of the point about which the function is being expanded; and second, the remainder term must tend quickly to zero (Khuri 1993, p. 111). Thus if the parameter space, 0, is restricted (i.e., bounded below by zero) and the true value of 0 falls on the boundary of the parameter space, the Taylor series expansion is not valid. Assessing the effect of the parameterization and parameter estimates on the estimates of the MSEP is one goal of this dissertation. When 0 is unbiased for 0 and cov d (y; 0) d (y; 0)', ( 0) ) = 0 we have E[ii]2 =E d(y;0)'(00)] = Ed (y;0) d(y; 0)' E [(00)(0). Noting that E[d(y;0)] = 0, Kackar and Harville (1984) propose the following approximation to C () : C(0) tr [A(0)B(0)] (27) where, A(0) = var[d(y;0)] =E[d(y; )]2 and B(0) is either an approximation to or the exact value of the MSE matrix, E 0 0 0 0 The inverse of the observed information matrix, I 1 (0), is commonly used for B(0). Equation (27) will be referred to as the KackarHarville approximation for the correction term. Thus the approximation of the MSEP of i proposed by Kackar and Harville (1984) is M, (0) +tr [A(0)B(0)]. (28) The accuracy of the KackarHarville approximation of the correction term in Equation (27) depends on the following conditions: 1. f linear in 0; 2. the validity of Taylor series expansion of T about 0; 3. 0 unbiased for 0; 4. cov d(y;0) d(y;0)',( 0)( 0) =0; 5. the tractability of the exact MSE of 0 . If all of these criteria are met, the KackarHarville approximation exactly equals the correction term it is approximating. 2.1.3 Estimating C(0) Because the calculation of the KackarHarville approximation to the MSEP in Equation (28) requires knowing 0, it is not directly functional. An estimator based on the Kackar Harville approximation was developed by Harville and Jeske (1990) and Prasad and Rao (1990) and is now implemented in the MIXED procedure of SAS. Following developments from Prasad and Rao (1990), an estimator for tr [A(O)B(O)] is given by tr A( )B(0)1 ; where 0 is substituted for 0 in the KackarHarville approximation for the correction term. The justification for this substitution is not directly evident. Prasad and Rao (1990) show that in the special case of the FayHerriot small area model, in which V(e) = R is known, the order of approximation is o(q 1) where q is the number of small areas, or in general the number of levels of the random effect. This result has not been generalized for the typical case when R is unknown. The validity of this substitution and the consequences on the accuracy of the MSEP estimate will be examined for the balanced oneway random effects model in Chapter 4. 2.1.4 Estimating M1 (0) Historically, M, (0) has often been estimated by M, (6), that is, by substituting 6 for 0 in M, (0); however, M, (6) may have bias of the same or higher order as the correction term (Prasad and Rao 1990; Booth and Hobert 1998). Consequently, it is just as important to account for the bias introduced by estimating M1 (0) with M1 (6) as it is to account for C(0). Prasad and Rao (1990), Harville and Jeske (1992), and Kenward and Roger (1997) all approximate the bias of M1 (6) by taking the expectation of a Taylor series of M1 (6) about 0 (assuming 0 is unbiased for 0). Harville and Jeske (1992) developed the following expression for the bias of E M, ( M, (0) + tr[A (0) B (0)] (29) 82, B (9) where A( (0) They show that when cov(y,t), var(t) and var(y) are all linear in 802 0, A (0) = 2A (0) leading to E[M, (6)] _M, (0) tr[A()B()] (210) Prasad and Rao (1990) and Kenward and Roger (1997) developed expressions similar to Equation (210). Again by substituting 6 for 0, possible estimates for M1 (0) are or M, ()+tr[A (0)B()]. (212) Notice again that there is bias introduced by substituting 0 for 0 in Equations (211) and (212). This bias is often assumed to be negligible, although the only noted justification, previously mentioned in Section 2.1.3, comes from Prasad and Rao (1990) and only applies to Equation (212) in the FayHerriot model. We will study this issue further through a special case in Chapter 4. 2.1.5 Estimators of M (0) Based on the KackarHarville Approximation Based on Equations (211) and (212), respectively, the following estimators of the MSEP follow from the KackarHarville approximation and the bias correction approximations: M () tr [A()B ()] + tr [A ()B() (213) and M, (e)+2tr [A ()B() (214) The DDFM = KR option of the MIXED procedure in SAS utilizes Equation (213) which reduces to Equation (214) when the model covariance structure is linear in 0. Chapter 3 explores the Kenward and Roger (1997) and Fai and Cornelius (1996) expansion of Equation (213) into multiple dimensions. Therein we establish the equality of the KenwardRoger and FaiComelius approximations to the correction term and expand the term to include linear combinations containing random effects. We also provide a proof that the most general form of the correction term approximation is nonnegative definite and discuss the definiteness properties of the overall MSEP estimator. 2.2 Hypothesis Testing for Single Dimension Linear Combinations Several methods of approximating the distributions for test statistics have been developed for single and multiple dimension linear combinations. The accuracy of these methods has been largely untested. Following is a summary of the available methods for approximating the distributions for single dimension linear combinations. Giesbrecht and Burns (1985) and Jeske and Harville (1988) provide approximations for t type test statistics based on the Satterthwaite (1941) method. Giesbrecht and Burns (1985) present a test statistic for fixed effects with the "naive" variance estimate: tGB 1 G t (215) x' (x'V x) ) When V is known and used in place of V in Equation (215), the test statistic has a normal distribution. Thus, it is assumed that Equation (215) has a tdistribution with the degrees of freedom derived from the approximate distribution of the denominator. To estimate the degrees of freedom, a "Satterthwaite" method is used. The variance of the denominator is matched with that of the approximating 7 distribution as follows: x' X'Vix) 1 df 7 1 var dfGB 'X) = 2dfGB ( VGB ) var L] XVX' x ) = 2dfGB k,(,'2 k'(X'V'X)  SdfGB 2 ( ) _ var (X'ViX) The term var [X'X'V iX) k is evaluated using a Taylor series approximation and the asymptotic covariance matrix of 0. In the GiesbrechtBurns case, this yields var[ '(' X'x)) ] cov 1 'Vz V Z'Vx(xv11 XV 1 (217) 'X'V1Z V z'Vx(x V121 The underlying assumption in developing this procedure is that the distribution of 'X'(XViX)) k is a multiple of a X2 distribution. Jeske and Harville (1988) also use a Satterthwaite method to approximate the degrees of freedom of a tstatistic; however, they include fixed and random effects in their test statistic with a corrected version of the MSEP estimate. The approximate ttest is given by tm = ~ t (218) M 9 + tr [A (6) B( where 2\M, (Q)+,)tA(Q),(Q)1}}2 dfJH L] (219) var M, (0)+ tr A(0)jB( )l As in the GiesbrechtBurns ttest, the denominator of Equation (219) is approximated by a Taylor series expansion yielding var M (0)+ tr A(0)B() h ()' (B(0)h (), (220) where h(0) = [M ()+ tr A(0)B()] and B(0) is an approximation ofvar(). 90 (6) Again, an underlying assumption is that the distribution of M, o)+ tr A () B (6)] follows a multiple of a _2 distribution. An extension of this method would be to incorporate the bias correction into the MSEP as seen in Equations (213) and (214). This extension is considered in Chapter 4. 2.3 Hypothesis Testing Methods for Multiple Dimension Linear Combinations Kenward and Roger (1997) and Fai and Cornelius (1996) offer Ftests to handle multiple dimension as well as single dimension cases; however, the methods for determining the distribution of the test statistics differ. One of the Fai and Cornelius methods (DDFM = SATTERTH option) and the KenwardRoger method (DDFM = KR option) are available in the SAS MIXED procedure. A goal of this dissertation is to compare these methods with other modified methods from Fai and Cornelius (1996) for the balanced oneway random effects model. We will also assess the effect of sample size, parameterization, and parameter value on these hypothesis testing methods. 2.3.1 KenwardRoger Multiple Dimension Hypothesis Testing Method The test statistic developed by Kenward and Roger (1997) utilizes an MSEP estimate for multiple dimension linear combinations that is adjusted for the correction term and the bias of M, (o). This MSEP estimate is thoroughly discussed in Chapter 3, but for now will be denoted by MK (). Note that these methods are only developed for tests of fixed effects; thus the MSEP can be thought of as a variance. Theoretically these methods could easily extend to include random effects. We include a simulation study for the oneway random effects illustration to determine the validity of including random effects in these procedures. Consider the hypothesis Ho:LP=0 vs. Ha:Lp#0, where L is an /dimension matrix of constants representing the linear combination of interest. Kenward and Roger (1997) propose the following Waldtype test statistic: F KR 0)'L'(LMKR (6)L) 1 L(). An Fdistribution is approximated for the test statistic to take into account the random structure of MK (O) and to match known cases where the test statistic has an exact Fdistribution. We will briefly review the procedure to allow comparisons to alternative methods and to examine any shortcomings. Kenward and Roger (1997) use a Taylor series expansion to approximate (LMK () L') ; however, it is unclear about which point the series is expanded and which higher order terms are dropped. It appears that the series is expanded about M, (0) rather than the true value, M (0). This may be justified by dropping higher order terms; however, this step is the only place where the corrected MSEP is taken into account in the distribution of the test statistic. Using this Taylor series approximation, Kenward and Roger (1997) use the identities E[FK] = E[E[FK M ()]] V [F] = E[4FKF MR() + V[E[FK MKR()]] to approximate the moments of the test statistic. A twomomentmatching method is then used to obtain estimates for a coefficient, K and degrees of freedom, m, so that c KR l,dfIR To make the approximating distribution match the exact cases where the distribution is known, selected higher degree terms are added to the Taylor series expansion. The following summarizes the results of the KenwardRoger procedure: 1+2 df, df = 4 + and K= p 1 E[FKR](dfR2) where V[F KR] 2E[FKR]2 and E[F} ] 1 ] and V[FKR]= C B(1 B)] ( 1 (1 ) (IB 3B) where S=1 J=1 L(V 00 (x ' (221) *tr L' L(X'VIX)L ]L(X'V1X) 1X' V X(XV X) A2 =Y Btr L VLX'VX L L(X'VX) X' X(XVX )1 1=1 ]=1 1 *LLL(X'VXX)L L L(XV1X) Xavx(x'V'x)1, L 00j J B B (A + 6A2), 21 g 31+ 2(1g)' 1g 231+2(1g)' l+2g 33/+2(1g)' (l +1)A (1+4)A2 g= (1+2)A2 and B, is the ijh element of the matrix B(0). The KenwardRoger method is available in the PROC MIXED procedure in SAS as the DDFM = KR option in the model statement. This option performs the adjustment to the MSEP estimate and the hypothesis test as described above. We now demonstrate that in the singledimension case, the KenwardRoger degrees of freedom approximation reduces to the GiesbrechtBurns degrees of freedom approximation reviewed in Section 2.2.1 (Kenward and Roger 1997, p. 988). Recall that the GiesbrechtBurns test statistic did not use an adjustment for unknown 0 in the MSEP. Thus, while the Kenward Roger and GiesbrechtBurns test statistics differ in the estimate of standard error, the approximated distributions are identical. Consider again a hypothesis test for the single dimension linear combination Ho : ,' = 0, under the linear mixed model. The KenwardRoger test statistic is t = (222) VA KR where VARKR = (XvlX)i1+ 21x (XVlX) B X'v V V 80, 800 X OV1 XX'V1X X 8V1 X X'V14 2V VXIX V1X = 90 / 0 4 86 (v v In this case A, = cov( )tr X'V)'1 XVZ Z'Vx 'Vx *tr ?. 1' X'V1X) 1 'X'ViZ ZV X (X'xVX1 XV*trX cov,) xvzX('v x)V 'iixfvz 'V zvx(i)1 1=1 J=1 9 x1'X'VZ Vz'vlx x'vlx)1 =A since the quantities inside the trace functions are scalars. Note also that 1 7 B= (A,+6A)= A2, 2 2 2A2 5A2 g= = 1, 3A2 1 1 cl 3(1)+2(1+1) 7' 2 C2  7 and 4 C3 = 7 Thus, E[F] = (1 A2)1, 1+ (1)( )A 22 V[FKR]=2 7 C2( 2A2 and P l 2 2A (1A2)2 2A, (1 A, )12A 2, 2 2(12A2 ) The degrees of freedom for the KenwardRoger hypothesis test are then 3 2 df = 2A 2 1 2(1 2A) =2 ('(X'VX)l j 1 2/ cov( ) I'XV1Z' VZ'V1X(X'V1X)1 (223) 1 [=1 0 1= 00, *I 'X'VvZ 0V X'V ) J{ Note that the denominator of Equation (223) is the Taylor series expression for var[ 'X'VX X)x This is the same as the degrees of freedom estimate from the Giesbrecht Burns test in Equations (216) and (217); however, recall that the GiesbrechtBurns test statistic is tGB x'(x'vix) x Thus df = dfGB but tK< ltGB Often, the discrepancy in these test statistics may not be enough to make a significant difference in the test results; however, there are cases when the discrepancy changes the conclusion of the test. The question becomes, which test is better? The accuracy of the estimating distribution must also be questioned in either test. Note that because the estimated distributions are equal, we can isolate the impact of modifying the estimated variance of the EBLUP on the hypothesis test. We explore that situation further though a simulation study in Chapter 4. 2.3.2 FaiCornelius Multiple Dimension Hypothesis Testing Methods Fai and Cornelius (1996) also use a moment matching method to determine the distribution for their multiple dimension test statistics. The differences between the Kenward Roger method and the FaiCornelius method come in the particular MSEP estimate used in the test statistic, in the number of moments evaluated, and in the techniques used to evaluate the moments of the test statistic. Fai and Cornelius (1996) offer two alternatives for the test statistic: one uses the naive MSEP estimate in matrix form, M1 (0), and the other uses the naive estimate plus a correction term denoted by, MFC () = M1 () + CF (6), which will be discussed thoroughly in Chapter 3. Note that neither of these takes into account the bias of M1 (0) for M, (0), as in the Kenward Roger procedure. The test statistics are FFCl =()'L'(LM()L) 1L(p P) (224) and FFC 0)'L'(LMFC 6 L') L pP). (225) For each of these test statistics, both one and two momentmatching methods are offered for estimating an Fdistribution, resulting in four alternatives for conducting a multiple dimension hypothesis test. The method of finding the moments is the same for both test statistics, so for simplicity let MSEP denote the estimator of the MSEP in either FaiCornelius test statistic. To evaluate the moments of the test statistics, first a spectral decomposition is performed so that P'L(MSEP)L'P = diag() where P is an orthogonal matrix of eigenvectors and 4, k = 1, 2,...1, are the eigenvalues of L(MSEP)L'. The test statistic is rewritten so that IF, =( P)'PP'L' (L (MSEP)L' PP'L p) =(P'L P'Lp) {P'L (MSEP)L'P1 (P'L P'LP) = (P'L P'Lp) diag ( )}1 (P'L P'LP) S(P'P L2 (226) k=l k = 1 t2k2 k=1 =Q. They then note that tk would have an independent standard normal distribution if the MSEP were known. It is reasonable then to approximate the distribution of tv with a tdistribution and estimate the degrees of freedom, vk, with a Satterthwaite procedure, i.e., k L(P'L'LP'LP) k 2 [(MSEP) V[(V(P'LpP'Lp)) V (MSEP) As in the KenwardRoger procedure, we have the underlying assumption that MSEP has a X2 distribution. If v > 2 for all k= 1, 2, ., 1, then the moments of Q can be approximated as follows: E(Q)= iE(t2 = lVk k=1 k=l Vk 2 and (Q)i 2v (vk 1) k( 1 1 (Vk 2)(v 4) From Equation (226) we see that the moments of the FaiComelius test statistics are E(FF)= E(Q) and 1 V (FFC)= V(Q). One or both of these moments are matched to the moments of the approximating Fdistribution, as in the KenwardRoger procedure, to produce an estimate of degrees of freedom (and an estimated coefficient in the twomoment case). The onemoment method with the test statistic in Equation (224) is available in the PROC MIXED procedure of SAS as the DDFM = SATTERTH option in the model statement. Note that in the single dimension case, the Fai Cornelius test statistic in Equation (224) reduces to the GiesbrechtBurns test statistic in Equation (215) and that the FaiComelius onemoment method of estimating the distribution also produces the same results as the GiesbrechtBurns method. Thus we are avoiding the discrepancies that occur between the KenwardRoger method and the GiesbrechtBums method described in Section 2.3.1. We are interested in comparing the available hypothesis testing methods to determine which one provides the most accurate TypeI error rate. Some work has already been done to this end by Schaalje, McBride, and Fellingham (2002). Schaalje et al. (2002) compare the one moment FaiCornelius method with the test statistic in Equation (224) (using the DDFM = SATTERTH option in SAS) with the KenwardRoger method (using the DDFM = KR option in SAS). They performed simulations for four splitplot designs, two of which were balanced and two of which were unbalanced. The simulation is repeated for five covariance structures to study the effect of the complexity of the covariance structure. Hypothesis tests were computed for whole plot effect and subplot effect using the DDFM = KR and DDFM = SATTERTH methods. Schaalje et al. (2002) calculated simulated Type I error rates and lackoffit tests to determine if the simulated pvalues followed a uniform (0,1) distribution. Both methods performed equally well for the simple Compound Symmetric covariance structure, regardless of the design. For the other four covariance structures the KenwardRoger method outperforms the FaiCornelius method in the lackoffit test and the type I error rates; however, as the sample size becomes smaller or the covariance structure becomes more complex, neither method performs well. Type I error rates are severely inflated under both methods when complex covariance structures are coupled with small sample sizes. Note that when random effects are considered, it is more appropriate to focus the investigation on prediction limits. We will shift the focus to prediction intervals and expand the comparison to include modified FaiCornelius methods with adjusted MSEP estimates in Chapter 4. Although we confine the comparison to a single dimension linear combination in the balanced oneway random effects model, this comparison will paint a more complete picture of the methods considered and show the need for further investigation into the proposed methods. 2.4 Negative Variance Component Estimates Very little research has focused on the impact of negative variance parameter estimates on BLUP procedures, the MSEP estimation methods, and hypothesis testing. Stroup and Littell (2002) begin to explore the impact of the method of variance component estimation and negative variance parameter estimates on tests of fixed effects. Their exploration reveals that REML may not always be the best choice in terms of power and true Type I error values for tests of fixed effects when negative variance parameter estimates are prevalent. The impact of negative variance parameter estimates on the EBLUP and MSEP estimation for linear combinations involving random effects has not been thoroughly investigated. In Chapter 5, we delve into the complexities of negative variance component estimates and the impact on EBLUP and the associated MSEP estimates. Smith and Murray (1984) develop a method for handling negative covariance estimates. They propose that the variance parameter in question be considered as a covariance parameter, thereby allowing negative estimates. The impact of this strategy on EBLUP methods is considered in Chapter 5, leading to an alternative model and analysis procedure. Table 21. Summary of developments for MSEP estimation and hypothesis testing Fixed or Paper Random Correction Term Bias Term Effects Kackar and Harville 1984 Giesbrecht and Burns 1985 Jeske and Harville 1988 Fixed and random Fixed tr[A(0)B( )] None Fixed and tr[A(0)B(0)] random MSEP Estimate M, (0) +tr [A (0)B (0)] None None None k,(XV1x)" k Prasad and Rao 1990 Fixed and Random Etr[A(O)B(O)]j tr [A (0)B()] +o (q1) For q = # small areas, in the Fey Herriot model E[EM(6)] = M (0) tr[A(O)B(O) +o(q ) For q = # small areas, in the FeyHerriot model Harville and o Jeske 1992 Fai and Cornelius 1996 Kenward and Roger 1997 Fixe ad and tr[A()B()] random Fixed Fixed tr[A(0)B(0)] defined for multiple dimension linear combination, denoted cr (); shown to be nnd tr[A(0)B(0)] defined for multiple dimension linear combination; c" (e) M,(e )tr [A(6)B6)+tr [A()B(6)B or M,() 2tr[A (6B M,(e) None E [M,()1 M,(0)+tr[A(O)B(O)] M,(0)tr [A(0)B(0)] Defined for multiple dimension linear combination, BIAS (e) M '()=M(, )+C(6e) = M ()+CR (e)+BIAS" (e E [l (e)]M, (0)+tr[A(e)B(0) SM,(0)tr[A(0)B(0)] M(6) + tr[A(6) B() I (6) +i2tr [A ( B Table 21 cont. Paper Kackar and Harville 1984 Giesbrecht and Burns 1985 Jeske and Harville 1988 Hypothesis H,: X'= 0 Ho: 'p + 6'u = 0 Test Stat tGB '(xv iX)V 1 Al= (6)+tr[A(6)B(6)] Distribution of Test Stat t where lLx ix)W] var '(X'V X) k Evaluated with Satterthwaitetype method tdf H 2[M1 (9)Itr[A(9)B(9)]] 2 Mi a (e)+tr[A (e)B ()] Evaluated with Satterthwaitetype method Prasad and Rao 1990 Harville and Jeske 1992 Fai and Cornelius 1996 H : L= 0 )1 ^ L'(LM ^e)L) L( p F2 1(pP)L'(LM ( eL')L L(p) One and Two moment matching; Spectral decomposition, Taylor series and Satterthwaite techniques used to determine moments. Note that this yields two choices of distributions for each test statistic. Kenward and Roger 1997 Ho: Lp=0 F 1 L'(LM (6^L') 1 ( Two moment matching; Taylor series and conditional expectations KR L used to determine moments. CHAPTER 3 ESTIMATES OF MSEP FOR MULTIPLE DIMENSION LINEAR COMBINATIONS 3.1 Reconciling KenwardRoger (1997) and FaiCornelius (1996) Definitions of the Correction Term Approximation 3.1.1 Linear Combinations of Fixed Effects Kenward and Roger (1997) and Fai and Cornelius (1996) extend the methods for estimating the MSEP summarized in Section 2.1 to multiple dimension linear combinations. Both methods require expanding the definition of the KackarHarville approximation of the correction term, and in the KenwardRoger case, the bias correction term, into a matrix form. Recall that the KackarHarville approximation of the correction term for single dimension linear combinations is C(0) tr[A(0)B(0)] where A () var[d(y;0)] =Ed(y;0)d(y;0)', d(y;0) = 0f(y; )/80, and B(0) is either an approximation to or the exact MSE matrix, E 0 0) Extending this definition to multiple dimension linear combinations is not necessarily unique. In both papers, only linear combinations of the vector of fixed parameters are considered. In Section 3.1.2 we extend their method to include linear combinations involving both fixed and random effects. Noting that tr [A (0) B (0) = b cov , where b,, is the it element of B (), Kenward and Roger expand this definition to the multiple dimension case by C((cov ), where L is an /dimension matrix of constants representing the linear combinations of interest. We will denote this approximation as C ( representing the linear combinations of interest. We will denote this approximation as CKR (0) . Note that, COVf Lp a 80, 0J co L,p aL,p a cov L L "o COV BL2,p OLap cov  cov 0J [80, 80 rLip aL2p (aLp aLp c cov L cov, co o' ) J Io, oj aL2p aL2p aLp aLp cov 2 L cov L2 OL c aL, aL, p cov O L,P L Sa0, a0o, 0, a 00 o where L, is the ith row of L. Thus we can write the kmth element of C ( (0) as {Ck (O)}= bl cov aL +b2 aCOV Lp b +*..+b cov Lk ,Lm 00, 80, 80, 802 qq 8q 80' q Fai and Cornelius (1996) define the multiple dimension approximation to the correction term differently and restrict their definition to a compound symmetric covariance structure with two covariance parameters. We now expand that definition to include any covariance structure with r parameters and show its equivalence to CR (0). They also establish the nonnegative definiteness of their correction term definition, which also applies in this more general setting. Let CFC (0) denote the Fai and Cornelius multiple dimension approximation to the correction term. Then, Fc () = (31) tr {A (0) B (0))} tr {A (0) B (0)} tr({Al(0)B(O)} ... tr{All,()B(O)} where COV ,Lk *L* COV ,O , Ak, (0)= cov : "0 : km{ [ 8Q 8Q Cov Lk 0L0 .P CO Lk d Lmp 0cov o, cov 0, 0 and B(O) is as before. Then the kmnth element of CFC (0) is {CFC (O)}k =tr{Ak( ()B(O)}= b, cov ,k O + bacov a aL + 1+ b ,, cov , O8q Odq Thus, clearly Ct (0) = CFC (0). Fai and Cornelius (1996) have shown that this definition of the correction term approximation is nonnegative definite. This becomes an important factor when the MSEP estimate is to be used in a test statistic or confidence interval. We define the most general form of this approximation and prove that it is nonnegative definite in the Section 3.1.2. 3.1.2 Linear Combinations of Fixed and Random Effects We can now generalize the definition of the multipledimension correction term approximation to include random effects and establish that it is nonnegative definite. Let K = [L I M] denote an /dimension matrix of constants. We now consider the linear combination, K = LB + Mu. To expand the definition to random effects, we need to redefine Ak (0) as Aa(Lk + Mki) (Lp+M i) Akm (0) = cova A 0 o (LP + Mu) I(LO+M ,u) a(Lp+Mu) (LP+MU) 80 80 cov O(Lk+ Mku) O(L0P+Miu) cov (L + Mu ) c(L0P+M iu) 0V' 00 J, 0o 2 cov O(Lk kii) o(L O+o i) ... cov '(Lk ki) O(L.0 + Mji) where Lk is the kth row of the fixed effects portion of the matrix of constants, L, and Mk is the kth row of the random effects portion of the matrix of constants, M. We now follow a similar procedure as used in Fai and Cornelius (1996) to establish that this generalized definition of CR (0) is nonnegative definite and thus the MSEP estimator utilizing this correction term estimator is nonsingular. First, however, we need to broaden the definition of C' (0) to incorporate linear combinations of fixed and random effects. First, using the result that = A A', we have ax ax OLkP OLk (X'VX) 1 X'Vy LkXV1 (X'VX) 1 cP^_,_  )_  = Lk(X'V1X) X' y+Lk c X'V ly 80, 80, 00, V =Lk (X'VX) X' y Lk (X'V1X) X'fV1 X (X'VX) X'V1y =Lk (X'V1X) X' (IP)y Lk(XVX) X'V1 V V IP)y = V1(IP)y CX1V ( P)y where C' =Lk(X'VX) X'V1 and P= X('V X) X1 1 and 8V1 V/? = Mk ZVl(I P)y+MkGZ' (I P)y MkGZ'V X =MG Z'V (IP)yMkGZ'V1 V1 (I Py Mk 0, ~V ,X(X( +MkGZ'V1X(X'V1X)1 XV1 v V1 (P)y v /rBy D'( D V (I P) y + E V1 (I P)y + FV1 (I (D + E + F)V1 (I P)y P)y D' = M Z', E' = MkGZ'V1 F' = MkGZ'V X(X'V 1X) 0X'V1 V fa~~ 6,V Thus, a(Lk, +Mku) 80, We can now assess the ijth element of Ak (0) as {Akn(zj) (0)} Scov(Lkp+ Mki) a(L + Mii) a6/ o0J cov[(C', +D' +,E' +F') V1 (IP)y, (C D' +E' +F)V1(IP)y] (C' + D + E' + F',)V1 (IP)V(I P)'V1 (Cm + Dm + Emj +F,) (Ck, + D, +E, + F,) V (I P)(Cm + D + Ej + F,) since aMkfl S0, where (C' +D' +E' +F'VI( P) y. b bk i ) II PY (I P)V(IP) v1 =(v x(x'Vx)' ') (1 1X(X 'Vx) X''1) =(IP). We now have (Ck + Dk + Ekl+ Fk ) Akm (0) (32) (32) (Ckr +Dkr +Ekr +Fkr) xV1(IP) (C, +D, +E, +Fm ) . (C,1 +Dm +Em +F')] We can now generalize the definition of the KackarHarville correction term approximation as follows. Definition 3.1. The multipledimension KackarHarville approximation for a linear combination including fixed and random effects is CK (0)= (trAkm (0)B(0)]}, (33) where Ak (0) is defined in Equation (32). We now prove that C' (0) is nonnegative definite. Theorem 3.1. The multipledimension KackarHarville approximation for the correction term, as defined in Equation (33), is nonnegative definite. Thus, the MSEP approximation utilizing this correction, M1 (0) + C' (0), is nonsingular. Proof. We begin by examining the kmth term of C"K () We can write V1 (I P) = Q1/2Q1/2 and B(0)= R1/2R1/2 where Q12 and R12 both exist since V1 (I P) is nonnegative definite and B(0) is positive definite (e.g., Schott 1997, p.16). Consider {(c (o)} = rA,, (0)B(0)} (Ckl + Dkl + Ekl + Fkl)' = tr (C, +D, +E ,, +F )I =tr(C + Dk + Ek+ Fkl) (C,+D, +E, +F ,)' V1 (I P)[(C,,D + D, +E., +lF1) ... (Cmr + Dmr + Emr + F,, )]B(0) Q Q [(Cm, + Dml m tE F ) ... (Cml, Dml, Em, + F ml) R2R1 R + F1,) (Cmr + Dmr + Emr + Fr R1/2R/ (Ckl + Dkl +Ekl +Fk ) Q(CD+E (C,h + D, + E + F ,)' Str (Q 12[(C,, +Dml +Em, + F1) ... (C,,, +Dr +Emr +F r)]R12) *(Q (Ckl+Dl +Ek +F) (C Fk+D +E, +F,)R12)'} = vec[(Q1/2 [(Cm + Dm1 + Eml + ) .. (Cm, +D, +E m +F,,)]R1/2)]' *{vec[(Q1/2 [(Ckl+ E ) ( +D +El +Fk) .(C +D +E, +F,)]R1 2)] where =vec[(Q1/2 [(C, + D, + E, +F,) ... (C, +D,, +E,+ +F,J R1/2)] Thus we can write CKR (0) as cK (0) = A'A where A=(s,,...,,) (34) Writing C' (0) in this form proves that it is nonnegative definite (e.g., Schott 1997, p. 16). Since M, (0) = E((Lp + Mi) (LP + Mu))((LI + Mu) (LP + Mu)) is positive definite, we have now established that the general approximation of MSEP, M,(0) + C (0) (35) is positive definite, and hence nonsingular, and thus is a valid measure of prediction error. o 3.2 Multiple Dimension Bias Correction Recall that it is often necessary to correct for the bias introduced into the estimate of MSEP by using M1 (6) to estimate M1 (0). Kenward and Roger (1997) define a bias correction term for multiple dimension linear combinations akin to the single dimension bias correction offered by Harville and Jeske (1992) found in Equation (211). We now rewrite this term to resemble the correction term in Equation (31), in order to maintain consistency in our notation and enable expansion to include linear combinations containing random effects. The definiteness properties of either the bias correction term or the estimate of MSEP containing the bias correction term has not previously been addressed. We will address that issue in Section 3.2.3. We first consider linear combinations containing only fixed effects. 3.2.1 Bias Correction Estimator for Multiple Dimension Linear Combinations of Fixed Effects We let b, again be the IJh element of B(0). Kenward and Roger (1997) state the bias correction term as 1 2Cov(LBLLP) BIASKrr 0 2 Cv (LO',L) (36) BIASKR (0) 1 Y b (36) 2 where where SCcov(LO,LO) 00,80o a2 var(Lp) 2 'cov (L, Lp) o0, o, a2 cov(Lia, L2) 2 cov(LJ, LP) 00,00JUp 02 cov (L,, LI,) So80,80, a2 var (L,) o0,o0, Thus, the knth element of BIASK (0) is i l cov(Lk,LmIJ) S0 cov (L OL,0p) COV(,L 00,0oJ a2 cov LkL ,J) a 0 0,2 co lJ)] B^2(LPLm~~ J We now rewrite BIASR (0) as BIASKR (0) where tr {Ai (0) B (O)} 2 tr{A, (0)B(0)} ... 02 cov(LkO, Lmp) 2 cov(LkO, L.m) 02 cov(Lk,,Lm) a 0ao, 2 cov (LkO, Lm) 2 cov(Lk ,Lm) a0, To confirm the equality of Equations (38) and (36) note that the kmth element of Equation (38) is also {BIASk (0)} (37) tr {Al, tr {A1 (38) BaK1 b 2 cov Lk,Lm) {BASk (O) = tr Akm (O)B(O))= 2 bi, '0 ) '2 cov (LkO, Lm) b2 cov (Lk, L,) + b12 + ..+b (39) aC2 Cov(Lk,, Ln)_ ++b j C 802 which is identical to Equation (37). It has been shown by Kenward and Roger (1997) that Equation (36) reduces to the multiple dimension correction term (C' (0)) when the covariance matrix, V is linear in 0, which gives MR (o) = M1 (o) + 2CR (6) as an estimator of the MSEP. In the next section, we use Equations (38) and (39) to define the estimator for bias correction for a multipledimension linear combination involving fixed and random effects and confirm that the resulting bias correction term continues to reduce to the correction term in linear covariance cases. 3.2.2 Bias Correction Estimator for Multiple Dimension Linear Combinations of Fixed and Random Effects The extension to include random effects is a straightforward but tedious task. However, it is important to explore fully the functions involved to determine when the term simplifies to the correction term approximation in Definition 3.1, and also to assess the effect on the definiteness properties of the MSEP estimator. The generalized bias correction term has the same form as in the fixed effect case: 1tr {A 1(0)B(0)} ... tr {AI, (O)B(0)}) BIAS 9(0) = (310) 2 Atr (A, (0) B(o)} tr {Ah,, () B(O)} but now, A covO(LO + Mki, L + M u) Ak,(0) = 0' a' cov(LP + Mi, Lj + M u) 02 cov(Lkp +M Ljm + M iu) a02 0 0,0, (311) a2 cov(Lk +Mki, LmP + Mm ) a2 cov(Lkp+Mki, LmP + Mmi) 8600 06,2 Now considering a single element of this matrix we see that C2 COV LkO +Mki, Lm +Msfu) A(Aj) (0)} 10,0 a2 [cov (Lk, L mP)+cov(Mii, Lm) + cov(Lk, Mrii) + cov (Mkii, Mii)]  c2 a)] 2 c)1 (312) co (LO2, L]P + COV(Mku, Lm 0 J 0 J 1 2 + a v(LkpM i+) [o(Mk,M)] 3 4 We now investigate each piece of Equation (312) using Littell et al. (2006) as a reference for the covariances involved. Consider Part (1) of Equation (312): (1) o= c L cov(y L,,) =, cov ,XV XVVy, L,,XV XVIl _ [L (X'VlX)1 X'VX x)Vx1 L', = 'VX L' 8= L (XV1X) 1t X 1v VX(XX 1X)Lm =L (XViX) XV1 W VIx(xvx) XV1 NVx(x'VX)1 L'X L (xVlx)' xVv1 VVI x(xVx) L't+ eL( (XVlX) XrV1 NGV, NVxxvx( e4 +Le (XVX)1 XV1 N XVx1 XV(X1 VxX L' X ( vl(vl)V L =CqVI1PC, C VICC CVPC,, +C VPC, +L, (XVX) XVi a Xv X(XVX)1L' (313) = v'(IP),, Cqv(iP)c ,,+4(xVx)V' Vl v lx(xy'x)l' where C' is as defined in Section 3.1.2. Now moving on to part (2) of Equation (312), we have O MzV1xvx1 e (2) = cov( j i,LB) N= GZVR j(X l1x = M zrvx(x'Vx)i L' +MGZV aVx(xV1X)I L' (314) +MGZVX(XV XVXVj a X XVxX)i1 Lm = ,Z'V~1 V1XXVXL k aO0 ao M k zV 1 V V OV V 'X(X v x Lf +MkGZ'V O VV VX~XVX) Lm a o ao, MkGZ'V 1 V V'X(XfV'X) LL aoj ao0 MkGZ'V l Vtx (xv tx)1XfV1avvx(xfv x)1L aoj00 1av 1X Lf +MkG~ V 1 v v lX (xlr x)l x fv d v lx(x fv lx) mb 1M GZ'V1 v Mk zvV'X(Xfvx) l Xfv l aoj vx(xfv 'x) f 0 Lb ~f l ) MkG'V '11'~ ao, o~VX 1 V1(XVX) a~ ao M kG~ fV 'X(X fv lx )X f la v lvx v ) fv l v lx~xlvlx ) m +MkGZ'V 1(XVlixv VXV1v 0, aoV V M~z' lxxv l) x l1 ov ov 1 L frl'3 VXX IIX~ 1L MkGZ'V 1X(X'V 1X)1 X'V'?V V1 aV V1X(X'V1X) Lb +MkGZfVIX(XY1X) XfV1 x0)1L0,x1 x l xvv aok ao ao xvx a~av 1MGZ'V1 V SMkGZ'V 1X(XfV iX) a 2V V X(XfV'X) D'V 1Cm,,, + D' V1PCm, E'VC, +E'VVPC E'VC', + E'VPCm, F'V1CmI + F'V1PCm, F'V1C, + F' V1PCm ^h ~ ~ ~ A m' k^i ^i' ^^ h*i Ct ^ mi 2 V Xt'XV _1 1 L' MkGZ'V1 VXVX'VX L' aoa 9019(X V1X iL MkGZ'V1X(X'VlX) XV1 2V V(xvlx) L' 8o, 8o0 D V1 (I P) C, EV1 (IP) C,   D V1(IP)Cm, E V1 (I P)Cm, k2V 0(xlx) 1 L' MkGZ'V1 V X V X'V1XV) L aoao 90(,90^iL MkGZ'V1X(X'V1X)1 XV1 d V VI(X'VX 1 L' . w' an 90,9, v" where D E,, and F' are as defined in Section 3.1.2. Due to the symmetry of parts (2) and (3) of Equation (312), we combine (2) and (3) as follows: (2)+ (3) (D, +E + F+ ,) V(I P)CC, (D, +E ++ F) V1 (IP)C,, (317) CV1(I P)(Dk +EE, +Fj) CVI(IP)(Dk+Ek +Fk) ( a2G V( aO^GO}_ { ao o\ and g, , are linear functions of 00,00OJ 02G 0oo o,0 02v and respectively. aoo ao Part (4) of Equation (312) tediously completes {Akm(,j) (0) as follows: (315) (316) where where /f  80,80J D'VC, + D'V PC bV CMi h CM M 2G Z'VX (X'VIX)I L' M 0,0, v F,'V1 (IP) Cmj F V1(IP) Cm, Mk z0~J'xx'VxX" L'  M 9,9 v / i (4) = cov (M,u, M )] M [G GZ'V ( P) ZG] M' 82G 82G = M, G M' Mk Z'V1 (I P)ZGM, k 60 ] m k80 80 m 02G 02p MkGZ'V1(IP)Z M' +MkGZ'V1 ZGM' aao, ao o, a a2V1 MkGZ' (IP)ZGM' D' V(IP) Dj D V(I P) Dm, EV1(I P) Dmj E' V1(I P) Dm, (318) F',V1(IP) D, F V1 (I P) Dm, D' V1 (I P) Ej D V1 (I P) Em DVIP)F D' V1(IP)F +Ef'V1PEj + E' V1PEm, +E V1PF + E' VPF. Now note that CP C X Lx(Xvlx) xv1' 80, 80, = x(x'vlx) x'v Ov Vx(xvlx)l x'v x(xvix) 1 xv1 v v1 which leads to 02p = x(xvix1 xv1 Ov VIx(xvix)1 xv1 v imx(xvix) 1 x,v1 X(XVlX) X ) V1 OV V1 V V(X'V)lx X1 XVX1 +x(Xfvlxx1 XVX 02V viX(XV1xX)XV1 ao a9 J +X(X'Vlx)1 XV1 V V1x xvix1 XV1 V1 xfxvx1 xV1 ao, a00 X XViX 1 X'V1 ViX XViXy1 x'V1 V Vi1 +x(x'Vlx) x'v1 Ov Vi vlx (x lx)lx' 1 v 1 l S, 0 9 XX'Vlx)l X'V1 V V XX'Vlx l X'V1 OV V1 a, a e, +X (X'V X)1 X'V1 V V1 V V1 +x+ X(X'V X)1 X'V1 OV V1 V V1 a0 00, 00, a00 x(xvlx)lx'v1 2V 1 Now we have 02p MkGZ'V1 ZGM' =F'V1(IP)F F'Vi (IP)Fm, FV1 (IP)E Fi'V1(IP)E, +MkX(X'VlX) X'V1 2 VixfxVix)1 XViM' 0oo v (2V x' MkX (XVlX)' XV1 ViM' , a2V 1 To complete Equation (318) we also must investigate : 90,90 a2V1 1 'V V 1 VV1 V 1 1 VV1 V V1 V V1 _V1 2V V1 ao0,ao ao0 a0, ao, ao0 ao0 ao, ao0,a This gives us 82V1 BV 9V MkGZ' (IP)ZGM' = MkGZ'V1 V1 V1(IP)ZGM' GV 1V 1 2V V 1 +MkGZ'V1 VV1 V V1 (IP)ZGM' MkGZ'V1 V (IP)ZGM' 0oj 00, o 00,08J k71 1 1 1 1 1 =E'V1Em +EV1E +EV VF +EVF MkGZ'V1 V (IP)ZGM'. E2V+VE Fo a2v1 2p Utilizing these expressions for and  in Equation (318) results in a2G a2G (4) = Mk M' Mk Z'V1 (I P) ZGM' S90,90 m k 00,0 m M2G 02p MkGZ'V1(I P)Z MG +MkGZV1 2 ZGM' ao, a~a ao, aoo a2V1 MkGZ' (IP)ZGM' D V1 (IP)Dmj DV1 (IP)Dm, EV1 (I P) Dmj EV1 (I P) Dm, F' V1(I P)Dmj F' V1 (IP) Dm, D'fV1(IP)Emj DfV1 (IP)Emz D',V1(IP)Fmv D V1(IP)F, E',V1(I P) Emj E'kV1(I P)Em, E',V1(IP)Fm E'kV1(IP) F, F' V1(IP) Ea F' V1(IP)Em, 82V F' V1(IP)FM + F' V 1(IP)F + MkGZ'V1 00V1(IP)ZGM' +MkGZ'X(X'V'Xxl X'V1 x 2V V XXVX1 X'VZGM' ( l2Vm (319) MkGZ'X(X'VIX) XV1 2 V'ZGM' (319) which reduces to (4) (Dk,+E +F) V(IP)(D +Emj + Fmj) (D, +E, Fk' V1(IP)(D, +Ek, +Fk,) (320) 82G ( 82V +f2 2 9 Piecing together (1) (4) in Equation (312) results in: Ak( cov (LP + Mkii, L m + Mmiu) (A y) (0)) =  S(Ck, + Dh + Eb + F, ) ) (I P)(C, + D,, + Emi + FmJ) (321) (C, +D +Ek +F )'V1(IP)(Cm +Dm +E +F,) +f 02G 02V +f a +g av . S2G 2V When G and V are linear in 0 the functions of f and g  will be zero. In that case (Ckl + Dkl +Ekl +Fk) Akn (0) =2 2 F (322) (Ckr +Dkr +Ekr+ Fkr) xV 1(IP)[(Cm +Dm +E, +F F) . (Cr +Dm +Em +F, )]. Comparing Equation (322) to Equation (32) gives the desired result that in the most general case involving multiple dimension linear combinations of fixed and random effects BIASKR (0) = C' (0) when G and V are linear in 0. Thus, just as in the single dimension fixed effects linear combinations, when G and V are linear in 0, MK (o) = M, (0)+ 2C"K () is an estimator of the MSEP of the linear combination of fixed and random effects that accounts for both the correction term and the bias correction term. 3.2.3 Definiteness Properties of the Estimator for MSEP for MultipleDimension Linear Combinations of Fixed and Random Effects Now that all pieces of the estimator have been defined in the multiple dimension setting an estimator for the MSEP can be defined as MK' (0) = M ()+C (C ) + BIASK (6) (323) It has not been proven that this MSEP estimator, either in single or multiple dimensions, is nonnegative definite. Just as in the single dimension case, we have shown above that when the variance matrix for the model is linear in 0, the bias correction estimator, BIASR (0), reduces to the correction term, C (0) which we have shown is nonnegative definite in its most general form. Hence the simplified MSEP estimator, M M (0) = M () +2C () (324) is nonsingular and therefore a valid measure of prediction error. However, when the covariance matrix, V, is not linear in 0, neither the bias correction term nor the MSEP estimator including the bias correction term has been previously shown to be nonnegative definite. In Chapter 4, by exploring the special case of the balanced oneway random effects model, we provide a counter example to prove that the bias correction is, in fact, indefinite. Indeed, we show that in a single dimension case the estimate of MSEP given in Equation (323) can be smaller than the naive estimate, M (9). From Equation (321) it is clear that the indefiniteness of BIAS' (0) is S2G ( 2V caused by f and g Note that it is not necessary for the bias correction term to be nonnegative definite for the overall estimator of the MSEP to be nonnegative definite. In fact, it is reasonable for the bias correction term to be indefinite since it is estimating the bias of M, (6), which is also indefinite. In Chapter 4, we provide a single dimension case where the negativity of the bias correction term causes the entire estimate of M1 (0)to be negative, proving that M (o)+ BIAS" (o), the overall MSEP estimator for M1 (0), is indefinite and a poor choice for estimating the MSEP. 3.3 Transform Invariance of MSEP Estimator Another important issue concerning these estimators is transform invariance. Consider two parameterizations of a model, 0 and D = g() Kackar and Harville (1984) showed that the approximation for the correction term in the single dimension case, tr A(0)B(0)], is transform invariant under the conditions that 4 = g( ) and that B* (0) =[J () B(g(0)) J(0) where B*and B are as in tr[A(0)B(0)] and J(0) is the Jacobian of the transformation. Both of these conditions are satisfied by using either REML or ANOVA variance component estimation and with B(0)= I 1 (), the inverse of the observed information matrix. This proof applies to the multiple dimension case, as we have shown that the multiple dimension correction term approximation is simply a matrix composed of the single dimension approximations as the components. Since each element of the matrix is transformation invariant, the entirety is transform invariant as well. It is also easy to show that the naive estimator of the MSEP, M1 (0), is also transform invariant under the condition that F = g( ) The invariance of the naive estimator indicates that the true bias of M, (6), E[M ()] M1 (0), will also be transform invariant (again with the condition, 4 = g ()). Thus a desirable property for the estimator of the bias correction term is transformation invariance. Clearly, BIAS" (0) is not transform invariant. In Chapter 4, we demonstrate the lack of transform invariance for a single dimension linear combination in the one way random effects model. This is an undesirable quality of the bias correction term estimator. We will investigate its effects on the balanced oneway random effects model in Chapter 4. CHAPTER 4 BALANCED ONEWAY RANDOM EFFECTS MODEL Many of the methods summarized in Chapter 2 were developed only for estimating and testing fixed effects (e.g., Giesbrecht and Bums 1985; Kenward and Roger 1997); however, the methods are often extended to include random effects without documented justification (SAS 2003). In Chapter 3, the elements involved in extending these methods to the most general cases were derived. By looking at the simplest model including random effects, namely the balanced oneway randomeffects model, we can examine if the use of these methods is as straightforward as previously claimed or if additional issues arise. The simplicity of this model often allows closed form expressions to be obtained for the approximations and estimators of the MSEP and their expected values, in addition to the true value of the MSEP being estimated. Peixoto and Harville (1986) derived closed form expressions for the true MSEP for a class of models including the balanced oneway random effects model. Similar methods are used here to enable direct comparisons of various MSEP estimators. By using this simple, balanced model, we can measure the result of including random effects on the MSEP estimates, as well as identify issues that may be more difficult to handle in more complex models. Consider the model: y, = + + e,, i = 1,...,k, j= 1,...,n e N( 0,o2), a N(0, 2) where oc < / < o, o2 > 0, o > 0. We use the parameterization 0 = (o2, ) where 2 v = 2 and consider the problem of predicting the mean of the ith level of the random a2 + a effect. Note that the variance matrix, V, is not linear in this parameterization, nor is this the parameterization used by SAS". This parameterization is chosen to demonstrate the impact of a nonlinear parameterization on the bias correction term and for consistency with past work on this topic. Comparisons will be made to results under the alternative parameterization, D = ("2, ) for which V is linear. The BLUP and the naive MSEP, M, (0), for t = a are straightforward to derive, as demonstrated in Kackar and Harville (1984, p. 857): (0)= (v)(y,), where =, andy =(42) n k M,(0) = 1 [l +(k 1)] (43) nkv Noting that (0) is linear in 0, the accuracy of the KackarHarville approximation of the correction term depends on four issues: 1) using the exact MSE of 6 versus an approximation, such as 1V (0), for B(0); 2) the appropriateness of the Taylor series approximation, depending in part on how close 0 is to the boundary of 0; 3) whether 6 is unbiased for 0; 4) the proximity of cov d (y; 0) d (y; 0)', 0) to zero. We can evaluate these four criteria to determine their effects on the accuracy of the KackarHarville approximation. We shall examine each of these criterion using both REML and ANOVA methods of variance component estimation. 4.1 KackarHarville Approximation Based on REML Estimation Method Using REML methods to estimate the variance components results in 0 = (i2, i3) where 2 = m, ifma > m (k 1)m +k(n1)me im < nk 1 Smin(me / m, 1) (44) where me = 1 and ma = k(n 1) a k1 (e.g., Kackar and Harville 1984; Searle, Casella and McCulloch 1992). Note that these estimators remain within the parameter space, 0, by definition. The EBLUP for t=a is = )= (1 )(y, y) (45) and the naive estimate of the MSEP of the EBLUP is M(2() = [1+(k 1),6]. (46) nk6 The true correction term in the REML case, as derived by Kackar and Harville (1984, p. 731), is / c v (nk1) C (0)= E[ ]2 = (k 1) (47) nk where Y' ( ) 2 (V 1) 1 SB(ve,v)u v ve (v (1) v(v1) { S i. v Iv (ve +1) (1 2)+ (1 )2f v 1 + 1 ( I, + ve + 1) 22 2 + ( 2 + V +V Ve a 1) V 2 V + V and B(,b)= F(a)F(b) (a,b)=[B (a,b)]1 t (1t)b1dt F(a + b) 0 k(n 1) k1 V Ve V e 2 2 Ve +VV This results in a true MSEP, MR (0), where M (0) (0) + CR (0) M (0)+ c2v(nk 1) (48) nk (Kackar and Harville 1984, p. 857). In this model, the exact MSE matrix of 6 is tractable when using REML estimates for the variance parameters. Thus assuming that 0 is sufficiently far from the boundary of 1 and choosing BR (0) to be the exact MSE matrix of 6, we can assess the accuracy of tr [A(0)BR (0) for CR (0) by simply assessing the proximity of cov d (y; 0)d (y; 0)',(0)(0) to zero. To evaluate tr [A(0)BR (0)] first note that A (0) = 22 (49) 0 a22 and BR (0) E (410) E( ^(2 _2)] E(3 v)2 var = var(y_ 2 (k1). As shown by Kackar and Harville (1984, p. var nkv v)2 = v2 where 'Ve (1 f)v 2 Va(Ve+1) 1 Bo B(v,,va)v (v 1) v (v 1)(v2) v(v 2) + (2 (V, + 1) 2v, (1 2) (1 )2 +I (v,) (vo,2)(v2)(v1) v2 v 2 tr[A(0)BR (0)] = a22vc = 2k 1) nk A KackarHarville approximation to the MSEP is then o2v(k 1)^ M (0) + (k 1) nk (411) (412) (Kackar and Harville 1984, p. 857). Now note that the only nonzero component of cov d (y;0) d (y;0)', ( ))' is cov[(y y)2,( u)2 =E(y, y)(v)2 ]E[(y,y)2[E(Ov) Kackar and Harville (1984, p. 731) show that E[(y y)2(vu) =] CR(0) k = cR (o) where a 22 731), E(3( As a result, It is also easy to verify that E (y y)2 = k(2 +n 2), since ~ Thus nk a +no, o v (nk 1) oC ug (k 1) nk nk Notice that this is simply the difference between the true correction term in the REML case, CR (0), and the approximation, tr [A(O)BR (0)]. This is only true when the exact MSE of 6 is used for BR () Thus the condition for CR (0) = tr [A(0)B (0)], i.e., the condition for cov d(y;0)dty;0 ),( 0 00 0)'=0 is S(nk1)= k (k 1). (414) Figure 41 shows Equation (413) as a function ofu for several values of k, holding n = 6, with the condition in Equation (414) met when the functions cross the zero line. Note that throughout, without loss of generality, we hold C2 = 1. As k increases, the functions move closer to zero, indicating that the KackarHarville approximation for CR (0) improves as k increases. However, as k increases, the amount of information about the random effect increases, reducing the uncertainty about the covariance parameter estimates. Hence CR (0) tends toward zero and accounting for the correction term becomes less important, rendering the correction term approximation irrelevant. In the remainder of this chapter, the analysis will focus on a model with k = 6 levels of the random effect to demonstrate the impact of the different estimators for the MSEP. In most models, the exact MSE of 0 is not tractable and an approximation of the MSE of 0 must be used. A common choice is I'1 (0), the inverse of the observed information matrix, with 0 replaced by 0 in practice. Consider the KackarHarville approximation of CR (0) where B(O) is the asymptotic MSE of 0 IV1 (0). Kackar and Harville (1984, p. 857) show that tr [A(0) ()0)] 22(kn 1) (415) k2n(n 1) where 2( 2)2 2a"2 k(n1) k(nl1) T1( (k) (416) 2U20 2v2(k 1) k(n1) k(n1)(k1) Figure 42 compares the two KackarHarville approximations in Equations (411) and (415) with the correction term, CR (0), for k = 6 and n = 6. The success of the approximation is greatly influenced by the value of v. For small values of v, which correspond to large values of a2 relative to "2, tr [A (0)11 (0)], is superior to tr [A (0) B (0)] for approximating the true correction term, CR (0). However, as v moves toward 1, which corresponds to the value of 02 growing relatively smaller, the roles reverse and tr[A(0)BR (0)] becomes the better approximation. In some cases, the choice of how to measure the variability of 0 may cause severe overestimation of the correction term, in turn causing a greatly inflated value for the MSEP estimate. SAS PROC MIXED uses tr [A(0) 1 (0)] which overestimates the true correction term by over 100% for half of the parameter space. Using an inflated estimate for the MSEP could lead to overly conservative interpretation. Because the correction term approximations in Equations (411) and (415) make use of the unknown variance parameters, estimates of the variance parameters must be substituted into these approximations. Note that there are now two levels of error being introduced into the correction term estimators: first, the true correction term is approximated by the Taylor series methods to yield the KackarHarville approximation; second, the variance parameters involved in the approximations are replaced by their estimators. In the case of tr [A (0)I1 (O) there is a third contribution to the error in that V1 (0) is an approximation of the MSE of 0. Substituting the REML estimates of the variance components into Equations (411) and (415) gives two possible estimators for the correction term: tr [A () BR (6= k (417) nk and [r A()1 ()1 2&2 (kn 1)8 Lr A (o)I() k 1)(418) k 2n(n 1) It is suggested by Prasad and Rao (1990) that the bias introduced by substituting variance parameters in these approximations is o(k ). To assess this assertion, we investigate the accuracy of Equations (417) and (418) by considering the expected values of both estimators. The expected value of Equation (417) is not available in closed form, so a simple Monte Carlo study was performed to evaluate the accuracy of the estimator for the oneway random effects model in Equation (41) with k = 6 and n = 6. Using the RANNOR function in SAS, 10,000 independent sets of e, and a, were generated from N(0,o2 = 1) and N(0, ua) for several values of a respectively. The values of a2 correspond to v = {0.05, 0.1, 0.15..., 1.0}. Setting i = 0 10,000 sets of {y, : i = 1,...6, j = 1... 6 were created from the model y, =a + ej The Monte Carlo value of E tr[A (0BR (Oj=) E 2 (k 1)o and the Monte Carlo standard error were obtained from the 10,000 replicates. Selected results of the study are contained in Table 41 (standard errors contained in parentheses) along with the values of CR (0) (Equation (47)) and tr[A(0)B (0)] (Equation (411)). Table 41. Accuracy of KackarHarville estimator (417) for CR (0) V 0.2 0.4 0.6 0.8 1.0 CR (0) .01537 .01890 .01758 .01660 .01850 tr A(0)BR (0)] .04447 .03384 .01950 .01097 .00954 E AtrA(0)BR (0} .03567 .02652 .01951 .01513 .00954 (.00013) (.00013) (.00011) (.00009) (.00002) **Monte Carlo value. Standard error of simulation in parentheses. Figure 43 shows the results of the Monte Carlo study graphically. The accuracy of the estimator again largely depends on the value of v. For small values of v, which correspond to values of a2 that are large relative to 2, the estimator for the correction term is not accurate with expected values more than twice the function it is estimating. However, as v approaches 1, the estimator for the correction term significantly underestimates the true correction term. The expected value of Equation (418) can be expressed in closed form if we assume normal distributions. First define k(n )me (k )ma Ue e 2 + 2 W = +2 e +U S = Ue + Ua It is easy to show that ue and u, are independent chisquare random variables with k(n1) and (k 1) degrees of freedom, respectively (e.g., Searle 1971, p. 410). Also, w and s are independently distributed where w has a chisquare distribution with (kn1) degrees of freedom, and s has a Beta(ve, Va) distribution (e.g., Johnson and Kotz 1970, Sec. 24.2). Now note that if m, > me then me <1 so that = min e, = e but when ma < me, >1 so that = min ,1 = 1. ma m ma ma ) 0 ifma >me [0ifm, < m, E tr A V kx k22 n) E^u} k^7 1 K mame) +E ( 3(mo,me) . k2 n) a Noting that if m> me then s < so that 3(s, )= < =(ma,me) 0 if s > 20ifm m2 (2Va ws 2 and that = 2 we see that ma k(n 1)v, (1 s) we see that / e 2 V 2 [ WS2 (1) E (n,,e) = E 9(s, ) ( nmz k(n 1)v, 1s k(n l)ve ) I S I (420) K k(kn  k(n l)v, a~  (kn  k(n 1J)Ve J s2 1 ( s) 1 0 ls B(,ve, V) B(v + 2, v 1) 1) .. I,(ve + 2, va Now note that ue a sw sw so = and ma k(n 1) (a2 +na2)(1 s)w = which leads to (k 1) E (k 1)n M'(1g M IM ))= E .m 13(m n( ,n))) +E k(n)Me (1(M, me)) \ kn1 1)ma + k(n 1)me1 (m,me kn1 (421) C2 + 2 E(w(l1s)(1 (s, )))+ 2 E ws(1 (s, ))) kn 1 knI1 (2 +n2) (1 S)S d a ds SB(ve, v) B(ve,va) ) B(ve, v +1) + 2 B(ve +1, v) + +n ) [Iv, +)] + [1I,(V, + ,V))] B(v,,v,) B(vev ) ( 2+na ) Va Ve + Va IT(Ve,V +1)] +2 e [1I (Ve+1,Va)]. Ve + Va Substituting Equations (420) and (421) for (1) and (2) in Equation (419), respectively, we have 2(kn 1) k2n(n 1) 2(kn 1) { 2vu(kn 1)vB(e + 2, k n(n 1) k(n )vB( 1) ,(ve + 2,va Ve, va) (422) 2V V +a e+ [I,(v, +1,v,)]+ v [1I,(ve,v+l)] . Ve [1 (Ve + 1,V)] +01 V [ Ve +Vo Ve + Va (2)=E (k tr (A(6)1r (6))] Figure 44 shows the accuracy of the KackarHarville estimator (Equation (422)) for the KackarHarville approximation (Equation (415)) and the correction term (Equation (47)) holding k= 6 and n= 6. Just as for the approximation, the accuracy of E r (A ()) ()) depends largely on the value of v. While tr [A (0) ()] is a fairly accurate estimator for tr [A(0) 1 (0)1 over the entire parameter space, it may severely overestimate CR (0) for larger values of v. We wish to assess the assertion of Prasad and Rao (1990) that E[tr(A(O), ())]= ,A(0) 1 (0)+o(kk ) forthis model. Notingthat lim Y = yieldsthe result that lim I, = 1 so that k> E[tr (A() 1 ())]tr [A( 0)1 (0)] 2(kn 1) (kn )vB(ve + 2, v 1)I, (v +2, v 1) 1 k2n(n 1) k(n )vB(v, va) +a2U1 v [l_ (V +l ]+a202 [1I (V",v+)] Ve+Va Ve+a J =(kl) confirming, for this example, the assertion of Prasad and Rao (1990). These results also show the importance of studying the effect of substituting variance parameter estimates for the unknown values in the approximations for the correction term, the impact of which has been typically overlooked. In the case of tr [A(0)BR (0)1 substituting variance parameter estimates improves the accuracy for CR (0) (in the long run). However, in the case of tr [A(0)11 (0) the substitution creates more inaccuracy for CR (0) for most values ofu. Figure 45 compares the accuracy of Equations (417) and (418) for the correction term. The plot shows that tr A (0)BR (0) is more accurate (in the longrun) than tr LA (0) 11 (0) for ) > 0.30; however, for u < 0.30, the reverse is true. This is the same phenomenon that we noticed with the approximations in Figure 42. Note that SAS PROC MIXED by default uses 11 (0) with REML estimates for 0 in the calculation of the MSEP estimator. 4.2 KackarHarville Approximation Based on ANOVA Estimation Methods The best choice of method for estimating 0 may depend on the model of interest, the level of balance in the design, and the goals of the analysis. Stroup and Littell (2002) demonstrate the effect of different estimation methods on the power and control of type I error for hypothesis tests on the fixed effects in analyzing an unbalanced, multilocation experiment. Their study indicates that REML and ML are not always the best choices when negative estimates of variance components are likely, and may lead to inflated Type I error rates and low power. The impact of the choice of variance component estimation method and negative variance component estimates on the KackarHarville approximation has not been investigated thoroughly. In Chapter 5 we elaborate on the impact of negative variance component estimates. To begin, however, we develop the correction term, several approximations, and the expected values of the estimators under ANOVA estimation for the balanced oneway random effects model. The ANOVA estimators of 0 = (2, ) are = (2, j) where k n yZc, )2 =1 1 =1 me k(n 1) (423) (2 + n(,2 m a a ma me n Sma k =1 l k1 These estimates are obtained under the METHOD = TYPE 3 option in SAS PROC MIXED. The true correction term, CA(0), is derived under normality assumptions as follows: k CA (0)= E (y,y)2 )2 (kn)E nE (y t)2i )2 c2 +n2 (k )ma + o [2 [ )2] a2 E = E 2u () kIn a2 kn I 2n  S aE [w(1s)(_t ) kn '1 72 2 av+ 2E w(1 kn .2 2 c2+n a v2E w(1 kn s V(1) s) Ve (1 s) 5 1)2 <"< ^ y S2 2E(w)E (1 kn SC2E(kn 1) E 1 asS(1 1 2, kn Sve (1s) by independence of w and s, and noting that w ~ k Recalling that s Beta(v, ,) we find that vs V 2S2 E ( ls) s 1 =E Ve(s) ve (1S) vs 2 + (1 s) vB (ve +2,v v2B (ve, v ) VB(ve +2,v 2B ( v qB(v v) 1) + V V + Va 2 2 +V 1) Va Ve + Va VaS ve( s) Thus the correction term is a 2v(kn 1) vB[(v, +2,a 1) V, kn L vB (v,vo) +vo f^ (4 24) Skn v, (v (424) 2(kn1)(k 1)C 2 k n(n1)(k 3) This gives the true MSEP for i&, M (0) = M (0) + CA (0) S2(kn 1)(k 1) 2 (425) = M, (Q+ u. k n(n1)(k3) To evaluate the KackarHarville approximation for the correction term, we need to determine how to evaluate B(0). We consider three alternatives for B(0): the exact MSE matrix E o 0o) ( o), the variancecovariance matrix of 0, and 1 (0). Recall from Equation (49) that due to the form of A (0) in this model we are only concerned with assessing the variability of 5 in order to evaluate the KackarHarville approximation to the correction term. We begin with the exact MSE of u, E [? v] Noting that m = Fkn1),k it is ma E k v(1)\ 2 (k 1) (kn 3) easily shown that E(?) = V (u) = (k 2k and Sk3 k(n1)(k 3) (k 5) and U (k1)2 (k(n1)+2) E v (k 1)2 (k(n1)+ 2) After algebraic simplification this leads to Sk(n 1)(k 3)(k 5) E[ v]2 = E ( 2)2vE ()+2 k k(kn+3n 5)+ ~1 (426) k(nl)(k3)(k5) Letting BA (0) = E ()( 0) ] we have the following approximation for C (0): S( B 2(k1)(k(kn+3n5)+1) trLA )B )] k n(n 1)(k3)(k 5) (427) Substituting the ANOVA estimates in Equation (423) for 0 yields this estimator for CA (0): S2(kl) (k(kn + 3n 5)+1)( S[A(6)n k(n 1) (k 3)(k 5) To assess the accuracy of Equation (428) for estimating Equation (427) and CA (0), we 2 2 2 VCUU Va WS evaluate the expected value. Recalling the relationship = k(Tv where w and s ma k(n 1)v, (1s) are independently distributed with ~ 1_, and s ~ Beta(ve, va), we have E \~ \=E E[w]E Im a I= k(n l)v, (1s) ( 2 v B (v, + 2, va 1) =k(n )v (kn 1 B(v,, va (429) (k1)(k(n1)+2)C k(n1)(k3) Thus, r A(F)B (u )= 2(k 1)2 (k(kn + 3n 5)+ 1)(k (n1)+2) 2 E trAn(n 2 (kA 3)2 (k_ 5)v. k n(n 1)(k 3)(k 5) (430) Again, we confirm the asymptotic assertion by Prasad and Rao (1990), noting that E tr [A()B,()] tr [A(0)B, (0)] 2(k 1)2(k(kn+3n5)+1)(k(n 1)+2) 2 2(kl)(k(kn+3n5)+1) 2 k3(n 1)2(k 3)2(k 5) k2n(n1)(k3)(k5) 2(k1)(k(kkn+3n5)+1) (k1)(k(n1)+2) 1 k2n(n 1)(k 3)(k 5) k(n1)(k3) J o(k 1) The second alternative for B(0) is the exact variancecovariance matrix of 0, B* (0). Because 5 is not unbiased for v, this is not the same as the MSE matrix for 0. As noted above, 2t2 (k 1)2 (kn 3) V () 2 = )2(k 3) which leads to the following approximation for CA (0): k(n 1)(k 3)2(k 5) 2(k 1)3 (kn 3) tr A(0)B* (0)= C2V. (431) Skv 2 n(n1)(k3)2(k 5) Again substituting the ANOVA estimates for 0 in Equation (431) yields an estimator for CA(0): tr[A()B (62 (k1)3 (kn 3) tr A()B ()1= ~2)_3) z. (432) A u M k2 n(n1)(k3)2(k5) Using Equation (429) we have Etr[A()B ()} 2(k1)4 (kn3)(k(n1)+2) Etr A 0) B 0)) = v (433) Sk3 n(nu l)2 (k 3)3 (k 5) Again we find that tr [A () B ()] tr [A (0)B ()] 2(k )4(kn3)(k(n1)+2) 2(k )3 (kn 3) k3n(n1)2 (k3)3 (k 5) k2n(n1)(k3)2 (k5) 2(k )3 (kn3) 2 (k 1)(k(n1)+2) 1 k n(n 1)(k 3)2 (k 5) k(n 1)(k3) =o(k ). The third alternative we consider for B(0) is to use the inverse of the observed information matrix, IV' (0), as an approximation. Note that I1 (0) is the same as the asymptotic variance of the REML estimators of 0, in Equation (416). Thus the KackarHarville approximation with ANOVA variance component estimates using V1 (0) is the same as in the REML case in Equation (415), i.e., tr [A(0) (0) = 2(kn 1) a2 (434) k k2n(n 1) and the estimator formed by substituting ANOVA estimates for 0 is tr A( )T1' 2(kn 1) (435) L ] k2n(n 1) Using Equation (429) we have k2n(n1) 2(kn 1)(k 1) (k (n 1)+ 2) (436) = 0.2/ ~)  o k3 n(n 1)2 (k 3) which again yields E{tr [A )] t[A(o)I1()] 2(kn1)(k1)(k(n1)+2) 2(kn 1) 2 k3n(n )2 (k 3) k2n(n 1) 2(kn 1) (k1)(k(n1)+2) 1 k2n(n1) L k(n1)(k3) o (k ) Figure 46 compares the three KackarHarville approximations in Equations (427), (431) and (434) to CA (0) for k = 6 and n = 6. Clearly tr [A()I 1 (0)] more closely approximates CA (0) than the other two approximations, for all values of v. Thus, although we would expect 11 (0) to be less accurate than BA (0) or B* (0) in evaluating the MSE of 0, especially for small values of k, the approximation of CA (o) utilizing 11 (0) is more accurate than the either of the other choices across the entire parameter space. In fact, the Kackar Harville approximation utilizing the exact MSE of 0 is the least accurate of all three approximations. Note that this is in contrast to the results from REML variance component estimates in Figure 42, where the best approximation depended on the value of u. Figure 47 compares E{tr[A())BA (6)) (Equation (430)) to tr[A(0)BA ()] and the correction term, CA (o) The bias for the correction term introduced by replacing the unknown variance components with ANOVA estimates in the correction term approximation is significant and increases as v moves toward 1. Figure 48 demonstrates a similar result for the Kackar Harville approximation and estimator utilizing the variance of 0. E tr [A () B* (0)] (Equation (433)) is again inflated over the approximation and they both overestimate the true correction term increasingly in v. Figure 49 shows a much different result than the previous two figures since the accuracy of the estimator utilizing I' (8) (Equation (436)) is improved over that of the approximation (Equation (434)). While tr [A( 0) (0) underestimates CA (0) by up to 50%, E trA(0) ( ) is much more accurate and consistently overestimates CA (0) by only 6.67% across the entire parameter space. Figure 410 compares all three estimators (Equations (430), (433), and (436)) relative to CA (0). Clearly the KackarHarville estimator utilizing IV' (8) is far more accurate than either of the other two choices. This is counterintuitive: the KackarHarville estimator using an approximation of the MSE of 0 outperforms the other estimators using exact measures of the variation of 0. Note again that this is a different result than seen with REML variance component estimates where the results were dependent on the value of u. There, we saw that the KackarHarville estimator utilizing the exact MSE of 0, BR (0), often performed better than the one using an approximation of the MSE of 6, I1 (0) (see Figure 45). The choice of which KackarHarville estimator for the correction term is best, therefore, depends, in part, on what method is used for variance component estimation. The final step in this assessment is to compare the ANOVA results to the REML results. Figure 411 compares the accuracy of tr [A(O)I' ()1 (Equation (436)) for CA (0) with the accuracy of the two REML estimators, tr A (0)BR ()] (Monte Carlo expected value) and tr IA(0)I' (0)1 (Equation (419)) for CR (0), with k= 6 and n = 6. The ANOVA estimator utilizing the information matrix, tr A () 1 (o)1, appears to perform better than either REML based estimator, although the added variation to the MSEP of i is larger for the EBLUP with ANOVA variance components, i.e., CA (0) > CR (0). We can assess the accuracy relative to the appropriate correction term to better compare the performance of the estimators. The relative bias of the two REML based estimators of the correction term and the ANOVA based estimator with 11 (8) is tabulated in Table 42 and graphically depicted in Figure 412. Relative bias was calculated as E t[AB]) C(0) C(0) The ANOVA estimator of the correction term has consistently lower relative bias than either of the REML estimators. Thus, although the variation added to the MSEP by using ANOVA variance component estimates is larger than when using REML variance component estimates, a more consistently accurate estimate of the added variation is available in the ANOVA case. While the accuracy of both REML estimators depends largely on the value of u, the ANOVA estimator is significantly more accurate across all values ofu. This special case has demonstrated the importance of the choice of variance component estimation method, the choice of measure of variation of 0, and the added variability of substituting parameter estimates for unknown variance components in the correction term estimates. While ANOVA methods add more variability to the EBLUP than REML estimation, the estimator for the MSEP of the EBLUP is more accurate with ANOVA methods than with REML methods. The best choice of how to measure the variability of 6 was not consistent across variance component estimation methods. For ANOVA methods, a better choice for B(0) was I1 (0), while for REML estimation the better choice depended on the value of v. We also discovered that the effect of substituting parameter estimates depends, in part, on the choice for B(90)and the variance component estimation method. The impact of this substitution is often considerable and could cause the MSEP estimate to be significantly inflated. Table 42. Relative bias (%) of estimators of correction term 0.2 0.4 0.6 0.8 1.0 REML** EtrE A( )BR( 132.0 40.3 11.0 8.8 48.5 E tr [A( )B,()] REML E Itr A()I1 ()l 33.7 85.0 152.3 202.8 220.8 ANOVA E tr A() l()] 6.7 6.7 6.7 6.7 6.7 **Monte Carlo values used to compute relative bias. 4.3 Bias Correction Term for the Balanced OneWay Random Effects Model 4.3.1 Bias Correction Term Approximation under 0 Parameterization We now address the bias correction term approximation for the oneway random effects model. We will consider the case where B(0) = V1 (0) so that the calculations for the approximation will be the same for the REML and ANOVA, and so that closedform expressions are attainable. The difference comes in the expected values of the estimators of the bias correction term due to the truncation of the REML estimates. We will show that in either case the bias correction estimator can be negative, thus proving by counterexample that the bias correction estimator is indefinite. We begin however, with demonstrating the lack of transform invariance. Recall that the parameterization we are working with is 0 = (o2, ). If we were considering the parameterization ( = (C2, ) wherein the variance matrix, V, is linear in the parameters, the bias correction term would be identical to the correction term approximation, which is transform invariant, i.e., tr (A ()B(())= tr (A ()B( ))= tr (A(0)B(0)), and nonnegativity would not be an issue since tr (A (0)B (0))is nonnegative definite in its most general definition as proven in Chapter 3. This is not the case with 0 as the parameterization, as will be demonstrated in this chapter, since tr (A (0) B (0)) # tr (A(0) B (0)). We now give a simple demonstration that the bias correction term approximation is not transform invariant, as discussed in Section 3.3, through the continuing example of the balanced oneway random effects model. The true bias of the naive MSEP, EM1 () M1 (0), is invariant to transformation as long as r ()= r( ). This follows from the definition of M, (0) = E [(0) t]2 Thus the lack of transform invariance of the bias estimate is a concern. To calculate the bias correction term approximation under the 0 parameterization, we first compute the second derivatives of the naive MSEP,M1 (0), with respect to the variance parameters: M, (0) 1 2v+kv ku2 +2 dO2 nkv 2M (0) = 0, (aoU2)2 and M, (0) _ (k 2 + 2v(1k)) a2 (1 2u+kkv kv +2) uv nkv nkv2 2(v2(1k)1) nkv2 a2M1 (0) 202 (aU)2 nkv3 2M (0) _(1 B02av k)v 1 nkV2 So, 0 (0) (1k)2 1 nkv2 Utilizing I11 (0) from Equation (416), we have 202 {(1k)v2 1 nk2 (n 1) A(0)IT( (0) S20 C f (C v) nk (1 k) 2 nku2 k2 202 nku3 f( v) (1k)v 1) 402 (kn 1) 2 (n1) nk2 (n1)(kl)u where f(o, v) is a function of 0 = ( v) that is inconsequential in this calculation. This gives (1k)v 1 kn1 tr [A(0) 1 (0)] = 4C2 F + 21 kn 1 L nk (n1)v nk (n1)(k 1) Thus the bias correction term for the naive estimator of the MSEP under the 0 parameterization is (439) (,2v) (437) (438) 1F(kl)v2 +1 kn1 BA2 (0)nk (n1)v nk (n1)(kl)u (44 which is obviously different from the bias correction term approximation used under the ( parameterization, which would be identical to Equation (415). Clearly the lack of transform invariance is attributable to the higher order derivatives in A (0). We will examine the impact of the parameterization by comparing each bias correction estimate to the true bias. 4.3.2 Expected Value of BIASR (0) with REML Covariance Parameter Estimation We begin by calculating the expected value of the bias correction term estimator once the REML estimates are substituted for the unknown variance components. Recall from Equations (419), (420), and (421) that E (j6 c 2 o2v(kn1)vB(v, + 2, v 1)I, (v + 2, v 1) Sk(n 1)vB(v, v,)) (441) +2 [1 _I(V+1,V)]+U2v1 [1I J(Ve,( +)]V . Ve +Va Ve +Va We also need EK2 (m(k 1) m + k(n 1)mn (i (mm)) (442) E = E (nMana IMe +E (1 '(n^,Ma ))e (442) v kn 1 where )=2 + E(w s)(s,2)) =2 1 /l, 1(ls)l  k1l B(ve,v) = 02 ll y(Ve,Va +1) and from Equation (421), (2)=(o2 +na) V' [I_,(v,",+l)]+C2 V, [_1I(v +1,v)]. Ve + Va Ve + Va We can utilize Equations (441) and (442) to compute the expected value of BIAS (0) as follows: E[BIAS (0)= E ltr A(0I) 0)( =2Ej2 (k 1)i +1 kn1 [nk 2(nl ) nk (n1)(k 1 =2 k( I E (/2\ 2 E /2) nk2 (n 1) v nk(k 1) =2 k1 C o(kn 1)vB(v, + 2, v, 1) (Ve + 2, v 1) nk (n1) k(n )veB(ve, va) nk (k 1) Ve + Va +*C2 Ve [1 (Ve+ l,)] 4 4) Ve + V aI (443) To evaluate the accuracy of the bias correction term estimator, the true bias correction term (which is the negative true bias) of M1 (0) for M1 (0) can also be calculated as M, (0) EM1 (0)], for REML variance component estimates of 0 Because M () = 0 when ma E[M, E I 1+(k1)"]} =E ( ) +(k l1)6 l) 3(],,(m,) (444) E na (k 2)m, (k 1)n = E + 3(Mn) . nk nk nkmn We establish that M (,VI) = (ve, +1), (445) nk nk (k 2)me M (k 2)2 E(sw)(s,) nk nk (n 1) (k 2)(kn 1) )] nk2 (n 1) (446) (k2)(kn 1)2 B(V, +1,v ) nk (n1) B(v ,,va) (k 2) 2( I,(ve +1, Va) nk and 7( 1 nk l( \)77 (kn 1) Snkma nk k(n )v) ,(B(ve +2, v,) (ve +2, vo1)2t (447) B(ve,va) (k 1) (knk +2)ac v 2 . I, (v + 2, v 1). nk (n1)(k3) Combining Equations (445), (446), and (447) into Equation (444) gives us the expected value for M, (9) with REML variance component estimates: 0 21 (k2)o2 E M, = I(e,Va + 1+ (,(ve ,+1, a) nk nk (448) (k 1)2 (knk + 2)2v nk2 (n1)(k 3) Thus the true bias correction term of M (0) for M, (0) is M,(0)E[M,(0)]= 1+k[l l)] (k1)2 (knk+2)v + IT,(ve +72,v, 1) (449) k(n 1)(k 3) VlI,(v, +1) (k 2) I, (ve +1,va)}. We can now make comparisons among the proposed bias correction term estimators of the naive MSEP, M1 (0). Figure 413 pictorially demonstrates the bias of M1 (o) for M, (0) under REML methods. From this figure it is easy to see that M, (0) has both positive and negative bias depending on the value of v. Figure 414 depicts the true bias correction term, M1 (0) E M1 (0), from Equation (449), along with the bias correction term approximation, tr[A(0)I1 (0) (Equation (440)) and its expected value under REML estimation (Equation (443)). First note that the effect of substituting variance parameter estimates in the bias correction approximation does not have much of an impact, unlike the results of the correction term approximation. Although tr A (0) (0) and E tr1(A 1 0 1 ) arenearly equal over the entire span of v, both significantly underestimate the true bias correction term over most of the parameter space. In fact, the true bias correction term is positive for v < 0.75 reflecting that M, (0) is underestimating M1 (0). However, the bias approximation and expected value are negative. Thus by using this bias correction term estimator, we would actually exacerbate the problem and significantly increase the amount by which M, (9) is already underestimating M, (0). That is, while M, (9) tends to underestimate M, (0) for v < 0.75, the "correction" by the bias correction term actually causes further underestimation of M1 (0), especially for small values of v. The use of this form of the bias correction term would actually produce a poorer estimate for M, (0) than using M, (9) alone. As v increases, we see that the true bias correction term becomes negative, reflecting that M, (9) is overestimating M1 (0); thus it is appropriate for the bias correction term estimator to be negative. As shown in Figure 414, the bias correction term approximation performs significantly better in this upper range of the parameter space than for v <.75. Now we compare this result to the bias adjustment under the linear parameterization, S=(o"2, C ), recalling that tr(A () I (0))= tr(A (D)I ())= tr(A (0)1(0)), where tr (A(0) 11(0)) is in Equation (415). Figure 415 shows M, (0) EM (o)1 along with tr [A (0)1 (0)] (Equation (415)) and E tr A () I1 (O)J (Equation (422)). We see a much different result for this bias correction term than we did for tr[A(0)T1 (0)1. Now we see the bias correction term approximation and estimator significantly overestimating the true bias correction term, especially for large values of v. For small values of v, the estimator is fairly accurate. This is the exact opposite of the previous result in Figure 414. Figure 416 compares the accuracy of tr A(0)11 ()] and tr A(0)I (E)] for M, () E[M (O). For v<.75, M, (0) EM (O) is positive. In this range, tr A (6) I1 (0)] is a significantly more accurate estimator because E Itr A(O)I(o)]l} is negative. For v>.75, M, (0)EM, (o)1 is negative, and Strip A(0)I1 (e ) is significantly more accurate than tr IA(0)I1 ()1. Thus the choice of which estimator is better depends on the value of v and also on whether it is better to over or underestimate M1 (0). Note that SAS uses the 0 parameterization in the DDFM = KR option of PROC MIXED. Hence the results from Figure 415 are applicable to SAS procedures. 4.3.3 Expected Value of BIAS"' (0) with ANOVA Covariance Parameter Estimation We can do this same comparison for ANOVA variance parameter estimates. Recall that the bias correction approximation in Equation (440) is the same under ANOVA and REML variance parameters due to the choice of 11 (0). However, because the ANOVA variance component estimates are not truncated, the expected values of the bias correction term estimator, E tr [A(0)1 (e)1 and the naive MSEP, ELM, (8)1 will differ from the REML case. First, recall from Equation (429) that S(k1)(k(n1)+2) ( k (n 1)(k3) and note that E[J2z13= E(m)= E kl a 2 U1 (451) Using Equations (450) and (451), the expected value for the bias correction term estimator is E[BIAS(6) = E tr[A(6) 16)] =2E (2 (kl) 132+ kn1 i Snk2(n1)u nk2 (nl)(kl)0 = 2 k1( E ( 2) 2 )E( < )}) (452) [[nk2 (n1)) nk (k1) 2f k1 (k 1)(k(n 1)+2) nk (n1) k(n1)(k3) nk(k ) 1) Note that by allowing the variance parameter estimates to take values outside of the parameter space, M, (8) does not maintain the nonnegative definite property. That is, in taking the expected value of M1 (8) with ANOVA variance parameter estimates, we are including negative values of M1 (8). Allowing variance parameter estimates outside of the parameter space in expected value calculations does not affect the definiteness properties of the bias correction term,  triA()I1 ()] since it is indefinite as will be discussed in Section 4.3.4. The calculation for the expected value of M1 (8) proceeds as follows: E[M ()] = E 1+(k1) nkt3 m (k2) m (k 1) m2 = E { ( 2)e m (453) nk nk nkm, C2V1 (k2) (k 1) (kn k+2)aCv nk nk nk (nl)(k3) nk nk nk (n1)(k3) Thus the true bias correction term (negative bias) of M1 (8) for M, (0) is M,(0) [M,(0)] =2 ) 1[+(k1)u] nkv 2 1 (k2)a2 (k1)2(knk+2)a2u nk nk nk2 (n1)(k 3) 2 (k 1)(kn 1)a2 nk (n1)(k 3) We can now compare this to the bias correction approximation, 2tr [A(0)1(0)] and the expected value of the bias correction estimator, E tr A() ()1 ) ,as well as tr [A(0)I1 (0)]and E{tr[A( 0)I()) . Figure 417 shows M, (0) and E[M, (6)]. We see that E[M1 (0)] consistently underestimates M1 (0) over the entire parameter space, growing more severe as v approaches 1. Figure 418 compares the true bias correction term, M, (0) [M, ()] to tr[A(0) 1 (0)] and Etr A(0)I (eo)]}. We see a very similar result as in Figure 414 for REML methods. Again, while MI () underestimates M, (0), adding tr[A(0)11 (0)] as a bias correction would increase the amount by which M1 (0) is underestimated. The bias correction term improves as v increases but never becomes positive, and therefore never improves the unadjusted estimator, M1 (). (454) Again we want to compare this to the bias correction under the linear parameterization ( = (2,2) Figure 419 shows M, (0) ELM, (o)] along with tr[A(0) 1 (0)] (Equation (415)) and E{tr (A )1I (6)]) (Equation (436)). The result is much improved in this parameterization. We can see that tr A( 0) ()] is a highly accurate estimator for the true bias correction term, M, (0) E[M (6)] and using this estimator would improve the estimator for M, (0) over the entire parameter space. That is, M, () )+tr A () V ()] is a more accurate estimator of M1 (0) than M, (0). Figure 420 compares the accuracy of tr [A(0)I1 (0) and  tr A(0)I J0 for M (0) EiM, .)1. Because M,(0) E M, (O) is positive over the entire parameter space, tr [A( )11 ()] is an unacceptable estimator of M (0) EM ()]. However, tr A(0)I'1 ()] is a highly accurate estimator for M, (0) ELM (0)] with nearly constant relative bias of 6.67%. 4.3.4 Impact of Negative Values of Bias Correction Term Approximation As mentioned in Section 3.2.3 the bias correction approximation given by Equation (310) in indefinite. The balanced oneway random effects model under the 0 = (o2,v) parameterization provides a sufficient example. Figure 415, shows that  tr A(0) 1 (0)1 takes negative values over the entire range of v. We can determine from Equation (440) when the bias correction term estimator will be negative in terms of k and n. First rewrite Equation (440) as ltr A(0)I (0)]= 2C2(k21)02+1 2 k1 2 nk2 (n1)v nk2(n1)(k1)v 2r2 [(k1)2 v2 k(n1)] nk2 (n 1) (k 1)v Noting that the denominator is always positive when v is in its parameter space, we determine that the bias correction term estimator in Equation (440) is negative when k(n1) u< \ . (k 1)2 For k = 6 and n = 6, as in our example, this value is v < 1.09 which means that the bias correction term estimator is negative over the entire (0, 1] parameter space for u. This counterexample suffices to prove that the bias correction term approximation in its most general form, 1tr (A (0) B (0)) tr {AI, (0) )B (0)) BIAS KR(0) 1 2 tr {(A, (0) B(0)} tr {A,, (0) B(O)} from Equations (310) and (311), is indefinite. This in itself is not necessarily problematic since the true bias correction term, M, (0) E M ( )], is also indefinite. However, if the magnitude of this piece is large enough to cause the entire MSEP estimator to be indefinite, then problems arise. It is beyond the scope of this dissertation to determine if the MSEP estimator utilizing the correction term estimator and the bias correction term estimator, M (0) + CK (0) + BIASK (6), is nonnegative definite. However, it can be shown that as an estimator for M1 (0), M, (0)+BIASf (0) (455) is indefinite by looking at the same special case of the balanced oneway random effects model. The form of Equation (455) for this model is M,( + BIAS' (6)= t ) [1+(k 1) 0] +2'2 (k 1)v2+1 kn 1 (456) nk2(n)v nk2(n1)(k1) C 2 ((k 1)2 (2 k(n 1))2 + k(n 1)(k 1)(k 2)v + k2(n 1) 3k(n 1)) nk2(n 1)(k 1) To determine when Equation (456) is negative, we find the roots of the numerator (noting that the denominator is greater than zero for k, n > 1) by solving the quadratic equation in v. This yields the following result: M,) + BIAS (6) < 0 where 3k(n1) k2(n 1) (k(n 1)(k1)(k1))2 k(n1)(k 1)(k 2) 57) (k1)2(2k(n1)) 4((k 1)2(2 k(n 1)))2 2(k 1)2(2k(n 1)) or S3k(n 1) k2 (n 1) (k(n 1)(k 1)(k 1))2 k(n 1)(k 1)(k 2) i < + (k1)2(2k(n1)) 4((k 1)2(2k(n1)))2 2(k 1)2(2k(n 1)) For k = 6 and n = 6 this gives 63 > 0.987 and 6 < 0.130. While the lower bound on 6 is outside the parameter space, the upper bound is within the parameter space and a valid result for the parameter estimate. In the case ofk = 3 and n = 3 the result is 6 > 0.75 and 6 < 0. Again, the lower bound is outside of the parameter space but the upper bound is well within the parameter space. This demonstrates the danger in using M (6) + BIASK (6) as an estimator for M1 (0) or M (0) and serves as a general warning of the possible effect of the indefiniteness of BIASR (0). The simplicity of this example raises even more cause for concern in more complex models. 4.4 Performance of Overall Estimators for the MSEP Now that all the pieces of the MSEP estimators have been evaluated, we can combine these results to get an overall picture for how accurate each MSEP estimator is for the true MSEP. Recall that the true MSEP values will differ for REML and ANOVA variance components, since each variance component estimation method adds a different amount of uncertainty to the MSEP of the EBLUP and a different amount of bias to M1 (0). For REML methods, we will look at MSEP estimators utilizing both BR (0) and I1 (0) in the correction term estimator since there was no clear preference (see Figure 45). However, for ANOVA methods, the estimators using I1 (0) were clearly superior to other choices of B(0) and thus we will limit our comparison to V1 (0) for ANOVA methods. Thus the following equations list the possible estimators for the MSEP of the EBLUP of t = a For REML variance parameter estimates we have three options: M, ()+2tr[A(0)BR ()], (458) M, (6) +2tr [A (6) ()], (459) and M, ) + tr [A (0)I ] tr A f0)rI 1]. (460) For ANOVA variance parameter estimates we have 2 choices for MSEP estimator: M, ()+2tr [A ()()], (461) and M(0)+ tr[A (0)(0)] tr[A (0)I(0) (462) Note that while Equations (458), (459), and (461) are derived under the D = (2, ) parameterization, due to the translation invariance of the elements involved, for simplicity we will use 0 notation. We begin with the REML comparison. Figure 421 shows the true MSEP for the EBLUP of t = a using REML variance parameter estimates, MR (0), along with the expected values of the three options in equations (458), (459), and (460). For completeness, E[M (6)] is included as well. As we have seen throughout with REML variance parameter estimates, the results are dependent on the values of u. For small values of v, Equation (459) is most accurate, almost matching the true MSEP exactly. As u grows, though, both Equation (458) and (460) become more accurate than Equation (459). Note that SAS uses Equation (459) as the estimator of the MSEP in the DDFM = KR option of PROC MIXED, which is not the best choice over a large range of u. It is especially problematic for u close to one, which again corresponds to values of crO small relative to 2. Now looking at ANOVA comparisons in Figure 422, we see that Equation (461) is most accurate for the true MSEP, MA (0), over the entire range of u. It is clearly the best choice for MSEP estimator when ANOVA methods are used. We also want to determine if REML or ANOVA methods are preferred when comparing MSEP estimators. To do so, we look at the relative bias of each of the five MSEP estimators listed above for their respective true MSEP values. Table 43 summarizes the results for select values of v and Figure 423 illustrates this comparison. The table and figure clearly shows that the MSEP estimator with the smallest relative bias over the entire range of v is M, (0)+ 2tr [A (0)1 (6)] which uses ANOVA methods. Thus for accuracy of the MSEP estimator, ANOVA variance component estimation under the 0 parameterization with MSEP estimators utilizing B(D) = I 1 () are the best choice among the ones compared here. Table 43. Relative bias (%) of MSEP estimators under REML and ANOVA estimation 0.2 0.4 0.6 0.8 1 M(O) 13.28 26.27 30.88 22.76 37.38 M, )+2tr A )BR 16.75 10.59 11.71 36.21 140.50 M, ()+22tr [A ( )V () 4.03 22.33 65.91 173.14 620.57 Mi, )+tr A (6) ( tr A 0)' 26.84 18.26 2.72 57.93 294.72 M (8) 17.72 51.38 93.33 142.68 200.00 M,1 ()+2tr[A( )1(e)] 1.19 3.42 6.24 9.51 13.35 M, ()+tr [A (0)(0) tr [A(0)(0) 29.70 36.58 49.77 67.18 88.37 **Monte Carlo values used to compute relative bias. It is important to note that while this is the MSEP estimator used by SAS in the DDFM = KR option when ANOVA methods are chosen with this model, the MSEP estimates are truncated at zero which changes the expected value of the MSEP estimator from the result given here. The disadvantage of using ANOVA methods is the likelihood of negative variance parameter estimates. However, by truncating the MSEP estimates, SAS negates the advantage of increased accuracy associated with ANOVA methods. The implication of allowing untruncated MSEP values is that negative estimates of the MSEP are possible. The impact of negative variance parameter estimates on EBLUP procedures in the balanced oneway random effects model are further explored in Chapter 5. 4.5 Comparison of Prediction Interval Methods Several methods for conducting hypothesis tests in linear mixed models are reviewed in Chapter 2. Table 21 contains a summary of the methods, including how the distribution is determined and which standard error is used in the test statistic. We now want to examine these methods in terms of the balanced oneway random effects model. Because we are estimating the realized value of a random effect, it is more appropriate to examine the methods in terms of prediction intervals. The goal of this examination is to determine the accuracy of the prediction interval estimation methods and determine where further investigation is necessary. Four prediction intervals will be compared in a Monte Carlo simulation study to determine which method has the most accurate coverage rate. All of the prediction intervals have a similar structure. The estimate of the realized value of the random effect, a,, is i ,the EBLUP for a,, as in Equation (42). This is consistent in all four methods. The structure for the prediction intervals is (MSEP)t 2,. (463) The estimate of MSEP and the degrees of freedom for the tdistribution depend on the method used to calculate the prediction interval. The methods used to determine the MSEP estimate and the degrees of freedom are summarized in Table 44. Table 44. Summary of prediction interval procedures. GB KR MSEP MI(4'D) FC(0) M (6) M (') +2tr A() B (c)] +2tr [A () B ()] dfG ~J4C J4 dfFC() F 2 FC(0) M, () +tr[A(0)B()] tr A(o)B()] 22 dfFc(o) 4 Zvi, LFr 9FA2ST, 2 "fJ UJGB z IV FFC(D) ] MLIV PF FC(O) var MSEPFC(S) var LMSEPFc(F ) var L I ] I S[MSEP(.)] Note: var [MSEP h(.) B(.)h(. ) where h (,) = and B(.) = 1 (*). Note that all of the methods use a "Satterthwaitetype" method for estimating the degrees of freedom. The degrees of freedom are calculated using 2 MSEP~ df= L (464) var [MSEP 0 [MSEP(.) where var[MSEP] h(.)'B(.)h(.) for h(.) = a. and B(.)=I (.). The disparity between the methods comes in the parameterization, and the estimate of the MSEP (or standard error) which subsequently impacts the degrees of freedom. We again study results under two parameterizations, = (2, v) and = (D2, C ). Note that due to the translation invariance of M1 (.) and tr [A(.)B(.)], the figures and results demonstrated earlier in this chapter pertaining to M, (o) and M, () + 2tr [A()B(O)] still apply to the standard error estimates used in the GiesbrechtBurns, KenwardRoger, and FC (0) prediction interval procedures. ] Two of the methods for formulating the prediction interval are used in SAS PROC MIXED. The first, from the DDFM = SATTERTH option, is similar to the GiesbrechtBurns (1985) procedure outlined in Chapter 2; however, in this application we are calculating prediction limits for the realized value of a random effect rather than a fixed effect, as in that paper. The method is also developed in Jeske and Harville (1988) and Fai and Cornelius (1996). We will refer to it as the GiesbrechtBurns prediction interval method, since it uses the naive MSEP estimate. The standard error is estimated by the naive MSEP, (M 1 ())/2 where M, (0) is in Equation (43). The other method available in PROC MIXED through the DDFM = KR option is the KenwardRoger (1997) approach. The standard error estimate includes a correction term and a bias correction term, as in Equation (459). Note that the KenwardRoger prediction interval is determined under the D parameterization. Recall that under the parameterization, = (o2, a), the bias correction term estimate is equal to the correction term estimate, thus the MSEP estimate used in the KenwardRoger prediction interval is M, (o) +2tr A (i)B o)] =M () +2tr A(6)B(O)]. It is important to keep the parameterization in mind throughout, as the MSEP estimate and the degrees of freedom calculations depend on which parameterization is used. It is demonstrated in Chapter 2 that in a singledimension linear combination, the KenwardRoger method for determining the degrees of freedom reduces to the GiesbrechtBurns method. Thus, although the KenwardRoger and GiesbrechtBurns methods for determining the prediction limits use different estimates for the standard error, the degrees of freedom for the tdistribution will be the same. KenwardRoger prediction limits adjust the MSEP estimate for estimating unknown variance components but do not adjust the tdistribution for the change in standard error. The other two procedures for creating prediction intervals, FC (0) and FC (0), are based on the FaiCornelius (1996) hypothesis testing methods. For the FC intervals, both the MSEP estimate and the degrees of freedom of the tdistribution account for the correction term and the bias correction term. Note that this is actually an extension of the methods developed by Fai and Cornelius (1996). The bias correction term was not considered as an adjustment to the MSEP estimate in the methods they proposed. The bias correction term is added here to be consistent with MSEP estimator used in the KenwardRoger method. The FC methods differ in the parameterization. Under the c = (o2, a) parameterization, the MSEP estimate is the same as in the KenwardRoger parameterization, M1 () + 2tr [A () B () =M, () + 2tr A (o) B (). Unlike the KenwardRoger prediction interval, the FC (D) procedure adjusts the degrees of freedom of the tdistribution for the adjustments to the MSEP estimate. The degrees of freedom are still calculated using a "Satterthwaite" method; however, now the corrected MSEP estimate, M, () + 2tr [A (0)B( ) is used in the calculation of the degrees of freedom, rather than the naive MSEP estimate, M1 (D). The final prediction limit method considered, FC (0), differs from the other three in the parameterization, 0 = (o2, v). Recall that the bias correction term estimate is not translation invariant, and thus by changing the parameterization, we also change the MSEP estimate. The MSEP estimate for the FC(O) procedure is M,(O)+t [Atr ) B ()1 tr A()B(O). This is used in both the standard error estimate and the degrees of freedom calculation. A Monte Carlo simulation study was conducted to compare the four prediction interval methods. The study is similar to the one conducted in Section 4.1 to evaluate the correction term estimators. Using the RANNOR function in SAS, 10,000 independent sets of e, and a, were generated from N(0, '2 =1) and N(0,o,) for several values of O, respectively. The values of a correspond to v= {0.05, 0.1, 0.15..., 1.0}. Setting = 0, 10,000 sets of y, : i = 1,...k, j = 1...6) were created from the model y, = a + e The number of levels of the random effect is varied to study the impact on the performance of the prediction interval procedures. The values for k are k = 3, 6, 15, or 30, while n is held constant at n = 6. Simulating the data in this way allows the realized values generated for the a, 's to be tracked. These are the realized values of the random effect for which the prediction intervals are constructed. By tracking the realized values of the random effect, we can determine how often the prediction intervals contain the true value of the realized random effect, providing the actual (simulated) coverage rate for a nominal 95% prediction interval. The prediction limits for a, for the GiesbrechtBurns and KenwardRoger procedures are produced by the CL option in the ESTIMATE statement in PROC MIXED in SAS, with the DDFM = SATTERTH or DDFM = KR options, respectively. To calculate the FC prediction limits for either parameterization, the degrees of freedom must be calculated directly and then used to determine the critical value from the tdistribution. For the FC (0) procedure, the standard error from the KenwardRoger procedure is used to calculate the prediction limits. For the FC (0) procedure, the standard error also has to be calculated directly in addition to the degrees of freedom because of the change in parameterization. The SAS programming statements used to perform the simulation study are contained in Appendix A along with the derivatives needed to compute the degrees of freedom for the FC procedures. The results of the simulation study are best demonstrated pictorially. Figures 424 through 427 contain the results of the simulation study for k = 3, 6, 15, and 30, respectively. It is clear that the KenwardRoger procedure produces the most inflated coverage rates consistently across values of k and for the entire parameter space of v. The prediction limits for the Kenward Roger procedure will always be larger than the GiesbrechtBurns procedure since M, () + 2tr [A (O) B () M, (). Adjusting the degrees of freedom in the FC (D) procedure for the adjusted MSEP estimate generally increases the degrees of freedom, causing the critical value from the tdistribution to shrink and thus reducing the width of the prediction interval. The coverage rates for the FC (D) procedure are generally closer to 95% than the coverage rates for the KenwardRoger procedure. The performance of the FC(0) prediction interval is more dependent on the value for k and the parameter v than the other intervals. For k = 3, the coverage rate for small values of v is exceedingly small. As v increases, the performance improves dramatically. This is due to the inclusion of the bias correction term under the 0 = (C2, v) parameterization. Small values ofk combined with small values of v produce small MSEP estimates. Recall from Figure 423 that the relative bias of M ()+ tr[A(0)B()] [tr[A(0)B(6)] approached 50% for small values of v. This is caused by the significant underestimation of the true bias correction term by  tr A ()B ( (see Figure 414). For larger values of k, the FC(0) prediction interval procedure compares favorably with the other procedures and consistently improves on the performance of the KenwardRoger prediction interval for larger values of v. For larger values of v recall that M ()+ 2tr [A ()B (O) overestimated the true MSEP more so than M, ()+tr [A ()B() tr [A ()B )l Thus we would expect a more accurate prediction interval with the FC(0) method in this range of the parameter space. To demonstrate the relationship between the prediction intervals more concretely, consider the hypothetical case where k = 6, n = 6, with data producing variance parameter estimates of a2 = 1 and (,2 = .5. Table 45 contains the values for the standard error, degrees of freedom, critical tvalue and the width of the prediction interval produced by each of the four procedures in this scenario. The width of the KenwardRoger prediction interval increases over the width of the GiesbrechtBurns interval due to the increase in the standard error estimate. The width of the FC (0) prediction interval is smaller than both the GiesbrechtBums and the KenwardRoger intervals because of the increased degrees of freedom and subsequent decrease in the critical t value. The width of the FC(0) interval is by far the smallest due to the decrease in the standard error estimate and the critical tvalue. Finally, to demonstrate the effect of k on the performance of the prediction interval procedures, Figure 428 shows the KenwardRoger coverage rates for all values of k considered in the simulation study. This is demonstrative of all the procedures in that as k increases, the procedures perform better and the coverage rates move closer to the nominal value of 95%. This study shows the importance of investigating the FaiComelius methods for hypothesis tests and prediction intervals in more complex situations. These preliminary results indicate that the FaiComelius methods often perform better than the KenwardRoger and the Giesbrecht Burns methods used in SAS. The KenwardRoger procedure produces an overlyconservative prediction interval because it does not adjust the degrees of freedom for the inflated MSEP estimate. The study also demonstrates the impact of the parameterization and the lack of transform invariance of the bias correction term on accuracy of the prediction interval methods. In more complex covariance structures, the parameterization may be vital to an accurate analysis. Table 45. Effect of prediction interval procedures for k = 6, n = 6, (2 = 1 and ^ 2 = .5 GB KR FC(0) FC(0) Standard Error estimate 0.43301 0.46894 0.46894 0.40196 Degrees of Freedom 8.15 8.15 16.39 16.52 t025,df 2.298 2.298 2.116 2.114 Width of CI 1.99 2.16 1.98 1.70 0 1 1 ~~~~~~~~ ~ ~   2 3 4 5 6 7 0 0.2 0.4 0.6 0.8 1 1.2 k=6 k=9 k=12 k=15 k=18 Figure 41. Equation (413) as a function of v for several values of k, holding n = 6. Equation (414) holds when functions cross zero line. 0.07 0.06 0.05 0.04 0.03 0.02 0.01 0 tr [A(0)I1 ()] S tr A( (0) 0.2 0.4 0.6 0.8 Figure 42. Accuracy of approximations tr [A(0)BR (0)] and tr [A(0)11 (0)] for the correction term, CR (0), under REML variance component estimation. 0.045 0.04 0.035 0.03 0.025 0.02 0.015 0.01 0.005 Accuracy of approximation tr [A(0)BR (0) and estimator E tr [A ()BR) () for the correction term, CR (), under REML variance component estimation. E{tr[A(0)BR()]} tr[A(9)BR ()] Figure 43. 0.06 0.05 0.04 0.03 0.02 0.01 S[A (O)r ()] 0.2 0.4 0.6 0.8 u Figure 44. Accuracy of approximation tr [A(0)' (0)] and estimator E tr A(o)I1 (0) for the correction term, CR (0), under REML variance component estimation. I I I E tr A(6) (6)) IC (8,1 E Atr(A (6) B, 0.2 0.4 Figure 45. Comparison of accuracy of estimators for the correction term, CR (0), under REML variance component estimation. 0.06 0.05 0.04 0.03 0.02 0.01 0 0.9 tr [A(0)BA (0)] 0.8 tr[A(0)B (0)] 0.7 0.6 0.5 0.4 0.3 0.2 ( 0.1 0 tr[ A(0)I1 (0) 0 a 0 0.2 0.4 0.6 0.8 1 1.2 Figure 46. Comparison of accuracy of approximations tr[A(0)I1 (0)], tr A(o)B* (0)], and tr A(0)BA (o)] for the correction term, CA (0), under ANOVA variance component estimation. 1.6 1.4 E tr [A 0)BA ) )] 1.2 1 tr [A ()B, ()] 0.8 0.6 0.4 0.2 0 0.2 0.4 0.6 0.8 1 1.2 Figure 47. Accuracy of approximation tr A(0)BA (0)] and estimator E tr [A()BA ()} for the correction term, CA (0), under ANOVA variance component estimation. 1.6 1.4 E tr[ A)B (B)]6 1.2 1 tr [A ()BA (0)] 0.8 0.6 0.4 0.2 0 0 0.2 0.4 0.6 0.8 1 1.2 U Figure 48. Accuracy of approximation tr A(0)B* (0)1 and estimator E tr [A()B* (6)]) for the correction term, CA (0), under ANOVA variance component estimation. 0.12 TrLAU MUj 0.1  0.08 0.06 tr[A (0)1 ()] 0.04 0.02 0 0.2 0.4 0.6 0.8 1 1.2 U Figure 49. Accuracy of approximation tr [A(0)I1 (0)] and estimator E tr [A () 1 (8)] for the correction term, CA (0), under ANOVA variance component estimation. 1.6 E tr[A(0)B, ()]} 1.4 E tr [A 0) B ()] 1.2 0.8 0.6 0.4 0.2 E tr[A(e) (e)] 0 0.2 0.4 0.6 0.8 1 1.2 Figure 410. Comparison of accuracy of estimators for the correction term, CA (0), under ANOVA variance component estimation. 0.14 Etr[A ())I1()]} 0.12 0.1 c 0.08 0.06 0.04 0.02 E tr A () B, ( 0 j 0 0.2 0.4 0.6 0.8 1 1.2 Figure 411. Accuracy of estimators for the correction term under REML, and ANOVA, variance component estimation. Figure 411. Accuracy of estimators for the correction term under REML, 0, and ANOVA, 6, variance component estimation. 500.00% 400.00% 300.00% 200.00% 100.00% 0.00% 100.00% Figure 412. Relative bias of estimators for the correction term under REML, 0, or ANOVA, 0, variance component estimation. 0.8 0.6 0.4 0.2 7l IE[Mi (0)]] 0 0 0.2 0.4 0.6 0.8 1 1.2 Figure 413. Bias produced by estimating M1 (0) with M1 (). 0.1 M,(6)EM:6)] 0 1 _ 0.2 0.4 6 0.8 1 12 0.E1 t[A (1)I1 () 0.2 1r A 0.3 0.4 0.5 0.6 Figure 414. Accuracy of approximation tr [A(0)I1 (0)] and estimator E tr A )I1 () for the true bias correction term, M () 2, under REML variance component estimation. M, (0) E[MI ()], under REML variance component estimation. 0.06  0.05 0.04 00.2 0.4 0.6 't 1 0.01 M (0)[M()] 0.02 0.03 Figure 415. Accuracy of approximation tr [A(0)I(0)] and estimator E {tr A A()I )l for the true bias correction term, M, (0) E[M ()] under REML variance component estimation. IjEtrFA(0)I 1 E I tr [A()I1() U Figure 416. Comparing the accuracy of E tr [A ()I 1 ()] and E tr A (0)I1 ()) for the true bias correction term, M, (0) E[M ()]. 0.1 0.2 0.3 0.4 0.5 0.6 M, () EM() 0.75 0.65 0.55 0.45 0.35 0.25 0.15 ul 0.05 0.05 0.15 Figure 417. Bias produced by estimating M1 (0) with M1 (6). M (e) M, (0)E[M,()] 0.1 0.2 0.4 0.6 0.8 1 12 0.2 E{ ttr [A(A)1I()]} 0.3 0.4 0.5 0.6 Figure 418. Accuracy of approximation tr[A(0)I1 (0) and estimator E tr A()I1 () for true bias correction term, S() M ( under ANOVA variance component estimation. M, (0) E[MM (6)], under ANOVA variance component estimation. 0.12 "\" L7\ I \j)J 0.1 IM,()E[M(0)]I 0.08 0.06 Itr[A(e )I1 0.04 4 0.02 0 0 0.2 0.4 0.6 0.8 1 1.2 Figure 419. Accuracy of approximation, tr [A (0) 1'(0)] and estimator E tr A () I1 ())} for true bias correction term, M, (0) E [M () under ANOVA variance component estimation. 01 0 01 02 03 04 05 06 Figure 420. Figure 420. Comparing the accuracy of E  tr A(6)Ie) )] and E tr A ()I ()] for the true bias correction term, 2LIJL1 M, (0 E[M I (6) ] 0.2 0.4 0.6 0.8 1 u Figure 421. Comparison of estimators of the MSEP for the EBLUP of a utilizing REML variance component estimation. Figure 422. Comparison of estimators of the MSEP for the EBLUP of ac utilizing ANOVA variance component estimation. 300.00% 250.00% 200.00% 150.00% 100.00% 50.00% 0.00% 50.00% 100.00% Figure 423. Relative bias of possible estimators for MSEP of EBLUP for a 1 0.98 80.96 0 0.94 0.92 0.9 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Simulated coverage rates of prediction interval procedures for t = a, with k Figure 424. 1 0.98 0.96 FC (D) o0.94 0.92 0.9 0.88 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Figure 425. Simulated coverage rates of prediction interval procedures for t = a, with k 1 0.99 0.98 0.97 0 U 0.96 0.95 0.94 0.93 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1) Simulated coverage rates of prediction interval procedures for t = a, with k = 15. Figure 426. 1 0.99 s 0.98 go o 0.97 U 0.96 0.95 0.94 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 u Figure 427. Simulated coverage rates of prediction interval procedures for t = a with k = 30. 1.U1 1 0.99 S 0.98 0.97  0.96 0.95 0.945   0.94 1 1 1 Figure 428. Simulated coverage rates for the KenwardRoger prediction interval procedure. CHAPTER 5 THE EFFECT OF NEGATIVE VARIANCE COMPONENTS ON EBLUP The effect of negative variance component estimates on EBLUP procedures is a topic that has been largely neglected (Stroup and Littell 2002). While the use of REML estimates, which are truncated at zero, would seem to negate this issue, there are still many circumstances when other variance parameter estimation methods are chosen and negative variance components may arise. For example, as shown in Stroup and Littell (2002) and in the results on the accuracy of the correction term and bias correction term estimates in Chapter 4, REML may not always be the preferred choice for variance component estimation. One strategy for handling negative variance component estimation is to allow that the component may be a covariance rather than a variance, thus alleviating the problem of truncating at zero. We begin to look at the impact of that strategy by again turning to the special case of the balanced one way random effects model. 5.1 Negative Variance Component Estimates and the Balanced One Way Random Effects Model To demonstrate the current issues with negative variance component estimation, we first look at the current SAS 9.1 output for PROC MIXED, both when the variance component estimates are truncated at zero and when they are not. Using the REML NOBOUND statement in the procedure statement allows the variance component estimates to take any value on the real line. Note that because we are in a balanced situation, the ANOVA variance component estimation would produce the same results as REML NOBOUND. We compare the effect of truncating using simulated data, generated from a normal distribution with o2 =.01 and cO2 =1. This distribution produces a high likelihood that the data generated will have larger variation within a block than between the blocks, hence often producing a negative estimate for Ca2. For demonstration purposes, the number of levels of the random effect is k = 5 and the number of observations at each level is n = 6. A dataset that meets the above condition is given in Table 5 1. It is easy to spot that more variation occurs within the block levels than between them. Table 51. Generated data that produces negative variance component estimate Replicate 1 2 3 4 5 6 Block 1 2.34754 0.24672 0.5585 0.60564 0.9062 1.37627 Block 2 1.01248 0.0493 0.49583 1.3952 0.8644 1.93825 Block 3 0.18826 0.91668 0.68925 0.96593 0.95929 0.7838 Block 4 0.4741 1.31312 1.5036 0.06752 1.2383 0.7888 Block 5 1.2176 0.64999 0.32013 1.4081 0.1774 1.77956 The results from the REML option in the MIXED procedure of SAS are contained in Tables 52 and 53 and the results from the REML NOBOUND options in the MIXED procedure are contained in Tables 54 and 55. Also, for comparison, least square means (LSMEANS) generated by PROC GLM are given in Table 56. These give a starting point from which to compare the EBLUPS resulting from the REML and REML NOBOUND options. The following SAS code generates the output in Tables 52 and 53: proc mixed data = &sample method = reml ; class blk trt; model y = /ddfm = kenwardroger ; random blk / s ; Table 52. REML variance parameter estimates Covariance parameter Estimate Block 0 Residual 1.0643 Table 53. Solution for random effects with REML variance parameter estimates Effect Block Estimate Std Err Pred DF t Value Pr > It Block 1 0 Block 2 0 Block 3 0 Block 4 0 Block 5 0 The NOBOUND option in the following SAS code is the only difference from the previous code and generates the SAS output in Tables 54 and 55: proc mixed data = &sample method = reml nobound ; class blk trt; model y = /ddfm = kenwardroger ; random blk / s ; Table 54. REML NOBOUND variance parameter estimates Covariance Parameter Estimate Blk 0.1410 Residual 1.1810 Table 55. Solution for random effects with REML NOBOUND variance parameter estimates Effect Block Estimate Std Err Pred DF t Value Pr > It Block 1 0.4846 0 29 Infty <.0001 Block 2 0.3461 0 29 Infty <.0001 Block 3 0.4106 0 29 Infty <.0001 Block 4 0.8870 0 29 Infty <.0001 Block 5 0.3379 0 29 Infty <.0001 Table 56. LSMEANS from PROC GLM Block LSMEAN 1 0.51858487 2 0.18960919 3 0.48927203 4 0.02458600 5 0.46046652 The results from the REML option, which truncates the variance component estimate for block effect at zero, are straightforward. The solution statement produces zero estimates for all blocks and missing values for the standard error, tvalues, degrees of freedom, and pvalues. The REML NOBOUND option is not quite so straightforward. When variance component estimates are positive, we can think of the BLUP as a "shrinkage" predictor. In other words, using the variance component estimate as a weight, the block means are shrunk toward the overall mean. In addition, in a balanced model, the BLUPS maintain the rank order of the Ismeans. This can be easily seen in the structure of the BLUP for the balanced oneway random effects model in Equation (42). Now, looking at the EBLUP estimates in Table 55, the first indication of a problem is that the standard errors are listed as zero and hence the tvalues are listed as infinity. The calculations for standard error in this case are negative, causing SAS to truncate the standard error at zero. Comparing the estimates in Table 55 to the LSMEANS from PROC GLM in Table 56 shows that the negative variance component estimate causes the EBLUP estimates to invert their rank order and expands the range of the estimates. The block with the largest LSMEAN (0.5186) now has the smallest EBLUP (0.4846) and, likewise, the block with the smallest LSMEAN (0.0246) now has the largest EBLUP (.8870). Also, the range of the LSMEANS is 0.5432 while the range of the EBLUP estimates is 1.3716. We normally expect the BLUP procedure to "shrink" the block means toward the overall mean according to the variation in the model; however, in this case, the estimates are inverted about the mean and expanded away from the overall mean. While the standard errors and tvalues listed with the EBLUPs are a clear red flag, the unexpected results of the EBLUPS require further investigation. Note also that the standard errors may be positive even when <2a < 0; this would eliminate the red flag and leave more room for error in the interpretation of the results. If the data analyst believes that a oneway random effects model is the true model for the research situation, then the recommendation from an EBLUP standpoint is to truncate at zero rather than proceed with a negative variance component estimate. Any use of EBLUPS while allowing the variance component to give a negative estimate is likely to yield misleading results. The LSMEANS likely yield more reliable estimates than the EBLUPS from the REML NOBOUND option. Keep in mind that it is not always the best choice to truncate at zero when variance component estimates are negative if the analyst is interested in fixed effects (Stroup and Littell 2002). 5.2 Considering the Variance Parameter as a Covariance One idea that surfaces when faced with negative variance component estimates is considering the parameter as a covariance rather than a variance, thus allowing the parameter estimate to take on values over the entire real line. Smith and Murray (1984) consider this option and the impact on estimation and hypothesis testing of the covariance component. For the balanced one way random effects model discussed in Chapter 4, the covariance structure would be modified from 2 . cov(yj ,) = 0 +2, i = iU j = j, 2a, i j',jjf = 0, i i' to cov(y, jy,),) = 0 + 2, i ', j = = 0, i=i', ij' =0, i i' where 0_o 1_0a" : U2 + 0a 0<00 <&0, One problem overlooked by Smith and Murray (1984) is that in a one way model, if the variance structure is modified in this way, the model is no longer a typical random effects model. The motivating example used in Smith and Murray (1984) is weaning rates of twin calves for 40 cows. In the original model, the cow effect is considered random, with the cows regarded as a random sample from a population of cows, and c,, the variance associated with this population. When the estimate for o2 is negative, the definition of the parameter is changed from the population variance of the cows to the covariance between twins born to the same cow. Thus the population variance for the cows is ignored all together and the negative covariance estimate is interpreted as reflecting competition for nutrition between the twin calves. The negative value of the variance parameter estimate is likely reflecting the combination of the positive cow population variance and the negative correlation between twins born to the same cow. That is, there is more negative intraclass variability than positive interclass variability resulting in a negative parameter estimate. The inversion of the LSMEANs that we see with the EBLUPs in PROC MIXED when the variance component estimate is negative occurs because the model is effectively changed to this Smith and Murray (1984) model. The EBLUP calculated by PROC MIXED for a in this case is a +n0o giving a negative coefficient when  < 6 < 0. Furthermore, may be sufficiently large n nO so that n < 1. Thus rather than "shrinking" the means, as we expect EBLUP procedures 2 +na to do, the EBLUP inverts and expands the range of the means. Clearly this is an unacceptable result. The question remains: what is an analyst to do when faced with this result. One possible solution is derived in the next section. 5.3 BLUP derivation for Random Effects Model with Correlated Errors We now present a model that would account for both the population variance of the random effect and the correlation of replicates within a level of the random effect. Consider a model with a random effect, as well as correlated errors. The model and variance structure in Equation (41) would be modified to: ye, = +a + e,i =l,..., k, j=l ,...,n a ~N( 0,o where e are normally distributed with E(e) = 0 2 + 0, i=i',j=j' ( (51) cov(e,,), i = i', j=I + j 0, i # i' In mixed models notation, this equates to y = Xp + Zu + e where x =1', p = Z=Ik 01 u=[ala2 ...ak N(0,G) where G = OIk and e=[elel2...ek '~ N(0,R) where R=Ik ,(o2I + J) so that V=ZGZ'+ R =Ik {2n ( + )J From general matrix theory (e.g., Searle 1982), we have v iC 2 \ + 08 V 1= 1 2 'n 2 (C2 + \"" r {2 2 +nj which gives the BLUP for u as 2 0a ii= GZV1 (y )= Ik ( or the BLUP for ac as a2 anu, (3 y) (52) Note that for the model structure given in Equation (51) to be valid, V must be positive definite. Therefore, the parameter space for which the model is valid is 02 2 2>0, a2>0, a2 + <  n In this parameter space the coefficient for the BLUP, given in Equation (52), is nonnegative. Thus the order of the LSMEANS for the blocks is preserved in the BLUP. One key consideration for this model is the problem of overparameterization. The parameters C02 and 08 are confounded and cannot be simultaneously estimated. One solution is to use an estimate for 02 from previous experience and consider the parameter as known. This leaves only 0 and 2 as unknown and the EBLUP analysis can be performed where the unknown variance parameters are estimated by an unbounded method, such as REML NOBOUND in PROC MIXED. There are many situations, such as agriculture or education, where using a previous estimate as the parameter is realistic. Data contained in Table 55 are generated from a distribution as described in Equation (51) with o2 =.9, 0O .13 and oC =.07. Table 57. Data generated from distribution in Model (51) Replicate 1 2 3 4 Block 1 1.77754 0.72357 0.79317 0.79674 Block 2 0.62131 1.15758 1.21619 0.24776 Block 3 0.3412 0.7861 0.41517 0.75414 Block 4 0.05152 0.9336 0.31271 0.3097 Block 5 0.75952 0.27854 0.28914 1.03234 5 1.31455 0.5867 1.34176 1.04458 0.88724 6 0.98605 0.12472 0.75806 1.07845 2.24931 The appropriate SAS statements to analyze this data in the manner described in Equation (52) are data gmat ; input row col value; cards; 1 1 .07 2 2 .07 3 3 .07 4 4 .07 5 5 .07 proc mixed data = &sample method = reml ; class blk trt by rn ; model y = /ddfm = kenwardroger ; random blk / gdata=gmat g s ; repeated trt / sub = blk type=cs ; run ; The gdata option after the random statement provides the known G matrix which in this case is G =.07I, where In is an ndimensional identity matrix. The output from this analysis of the data in Table 57 are contained in Tables 58 and 59. The EBLUPS for the levels of the random effect are generated by the solutions or s option in the random statement of the SAS code, and are given in Table 59 under "Estimate." Comparison of these values to the LSMEANS generated by GLM in Table 510 show that the function worked as expected, maintaining the rank order of the means and "shrinking" the predictors toward the overall mean. Table 58. Covariance parameter estimates under Model (51) Covariance Parameter Subject Estimate Variance Block 0.9497 CS Block 0.1311 Table 59. Solution for random effects under Model (51) Effect Block Estimate Std Err Pred DF t Value Pr > Itl Block 1 0.00593 0.2646 2.16 0.02 0.9840 Block 2 0.2849 0.2646 2.16 1.08 0.3868 Block 3 0.2761 0.2646 2.16 1.04 0.3991 Block 4 0.1560 0.2646 2.16 0.59 0.6113 Block 5 0.1412 0.2646 2.16 0.53 0.6433 Table 510. LSMEANS from PROC GLM Block y LSMEAN 1 0.03457327 2 0.42190439 3 0.35697247 4 0.19015370 5 0.22234738 Note that the DDFM = KR method was used here which uses an adjusted standard error. If the DDFM = SATTERTH option is used, the estimates and degrees of freedom remain the same, but the standard error is reduced to .1647 which would impact the prediction limits as shown in Section 4.5. Given the conclusion in Chapter 4 regarding the inaccuracy of the adjustment under certain circumstances, the accuracy of either standard error estimate and their effect on prediction intervals for the realized values of the random effect needs further study. This remains as future research. For comparison, Tables 511 and 512 contain the results of the analysis of the data in Table 57 if the correlation were ignored and the model were run as a random effects model with the REML NOBOUND option. We see that the variance parameter estimate is negative and the EBLUPs are inverted and expanded as shown in Section 5.1. Table 511. Variance parameter estimates under random effects model Covariance Parameter Estimate Block 0.06110 Residual 0.9497 Table 512. Solution for random effects under random effects model Effect Block Estimate Std Err Pred DF t Value Pr > It Block 1 0.005176 0 29 Infty <.0001 Block 2 0.2487 0 29 Infty <.0001 Block 3 0.2410 0 29 Infty <.0001 Block 4 0.1361 0 29 Infty <.0001 Block 5 0.1232 0 29 Infty <.0001 One issue that occurs with analyzing data in this way is that it is prone to convergence failure. Of 10,000 simulated data sets from the distribution described in Equation (51) and analyzed with this procedure, 21.5% failed to converge when wither REML or REML NOBOUND procedures were used. Even with this complication, this parameterization remains one of the only viable options if predictors of the realized values of the random effects are desired in a case with negative variance component estimation. CHAPTER 6 CONCLUSION AND FUTURE WORK The primary goal of this dissertation was to examine the impact of estimating variance components on the estimators of the MSEP for the EBLUP. This was accomplished by investigating properties of generalized components of the MSEP estimator, examining the behavior of various procedures in the balanced oneway random effects model, and determining the impact of negative variance components on EBLUP procedures. Following the work by Kackar and Haville (1984), Kenward and Roger (1997), Jeske and Harville (1988), and Fai and Cornelius (1996), estimators for the correction term and the bias correction term were generalized to accommodate multiple dimension linear combinations of fixed and random effects. Properties of these generalized estimators were investigated. We showed that while for linear covariance structures the bias correction term estimator, BIAS( (0), continues to reduce to the correction term estimator, C" (0), which is non negative definite, BIASR (0), in general, is indefinite. The indefiniteness of BIASK (0) causes the estimator, M1 (0)+ BIASR (0), of the naive MSEP, M1 (0), to be indefinite. Future work includes the investigation of definiteness properties of the MSEP estimator M1 (0) + C (0) + BIAS' (0), which includes both the correction term and the bias correction term. The magnitude of M1 (0) +C' (0) may outweigh that of BIAS" (0) to produce a positive definite estimator. We also demonstrated that BIAS (0) is not transform invariant, and that the choice of parameterization can have a significant impact on the bias correction term, the MSEP estimate, and the coverage rates of prediction intervals. The balanced oneway random effects model provided a simple setting in which the various components of MSEP estimators could be thoroughly explored. Closed form expressions allowed for direct examination of the correction term and bias correction term estimators. Through this example, the impact of variance component estimation techniques, the values of the variance components, and the parameterization was demonstrated. We determined that the MSEP estimator may significantly over or underestimate the true MSEP depending on these factors. We showed that truncating variance parameter estimates under REML estimation can cause significant (longrun) overestimation of both the correction term and the bias correction term, over most of the parameter space. While these procedures have been considered suspect near the boundary of the parameter space, this study showed that the accuracy of the methods is questionable over a much greater range of the parameter space. We also discovered that the choice of how to measure the MSE of the variance parameter estimates can significantly impact the accuracy of the correction term estimator. In REML situations, using the exact MSE of the variance parameter estimates in the correction term estimator resulted in a more accurate correction term estimator than using the asymptotic MSE over twothirds of the parameter space. However, if variance parameter estimates are left untruncated, as in ANOVA variance component estimates, the asymptotic MSE provided a more accurate correction term estimator over the entire parameter space. The importance of the parameterization of the model is apparent in the bias correction term estimator since it is not transform invariant. Under a linear parameterization, BIAS (0) slightly overestimates the true bias correction term; while under a nonlinear parameterization BIAS( (0) significantly underestimates the true bias correction term for a large portion of the parameter space when using REML variance component estimates. Under ANOVA variance parameter estimation, the linear parameterization provides a significantly more accurate bias correction term estimator. Even with this simple model, we see how the interaction of a few factors complicates the results. The question of which MSEP estimator is best depends on the parameterization, the true value of the variance parameter, and the choice of variance parameter estimation method. Each of these factors deserves investigation in more complex models. These same factors impact the coverage rates of prediction intervals for a realized value of the random effect. We introduced a modification of the FaiCornelius methods for approximating the degrees of freedom that includes a bias correction term in the MSEP estimator. The results of a Monte Carlo simulation study showed that the FaiCornelius based methods compare favorably with the GiesbrechtBurns and KenwardRoger methods currently available in SAS. The simulation study also demonstrated a shortcoming of the Kenward Roger method in that the degrees of freedom are not adjusted for the increased MSEP estimate for a single dimension linear combination. In fact, of the four prediction interval procedures included in the simulation study, the KenwardRoger method consistently performed the poorest, producing the most inflated coverage rates. Because the modified FaiCornelius method adjusts both the MSEP and the distribution for the variance component estimates, this method deserves more attention and may significantly improve the coverage rates or Type I error for hypothesis testing in many situations. Future work includes further investigation of these methods in more complex settings. The impact of negative variance component estimates on the EBLUP and the MSEP estimators was studied. We show that the EBLUP is no longer a reliable or valid estimator of the random effect when negative variance component estimates are present. The proposed solution by Smith and Murray (1984) of regarding the variance parameter as a covariance is shown to be invalid for EBLUP procedures. An alternative solution of a random effects model with correlated errors was presented. Further study of this proposed method is indicated. Future work also includes the investigation of the impact of negative variance parameter estimates on fixed effects estimators, their standard errors, and distribution estimation techniques. APPENDIX DETAILS FOR MONTE CARLO SIMULATION STUDY ON PREDICTION INTERVALS The calculations necessary for the FC (0) and FC (0) prediction intervals are detailed below, followed by an example of the SAS code used for the Monte Carlo simulation study to determine coverage rates for the four prediction interval procedures in Section 4.5. Recall from Equation (464) that the inverse of the observed information matrix, B() = I' (*) and the vectors of derivatives of the MSEP, h(.), are necessary for the "Satterthwaite"method of calculating degrees of freedom. First we look at the FC (D) method. The inverse of the observed information matrix for the D = (2, 2) is B (0) = I1 (0) 2(2 )2 2( 2 )2 k(n1) kn(n 1) (Al) 2(C2) 22 2 ( 2 kn(nl) n2 k1 k(n1) L82 2a, where, due to translation invariance, M1 (0) is as in Equation (43) and tr[A ()B(()] is as in Equation (415). Thus, a8M, () + 2tr [A(D)B(()]} (1 u)[ +(k 1) ] O82 nkv 2(kn 1)a2[2 +2na2 (A2) k2 2 2 k2n(nl)(c2 +na)2 and a{M, (4)+2tr[A(4V)B(04)]} a'(2nu +ka2) +n 2(a)2 0: k(oZ+. o )2 S (A3) 2n(kn 1)(C2) k n(n( 1)2 n)2 The degrees of freedom for the FC (0) method are calculated as 2 [M () + 2tr [A () B (Q)]]2 dc(I) = var [M, ()+ 2tr [A(() B() where varM [M (4)+2tr A ()B ()1] h ) ()h (4)) with the elements of h (D) from Equations (A2) and (A3) and T' (D) from Equation (Al). For the FC (0) method, we turn to the 0 = (o, v) parameterization. The inverse of the observed information matrix is now B(0) = 1 (0) 2(2)2 2(a2)v k(n1) k(n1) (A4) 2(C )v 2vt(kn1) k(n1) k(k1)(n1) We use the MSEP estimator utilizing both the correction term and the bias correction term elements to be consistent with the KenwardRoger and the FC (D) methods. Because the bias correction term estimator is not transform invariant, the MSEP estimate is different than the one used in the D parameterization. The MSEP estimator in the 0 = (2, v) parameterization is given in Equation (460). Thus S (0)+ tr [A(0)B(0)]ltr [A(0) B(0)]1 h(O) = a2 00 ( I1 C j (0) +tr [A(0) B(0)] r [A(0) B(0)] 2 where S{M, (0) +tr[A(0)B(0)] 8a2 trA()B(0) (1 v k 1 nkv 2v(kn1) 4[k(n1)+(k1)v2 k2n(n 1) nk2 (n 1)(k 1)2 { M, (0)+ trA(0) B(0)] trA(0) B(0)] 2 [E(1 k) v2 gv nkv2 (A6) 2o(kn)_ 42[ 2(k1)2 k(nl)+(kl)22] Sk2n(n1) nk2(n1)(k1)v2 The degrees of freedom for the FC(0) method are calculated as 2 2f() [M () + tr [A r )B( ) tr A()B ) var MI (6) + tr [A ()B()] tr [A ()B( where var[M, ()+tr[A(0)B() tr A)B(0) h)B(0)h () with the elements of h () from Equations (A5) and (A6), and IV1 () from Equation (A4). (A5) An example of the SAS code used for the Monte Carlo simulation study to determine coverage rates for the four prediction interval procedures, utilizing the above derivations for the FC(D) and FC(O) methods, follows. /*Sample program for Monte Carlo simulation study of prediction intervals*/ /*Generating Data for random effect and error from normal distributions and creating y ij, observations*/ /*k=3,n=6,sigma(a) squared=10*/ data normal ; do mc = 1 to 10000; do sigma a = (10)**.5 ; do i = 1 to 3 ; stda = rannor(17) do j = 1 to 6 ; e ij = rannor (18) a = sigma a*stda y ij = a +eij ; output ; end end ; end end ; run /*Creates Satterthwaite prediction limits on al*/ proc mixed data = normal method=REML noclprint noinfo noitprint; class i ; model y ij = / ddfm=Satterth ; random i ; estimate 'effect 1'  i 1 0 0 /cl; by mc ; ods listing exclude estimates ods listing exclude fitstatistics ; ods listing exclude covparms ods output estimates=estl ; run ; /*Creates KenwardRoger prediction limits on al*/ /*Outputs KenwardRoger Standard Error for use in FC(() prediction intervals*/ /*Outputs Covariance Parameter Estimates to use in FC(G) calculations*/ proc mixed data = normal method=REML noclprint noinfo noitprint; class i ; model y ij = / ddfm=kenwardroger ; random i ; estimate 'effect 1'  i 1 0 0 /cl; by mc ; ods listing exclude estimates ; ods listing exclude fitstatistics ods listing exclude covparms ods output estimates=est2 ; ods output covparms=cp ; run ; data est ; set estl ; drop Label rename tValue=Satt tval ; rename DF=Satt df ; rename Probt = Satt pvl ; rename StdErr=seSATT ; rename upper = satt uppercl ; rename lower = satt lowercl ; run ; data est2 ; set est2 drop Label estimate; rename StdErr=seKR ; rename tValue=KR tval rename DF=KR df ; rename Probt = KR pval ; rename upper = KR uppercl ; rename lower = KR lowercl ; run ; proc sort data=cp ; by covparm ; data siga resid out set cp ; if covparm='i' then output siga ; else if covparm='Residual' then output resid ; run ; data siga ; set siga ; siga = estimate ; drop covparm estimate ; run ; data resid ; set resid resid = estimate ; drop covparm estimate run ; /*Isolates al for each data set*/ data a ; set normal ; where i = 1 and j = 1 ; run ; data est ; merge estl est2 a siga resid; by mc ; k=3; n=6; upshat = resid/(resid + n*siga) ml = resid*(lupshat)*(1+((kl)*upshat))/(n*k*upshat) /*elements of inverse information matrix for <*/ bll = 2*resid*resid/(k*(n1)) ; bsigsiga = 2*resid*resid/(k*n*(n1)) ; bsiga = 2*resid*resid*(i/(n*n))*((1/(upshat*upshat*(k1)))+(1/(k*(n1)))) /*derivatives of Ml, ( parameterization*/ dl = n*(kl)*siga*siga/(k*(resid+n*siga)*(resid+n*siga)) ; d2 = (resid*(2*n*siga + k*resid) + n*n*siga*siga)/(k*(resid+n*siga)*(resid+n*siga)) ; /*derivatives of 2*trab, ( parameterization*/ dl FC2 = dl + 2*((2*(k*nl)*resid*resid + 4*(k*n l)*n*resid*siga)/(k*k*n*(nl)*(resid+n*siga)*(resid+n*siga))) ; d2 FC2 = d2 2*(2*n*(k*nl)*upshat*upshat/(k*k*n*(n1))) /*dfFC2 = FC(() degrees of freedom*/ dbdFC2 = dl FC2*(dl FC2*bll +d2 FC2*bsigsiga) + d2 FC2*(dl FC2*bsigsiga +d2 FC2*bsiga) ; dfFC2 = 2*(seKR**2)*(seKR**2)/dbdFC2 ; /*critical tvalue for alpha=.05, FC(() degrees of freedom*/ tFC025 = quantile('T',.975,dfFC2); /*FC(() prediction limits*/ FC phi uppercl= estimate +tFC025*seKR ; FC Phi lowercl = estimate tFC025*seKR ; /*MSEP estimate for 8 parameterization*/ msekrbias = ml + 2*resid*upshat*(k*nl)/(k*k*n*(n1)) 2*resid*((k*(n1))((kl)*(k1)*upshat*upshat))/(n*k*k*(n l)*(kl)*upshat) ; sqmsekrbias = msekrbias**(1/2) ; /*elements of inverse information matrix for 8 (along with bll)*/ b12 = 2*resid*upshat/(k*(n1)) ; b22 = 2*upshat*upshat*(k*n 1)/(k*(n1)*(k1)) /*derivatives of Ml, 8 parameterization*/ dlsatt = (1upshat)*(l+(kl)*upshat)/(n*k*upshat) ; d2satt = resid*((1k)*upshat*upshat 1)/(n*k*upshat*upshat) /*derivative of trab + bias term, 8 parameterization*/ dl FC3 = dlsatt + 2*upshat*(k*nl)/(k*k*n*(n1)) (2*k*(n1) 2*(k1)*(k 1)*upshat*upshat)/(n*k*k*(n1)*(k1)*upshat) ; d2 FC3 = d2satt + 2*resid*(k*nl)/(k*k*n*(n1)) + 2*resid*(2*(k1)*(k1)*upshat*upshat + k*(n1) (kl)*(k 1)*upshat*upshat)/(n*k*k*(n1)*(k1)*upshat*upshat) ; /*dfFC3 = FC(G) degrees of freedom*/ dbdFC3 = dl FC3*(dl FC3*bll +d2 FC3*bl2) + d2 FC3*(dl FC3*bl2 +d2 FC3*b22) dfFC3 = 2*msekrbias*msekrbias/dbdFC3 ; /*critical tvalue, alpha .05, FC(G) degrees of freedom*/ tFC025bias = quantile('T',.975,dfFC3); /*FC(8) prediction limits*/ FC theta uppercl FC theta lowercl estimate +tFC025bias*sqmsekrbias ; estimate tFC025bias*sqmsekrbias ; /*Flags each prediction interval containing al*/ if satt uppercl = or satt lowercl = then satt cover = ; else if satt uppercl > a and satt lowercl < a then satt cover = 1 else satt cover=0 ; if KR uppercl = or KR lowercl = then KR cover = ; else if KR uppercl > a and KR lowercl < a then KR cover = 1 ; else KR cover=0 ; if FC phi uppercl = or FC phi lowercl = then FC phi cover = ; else if FC phi uppercl > a and FC phi lowercl < a then FC phi cover = 1 ; else FC phi cover=0 ; if FC theta uppercl = or FC theta lowercl = then FC theta cover = ; else if FC theta uppercl > a and FC theta lowercl < a then FC theta cover else FC theta cover run ; /*Calculates True Coverage Rates for Each Prediction Interval Method*/ proc means data = est n mean stderr 1clm uclm; var satt cover KR cover FC phi cover FC theta cover ; title 'Sigmasq a=10 k=3 n=6' ; title 'Simulated true coverage rates' ; title 'BLUP al' run ; LIST OF REFERENCES Booth, J. G. and Hobert, J. P. (1998), "Standard Errors of Prediction in Generalized Linear Mixed Models," Journal of the American Statistical Association, 93, 262272. Corbeil, R. R. and Searle, S. R. (1976), "Restricted Maximum Likelihood Estimation of Variance Components in the Mixed Model," Technometrics, 18, 3138. Fai, H. T. and Cornelius, P. L. (1996), "Approximate Ftests for Multiple Degree of Freedom Hypotheses in Generalized Least Squares Analyses of Unbalanced SplitPlot Experiments," Journal of Statistical Computation and Simulation, 54, 363378. Ferguson, T. S. (1967), Mathematical Statistics, New York: Academic Press. Giesbrecht, F. G. and Burns, J. C. (1985), "TwoStage Analysis Based on a Mixed Model: LargeSample Asymptotic Theory and Small Sample Simulation Results," Biometrics, 41, 477 486. Harville, D. A. (1976), "Extension of the GaussMarkov Theorem to Include the Estimation of Random Effects," The Annals of Statistics, 4, 384395. Harville, D. A. (1977), "Maximum Likelihood Approaches to Variance Component Estimation and to Related Problems," Journal of the American Statistical Association, 72, 320338. Harville, D. A. (1985), "Decomposition of Prediction error," Journal of the American Statistical Association, 80, 132138. Harville, D. A. (1990), "BLUP and Beyond," In Advances in Statistical Methodsfor Genetic Improvement ofLivestock, eds. D. Gianola and K. Hammond, 239276. Harville, D. A. and Carriquiry, A. L. (1992), "Classical and Bayesian Prediction as Applied to an Unbalanced Mixed Linear Model," Biometrics, 48 ,9871003. Harville, D. A. and Jeske, D. R. (1992), "Mean Squared Error of Estimation or Prediction Under General Linear Model," Journal of the American Statistical Association, 87, 724731. Henderson, C. R. (1950), "Estimation of Genetic Parameters," Annals ofMathematical Statistics, 21, 309310. Hulting, F. L. and Harville, D. A. (1991), "Some Bayesian and NonBayesian Procedures for the Analysis of Comparative Experiments and for SmallArea Estimation: Computational Aspects, Frequentist Properties, and Relationships," Journal of the American Statistical Association, 86, 557568. Jeske, D. R. and Harville, D. A. (1988), "PredictionInterval Procedures and (FixedEffects) ConfidenceInterval Procedures for Mixed Linear Models," Communications in Statistics A. Theory and Methods, 17, 10531087. Johnson, Norman L. and Kotz, Samuel (1970), Distributions in Statistics: Continuous Univariate Distributions (Vol. 2), New York: John Wiley. Kackar, R. N. and Harville, D. A. (1981), "Unbiasedness of TwoStage Estimation and Prediction Procedures for Mixed Linear Models," Communications in Statistics A. Theory and Methods, 10, 12491261. Kackar, R. N. and Harville, D. A. (1984), "Approximations for Standard Errors of Estimators of Fixed and Random Effects in Mixed Linear Models," Journal of the American Statistical Association, 79, 853862. Kenward, M. G., and Roger, J. H. (1997), "Small Sample Inference for Fixed Effects from Restricted Maximum Likelihood," Biometrics, 53, 983997. Khatri, C. G. and Shah, K. R. (1981), "On the Unbiased Estimation of Fixed Effects in a Mixed Model for Growth Curves," Communications in Statistics A. Theory and Methods, 10, 401406. Khuri, A. I. (1993), Advanced Calculus in ith Applications in Statistics, New York: John Wiley & Sons. Khuri, A. I. and Sahai, H. (1985), "Variance Component Analysis: A Selective Literature Survey," International Statistical Review, 53, 279300. Lahiri, P. and Rao, J. N. K. (1995), "Robust Estimation of Mean Squared Error of Small Area Estimators," Journal of the American Statistical Association, 90, 758766. Littell, R. C., Milliken, G. A., Stroup, W. W., Wolfinger, R. D., and Schabenberger, O. (1996), SAS for Mixed Models (2nd ed.), Cary, NC, SAS Institute Inc. Peixoto, J. L., and Harville, D. A. (1985), "Comparison of Alternative Predictors Under the Balanced OneWay Random Model," Journal of the American Statistical Association, 81, 431 436. Prasad, N. G. N. and Rao, J. N. K. (1990), "The Estimation of Mean Square Error of Small Area Estimators," Journal of the American Statistical Association, 85, 163171. Puntanen, S. and Styan, G. P. H. (1989), "The Equality of the Ordinary Least Squares Estimator and the Best Linear Unbiased Estimator," The American Statistician, 3, 153164. Robinson, D. L. (1987), "Estimation and Use of Variance Components," The Statistician, 36, 3 14. Robinson, G. K. (1991), "That BLUP is a Good Thing: The Estimation of Random Effects," Statistical Science, 6, 1551. SAS Institute, Inc. (2003), SAS/STAT User 's Guide in SAS OnlineDoc, Version 9.1.3, Cary, NC. Satterthwaite, F. E. (1941), "Synthesis of Variance," Psychometrika, 6, 309316. Satterthwaite, F. E. (1946), "An Approximate Distribution of Estimates of Variance Components," Biometrics Bulletin, 2, 110114. Searle, S. R. (1971), Linear Models, New York: John Wiley & Sons. Searle, S. R. (1987), Linear Models for Unbalanced Data, New York: John Wiley & Sons. Searle, S. R., Casella, G. and McCulloch, C. E. (1992), Variance Components, New York: John Wiley & Sons. Schaalje, G.B., McBride, J.J. and Fellingham, G.W. (2002), "Adequacy of approximations to distributions of test statistics in complex mixed linear models," Journal ofAgricultural, Biological and Environmental Statistics, 7, 512524. Smith, David W., and Murray, Leigh W. (1984), "An Alternative to Eisenhart's Model II and Mixed Model in the Case of Negative Variance Estimates," Journal of the American Statistical Association, 79, 145151. Stroup, W. and Littell, R. C. (2002), "Impact of Variance Component Estimates on Fixed Effect Inference in Unbalanced Mixed Models," Proceedings of the 14 Annual Kansas State University Conference in Applied Statistics in Agriculture, 3248. Tuchscherer A., Herrendorfer, G., and Tuchscherer, M. (1998), "Evaluation of the Best Linear Unbiased Prediction in Mixed Linear Models with Estimated Variance Components by Means of MSE of Prediction and the Genetic Selection Differential," Biometrical Journal, 40, 949962. Wolfe, D. A. (1973), "Some General Results About Uncorrelated Statistics," Journal of the American Statistical Association, 68, 10131018. BIOGRAPHICAL SKETCH Jamie McClave Baldwin is a native of Gainesville, Florida. She is the daughter of Dr. James T. McClave and Mary Jay McClave, both of whom hold graduate degrees from the University of Florida. She is married to Ian Baldwin and mother to Caryss and Zach Baldwin. She will deliver their third child in January, 2008. She is a graduate of Gainesville High School, Vanderbilt University (B. A., mathematics and economics, magna cum laude, 1997) in Nashville, Tennessee, and the University of Florida (M. Stat., statistics, 1999; Ph.D., December 2007). Jamie began her career in statistics while still in high school, as a data entry assistant at Info Tech, Inc. She has held several internships with Info Tech, Inc., allowing her to gain understanding of each step of the statistical analysis process. While at U. F., she worked as a graduate teaching assistant in the Department of Statistics and as a graduate research assistant in IFAS, Statistics. After graduation, Jamie will remain in Gainesville as a Statistical Consultant at Info Tech, Inc. PAGE 1 1 EVALUATING ADJUSTMENTS TO THE MEAN SQUARED ERROR DUE TO ESTIMATING VARIANCE PARAMETER S IN LINEAR MIXED MODELS By JAMIE MCCLAVE BALDWIN A DISSERTATION PRESENTED TO THE GRADUATE SCHOOL OF THE UNIVERSITY OF FLOR IDA IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY UNIVERSITY OF FLORIDA 2007 PAGE 2 2 2007 Jamie McClave Baldwin PAGE 3 3 To my husband, Ian, and to my children, Caryss, Zachary, and Baby Baldwin PAGE 4 4 ACKNOWLEDGMENTS First and m ost importantly, I thank my husband, Ian Baldwin, for providing endless patience over the last 10 years whil e I have pursued this degree. I would not have been able to endure this process without his unwavering supp ort and constant belief in me, even when I couldnt believe in myself. I thank him for ma intaining both the house and my sanity when times were hard and for making me smile when days were too long to manage. I express my eternal love and gratitude to my children, Cary ss, Zach, and Baby Number 3, for their infinite supply of laughter and hugs. They have insp ired me to push beyond a ny limitation and live life to the fullest. They are the lights of my life a nd the most beautiful souls I have ever encountered and I give thanks for them everyday. My family is a daily demonstrati on of Gods Grace; there is nothing I could ever do to deserve their unconditional love and support. I thank my parents, Jim and Mary Jay McClav e, for their hard wor k, love, patience, and constant support. To my father, I am so grat eful to have such a wonderful role model, both professionally and personally. I am forever indebted to him for continual editing and mathematical advice. I am so fortunate to ha ve the opportunity to work with the smartest and bravest man that I know, not to mention the best cons ultant in the industry. To my mom, I am so thankful for her wise parenting advice, too many hours of childcare to even count, and girl time when it was most needed. I literally could not ha ve finished this monstrous endeavor without her emergency child care services. I also thank them for all of the generosity they have shown me and my family in so many ways. I would also like to thank my brother, Will McClave, his wife, Amber, and my awesome nephews, Jake and Luke. They have been a cons tant support for my family and I look forward to the next chapter of our lives together. I am so blessed to have such a wonderful, supportive family that lives so close by. It is a rarity in this world which I truly appreciate. PAGE 5 5 To the best friends anyone could ask for, I am thankful for their friendship, love, and encouragement through this entire process. I th ank Holly Holly for always helping me see the bright side, laughing at my stories, listening to my endless complaints, and allowing me to learn so much about motherhood and balance from her w onderful example. I thank Heather Bristol for being our pioneer through the statistics program. She has shown me what true bravery is and taught me how to deal with the most unexpected of situations with grac e and strength. I thank Dr. Patches Johnson for encouraging my continua tion when I thought I would go no further, and for being my mentor in so many ways. I tha nk her for being a fantas tic and fun traveling companion and for her contagious laugh. I thank Meghan Medlock Abraham, my college roommate, for a friendship that picks up right wh ere it was left off no matter how much time has passed and for having a mouth big enough to eat her own fist. I thank Brooke Bloomberg, my oldest and dearest friend, for he r sisterly bond and for beating me to the finish line; it was all the motivation I needed! It is amazing that we have both accomplished our childhood ambitions. We rock!! I am grateful for other friends that have he lped me along the way w ith their support and laughter. My sincerest appreciation goes to Andy and Tracy Bachmann, Eric and Ann Bloomberg Beshore, Nicole Provost and Kathy Carroll, Debbie Hagan, Lisa Jamba, Ron and Cindy Marks, Larry and Sandy Reimer, and Yvette and Scott Silvey. I co nsider each one of them family and I am so lucky to have them in my life. Finally, I thank Dr. Ramon Littell for his patience and guidance during this seemingly endless project. Without his loyal support and en during belief, this disse rtation could not have been completed. This degree would not have be en possible without his wealth of knowledge, generous spirit, and encouragement. PAGE 6 6 TABLE OF CONTENTS page ACKNOWLEDGMENTS...............................................................................................................4 LIST OF TABLES................................................................................................................. ..........8 LIST OF FIGURES.........................................................................................................................9 ABSTRACT...................................................................................................................................12 CHAP TER 1 INTRODUCTION AND BACKGROUND........................................................................... 14 1.1 Introduction to Mixed Models ......................................................................................14 1.2 The Linear Mixed Model .............................................................................................. 15 1.3 Best Linear Unbiased Predictors (BL UPs) When is Known: Mixed Model Equations.......................................................................................................................15 1.4 Empirical Best Linear Unbiased Estimation (EBLUP)................................................. 16 1.5 TwoStage Estim ation and Unbiasedness..................................................................... 17 2 LITERATURE REVIEW.......................................................................................................19 2.1 Precision of EBLUPs .................................................................................................... 19 2.1.1 Mean Squared Error of Pred iction (MSEP) of EBLUPs ...................................19 2.1.2 KackarHarville Approxim ation to MSEP........................................................ 22 2.1.3 Estim ating C ...............................................................................................24 2.1.4 Estim ating 1M .............................................................................................25 2.1.5 Estim ators of M Based on the KackarHarville Approximation............... 26 2.2 Hypothesis Testing for Single Di m ension Linear Combinations................................. 27 2.3 Hypothesis Testing M ethods for Multiple Dimension Linear Combinations............... 29 2.3.1 KenwardRoger Multiple Dim ens ion Hypothesis Testing Method.................. 29 2.3.2 FaiCornelius Multiple D imension Hypothesis Testing Methods.................... 35 2.4 Negative Variance Component Estim ates.....................................................................38 3 ESTIMATES OF MSEP FOR MULTIPLE DIMENSION LI NEAR COMBINATIONS.... 42 3.1 Reconciling KenwardRoger (1997) and Fa iCornelius (1996 ) Definitions of the Correction Term Approximation................................................................................... 42 3.1.1 Linear Combinations of Fixed Effects ..............................................................42 3.1.2 Linear Combinations of Fixed and Random Effects.........................................44 3.2 Multip le Dimension Bias Correction............................................................................ 49 3.2.1 Bias Correc tion Estimator for Multiple Dimension Linear Combinations of Fixed Effects.....................................................................................................49 PAGE 7 7 3.2.2 Bias Correc tion Estimator for Multiple Dimension Linear Combinations of Fixed and Random Effects................................................................................ 51 3.2.3 Definiteness Properties of the Estimator for MSEP for MultipleDimension Linear Combinations of Fixed and Random Effects...................... 59 3.3 Transform Invariance of MSEP Estimator.................................................................... 61 4 BALANCED ONEWAY RANDOM EFFECTS MODEL................................................... 63 4.1 KackarHarville Approxim ation Ba sed on REML Estimation Method........................ 64 4.2 KackarHarville Approxim ation Based on ANOVA Estimation Methods...................75 4.3 Bias Correction Term for the Balanced OneWay Random Effects Model.................. 84 4.3.1 Bias Correction Term Approximation under Parameterization..................... 84 4.3.2 Expected V alue of KRBIAS with REML Covariance Parameter Estimation......................................................................................................... 87 4.3.3 Expected V alue of KRBIAS with ANOVA Covariance Parameter Estimation......................................................................................................... 92 4.3.4 Impact of Negative Values of Bias Correction Term Approximation.............. 95 4.4 Perform ance of Overall Estimators for the MSEP........................................................ 98 4.5 Comparison of Prediction Interval Methods............................................................... 101 5 THE EFFECT OF NEGATIVE VAR IANCE COMPONENTS ON EBLUP...................... 137 5.1 Negative Variance Component Estim ates and the Balanced One Way Random Effects Model..............................................................................................................137 5.2 Considering the Variance Param eter as a Covariance................................................ 141 5.3 BLUP derivation for Random Effect s Model with Correlated Errors........................ 143 6 CONCLUSION AND FUTURE WORK............................................................................. 148 APPENDIX DETAILS FOR MONTE CARLO SI MULATION ST UDY ON PREDICTION INTERVALS........................................................................................................................152 LIST OF REFERENCES.............................................................................................................159 BIOGRAPHICAL SKETCH.......................................................................................................162 PAGE 8 8 LIST OF TABLES Table page 21 Summary of developments for MS EP estim ation and hypothesis testing......................... 40 41 Accuracy of KackarHarville estimator (417) for RC ................................................71 42 Relative bias (%) of estimators of correction term............................................................ 84 43 Relative bias (%) of MSEP estimators under REML and ANOVA estimation.............. 100 44 Summary of prediction interval procedures. .................................................................... 102 45 Effect of prediction interval procedures for k = 6, n = 6, 2 1 and 2 .5a ................ 108 51 Generated data that produces negative variance component estimate............................. 138 52 REML variance parameter estimates............................................................................... 138 53 Solution for random effects with RE ML variance parameter estimates.......................... 138 54 REML NOBOUND variance parameter estimates.......................................................... 139 55 Solution for random effects with REML NOBOUND variance param eter estimates.....139 56 LSMEANS from PROC GLM......................................................................................... 139 57 Data generated from distribution in Model (51)............................................................. 145 58 Covariance parameter es tim ates under Mode1 (51)....................................................... 146 59 Solution for random effects under Model (51)............................................................... 146 510 LSMEANS from PROC GLM......................................................................................... 146 511 Variance parameter estimates under random effects model............................................147 512 Solution for random effect s under random effects model............................................... 147 PAGE 9 9 LIST OF FIGURES Figure page 41 Equation (413) as a function of for several values of k, holding n = 6. Equation (414) holds when functions cross zero line.................................................................... 109 42 Accuracy of approximations Rtr A B and tr 1A I for the correction term, RC under REML variance component estimation......................... 110 43 Accuracy of approximation Rtr A B and estimator REtr A B for the correction term, RC under REML variance component estimation.................... 111 44 Accuracy of approximation tr 1A I and estimator Etr 1A I for the correction term, RC under REML variance component estimation.................... 112 45 Comparison of accuracy of estimators for the correction term, RC under REML variance component estimation........................................................................................ 113 46 Comparison of accuracy of approximations tr 1A I Atr A B and Atr A B for the correction term, AC under ANOVA variance component estimation...................................................................................................... 114 47 Accuracy of approximation Atr A B and estimator AEtr A B for the correction term, AC under ANOVA variance component estimation............... 115 48 Accuracy of approximation Atr A B and estimator AEtr A B for the correction term, AC under ANOVA variance component estimation................ 116 49 Accuracy of approximation tr 1A I and estimator Etr 1A I for the correction term, AC under ANOVA variance component estimation................ 117 410 Comparison of accuracy of estimators for the correction term, AC under ANOVA variance component estimation........................................................................ 118 411 Accuracy of estimators for the correction term under REML, and ANOVA, variance component estimation........................................................................................ 119 PAGE 10 10 412 Relative bias of estimators fo r the correction term under REML, or ANOVA, variance component estimation........................................................................................ 120 413 Bias produced by estimating 1M with 1M.......................................................... 121 414 Accuracy of approximation 1 2 tr 1 I and estimator 1 2 Etr 1 I for the true bias correction term, 11 MEM under REML variance component estimation................................................................. 122 415 Accuracy of approximation tr 1A I and estimator Etr 1A I for the true bias correction term, 11 MEM under REML variance component estimation..................................................................................................................... ....123 416 Comparing the accuracy of 1 2 Etr 1 I and Etr 1A I for the true bias correction term, 11 MEM .......................................................... 124 417 Bias produced by estimating 1M with 1M .......................................................... 125 418 Accuracy of approximation 1 2 tr 1 I and estimator 1 2 Etr 1 I for true bias correction term, 11MEM under ANOVA variance component estimation........................................................................ 126 419 Accuracy of approximation, tr 1A I and estimator Etr 1A I for true bias correction term, 11MEM under ANOVA variance component estimation..................................................................................................................... ....127 420 Comparing the accuracy of 1 2 Etr 1 I and Etr 1A I for the true bias correction term, 11MEM .......................................................... 128 421 Comparison of estimators of the MSEP for the EBLUP o f ia utilizing REML variance component estimation........................................................................................ 129 PAGE 11 11 422 Comparison of estimators of the MSEP for the EBLUP o f ia utilizing ANOVA variance component estimation........................................................................................ 130 423 Relative bias of possible es tim ators for MSEP of EBLUP for ia ................................... 131 424 Simulated coverage rates of pr ediction interval p rocedures for 1ta with k = 3...........132 425 Simulated coverage rates of pr ediction interval p rocedures for 1ta with k = 6...........133 426 Simulated coverage rates of pr ediction interval p rocedures for 1ta with k = 15.........134 427 Simulated coverage rates of pr ediction interval p rocedures for 1ta with k = 30.........135 428 Simulated coverage rates for the Kenw ardRoger prediction in terval p rocedure............ 136 PAGE 12 12 Abstract of Dissertation Pres ented to the Graduate School of the University of Florida in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy EVALUATING ADJUSTMENTS TO THE MEAN SQUARED ERROR DUE TO ESTIMATING VARIANCE PARAMETERS IN LINEAR MIXED MODELS By Jamie McClave Baldwin December 2007 Chair: Ramon Littell Major: Statistics Several methods are commonly used to handl e the propagation of va riance stemming from estimating unknown variance componen ts in Linear Mixed Models (LMM). The accuracy and consequences of using these methods, however have not been thoroughly investigated. Empirical Best Linear Unbiased Predictors (E BLUP) for analyzing LMMs are widely used, yet the best way to evaluate the precision of the EBLUP is not generally understood. Many developments in the estimation of the Mean Squa red Error of Prediction (MSEP) of the EBLUP, and the use of these estimates for hypothesis testi ng, have occurred during the last two decades. This dissertation begins with a thor ough review of these developments. Existing methodologies for evaluating the precis ion of the EBLUP are generalized to apply to multiple dimension linear combinations of fixed and random effects, and the definiteness properties of these methodologies are examined. The methods for evaluating the MSEP of the EBLUP are examined thoroughly for the balanced oneway random effects model to assess the accuracy of the methods, the effect of the para meterization of the model, and the impact of variance parameter estimation techniques on the components of the MSEP estimators. The impact of negative variance component estimates on EBLUP methodologies is examined, and an PAGE 13 13 alternative solution to account for negative variance estimates is developed for the oneway model. PAGE 14 14 CHAPTER 1 INTRODUCTION AND BACKGROUND 1.1 Introduction to Mixed Models In 1984, Kackar and Harville introduced a new m ethod to handle the propagation of variance stemming from estimating unknown variance parameters in linear mixed models. In the nearly quarter century since this development, there have been many adjustments, improvements and new techniques resulting from their idea. Lin ear mixed models and th e procedures used to analyze them are full of complexities and nuances that can confuse even the most experienced analyst. For example, complex covariance struct ures coupled with unbalanced data and variance parameter estimates on or near the boundary of the parameter space can produce confounding results with many mixed models procedures. Th is paper begins to unwind and isolate these complexities using simple cases to explore th e best options available and determine when procedures for estimating the realized values of the random effects and evaluating the variability of these estimates succeed and fail. Mixed models methodologies have develope d over time to accommodate situations where prediction of the realized va lues of the random effects is requ ired. These situations arise, for example, in agriculture, education, and clinic al trials. A typical ag riculture example is the need to assess the genetic contribut ion of a particular animal for br eeding, which is considered to be drawn from a population of animals. An exam ple from education is th e desire to assess the effect of a particular teacher on the test scores of his or her pupils. A clinical trial with multiple clinics and varying patient dropout rates gives another arena where mixed models are frequently used to analyze data. Many of these analyses involve complex c ovariance structures and almost all analyses involve unknown cova riance parameters. It is im portant not only to develop accurate methods for evaluating the realized value of the random effect, but also to evaluate the PAGE 15 15 associated mean squared error to determine the precision of the estimate. We begin by describing the framework of the model. 1.2 The Linear Mixed Model The general for m of the linear mixed model is yX Zue (11) where X is a known (n x p ) design matrix for the fixed effects, is a ( p x 1) vector of fixed effect parameters, Z is a known (n x q ) design matrix for the random effects, u is a ( q x 1) vector of random effects, and e is an ( n x 1) vector containing the ra ndom error terms. We will assume that Eu0, VuG, and E e0, V eR, cov u,e0. From the model in Equation (11), it follows that EyX and V y VZGZR. The matrices G and R are taken to have known structures but depend on an unknown dispersion parameter, for some known parameter space Multivariate normal distributions on u and e are often assumed and many of the results are derived under normality assumptions; the validity of these results under other distributional assumptions is questionable. For simplicity, all matrices are assumed to be of full rank. 1.3 Best Linear Unbiased Predictors (BLUPs) When is Known: Mixed Model Equations Best Linear Unbiased Predictors (BLUPs) originated from the desire to estimate the realized values of the random effects along with the fixed eff ects, or predictable linear combinations of both types of effects. Th is method was originally developed by Henderson (1950) for estimating genetic merits in animal breeding. Henderson approached the problem using maximum likelihood methodology, although he later acknowledged that his method cannot be called maximum likelihood since the func tion being maximized is not a likelihood (Robinson 1991, p.18). Assuming that u and e are normally distributed, Henderson maximized PAGE 16 16 the joint density of y and u with respect to and u. This method results in Hendersons mixed models equations (MMEs): 1 1 1 1 11 1XRX XRZuXRy ZRX ZRZGuZR y (12) Note here that G and R are assumed fully known. The solutions to these equations, 1 1 1 1 =XVXXV y u=GZVyX (13) are the Best Linear Unbiased Estimate (BLUE) and Best Linear Unbi ased Predictor (BLUP), respectively: they are linear in the data, unbiased, and achieve minimum variance or mean squared error (MSE), respectivel y, among all linear, unbiased estimat ors or predictors (see, for example, Harville 1990 and Robinson 1991). Note that the solution for in Equation (13) gives th e GLS estimate. Robinson summarized several other justifications for the BLUP estimates, including a Bayesian method with a flat prior on and a N0,G prior on u yielding the same result as the MMEs, as well as justif ications that do not require normality. A linear combination of fixed and random effects is pr edictable when the fixed effect portion is estimable. That is, for vectors and + u is predictable if =aX for some a (Littell, Milliken, Stroup, Wolfinger, and Schabenberger 2006, p. 211). When estimation of a predictable linear combination of fixed and random effects is desired, + u gives the BLUP for the predictable function. 1.4 Empirical Best Linear U nbiased Estimation (EBLUP) The estimates in Equation (13) assume G and R (and thus V) are fully known; this assumption, however, is rarely satisfied. A common alternative is to assume G and R are known PAGE 17 17 up to a dispersion parameter 1,,r thus G=G and R=R The BLUP solutions depend on G and R, and therefore which is typically unknown. The standard way of resolving this issue is to employ twostage es timation; that is, first estimate the dispersion parameter by, say, then use these estimates in the equations for the BLUP of the predictable function, yielding + u, where and u are the solutions to Equations (12) with estimating : G=G R=R V=V 1 1 1 1 =XVXXV y u=GZVyX These predictors are often referred to as Em pirical BLUPs (EBLUP), although they are no longer linear functions of the data, th ey are not necessarily unbiased, and they no longer necessarily have minimum MSE. There are many ways to estimate including ANOVA, MINQUE, MIVQUE, maximum likelihood (ML), and restricted maximum likeli hood (REML). The REML methods tend to be favored from a theoretical perspective (Searle, Casella, and McCulloch, 1992). However, from a data analytic perspect ive it is unclear which method is most advantageous. The technicalities of these methods will not be reviewed here. Fo r an overview of the technical details of these estimation methods see, for example, Khuri and Sahai (1985) and Robinson (1987). We will review the advantages and disadvantages of the methods as necessary in the context of evaluating the precision of the EBLUP methodology. 1.5 TwoStage Estimation and Unbiasedness The consequ ences of twostage estimation on the estimates of the linear combination, + u, are discussed in Kackar and Harville ( 1981). They show that when the dispersion PAGE 18 18 parameter estimates are translation invariant, even functions of the data, and the data are symmetrically distributed, the twostage estima tion procedure resulting in the EBLUPs provides unbiased predictors of linear co mbinations of effects; i.e., E + u Khatri and Shah (1981) provide similar results. Kackar and Ha rville (1981) show that these conditions are satisfied by assuming normality of the random effects and error and using REML, ML, ANOVA, MINQUE or MIVQUE variance component estimation methods. Note that this result can be extended to the broader class of location equivariant estimators, as shown in Kackar and Harville (1984), citing a theorem by Wolfe (1973). In evaluating the precision of EBLUPs, we must account for sampling variability contributed by the estimation of When this added variability is ignored, the true precision is often grossly underestimated, as demonstrated by Kackar and Harville (1984), Harville and Jeske (1992), and Tuchscherer, Herrendorfer, and Tuchsc herer (1998). Goals of this dissertation include clarifying and compari ng existing methods of estimating the precision of the EBLUP, assessing the accuracy of these methods and ev aluating the methods of hypothesis testing which utilize the various methods of estimating the precisi on. After reviewing th e related literature in Chapter 2, we generalize results on evaluating the precision of the EBLUP to include multiple dimensional cases and random effects in Chapte r 3. A case study of the balanced oneway random effects model in Chapter 4 sheds light on the problems with existing methods and suggests possibilities for improvement. Chapte r 5 tackles the issue of negative variance component estimates and the impact on EBLUP me thodology. Conclusions and future work are summarized in Chapter 6. PAGE 19 19 CHAPTER 2 LITERATURE REVIEW 2.1 Precision of EBLUPs 2.1.1 Mean Squared Error of P rediction (MSEP) of EBLUPs Let t + u and let + u denote the estimate of t when (and hence V ) is known. When is unknown, denote the estimate of t, the EBLUP, by u. The prediction error for the EBLUP is t and the mean squared error of prediction (MSEP) is 2 M Et. By decomposing the prediction error into four components, each assuming a different, successive state of knowledge, Harville (1985) shows how each component adds to the variability. The first state of knowledge supposes full knowledge of the joint distribution of (y, t). The unbiased minimum MSE predictor of t in this case is clearly EtE y + uy(Harville 1985, sec. 2.1). This is known as the Best Predictor. The second state of knowledge supposes th e first and second mome nts are known, i.e., and are known but the functional form of the joint distribution of (y, t) is unknown. In this case the predictor with minimum MS E among all linear predictors is where 1 1 1 + GZZGZ+R y X GZZGZ+Ry GZZGZ+RX (21) (Harville 1985, sec. 2.2). This is known as the Best Linear Predictor. PAGE 20 20 The third state of knowledge assumes only th e variances and covari ances are known, i.e., is known but and the functional form of the joint distribution of (y, t) are unknown. Since depends on we must replace with an estimate. Choosing the BLUE for results in where B B 1 1y GZZGZ+Ry y GZZGZ+RX (22) (Harville 1985, sec. 2.3). This is the Best Linear Unbiased Predictor (BLUP). The fourth state of knowledge represents th e typical situation in data analysis: all moments and the form of the joint distribution of (y, t) are unknown. That is, now is also unknown. Because the BLUP in Equation (22) depends on we cannot direc tly use it without further adjustment. Let z be maximal invariant, that is, let z L y where L is an n x (n p) matrix with rank(L) = n p and 0 LX. A function g y is translation invariant if and only if gh y L y Thus let z be an arbitrary translation invariant estimate of Then BBdPzyy; ; where ; Pz is an arbitrary probability distribution on gives a feasible estimate of t. The Empirical Best Linear Unbiased Predictor, EBLUP, corresponds to ;By the value of B y when ;Pz is the degenerate probability distribution assigning all mass to (Harville 1985, sec. 2.4). Note that B y is translation invariant, thus preserving the unbiasedness of the predictor. With this notation, we now have the f our components of th e prediction error: 3 12 4 BBBtEttEt yyy; (23) PAGE 21 21 Each successive component is associated with moving from one state of knowledge to the next. This decomposition allows analysis of the role of each component in the MSEP. Component (1) of Equation (23) is uncorrelated with all other com ponents by the following result (Harville 1985, p. 135): cov((),()))()() ()(()) 0. hEttEhEtt EEhEtt yyyy yyy (24) Components (2) and (3) of Equation (23) are also uncorrelated since cov(,())0Et yy and B is linear in y (Harville 1985, p. 135). Thus the Mean Squared Error of Prediction (MSEP), M can be written as the following six components: 2 2 222 (1) (2) (3) (4) (5) (6) () () ; (); ;.BBB BBBBBMEt EEtytEEtyEEy EEtyyEy (25) In the development of estimators for M it is often assumed that the data follow a normal distribution. When the joint distributions of zt and ,Bz y are multivariate normal, as is the case when u and e are normally distributed, then components (5) and (6) of Equation (25) are 0. However, without normality, th ere is no guarantee that these crossproduct term s disappear. The remainder of this chapter will review the developments over the last two decades for evaluating the MSEP of the EBLUP and conducting tests of hypotheses. Table 21 contains a PAGE 22 22 summary of the developments reviewed herein and stands as a helpful reference for these developments and the notation used throughout. 2.1.2 KackarHarville Approximation to MSEP Kackar and Harville (1984) and Harville a nd Jeske (1992) assume normality in developing an approximation and estimator for M Thus, only components (1) (4) of Equation (25) need to be assessed for the approxim ation. Components (1), (2), and (3) of Equation (25) com prise the MSEP when is known. This is referred to as the nave MSEP and will be denoted by 2 1 B M Et In the mixed model case with V(u) = G and V(e) = R, this is 1M 1 1 1 1 1 G GZVZG + GZVXXVX XVZG When normality is assumed, the crossproduct te rms are zero, and the only contribution to the MSEP of estimating is component (4) of Equation (25). We will refer to this component as the correction term and denote it by C. 1M is often used as an approximation for the MSEP. When the data are normally distributed, this can seriously underestimate the MSEP since it ignores the contri bution of estimating captured by C, as shown by Kackar and Harville (1984), Harville and Jeske (1992), and Tuchscherer et al. (1998). Peixoto and Harville (1986) expand on results in Kackar and Harville (1984) to derive exact, closedform expressions of the MSEP for certain predictors for the special case of the balanced oneway random effects model. We use similar methods in Chapter 4 to enable comparisons of approximations and estimators to the exact MSEP in the balanced oneway random effects model. Assuming normality, Kackar and Harville (1984) provide an approximation to C using Taylor series methods. Using a Taylor series expansion of about PAGE 23 23 yields () d y ; where ()() dy; y; This leads to the approximation, 2 2 2 = d( y ; ) (26) Note that when is linear in this Taylorseries approximati on is exact. The validity of the Taylorseries approximation is contingent on two conditi ons: First, the function being expanded must have derivatives of all orders defined in a neighborhood of the point about which the function is being expanded; and second, the remainder term mu st tend quickly to zero (Khuri 1993, p. 111). Thus if the parameter space, is restricted (i.e., bounded below by zero) and the true value of falls on the boundary of the parameter space, the Taylor series expansion is not valid. Assessing the effect of the parameterizat ion and parameter estimates on the estimates of the MSEP is one goal of this dissertation. When is unbiased for and cov,0 dy; dy; we have 2 EEEE 2 d( y ; ) d y ; d y ; Noting that 0E dy; Kackar and Harville (1984) propose the following approximation to C: Ctr A B (27) where, 2var() = E A dy; d(y; ), and B is either an approximation to or the exact value of the MSE matrix, E The inverse of the observed information matrix, 1I is commonly used for B Equation (27) will be referred to as the KackarHarville PAGE 24 24 approximation for the correction term. Thus th e approximation of the MSEP of proposed by Kackar and Harville (1984) is 1Mtr A B (28) The accuracy of the KackarHarville approximation of the correcti on term in Equation (27) depends on the following conditions: 1. linear in ; 2. the validity of Taylor series expansion of about ; 3. unbiased for ; 4. cov 0 dy; dy; ; 5. the tractability of the exact MSE of If all of these criteria are me t, the KackarHarville approximation exactly equals the correction term it is approximating. 2.1.3 Estimating C Because the calculation of th e KackarHarville approximati on to the MSEP in Equation (28) requires knowing it is not directly functional. An estimator based on the KackarHarville approximation was developed by Harville and Jeske (1990) and Prasad and Rao (1990) and is now implemented in the MIXED procedure of SAS. Following developments from Prasad and Rao (1990), an estimator for tr A B is given by tr A B ; where is substituted for in the KackarHarville approximation for the correction term. The justification for this substitution is not directly evident. Pr asad and Rao (1990) show that in the special case of the FayHerriot small area model, in which V(e) = R is known, the order of approximation is 1()oq where q is the number of small areas, or in ge neral the number of levels of the random effect. This result has not been generalized for the typical case when R is unknown. The PAGE 25 25 validity of this substitution and the consequences on the accuracy of the MSEP estimate will be examined for the balanced oneway random effects model in Chapter 4. 2.1.4 Estimating 1M Historically, 1M has often been estimated by 1M that is, by substituting for in 1M; however, 1M may have bias of the same or hi gher order as the correction term (Prasad and Rao 1990; Booth and Hobert 1998). Consequently, it is just as important to account for the bias introduced by estimating 1M with 1M as it is to account for C Prasad and Rao (1990), Harville and Jeske (1992), and Kenward and Roger (1997) all approximate the bias of 1M by taking the expectatio n of a Taylor series of 1M about (assuming is unbiased for ). Harville and Jeske (1992) developed the following expression for the bias of 1M : 111 2EMMtr B (29) where 2 1 2M They show that when cov,t y vart and var y are all linear in 2 =A leading to 11EMMtr A B (210) Prasad and Rao (1990) and Kenward and Roge r (1997) developed expressions similar to Equation (210). Again by substituting for possible estimates for 1M are PAGE 26 26 11 2Mtr B (211) or 1Mtr A B (212) Notice again that there is bias introduced by substituting for in Equations (211) and (212). This bias is often assumed to be negligible, although the only noted justif ication, previously m entioned in Section 2.1.3, comes from Prasad and Rao (1990) and only applies to Equation (212) in the FayHerriot model. We will stud y this issu e further through a special case in Chapter 4. 2.1.5 Estimators of M Based on the KackarHarville Approximation Based on Equations (211) and (212), respectively, the following estimators of the MSEP follow from the KackarHarville approxim ation and the bias correction approximations: 11 2Mtrtr B A B (213) and 1 2Mtr A B (214) The DDFM = KR option of the MIXED procedure in SAS utilizes Equation (213) which reduces to E quation (214) when the model covariance structure is linear in Chapter 3 explores the Kenward and Roge r (1997) and Fai and Cornelius (1996) expansion of Equation (213) into multiple dimensions. Therein we establish the equality of the KenwardRoger and FaiCornelius approxim ations to the correction term and expand the term to include linear combinations containing random eff ects. We also provide a proof that the most PAGE 27 27 general form of the correction term approximation is nonnega tive definite and discuss the definiteness properties of the overall MSEP estimator. 2.2 Hypothesis Testing for Single Dimension Linear Combinations Several methods of approximating the distributions for test statistics have been developed for single and multiple dimension linear combinatio ns. The accuracy of these methods has been largely untested. Following is a summary of the available methods for approximating the distributions for single dimension linear combinations. Giesbrecht and Burns (1985) and Jeske and Ha rville (1988) provide approximations for ttype test statistics based on the Satterthwaite (1941) method. Giesbrecht and Burns (1985) present a test statistic for fixed effect s with the nave variance estimate: GBGB dftt 1 1 XVX (215) When V is known and used in place of Vin Equation (215), the test statistic has a normal distribution. Thus, it is assum ed that Equation (215) has a tdistribution with the degrees of freedom derived from the approximate distribution of the denominator. To estimate the degrees of freedom, a Satterthwaite method is used. Th e variance of the denominator is matched with that of the approximating 2 distribution as follows: 2 2 2 var 2 var 2 2 varGB GB GB GB GBdf df df df df 1 1 1 1 1 1 1 1 1 1 1 1 XVX XVX XVX XVX XVX XVX (216) PAGE 28 28 The term var 1 1 XVX is evaluated using a Taylor series approximation and the asymptotic covariance matrix of In the GiesbrechtBurns case, this yields 11 var cov, .rr ij ij i j 1 1 1111 1 1 1 1V XVX XVZZVXXVX V XVZZVXXVX (217) The underlying assumption in developing th is procedure is that the distribution of 1 1 XVX is a multiple of a 2 distribution. Jeske and Harville (1988) also use a Satterthwaite method to approximate the degrees of freedom of a tstatistic; however, they include fixed and random effects in their test statistic with a corrected version of the MSEP estimate. The approximate ttest is given by 1 J H J Hd ftt Mtr A B (218) where 2 1 1 2 varJHMtr df Mtr A B A B (219) As in the GiesbrechtBurns t test, the denominator of Equation (219) is approximated by a Taylor series expansion yielding 1 var,Mtr A B h B h (220) where 1 and is an approximation of var. Mtr A B h B PAGE 29 29 Again, an underlying assumption is that the distribution of 1 Mtr A B follows a multiple of a 2 distribution. An extension of this method would be to incorporate the bias correction into the MSEP as seen in Equations (213) and (214). This extension is considered in Chapter 4 2.3 Hypothesis Testing Methods for Mult iple Dim ension Linear Combinations Kenward and Roger (1997) and Fai and Cornelius (1996) offer F tests to handle multiple dimension as well as single dimension cases; however, the methods for determining the distribution of the test statis tics differ. One of the Fai and Cornelius methods (DDFM = SATTERTH option) and the Kenwar dRoger method (DDFM = KR op tion) are available in the SAS MIXED procedure. A goal of this dissertati on is to compare these methods with other modified methods from Fai and Cornelius ( 1996) for the balanced oneway random effects model. We will also assess the effect of sample size, parameterization, and parameter value on these hypothesis testing methods. 2.3.1 KenwardRoger Multiple Dime nsion Hypothesis Testing Method The test statistic develope d by Kenward and Roger (1997) utilizes an MSEP estimate for multiple dimension linear combinations that is ad justed for the correction term and the bias of 1 M This MSEP estimate is thoroughly discussed in Chapter 3, but for now will be denoted by KRM Note that these methods are only devel oped for tests of fixed effects; thus the MSEP can be thought of as a variance. Theoretically these methods could easily extend to include random effects. We include a si mulation study for the oneway random effects illustration to determine the validity of in cluding random effects in these procedures. PAGE 30 30 Consider the hypothesis vs. 0aH:L =0H:L 0, where L is an l dimension matrix of constants representing the linear combination of interest. Kenward and Roger (1997) propose the following Waldtype test statistic: 11 KR KRF l LLM LL An F distribution is approximated for the test statistic to take in to account the random structure of KRM and to match known cases where the test statistic has an exact F distribution. We will briefly review the procedure to allow comp arisons to alternative methods and to examine any shortcomings. Kenward and Roger (1997) use a Taylor series expansion to approximate 1KR LM L; however, it is unclear about which point the series is expanded and which higher order terms are dropped. It app ears that the series is expanded about 1M rather than the true value, M This may be justified by dropping higher order terms; however, this step is the only place where the corrected MSEP is take n into account in the distribution of the test statistic. Using this Taylor series approximati on, Kenward and Roger (1997) use the identities  KR KRKR KR KR KR KR KREFEEF VFEVF VEF M M M to approximate the moments of th e test statistic. A twomoment matching method is then used to obtain estimates for a coefficient, and degrees of freedom, m so that K R K RldfFF PAGE 31 31 To make the approximating distribution match th e exact cases where the distribution is known, selected higher degree terms are added to th e Taylor series expans ion. The following summarizes the results of th e KenwardRoger procedure: 2 4 and = 12KR KR KRKRdf l df lE Fd f where 2, 2KR KRVF EF and 1 21 2 2312 1 and 11KR KRAc B EF VF ll cBcB where 1 11 i j ,rr ij ijABtr tr 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1V LLXVXLLXVXXXXVX V LLXVXLLXVXXXXVX (221) 2 11 i j ,rr ij ijABtr 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1V LLXVXLLXVXXXXVX V LLXVXLLXVXXXXVX PAGE 32 32 12 1 2 3 12 21 6, 2 321 321 2 321 14 2 BAA l g c lg lg c lg lg c lg lAlA g lA and Bij is the ijth element of the matrix B The KenwardRoger method is available in the PROC MIXED procedure in SAS as the DDFM = KR option in the model statement. This option performs the adjustment to the MSEP estimate and the hypothesis test as described above. We now demonstrate that in the singledim ension case, the Kenwar dRoger degrees of freedom approximation reduces to the Giesbrec htBurns degrees of freedom approximation reviewed in Section 2.2.1 (Kenward and Roger 1997, p. 988). Recall that the GiesbrechtBurns test statistic did not use an adjustment for unknown in the MSEP. Thus, while the KenwardRoger and GiesbrechtBurns test statistics di ffer in the estimate of standard error, the approximated distributions are identical. Consider again a hypothesis test for th e single dimension linear combination :0oH under the linear mixed model. The Ke nwardRoger test statistic is KR K Rt VAR (222) where PAGE 33 33 1 11 2 11 2 1 4KR ij ij ijijVAR B 1 11 11 1111 V=VVV XVX XVXXVX VVV XXXVXXXXVVXXVX In this case 1 11 1 cov, *rr ij ij i j r ijAt r tr 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 1V XVX XVZZVXXVX V XVX XVZZVXXVX XVX 1 2 cov, *r ij i jA 1 111 1 111V XVZZVXXVX V XVZZVXXVX since the quantities inside the trace func tions are scalars. Note also that 122 22 2 1 217 6, 22 25 1, 3 11 3(1)2(11)7 2 7 B AAA AA g A c c and 34 7c PAGE 34 34 Thus, 1 2 2 2 22 22 221, 17 1 2 72 2, 112 2747 11 7272KR KREFA A A VF AA AA and 2 2 22 2 2 221 22 2212 112 A AA A AA The degrees of freedom for the Kenw ardRoger hypothesis test are then 2 2 2 1132 4 2 1 212 2c o v KR rr ij ij i jdf A A A 2 1 1 1 1 1 1 1 1 1 1V XVX XVZZVXXVX V XVZZVXXVX (223) Note that the denominator of Equation (223) is the Taylor series expression for var 1 XVX This is the same as the degrees of freedom estimate from the GiesbrechtBurns test in Equations (216) and (217); however, recall that the GiesbrechtBurns test statistic is GBt 1 1 XVX Thus K RGBdfdf but K RGBtt Often, the discrepancy in these test statistics may not be enough to make a significant diffe rence in the test results; how ever, there are cases when the PAGE 35 35 discrepancy changes the conclusion of the test. The question becomes, which test is better? The accuracy of the estimating distribution must also be questioned in either test. Note that because the estimated distributions are equal, we can isolate the impact of modifying the estimated variance of the EBLUP on the hypothesis test. We explore that situ ation further though a simulation study in Chapter 4. 2.3.2 FaiCornelius Multiple Dime nsion Hypothesis Testing Methods Fai and Cornelius (1996) also use a moment matching method to determine the distribution for their multiple dimension test st atistics. The differences between the KenwardRoger method and the FaiCornelius method come in the particular MSEP estimate used in the test statistic, in the number of moments evaluated, and in the techniques used to evaluate the moments of the test statistic. Fai and Cornelius (1996) offer two alternatives for the test statistic: one uses the nave MSEP estimate in matrix form, 1M and the other uses the nave estimate plus a correction term denoted by, FC FC1M M C which will be discussed thoroughly in Chapter 3. Note that neither of these ta kes into account the bias of 1M for 1M as in the KenwardRoger procedure. The test statistics are 1 111 FCF l LLM LL (224) and 1 21 FC FCF l LLM LL (225) For each of these test statistics, both one and two momentmatching methods are offered for estimating an F distribution, resulting in four alternatives for conducting a multiple dimension hypothesis test. The method of finding the moments is the same for both test statistics, so for PAGE 36 36 simplicity let MSEP denote the estimator of the MSEP in eith er FaiCornelius test statistic. To evaluate the moments of the test statistics, fi rst a spectral decomposition is performed so that kdiag PLMSEPLP where P is an orthogonal matrix of eigenvectors and k 1,2,, kl are the eigenvalues of ) L(MSEPL. The test statistic is rewritten so that 1 1 1 2 1 2 1 .kFC k l k k k l klF diag t Q PPLLMSEPLPPL PL PL PLMSEPLPPL PL PL PL PL PL PL PL (226) They then note that kt would have an independe nt standard normal distribution if the MSEP were known. It is reasonable then to approximate the distribution of kt with a tdistribution and estimate the degrees of freedom, k with a Satterthwaite procedure, i.e., 2 2 kk k k kV V VV MSEP PL PL MSEP PL PL As in the KenwardRoger procedure, we have the underlying assumption that MSEP has a 2 distribution. If 2k for all k= 1, 2, ., l, then the moments of Q can be approximated as follows: PAGE 37 37 2 112kll k kk kEQEt and 2 2 2 1121 24kll kk kk kkVQVt From Equation (226) we see that the moments of the FaiCo rnelius test statistics are 1FC E FEQ l and 21FCVFVQ l. One or both of these moments are matche d to the moments of the approximating Fdistribution, as in the KenwardRoger procedure, to produce an estimate of degrees of freedom (and an estimated coefficient in the twomoment case). The onemoment method with the test statistic in Equation (224) is available in the P ROC MIXED procedure of SAS as the DDFM = SATTERTH option in the model statement. Note that in the single dimension case, the FaiCornelius test statistic in Equation (224) reduces to the GiesbrechtBurns test statistic in Equation (215) and that the FaiCornelius onemo m ent method of estimating the distribution also produces the same results as the GiesbrechtBurns method. Thus we are avoiding the discrepancies that occur betw een the KenwardRoger method a nd the GiesbrechtBurns method described in Section 2.3.1. We are interested in comparing the available hypothesis testing methods to determine which one provides the most accurate TypeI error rate. Some work has already been done to this end by Schaalje, McBride, and Fellingham ( 2002). Schaalje et al. (2002) compare the one moment FaiCornelius method with the test statistic in Equation (224) (using the DDFM = SATTERTH option in S AS) with the KenwardRoger method (using the DDFM = KR option in PAGE 38 38 SAS). They performed simulations for four splitp lot designs, two of which were balanced and two of which were unbalanced. The simulation is re peated for five covariance structures to study the effect of the complexity of the covariance st ructure. Hypothesis tests were computed for whole plot effect and subplot effect using the DDFM = KR and DDFM = SATTERTH methods. Schaalje et al. (2002) calculated simulated Type I error rates and l ackoffit tests to determine if the simulated pvalues followed a uniform (0,1) distribution. Both me thods performed equally well for the simple Compound Symmet ric covariance structure, regardless of the design. For the other four covariance structur es the KenwardRoger method out performs the FaiCornelius method in the lackoffit test a nd the type I error rates; howev er, as the sample size becomes smaller or the covariance structure becomes more complex, neither method performs well. Type I error rates are severe ly inflated under both methods when complex covariance structures are coupled with small sample sizes. Note that when random effects are consider ed, it is more appropriate to focus the investigation on prediction limits. We will shift the focus to prediction intervals and expand the comparison to include modified FaiCornelius methods with adjusted MSEP estimates in Chapter 4. Although we confine the comparison to a single dimension linear combination in the balanced oneway random effects model, this comparison will paint a more complete picture of the methods considered and show the need for further investigation in to the proposed methods. 2.4 Negative Variance Component Estimates Very little research has focused on the imp act of negative variance parameter estimates on BLUP procedures, the MSEP estimation met hods, and hypothesis testing. Stroup and Littell (2002) begin to explore the impact of the met hod of variance component estimation and negative variance parameter estimates on tests of fixed eff ects. Their exploration reveals that REML may not always be the best choice in terms of power and true Type I error va lues for tests of fixed PAGE 39 39 effects when negative variance parameter estimat es are prevalent. The impact of negative variance parameter estimates on the EBLUP an d MSEP estimation for linear combinations involving random effects has not been thoroughly investigated. In Chapter 5, we delve into the complexities of negative variance component estimates and the impact on EBLUP and the associated MSEP estimates. Smith and Murray (1984) develop a method for handling negative covariance estimates. They propose that the vari ance parameter in question be considered as a covariance parameter, thereby a llowing negative estimates. The impact of this strategy on EBLUP methods is considered in Chapter 5, l eading to an alternative model and analysis procedure. PAGE 40 40Table 21. Summary of de velopments for MSEP estimation and hypothesis testing Paper Fixed or Random Effects Correction Term Bias Term MSEP Estimate Kackar and Harville 1984 Fixed and random tr A B None 1Mtr A B Giesbrecht and Burns 1985 Fixed None None 1 1 XVX Jeske and Harville 1988 Fixed and random tr A B None 1 Mtr A B Prasad and Rao 1990 Fixed and Random 1 Etr tr oq A B A B For q = # small areas, in the FeyHerriot model 1 1 1 = () EM M tr oq A B For q = # small areas, in the FeyHerriot model 1 2 Mtr A B Harville and Jeske 1992 Fixed and random tr A B 11 11 2 EMMtr Mtr B A B 11 2 Mtrtr B A B or 1 2 Mtr A B Fai and Cornelius 1996 Fixed tr A B defined for multiple dimension linear combination, denoted FCC ; shown to be nnd None 1M or FC FC1M M C Kenward and Roger 1997 Fixed tr A B defined for multiple dimension linear combination; KRC 1 2 Etr tr 11 1M M B M A B Defined for multiple dimension linear combination, KRBIAS KR KRKR1M M C BIAS PAGE 41 41Table 21 cont. Paper Hypothesis Test Stat Distribution of Test Stat Kackar and Harville 1984 N/A N/A N/A Giesbrecht and Burns 1985 0: 0 H GBt 1 1 XVX GBdft where 2 2 varGBdf 1 1 1 1 XVX XVX Evaluated with Satte rthwaitetype method Jeske and Harville 1988 0: 0 H u 1 JHt Mtr A B J Hdft where 2 1 1 2 varJHMtr df Mtr A B A B Evaluated with Satte rthwaitetype method Prasad and Rao 1990 N/A N/A N/A Harville and Jeske 1992 N/A N/A N/A Fai and Cornelius 1996 0: H L 0 1 111 FCF l LLM LL 1 21 FC FCF l LLM LL One and Two moment matching; Spectral decomposition, Taylor series and Satterthwaite techniques used to determine moments. Note that this yields two choices of distributions for each test statistic. Kenward and Roger 1997 0: H L 0 11 KR KRF l LLM LL Two moment matching; Taylor seri es and conditional expectations used to determine moments. PAGE 42 42 CHAPTER 3 ESTIMATES OF MSEP FOR MULTIPLE DI MENSION LINEAR COMBINATIONS 3.1 Reconciling KenwardRoger (1997) and FaiCornelius (1996) Definitions of the Correction Term Approximation 3.1.1 Linear Combinatio ns of Fixed Effects Kenward and Roger (1997) and Fai and Cornelius (1996) extend the methods for estimating the MSEP summarized in Section 2.1 to multiple dimension linear combinations. Both methods require expanding the definition of the KackarH arville approximation of the correction term, and in the KenwardRoger case, the bias correction term, into a matrix form. Recall that the KackarHarville approximation of the correction term for single dimension linear combinations is Ctr A B where var() = dE A y; dy; dy; ()() dy; y; and B is either an approximation to or the exact MSE matrix, E Extending this definition to multiple dimension linear combinations is not necessarily unique. In both papers, only linear combinations of the vector of fixed parameters are considered. In Section 3.1.2 we extend their method to in clude linear combinations involving both fixed and random effects. Noting that 11cov,rr ij ji ijtr b A B where bij is the ijth element of B Kenward and Roger expand this definition to the multiple dimension case by 11cov,rr KR ij ji ijb L L C C where L is an ldimension matrix of constants representing the linear combinations of interest. We will denote this approximation as KRC PAGE 43 43 Note that, 11 12 1 21 22 21cov,cov, cov, cov,cov, cov, cov,l ij ij ij ijij ij ij L L L L L L L L L L L L L L 12cov,cov, cov,lll l ij ij ij L L L L L L where Li is the ith row of L. Thus we can write the kmth element of KRC as 11 12 11 12cov,cov, cov,.KR kmkmkm km qq qqbbb L L L L L L C Fai and Cornelius (1996) define the multiple dimension approximation to the correction term differently and restrict their definition to a compound symmetric covariance structure with two covariance parameters. We now expand that definition to include a ny covariance structure with r parameters and show its equivalence to KRC They also establish the nonnegative definiteness of their correction term definition, wh ich also applies in this more general setting. Let FCC denote the Fai and Cornelius multiple dimension approximation to the correction term. Then, 11 1 1 l FC ll ltr tr tr tr A B A B C A B A B (31) where PAGE 44 44 11 1 1cov,cov, cov, cov,cov,km km r km km km km rr r L L L L L L A L L L L and B is as before. Then the kmth element of FCC is 11 12 11 12cov, cov, cov,.FC km km km km km qq qqtr b b b L L L L C A B L L Thus, clearly KRC = FCC Fai and Cornelius (1996) have shown that this definition of the correction term approximation is nonnegative definite. This b ecomes an important factor when the MSEP estimate is to be used in a test statistic or conf idence interval. We define the most general form of this approximation and prove that it is non negative definite in the Section 3.1.2. 3.1.2 Linear Combinations of Fixed and Random Effects We can now generalize the definition of the multipledimension correction term approximation to include random effects and esta blish that it is nonnegative definite. Let  KLM denote an ldimension matrix of constants. We now consider the linear combination, KL Mu u. To expand the definition to random effects, we need to redefine kmA as PAGE 45 45 11 1 1()() cov ()()()() cov cov ()()()() cov cov ,kkmm km kkmmkkmm r kkmm kkmm rr L MuL Mu A L MuL Mu L MuL Mu L MuL Mu L MuL Mu r where kL is the kth row of the fixed effects portion of the matrix of constants, L, and kM is the kth row of the random effects portion of the matrix of constants, M. We now follow a similar procedure as used in Fai and Cornelius (1996) to establish that this generalized definition of KRC is nonnegative definite and thus the MSEP estimator utilizing this correction term es timator is nonsingular. First, how ever, we need to broaden the definition of KRC to incorporate linear combinati ons of fixed and random effects. First, using the result that 1 11xx AA AA, we have 1 1 1 111 1 1 where k k kk iiii kk ii k i k i ki 11 1 1 11 11 111 1 1 1 111 1LXVXXVy XVX L V LXVXX y LX V y VV LXVXXyLXVXXXXVXXVy V LXVXXIPy V LXVXXVVIPy CVIPy 11 and kik i 11 11V CLXVXXVPXXVXXV and PAGE 46 46 1 k kkk ii i i kk ii k i ki ki ki kikiki 1 11 111 1111 111 1Mu GV MZVIPyMGZIPyMGZVX GV MZVIPyMGZVVIPy V MGZVXXVXXVVIPy DVIPyEVIPyFVIPy DEFVIPy where ,kik i kik i 1G DMZ V EMGZV and 1.kik i 111V FMGZVXXVXXV Thus, ()kk kikikiki i 1L Mu CDEFVIP y We can now assess the ijth element of kmA as ()()() cov cov kkmm kmij ij kikikiki mjmjmjmj kikikiki mjmjmjmj kikikiki mjmjmjmj 11 11 1L MuL Mu A CDEFVIP y CDEFVIP y CDEFVIPVIPVCDEF CDEFVIPCDEF since PAGE 47 47 11. 111 1 11IPVIPVVXXVXXVVXXVXXV IP We now have 1111 1111 kkkk km krkrkrkr mmmm mrmrmrmr 1CDEF A CDEF VIPCDEFCDEF (32) We can now generalize the definition of the K ackarHarville correction term approximation as follows. Definition 3.1. The multipledimension KackarHarville approximation for a linear combination including fixed and random effects is ,KR kmtr C A B (33) where kmA is defined in Equation (32). We now prove that KRC is nonnegative definite. Theorem 3.1. The multipledimension KackarHarville approximation for the correction term, as defined in Equation (33), is nonnegative definite. Thus, the MSEP approximation utilizing this correction, KR1M C is nonsingular. Proof. We begin by examining the kmth term of KRC We can write //11 2 1 2VIPQQ and //1212B RR where Q1/2 and R1/2 both exist since 1VIP is nonnegative definite and B is positive definite (e.g., Schott 1997, p.16). PAGE 48 48 Consider 1111 1111 1111 1/21/2 1111 111 km kkkk mmmm mrmrmrmr krkrkrkr kkkk mmmm mmm krkrkrkrKR kmtr tr tr C A B CDEF 1 VIPCDEFCDEFB CDEF CDEF QQCDEFCDE CDEF 1/21/2 1 1111 1/2 1/21/2 1/2 1111 1/2 1/2 1111 m kkkk mmmmmrmrmrmr krkrkrkr mmmmmrmrmrmrtr tr FRR CDEF QCDEFCDEFRR Q CDEF QCDEFCDEFR 1/2 1/2 1111 1/2 1/2 1111 1/2 1/2 1111 *kkkk krkrkrkr mmmm mrmrmrmr kkkk krkrkrkr mkkvec vec QCDEFCDEFR QCDEFCDEFR QCDEFCDEFR m where 1/2 1/2 1111 iiii iririririvec QCDEFCDEFR Thus we can write KRC as 1where ,,.KR l C (34) PAGE 49 49 Writing KRC in this form proves that it is nonnega tive definite (e.g., Schott 1997, p.16). Since E 1M L +MuL +MuL +MuL +Mu is positive definite, we have now established that the general approximation of MSEP, KR1M C (35) is positive definite, and hence nonsingular, and t hus is a valid measure of prediction error. 3.2 Multiple Dimension Bias Correction Recall that it is often nece ssary to correct for the bias introduced into the estimate of MSEP by using 1M to estimate 1M Kenward and Roger (1997) define a bias correction term for multiple dimension linear combinations akin to the single dimension bias correction offered by Harville and Jeske (1992) found in Equation (211). We now rewrite this term to resem ble the correction term in Equation (31), in order to maintain consistency in our notation and enable expansion to include linear com b inations containing random effects. The definiteness properties of either the bias correction term or the estimate of MSEP containing the bias correction term has not previously been addr essed. We will address that issue in Section 3.2.3. We first consider linear combinations containing only fixed effects. 3.2.1 Bias Correction Estimator for Multiple Dimen sion Linear Combinations of Fixed Effects We let bij again be the ijth element of B Kenward and Roger (1997) state the bias correction term as 2 11cov 1 2rr KR ij ji ijb L ,L BIAS (36) where PAGE 50 50 222 1121 2 222 12var cov, cov, cov cov,cov, varl ijijij ij lll ijijij L L L L L L ,L L L L L L Thus, the kmth element of KRBIAS is 22 11 12 2 11 2 22 2cov cov 1 2 cov cov .km km KR km km km ij rr ij rbb bb L ,L L ,L BIAS L ,L L ,L (37) We now rewrite KRBIAS as 1 11 2l KR ll ltr tr tr tr 11 B B BIAS B B (38) where 22 2 2 11 22 2 1cov, cov, cov, cov, cov,km km r km km km km rr L L L L L L L L L L To confirm the equality of Equations (38) and (36) note that the kmth element of Equation (38) is also PAGE 51 51 2 11 2 1 22 12 12cov, 1 2 cov,cov, km km km ij ijKR km kmb bbtrB L L L L L L BIAS 2 2cov,km rr rb L L (39) which is identical to Equation (37). It has been shown by Ke nward and Roger (1997) that Equation (36) reduces to the multiple dimension correction term ( KRC ) when the covariance matrix, V is linear in which gives KR KR 1M =M +2C as an estimator of the MSEP. In the next section, we use Equations (38) and (39) to define the estimator for bias correction for a m ultipledimension linear combination involvi ng fixed and random effects and confirm that the resulting bias correction term continues to reduce to the correction te rm in linear covariance cases. 3.2.2 Bias Correction Estimator for Multiple Dimen sion Linear Combinations of Fixed and Random Effects The extension to include random effects is a straightforward but tedi ous task. However, it is important to explore fully the functions invo lved to determine when the term simplifies to the correction term approximation in Definition 3.1, and also to assess the effect on the definiteness properties of the MSEP estimator. The generalized bias co rrection term has the same form as in the fixed effect case: 11 1 11 2l KR llltr tr tr tr B B BIAS B B (310) but now, PAGE 52 52 2 22 2 11 22 2 1cov, cov,cov, cov, cov,kkmm km kkmmkkmm r kkmm kkmm rr L MuL Mu L MuL Mu L MuL Mu L MuL Mu L MuL Mu (311) Now considering a single element of this matrix we see that 2 () 2 22 12cov, cov,cov,cov,cov, cov,cov, kkmm kmij ij kmkmkmkm ij kmkm ijij L MuL Mu L L MuL L MuMuMu L L MuL 22 34 cov, cov,.km km ij ij L MuMuMu (312) We now investigate each piece of Equation (312) using Littell et al. (2006) as a reference for the covarian ces involved. C onsider Part (1) of Equation (312): PAGE 53 53 22 221cov, cov ,km k m ij ij km k m ij ij km ji k 1 1 1 1 1 1 1 1 1 1 11 1 1 1 1 111 1 1 1L L LXVXXVyLXVXXVy LXVXXVXXVXLLXVXL V LXVXXVVXXVXL LXVXXV 2 m ji km ji km ij km ij k j 1 1 11 111 1 1 1 1111 1 1 1 1 11 1 1 1 1111 1 1 111VV VXXVXXVVXXVXL VV LXVXXVVVXXVXL V LXVXXVVXXVXL VV LXVXXVVVXXVXL V LXVXXVVXXVX 2 2,m i kjmikjmikimjkimj km ij kimjkjmikm ij 1 1 111 1 1 1 1 1 1 1 1 11 1 1 1 1 1 1 11V XVVXXVXL CVPCCVCCVPCCVPC V LXVXXVVXXVXL V CVIPCCVIPCLXVXXVVXXVXL (313) where ki C is as defined in Section 3.1.2. Now moving on to part (2) of Equation (312), we have 222cov km k m ij ij km km ji i k i 1 11 1 1 11 111 1 1 11 111Mu,L MGZVXXVXL GV MZVXXVXLMGZVVXXVXL V MGZVXXVXXVVXXVX m L (314) PAGE 54 54 km ij k m ij km ji km ji km ij k 1 11 1 1 1 1 1 11 1 1 11 1 1 111 1 1 111 1 1GV MZVVXXVXL GV MZVXXVXXVVXXVXL GV MZVVXXVXL VV MGZVVVXXVXL VV MGZVVVXXVXL V MGZV m ji k m ji k m ij k ji 1 1 1 1 11 1 1 1 1 1 11 1 1 1 11 1 11 1 1 1 1 1 1 11 1 11 1V VXXVXXVVXXVXL GV MZVXXVXXVVXXVXL VV MGZVVXXVXXVVXXVXL VV MGZVXXVXXVVXXVXXVVXXVXL 2 m k m ji k m ij k m ij km ij k 1 1 1 1 111 1 1 1 1 1 1 11 1 11 1 1 1 1 1 111 1 1 1 1VV MGZVXXVXXVVVXXVXL VV MGZVXXVXXVVXXVXXVVXXVXL VV MGZVXXVXXVVVXXVXL G MZVXXVXL M 2 2 m ij k m ij 1 1 1 1 1 1 1 1 1 1 1V GZVVXXVXL V MGZVXXVXXVVXXVXL PAGE 55 55 22 kimjkimjkjmikjmi kimjkimjkjmikjmi kimjkimjkjmikjmi km k ij ij 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1DVC+DVPCDVC+DVPC EVC+EVPCEVC+EVPC FVC+FVPCFVC+FVPC GV MZVXXVXLMGZVVXXVXL 2 m k m ij 1 1 1 1 1 1 1V MGZVXXVXXVVXXVXL (315) 22 2.ki mjkj mi ki mjkj mi ki mjkj mi km km ij ij k m ij 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1DVIPCDVIPC EVIPCEVIPC FVIPCFVIPC GV MZVXXVXLMGZVVXXVXL V MGZVXXVXXVVXXVXL (316) where ki D ki E and ki F are as defined in Section 3.1.2. Du e to the symmetry of parts (2) and (3) of Equation (312), we combine (2) and (3) as follows: 22 11(2)(3) ,kikiki mj kjkjkj mi ki kjkjkj kj kikiki ij ijfg 1 1 1 1DE+FVIPC DE+FVIPC CVIPD+E+F CVIPD+E+F GV (317) where 2 1 ijf G and 2 1 ijg V are linear functions of 2 ij G and 2 ij V, respectively. Part (4) of Equation (312) tediously completes ()kmij as follows: PAGE 56 56 22 22 22 24cov,km k m ij ij kmk m ij ij km km ij ij km ij ki mjkj mi 1 1 1 1 1 1 1MuMuMGGZVIPZGM GG MMMZVIPZGM GP MGZVIPZMMGZVZGM V MGZIPZGM DVIPDDVIPD .ki mjkj mi ki mjkj mi ki mjkj mi ki mjkj mi kimjkjmi kimjkjmi 1 1 1 1 1 1 1 1 1 1 1 1EVIPDEVIPD FVIPDFVIPD DVIPEDVIPE DVIPFDVIPF +EVPE+EVPE +EVPF+EVPF (318) Now note that ii i i 1 11 1 1 1 1 11 1 1 1 11P XXVXXV VV XXVXXVVXXVXXVXXVXXVV which leads to PAGE 57 57 2 2ij j i ji ij ij 1 1 1 1 11 1 11 1 1 1 1 1 111 1 1 1 1 1 1 1 1 1 1 1 1 111 1 1PVV XXVXXVVXXVXXVVXXVXXV VV XXVXXVVVXXVXXV V XXVXXVVXXVXXV VV XXVXXVVVXXVXXV ij ij ji ji i 1 1 1 1 11 1 11 1 1 1 1 1 11 1 11 1 1 1 11 1 11 1 1 1 111 1 1VV XXVXXVVXXVXXVVXXVXXV VV XXVXXVVXXVXXVV VV XXVXXVVXXVXXVV VV V XXVXXVVVXXVXXVV 2.j ij 11 1 1 1 1V V V XXVXXVV Now we have 2 2 2.k mkj miki mjkj mi ij ki mjk m ij km ij 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1P MGZVZGMFVIPFFVIPFFVIPE V FVIPEMXXVXXVVXXVXXVM V MXXVXXVVM To complete Equation (318) we also must investigate 21 ij V: 2 2 ijji ij ji ij 1 1111111111VVVVVVV VVVVVVVVVV. PAGE 58 58 This gives us 2 2 2.km k m ij ij km k m ji ij kimjkjmikimjkjmik m ij 1 111 111111 1 1 1 1 1 1VV V MGZIPZGMMGZVVVIPZGM VV V MGZVVVVIPZGMMGZVVIPZGM V EVE+EVE+EVF+EVFMGZVVIPZGM Utilizing these expressions for 21 ij V and 2 ij Pin Equation (318) results in 22 22 2(4)kmk m ij ij km km ij ij km ij ki mjkj miki mjkj mi ki mjkj mi 1 11 1 1 1 1 1 1 1GG MMMZVIPZGM GP MGZVIPZMMGZVZGM V MGZIPZGM DVIPDDVIPDEVIPDEVIPD FVIPDFVIPD 2 kimjkjmi ki mjkj miki mjkj mi ki mjkj miki mjkj mi ki mjkj mik m ij k 1 1 1 1 1 1 1 1 1 1 1 1 1 1DVIPEDVIPE DVIPFDVIPFEVIPEEVIPE EVIPFEVIPFFVIPEFVIPE V FVIPF+FVIPFMGZVVIPZGM M 2 2 m ij km ij 1 1 1 1 1 1 1 1 1 1 1V GZXXVXXVVXXVXXVZGM V MGZXXVXXVVZGM(319) which reduces to 22 22(4) .kikiki mjmjmj kjkjkj kikiki ij ijfg 1 1DE+FVIPD+E+F D+E+FVIPD+E+F GV (320) PAGE 59 59 Piecing together (1) (4) in Equation (312) results in: 2 () 22cov, .kkmm kmij ij kikikiki mjmjmjmj kjkjkjkj mimimimi ijijfg 1 1L MuL Mu CDE+FVIPCD+E+F CD+E+FVIPCD+E+F GV (321) When G and V are linear in the functions of 2 ijf Gand 2 ijg Vwill be zero. In that case 1111 11112 .kkkk km krkrkrkr mmmm mrmrmrmr 1CDEF CDEF VIPCDEFCDEF (322) Comparing Equation (322) to Equation (32) gives the desired result that in the most general case invo lving multiple dimension linear combinations of fixed and random effects KR BIAS KRC when G and V are linear in Thus, just as in the single dimension fixed effects linear combinations, when G and V are linear in KR KR 1M =M +2C is an estimator of the MSEP of the linear combinat ion of fixed and random effects that accounts for both the correction term and th e bias correction term. 3.2.3 Definiteness Properties of the Estima tor for MSEP for Mult ipleDimension Linear Combinations of Fixed and Random Effects Now that all pieces of the estimator have been defined in the multiple dimension setting an estimator for the MSEP can be defined as PAGE 60 60 KR KR KR1M M C BIAS (323) It has not been proven that th is MSEP estimator, either in single or multiple dimensions, is nonnegative definite. Just as in the single dimension case, we have shown above that when the variance matrix for the model is linear in the bias correction estimator, KRBIAS reduces to the correction term, KRC which we have shown is nonnegative definite in its most general form. Hence the simplified MSEP estimator, 2KR KR1M M C (324) is nonsingular and therefore a valid measure of prediction error. However, when the covariance matrix, V, is not linear in neither the bias correction term nor the MSEP estimator including the bias correction term has been previously show n to be nonnegative definite. In Chapter 4, by exploring the special case of th e balanced oneway random effects model, we provide a counter example to prove that the bias correction is, in fa ct, indefinite. Indeed, we show that in a single dimension case the estimate of MSEP given in Equation (323) can be smaller than the nave estim ate, 1M From Equation (321) it is clear that the indefiniteness of KRBIAS is caused by 2ijf G and 2ijg V. Note that it is not nece ssary for the bias correction term to be nonnegative definite for the overa ll estimator of the MSEP to be nonnegative definite. In fact, it is reasonable for the bias correction term to be indefinite since it is estimating the bias of 1M which is also indefinite. In Chapte r 4, we provide a single dimension case where the negativity of the bias correct ion term causes the entire estimate of 1M to be PAGE 61 61 negative, proving that KR1M BIAS the overall MSEP estimator for 1M is indefinite and a poor choice for estimating the MSEP. 3.3 Transform Invariance of MSEP Estimator Another important issue concerning these esti mators is transform invariance. Consider two parameterizations of a model, and g Kackar and Harville (1984) showed that the approximation for the correction term in the single dimension case, tr A B is transform invariant under the conditions that g and that 1 1g B J B J where B*and B are as in tr A B and J is the Jacobian of the transformation. Both of these co nditions are satisfied by using either REML or ANOVA variance component estimation and with 1 B I the inverse of the observed information matrix. This proof applies to the multiple dimension case, as we have shown that the multiple dimension correction term approximation is simply a matrix composed of the single dimension approximations as the components. Since each element of the matrix is transformation invariant, the entirety is transform invariant as well. It is also easy to show that the nave estimator of the MSEP, 1M is also transform invariant under the condition that g The invariance of the nave estimator indicates that the true bias of 1M E 11M M will also be transfor m invariant (again with the condition, g ). Thus a desirable property for the estimator of the bias correction term is transformation invariance. Clearly, KRBIAS is not transform invariant. In Chapter 4, we demonstrate the lack of transform invariance fo r a single dimension linear combination in the PAGE 62 62 one way random effects model. This is an undesirable quality of th e bias correction term estimator. We will investigate its effects on the balanced oneway random effects model in Chapter 4. PAGE 63 63 CHAPTER 4 BALANCED ONEWAY RANDOM EFFECTS MODEL Many of the methods summarized in Chapter 2 were developed only for estimating and testing fixed effects (e.g., Gi esbrecht and Burns 1985; Kenward and Roger 1997); however, the methods are often extended to include random effects without documented justification (SAS 2003). In Chapter 3, the elements involved in ex tending these methods to the most general cases were derived. By looking at the simplest mode l including random effects, namely the balanced oneway randomeffects model, we can examine if the use of these methods is as straightforward as previously claimed or if additional issues ar ise. The simplicity of this model often allows closed form expressions to be obtained for the approximations and estimators of the MSEP and their expected values, in addition to the true value of the MSEP being estimated. Peixoto and Harville (1986) derived closed form expressi ons for the true MSEP for a class of models including the balanced oneway ra ndom effects model. Similar methods are used here to enable direct comparisons of various MSEP estimators. By using this simple, balanced model, we can measure the result of including ra ndom effects on the MSEP estimates, as well as identify issues that may be more difficult to handle in more complex models. Consider the model: 22, 1,,, 1,, 0,, 0,ij iij ij i a y aeikjn eNaN (41) where 22, 0, 0a. We use the parameterization 2, where 2 22an and consider the problem of predicting the mean of the ith level of the random effect. Note that the variance matrix, V, is not linear in this para meterization, nor is this the parameterization used by SAS. This parameterization is chosen to demonstrate the impact of a PAGE 64 64 nonlinear parameterization on the bias correction term and for consistency with past work on this topic. Comparisons will be made to results under the altern ative parameterization, 22,a for which V is linear. The BLUP and the nave MSEP, 1M, for ita are straightforward to derive, as demonstrat ed in Kackar and Harville (1984, p. 857): 1 11 where and n k ij i j i iiy y yyy y nk (42) 2 11 11Mk nk (43) Noting that is linear in the accuracy of the KackarHarville approximation of the correction term depe nds on four issues: 1) using the exact MSE of versus an approximation, such as 1I for B ; 2) the appropriateness of the Taylor series approximation, depending in part on how close is to the boundary of ; 3) whether is unbiased for ; 4) the proximity of cov, dy; d y ; to zero. We can evaluate these four criteria to de termine their effects on the accuracy of the KackarHarville approximation. We shall examin e each of these criterion using both REML and ANOVA methods of variance component estimation. 4.1 KackarHarville Appr oximation Based on REML Estimation Method Using REML methods to estimate the variance components results in 2 where PAGE 65 65 2 2 2 11 1 if (1)(1) if 1 min(/,1) where and (1)1eae ae ae ea kn k iji i ij i eammm kmknm mm nk mm yy nyy mm kn k (44) (e.g., Kackar and Harville 1984; Searle, Casella and McCulloch 1992). Note that these estimators remain within the parameter space, by definition. The EBLUP for ita is 1i y y (45) and the nave estimate of the MSEP of the EBLUP is 2 1 1 11 Mk nk (46) The true correction term in the REML case, as derived by Kackar and Harville (1984, p. 731), is 2 2 11 Rnk CE nk (47) where 1 1 1 2 2211 21 ,11 1121 I,1 2 1a ee eaaea a ae aa ea eaea eaB PAGE 66 66 and 1 11 0()() ,, I,,( 1) () (1)1 22x ab x e ea eaab B ab abBabttdt ab knk This results in a true MSEP, RM where 1 2 1 11RRMMC nk M nk (48) (Kackar and Harville 1984, p. 857). In this model, the exact MSE matrix of is tractable when using REML estimates for the variance parameters. Thus assuming that is sufficiently far from the boundary of and choosing RB to be the exact MSE matrix of we can assess the accuracy of Rtr A B for RC by simply assessing the proximity of cov, dy; d y ; to zero. To evaluate Rtr A B first note that 2200 0 a A (49) and 2 22 22 2 22 REE EE B (410) PAGE 67 67 where 2 221 varvarik ay y nk As shown by Kackar and Harville (1984, p. 731), 2 2 0E where 1 0 2 2 2211 21 ,1122 11 21 2 I, 121a eae eaaeaa a ae a ea eaaaB As a result, 2 2 0 220(1)Rk tr a nk A B (411) A KackarHarville approximation to the MSEP is then 2 0 1(1)k M nk (412) (Kackar and Harville 1984, p. 857). Now note that the only nonzero component of cov, dy; d y ; is 222222 cov,iiiyyEyyEyyE Kackar and Harville (1 984, p. 731) show that 2 22 11 iRnk EyyC nk PAGE 68 68 It is also easy to verify that 2 221iak Eyy n nk since 2 1 221a k akm n Thus recalling that 2 2 0E and combining results, we have 22 22 10 22 22 1011 cov, 11 .i ankk yy n nk nk nk k nk nk (413) Notice that this is simply the difference be tween the true correcti on term in the REML case,RC and the approximation, Rtr A B This is only true when the exact MSE of is used for RB Thus the condition for RRCtr A B i.e., the condition for cov,0 dy; dy; is 1011 nkk (414) Figure 41 shows Equation (413) as a function of for several values of k, holding n = 6, with the condition in Equation (414) met when the functions cr oss th e zero line. Note that throughout, without loss of generality, we hold 21 As k increases, the functions move closer to zero, indica ting that the KackarHarville approximation for RC improves as k increases. However, as k increases, the amount of information about the random effect increases, reducing the uncertainty about the cova riance parameter estimates. Hence RC tends toward zero and accounting for the correction term b ecomes less important, rendering the correction term approximation irrelevant. In the remainder of this chapter, the analysis will focus on a model with k = 6 levels of the random effect to demonstr ate the impact of the different estimators for the MSEP. PAGE 69 69 In most models, the exact MSE of is not tractable and an approximation of the MSE of must be used. A common choice is 1I the inverse of the observed information matrix, with replaced by in practice. Consider the KackarHarville approximation of RC where B is the asymptotic MSE of 1I Kackar and Harville (1984, p. 857) show that 2 22(1) (1) kn tr knn 1A I (415) where 2 2 2 2 22 2 11 21 2 111knkn kn knknk 1I (416) Figure 42 compares the two KackarHarville approximations in Equations (411) and (415) with the co rrection term,RC for k = 6 and n = 6. The success of the approximation is greatly influenced by the value of For small values of which correspond to large values of 2a relative to 2 tr 1A I is superior to Rtr A B for approximating the true correction term, RC However, as moves toward 1, which corresponds to the value of 2 a growing relatively smaller, the roles reverse and Rtr A B becomes the better approximation. In some cases, the choice of how to measure the variability of may cause severe overestimation of the correction term, in turn causing a greatly inflated value for the MSEP estimate. SAS PROC MIXED uses tr 1A I which overestimates the true PAGE 70 70 correction term by over 100% for half of the parameter space. Using an inflated estimate for the MSEP could lead to overly conservative interpretation. Because the correction term approximations in Equations (411) and (415) make use of the unknown variance param eters, estimates of the va riance parameters must be substituted into these approximations. Note that there are now two levels of error being introduced into the correction term estimators: first, the true corr ection term is approximated by the Taylor series methods to yield the KackarHarville approximati on; second, the variance parameters involved in the approximations are replaced by their estimators. In the case of tr 1A I there is a third contribution to the error in that 1I is an approximation of the MSE of Substituting the REML estimates of the variance components into Equations (411) and (415) gives two possible estim ators for the correction term: 2 0 (1) Rk tr nk A B (417) and 2 2 2(1) (1) kn tr knn 1A I (418) It is suggested by Prasad and Rao (1990) that the bias introduced by substituting variance parameters in these approximations is 1ok. To assess this assertion, we investigate the accuracy of Equations (417) and (418) by considering the expected values of both estim ators. The expected value of Equation (417) is not available in closed for m, so a simple Monte Carlo study was performed to evaluate the accuracy of the estimator for the oneway random effects model in Equation (41) with k = 6 and n = 6. Using the RANNOR function in SAS, 10,000 PAGE 71 71 independent sets of eij and ai were generated from 2(0,1)N and 20,aN for several values of 2a respectively. The values of 2a correspond to 0.05, 0.1, 0.15, 1.0. Setting 0 10,000 sets of :1,6,16ijyij were created from the model ijiij y ae. The Monte Carlo value of 2 0 (1) Rk Etr E nk A B and the Monte Carlo standard error were obtained from the 10,000 replicates. Selected results of the study are contained in Table 41 (standard errors contained in parentheses) along with the values of RC (Equation (47)) and Rtr A B (Equation (411)). Table 41. Accuracy of KackarHarville estim ator (417) for RC 0.2 0.4 0.6 0.8 1.0 RC .01537 .01890 .01758 .01660 .01850 Rtr A B .04447 .03384 .01950 .01097 .00954 **REtr A B .03567 (.00013) .02652 (.00013) .01951 (.00011) .01513 (.00009) .00954 (.00002) **Monte Carlo value. Standard error of simulation in parentheses. Figure 43 shows the results of the Monte Ca rlo study graphically. The accuracy of the estimator again largely depends on the value of For small values of which correspond to values of 2a that are large relative to2 the estimator for the correction term is not accurate with expected values more than twice th e function it is estimating. However, as approaches 1, the estimator for the correcti on term significantly underestimat es the true correction term. PAGE 72 72 The expected value of Equation (418) can be expressed in closed form if we assume norm al distributions. First define 222(1)(1) eae eae a ae aknmkm u uuw u u s nu u It is easy to show that ue and ua are independent chisqua re random variables with k(n1) and (k1) degrees of freedom, respectivel y (e.g., Searle 1971, p. 410). Also, w and s are independently distributed where w has a chisquare distribution with (kn1) degrees of freedom, and s has a Beta(e, a) distribution (e.g., Johnson and Kotz 1970, Sec. 24.2). Now note that if ma > me then 1e am m so that min,1ee aamm mm but when ma < me, 1e am m so that min,11e am m Defining 1 if 0 if ae ae aemm mm mm gives 2 2 2 2 1 221 (1) 21 1(1) ,1 (1) 1ae e ae ae akn Etr E knn kn kmknm m EmmE mm knnm kn 1A I (419) Noting that if ma > me then s < so that 1 if 1 if ,, 0 if 0 if ae ae e aemm s sm m s mm and that 22 2(1)(1)ea aem ws mkns we see that PAGE 73 73 22 2 2 2 1 1 2 2 0 21, (1)1 (1)1 1 (1) (1)1(,) (2,1) (1) ( (1) (,)a eea ae ae a e a ee a ae a ee am ws EmmEs mk ns s EwEs kn s ss s kn ds kn sB B kn I kn B 2,1).ea (420) Now note that 2 so (1)eesw uswm kn and 221 (1)a answ m k which leads to 22 21(1) 1 21 ,1 11 (1) 1, 1 11, 11ae a ae ae e ae akmknm km Em m Em m kn kn knm Em m kn n Ewss Ews kn kn 11 11 11 22 2 22 2 22 21, 11 1 (,)(,) (,1)(1,) 1(,1)1(1,) (,)(,) 1(,1)1(1,).aa eea eaea eaea aeaea eaea ae aeaea eaeas ssss nsdssds BB BB nII BB nII (421) Substituting Equations (420) and (421) for (1) and (2) in Equation (419), respectively, we have 2 2 2 2 22 121 (1) 21 (1)(2,1)(2,1) (1) (1)(,) 1(1,) 1(,1).aeaea eea ea ea ea ea eakn Etr E knn kn knB I knn knB II 1A I (422) PAGE 74 74 Figure 44 shows the accuracy of th e KackarHarville estimator (Equation (422)) for the KackarHarville approxim ation (Equation (415)) and the correction term (Equation (47)) holding k = 6 and n = 6. Just as for the approxim ation, the accuracy of Etr 1A I depends largely on the value of While tr 1A I is a fairly accurate estimator for tr 1A I over the entire parameter space, it may severely overestimate RC for larger values of We wish to assess the asserti on of Prasad and Rao (1990) that 1 E tr tr ok 1 1A I A I for this model. Noting that lim1k yields the result that lim1kI so that 2 2 2122 1 21 (1)(2,1)(2,1) 1 (1) (1)(,) 1(1,) 1(,1)aeaea eea ea ea ea ea eaEtr tr kn knB I knn knB II ok 1 1A I A I confirming, for this example, the assertion of Prasad and Rao (1990). These results also show the importance of studying the effect of substituting variance parameter estimates for the unknown values in th e approximations for th e correction term, the impact of which has been typically overlooked. In the case of Rtr A B substituting variance parameter estimates improves the accuracy for RC (in the long run). However, in the case of tr 1A I the substitution creates more inaccuracy for RC for most values of PAGE 75 75 Figure 45 compares the accuracy of Equations (417) and (418) for the correction term. The plot shows that Rtr A B is more accurate (in the longrun) than tr 1A I for > 0.30; however, for < 0.30, the reverse is true. This is the same phenomenon that we noticed with the approximations in Figure 42. Note that SAS PROC MIXED by default uses 1I with REML estimates for in the calculation of the MSEP estimator. 4.2 KackarHarville Approximation Ba sed on ANOVA Estimation Methods The best cho ice of method for estimating may depend on the model of interest, the level of balance in the design, and the goals of the an alysis. Stroup and Littell (2002) demonstrate the effect of different estimation methods on the pow er and control of type I error for hypothesis tests on the fixed effects in analyzing an unba lanced, multilocation experiment. Their study indicates that REML and ML ar e not always the best choices when negative estimates of variance components are likely, and may lead to inflated Type I error rates and low power. The impact of the choice of variance component estimation method and negative variance component estimates on the KackarHarville approximation has not been investig ated thoroughly. In Chapter 5 we elaborate on the impact of nega tive variance component estimates. To begin, however, we develop the correction term, several approximations, and the expected values of the estimators under ANOVA estimation for the balanced oneway random effects model. The ANOVA estimators of 2, are 2, where PAGE 76 76 2 11 2 2 22 2 2 1, (1) = 1kn iji ij e e aa k i aei aayy m kn m nm nyy mm m nk (423) These estimates are obtained under the METHOD = TYPE 3 option in SAS PROC MIXED. The true correction term, CA( ), is derived under normality assumptions as follows: 22 22 1 1 22 22 22 22 2 22 22 2 2 22 2() (1) (1) (1)1 (1) (1k Ai i i aa a a a aa aa eCEyyknEnyy nkmn EEu knnkn nn EwsEws knkn ns Ews kn s 22 22 2 2 21( 1 ) 1 )( 1 ) (1) (1)1, (1)aa e a ens EwEs kn s s kn Es kn s by independence of w and s and noting that 2 1knw Recalling that ,easBeta we find that 2 22 2 2 2 2 2(1)1 2(1) (1) (1) 2,1 2 2,1 ,aa a eee aea aa eeaeaea aea a eeaeasss Es E s ss B B B B PAGE 77 77 Thus the correction term is 2 2 2 2 2 22,1 (1) (1) 1 211 13aea a A eeaea a eaB kn C knB kn kn knk knnk (424) This gives the true MSEP for 1 2 1 2211 13AAMMC knk M knnk (425) To evaluate the KackarHarville approxima tion for the correction term, we need to determine how to evaluate B We consider thre e alternatives for B : the exact MSE matrix E the variancecovariance matrix of and 1I Recall from Equation (49) that due to the form of A in this model we are only concerned with assessing the variability of in order to evaluate the KackarH arville approximation to the correction term. We begin with the exact MSE of 2E Noting that (1),1 e knk am F m it is easily shown that 1 3 k E k 22 22(1)(3) (1)(3)(5) kkn V knkk and 2 2 2112 135 kkn E knkk After algebraic simplification this leads to PAGE 78 78 2 22 22 351 2. 135EEE kknn knkk (426) Letting AE B we have the following approximation forAC: 2 221351 135Akkknn tr knnkk A B (427) Substituting the ANOVA estimates in Equation (423) for yields this estimator for AC: 2 221351 135Akkknn tr knnkk A B (428) To assess the accuracy of Equation (428) for estimating Equation (427) and AC, we evaluate the expected value. Recalling the relationship 22 2(1)(1)ea aem ws mkns where w and s are independently distributed with2 1knw and ,easBeta we have 22 2 2 2 2(1)(1) 2,1 1 (1), 112 13ea ae ea a ee am s EEEwE mkn s B kn kn B kkn knk (429) Thus, 2 2 22 32135112 135Akkknnkn Etr knnkk A B (430) PAGE 79 79 Again, we confirm the asymptotic assert ion by Prasad and Rao (1990), noting that 2 22 22 2 3 2 2 1213511221351 135 135 21351112 1 13513 .AAEtr tr kkknnknkkknn knnkk knnkk kkknnkkn knnkkknk ok A B A B The second alternative for B is the exact variancecovariance matrix of *AB Because is not unbiased for this is not the same as the MSE matrix for As noted above, 22 22(1)(3) (1)(3)(5) kkn V knkk which leads to the following approximation for AC: 3 *2 2 2213 135Akkn tr knnkk A B (431) Again substituting the ANOVA estimates for in Equation (431) yields an estimator for AC: 3 *2 2 2213 135Akkn tr knnkk A B (432) Using Equation (429) we have 4 *2 23 321312 135Akknkn Etr knnkk A B (433) Again we find that PAGE 80 80 ** 4 3 22 23 2 32 3 2 2 2 121312 213 135135 112 213 1 13 135 .AAEtr tr kknkn kkn knnkk knnkk kkn kkn knk knnkk ok A B A B The third alternative we consider for B is to use the inverse of the observed information matrix, 1I as an approximation. Note that 1I is the same as the asymptotic variance of the REML estimators of in Equation (416). Thus the KackarHarville approxim ation with ANOVA variance component estimates using 1I is the same as in the REML case in Equation (415), i.e., 2 22(1) (1)kn tr knn 1A I (434) and the estimator formed by substituting ANOVA estimates for is 2 22(1) (1)kn tr knn 1A I (435) Using Equation (429) we have 2 2 2 322(1) (1) 2(1)112 (1)3 kn Etr E knn knkkn knnk 1A I (436) which again yields PAGE 81 81 22 322 2 2 12(1)112 2(1) (1)3 (1) 112 2(1) 1 (1)13 Etr tr knkkn kn knnk knn kkn kn knnknk ok 1 1A I A I Figure 46 compares the three Kacka rHarville approximations in Equations (427), (431) and (434) to AC for k = 6 and n = 6. Clearly tr 1A I more closely approximates AC than the other two approxi mations, for all values of Thus, although we would expect 1I to be less accurate than AB or *AB in evaluating the MSE of especially for small values of k, the approximation of AC utilizing 1I is more accurate than the either of the other choices across the entire parameter space. In fact, the KackarHarville approximation utilizing the exact MSE of is the least accurate of all three approximations. Note that this is in contrast to the results from REML variance component estimates in Figure 42, where the best approximation depended on the value of Figure 47 compares AEtr A B (Equation (430)) to Atr A B and the correction term, AC The bias for the correction te rm introduced by replacing the unknown variance components with ANOVA es timates in the correction term approximation is significant and increases as moves toward 1. Figure 48 demonstr ates a similar result for the KackarHarville approximation and estimat or utilizing the variance of AEtr A B (Equation (433)) is again inflated over the approx im ation and they both overestimate the true PAGE 82 82 correction term increasingly in Figure 49 shows a much diffe rent result than the previous two figures since the accuracy of the estimator utilizing 1I (Equation (436)) is improved over that of the approxim ation (Equation (434)). While tr 1A I underestimates AC by up to 50%, Etr 1A I is much more accurate and consistently overestimates AC by only 6.67% across the entire parameter space. Figure 410 compares all three estimators (Equations (430), (433), and (436)) relative to AC Clearly the KackarHarville estimator utilizing 1I is far more accurate than either of the other two choices. This is counter intuitive: the KackarHarville estimator using an approximation of the MSE of outperforms the other estimators using exact measures of the variation of Note again that this is a different result than seen with REML variance component estimates where the results were dependent on the value of There, we saw that the KackarHarville estimato r utilizing the exact MSE of RB often performed better than the one using an approxi mation of the MSE of 1I (see Figure 45). The choice of which KackarHarville estimator for th e correction term is best, theref ore, depends, in part, on what method is used for variance component estimation. The final step in this assessment is to co mpare the ANOVA results to the REML results. Figure 411 compares the accuracy of tr 1A I (Equation (436)) for AC with the accuracy of the two REML estimators, Rtr A B (Monte Carlo expected value) and tr 1A I (Equation (419)) for RC with k = 6 and n = 6. The ANOVA estimator PAGE 83 83 utilizing the information matrix, tr 1A I appears to perform better than either REML based estimator, although the a dded variation to the MSEP of is larger for the EBLUP with ANOVA variance components, i.e., AC > RC We can assess the accu racy relative to the appropriate correction term to better compare th e performance of the estimators. The relative bias of the two REML based estimators of the correction term and the ANOVA based estimator with 1I is tabulated in Table 42 and graphically depicted in Figu re 412. Relative bias was calculated as EtrC C AB The ANOVA estimator of the correction term has consistently lower relative bias than either of the REML estimators. Thus, although the variation added to the MSEP by using ANOVA variance component estimates is la rger than when using REML variance component estimates, a more consistently accurate es timate of the added variation is available in the ANOVA case. While the accuracy of both REML estim ators depends largely on the value of the ANOVA estimator is significantly more accurate across all values of This special case has demonstrated the im portance of the choice of variance component estimation method, the choice of measure of variation of and the added variability of substituting parameter estimates for unknown va riance components in the correction term estimates. While ANOVA methods add more variability to the EBLUP than REML estimation, the estimator for the MSEP of the EBLUP is more accurate with ANOVA methods than with REML methods. The best choice of how to measure the variability of was not consistent across variance component estimation methods. For ANOVA methods, a better choice for B PAGE 84 84 was 1I while for REML estimation the bette r choice depended on the value of We also discovered that the effect of substituting parame ter estimates depends, in part, on the choice for B and the variance component estimation method. The impact of this substitution is often considerable and could cause the MSEP es timate to be significantly inflated. Table 42. Relative bias (%) of estimators of correction term 0.2 0.4 0.6 0.8 1.0 REML** REtr A B 132.0 40.3 11.0 8.8 48.5 REML Etr 1A I 33.7 85.0 152.3 202.8 220.8 ANOVA Etr 1A I 6.7 6.7 6.7 6.7 6.7 **Monte Carlo values used to compute relative bias. 4.3 Bias Correction Term for the Ba la nced OneWay Random Effects Model 4.3.1 Bias Correction Term Approximation under Parameteriz ation We now address the bias co rrection term approximation for the oneway random effects model. We will consider the case where 1B I so that the calculations for the approximation will be the same for the REML a nd ANOVA, and so that closedform expressions are attainable. The difference comes in the ex pected values of the estimators of the bias correction term due to the truncation of the REML estimates. We will show that in either case the bias correction estimator ca n be negative, thus proving by counterexample that the bias PAGE 85 85 correction estimator is indefinite We begin however, with demons trating the lack of transform invariance. Recall that the parameteriza tion we are working with is 2, If we were considering the parameterization 22,a wherein the variance matrix, V, is linear in the parameters, the bias correction term would be identical to the correction term approximation, which is transform invariant, i.e., 1 2tr tr tr B A B A B and nonnegativity would not be an issue since tr A B is nonnegative definite in its most general definition as proven in Chapte r 3. This is not the case with as the parameterization, as will be demonstrated in this chapter, since 1 2tr tr B A B We now give a simple demonstration that the bias correction term approximation is not transform invariant, as discusse d in Section 3.3, through the con tinuing example of the balanced oneway random effects model. The true bias of the nave MSEP, E 11M M is invariant to transformation as long as This follows from the definition of 2Et 1M Thus the lack of transform invariance of the bias estimate is a concern. To calculate the bias correction term approximation under the parameterization, we first compute the second deri vatives of the nave MSEP, 1M with respect to the variance parameters: 22 1 2 2 1 2 212 0,M kk nk M PAGE 86 86 and 22 2 2 1 2 22 212 221 11 kk kk M nk nk k nk 2 2 1 2 32 ,M nk and 22 1 2211Mk nk So, 2 2 2 2 2311 0 11 2k nk k nknk (437) Utilizing 1I from Equation (416), we have 22 2 2 22 2 2 22211 1 211 41 111k f nkn k kn f nknnknk 1 I (438) where 2,f is a function of 2, that is inconsequential in this calculation. This gives 2 2 2211 1 4. 111k kn tr nknnknk 1 I (439) Thus the bias correction term for the nave estimator of the MSEP under the 2, parameterization is PAGE 87 87 2 2 2211 11 2 21 1 1KRk kn tr nknnknk 1BIAS I (440) which is obviously different from the bias correction term approximation used under the parameterization, which would be identical to Equation (415). Clearly the lack of transform invariance is attributable to the higher order derivatives in We will examine the impact of the parameterization by comparing each bias co rrection estimate to the true bias. 4.3.2 Expected Value of KRBIAS with REML Covariance Parameter Estimation We begin by calculating the expected value of the bias correction term estimator once the REML estimates are substituted for the unknown variance components. Recall from Equations (419), (420), and (421) that 2 2 221(1)(2,1)(2,1) (1)(,) 1(1,) 1(,1).aeaea eea ea ea ea ea eaknB I E knB II (441) We also need 2 1 21(1) ,1 1ae aae aekmknm EEmmmE mm kn (442) where 22 1 1 21 0 2111, 1 1 1 1 1(,) (,1)a ea ea ean Ewss k ss kn sd s kB I and from Equation (421), PAGE 88 88 22 221 ( 1 )1 ( 1 )ae ae a e a ea eanI I We can utilize Equations (441) and (442) to compute th e expected valu e of KRBIAS as follows: 2 2 22 22 1 2 2 21 2 11 1 2 111 12 2 11 (1)(2,1)(2,1) 1 2 1( 1 )KR aeaea eEEt r k kn E nknnknk k EE nkn nkk knB I k nkn kn 1BIAS I 22 1 21 22 2(,) 1(1,) 1(,1) 1 (,1) 1(,1) 1 1(1,).ea ea ea ea ea ea a ea a ea ea e ea eaB II InI nkk I (443) To evaluate the accuracy of the bias corre ction term estimator, the true bias correction term (which is the negative true bias) of 1M for 1M can also be calculated as 11 MEM for REML variance co mponent estimates of Because 1 0M when aemm the expected value of 1M is PAGE 89 89 2 1 2 2 1 11 1 11, 21 ,.ae ee a ae aEME k nk Ekm m nk kmkm m Em m nknknkm (444) We establish that 21,( ,1 ) ,a ae eam EmmI nk nk (445) 2 2 2 2 2 2 222 ,, 1 21 1 21 (1,) (1,) 1(,) 2 (1,),e ae ea ea ea eakmk Em mEs ws nk nkn kkn Ess nkn kkn B I nknB k I nk (446) and 2 2 2 2 21 1 ,( 1 ) (1) (2,1) (2,1) (,) 12 (2,1). 13e a ae ae ea ea ea eakm k Em mk n nkm nkkn B I B kknk I nknk (447) Combining Equations (445), (446), and (447) into Equation (444) gives us the expected value for 1M with REML variance component estimates: PAGE 90 90 2 21 1 2 2 22 (,1) (1,) 12 (2,1). 13ea ea eak EMII nknk kknk I nknk (448) Thus the true bias correction term of 1M for 1M is 2 11 2 11 11 12 (2,1) 13 (,1)2(1,).ea eaeaMEM k nk kknk I knk IkI (449) We can now make comparisons among the proposed bias correcti on term estimators of the nave MSEP, 1M Figure 413 pictorially demonstrates the bias of 1M for 1M under REML methods. From this figure it is easy to see that 1M has both positive and negative bias depending on the value of Figure 414 depicts the tr ue bias correction term, 11 MEM from Equation (449), along with the bias correction term approximation, 1 2tr 1 I (Equation (440)) and its expected valu e under REML estim ation (Equation (443)). First note that the effect of substi tuting varian ce parameter estimates in the bias correction approximation does not have much of an impact, unlike the results of the correction term approximation. Although 1 2tr 1 I and 1 2Etr 1 I are nearly equal over the entire span of both significantly underestimate the true bias correction term over most of the parameter space. In fact, the true bias correction term is positive for 0.75 reflecting that 1M is underestimating 1M However, the bias approximation and PAGE 91 91 expected value are negative. Thus by using th is bias correction term estimator, we would actually exacerbate the problem and si gnificantly increase the amount by which 1M is already underestimating 1M That is, while 1M tends to underestimate 1M for 0.75 the correction by the bias correction term actually causes further underestimation of 1M especially for small values of The use of this form of the bias correction term would actually produce a poorer estimate for 1M than using 1M alone. As increases, we see that the true bias correction term becomes negative, reflecting that 1M is overestimating 1M ; thus it is appropriate for the bias correction term estimator to be negative. As shown in Figure 414, the bias correction term approxima tion performs significantly better in this upper range of the parameter space than for .75 Now we compare this result to the bias adjustment under the lin ear parameterization, 22,a recalling that 1 2tr tr tr 1 1 1 I A I A I where tr1A I is in Equation (415). Figure 415 shows 11 MEM along with tr 1A I (Equation (415)) and Etr 1A I (Equation (422)). We see a much different result for this bias correction term than we did for 1 2tr 1 I Now we see the bias correction term approximation and estimator significantly overestimating the true bias correction term, especially for large values of For small values of the estimator is fairly accurate. This is the exact opposite of PAGE 92 92 the previous result in Figure 414. Figure 416 compares the accuracy of 1 2tr 1 I and tr 1A I for 11 MEM For .75 11 MEM is positive. In this range, tr 1A I is a significantly more accurate estimator because 1 2Etr 1 I is negative. For .75 11 MEM is negative, and 1 2tr 1 I is significantly more accurate than tr 1A I Thus the choice of which estimator is better depends on the value of and also on whether it is better to overor underestimate 1M Note that SAS uses the parameterization in the DDFM = KR option of PROC MIXED. Hence the results from Figure 415 are a pplicable to SAS procedures. 4.3.3 Expected Value of KRBIAS with ANOVA Covariance Parameter Estimation We can do this same comparison for ANOVA variance parameter estim ates. Recall that the bias correction approximation in Equation (440) is the same under ANOVA and REML varian ce parameters due to the choice of 1I However, because the ANOVA variance component estimates are not truncated, the expected values of the bias correction term estimator, 1 2Etr 1 I and the nave MSEP, 1EM will differ from the REML case. First, recall from Equation (429) that 22112 13kkn E knk (450) and note that 21 21211a au EEmE k (451) PAGE 93 93 Using Equations (450) and (451), the expected value for the bias correction term estimator is 2 2 22 22 1 21 2 11 1 2 111 12 2 11 2KREEt r k kn E nknnknk k EE nkn nkk k 1BIAS I 2 2 21112 1 113 2 1 kkn nknknk nkk (452) Note that by allowing the variance parameter estimates to take values outside of the parameter space, 1M does not maintain the nonnegative defini te property. That is, in taking the expected value of 1M with ANOVA variance parameter estimates, we are including negative values of 1M Allowing variance parameter estimates outside of the parameter space in expected value calculations does not affect the definiteness properties of th e bias correction term, 1 2tr 1 I since it is indefinite as will be discu ssed in Section 4.3.4. The calculation for the expected value of 1M proceeds as follows: 2 1 2 2 22 21 2 1 11 21 212 13ee a aEME k nk kmkm m E nknknkm kkknk nknknknk (453) PAGE 94 94 Thus the true bias correction term (negative bias) of 1M for 1M is 2 11 2 22 21 2 2 21 11 212 13 211 13 MEM k nk kkknk nknknknk kkn nknk (454) We can now compare this to the bias correction approximation, 1 2tr 1 I and the expected value of the bias correction estimator, 1 2Etr 1 I ,as well as tr 1A I and Etr 1A I Figure 417 shows 1M and 1EM We see that 1EM consistently underestimates 1M over the entire parameter space, growing more severe as approaches 1. Figure 418 compares the tr ue bias correction term, 11MEM to 1 2tr 1 I and 1 2Etr 1 I We see a very similar resu lt as in Figure 414 for REML methods. Again, while 1M underestimates 1M adding 1 2tr 1 I as a bias correction would increase the amount by which 1M is underestimated. The bias correction term improves as increases but never becomes positive, and therefore never improves the unadjusted estimator, 1M PAGE 95 95 Again we want to compare this to the bi as correction under the linear parameterization 22,a Figure 419 shows 11MEM along with tr 1A I (Equation (415)) and Etr 1A I (Equation (436)). The result is much improved in this param eterization. We can see that tr 1A I is a highly accurate estimator for the true bias correction term, 11MEM and using this estimator would improve the estimator for 1M over the entire parameter space. That is, 1Mtr 1 A I is a more accurate estimator of 1M than 1M Figure 420 compares the accuracy of tr 1A I and 1 2tr 1 I for 11MEM Because 11MEM is positive over the entire parameter space, 1 2tr 1 I is an unacceptable estimator of 11MEM However, tr 1A I is a highly accurate estimator for 11MEM with nearly constant relative bias of 6.67%. 4.3.4 Impact of Negative Values of Bias Correction Term Approximation As mentioned in Section 3.2.3 the bias correction approximation given by Equation (310) in indefinite. The balanced oneway random eff ects model under the 2, parameterization provides a sufficient example. Figure 415, shows that 1 2tr 1 I takes negative values ov er the entire range of We can determine from Equation (440) when the bias correction term estimator will be negative in terms of k and n. First rewrite Equation (440) as PAGE 96 96 2 2 22 2 22 211 11 2 2111 211 11k kn tr nknnknk kkn nknk 1 I Noting that the denominator is always positive when is in its parameter space, we determine that the bias correction term estimator in Equation (440) is negative when 21 1kn k For k = 6 and n = 6, as in our example, this value is 1.09 which means that the bias correction term estimator is negative over the entire (0, 1] parameter space for This counterexample suffices to prove that the bias correction term approximati on in its most general form, 11 1 11 2l KR ll ltr tr tr tr B B BIAS B B from Equations (310) and (311), is indefinite. This in itself is not necessarily prob lematic since the true bias correction term, 11 MEM is also indefinite. However, if the magnitude of this piece is large enough to cause the entire MSEP estimator to be indefinite, then problems arise. It is beyond the scope of this dissertation to determine if the MSEP estimator utilizing the correction term estimator and the bias correction term estimator, KR KR1M C BIAS is nonnegative definite. However, it can be shown that as an estimator for 1M KR1M BIAS (455) is indefinite by looking at the sa me special case of the balanced oneway random effects model. PAGE 97 97 The form of Equation (455) for this model is 2 1 2 2 22 2 222 2 1 11 11 1 2 111 12(1)(1)(1)(2)(1)3(1) (1)(1)KRMk nk k kn nknnknk kknknkkknkn nknk BIAS (456) To determine when Equation (456) is negative, we find the r oots of the num era tor (noting that the denominator is greater than zero for k, n > 1) by solving the quadratic equation in This yields the following result: 1 0KRM BIAS where 2 2 2 2 2 2(1)(1)(1) 3(1)(1)(1)(1)(2) (1)(2(1)) 2(1)(2(1)) 4(1)(2(1))knkk knknknkk kkn kkn kkn (457) or 2 2 2 2 2 2(1)(1)(1) 3(1)(1)(1)(1)(2) (1)(2(1)) 2(1)(2(1)) 4(1)(2(1))knkk knknknkk kkn kkn kkn For k = 6 and n = 6 this gives 0.987 and 0.130 While the lower bound on is outside the parameter space, the upper bound is within the parameter space and a valid result for the parameter estimate. In the case of k = 3 and n = 3 the result is 0.75 and 0 Again, the lower bound is outside of the parameter space but the upper bound is well within the parameter space. This demonstrates the danger in using 1KRM BIAS as an estimator for 1M or M and serves as a general warning of the possible effect of the indefiniteness of PAGE 98 98 KRBIAS The simplicity of this example raises even more cause for concern in more complex models. 4.4 Performance of Overall Estimators for the MSEP Now that all the pieces of the MSEP estimators have been evaluated, we can combine these results to get an overall picture for how accurate each MSEP estim ator is for the true MSEP. Recall that the true MSEP values will differ for REML and ANOVA variance components, since each varian ce component estimation method a dds a different amount of uncertainty to the MSEP of the EBLUP and a different amount of bias to 1M For REML methods, we will look at MSEP estimators utilizing both RB and 1I in the correction term estimator since there was no clear pref erence (see Figure 45). However, for ANOVA methods, the estimators using 1I were clearly superior to other choices of B and thus we will limit our comparison to 1I for ANOVA methods. Thus th e following equations list the possible estimators for the MSEP of the EBLUP of ita For REML variance parameter estimates we have three options: 1 2,RMtr A B (458) 1 2, Mtr 1 A I (459) and 11 2Mtrtr 1 1 A I I (460) For ANOVA variance parameter estimates we have 2 choices for MSEP estimator: PAGE 99 99 12, Mtr 1 A I (461) and 11 2Mtrtr 1 1 A I I (462) Note that while Equations (458), (459), and (461) are derived under the 22,a parameterization, due to the translation invariance of the elements involved, for simplicity we will use notation. We begin with the REML comparison. Figure 421 shows the true MSEP for the EBLUP of ita using REML variance parameter estimates, RM along with the expected values of the three options in equations (458), (459), and (460). For completeness, 1 EM is included as well. As we ha ve seen throughout with REML va riance parameter estimates, the results are dependent on the values of For small values of Equation (459) is most accurate, almost m atching the true MSEP exactly. As grows, though, both Equation (458) and (460) become more accurate than Equation (459). Note that SAS uses Equation (459) as the es timator of the MSEP in the DDFM = KR op tion of PROC MIXED, which is not the best choice over a large range of It is especially problematic for close to one, which again corresponds to values of 2a small relative to 2 Now looking at ANOVA comparisons in Figure 422, we see that Equation (461) is most accurate for the tru e MSEP, AM over the entire range of It is clearly the best choice for MSEP estimator when ANOVA methods are used. We also want to determine if REML or ANOVA methods are preferred when comparing MSEP estimators. To do so, we look at the rela tive bias of each of the five MSEP estimators PAGE 100 100 listed above for their respective true MSEP values Table 43 summarizes the results for select values of and Figure 423 illustrates this comparison. The table and figure clearly shows that the MSEP estimator with the smallest relative bias over the entire range of is 12 Mtr 1 A I which uses ANOVA methods. Thus for accuracy of the MSEP estimator, ANOVA variance component estimation under the parameterization with MSEP estimators utilizing 1B =I are the best choice among the ones compared here. Table 43. Relative bias (%) of MSEP estimators under REML and ANOVA estimation 0.2 0.4 0.6 0.8 1 1M 13.28 26.27 30.88 22.76 37.38 1 2* *RMtr A B 16.75 10.59 11.71 36.21 140.50 1 2 Mtr 1 A I 4.03 22.33 65.91 173.14 620.57 11 2Mtrtr 1 1 A I I 26.84 18.26 2.72 57.93 294.72 1M 17.72 51.38 93.33 142.68 200.0012 Mtr 1 A I 1.19 3.42 6.24 9.51 13.35 11 2Mtrtr 1 1 A I I 29.70 36.58 49.77 67.18 88.37 **Monte Carlo values used to compute relative bias. It is important to note that while th is is the MSEP estimator used by SAS in the DDFM = KR option when ANOVA methods are chosen w ith this model, the MSEP estimates are truncated at zero which changes the expected value of the MSEP estimator from the result given here. The disadvantage of using ANOVA methods is the likelihood of negative variance parameter estimates. However, by truncating the MSEP estimates, SAS negates the advantage PAGE 101 101 of increased accuracy associated with ANOVA methods. The implication of allowing untruncated MSEP values is that negative estimates of the MSEP are possible. The impact of negative variance parameter estimates on EBLUP procedures in the balanced oneway random effects model are further explored in Chapter 5. 4.5 Comparison of Prediction Interval Methods Several methods for conducting hypothesis tests in linear mixed models are reviewed in Chapter 2. Table 21 contains a summary of the methods, including how the distribution is determined and which standard error is used in th e test statistic. We now want to examine these methods in terms of the balanced oneway random effects model. Because we are estimating the realized value of a random eff ect, it is more appropriate to ex amine the methods in terms of prediction intervals. The goal of this examination is to determin e the accuracy of the prediction interval estimation methods and determine where further investigation is necessary. Four prediction intervals will be compared in a Monte Carlo simulation study to determine which method has the most accurate coverage rate. All of the prediction in tervals have a similar structure. The estimate of the realized value of the random effect, 1a, is ,the EBLUP for 1a, as in Equation (42). This is consistent in all four methods. The structure for the prediction inte rvals is 1/2, .dfMSEPt (463) The estimate of M SEP and the degrees of freedom for the tdistribution depend on the method used to calculate the pr ediction interval. The method s used to determine the MSEP estimate and the degrees of freedom are summarized in Table 44. PAGE 102 102 Table 44. Summary of pred iction interval procedures. GB KR FC FC MSEP 1M 1 2 M tr A B 1 2 M tr A B 1 1 2 M tr tr A B B DF 22 varGB GB GBdf MSEP MSEP K RGBdfdf 22 varFC FC FCdf MSEP MSEP 22 varFC FC FCdf MSEP MSEP Note: var B MSEP hh where MSEP h and 1BI. Note that all of the methods use a Sattert hwaitetype method for estimating the degrees of freedom. The degrees of fr eedom are calculated using 22 var MSEP df M SEP (464) where var B MSEP hh for MSEP h and 1BI. The disparity between the methods comes in the parameterizati on, and the estimate of the MSEP (or standard error) which subsequently impacts the degrees of freedom. We again study results under two parameterizations, 2, and 22,a Note that due to the translation invariance of 1M and tr AB, the figures and results demonstrated earlier in this chapter pertaining to 1 M and 1 2 Mtr A B still apply to the standard error estimates used in the GiesbrechtBurns, KenwardRoger, and FC prediction interval procedures. PAGE 103 103 Two of the methods for formulating the prediction interval are used in SAS PROC MIXED. The first, from the DDFM = SATTERTH option, is similar to the GiesbrechtBurns (1985) procedure outlined in Ch apter 2; however, in this a pplication we are calculating prediction limits for the realized value of a random effect rather than a fixed effect, as in that paper. The method is also developed in Jeske a nd Harville (1988) and Fai and Cornelius (1996). We will refer to it as the Gies brechtBurns prediction interval method, since it uses the nave MSEP estimate. The standard erro r is estimated by the nave MSEP, 1/2 1 M where 1 M is in Equation (43). The other m ethod available in PROC MI XED through the DDFM = KR option is the KenwardRoger (1997) approach. The standard error estimate includes a correc tion term and a bias correction term, as in Equation (459). Note that the KenwardRoger prediction interval is determ ined under the parameterization. Recall that under the parameterization, 22,a the bias correction term estimate is equal to the correction term estimate, thus the MSEP estimate used in the KenwardRoger prediction interval is 11 22 MtrMtr A B A B It is important to keep the parameterization in mind throughout, as the MSEP estimate and the degrees of freedom calculations depend on which parameteri zation is used. It is demons trated in Chapter 2 that in a singledimension linear combination, the Kenwar dRoger method for determining the degrees of freedom reduces to the GiesbrechtBurns me thod. Thus, although the KenwardRoger and GiesbrechtBurns methods for determining the prediction limits use different estimates for the standard error, the degrees of freedom for the t distribution will be the same. KenwardRoger PAGE 104 104 prediction limits adjust the MSEP estimate for estimating unknown variance components but do not adjust the t distribution for the cha nge in standard error. The other two procedures for creating prediction intervals, FC and FC are based on the FaiCornelius (1996) hypothe sis testing methods. For the FC intervals, both the MSEP estimate and the degrees of freedom of the t distribution account for th e correction term and the bias correction term. Note that this is actually an ex tension of the methods developed by Fai and Cornelius (1996). The bias correc tion term was not considered as an adjustment to the MSEP estimate in the methods they proposed. The bias co rrection term is added here to be consistent with MSEP estimator used in the KenwardR oger method. The FC methods differ in the parameterization. Under the 22,a parameterization, the MSEP estimate is the same as in the KenwardRoger parameterization, 11 22 MtrMtr A B A B Unlike the KenwardRoger prediction interval, the FC procedure adjusts the de grees of freedom of the t distribution for the adjustments to the MSEP estimate. The degr ees of freedom are still calculated using a Satterthwaite method; however, now the corrected MSEP estimate, 1 2 Mtr A B is used in the calculation of the degrees of freedom, rather than the nave MSEP estimate, 1 M. The final prediction limit method considered, FC differs from the other three in the parameterization, 2, Recall that the bias correction term estimate is not translation invariant, and thus by changing the parameterizati on, we also change the MSEP estimate. The PAGE 105 105 MSEP estimate for the FC procedure is 11 2 Mtrtr A B B This is used in both the standard error estimate and the degrees of freedom calculation. A Monte Carlo simulation study was conducted to compare the four prediction interval methods. The study is similar to the one conducted in Section 4.1 to evaluate the correction term estimators. Using the RANNOR function in SAS, 10,000 independent sets of eij and ai were generated from 2(0,1) N and 20,aN for several values of2 a respectively. The values of 2 a correspond to 0.05, 0.1, 0.15, 1.0. Setting 0 10,000 sets of :1,,16ijyikj were created from the model ijiij y ae The number of levels of the random effect is varied to study the impact on the performance of the prediction interval procedures. The values for k are k = 3, 6, 15, or 30, while n is held constant at n = 6. Simulating the data in this way allows the realized values generated for the 'ias to be tracked. These are the realized values of the random effect for whic h the prediction interval s are constructed. By tracking the realized values of the random effect, we can dete rmine how often the prediction intervals contain the true valu e of the realized ra ndom effect, providing th e actual (simulated) coverage rate for a nominal 95% prediction interval. The prediction limits for 1a for the GiesbrechtBurns and KenwardRoger procedures are produced by the CL option in the ESTIMATE statement in PROC MIXED in SAS, with the DDFM = SATTERTH or DDFM = KR options, respec tively. To calculate the FC prediction limits for either parameterization, the degrees of freedom must be calculat ed directly and then used to determine the critical value from the t distribution. For the FC procedure, the standard error from the KenwardRoger procedure is used to calculate the prediction limits. For PAGE 106 106 the FC procedure, the standard error also has to be calculated directly in addition to the degrees of freedom because of the ch ange in parameterization. The SAS programming statements used to perform the simulation st udy are contained in Appendix A along with the derivatives needed to compute the degrees of freedom for the FC procedures. The results of the simulation study are best de monstrated pictorially. Figures 424 through 427 contain the results of the simulation study for k = 3, 6, 15, and 30, respectively. It is clear that the KenwardRoger procedure produces the most inflated cove rage rates consistently across values of k and for the entire parameter space of The prediction limits for the KenwardRoger procedure will always be larger than the GiesbrechtBurns procedure since 11 2 MtrM A B Adjusting the degrees of freedom in the FC procedure for the adjusted MSEP estimate generally increas es the degrees of freedom, causing the critical value from the t distribution to shrink and thus reducing the width of the prediction interval. The coverage rates for the FC procedure are generally closer to 95% than the coverage rates for the KenwardRoger procedure. The performance of the FC prediction interval is more dependent on the value for k and the parameter than the other intervals. For k = 3, the coverage rate for small values of is exceedingly small. As increases, the performance improves dramatically. This is due to the inclusion of the bias correction term under the 2, parameterization. Small values of k combined with small values of produce small MSEP estimates. Recall from Figure 423 that the relative bias of 11 2 Mtrtr A B B approached 50% for small values of This is caused by the significant underestimation of the true bias correction term by PAGE 107 107 1 2 tr B (see Figure 414). For larger values of k, the FC prediction interval procedure compares favorably with the other procedures and consistently improves on the performance of the KenwardRoger predic tion interval for larger values of For larger values of recall that 1 2 Mtr A B overestimated the true MSEP more so than 11 2 Mtrtr A B B Thus we would expect a more accurate prediction interval with the FC method in this range of the parameter space. To demonstrate the relationship between the pr ediction intervals more concretely, consider the hypothetical case where k = 6, n = 6, with data producing vari ance parameter estimates of 2 1 and 2 .5a Table 45 contains the values for the standard error, degrees of freedom, critical t value and the width of the prediction interv al produced by each of the four procedures in this scenario. The width of the KenwardRoge r prediction interval in creases over the width of the GiesbrechtBurns interv al due to the increase in the standard error estimate. The width of the FC prediction interval is smaller than both the GiesbrechtBurns and the KenwardRoger intervals because of the increased degrees of freedom and subsequent de crease in the critical t value. The width of the FC interval is by far the smallest due to the decrease in the standard error estimate and the critical t value. Finally, to demonstrate the effect of k on the performance of th e prediction interval procedures, Figure 428 shows the KenwardR oger coverage rates for all values of k considered in the simulation study. This is demonstr ative of all the procedures in that as k increases, the procedures perform better and th e coverage rates move closer to the nominal value of 95%. PAGE 108 108 This study shows the importance of investigating the FaiCornelius methods for hypothesis tests and prediction intervals in more complex situ ations. These preliminary results indicate that the FaiCornelius methods often perform bette r than the KenwardRoger and the GiesbrechtBurns methods used in SAS. The KenwardRoger procedure produces an overlyconservative prediction interval because it does not adjust the degrees of freedom for the inflated MSEP estimate. The study also demonstrates the imp act of the parameterization and the lack of transform invariance of the bias correction term on accuracy of the predic tion interval methods. In more complex covariance structures, the paramete rization may be vital to an accurate analysis. Table 45. Effect of predic tion interval procedures for k = 6, n = 6, 2 1 and 2 .5a GB KR FC FC Standard Error estimate 0.43301 0.46894 0.46894 0.40196 Degrees of Freedom 8.15 8.15 16.39 16.52 .025,dft 2.298 2.298 2.116 2.114 Width of CI 1.99 2.16 1.98 1.70 PAGE 109 109 7 6 5 4 3 2 1 0 1 0 0.2 0.4 0.6 0.8 1 1.2 k=6 k=9 k=12 k=15 k=18 Figure 41. Equation (413) as a function of for several values of k, holding n = 6. Equation (414) hol ds when functions cross zero line. PAGE 110 110 0 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0 0.2 0.4 0.6 0.8 1 1.2 tr 1A I RC Rtr A B Figure 42. Accuracy of approximations Rtr A B and tr 1A I for the correction term, RC under REML variance component estimation. PAGE 111 111. 0 0.005 0.01 0.015 0.02 0.025 0.03 0.035 0.04 0.045 0.05 0 0.2 0.4 0.6 0.8 1 1.2 REtr A B RC Rtr A B Figure 43. Accuracy of approximation Rtr A B and estimator REtr A B for the correction term, RC under REML variance component estimation. PAGE 112 112 0 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0 0.2 0.4 0.6 0.8 1 1.2 Etr 1A I tr 1A I RC Figure 44. Accuracy of approximation tr 1A I and estimator Etr 1A I for the correction term, RC under REML variance component estimation. PAGE 113 113 0 0.01 0.02 0.03 0.04 0.05 0.06 0 0.2 0.4 0.6 0.8 1 1.2 REtr A B RC Etr 1A I Figure 45. Comparison of accuracy of estimators for the correction term, RC under REML variance component estimation. PAGE 114 114 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 00.20.40.60.811.2 Atr A B Atr A B AC tr 1A I Figure 46. Comparison of accuracy of approximations tr 1A I Atr A B and Atr A B for the correction term, AC under ANOVA variance component estimation. PAGE 115 115 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 00.20.40.60.811.2 AC Atr A B AEtr A B Figure 47. Accuracy of approximation Atr A B and estimator AEtr A B for the correction term, AC under ANOVA variance component estimation. PAGE 116 116 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 00.20.40.60.811.2 AC Atr A B AEtr A B Figure 48. Accuracy of approximation Atr A B and estimator AEtr A B for the correction term, AC under ANOVA variance component estimation. PAGE 117 117 0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0 0.2 0.4 0.6 0.8 1 1.2 AC Etr 1A I tr 1A I Figure 49. Accuracy of approximation tr 1A I and estimator Etr 1A I for the correction term, AC under ANOVA variance component estimation. PAGE 118 118 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 0 0.2 0.4 0.6 0.8 1 1.2 AC AEtr A B AEtr A B Etr 1A I Figure 410. Comparison of accuracy of estimators for the correction term, AC under ANOVA variance component estimation. PAGE 119 119 0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0 0.2 0.4 0.6 0.8 1 1.2 AC RC REtr A B Etr 1A I Etr 1A I Figure 411. Accuracy of estimator s for the correction term under REML, and ANOVA, variance component estimation. PAGE 120 120 100.00% 0.00% 100.00% 200.00% 300.00% 400.00% 500.00% 0 0.2 0.4 0.6 0.8 1 1.2Relative Bias REtr A B Etr 1A I Etr 1A I Figure 412. Relative bias of estimat ors for the correction term under REML, or ANOVA, variance component estimation. PAGE 121 121 0 0.2 0.4 0.6 0.8 1 1.2 0 0.2 0.4 0.6 0.8 1 1.2 1M 1 EM Figure 413. Bias produced by estimating 1M with 1 M. PAGE 122 122 0.6 0.5 0.4 0.3 0.2 0.1 0 0.1 0 0.2 0.4 0.6 0.8 1 1.2 1 2 tr 1 I 1 2 Etr 1 I 11 MEM Figure 414. Accuracy of approximation 1 2 tr 1 I and estimator 1 2 Etr 1 I for the true bias correction term, 11 MEM under REML variance component estimation. PAGE 123 123 0.03 0.02 0.01 0 0.01 0.02 0.03 0.04 0.05 0.06 0.07 00.20.40.60.811.2 tr 1A I Etr 1A I 11 MEM Figure 415. Accuracy of approximation tr 1A I and estimator Etr 1A I for the true bias correction term, 11 MEM under REML variance component estimation. PAGE 124 124 0.6 0.5 0.4 0.3 0.2 0.1 0 0.1 0 0.2 0.4 0.6 0.8 1 1.2 11 MEM 1 2 Etr 1 I Etr 1A I Figure 416. Comparing the accuracy of 1 2 Etr 1 I and Etr 1A I for the true bias correction term, 11 MEM PAGE 125 125 0.15 0.05 0.05 0.15 0.25 0.35 0.45 0.55 0.65 0.75 00.20.40.60.811.2 1M 1EM Figure 417. Bias produced by estimating 1M with 1M. PAGE 126 126 0.6 0.5 0.4 0.3 0.2 0.1 0 0.1 0.2 00.20.40.60.811.2 1 2 Etr 1 I 1 2 tr 1 I 11MEM Figure 418. Accuracy of approximation 1 2 tr 1 I and estimator 1 2 Etr 1 I for true bias correction term, 11MEM under ANOVA variance component estimation. PAGE 127 127 0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0 0.2 0.4 0.6 0.8 1 1.2 Etr 1A I tr 1A I 11MEM Figure 419. Accuracy of approximation, tr 1A I and estimator Etr 1A I for true bias correction term, 11MEM under ANOVA variance component estimation. PAGE 128 128 0.6 0.5 0.4 0.3 0.2 0.1 0 0.1 0.2 0 0.2 0.4 0.6 0.8 1 1.2 11MEM 1 2 Etr 1 I Etr 1A I Figure 420. Comparing the accuracy of 1 2 Etr 1 I and Etr 1A I for the true bias correction term, 11MEM PAGE 129 129 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 0.2 0.4 0.6 0.8 1 1.2 1 2 Mtr 1 A I 11 2 Mtrtr 1 1 A I I 1 RRMMC 1 M 1 2RMtr A B Figure 421. Comparison of estimator s of the MSEP for the EBLUP of ia utilizing REML variance component estimation. PAGE 130 130 0.2 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 1.2 1 AAMMC 12 Mtr 1 A I 11 2 Mtrtr 1 1 A I I 1M Figure 422. Comparison of estimator s of the MSEP for the EBLUP of ia utilizing ANOVA variance component estimation. PAGE 131 131 100.00% 50.00% 0.00% 50.00% 100.00% 150.00% 200.00% 250.00% 300.00% 0 0.2 0.4 0.6 0.8 1 1.2% Relative Bias 11 2 Mtrtr 1 1 A I I 12 Mtr 1 A I 1 2 Mtr 1 A I 11 2 Mtrtr 1 1 A I I 1 2RMtr A B 1 M 1M Figure 423. Relative bias of possibl e estimators for MSEP of EBLUP for ia PAGE 132 132 0.9 0.92 0.94 0.96 0.98 1 1.02 00.10.20.30.40.50.60.70.80.91Coverage Rate FC FC K R GB Figure 424. Simulated coverage rates of prediction interval procedures for 1ta with k = 3. PAGE 133 133 0.88 0.9 0.92 0.94 0.96 0.98 1 1.02 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1Coverage Rate FC FC KR GB Figure 425. Simulated coverage rates of prediction interval procedures for 1ta with k = 6. PAGE 134 134 0.93 0.94 0.95 0.96 0.97 0.98 0.99 1 1.01 00.10.20.30.40.50.60.70.80.91Coverage Rate FC FC KR GB Figure 426. Simulated coverage rates of prediction interval procedures for 1ta with k = 15. PAGE 135 135 0.94 0.95 0.96 0.97 0.98 0.99 1 1.01 00.10.20.30.40.50.60.70.80.91Coverage Rate FC FC KR GB Figure 427. Simulated coverage rates of prediction interval procedures for 1ta with k = 30. PAGE 136 136 0.94 0.95 0.96 0.97 0.98 0.99 1 1.01 00.10.20.30.40.50.60.70.80.91Coverage Rate k = 30 k = 15 k = 6 k = 3 Figure 428. Simulated coverage rates for the KenwardRoger prediction interval procedure. PAGE 137 137 CHAPTER 5 THE EFFECT OF NEGATIVE VARIANCE COMPONENTS ON EBLUP The effect of negative varian ce component estim ates on EBLUP procedures is a topic that has been largely neglected (Stroup and Littell 20 02). While the use of REML estimates, which are truncated at zero, would seem to negate this issue, there are still many circumstances when other variance parameter estimation methods ar e chosen and negative variance components may arise. For example, as shown in Stroup and Litt ell (2002) and in the resu lts on the accuracy of the correction term and bias correction term estim ates in Chapter 4, REML may not always be the preferred choice for varian ce component estimation. One strategy for handling negative variance component estimation is to allow that th e component may be a covariance rather than a variance, thus alleviatin g the problem of truncating at zero. We begin to look at the impact of that strategy by again turning to the special case of the balanced one way random effects model. 5.1 Negative Variance Component Estimates and the Balanced One Way Random Effects Model To demonstrate the current issues with ne gative variance component estimation, we first look at the current SAS 9.1 output for PROC MIXED, both when the variance component estimates are truncated at zero and when they are not. Using the REML NOBOUND statement in the procedure statement allows the variance co mponent estimates to take any value on the real line. Note that because we are in a bala nced situation, the ANOVA variance component estimation would produce the same results as REML NOBOUND. We compare the effect of truncating using simulated data, gene rated from a normal distribution with 2.01a and 21 This distribution produces a high likelihood that the data generate d will have larger variation within a block than between the blocks, he nce often producing a negative estimate for 2a For demonstration purposes, the number of levels of the random effect is k = 5 and the number of PAGE 138 138 observations at each level is n = 6. A dataset that meets the a bove condition is given in Table 51. It is easy to spot that more variation occurs within the block le vels than between them. Table 51. Generated data that produ ces negative variance component estimate Replicate 1 2 3 4 5 6 Block 1 2.34754 0.24672 0.5585 0.60564 0.9062 1.37627 Block 2 1.01248 0.0493 0.49583 1.3952 0.8644 1.93825 Block 3 0.18826 0.91668 0.68925 0.96593 0.95929 0.7838 Block 4 0.4741 1.31312 1.5036 0.06752 1.2383 0.7888 Block 5 1.2176 0.64999 0.32013 1.4081 0.1774 1.77956 The results from the REML option in the MIXED procedure of SAS are contained in Tables 52 and 53 and the results from the REML NOBOUND options in th e MIXED procedure are contained in Tables 54 and 55. Also, for comparison, least square means (LSMEANS) generated by PROC GLM are given in Table 56. These give a starting point from which to compare the EBLUPS resulting from the REML and REML NOBOUND options. The following SAS code generates the output in Tables 52 and 53: proc mixed data = &sample method = reml ; class blk trt; model y = /ddfm = kenwardroger ; random blk / s ; Table 52. REML variance parameter estimates Covariance parameter Estimate Block 0 Residual 1.0643 Table 53. Solution for random effects w ith REML variance parameter estimates Effect Block Estimate Std Err PredDF t Value Pr > t Block 1 0 . Block 2 0 . Block 3 0 . Block 4 0 . Block 5 0 . PAGE 139 139 The NOBOUND option in the following SAS code is the only diffe rence from the previous code and generates the SAS output in Tables 54 and 55: proc mixed data = &sample method = reml nobound ; class blk trt; model y = /ddfm = kenwardroger ; random blk / s ; Table 54. REML NOBOUND variance parameter estimates Covariance Parameter Estimate Blk 0.1410 Residual 1.1810 Table 55. Solution for random effects with REML NOBOUND variance parameter estimates Effect Block Estimate Std Err PredDF t Value Pr > t Block 1 0.4846 0 29 Infty <.0001 Block 2 0.3461 0 29 Infty <.0001 Block 3 0.4106 0 29 Infty <.0001 Block 4 0.8870 0 29 Infty <.0001 Block 5 0.3379 0 29 Infty <.0001 Table 56. LSMEANS from PROC GLM Block LSMEAN 1 0.51858487 2 0.18960919 3 0.48927203 4 0.02458600 5 0.46046652 The results from the REML option, which trunc ates the variance component estimate for block effect at zero, are straightforward. The solution statement produces zero estimates for all blocks and missing values for the standard error, tva lues, degrees of freedom, and pvalues. The REML NOBOUND option is not quite so straightforward. When variance component estimates are positive, we can think of the BLUP as a shr inkage predictor. In other words, using the PAGE 140 140 variance component estimate as a weight, the bl ock means are shrunk towa rd the overall mean. In addition, in a balanced model, the BLUPS mainta in the rank order of the lsmeans. This can be easily seen in the structure of the BLUP for the balanced oneway random effects model in Equation (42). Now, looking at the EBLUP estimates in Tabl e 55, the first indication of a problem is that the standard errors are listed as zero and hence the tvalues are listed as infinity. The calculations for standard error in this case are negative, causing SAS to truncate the standard error at zero. Comparing the estimates in Table 55 to the LSMEANS from PROC GLM in Table 56 shows that the nega tive variance component estimate causes the EBLUP estimates to invert their rank order and expands the range of the estimates. The block with the largest LSMEAN (0.5186) now has the smallest EBLUP (0 .4846) and, likewise, the block with the smallest LSMEAN (0.0246) now has the larges t EBLUP (.8870). Also, the range of the LSMEANS is 0.5432 while the range of the EBLU P estimates is 1.3716. We normally expect the BLUP procedure to shrink the block m eans toward the overall mean according to the variation in the model; however, in this case, the estimates are inverted about the mean and expanded away from the overall mean. While the standard errors and tvalues listed with the EBLUPs are a clear red flag, the unexpected results of the EBLUPS require further investigation. Note also that the standard e rrors may be positive even when 20a ; this would eliminate the red flag and leave more room for error in the interpretation of the results. If the data analyst believes that a oneway random effects model is the true model for the research situation, then the r ecommendation from an EBLUP sta ndpoint is to truncate at zero rather than proceed with a negative variance component estimate. A ny use of EBLUPS while allowing the variance component to gi ve a negative estimate is likely to yield misleading results. PAGE 141 141 The LSMEANS likely yield more reliable estim ates than the EBLUPS from the REML NOBOUND option. Keep in mind that it is not always the best choice to truncate at zero when variance component estimates are ne gative if the analyst is interested in fi xed effects (Stroup and Littell 2002). 5.2 Considering the Variance Parameter as a Covariance One idea that surfaces when faced with negative variance component estimates is considering the parameter as a covariance rather than a variance, thus allowing the parameter estimate to take on values over the entire real line Smith and Murray (1984) consider this option and the impact on estimation and hypothesis te sting of the covarian ce component. For the balanced one way random effects model discussed in Chapter 4, the covariance structure would be modified from 22 2cov(,), 0, ijija a y yiijj iijj ii to 2cov(,), 0, ijija a y yiijj iijj ii where 20aa One problem overlooked by Smith and Murray (1984) is that in a one way model, if the variance structure is modified in this way, the model is no longer a typical random effects model. The motivating example used in Smith and Murray (1984) is weaning rates of twin calves for 40 cows. In the original model, the cow effect is considered random, with the cows regarded as a PAGE 142 142 random sample from a population of cows, and 2 a the variance associated with this population. When the estimate for 2a is negative, the definition of the parameter is changed from the population variance of the cows to the covariance between twins born to the same cow. Thus the population variance for the cows is ignored all to gether and the negative covariance estimate is interpreted as reflecting competition for nutrition between the twin calves. The negative value of the variance parameter estimate is likely reflecting the combination of the positive cow population variance and the negative correlation between twins born to the same cow. That is, there is more negative intraclass variability th an positive interclass variability resulting in a negative parameter estimate. The inversion of the LSMEANs that we s ee with the EBLUPs in PROC MIXED when the variance component estimate is negative occurs because the model is effectively changed to this Smith and Murray (1984) model. The EBLUP calculated by PROC MIXED for ia in this case is ... 2 a i an y y n giving a negative coefficient when 2 0an Furthermore, a may be sufficiently large so that 2 1 a an n Thus rather than shrinking the means, as we expect EBLUP procedures to do, the EBLUP inverts and expands the range of the means. Clearly this is an unacceptable result. The question remains: what is an analyst to do when faced with this result. One possible solution is derived in the next section. PAGE 143 143 5.3 BLUP derivation for Random Effects Mo del with Correlated Errors We now present a model that would account for both the population variance of the random effect and the correlation of replicates with in a level of the random effect. Consider a model with a random effect, as we ll as correlated errors. The model and variance structure in Equation (41) would be modified to: 2,1,,,1,, 0,ij iij ia y aeikjn aN where are normally distributed withije 20 cov(,), 0, ij a ijijaEe iijj eeiijj ii (51) In mixed models notation, this equates to y=X +Zu+e where 12, ,,N kn kaaa X1 ZI1 uN0,G where 2 akGI and 11,12,kneee eN 0 R where 2()knanRIIJ so that PAGE 144 144 22 knaan V=ZGZ+RIIJ. From general matrix theory (e.g., Searle 1982), we have 2 1 2 221aa kn n aan VII J which gives the BLUP for u as 2 .. 22 1a kn aa y n 1uGZVyI y or the BLUP for ia as 2 ... 22 a ii aan ay y n (52) Note that for the model structure given in Equation (51) to be valid, V m ust be positive definite. Therefore, the parameter space for which the model is valid is 2 2220, 0, aaan In this parameter space the coefficient for the BLUP, given in Equation (52), is nonnegative. Thus the order of the LSMEANS for the bloc ks is preserved in the BLUP. One key consideration for this model is the problem of overparameterization. The parameters 2a and a are confounded and cannot be simu ltaneously estimated. One solution is to use an estimate for 2 a from previous experience and consider the parameter as known. This leaves only a and 2 as unknown and the EBLUP analysis can be pe rformed where the unknown variance parameters are estimated by an unbounded method, such as REML NOBOUND in PROC MIXED. There PAGE 145 145 are many situations, such as agriculture or educ ation, where using a previous estimate as the parameter is realistic. Data contained in Table 55 are generated from a distribution as described in Equation (51) with 2.9, .13a and 2.07a Table 57. Data generated from distribution in Model (51) Replicate 1 2 3 4 5 6 Block 1 1.77754 0.72357 0.79317 0.79674 1.31455 0.98605 Block 2 0.62131 1.15758 1.21619 0.24776 0.5867 0.12472 Block 3 0.3412 0.7861 0.41517 0.75414 1.34176 0.75806 Block 4 0.05152 0.9336 0.31271 0.3097 1.04458 1.07845 Block 5 0.75952 0.27854 0.28914 1.03234 0.88724 2.24931 The appropriate SAS statements to analyze this data in the manner described in Equation (52) are data gmat ; input row col value; cards; 1 1 .07 2 2 .07 3 3 .07 4 4 .07 5 5 .07 ; proc mixed data = &sample method = reml ; class blk trt ; by rn ; model y = /ddfm = kenwardroger ; random blk / gdata=gmat g s ; repeated trt / sub = blk type=cs ; run ; The gdata option after the random st atement provides the known G matrix which in this case is .07 5GI where nI is an ndimensional identity matrix. The output from this analysis of the data in Table 57 are contained in Tables 58 and 59. The EBLUPS for the levels of the random effect are generated by the solutions or s option PAGE 146 146 in the random statement of the SAS code, and are given in Table 59 under Estimate. Comparison of these values to the LSMEANS ge nerated by GLM in Table 510 show that the function worked as expected, maintaining the rank order of the means and shrinking the predictors toward the overall mean. Table 58. Covariance parame ter estimates under Mode1 (51) Covariance Parameter Subject Estimate Variance Block 0.9497 CS Block 0.1311 Table 59. Solution for ra ndom effects under Model (51) Effect Block Estimate Std Err PredDF t Value Pr > t Block 1 0.00593 0.2646 2.16 0.02 0.9840 Block 2 0.2849 0.2646 2.16 1.08 0.3868 Block 3 0.2761 0.2646 2.16 1.04 0.3991 Block 4 0.1560 0.2646 2.16 0.59 0.6113 Block 5 0.1412 0.2646 2.16 0.53 0.6433 Table 510. LSMEANS from PROC GLM Block y LSMEAN 1 0.03457327 2 0.42190439 3 0.35697247 4 0.19015370 5 0.22234738 Note that the DDFM = KR method wa s used here which uses an adju sted standard error. If the DDFM = SATTERTH option is used, the estimates a nd degrees of freedom remain the same, but the standard error is reduced to .1647 which would impact the prediction limits as shown in Section 4.5. Given the conclusion in Chapter 4 regarding the inaccuracy of the adjustment under PAGE 147 147 certain circumstances, the accuracy of either standard error estimate and their effect on prediction intervals for the rea lized values of the random effect needs further study. This remains as future research. For comparison, Tables 511 and 512 contain the results of the analysis of the data in Table 57 if the correlation were ignored and the model were run as a random effects model with the REML NOBOUND option. We see that the va riance parameter estimate is negative and the EBLUPs are inverted and expanded as shown in Section 5.1. Table 511. Variance parameter estimates under random effects model Covariance Parameter Estimate Block 0.06110 Residual 0.9497 Table 512. Solution for random effects under random effects model Effect Block Estimate Std Err PredDF t Value Pr > t Block 1 0.005176 0 29 Infty <.0001 Block 2 0.2487 0 29 Infty <.0001 Block 3 0.2410 0 29 Infty <.0001 Block 4 0.1361 0 29 Infty <.0001 Block 5 0.1232 0 29 Infty <.0001 One issue that occurs with an alyzing data in this way is th at it is prone to convergence failure. Of 10,000 simulated data sets fr om the distribution described in Equation (51) and analyzed with this procedure, 21.5% failed to con verge when wither REML or REML NOBOUND procedures were used. Even with th is complication, this parameterization remains one of the only viable options if predictors of the realized values of the random effects are desired in a case with negative variance component estimation. PAGE 148 148 CHAPTER 6 CONCLUSION AND FUTURE WORK The prim ary goal of this dissertation was to examine the impact of estimating variance components on the estimators of the MSEP for the EBLUP. This was accomplished by investigating propertie s of generalized components of the MSEP estimator, examining the behavior of various procedures in the balanced oneway random effects model, and determining the impact of negative variance components on EBLUP procedures. Following the work by Kackar and Haville (1984), Kenward and Roger (1997), Jeske and Harville (1988), and Fai and Cornelius (1996), estimators for the correction term and the bias correction term were generalized to accommodate multiple dimension linear combinations of fixed and random effects. Properties of these ge neralized estimators were investigated. We showed that while for linear covariance stru ctures the bias correction term estimator, KRBIAS continues to reduce to th e correction term estimator, KRC which is nonnegative definite, KRBIAS in general, is indefin ite. The indefiniteness of KRBIAS causes the estimator, KR1M BIAS of the nave MSEP, 1M to be indefinite. Future work includes the investigation of defin iteness properties of the MSEP estimator KR KR1M C BIAS which includes both the correcti on term and the bias correction term. The magnitude of KR1M C may outweigh that of KRBIAS to produce a positive definite estimator. We also demonstrated that KRBIAS is not transform invariant, and that the choice of parameteri zation can have a significant imp act on the bias correction term, the MSEP estimate, and the coverage rates of prediction intervals. The balanced oneway random effects model provided a simple setting in which the various components of MSEP estimators coul d be thoroughly explored. Closed form PAGE 149 149 expressions allowed for direct examination of the correction term and bias correction term estimators. Through this example, the impact of variance component estimation techniques, the values of the variance components, and the para meterization was demonstrated. We determined that the MSEP estimator may significantly ove ror underestimate the true MSEP depending on these factors. We showed that truncating variance parameter estimates under REML estimation can cause significant (longrun) overestimation of both the correction term and the bias correction term, over most of the parameter space. While these procedures have been considered suspect near the boundary of the parameter space, this study showed that the accuracy of the methods is questionable over a much greater range of the parameter space. We also discovered that the choice of how to measure the MSE of the variance parameter estimates can significantly impact the accuracy of the correction term estimator. In REML situations, using the exact MSE of the varian ce parameter estimates in the correction term estimator resulted in a more accu rate correction term estimator than using the asymptotic MSE over twothirds of the parameter space. Howeve r, if variance parameter estimates are left untruncated, as in ANOVA variance component estimates, the asymptotic MSE provided a more accurate correction term estimator over the entire parameter space. The importance of the parameterization of the m odel is apparent in th e bias correction term estimator since it is not transform invari ant. Under a linear parameterization, KRBIAS slightly overestimates the true bias correcti on term; while under a nonl inear parameterization KRBIAS significantly underestimates the true bias correction term for a large portion of the parameter space when using REML variance component estimates. Under ANOVA variance parameter estimation, the linear parameterizatio n provides a significantly more accurate bias correction term estimator. Even with this simp le model, we see how the interaction of a few PAGE 150 150 factors complicates the results. The question of which MSEP estimator is best depends on the parameterization, the true value of the variance parameter, and the choice of variance parameter estimation method. Each of these factors deserves investigation in more complex models. These same factors impact the coverage rates of prediction intervals fo r a realized value of the random effect. We introduced a modifi cation of the FaiCornelius methods for approximating the degrees of freedom that in cludes a bias correction term in the MSEP estimator. The results of a Monte Carlo simula tion study showed that the FaiCornelius based methods compare favorably with the Giesbrecht Burns and KenwardRoger methods currently available in SAS. The simulation study also demonstr ated a shortcoming of the KenwardRoger method in that the degrees of freedom ar e not adjusted for the increased MSEP estimate for a single dimension linear combination. In f act, of the four predicti on interval procedures included in the simulation study, the KenwardR oger method consistently performed the poorest, producing the most inflated covera ge rates. Because the modified FaiCornelius method adjusts both the MSEP and the distribution for the variance component estimates, this method deserves more attention and may significantly improve the coverage rates or Type I error for hypothesis testing in many situations. Futu re work includes further investig ation of these methods in more complex settings. The impact of negative va riance component estimates on the EBLUP and the MSEP estimators was studied. We show that the EBLUP is no longer a reliable or valid estimator of the random effect when negative variance component estimates are present. The proposed solution by Smith and Murray (1984) of rega rding the variance parameter as a covariance is shown to be invalid for EBLUP procedures. An alterna tive solution of a rando m effects model with correlated errors was presented. Further study of this proposed method is indicated. Future PAGE 151 151 work also includes the investigation of the impa ct of negative varian ce parameter estimates on fixed effects estimators, their standard errors, and distribution estimation techniques. PAGE 152 152 APPENDIX DETAILS FOR MONTE CARLO SIMULATION STUDY ON PREDICTION INTERVALS The calcu lations necessary for the FC and FCprediction intervals are detailed below, followed by an example of the SAS code used for the Monte Carlo simulation study to determine coverage rates for the four prediction interval procedures in Section 4.5. Recall from Equation (464) that the inverse of th e observed infor mation matrix, 1BI and the vectors of derivatives of the MSEP, h, are necessary for the Satterthwaitemethod of calculating degrees of freedom. First we look at the FC method. The inverse of the observed information matrix for the 22,a is 22 22 22 22 2 222 11 22 1 111 kn knn knnnkkn 1B I (A1) We also need 11 2222aMtr Mtr A B A B h where, due to translation invariance, 1M is as in Equation (43) and tr A B is as in Equation (415). Thus, 1 2 222 2 2222 111 212 1a aMtr k nk kn n knnn A B (A2) and PAGE 153 153 2 22222 1 2 2 22 2 2 2 2222 2 21 1aa a a aMtr nkn kn nkn knnn A B (A3) The degrees of freedom for the FC method are calculated as 2 1 1 22 var2FCMtr df Mtr A B A B where 1 var2 Mtr 1 A B h I h with the elements of h from Equations (A2) and (A3) and 1I from Equation (A1). For the FC method, we turn to the 2, parameterization. The inverse of the observed information matrix is now 2 22 2 222 11 2 21 111 knkn kn knkkn 1B I (A4) We use the MSEP estimator utilizing both the correction term and the bias correction term elements to be consistent w ith the KenwardRoger and the FC methods. Because the bias correction term estimator is not transform invariant, the MSEP estimate is different than the one used in the parameterization. The MSEP estimator in the 2, parameterization is given in Equation (460). Thus PAGE 154 154 1 1 211 22 Mtr tr Mtr tr A B B A B B h where 1 2 2 2 2221 111 2 411 21 111Mtrtr k nk knk kn knnnknk A B B (A5) and 22 1 2 22 22 2 22 21 11 2 42111 21 11 1Mtrtr k nk kknk kn knnnknk A B B (A6) The degrees of freedom for the FCmethod are calculated as 2 1 11 2 2 1 var 2FCMtrtr df Mtrtr A B B A B B where 11 var 2 Mtrtr A B B h B h with the elements of h from Equations (A5) and (A6), and 1I from Equation (A4). PAGE 155 155 An example of the SAS code used for the Monte Carlo simulation study to determine coverage rates for the four prediction interval procedures, utilizing the above derivations for the FC and FC methods, follows. /*Sample program for Monte Carlo simulation study of prediction intervals*/ /*Generating Data for random effect and error from normal distributions and creating y_ij, observations*/ /*k=3,n=6,sigma(a)_squared=10*/ data normal ; do mc = 1 to 10000; do sigma_a = (10)**.5 ; do i = 1 to 3 ; stda = rannor(17) ; do j = 1 to 6 ; e_ij = rannor (18) ; a = sigma_a*stda ; y_ij = a +e_ij ; output ; end ; end ; end ; end ; run ; /*Creates Satterthwaite prediction limits on a1*/ proc mixed data = normal method=REML noclprint noinfo noitprint; class i ; model y_ij = / ddfm=Satterth ; random i ; estimate 'effect 1'  i 1 0 0 /cl; by mc ; ods listing exclude estimates ; ods listing exclude fitstatistics ; ods listing exclude covparms ; ods output estimates=est1 ; run ; /*Creates KenwardRoger prediction limits on a1*/ /*Outputs KenwardRoger Standard Error for use in FC( ) prediction intervals*/ /*Outputs Covariance Parameter Estimates to use in FC( ) calculations*/ proc mixed data = normal method=REML noclprint noinfo noitprint; class i ; model y_ij = / ddfm=kenwardroger ; PAGE 156 156 random i ; estimate 'effect 1'  i 1 0 0 /cl; by mc ; ods listing exclude estimates ; ods listing exclude fitstatistics ; ods listing exclude covparms ; ods output estimates=est2 ; ods output covparms=cp ; run ; data est1 ; set est1 ; drop Label ; rename tValue=Satt_tval ; rename DF=Satt_df ; rename Probt = Satt_pvl ; rename StdErr=seSATT ; rename upper = satt_uppercl ; rename lower = satt_lowercl ; run ; data est2 ; set est2 ; drop Label estimate; rename StdErr=seKR ; rename tValue=KR_tval ; rename DF=KR_df ; rename Probt = KR_pval ; rename upper = KR_uppercl ; rename lower = KR_lowercl ; run ; proc sort data=cp ; by covparm ; data siga resid out ; set cp ; if covparm='i' then output siga ; else if covparm='Residual' then output resid ; run ; data siga ; set siga ; siga = estimate ; drop covparm estimate ; run ; data resid ; set resid ; resid = estimate ; drop covparm estimate ; run ; /*Isolates a1 for each data set*/ data a ; set normal ; PAGE 157 157 where i = 1 and j = 1 ; run ; data est ; merge est1 est2 a siga resid; by mc ; k=3 ; n=6 ; upshat = resid/(resid + n*siga) ; m1 = resid*(1upshat)*(1+((k1)*upshat))/(n*k*upshat) ; /*elements of inverse information matrix for */ b11 = 2*resid*resid/(k*(n1)) ; bsigsiga = 2*resid*resid/(k*n*(n1)) ; bsiga = 2*resid*resid*(1/(n*n))*((1/(upshat*upshat*(k1)))+(1/(k*(n1)))) ; /*derivatives of M1, parameterization*/ d1 = n*(k1)*siga*siga/(k*(resid+n*siga)*(resid+n*siga)) ; d2 = (resid*(2*n*siga + k*resid) + n*n*siga*siga)/(k*(resid+n*siga)*(resid+n*siga)) ; /*derivatives of 2*trab, parameterization*/ d1_FC2 = d1 + 2*((2*(k*n1)*resid*resid + 4*(k*n1)*n*resid*siga)/(k*k*n*(n1)*(resid+n*siga)*(resid+n*siga))) ; d2_FC2 = d2 2*(2*n*(k*n1)*upshat*upshat/(k*k*n*(n1))) ; /*dfFC2 = FC( ) degrees of freedom*/ dbdFC2 = d1_FC2*(d1_FC2*b11 +d2_FC2*bsigsiga) + d2_FC2*(d1_FC2*bsigsiga +d2_FC2*bsiga) ; dfFC2 = 2*(seKR**2)*(seKR**2)/dbdFC2 ; /*critical tvalue for alpha=.05, FC( ) degrees of freedom*/ tFC025 = quantile('T',.975,dfFC2); /*FC( ) prediction limits*/ FC_phi_uppercl= estimate +tFC025*seKR ; FC_Phi_lowercl = estimate tFC025*seKR ; /*MSEP estimate for parameterization*/ msekrbias = m1 + 2*resid*upshat*(k*n1)/(k*k*n*(n1)) 2*resid*((k*(n1))((k1)*(k1)*upshat*upshat))/(n*k*k*(n1)*(k1)*upshat) ; sqmsekrbias = msekrbias**(1/2) ; /*elements of inverse information matrix for (along with b11)*/ b12 = 2*resid*upshat/(k*(n1)) ; b22 = 2*upshat*upshat*(k*n 1)/(k*(n1)*(k1)) ; /*derivatives of M1, parameterization*/ PAGE 158 158 d1satt = (1upshat)*(1+(k1)*upshat)/(n*k*upshat) ; d2satt = resid*((1k)*upshat*upshat 1)/(n*k*upshat*upshat) ; /*derivative of trab + bias term, parameterization*/ d1_FC3 = d1satt + 2*upshat*(k*n1)/(k*k*n*(n1)) (2*k*(n1) 2*(k1)*(k1)*upshat*upshat)/(n*k*k*(n1)*(k1)*upshat) ; d2_FC3 = d2satt + 2*resid*(k*n1)/(k*k*n*(n1)) + 2*resid*(2*(k1)*(k1)*upshat*upshat + k*(n1) (k1)*(k1)*upshat*upshat)/(n*k*k*(n1)*(k1)*upshat*upshat) ; /*dfFC3 = FC( ) degrees of freedom*/ dbdFC3 = d1_FC3*(d1_FC3*b11 +d2_FC3*b12) + d2_FC3*(d1_FC3*b12 +d2_FC3*b22) ; dfFC3 = 2*msekrbias*msekrbias/dbdFC3 ; /*critical tvalue, alpha = .05, FC( ) degrees of freedom*/ tFC025bias = quantile('T',.975,dfFC3); /*FC( ) prediction limits*/ FC_theta_uppercl = estimate +tFC025bias*sqmsekrbias ; FC_theta_lowercl = estimate tFC025bias*sqmsekrbias ; /*Flags each prediction interval containing a1*/ if satt_uppercl = or satt_lowercl = then satt_cover = ; else if satt_uppercl > a and satt_lowercl < a then satt_cover = 1 ; else satt_cover=0 ; if KR_uppercl = or KR_lowercl = then KR_cover = ; else if KR_uppercl > a and KR_lowercl < a then KR_cover = 1 ; else KR_cover=0 ; if FC_phi_uppercl = or FC_phi_lowercl = then FC_phi_cover = ; else if FC_phi_uppercl > a and FC_phi_lowercl < a then FC_phi_cover = 1 ; else FC_phi_cover=0 ; if FC_theta_uppercl = or FC_theta_lowercl = then FC_theta_cover = ; else if FC_theta_uppercl > a and FC_theta_lowercl < a then FC_theta_cover = 1 ; else FC_theta_cover = 0 ; run ; /*Calculates True Coverage Rates for Each Prediction Interval Method*/ proc means data = est n mean stderr lclm uclm; var satt_cover KR_cover FC_phi_cover FC_theta_cover ; title 'Sigmasq_a=10 k=3 n=6' ; title2 'Simulated true coverage rates' ; title3 'BLUP a1' ; run ; PAGE 159 159 LIST OF REFERENCES Booth, J. G. and Hobert, J. P. (1998), Standard E rrors of Prediction in Generalized Linear Mixed Models, Journal of the American Statistical Association 93, 262272. Corbeil, R. R. and Searle, S. R. (1976), Restr icted Maximum Likelihood Estimation of Variance Components in the Mixed Model, Technometrics 18, 3138. Fai, H. T. and Cornelius, P. L. (1996), Appr oximate Ftests for Multiple Degree of Freedom Hypotheses in Generalized Least Squares Analyses of Unbalan ced SplitPlot Experiments, Journal of Statistical Co mputation and Simulation 54, 363378. Ferguson, T. S. (1967), Mathematical Statistics New York: Academic Press. Giesbrecht, F. G. and Burns, J. C. (1985), TwoStage Analysis Based on a Mixed Model: LargeSample Asymptotic Theory and Small Sample Simulation Results, Biometrics 41, 477486. Harville, D. A. (1976), Extension of the Gaus sMarkov Theorem to Include the Estimation of Random Effects, The Annals of Statistics 4, 384395. Harville, D. A. (1977), Maximum Likelihood A pproaches to Variance Component Estimation and to Related Problems, Journal of the American Statistical Association 72, 320338. Harville, D. A. (1985), Decom position of Prediction error, Journal of the American Statistical Association 80, 132138. Harville, D. A. (1990), BLUP and Beyond, In Advances in Statistica l Methods for Genetic Improvement of Livestock eds. D. Gianola and K. Hammond, 239276. Harville, D. A. and Carriquiry, A. L. (1992), Cla ssical and Bayesian Predic tion as Applied to an Unbalanced Mixed Linear Model, Biometrics 48 ,9871003. Harville, D. A. and Jeske, D. R. (1992), Mean Squared Error of Estimation or Prediction Under General Linear Model, Journal of the American Statistical Association 87, 724731. Henderson, C. R. (1950), Estimation of Genetic Parameters, Annals of Mathematical Statistics 21, 309310. Hulting, F. L. and Harville, D. A. (1991), Some Bayesian and NonBayesian Procedures for the Analysis of Comparative Experiments and for SmallArea Estimation: Computational Aspects, Frequentist Properties, and Relationships, Journal of the American Statistical Association 86, 557568. Jeske, D. R. and Harville, D. A. (1988), Predi ctionInterval Procedures and (FixedEffects) ConfidenceInterval Procedures for Mixed Linear Models, Communications in Statistics A. Theory and Methods 17, 10531087. PAGE 160 160 Johnson, Norman L. and Kotz, Samuel (1970), Distributions in Statisti cs: Continuous Univariate Distributions (Vol. 2), New York: John Wiley. Kackar, R. N. and Harville, D. A. (1981) Unbiasedness of TwoStage Estimation and Prediction Procedures for Mixed Linear Models, Communications in Statistics A. Theory and Methods, 10, 12491261. Kackar, R. N. and Harville, D. A. (1984), Approximations for Standard Errors of Estimators of Fixed and Random Effects in Mixed Linear Models, Journal of the Am erican Statistical Association 79, 853862. Kenward, M. G., and Roger, J. H. (1997), S mall Sample Inference for Fixed Effects from Restricted Maximum Likelihood, Biometrics 53, 983997. Khatri, C. G. and Shah, K. R. (1981), On the U nbiased Estimation of Fixed Effects in a Mixed Model for Growth Curves, Communications in Statistics A. Theory and Methods 10, 401406. Khuri, A. I. (1993), Advanced Calculus with Applications in Statistics New York: John Wiley & Sons. Khuri, A. I. and Sahai, H. (1985), Varian ce Component Analysis: A Selective Literature Survey, International Statistical Review 53, 279300. Lahiri, P. and Rao, J. N. K. (1995), Robust Estimation of Mean Squared Error of Small Area Estimators, Journal of the American Statistical Association 90, 758766. Littell, R. C., Milliken, G. A., Stroup, W. W., Wolfinger, R. D., and Schabenberger, O. (1996), SAS for Mixed Models (2nd ed.), Cary, NC, SAS Institute Inc. Peixoto, J. L., and Harville, D. A. (1985), C omparison of Alternative Predictors Under the Balanced OneWay Random Model, Journal of the American Statistical Association 81, 431436. Prasad, N. G. N. and Rao, J. N. K. (1990), The Estimation of Mean Squa re Error of Small Area Estimators, Journal of the American Statistical Association 85, 163171. Puntanen, S. and Styan, G. P. H. (1989), The Equality of the Ordinary Least Squares Estimator and the Best Linear Unbiased Estimator, The American Statistician 3, 153164. Robinson, D. L. (1987), Estimati on and Use of Variance Components, The Statistician 36, 314. Robinson, G. K. (1991), That BLUP is a G ood Thing: The Estimation of Random Effects, Statistical Science 6, 1551. PAGE 161 161 SAS Institute, Inc. (2003), SAS/STAT Users Guide in SAS OnlineDoc Version 9.1.3, Cary, NC. Satterthwaite, F. E. (1941), Synthesis of Variance, Psychometrika 6, 309316. Satterthwaite, F. E. (1946), An Approximate Distribution of Estimates of Variance Components, Biometrics Bulletin 2, 110114. Searle, S. R. (1971), Linear Models New York: John Wiley & Sons. Searle, S. R. (1987), Linear Models for Unbalanced Data New York: John Wiley & Sons. Searle, S. R., Casella, G. and McCulloch, C. E. (1992), Variance Components New York: John Wiley & Sons. Schaalje, G.B., McBride, J.J. and Fellingham, G.W. (2002), Adequacy of approximations to distributions of test statistics in complex mixed linear models, Journal of Agricultural, Biological and Environmental Statistics 7, 512524. Smith, David W., and Murray, Leigh W. (1984), An Alternative to Ei senharts Model II and Mixed Model in the Case of Negative Variance Estimates, Journal of the American Statistical Association 79, 145151. Stroup, W. and Littell, R. C. (2002), Impact of Variance Component Es timates on Fixed Effect Inference in Unbalanced Mixed Models, Proceedings of the 14th Annual Kansas State University Conference in Applied Statistics in Agriculture 3248. Tuchscherer A., Herrendorfer, G., and Tuchscherer, M. (1998), Evaluatio n of the Best Linear Unbiased Prediction in Mixed Linear Models with Estimated Variance Components by Means of MSE of Prediction and the Genetic Selection Differential, Biometrical Journal 40, 949962. Wolfe, D. A. (1973), Some General Re sults About Uncorrelated Statistics, Journal of the American Statistical Association 68, 10131018. PAGE 162 162 BIOGRAPHICAL SKETCH Jam ie McClave Baldwin is a native of Gainesv ille, Florida. She is the daughter of Dr. James T. McClave and Mary Jay McClave, both of whom hold graduate degrees from the University of Florida. She is married to Ian Ba ldwin and mother to Caryss and Zach Baldwin. She will deliver their thir d child in January, 2008. She is a graduate of Gainesville High School, Vanderbilt University (B. A., mathematics and economics, magna cum laude 1997) in Nashville, Tennessee, and the University of Flor ida (M. Stat., statistics, 1999; Ph.D., December 2007). Jamie began her career in statistics while stil l in high school, as a data entry assistant at Info Tech, Inc. She has held several internsh ips with Info Tech, Inc., allowing her to gain understanding of each step of the statistical analysis process. While at U. F., she worked as a graduate teaching assistant in the Department of St atistics and as a graduate research assistant in IFAS, Statistics. After graduation, Jamie will remain in Gainesville as a Statistical Consultant at Info Tech, Inc. xml version 1.0 encoding UTF8 REPORT xmlns http:www.fcla.edudlsmddaitss xmlns:xsi http:www.w3.org2001XMLSchemainstance xsi:schemaLocation http:www.fcla.edudlsmddaitssdaitssReport.xsd INGEST IEID E20101206_AAAABW INGEST_TIME 20101206T09:44:59Z PACKAGE UFE0021619_00001 AGREEMENT_INFO ACCOUNT UF PROJECT UFDC FILES FILE SIZE 842 DFID F20101206_AAASBG ORIGIN DEPOSITOR PATH baldwin_j_Page_056.txt GLOBAL false PRESERVATION BIT MESSAGE_DIGEST ALGORITHM MD5 b162db0c06ba6490790063d2edd75d62 SHA1 2e565837565819f8a7c7d8fdf66be9ec6c8aa5a8 768 F20101206_AAASAS baldwin_j_Page_039.txt 2ad1694fc5b13d1af8efed9c1cce837d 1ebed3db2a56216d86d2e574c31d80bd0b0a28f4 15692 F20101206_AAARVM baldwin_j_Page_033.pro 157ab832f209e1f2abb788c15edc467b da50f64323405f0961e750c97ba230e004e048c0 21093 F20101206_AAARUX baldwin_j_Page_011.pro 27ce79ac04a7e096268028f9c6699381 60bff6f3a412909311484e5c249a7db56d5a0307 41058 F20101206_AAARWA baldwin_j_Page_049.pro ab8289b3234afbb5cceff090176e10b1 1cae5f827f690427f9d193118954696bdf3c4f03 1141 F20101206_AAASBH baldwin_j_Page_057.txt b731f2bf21edce9c634c083d47844d57 2b292fab37256c8ba32b4ffec1ecbb3b85fa7c92 1544 F20101206_AAASAT baldwin_j_Page_040.txt f581c027944a7daecac37678eb8277e5 29f326480e8a3228b6d45e3fc108a9c07cf41b8d 23497 F20101206_AAARVN baldwin_j_Page_034.pro b2a0d99fbab9f94de1e6756d1f7d8d76 ada3dce51f4c646ab3ba2b7f248873b2890cf4ff 44953 F20101206_AAARUY baldwin_j_Page_012.pro aa9761c0356fb2a4f0ce9e3b57773e40 c308ded01b42987ab5fc88375e6e6edf8a6e64c6 38033 F20101206_AAARWB baldwin_j_Page_051.pro e745e489fdfc7199c30c3f59233ae74d 59875da6440ca5d4d5037d4b8ef5f9f7038b17bf 1668 F20101206_AAASBI baldwin_j_Page_058.txt fee01679de6102c327f8fac40d202a01 87ab8604e5dd5e378150a57681352f2c58753194 1101 F20101206_AAASAU baldwin_j_Page_041.txt 6a0b7a0864432005caabac8347ea33f1 82a7b54c54ddb05d84e2b94e4142ab9924a17451 44328 F20101206_AAARVO baldwin_j_Page_035.pro 417472c75df517ce3a1047c695f3224d 78a5041a80a01729e802e44e8bf77462940be9c6 52661 F20101206_AAARUZ baldwin_j_Page_014.pro 3dcf82903b6c875d171616dd28f37af0 1b2bdff25e0a9b2e00478b377b3c24ce24617f25 19602 F20101206_AAARWC baldwin_j_Page_052.pro 1b6a143eeeed017ace4ccd91921d6976 8e81375dca260fb13ebe22d41a6ae0b514e9d9fd 1849 F20101206_AAASBJ baldwin_j_Page_060.txt 8b47f8a97da072d926fbcb39cdd5ceff 313a0d411147130e08fe54a12c63df6755246769 1749 F20101206_AAASAV baldwin_j_Page_042.txt de9cd6a60cb8606c15b7df2adc8a6d04 a20adcd0b509f690e0099494693edc3ac51f8b60 19537 F20101206_AAARWD baldwin_j_Page_053.pro 4b2c8c146bead844a22d51b824b078f2 9480bcb36392b903bc0e6b09046dfbfaf36020c5 1784 F20101206_AAASBK baldwin_j_Page_061.txt fbc8e4a33a89ee87ffedbc72297bb2ba d9709f2f8079ca2bf0dc5f61901cfd78d8de990a 1485 F20101206_AAASAW baldwin_j_Page_044.txt d200a852383df42e89dc0abf9529c8df 59e2bc06745d78d1caeba696266d89eea7375f22 28457 F20101206_AAARVP baldwin_j_Page_036.pro 09bb1e72585515988acd8f67032ee9ca 9154f85fe5ba575b7f05a3c7ee063aafeaff768f 18793 F20101206_AAARWE baldwin_j_Page_054.pro e7741d984d4960874be1b4abdf63c08c 572e0004641a2396ea81343a5386f0c10ace1fbe 213 F20101206_AAASBL baldwin_j_Page_062.txt 2753e6d0a7eac398b4a21a24b1af10d0 93ec50f7321453e14d9c85b1cd7616311fa51cb2 1498 F20101206_AAASAX baldwin_j_Page_045.txt a153dd6b8e3febede6543c0e8f90cf42 b4aba47b20c83045c0867120c9414670db7e4040 40414 F20101206_AAARVQ baldwin_j_Page_037.pro f3d425f73651908eb6b8d2333dd2a8e7 30d6d62c40c2da28580f0597c07c64274b1a29fc 32026 F20101206_AAARWF baldwin_j_Page_055.pro 7b15e8071c5b8803bbcfbc5ea51499e1 8b9127bb4a3cf1ceed15809ba454fa75b90161eb 1504 F20101206_AAASCA baldwin_j_Page_079.txt 7cb33d07da889af6a81c7847f48ae4ea 7a190f1c6f1d876260a54718e4f973fbea8273d1 1989 F20101206_AAASBM baldwin_j_Page_063.txt 1582f0fe44f2b174fa562ac05a107f97 33cb86d1024608956f72b34c51c69e0fbfeecfdc 811 F20101206_AAASAY baldwin_j_Page_046.txt 60a06f81acc678f9917a2109bdc48c3a 741310f00ec04f278a594bd993b2c584fe5a5d0b 56641 F20101206_AAARVR baldwin_j_Page_038.pro f114d176d26294c3793bc54cd310cce2 1f237c6747887385cefaf10a8ce6a4bbda1baaa7 15806 F20101206_AAARWG baldwin_j_Page_056.pro 580ce2b6f80ade0a47b3decefa90213f a025d5487b10374905170a7f41bc625b71e9e909 1729 F20101206_AAASCB baldwin_j_Page_081.txt 5520470016c8959713c2e8c786c4a582 4aad226215c6c9dc8ffef6a4fb15c8b18597062d 1422 F20101206_AAASBN baldwin_j_Page_065.txt c01c2f9c5c1eafed3310dd0fde018355 fdb22bf0b0372716dc48f4b572f625011f6f83e9 19310 F20101206_AAARVS baldwin_j_Page_039.pro 691a03515a6505c5ed6649c5a3738a24 fd7862bb0884a192d0f643393f3ccb4cd74fcfe9 24323 F20101206_AAARWH baldwin_j_Page_057.pro 99f2f83edf4519a3c5b62db5feba8c60 dc2426ed7861166fb14c69f1ba3d921a9fb3cdf6 1742 F20101206_AAASCC baldwin_j_Page_082.txt 3c5fe2e66db4a1a41acecbc383c6bde3 7eb67f579dcef58a3ed2e2f28d4e0e0794ef9d93 1301 F20101206_AAASBO baldwin_j_Page_066.txt c1c193c24ba8c133eb5a2d5b324ca897 07683b7bc9d453320ab3a77cdac28e3572c57ec3 1374 F20101206_AAASAZ baldwin_j_Page_047.txt 98ed8ea7a85aa02c3031ed34665f65a3 42199bcac5405a8ca1d690d57a869e35c9885023 32555 F20101206_AAARVT baldwin_j_Page_040.pro 6de9d81b1266182f1f615a26273e7eca 79b38abaee4bae9832f3dc2226adcb538bf04752 34006 F20101206_AAARWI baldwin_j_Page_059.pro a1068b4fef8aa2234748097581e10ad2 dd262b4a761e8af6037a7a8d9be6686d502d0845 1874 F20101206_AAASCD baldwin_j_Page_083.txt a220da7977c29166878c16dae65b6ef5 fed59a8febf14f64dc10ead652ff41962f9ce999 1078 F20101206_AAASBP baldwin_j_Page_067.txt 582a064c8630c9f8f41200b9e6187540 a4b3ab9d28b89346bae08d7c5e123461dd1e9693 41048 F20101206_AAARVU baldwin_j_Page_042.pro 4cc214372eb78903b2bfe3d85ae2216c 8501483f928e07d55bbb0ecf293ced546c3cccaa 44133 F20101206_AAARWJ baldwin_j_Page_060.pro 196dc11a668de567df9a8c505de8324d 0a092fd95d7f6e729e29926ee22cf9615ec93e35 1688 F20101206_AAASCE baldwin_j_Page_085.txt b02c00322e71494a5ee4a2081e2f3973 95ea42e8dd1550401ded601599b430ae5ab147ce 1692 F20101206_AAASBQ baldwin_j_Page_069.txt c63bb8994850cfb45c778593638f2869 09ed44061bdb2e671adf974353ed326424127d3a 29145 F20101206_AAARVV baldwin_j_Page_044.pro 8ee3e5464aac5e7ec353ee31bf555516 d16690e2fea2d56df2bf222ed9a5d7292004f0ca 5258 F20101206_AAARWK baldwin_j_Page_062.pro d0405840ce451675e46d6a42e7848f73 a063023c623ef74b7dc0d42afbea8ab7863999bf 1647 F20101206_AAASCF baldwin_j_Page_087.txt bc098c8c75867d1e79387c466e9a88f5 795fe39615a6c12e58be37f6ed1deaba742e3431 1912 F20101206_AAASBR baldwin_j_Page_070.txt 495e4728728fdc2b177b65262ab45b4b ec37def725aa453c93a397ad89725c43d0246e26 32669 F20101206_AAARVW baldwin_j_Page_045.pro 7580e0a08a7203535eea3b0baf452cf7 319465bdd803ede66631ac0b233c930f9830bec9 47086 F20101206_AAARWL baldwin_j_Page_063.pro e9de6f55945a97e89b95b896a85896be b45dc52011e65a9e81bd76cdddf6619e978012ab 1867 F20101206_AAASCG baldwin_j_Page_090.txt a3cfc8c55fe2f723a77373b6d1a72e15 102c86a615ca5e3a148971bd16bfb48bdcbbba7b 1770 F20101206_AAASBS baldwin_j_Page_071.txt 217a2770c4d9fa487c85b7a5b7606290 53d677804e87234e27514a174ccfb3316b519fa5 19312 F20101206_AAARVX baldwin_j_Page_046.pro 03d848801b1a2d916284ae6cdb3ff31a 9fc59954a00e13db5fc513a579f93684e7bff876 24904 F20101206_AAARXA baldwin_j_Page_080.pro fb5c132a520570f536486d8295b09381 3a68e241bd6ab2db2fca8cd976efe91330e84c7f 33112 F20101206_AAARWM baldwin_j_Page_064.pro bb62facfa8fe01e2d53f32c1d0dd401a ba62c2021ce885c3c62aac57102ddcba9e57ce80 F20101206_AAASCH baldwin_j_Page_093.txt 2be1cdcac28fb008e27796e26ca18371 170c036c48d8a07c09cc8c2da50b7f61e294fe5e 1328 F20101206_AAASBT baldwin_j_Page_072.txt 15077f56d98ac02d72f2501342a06b18 6654d89e3fed94b44d64ff0aa19f345468b8723b 29374 F20101206_AAARVY baldwin_j_Page_047.pro 0c62c7043d36d2c4b2de37d4c55d208e 8c22218c8f2d880e8c0f20e8126dda089e016a0e 40348 F20101206_AAARXB baldwin_j_Page_081.pro 927c8229425524595d060a0d607ca4ee cc5a5c101c9094b78126337d8c004b7105226f90 21508 F20101206_AAARWN baldwin_j_Page_065.pro 618497fac8b178ac985b7b9ff7aa11a1 8fcb93c42e1447cd2c0924f135e9f6dece3448b8 1320 F20101206_AAASCI baldwin_j_Page_094.txt a1605e5b9d7c5ae69caf910bda398f31 1d905b7d9ce9e0fbaad91efba78ab288ebb57838 1508 F20101206_AAASBU baldwin_j_Page_073.txt 9168e0a0f52ff0c914687c50749e875d 01c600fade7a971f2b624c25ba589c14d6aa1e21 20097 F20101206_AAARVZ baldwin_j_Page_048.pro c6c477a645ddc4ca5ce8009426d99e11 88954f3dfc64d9859e01d3092322b7691351807f 44008 F20101206_AAARXC baldwin_j_Page_082.pro edd7959bac28e944346f1a1a69885b91 628233c4adeb733a1ee03f1953adbc72ea99c185 21674 F20101206_AAARWO baldwin_j_Page_066.pro 7be9fead9d80c883d35c0fea210b955d 491198e17deb8bbfc6db86e56fab970bc5283acf 1515 F20101206_AAASCJ baldwin_j_Page_095.txt 1070bcbe42c5a045e769ce681e0eb905 584e00cf57b93c42b6fed58d852e94e4e48e3276 1596 F20101206_AAASBV baldwin_j_Page_074.txt bfd603435ff98589694298d53b9afc14 3daef551f86a3dee58bf4c55157913819fcad20f 46061 F20101206_AAARXD baldwin_j_Page_083.pro cd91217e3d0edfc2d079da95a62d7cb7 e17c59eea7ec94bc06a7b8071f6d044da6e87ea3 18824 F20101206_AAARWP baldwin_j_Page_067.pro 57f2592dd43ee724cf52333da29f4ae0 7bdabd27b26c9044a5848fb7847fa08fad57723e 1840 F20101206_AAASCK baldwin_j_Page_096.txt bca1365ff6ed1ec168ab26fbbc2e2a04 8d8869e336af3e63a7d46783f39edeb63f671b2e F20101206_AAASBW baldwin_j_Page_075.txt ab224831b5a754197af71b0c36c297f9 b9ca89dc748c1ec989964f7322a874443873dce8 37830 F20101206_AAARXE baldwin_j_Page_084.pro f13bb5300caac439ff6722a317d5d534 9f4405f3947efb2f7d8fdd3e009e44a828778f10 1614 F20101206_AAASCL baldwin_j_Page_097.txt 160ecfd3fbf03bdc4561392a4ed8899c 250c6ee3562528bcde7f70460a6314dad3c777e9 1220 F20101206_AAASBX baldwin_j_Page_076.txt d8679128a2cc14cdd0f25d07da12cd28 e0fc2c868282544dd1ed2aa079b67fea85565064 38017 F20101206_AAARXF baldwin_j_Page_085.pro edf447638de8babe12fcc7c0d45be28e d459fe1b893f5f539f81c74dc5d4429dda43c186 36062 F20101206_AAARWQ baldwin_j_Page_068.pro 62265fdabb94dd53c6a3d9c1b8b8459e 2ca2f3ceb626188a54b699a685f10d2264b5c91e 599 F20101206_AAASDA baldwin_j_Page_116.txt 8ba3cb24917390f566c751f1a2570c98 3382dd73c0e5fe4edd2de48d7e8b75ce6a45c850 F20101206_AAASCM baldwin_j_Page_098.txt 771159e512c30cfc4402644ac06db547 f590157575f280bd9bf75a6603bc3fa9de385c3f 1699 F20101206_AAASBY baldwin_j_Page_077.txt e9186cf62b8daed6c80bff37cc595b95 d21b93d42cea7410c339d816023d9131ecba6034 19915 F20101206_AAARXG baldwin_j_Page_086.pro d82230b8b8432c06ef2a71719930c795 c0eb17ca48eff648c7ad5e6c7377b6a040f1f070 36052 F20101206_AAARWR baldwin_j_Page_069.pro 037b8396b2b7d70af56676af99844ef9 489a723ae37ab8ff0a3d36b0bba5da5fe4bead30 666 F20101206_AAASDB baldwin_j_Page_118.txt fab0274490ac08af5b0fc98000a14004 1809e4616fba400b80d67ec276b976c8f8ae7ffa 1831 F20101206_AAASCN baldwin_j_Page_101.txt f6d1fbbd58af9bfd344e5fc59350e6ea d214fdee9dfa624967a4f0afae0f3befa5cfb148 1579 F20101206_AAASBZ baldwin_j_Page_078.txt e210d314d16cdd4a1c51b7608b86f765 e01d93d850b14b2548e77837b609aeba65872179 37977 F20101206_AAARXH baldwin_j_Page_090.pro ef7bf21acdc0b13e6e43402f057fb7c9 7fe0047bcfa1492eaa309f13c878cea781e6d5eb 43270 F20101206_AAARWS baldwin_j_Page_070.pro abbe378169ff17ee7028d842d08de668 34e88d887a839f0414fa04ab81332df42fc68327 856 F20101206_AAASDC baldwin_j_Page_119.txt b25342cd79cb2d08d2bba505a58e5e62 028c83d54cd255572fae482d72dcf5c12995eeb8 1746 F20101206_AAASCO baldwin_j_Page_104.txt 3dff4fe603fa12fbc65c828c16c766d3 d795de606727d45a65b82595e303ff91b18fc546 41756 F20101206_AAARXI baldwin_j_Page_091.pro 1f2180cd69ca56cb97f40a76ca6f98f3 2a9bb874ff5b24ae26e34d37529f0f0b319e9410 26623 F20101206_AAARWT baldwin_j_Page_072.pro a1e28c9c4fbba685e2856005b92cc14c 767eb04a1ecec2b6041556eea98ecc70f2d0ee69 4806 F20101206_AAARAA baldwin_j_Page_078thm.jpg 8c0ad9a51f8461a470fdd5613465fece 72f8f17600660356ff4d6e7c3260af8b380298f0 260 F20101206_AAASDD baldwin_j_Page_120.txt 2078e93cc3f01bcaf17ebb7593504385 9bc7455662cff287ec9bd23d952a3d5cc4e38449 1960 F20101206_AAASCP baldwin_j_Page_105.txt fc5c305325cec52d7135c12e2b7b400c 8edc73518cffb470ecd90f4505936a5158428144 31001 F20101206_AAARXJ baldwin_j_Page_093.pro b09c0b69f598f926ea3d8f9b02857453 2f993a1bab89a29dd324a304dc1fbac2b3cf6b81 31019 F20101206_AAARWU baldwin_j_Page_073.pro 73f5db68234e1ec80f2397743ab5282c aab5fec8320c0f9d1fe495419d2d56f66f74d92b 63340 F20101206_AAARAB baldwin_j_Page_074.jpg df2ffd5c3961c12b3e25e05a5c49ed5a 751b776a5d6833e001720203dbfa047d3949ebe6 457 F20101206_AAASDE baldwin_j_Page_121.txt 7276db36f394bd899151f92b3da32baf 3d44655d2767878260e1f811340d173a037007a1 1950 F20101206_AAASCQ baldwin_j_Page_106.txt 93177f95215ee6c1ade5e769c4e0eddf 237040fd78fdcda6d6a2efdce052a0537890aecd 38455 F20101206_AAARXK baldwin_j_Page_095.pro f669488ec7bbead185e6dfc18bd9a6e2 20ed6f7265b14ed908121a124a7c2310c670de91 41329 F20101206_AAARWV baldwin_j_Page_075.pro 2a5cf762a04d61facbea08b14c9229a9 90d24d91fbe01262c2fe9af5dfa96343633b1a6e 1248 F20101206_AAARAC baldwin_j_Page_034.txt 2c15f85be4d5f091a93fde7f9aaac047 e034d67549723c3f7a4946bea80be63a0b3bc6d3 681 F20101206_AAASDF baldwin_j_Page_122.txt 97862457ccf15faf326813e732298f6c 179e8395018a5d2dfd979ccc7e45653dc2fa1e38 F20101206_AAASCR baldwin_j_Page_107.txt e3907de47595433424725b85f2020600 a81a78dbe8db545b1d9a1722280969dd2880b204 37727 F20101206_AAARXL baldwin_j_Page_096.pro ef94cc4a6f6a495a39f4b4770a4a5889 f20061ebf4abdab7898476bebd3e357e63811c63 25837 F20101206_AAARWW baldwin_j_Page_076.pro c1d7b560c39c5cf28b6d4ec79fdc86fe 3aa06a36e68f1a5276bfe9412414451eb5812554 5209 F20101206_AAARAD baldwin_j_Page_079thm.jpg f710f0e39f9c8f77cfa62e85d6437de0 fb0093be346b4f4dceaa6e3ae1d91fd8035a486b 540 F20101206_AAASDG baldwin_j_Page_123.txt 3a8f0e318c668491c878fb482ede58b8 71b6bdc5ac7c049f9792b516871ccf449686a54d 1321 F20101206_AAASCS baldwin_j_Page_108.txt 65b3175d79be04378f7e0bb33c1d1593 5f88f3d72fa1c86961376204f956c40377ac82dc 35218 F20101206_AAARXM baldwin_j_Page_097.pro 5b66380f60df7d934a23d12e1ac2c047 2b6fbaf42059cb370aacaa52786a62601a430e3d 30596 F20101206_AAARWX baldwin_j_Page_077.pro 7a3beff6f0e0d55b00c26bd720e13a36 ac424dfdca925a79aa6c1333909b079c056eadc2 819 F20101206_AAARAE baldwin_j_Page_127.txt f962c8cc27b4f29b40ece1f47ba5b12d 5fbd8d532a8e213daaeaeaba765c5762e29dd57a 8662 F20101206_AAARYA baldwin_j_Page_112.pro 91a0adfd301d96c858931f0f0ddfb55e 8646602123feb7f8e5b4da7aefb2b2318477d147 359 F20101206_AAASDH baldwin_j_Page_124.txt c79d3ebc00c9c0716821d1171d8f0f9f 34d51af7dee069193feb70bd2a042b0889016674 673 F20101206_AAASCT baldwin_j_Page_109.txt 3944a44a39b76df3de31e29742f7f44e 05de4688e2c66e6baa11b38b20c084ef51f90e58 41505 F20101206_AAARXN baldwin_j_Page_099.pro 66fabdd18d5d3ceaad266cb9f02ed2cb d8214a124ce5b1a5e5d719f0fa3267189e3c51a3 26407 F20101206_AAARWY baldwin_j_Page_078.pro 35d36f9b6fa4c1bc652a0dfc5ef7e8b9 666b43cf7a6071a95163996223d21fb59b4230f7 6410 F20101206_AAARAF baldwin_j_Page_081thm.jpg fa48c2a6fa145bea56a56c31974d5659 65245770cf9b798f959749929f01b6e6ddd901a9 14933 F20101206_AAARYB baldwin_j_Page_114.pro 3ffd916ccafdabf464e0b5b7e8547ac5 dff837631f9808800f62ed7465e80b81dfc98838 317 F20101206_AAASDI baldwin_j_Page_129.txt 25e01ec106a4da56345add919a0f2a12 57c1ff7f4c7ccc12f30fa8ae5803abec28cf18f8 390 F20101206_AAASCU baldwin_j_Page_110.txt f4b9fa6d5ebebf00291d875d36540a0c 338ef6719296bd1b37cff66c58552f54e00994e4 43095 F20101206_AAARXO baldwin_j_Page_100.pro bf40eacb586ea29a490c37dd4366b7b7 8327b264708527721b74cae02f409096b4b29553 29100 F20101206_AAARWZ baldwin_j_Page_079.pro 651d7fec1bc650908404bc48dd439f1f 3f77340dcaa759ec716dcca59b0a372e18af6b71 1800 F20101206_AAARAG baldwin_j_Page_017.txt c67d782301302158493a39fb5aa3aca1 d8065bb9fd10bd74878076800867b4c40b430104 10800 F20101206_AAARYC baldwin_j_Page_115.pro 2db18329e58afad42fd7d0809e4c4606 682b7d455d7496eb5c4e668f489c33603ab3c54a 195 F20101206_AAASDJ baldwin_j_Page_130.txt 528451591761d43dd286bf0101c40c4e aa745f985ad4b3cdcfdad22ce56b8c8fef323b57 321 F20101206_AAASCV baldwin_j_Page_111.txt 026e51e129d9881ba11b43dae58dda1d b6147234b544d010ebf9b1cc04bafcad715a9249 44238 F20101206_AAARXP baldwin_j_Page_101.pro c41bcaec5a7f8cd10aaa9830bbe3da48 cc21af9cb936f778bf61c46b7e866d4e2dc0948c 85205 F20101206_AAARAH baldwin_j_Page_007.jpg 3d60b72f675e648eb1dc1de81a9d9129 11875d451e20554ef0e99b96b689f6df66b179af 10910 F20101206_AAARYD baldwin_j_Page_117.pro 88e47ba1a01ec70966ff846ffaefdbe4 a8b92d5ffe70074da7f08f5aee9349a23cfb27fa 229 F20101206_AAASDK baldwin_j_Page_131.txt 0eecc2ee25671e78cde6102ca2e7b441 2ef1bf790cd4062c8f08ef89d4d2285d2c863909 468 F20101206_AAASCW baldwin_j_Page_112.txt 6c51030b0769518089c91426b7f9f4f0 fd1601c40bcaa9d618ba85b51b3b9937e0af8149 35258 F20101206_AAARXQ baldwin_j_Page_102.pro 31c0bf12cde39d79dc0787af2cfa4b66 3a887181ba82ff1a54eb843c3a976abe297fbcc0 1115 F20101206_AAARAI baldwin_j_Page_088.txt 135062fd9853859db01e0462ae414ad3 7d894594627ed9c39aa2a85f7ddfc969a0d40a95 12179 F20101206_AAARYE baldwin_j_Page_118.pro d7c744a449e7b229d15486f9e2714949 4a2795b9be146f03f05598fbbb59de6e63f73ab2 313 F20101206_AAASDL baldwin_j_Page_132.txt 25eff9783dd9cce9764424a1b586a151 2166e4acee6ffc7857ad25a095e665c15b30306e 300 F20101206_AAASCX baldwin_j_Page_113.txt d700a09334a6d31c88c14133af038ee2 0f2c637e2a2eebc2175f7c90369afe384d43f15b 25271604 F20101206_AAARAJ baldwin_j_Page_039.tif 4123def7d1ef8f1cecba46e2cdf83b5d e70b945a1312967a215c9564b821ed445fdd87c7 13031 F20101206_AAARYF baldwin_j_Page_119.pro 6593e6d57c4fbb8b071d1da1da8a4df1 dca3ed404ea093901743d04562a369bf3c25b835 203 F20101206_AAASEA baldwin_j_Page_151.txt f55942cc431c048847830cf4f9b7cfdd f79ef3eb4e6535935e87af05dd1d72a0aea9a47e 368 F20101206_AAASDM baldwin_j_Page_133.txt 8d184da07fab243becc2026154f4b477 67b1de2f6afb34fd5735f2b2eae8d07bbd0b1e96 964 F20101206_AAASCY baldwin_j_Page_114.txt 5479b4aeb70c744744d13a48f3044c1e 2c472c51b0ded9471cf59965204181c720c15f19 47555 F20101206_AAARXR baldwin_j_Page_103.pro 28b3e7da979dd6f897a92d5f34f409fb e87039f8ffc0e7aa986df2b3cba89a45c45b4634 2934 F20101206_AAARAK baldwin_j_Page_123thm.jpg b9ef8a0c3cfd24e371b6c0f4fc9192b0 17040a8810f9ed412f7237de594de9c5b51c910c 5150 F20101206_AAARYG baldwin_j_Page_120.pro 1531c30f8699d489a2ad98c3511129bf 64a80ce3d34fc3eca805f9d69f69176631091fa0 F20101206_AAASEB baldwin_j_Page_152.txt 047156126a0ebff1a832c0221498c5a0 bcf404c7120d6f58d76312816d707258ab2ccb10 411 F20101206_AAASDN baldwin_j_Page_134.txt 13881bb270f306e2906ca7c15f15b38a 9768957313c491319c28abcd73ffa340834b554c 610 F20101206_AAASCZ baldwin_j_Page_115.txt a9d6044d11b5949e312719fdf8c2e96a 3f37379e598a1125751dec889f2f363dde189cf2 44343 F20101206_AAARXS baldwin_j_Page_104.pro 4384e44f62396b9da70ed39a3fe8896e cfcbdddf1f7ed5688b2f4de89f8272dabdc5886e 22567 F20101206_AAARAL baldwin_j_Page_042.QC.jpg 03c29fc4d947fae1064e52426e2df01b fc4ad0b831c9928781da3827aff5a0f5b84c1c34 5531 F20101206_AAARYH baldwin_j_Page_121.pro 1674324a5165f0a362a8cfcd88bbedcc 5126be43cc87caa9e3f5d2c493e37f5fa1fcf338 1555 F20101206_AAASEC baldwin_j_Page_153.txt 31c8d4c68a447fb9cb1d65a1eb92cf16 8719e5d2448d5ca2d0b86144215bb22c31ae096b F20101206_AAASDO baldwin_j_Page_135.txt a4dc871b94e2334f786ab954af66b39c f05289ed66fccf972600455f26500317f647d89f 49699 F20101206_AAARXT baldwin_j_Page_105.pro 7989bbbcedf413db1e0df4d77372d402 5506d65343e9f0cba54c1dceb513a51be642fa01 F20101206_AAARBA baldwin_j_Page_137.tif ffff27c19663527d41d6148fdf9d8ad0 d99b7a3acc22ca26ab7fe5e6ba91ef52607ec8f6 877481 F20101206_AAARAM baldwin_j_Page_027.jp2 67ce761182bbc7bb18fd8c020951b840 eca5ee6ee31d6c60470eb95a12d46775161388f9 12326 F20101206_AAARYI baldwin_j_Page_122.pro 43da52dfa75d738873d49b896ee0bc58 16a28d782a67a283be9ce7d65217154af3704a78 1117 F20101206_AAASED baldwin_j_Page_154.txt 233a66a6a2b40e86212316aa9d9438cb a05e6a7d6bd007073cdb1875113c17af7b94c33e 2178 F20101206_AAASDP baldwin_j_Page_137.txt 26fdc93f02f0a24aeae734072c8397b2 5937ceba4c652ecbd5db7092a3745dd7033e18e3 49430 F20101206_AAARXU baldwin_j_Page_106.pro 0331f0f9c18c2c183bc602be3b69a6ed 6c6d21353422f7603b1e6d46647b88beecd666db 292 F20101206_AAARBB baldwin_j_Page_125.txt bc0963e0c33eaf005d5232c18e2ef877 a35e91d5bc9e3a04361fdb8d4e3e6a6f9b8de306 629 F20101206_AAARAN baldwin_j_Page_126.txt 841e9848d7f07016e5543a4018f95bc8 3c8ecec9c534c1b73504de86a099c97cce71e553 9921 F20101206_AAARYJ baldwin_j_Page_123.pro 129362bcc68251414f2179ee8e1f1e5b 41786b3abae05174b8cb6a2c250524622e0a7f55 1095 F20101206_AAASEE baldwin_j_Page_156.txt 58132221d7beff9cc8774e420a3356aa 66b294375aa38a4c714bd5f6f1cb756b801468cd 1952 F20101206_AAASDQ baldwin_j_Page_138.txt a03f095a2c95f42fed24d908ff9ddef7 1141d0be7e1dfd75780a30e40b39568663240a6b 47244 F20101206_AAARXV baldwin_j_Page_107.pro 7db2e409530340a95901d369f6679816 7e7dd8dd68ad55267f6a8a92246f9d7e18676e86 3700 F20101206_AAARBC baldwin_j_Page_003.QC.jpg c345751b7c0f4abe196a87119f9ee28a 8a45643d49e729cfeb6f0f36012d5d0cd9e74a48 48164 F20101206_AAARAO baldwin_j_Page_077.jpg 557648cc83ed0d82a0ca6b188d632387 c41a1d485d920c8d4d86f0aec2e09dd4c0da9713 6403 F20101206_AAARYK baldwin_j_Page_124.pro 423c052c6d739a48f2fe082ff15d9e8b 10bd7058ffd9752da5060b01af3b43ed5e5b1704 1938 F20101206_AAASEF baldwin_j_Page_158.txt 038d7696c3b4123add386f446ad74bd0 21bdbd2d20cd0b213b7aab37ebad2f2c40d0a5c0 1967 F20101206_AAASDR baldwin_j_Page_139.txt b6c0448b5bfeb3bd0284ad58fb99a96d 95c653a995d29dfaddc94db83d3bf37688c1975a 29385 F20101206_AAARXW baldwin_j_Page_108.pro 95e2abc615791d37d66e8f2e23143fb5 1fcb571d33463bb3f915c3cdffce0e7f72e1ca5d 20035 F20101206_AAARBD baldwin_j_Page_084.QC.jpg d624c132ab4fe0b231b39bfbfbb18de6 9bc0388a28dfe89fcfbfad5273245af983fb6cc3 43901 F20101206_AAARAP baldwin_j_Page_017.pro 18b211f06d3f017493bf7ab729a75a01 b39b93e2dce8f1da0a4ddbe9b461a78941948f14 3822 F20101206_AAARYL baldwin_j_Page_125.pro 59d3d9cb3be48969e591c03696f77df1 b3f7a63783870441500e9e536f66a0e87e042081 2472 F20101206_AAASEG baldwin_j_Page_159.txt 8252b4a85afa811ef300133213ab6cb9 c6076b62dd0428e3a2446c43c1563527fc5e5b66 2110 F20101206_AAASDS baldwin_j_Page_140.txt 9dda48fbbc6845ef34f6f3f2b0445f8f d1e150506b60e6e2f258cd5d785e9ded3f9d0e5c 10576 F20101206_AAARXX baldwin_j_Page_109.pro b24dfdc0aa9de9ea5f2df07f62def6e2 d0b23f6a0e9bddd295365f188671eee49a3c387c 1663 F20101206_AAARBE baldwin_j_Page_144.txt 6c0b88312ca60719e33527e9d4b7e11c 9e3743947e456d628ba5b34aa7e2348702ae2353 6404 F20101206_AAARAQ baldwin_j_Page_109.QC.jpg 174485c90daf9543425035e4002e2278 1e82d806d75b21ab75ce2f603c8cc7c36c186ed0 36607 F20101206_AAARZA baldwin_j_Page_141.pro 32c32d286c1453bcc0a030559f9a864e 7ca1c0dd6c4a3f98f7463cc7e416afc6c59d6f49 10060 F20101206_AAARYM baldwin_j_Page_126.pro 1996ee7ef2ce4e39cbe7ea7af9bc1c1e 6c56d01306885ff6def92cc7d2d6802bd193e6a5 2275 F20101206_AAASEH baldwin_j_Page_160.txt e9381517abee8cddb9dc7f07b022a9e7 5db9ae6131164662c2af8abbc8d9f20960f0b14a 1774 F20101206_AAASDT baldwin_j_Page_141.txt 387273351c1aa79e0fbb8ee3d6ee4e8e 2adb5939ef689f02d63caf22ed85fd72f715608b 7685 F20101206_AAARXY baldwin_j_Page_110.pro 3fccf14becb40adda53d4cca4f5fd5cd e36ee9fb4c6783e28fd547cd09741b83f8c9fffc 4680 F20101206_AAARBF baldwin_j_Page_088thm.jpg 907b97f9f8dba204bea4d3424b91b6cb 85e9e21475d441ca2748cc42aeb15ffa47106f7e 1880 F20101206_AAARAR baldwin_j_Page_103.txt c5eecf6fff8c7a4eb36c651e52438eff 63387ff2c4c70aa09ab6d8c67976987f89bf248d 19663 F20101206_AAARZB baldwin_j_Page_143.pro 61781f76cef419f8189bba660eaa359d 0b74295f2a18c4be233debc35004685fbb71b9f0 10127 F20101206_AAARYN baldwin_j_Page_127.pro a1b116faa441d63ce2beb17b7ec42e2f 2f5a26f4402c2580067881180523fca47184b8ca 1724 F20101206_AAASEI baldwin_j_Page_161.txt 0ed633a6a784155be89084a19b5ab48d a506856f63fa95da53b33094fc357abbd8a4c254 1653 F20101206_AAASDU baldwin_j_Page_142.txt b0f47e1ef7d55255705a8afd9961dbff 7fa84252f8695a071ac65bfee3c247b871f33d69 7085 F20101206_AAARXZ baldwin_j_Page_111.pro d4041aa8db548018aaea81959cc5d4ad 829e99f0fbce68ec59a0818f6bfdfca14f06a7e5 1947 F20101206_AAARBG baldwin_j_Page_016.txt 2fdf44cb343af5027fb6168b6da6a835 97f67abb73433ac915343dddf792eb9b66289a69 25078 F20101206_AAARAS baldwin_j_Page_106.QC.jpg e6545a8b1d1804b7fb7eb6ff59888831 4fb35050de92df0a243031c8f78ba431861f0476 36794 F20101206_AAARZC baldwin_j_Page_145.pro 543ecb435ab2ee006319ec730b5a468e 7375eb9d99a180feb7c0d3153b7205f931d21d9f 5441 F20101206_AAARYO baldwin_j_Page_128.pro fb1a82b71d8cbbd7037f6f7424b4a213 75ceba72a8132217cc0246465f1db6fbe22c9948 1164 F20101206_AAASEJ baldwin_j_Page_162.txt 3e2869cea54d647cbadccc2c0ab7e051 fd667cf609b70575cd7fde6936760fedd1983a09 1319 F20101206_AAASDV baldwin_j_Page_143.txt 26dead20cc28fdb6cdc5763f4b348f7b b306d3a78cb793a7a996562daa27769dae973afa 1072 F20101206_AAARBH baldwin_j_Page_052.txt 26d01225a169f342fa9e4336dba13448 163d975253efd820dbb6f14dbd93095fc95f8972 460555 F20101206_AAARAT baldwin_j_Page_065.jp2 bd162bab3f57e2a2bcadf29b97568f30 2d49b840abc0fa29f4ad199e472d5fda56dca4c9 54990 F20101206_AAARZD baldwin_j_Page_149.pro 7f9bdae5cc9c405bfb33ffa5f7da7f08 d8b76d136a376cf384be297ac4539d79b3ad5d96 4539 F20101206_AAARYP baldwin_j_Page_129.pro 252ea2078cbcc3b57aca1ed71d808570 f255f6fb290b9d3efed5f9c597ac96e403c62656 2383 F20101206_AAASEK baldwin_j_Page_001thm.jpg ed9dc699f7371744550ef8967805a426 8c89d7cf1274325512681cc536cdb8429431e555 1891 F20101206_AAASDW baldwin_j_Page_146.txt d7c760cb88e81064ce71f4f590fc97b0 0e216d624411595a25d620e34dca9b91d719b16c 1406 F20101206_AAARBI baldwin_j_Page_080.txt 7daf010d8e20f558434d87eb02eaefe2 afa48724f02f54804192479647fa037dde62b3e0 978877 F20101206_AAARAU baldwin_j_Page_100.jp2 648ae380cb899bd5847ecd5315432923 49093ae4ecefefc25a607aeefc9b4210c68739b9 53959 F20101206_AAARZE baldwin_j_Page_150.pro 86d4b1ad690d6c48eb926d4de20fb408 59e8e8ab320124fe87df1696305df441b6253de8 3337 F20101206_AAARYQ baldwin_j_Page_130.pro 5884b27485bdea9111445dc2b46ed159 1f54c2b267e33154121f0f46a931fc0b37e04e7a 1074325 F20101206_AAASEL baldwin_j.pdf ddc25e68d20c537a789bbe1ffbd13e9b a338b30f944bfa3985344ec9b7f83b435c1e2515 1934 F20101206_AAASDX baldwin_j_Page_147.txt 685bd5c90cc34f5b85b1fe518225f535 de43393674230830af90c0c951a33d32f738af26 118856 F20101206_AAARBJ baldwin_j_Page_151.jp2 ac187a69778b3a9e1f5cc70b26c85682 8a909b14dabfd55c38f4574ad1f0976df49b1b55 437740 F20101206_AAARAV baldwin_j_Page_114.jp2 ad6d03235d0b97aa383a63762af63873 8a335e019314ac0c183b29fe751457f870e13131 5061 F20101206_AAARZF baldwin_j_Page_151.pro 8fe31727d932a4487fc90fbcdd426c77 7e3812fb03a86dd35428535fe9f3a4ff87509a10 5407 F20101206_AAARYR baldwin_j_Page_132.pro bc48c0fb94a1f356a8dbd5134164fbe2 1b0a2e3c35b679eb7b42769b0e9e251e2d04d361 26870 F20101206_AAASFA baldwin_j_Page_140.QC.jpg f256414d4e9302dd7bd03b965cadb01a 50d31affbdf0d61eeff302c64deb718bc6bad79e 2564 F20101206_AAASEM baldwin_j_Page_124thm.jpg b6e0a940bcc6cc20abd022bdbeb74884 2d42f96a60c616bc1fa45782df9e07d2a4e2b778 2008 F20101206_AAASDY baldwin_j_Page_148.txt 8b14c0dc37b365aad5da4c0b4403d610 c685999c296baa35fedfc9a9ea574c27c8e8db6f F20101206_AAARBK baldwin_j_Page_093.tif 03bf20940527fc1850a322b6adab68f6 9c5932ff2dc1efa99755c2a45e50facfe74ffbfc 28587 F20101206_AAARAW baldwin_j_Page_094.pro b0876922f6368d249a3eddd51eb16103 5a353140ca555c94f9d972081fa49ef761ff5030 28424 F20101206_AAARZG baldwin_j_Page_152.pro b822983ac3ced8a87a89aeda2bf7ec18 0efa3fabfcd2d18b59e19618c8b3082c5846ff52 19073 F20101206_AAASFB baldwin_j_Page_064.QC.jpg 2b0c6b28ddf17cd458e280cf86fd6bea b7a20ff75c9752f82de67374e25e1e16d99e447c 6947 F20101206_AAASEN baldwin_j_Page_029thm.jpg 2c740ba605d8480c4c1ba6f38b36dd83 5b57c99deccac0b6e186ea9a5fbf14047aa0cd5d 2159 F20101206_AAASDZ baldwin_j_Page_149.txt 68cc619fb4c8fc2b7bbc5ebced5c077c a8b0d02a86f89367a4cadcdfde093ffbed7ddc13 46430 F20101206_AAARBL baldwin_j_Page_034.jpg e99cd9945bfb3e69566eebe50ff3ab28 76f1403e17c1b6e60c36fa68c75dcbdbf5520ab3 22569 F20101206_AAARAX baldwin_j_Page_088.pro 10664b29134c49e42f88b4fbff1500ff 76ee6990ca250c1f7e75baf770f2f0004825b737 27361 F20101206_AAARZH baldwin_j_Page_153.pro 40368b2b2aa5ff7eb79e5171fe547bd6 81a3261391a95245b6664e0291223333be7b509f 6345 F20101206_AAARYS baldwin_j_Page_133.pro db9f6e1f6329877cb563abfeb60d23e1 d37beef14e19c41c3b2630d5cb1f578d428a9d5f 8074 F20101206_AAASFC baldwin_j_Page_134.QC.jpg 3a29cf5f72d5c10a6d88c515d8cd3c4c eedc4ef00eb9e1aac4d8496d81730ede997c651c 5361 F20101206_AAASEO baldwin_j_Page_153thm.jpg 7cbf0f2685c1322e0dc1d2869e1eeed0 58747897ecdbebbf88c9c2c907ec7e680adceef0 65996 F20101206_AAARBM baldwin_j_Page_147.jpg 362aa2e5066e3e2d28734764056dd5ac 9e349232df2701018aed36ada78d8d8a85ebcec6 48519 F20101206_AAARAY baldwin_j_Page_162.jpg 301c86b7f0defc815a93f95e90a37536 4eb2d8033b647e13ee4c6d0a47af7f3e9b0b0012 19612 F20101206_AAARZI baldwin_j_Page_154.pro 67b11c3f4f21c305c6607af0a3b3734a f84090eddeefc5bb12abba474948a1e32350fdb5 6192 F20101206_AAARYT baldwin_j_Page_134.pro bd412255166658a200262666c5488f09 73699f8a88e8cb1328c034e003a484cdf868c9b2 19693 F20101206_AAARCA baldwin_j_Page_158.QC.jpg e602d97d2f150321c0178433aa28f549 deddebc387981308fa0fb0b344bc13697ccd4299 8122 F20101206_AAASFD baldwin_j_Page_001.QC.jpg 9d2a2cc9bdb44269e6d82c2c0f166cf8 459cf5e47d992e5833a9db07ea9563cdd35905e1 8184 F20101206_AAASEP baldwin_j_Page_133.QC.jpg a17b195120f988716e442a9131127c6e ac0b6c03e44e8a1f0d91be2648611b223ca9c403 F20101206_AAARBN baldwin_j_Page_066.tif 677792c6381f9b298bfbf81c9e73e2cf c4090bb8987433091411a99edd602aa2841d7325 1610 F20101206_AAARAZ baldwin_j_Page_068.txt f265db645627f9e86a61df5a86792f0b e2041921ae409bca1bdd184a0e10abfecaa416dd 33598 F20101206_AAARZJ baldwin_j_Page_155.pro e20c8a6212aa0a88979cc5bd5cc722fe fd4ea67d29a038ff17d8a24f9a77e36100cfb35d 5820 F20101206_AAARYU baldwin_j_Page_135.pro 994bcaa2d47fe30eb2a033b9563a07cc af60629f5e283fc3c6fb1d596b892ebbc5535248 60503 F20101206_AAARCB baldwin_j_Page_069.jpg 2c4e6fc67a9d1c3f34ad60d5212b7cd1 6c82c78c1bc4c3a1de391960f2181ff3e540d297 5982 F20101206_AAASFE baldwin_j_Page_007thm.jpg 498e936fdceb489d8bcbf9346bc89761 15788d7fa4906164f0e83c26c2ea62578ff6821e 6080 F20101206_AAASEQ baldwin_j_Page_102thm.jpg 9ba7d940b021898769f44e3d7b4e6e29 54af1e9207e94654e02b30f04b0819d52e6591b9 39373 F20101206_AAARBO baldwin_j_Page_142.pro 143fa8b4d49d86c816ec1f6c8bd8575e f0a463d60fbc6c714688737688ab7e7e4cd22191 1578 F20101206_AAAQWI baldwin_j_Page_145.txt ab388b4c46c8abe244ef080c615f5a8b d5bf1134f5cbe8353e6f74da10badcfc3cb6d1e1 24725 F20101206_AAARZK baldwin_j_Page_156.pro 885fa85bf9a399c01e3eca286b30ddf9 dba9fc211b39eccaddbb847d0f5108b7a9b74686 10908 F20101206_AAARYV baldwin_j_Page_136.pro a2261d9579d997e04d7d96fb0b64caf1 aab29d422df952f3fee08d85c298891254316634 7559 F20101206_AAARCC baldwin_j_Page_122.QC.jpg 0647e4838960dd795b60489200dfd66f 361b20ed9fd77903b31e9ede8827cc6f50c21d7d 26482 F20101206_AAASFF baldwin_j_Page_137.QC.jpg 824ef1b7928a28aaadeaa9c92bbf5dbe ea31e3e726b0a494c20019c5c555c4bf254b615d 2510 F20101206_AAASER baldwin_j_Page_110thm.jpg b015ac3a030a9c06198acb3c32e2d8c1 f0c31255387f14a192f0772441c0bf617ddc8e72 25265604 F20101206_AAARBP baldwin_j_Page_109.tif 4755f0c195a0a09884bc6dd423658438 5cd18656ce08c7aa845261bf15171b6d0b08a67e 18648 F20101206_AAAQWJ baldwin_j_Page_019.QC.jpg 6925269aac5a23afec132e859b7e9178 272388747a408ebff01fabb2b2432535497f7e0a 40122 F20101206_AAARZL baldwin_j_Page_157.pro 802b50c3339440049945b79b1e93bf26 a2a7c1d3811faac9ed0d7fd626ae98fab0914581 53147 F20101206_AAARYW baldwin_j_Page_137.pro 791c8938ece6043dd6276c5c75cd641b 3e3fefa14ddbdf2ea295ac59fb295e9b64deb45c 7645 F20101206_AAARCD baldwin_j_Page_149thm.jpg 7d01ed3585073ef5ff4de7a6dec172fc 0b490fdddc7a76e297cfda1a9275ed901908f707 6023 F20101206_AAASFG baldwin_j_Page_069thm.jpg cd484f3b1491fb6d0ca1e61b930a5774 6658317b7f0c378fb3f38f4e0c59bd4d68c3684a 24257 F20101206_AAASES baldwin_j_Page_101.QC.jpg 5c0305fe7299a0f302fff59fcab9cc16 f8fa5f41ab5948216023c3cfc158647657d02d8a 669496 F20101206_AAARBQ baldwin_j_Page_047.jp2 67dae05f52f70314cbfe0829d67ff5a6 cd5663f34c6763d847168089c0a31aacee4cc015 63259 F20101206_AAAQWK baldwin_j_Page_096.jpg 1c34f6813043abf8582ce90f823507fe de3f32eccbb57911e959144e7f05b5ceb98c8d35 46135 F20101206_AAARZM baldwin_j_Page_158.pro 9c163014361521f4a1cfa7f8ad1e941e 5e0007f7b544a85d7a9b27d2e51af667c199ce04 42778 F20101206_AAARYX baldwin_j_Page_138.pro 93d575137c2396a8909847431c299538 448927abdaf32bb93e908251fa35c91b1779a11a 669 F20101206_AAARCE baldwin_j_Page_117.txt febbb12cf53bdbc91b7b15f9e5ab4a38 fab532dec4d5fd381400e1a057dea85d46a5d282 21892 F20101206_AAASFH baldwin_j_Page_017.QC.jpg b939844985e0a902b8a48d99eefd896c f96e79d939775f259bb5c60eea6209aecb0e77b5 2698 F20101206_AAASET baldwin_j_Page_135thm.jpg d8619a2bb0bd18086ac1736599c55b8c f032e8ff4ad7a2e03e43e6b64cbba236efefc921 7336 F20101206_AAARBR baldwin_j_Page_114.QC.jpg 6270d5e24bc5ed3b5b99c57f899e1803 aa887366f0e44de8c42c13e7ff136feaddfd2973 95 F20101206_AAAQWL baldwin_j_Page_002.txt 016a4da100df461fb9cf5f1f70713b69 322afaeedb2fab2e08b2b2dedd829738c5f8b992 64065 F20101206_AAARZN baldwin_j_Page_159.pro b7cbc81a82b22dcd012df1c8499a6e06 2bfe2030d6c79f13d6eb9202ee724db1fe2810d8 41558 F20101206_AAARYY baldwin_j_Page_139.pro 83fafd85b4ca8a7b553a58abfaef9bf2 11456ca2d14c2f15bc0b67fa9e531de317b83233 1369 F20101206_AAARCF baldwin_j_Page_043.txt b7e1aa3ffa49e420aad91756d8c66f93 9e47c7569be0fe4c90d3183537a1a46ca8fa2059 18061 F20101206_AAASFI baldwin_j_Page_043.QC.jpg 63edd2b159371582acdf5df0a07ace22 66b315d8b60150f330188171256b027932b689a4 2958 F20101206_AAASEU baldwin_j_Page_133thm.jpg 6c2da7ca8bb60867b8f327dd5fb6468a 283bcc256d4edb561ec7f1bbb5785306a45ec8de 1503 F20101206_AAAQXA baldwin_j_Page_064.txt 9423ef2ef7b00622ae9e69b0dd9645f0 335369d119696f40877b6baa241c55563fec4f04 1051944 F20101206_AAARBS baldwin_j_Page_106.jp2 b597caba21705ee2a32e1094e9eab778 af5412e4f907ded00cbcd43255b5c7d3ab162a79 2875 F20101206_AAAQWM baldwin_j_Page_120thm.jpg f7e07e4749ad5d2c3b0d454ec467e56b 75dc2cb43b3c207e24eac100bac2b91de656206e 58356 F20101206_AAARZO baldwin_j_Page_160.pro 50d8f8259e54966d26cb7fea117a9fb1 7691d11ef49e1a2bee8d6e2ed8b9a3b3a047b37e 53649 F20101206_AAARYZ baldwin_j_Page_140.pro f80939a43b1a3006ceb0b52ccaa96639 e840b48be0cbf109b157f60d3df61a42052cfaeb 40963 F20101206_AAARCG baldwin_j_Page_146.pro ec050089a35f00b6502b96a1ba695bd8 7f54df6a2d9707f70913b8a59e10894c44f81cbd 7762 F20101206_AAASFJ baldwin_j_Page_041.QC.jpg 1c7b9f678f8bdd50123a4b79e717b670 8a91222213a18d69dc22cc5a3f2115dca6c5ed6e 23516 F20101206_AAASEV baldwin_j_Page_035.QC.jpg 5a2fc4bb4ff7dad518efa7281d0a2c31 69c3defb206286d00ae4760ef7ee0254c051a8d6 36690 F20101206_AAAQXB baldwin_j_Page_033.jpg 6eb318ce39a9d5bb0c1dbbf845218057 81b8f5eae6cb12a12d931bcea7ef2a342418b914 F20101206_AAARBT baldwin_j_Page_053.tif b811ad5bd30e2c9fbb0b5b4e95b80349 7b2f1d1fbc81d49c5ad3cd2c2bb2eaf6e7a893ec 12244 F20101206_AAAQWN baldwin_j_Page_067.QC.jpg edac5b2a57a5f27151ae15483902d75a 10d980a71eab6103d5439b3c8e258bba41a520fb 43859 F20101206_AAARZP baldwin_j_Page_161.pro b991f6bb4c5d913e58547ab16204f96a 104752fdbaa351770c2a4abb93432ff5f8c11dac 7464 F20101206_AAARCH baldwin_j_Page_112.QC.jpg 48b1faf2f4ea090627d909a351b3c303 de1ef8262952e4ad3ddc698761a60f6b572fdd5b 3597 F20101206_AAASFK baldwin_j_Page_131thm.jpg 02806ac6fde5e6f2ba8c2a709a00556c 734af6f480f77c6b06103e0ad9feac72f0f97995 12916 F20101206_AAASEW baldwin_j_Page_046.QC.jpg cbcc5f64720e031be61f609cee878c3c ba267c4af55a93cdd377cc2a8d098f977df01419 1613 F20101206_AAAQXC baldwin_j_Page_084.txt 5101235ceffa1b64807620100da2c1cf 14a1e51bf759e9856f8efc1c9ad6130c8f4003e1 44774 F20101206_AAARBU baldwin_j_Page_061.pro 7b734b919d704f247413aa074a931ee7 e859af84db36911eb2f57c6d05f4ca22c7ba5ef3 F20101206_AAAQWO baldwin_j_Page_141.tif a5766ad498e87632f0e5d32475600802 248bb585ec483cbc614c56b89cf7a13e0b2a260b 490 F20101206_AAARZQ baldwin_j_Page_001.txt cb39e225babbc41a1a81d232582698c1 ae31e52db828eed92fae214a5afcec2be48720bf F20101206_AAARCI baldwin_j_Page_107.tif 07b2147ed37f679395d28f742dc553d4 833084dcce4bf2beed452a80d5bba3de2743658b 6833 F20101206_AAASFL baldwin_j_Page_113.QC.jpg de4de1e2e1775bd7b590e744cc5d54cd 48468fc254e0cea7fb5394c8e320bb562061eaa2 16135 F20101206_AAASEX baldwin_j_Page_036.QC.jpg 7e89fd6cbdc95169f7fd57fdd1964d2e d5782c9fe75566a8f7c83ce41e681b17b3ec8931 15912 F20101206_AAAQXD baldwin_j_Page_078.QC.jpg 716a53b3a3c2882df7ddaecc0b716153 75e12cf129c4840aa3c57f25db686a95f6bbd127 F20101206_AAARBV baldwin_j_Page_024.tif 3671150f8ff50fbddf7677dbb15e2db9 288bbb73a821913e475e796d9deb1e7416d805d2 527 F20101206_AAAQWP baldwin_j_Page_136.txt 091de8a88de2b5497a77f8037cb2cbbe f68781f1722c4f8a24ccdf03d4923cde32be82e6 130 F20101206_AAARZR baldwin_j_Page_003.txt 39c64c38d16fed841947dca094f1bd04 db87f62e29885b7116f75b9d959159e3010dcc55 46770 F20101206_AAARCJ baldwin_j_Page_022.pro 326b98566e3154d5c4d7245bd0389dc2 269e30e0627e28a6901fbc71e3095c9d70808571 16282 F20101206_AAASGA baldwin_j_Page_079.QC.jpg 8284af1b020371ad6ebf3da8dc83c4c3 c1ac52873fe2cc56f32bfdc112da25ac5ec917b3 5393 F20101206_AAASFM baldwin_j_Page_043thm.jpg b98690a475c720812bc44527bddb1db0 2d25f434eba40bb4b4beb05272cd28794eb0ac17 5704 F20101206_AAASEY baldwin_j_Page_058thm.jpg ea87344198c9044f650d0466ae65a9b5 f45288e312770f86075702ba30c25ec6e3c18ae4 6259 F20101206_AAAQXE baldwin_j_Page_090thm.jpg 0670e4c636d9de50585877ee79d201d2 c44a7f671fa1f79376db9a5af7dd965110039b9b 2742 F20101206_AAARBW baldwin_j_Page_117thm.jpg 4cd9678aedb58100aab7ab92ed22b56c 2086c40b5b7663d35902ba52024c9440e64c0a29 6225 F20101206_AAAQWQ baldwin_j_Page_070thm.jpg fc544b1c320c5b5fe7328bbbf2d7c9c3 b93aea80fc9501cb07c71007bf123e98579759e7 2185 F20101206_AAARZS baldwin_j_Page_004.txt e74234373135c2f84d95c2998b8be656 bbe56471a469e204043a590b6b860361e2ccfaa4 15631 F20101206_AAARCK baldwin_j_Page_151.jpg 4a3f2be15ea5ad89b7ba93bd0dc8f8b9 121553f564f816d1239145c6d5700fd20538fb37 5225 F20101206_AAASGB baldwin_j_Page_151.QC.jpg a50e4e8cd21d2f5525c78af45e7d2e3c 78c8678e866fbac0f42b6b8c2f0565efe50fd2ee 1592 F20101206_AAASFN baldwin_j_Page_013thm.jpg 6e768389c38fb8e3f6e45a3f9eac646e 972d74de217b4e2f9e6e37fb1a904900bfb93db3 F20101206_AAASEZ baldwin_j_Page_096.QC.jpg 445e0e26725f821284795c9efe938fd4 9bdbff155361132fb7947f28374211fd65fdef61 84505 F20101206_AAAQXF baldwin_j_Page_006.pro 1c3de055dc792bb34a81d7cde414a9b5 0e08123927f605ba82b8ac51d58b780b0d4096c7 11593 F20101206_AAARBX baldwin_j_Page_143.QC.jpg eb5c932600fb842ea71ddbd86d8eb8a5 db339c8f7ce6e8df57d39d1f5ec47e58062fb308 30381 F20101206_AAARCL baldwin_j_Page_043.pro 165effa936d487567aeadcb8ec14237f 810efda924ca0d36748f5fb1e9832909191efbce 4937 F20101206_AAASGC baldwin_j_Page_055thm.jpg 686ba043400871efb9ff397e2e8efe87 826e6daa80ac4630013caec7f04914174031c2c5 7554 F20101206_AAASFO baldwin_j_Page_150thm.jpg 2476612b9cafe35b18bf751f28aa25e4 5a26c0eae1ccfe9d30ddc3190b38c08721142dfc 49241 F20101206_AAAQXG baldwin_j_Page_148.pro 753ffea0f74f657899773452ec082534 9292189b63cd1a6dd58b4c4fcb15dd335cd25e36 1571 F20101206_AAARBY baldwin_j_Page_003thm.jpg b262f3eaed810bbd151c306e38eeb0e3 c7e0bd861a9478834dd70ed2a3977142d2136de9 385766 F20101206_AAAQWR baldwin_j_Page_120.jp2 50ca4a2e7157c5eb29f991be664f983c 6dfdbe1cd07bf32d98d807847e74b3423aef8441 3527 F20101206_AAARZT baldwin_j_Page_006.txt 241f9ed10aa661dfbce7a1cda00048b1 dd76dba30115c25529534fffc71e43f699ead8d3 5528 F20101206_AAARDA baldwin_j_Page_152thm.jpg 2f475edeeb01d260e432c8d8b0d8980a f106b40a9c443f712887fe2bf96111b2cf8d0da1 3167 F20101206_AAARCM baldwin_j_Page_002.QC.jpg 8edc02cbf3f54196802646bdc70f85b4 1474ffd02194cbe1e47eaa129c108d5f88f89b36 11094 F20101206_AAASGD baldwin_j_Page_131.QC.jpg 5687c9252dcf6fc5a58ad19b2ee8858d 725cdef892c78fae9e33cb425bcb9f9e0996a530 27120 F20101206_AAASFP baldwin_j_Page_149.QC.jpg 2e75cfb3d9437cf80ca841152f0468b0 a0dd0438a98c616ee60ff76c51b449bce89b6cce 19791 F20101206_AAAQXH baldwin_j_Page_008.QC.jpg 7c9680e24a644312ad84e4852a5b479b 75efcef98ab3b0d8846b015e452e07e6296ee38f F20101206_AAARBZ baldwin_j_Page_001.tif 44f8112e911275ade77ffa390be0cc2e b04d96d5a36e65760ee518bba57fd9c237f00655 475065 F20101206_AAAQWS baldwin_j_Page_136.jp2 4fdc2098225a2937e6712bb192a345c6 ea9bf4e82bb2eca68384b30ab67814b1ef0110ae 2090 F20101206_AAARZU baldwin_j_Page_008.txt 65a5a9dc8c3daf1ca0cb95981a202ed5 bbf83a9b308ed669f937e08ecbd23c92dd99fdcb 5298 F20101206_AAARDB baldwin_j_Page_047thm.jpg 8fa97bff4b742077929631ee11c96c6f 847c9be26d8ecb97adb7dcf1fecfb17a612fa20d 1331 F20101206_AAARCN baldwin_j_Page_089.txt 424881092e595b503625167366da2bfc 795b2a9e60a3aedea02a8df8f4247f04fd512f69 17671 F20101206_AAASGE baldwin_j_Page_087.QC.jpg 211fc0659c405a1c11593ca029b12cc8 6efb672ae0d33532d5350c08f9ef170d93508239 7046 F20101206_AAASFQ baldwin_j_Page_100thm.jpg 99ceb41cbd0ae0faa1424a77bf08f810 21644988dbd46cd513663c2f52505bb92c0d68da F20101206_AAAQXI baldwin_j_Page_126.tif 3349d49dbc54a42962237cf7bd69fb1c fef79baac83f84fad0bd02ef5b5c5cd851865585 2674 F20101206_AAAQWT baldwin_j_Page_122thm.jpg 5d0d7a0a51947251a9a87a5a92bf8e98 d56196e6d7aa97998d1d164cf26370bc3073149c 2310 F20101206_AAARZV baldwin_j_Page_009.txt e413ea131cfd4489e9d86ae3f95556ec d64ad2d983f73cb42b7a3e4a3381fe575232a78a 16097 F20101206_AAARDC baldwin_j_Page_125.jpg 6414aba089fe5dbf3d5b2f0363fdb535 93ebbb22fc14a280ec2e4dcd07bb2aa8a9693be2 24617 F20101206_AAARCO baldwin_j_Page_015.QC.jpg 577ac8e8485e3182dabf3397b02c2eab 9839cd84b3f14573c493e1b07b84d9f7f678a373 18356 F20101206_AAASGF baldwin_j_Page_152.QC.jpg 05af9c2faa9d18e3cfb041666a7888ee 8977a08b5ebbf92d585555743ca9c650f0f8c724 4219 F20101206_AAASFR baldwin_j_Page_046thm.jpg aaf68c18222a3244e43fcd7bbf303282 e5442f782f7ba0bd2a56a5a88d2de570985de136 309 F20101206_AAAQXJ baldwin_j_Page_128.txt 49a0da0b52b25724ac9093cf1d377d9a c09f36697dc9df5a1510975b154e221e23fb5287 6832 F20101206_AAAQWU baldwin_j_Page_042thm.jpg 59f702fe4583ec7c8e12be5f695ce044 30274c785fb6b15ae3d1aecf525b53ce33413fb7 2203 F20101206_AAARZW baldwin_j_Page_010.txt 6d12ffce496a6bd5b61291467a64e50e 9f978cfcabcc7d0bf18e2189da51d62023e42615 19588 F20101206_AAARDD baldwin_j_Page_141.QC.jpg a0a33d19b06ae9aad9ba8b45e8101417 635a9a5e984271a7aeaa83a3d799d6e8999d40a5 29648 F20101206_AAARCP baldwin_j_Page_144.pro 270bc5c3ab32e88f39f938e1f77e0a26 af3407c9098d808bfbda830a7b724788d56923c2 7303 F20101206_AAASGG baldwin_j_Page_110.QC.jpg 7439834a6c20a1506ee6b3bf4fe42b5e af9801775a19e3ea6258ad4bfe6064d241c64386 16404 F20101206_AAASFS baldwin_j_Page_072.QC.jpg f5dab0a1014d3c4d7b1451e9894fe147 dd62eb79e501661bf6b853e251fd9ca1c6877d48 F20101206_AAAQXK baldwin_j_Page_047.tif 658424f0c7760d305e3546c4e258962e 0711419cdbfbc0f793ab97a7217fb368bf5c83b8 1051981 F20101206_AAAQWV baldwin_j_Page_148.jp2 f51d181e88f2c5501611657ec3a870c1 e30567618a1593e2e0f573ffe06aebe48c0df61b 123 F20101206_AAARZX baldwin_j_Page_013.txt 26166f03c40f06afc236ddeec3a6b283 e6c97f5911326371f4ca752543057b047453d9ec 663904 F20101206_AAARDE baldwin_j_Page_055.jp2 b25586ae75ba2496fe3ca2d458673377 b73842e7ec5d3f8b107687cc0246a8f869f1bdc7 36775 F20101206_AAARCQ baldwin_j_Page_039.jpg d51542308ed2e3487a1df5e08b63f5ee 6858be7739097c6b1ae12d78a63d76f87997679b 6905 F20101206_AAASGH baldwin_j_Page_091thm.jpg 6e9ba88db70544592f6ab10f6fe03714 5973aaf588c3d9d710365ea58cbda04803ed08ca 5291 F20101206_AAASFT baldwin_j_Page_044thm.jpg ac44e1f485cdd44bab9c7d3ba5130a92 70979d5b6e9a7bb703714946c3e537b23e48f501 25773 F20101206_AAAQXL baldwin_j_Page_018.QC.jpg e19d21e168d40343ee806876108a9a3a bc5e7a8a33b20b949afdfa65fc562f889927e063 F20101206_AAAQWW baldwin_j_Page_008.tif 7ca74389043e5383240e61b3ba8bc84f 9b159f6e742983b36be90023436f50374fd2c45a 2162 F20101206_AAARZY baldwin_j_Page_014.txt 9d8d6b6b0373c2705e613760e21db40b b21367d56ee2ba0a1c162a1768644a16b1a524ca 882908 F20101206_AAARDF baldwin_j_Page_068.jp2 ee60fd287384949ebb7de9c3089bd887 c8a8e781133bba8079471e7cbc94abe4bb718235 2121 F20101206_AAARCR baldwin_j_Page_150.txt 823f30aa4233114a2948e8331ee255dd df27f9ba1f88a4c3630e840e04876fb1577e777b 7849 F20101206_AAASGI baldwin_j_Page_159thm.jpg a0927929c4b65785f9c2afb664edac57 14d72fdb1d16a1d961eadc956a62931a31dee9de 7875 F20101206_AAASFU baldwin_j_Page_111.QC.jpg f3e842c97736b9630dc5e1212f6c2438 34b945dfa92445200db14ed8e436b9a328b351c3 1695 F20101206_AAAQXM baldwin_j_Page_157.txt aa6fa360378d1830a8f592d34ce8c566 90b6385f000dc682c296cd680f4c5c2a01074fa8 17882 F20101206_AAAQWX baldwin_j_Page_050.pro 6dbdc3028974e0a5880fcca28aa57033 0470099f0949111ccac3b31f7e9796d91ab638cf F20101206_AAARZZ baldwin_j_Page_015.txt 0e4237d5b8b3461355a62e2ae59626b6 ea95f2c7a48a02ee9185131e2b487b38b6765a0d 1051951 F20101206_AAARDG baldwin_j_Page_029.jp2 3637d0f94467ed65663ee7297a10dbca bb0fc7ce05a68eabc422c3948aeecf6811fce0d9 23554 F20101206_AAAQYA baldwin_j_Page_114.jpg 45d5ce2e71aa9487b93755ef3f82b67c a0d9c19b41da3eb9b54405f60871c26661bd33c9 4705 F20101206_AAARCS baldwin_j_Page_032thm.jpg d02a95cec6644fa2a5676106c6332ff9 1009d96c442821535df594b5f4dd288ebb42854d 1339 F20101206_AAASGJ baldwin_j_Page_002thm.jpg d95d8dca284fcb7c392d51deba05b8c3 7308fffa782147edd0cfa27790b61669e647c75f 21404 F20101206_AAASFV baldwin_j_Page_147.QC.jpg ee27ff16b5ca1ec4c95a790a2a6d68ca fd68471cd1bbe26c1dcecd69e990487eaef203d6 44095 F20101206_AAAQXN baldwin_j_Page_147.pro b4f66ed65e8f44b374d78d1e7972bbc2 eac0ed278be65260e345d5a533122f22e49856a2 600022 F20101206_AAAQWY baldwin_j_Page_080.jp2 117908b57c303f2e0cada4e06a6a86eb dbc6e516cab7afb35da9b74d36ebee53d2e66479 1948 F20101206_AAARDH baldwin_j_Page_012.txt 61f77ba4b6764fa283cd6d4611bdf225 d5e73a4f02c1ce9bb713e699ef63397ddaa24bfa 20697 F20101206_AAAQYB baldwin_j_Page_089.pro f85d865ece3c5ced747a8270a5723d33 5bb86be73cd056743e2953677a7e4b1b9878bf51 6135 F20101206_AAARCT baldwin_j_Page_051thm.jpg 91339999492c1fd711fe8dab46355509 b3b3fefe43cd962e95069be9139b43d6c4259999 2997 F20101206_AAASGK baldwin_j_Page_129thm.jpg cb04dfb3ae4dce2e4de6579467649f43 2121b10cf28a04297f442ffb3fba58f6b520f7fc 15960 F20101206_AAASFW baldwin_j_Page_153.QC.jpg 20e33dbb3ec2631ec47f37bba69691ab 292367fdf4f2bb63572f7b9006df9cad336394e4 69996 F20101206_AAAQWZ baldwin_j_Page_099.jpg fb2efd2706e2192e6763774869aa7351 cae8039b6e8a382b6f19c39c4162ecfb5de6e14f 2940 F20101206_AAARDI baldwin_j_Page_013.pro 33165318675303e22a6ef041f0aa260a 77de381027411312b04c8a59f747f459eb9beeba 1051982 F20101206_AAAQYC baldwin_j_Page_009.jp2 09fdf80b965ab83136ba371a358d4fe7 93f9ce878798117313aa562539ffff83c5dda73a F20101206_AAARCU baldwin_j_Page_105.jp2 9cce67c865b116c021a57292863d0062 34ddb98e2935f526e76b156beef239526d7ca20e 3926 F20101206_AAAQXO baldwin_j_Page_052thm.jpg c48040307fdd62c9ea1098ff0f400f9a f0abe2164dc0ccfe006508b48e39a7699e65dbfb 11553 F20101206_AAASGL baldwin_j_Page_011.QC.jpg 3f327940a50e7dc39075964d20624069 575d9b2f933af8640de2e8554a2ef2dcf44a03ae 4334 F20101206_AAASFX baldwin_j_Page_154thm.jpg 33ef1965163b6104d48a45cce6df6fa5 c3e6b5acf959ba593df2b730b0d98af525dd1591 1051979 F20101206_AAARDJ baldwin_j_Page_005.jp2 c14abafe6e422086cd8e90ed889d3f52 19e9221103cfc04f91c903078c161ac037e0e775 1689 F20101206_AAAQYD baldwin_j_Page_059.txt 4617861ae955dd0e631fd7eebf919565 01e7316ffbd89cd607f49d6439d713ad2bef046c 803054 F20101206_AAARCV baldwin_j_Page_146.jp2 ccaa4d9d3b2f3a8eee18d265652a8faf b374cd3dc36e5c91bc9b1edf147ccea0fffc39e1 1051974 F20101206_AAAQXP baldwin_j_Page_083.jp2 4d1e3f3de0b5a80ea90521e6f8435aae 2f5e3a9152bb3c3a978cbd088e5135ffc654c24f 27891 F20101206_AAASHA baldwin_j_Page_150.QC.jpg 4823e4f2261af2ed6f6624e583ed1b89 837d6aa2be884ce3abf5d214fea335761e4eeb09 8132 F20101206_AAASGM baldwin_j_Page_119.QC.jpg 7e71f2643553b5e2710b54749ef7f12d ad3ee85a4364a4f04df02e71e27e285651090fe8 13777 F20101206_AAASFY baldwin_j_Page_154.QC.jpg ce052a1a81152b04a0ed6ec7cc4e5d48 59c31ab4c19d4ba54a0e99f14b159d563fbbce8d 5561 F20101206_AAARDK baldwin_j_Page_094thm.jpg 6fa2a7cdd17621e0d690fc23e8b1facf 2d3e07f0b4c0bdf392e2cecc5774b5f5ed7e213f 2244 F20101206_AAAQYE baldwin_j_Page_038.txt 6ca107f97227f21f257e7d488d0f7aac e44c9a0c230dcb81b850a8f091bb766cb448297c F20101206_AAARCW baldwin_j_Page_099.tif 63ffc3eacf216e011ff3f4c76553203c 3e6049e241d1953ced92924e8cb8982ccc326436 23724 F20101206_AAAQXQ baldwin_j_Page_126.jpg 4fef8a3c18c10ae761f7e515b084e97a 45eeeab470c0d0092dad3c0a0617974fe907155e 24676 F20101206_AAASHB baldwin_j_Page_107.QC.jpg cb6b9be8626bdc51571b907bba3cacda 9e3d4315c91a9a1e81caca162e66077b9ca1610b 9953 F20101206_AAASGN baldwin_j_Page_040.QC.jpg 4cdbf0f00ea1eee28de5482f3c74b847 3a9694cf6059444914ca97053c6635e0af8d5567 2727 F20101206_AAASFZ baldwin_j_Page_111thm.jpg 8de71ba4bfcf2843725542c40574ca95 83fc7d08d20d49321d777a56567b788554ea3dca F20101206_AAARDL baldwin_j_Page_119.tif 0b0b867d50edcca1674952539521c0f0 b95d66443815db9d222e6ac8acc6ef56792d24f5 21715 F20101206_AAAQYF baldwin_j_Page_115.jpg 900f66ebee21607872a11910a88e4618 1ec228ea22b4a2bd0e64bb8112750a5adf536498 4503 F20101206_AAARCX baldwin_j_Page_050thm.jpg 674c0d1005f5725c9f44739dfb155dd3 25d84f7eaced27462bd1fb5dfde7fb0453477061 7065 F20101206_AAAQXR baldwin_j_Page_005thm.jpg 6fa33635c2420160122245590c3643d5 2e3c99649f1318fd884745ef7283829bf7d5e243 2632 F20101206_AAASHC baldwin_j_Page_116thm.jpg f65beb39823bde412b723f8b8bd096eb 2144deb62ac503fc947504cae3164f7ad9984f91 23896 F20101206_AAASGO baldwin_j_Page_020.QC.jpg 9cf8d0860742dfbac29beadeaaf1595e 02ee96aa10406253aa71d728db8b6327a524b7a9 F20101206_AAAREA baldwin_j_Page_062.tif 805ee84ec16668ce3e334798db9b896b f2439e3699f16a71fede85da1399efa81c032a0b 68107 F20101206_AAARDM baldwin_j_Page_070.jpg 55ef2e5b25ba024d6efd1f923748400c 71f20bcbbc80da56cc73513de2a72ff0e6535cd1 671809 F20101206_AAAQYG baldwin_j_Page_073.jp2 248c294f88def1bea5c4639a60fc7f17 e767162a8eea0e26188b9d59619d9d835f1f3230 1018 F20101206_AAARCY baldwin_j_Page_002.pro 751e8a4964a71c24927009aff0f8e55b 0afb4547e92c0e4ab418b95531e1e57d17831f3d 6481 F20101206_AAASHD baldwin_j_Page_101thm.jpg e973d6154090ff2590839887fbaadd45 307353fc8c2bc1a88f816b95d37f3b5813963408 17172 F20101206_AAASGP baldwin_j_Page_073.QC.jpg 52a75c27380d5d234b47c73c84234dff 04a3f4fd28b540945233f1050221e6c9c572c944 34185 F20101206_AAAREB baldwin_j_Page_098.pro 0b0d06df7f4ff962400982bd8dd6dc53 8683b7e1e2ee0a1f335738721aebb37df52e7fe9 41937 F20101206_AAARDN baldwin_j_Page_020.pro ee980eecca09e000ca74404acab4e167 7422c7f1772414fa3fbd15c1866a08922ee1217b 1419 F20101206_AAAQYH baldwin_j_Page_155.txt c15d581980e245312b87ce38448d2078 e6c75a9c2699861d4ed329aab8c80de44e7b6349 25933 F20101206_AAARCZ baldwin_j_Page_103.QC.jpg fcd60bfe08d8bce237732874990d35ee 402fc7b4ca0abd0bbdf486dd7e18c7234b431e90 1051986 F20101206_AAAQXS baldwin_j_Page_012.jp2 953489cbefe4f9f2a82a52a697461853 e7a5e84782653b7eb8f36487d6db88e2ce28347c 6216 F20101206_AAASHE baldwin_j_Page_023thm.jpg 65924eeff99f372d2e43c0fedd3ceeac 705a714cbf8dd2ee4d77825f85e8f3bd483637dc 2762 F20101206_AAASGQ baldwin_j_Page_132thm.jpg 141929cefa16b0becdd18844593dc8f8 ea3a7c4f9ad2cdc14dd4d22135d6fa258268a19b 849450 F20101206_AAAREC baldwin_j_Page_074.jp2 c97f2ecb2b50f4052ecc76a31a1cf269 488fde1a9825b083492302884dc7f1231dd1942b 72537 F20101206_AAARDO baldwin_j_Page_082.jpg b7254d818decc1c6163dea32e410ee31 29340dbae08d9fc61180350943edc72d1f0f837a 7639 F20101206_AAAQYI baldwin_j_Page_137thm.jpg c936c92e613bf3a5f7dc1a53ee28a104 e9b99f853f9d3ca40b016a199685beb7bde105c1 39957 F20101206_AAAQXT baldwin_j_Page_071.pro c2881397d109a1bc8e1800ec08025742 b5475cde35ff28acc1b7a9c9a3eba6deb24a5f58 2165 F20101206_AAASHF baldwin_j_Page_121thm.jpg c2e5571f20a3a78924ef21c9a124d9b5 a84062d20ddb840fc1aff7409e21f798144a0adc 2591 F20101206_AAASGR baldwin_j_Page_115thm.jpg 6fe1517075800f1160f5c936e50a180d 1bcf0bf2dc847ee1789239a8d81fbd0e2c8cbef4 12928 F20101206_AAARED baldwin_j_Page_065.QC.jpg 65d9855474df3d1349e5de3ce069111f 42d82622238df2bc3be00a765fbf9968967cee9d 19243 F20101206_AAARDP baldwin_j_Page_145.QC.jpg 46013e1eda0e718807f72c17a5a5df69 e7496e83a03501ac58ba1b2dd2c988cff41a4ba4 F20101206_AAAQYJ baldwin_j_Page_074.tif af67d5d58732a6226acf5f1c91032d1b 28c0ef7d3d276729617d367fbc6c3b84255fb301 56529 F20101206_AAAQXU baldwin_j_Page_009.pro 05e710cfa55eaf748cba6df389d59c30 79971299541af35b0db8c1aad5c03d94da61d073 3862 F20101206_AAASHG baldwin_j_Page_031thm.jpg bfa37ea52da282b83f9442f295feecd4 8409cc7a4032131062179dfaf3c9adcbc83cdeaf 5278 F20101206_AAASGS baldwin_j_Page_036thm.jpg 1f7aa458fb41edad1846c8b78ed2048f 60f522612b7bead32cf79da56de67290b6654d94 31065 F20101206_AAAREE baldwin_j_Page_087.pro bc6864019c46f6abbdd8ae88aba95a57 8ded851301fceab76db80dd4261c19c46f7d79dd 1869 F20101206_AAARDQ baldwin_j_Page_049.txt 0b4a4d6aba030dc7d8ca9e7d3c031ef8 8491a8487ff233892e0f4f777f0ae1940cfc5046 1657 F20101206_AAAQYK baldwin_j_Page_091.txt dcde74ccbdd77847392b24018c6666b0 e4cd310e3cf5e1b5db6927d45474cbe2eb655381 F20101206_AAAQXV baldwin_j_Page_150.tif 6edde70e091a026d08dba6bc5e96373d 927f41c5bc2c8014da9dc61e6328a44b29f3c52d 7165 F20101206_AAASHH baldwin_j_Page_103thm.jpg 5f77485828d3429e4a6694e7eab7e29d cdeb5c4e4954b27c277e5e6ee51e673a89ac7c86 6937 F20101206_AAASGT baldwin_j_Page_035thm.jpg 1c1e3060c7b93f4414484cbe82a19381 ef60a83822268d980e4a22eb426e97e506180c00 455560 F20101206_AAAREF baldwin_j_Page_067.jp2 07eb024d52404eef2984d0665913ba20 847eac1d9349dd8b6c336344a2ea0a52b6172686 F20101206_AAARDR baldwin_j_Page_005.tif e7ae462e2072efacc7f06823645efce8 5525937c1f24ab076bcad8cccd5addefeaeb791e F20101206_AAAQYL baldwin_j_Page_084.tif daee4708c394eece27532f552d7e90fc a1aaa386736cb4cc3c9b4c9cf97fd56c97bb73f0 2836 F20101206_AAAQXW baldwin_j_Page_128thm.jpg 5f0141239f836acde444186a5a2d6b5b 83df68b78284bb5f3b937cf759faa929f0079c3c 16014 F20101206_AAASHI baldwin_j_Page_047.QC.jpg 024b254558106b3ee57b4090ea9f1b2d b11587ab415fca35fa96c19772a9a33035cc01c9 4963 F20101206_AAASGU baldwin_j_Page_072thm.jpg a4dea0309bc11439307b34b1573177ca c45313b48f6a0b6309d50acda91ce1da9d492ee7 5989 F20101206_AAAREG baldwin_j_Page_138thm.jpg 94d6808a7cc27d1fec8cae7cd0a89d97 a449e6c455b855456da004877a908ceae477f944 7262 F20101206_AAAQZA baldwin_j_Page_115.QC.jpg 38b5ae1d2a8776345993ec38bd8909a5 95d922ccfcb061e80baf90eff65470f963ec82fe 1051985 F20101206_AAARDS baldwin_j_Page_140.jp2 ac02699281f7bb84929a5b84ffb21778 61db8b483f53223ed698952e8c6863e9921c6c53 27000 F20101206_AAAQYM baldwin_j_Page_004.QC.jpg 9d6ab11e9451548eb8e8509ca08ad25a d147610e7d01a952a9660e5b826fe0588a3295f4 76293 F20101206_AAAQXX baldwin_j_Page_107.jpg 5cb5f6130ab4a3b4576b6aa8b03faaa3 3461f78d9a9df9920a44c16d2826c6917f8b22b4 2746 F20101206_AAASHJ baldwin_j_Page_126thm.jpg 04598c50ba7d4b7e753842f50e538374 0ddb1acdcf350de084d0fcea55aebd45b9797e61 26117 F20101206_AAASGV baldwin_j_Page_014.QC.jpg 9c512aa9f96221b9bc4e2d195f31a819 373b52fbcf5e79039f24fedce3dd9c73906bb533 F20101206_AAAREH baldwin_j_Page_118.tif 73e84064757713b3c30dd1dea75e866e e501a046b9cd3ab9373b1313cb16ee5da2c0ce90 697424 F20101206_AAAQZB baldwin_j_Page_087.jp2 cb7c0b12189f68c0004683bc9424094d ab471523307586baaf382abc36033df07a188b88 F20101206_AAARDT baldwin_j_Page_121.tif f5e45637257ca650094abdaf1b3eb664 bc9932f8fd1bbe6f8b5413e469e7643ba2cf0c75 40420 F20101206_AAAQYN baldwin_j_Page_089.jpg d7bcfb2fec1fd70210ddcad7270cb737 9d071528d8c4eb5e1a5984bf3e4b72f9417ebfd9 4204 F20101206_AAAQXY baldwin_j_Page_131.pro d4adcf6b5f3f79fb2464585642e23196 7056aef148d4457e9838cdaa5f492bd57fa14061 20088 F20101206_AAASHK baldwin_j_Page_059.QC.jpg 28a6e4c45f63c65023848fe037a96952 43a7abd35d6671b9f794034aec9e636257a048ca 7055 F20101206_AAASGW baldwin_j_Page_061thm.jpg 53a816c14ce4960cdc5fb14996d16288 3494e2355f3783b67ac3f47a14c7fc9d434e7ae2 20062 F20101206_AAAREI baldwin_j_Page_146.QC.jpg 8352faaf6bb22878d12e7baed8f0845c c6657ce7d1156f367ea78e4d5f0abdd0cd19753e 6319 F20101206_AAAQZC baldwin_j_Page_006thm.jpg 1e111bf5345f33f4cfc7f124bf3b95b0 662bae570a1feebe400066a619fc294535898f76 24665 F20101206_AAARDU baldwin_j_Page_041.pro c3d8db95d6f47edaedf920b58ced277b 4347cd65bba8af9f9707d5d268785986981d1d56 1886 F20101206_AAAQYO baldwin_j_Page_100.txt 926b4091cb02c5ebae3f1463fa7fd320 32266b82b976a3124ae94c02212581ee5cd9631b 17543 F20101206_AAAQXZ baldwin_j_Page_144.QC.jpg 3bc13d779094325f233e202f227c319c 81e587e5175b487dd0ae29b470a2771693f4ff43 19447 F20101206_AAASIA baldwin_j_Page_102.QC.jpg 0cbb5f23618797df4646fff4778819bd 030512140b136228cfb97e871ce64ecb0c626715 5115 F20101206_AAASHL baldwin_j_Page_145thm.jpg 17683abbbe232c1624241d4323542fb7 c83ad537568f0a1867150a345312a859c58feae7 6654 F20101206_AAASGX baldwin_j_Page_012thm.jpg 25f38942e1a753f2e62dd12f8492e07b cdded35b4c44b92c01ceae7a19d2b2a4783c2632 28148 F20101206_AAAREJ baldwin_j_Page_162.pro ce2dcdff5d3a36bb21a5a7a5295f87ba c7fa9da3f95b8312bf8b51730bb0accad03576df 2987 F20101206_AAAQZD baldwin_j_Page_119thm.jpg 52c7ef1fe889be364b7fdf8e1dff844c dff07b2459b6af8ad9832576267ad95bbe7b68d6 75698 F20101206_AAARDV baldwin_j_Page_010.jpg 6c3cba59c67befd3498190989279e252 975674c9dd5d1a2c2f9f562c36c8e9718a7ddc99 6798 F20101206_AAAQYP baldwin_j_Page_104thm.jpg d52dffa284b8d7062bbe76e23a45b7d9 f3eaee3f4ecf04cf248ab296371ea8427fd05605 6665 F20101206_AAASHM baldwin_j_Page_099thm.jpg c25c0adeb28c1c809204c68f219010a7 d9cd0c46ab619026ea52aaf7496f6f6224bc45d4 9331 F20101206_AAASGY baldwin_j_Page_129.QC.jpg 45894f2b955d382cd2af4e6ee0328f02 896e2d2f8ebc7800ed02f635da895452625c1f53 21245 F20101206_AAAREK baldwin_j_Page_027.QC.jpg 69d8d5a5525ed0d36b230cf21474a0b8 33353abf90b3c7d5b73c74eab04e5606c0e8895c 7672 F20101206_AAAQZE baldwin_j_Page_038thm.jpg d4d0b1c525cb87502ae6de58fffb8ab6 034c820ac5026ebc113ce52dcceb44640486a734 17704 F20101206_AAARDW baldwin_j_Page_031.pro 735bc364a472d77fb38a75051faf433c 338c045c1a7c856211dce15a993d5f4222db743b 844 F20101206_AAAQYQ baldwin_j_Page_011.txt 46615488324d99832b445ce134a0ad72 0a942feb44c053b14b573979b8f0aa3af2cc2d4a 17836 F20101206_AAASIB baldwin_j_Page_055.QC.jpg 97d3c9aa93a43e43dc605065775738a5 c234b78f98f613306a9182572bfeaf0a8271e07f 18564 F20101206_AAASHN baldwin_j_Page_157.QC.jpg 0665884ac0b6336370315ffcab814c9a c5041d78f2137f258c0d091bfb6fe17b123ad5f9 24475 F20101206_AAASGZ baldwin_j_Page_009.QC.jpg 9b49d4c8c3800e64565b2ab4cd5da077 3639d53f3e60abf1727772917ce510a27fff6d4b 402685 F20101206_AAAQZF baldwin_j_Page_132.jp2 8d3c715118896d2e2b22bb0fea41f99f dd2eee6ce3abcbf0f3fdd893b4d01978421f9508 1051977 F20101206_AAARDX baldwin_j_Page_008.jp2 617e6488efc57b50939f136d2d459a4f fc5380404b133c0fa40fa80d5271106b127e8ff7 8318 F20101206_AAAQYR baldwin_j_Page_116.pro 9f8361453c4513b44d4608f2bbb6f533 4f7493d5f88f4e04efd293d8d0f50414acfbfdcd 7924 F20101206_AAAREL baldwin_j_Page_130.QC.jpg a92db07fc86c1e5bcae9f61df45a6680 4abe92c19043337484ee7085a28e4faff6f5e382 6077 F20101206_AAASIC baldwin_j_Page_074thm.jpg b5caa2e27b55d8973f39b77fd704905d 6f3316a633d94d5971386acf45c9df923db19336 23782 F20101206_AAASHO baldwin_j_Page_016.QC.jpg 43bc4d956e4d606baacef7a0c4832bb3 89a89252a1cee1f2c0a653415b7cf6b34d3c0c87 728753 F20101206_AAAQZG baldwin_j_Page_045.jp2 e36443c98d746b8116b624404a4aa3dd f3c12a959ef7cf782b60c5fb5a4373665e26e899 51227 F20101206_AAARDY baldwin_j_Page_057.jpg abb9dd7bb99c8a9eef774fa76d86dfe0 fa7cb5ff90fde64c20699f0c38f071d5c61f1332 11571 F20101206_AAAQYS baldwin_j_Page_039.QC.jpg 5f3ce5aa5f89108c51d91afd701315e7 b58bbf50e29ed4c22705255079b710db5c796f97 4455 F20101206_AAAREM baldwin_j_Page_162thm.jpg 216e54f937334161c84cc28203a7f9e2 e30dd0e138f3d417f4f2cc5bf89ef629e6af91d0 6068 F20101206_AAASID baldwin_j_Page_096thm.jpg 2e8e0463894f44c228896ace9d25c749 a57ffde3c80bdb306348b6032b8ca56e143be44b 5580 F20101206_AAASHP baldwin_j_Page_093thm.jpg bf0d8e5d152fcf531cf2dc4bc2b1523e d5d11074021d5e2c69ec0ce8da20ebad21c0b06f 2757 F20101206_AAAQZH baldwin_j_Page_007.txt 76115b9801e940a72c9a7a26444c7468 89cbd4f1952c4047230474ae4f98f3d8c4eead63 76824 F20101206_AAARDZ baldwin_j_Page_022.jpg b6662dd46b431f8c1e3f069159cc9004 b062767ff63d9e79715485f26ef697647dbed69a 6227 F20101206_AAAREN baldwin_j_Page_142thm.jpg 303243b76b4fb160ca6afaf307ebb225 efbeb225d1d3316f1519b5b658b3547bd8e89726 6507 F20101206_AAASIE baldwin_j_Page_092thm.jpg 8338be076eba5e7b208b991c4afc4c51 bcdc78fcbc4eef09e65b59ca8f75d6804fb62905 22773 F20101206_AAASHQ baldwin_j_Page_092.QC.jpg 46116b20d9c80ed687543cd0dc726995 5aa789eb691fa68af39ab18d67fb4b6dacd3868e 5173 F20101206_AAAQZI baldwin_j_Page_077thm.jpg f2f52577bfa1a06ea3ead14e55437d7d 66238d972fcb933993748365595a97b2b1c71c92 5626 F20101206_AAAQYT baldwin_j_Page_113.pro d339566878457c9b83f8f025d3b32ed4 a4f7e3de7ff0a98de923c23bac2f3adcf760b7a3 27152 F20101206_AAARFC baldwin_j_Page_001.jpg fe2e4246d6280699ec902e464187adb6 859a09465b8b6747d711d6e8f4be158f0d9ba9a0 467096 F20101206_AAAREO baldwin_j_Page_052.jp2 f6dcc0f5089e8feb2a9a0d7005cd1292 a40561f2bc8386fd2b39744dbc434c238e6629f7 12637 F20101206_AAASIF baldwin_j_Page_048.QC.jpg 7b1f3d3b879eaad359d70c1f29562afe e8814090f616f1e879b4f3c81b9012a07e8a0ad1 23308 F20101206_AAASHR baldwin_j_Page_012.QC.jpg 6fd13e8d1d0bf4e2ec9d5980410f1902 f7ad5a7c8420efa55691a259f6aea216b49075ad 1802 F20101206_AAAQZJ baldwin_j_Page_102.txt ade5251a8642b60c2b3f250694527339 5ece3a2dcff87b65eb97560267767fc3892828ba 1199 F20101206_AAAQYU baldwin_j_Page_086.txt 5033d3eae298718317d9f9793a08d892 ad25bb016fef44cf9d212f9b7bddf2dc8467f90c 10413 F20101206_AAARFD baldwin_j_Page_002.jpg b0b9764911719a0a8e90f184b3befcd0 c5aebbc51e747b52345127ea8eddea7da5eebdf7 6886 F20101206_AAAREP baldwin_j_Page_017thm.jpg cae3833022927caa84a73df29f6aa257 c05047258b9a44393f1d817a164c7f22ae99ae2e 7577 F20101206_AAASIG baldwin_j_Page_128.QC.jpg 2f42622d5377b14977aca9915ed9b1e4 28c7e61c26ae951f5a65d3f1d7d7d32060b9f077 8351 F20101206_AAASHS baldwin_j_Page_136.QC.jpg 8e1a4d5465edd4d80c8e9b04af949888 81019678bbd91161103a563253dd7ac17cc8a321 4547 F20101206_AAAQZK baldwin_j_Page_076thm.jpg 177111cf2fda8048308e66b92baf8a7a f2a9017a61b92e1b7df73f3633236c264ca97d11 35937 F20101206_AAAQYV baldwin_j_Page_030.pro e0f1a8eb40af1d04df8b77b2b864d6ab d56e194ea2f913ad4813be37fa8f3e776133a1a0 11228 F20101206_AAARFE baldwin_j_Page_003.jpg 27ef09cc5c3f520130042233818618bb 85f101931798a08f6900b3fe23e85e1081490aef 23918 F20101206_AAAREQ baldwin_j_Page_022.QC.jpg 38179855b947989f3909a5a92337ac5b a7ee24a905ccd333a7582e722b4b6d9a7b24a0a8 2305 F20101206_AAASIH baldwin_j_Page_125thm.jpg d70db7975361bc65b3db788825354788 b0fb17a60b2b51cde6b7b12e5aa135e5bbc138a5 4830 F20101206_AAASHT baldwin_j_Page_034thm.jpg e1d4215e2b6a9f62288216a6b47e8d5a 90119979fa25899d3cef7ad64308663c006927c0 5728 F20101206_AAAQZL baldwin_j_Page_064thm.jpg 4e581bf5105205fe4fe871e0d87160bd 38dc9353cef706f448c4f68087f4c24d2c25ecd7 2028 F20101206_AAAQYW baldwin_j_Page_005.txt 171bda878780c17d8cd9f4cf86cb3e3f 173d36e931bf75d0b796c4a96b859687267e316d 86601 F20101206_AAARFF baldwin_j_Page_004.jpg 640e72de25f27cddc1b50f82ae16c6e8 123c921e8a71f9b126042603986b72d06a3c7335 1705 F20101206_AAARER baldwin_j_Page_021.txt 156eb653772b510691052ce8f6cf4e97 5feff97a176d40aa82e12536fcbb311394ec70aa 22233 F20101206_AAASII baldwin_j_Page_049.QC.jpg 25b616d1bef1a151b7055ac7958b8eda b87a0527ee66666ff788b47caefc118efdfdca6d 21782 F20101206_AAASHU baldwin_j_Page_024.QC.jpg 7da525df2f22829672cbcd38e13aa3fd c46cad8a8df9d0c18b73eaa17c8e23401526751b 6146 F20101206_AAAQZM baldwin_j_Page_085thm.jpg a17047093544ebb1cfa710091a20b3b8 edbaea1849da0780f8c49681879c4b0a2b63f138 278607 F20101206_AAAQYX baldwin_j_Page_001.jp2 123b7b61b1ff2412fe3249ad6e408055 7b0b5738bc49eb95d4a991895b84f34c05d2e8d9 82619 F20101206_AAARFG baldwin_j_Page_005.jpg 7cd4de7476000c4895c072ded0fae151 74c4fe4b5f53ad19b88fed2e3f04b0769339c430 21465 F20101206_AAARES baldwin_j_Page_070.QC.jpg 2c1b670f444f5a659539c63382bebf2c 66e9ab12b1a9d8867330354ff87d237def73f7df 5568 F20101206_AAASIJ baldwin_j_Page_045thm.jpg 0c7d590c60b486b2b125346fcb636dda 0054bb0f30a011ee7c19f5d17b673de6dc42792d 5793 F20101206_AAASHV baldwin_j_Page_025thm.jpg f1e9303dba041df0db89adf29bcc21bb 9de7a72e3197fefe5abe89e46ed608bbf5bffd9c 6275 F20101206_AAAQZN baldwin_j_Page_049thm.jpg bd021c6f7b2234a4396973c2f3a8d116 b771814a75d3944555dc17d1618604fb4da3dfa4 36095 F20101206_AAAQYY baldwin_j_Page_092.pro 4be2a84fd1ab0544d7f3504244e8de31 ee9428b32ecd9d358103b8ac3f8972464d49fe5c 92674 F20101206_AAARFH baldwin_j_Page_006.jpg ca0ccb1f7b6b4d038f67fc52dcaa16b9 bbe9d9b57462963037c7d9eeee2f7721442451b2 F20101206_AAARET baldwin_j_Page_139.tif 88ca6fca08cb6a69ffbf8f3ad3db4446 033fef0afd17ab64d1af2314aabd2d8d05d73879 16668 F20101206_AAASIK baldwin_j_Page_057.QC.jpg 12328ef5e4799a4bd175f3263ddc91cb f0b2c06413aca36e89d89a8d4320f8e836b97f79 5720 F20101206_AAASHW baldwin_j_Page_161thm.jpg db84b01ee0e1e4ddbf2ceff270107dc3 186ede8ef8f89937b8647d3d03f73e48cdc58b2a 15399 F20101206_AAAQZO baldwin_j_Page_162.QC.jpg 93ef40ea2eda9cef12bbc4734c25ef15 a8568f75937539875950f3778c90bf3ba5b05a9e 1727 F20101206_AAAQYZ baldwin_j_Page_099.txt b2502e0f220d9bcead2a447428126fd3 7e40daa6468b9cfbd185dbf1d428d1d930a7b0a9 68657 F20101206_AAARFI baldwin_j_Page_008.jpg 1f8f91ae8e6726963c06d6a1aef2963d 122748fc706510e25052a909dfe8021fd9369a31 15368 F20101206_AAAREU baldwin_j_Page_034.QC.jpg bd069d862e74168673b8205c7fba5fb3 442bc3ce5b594811507c89b60ed15e615491d6db 22529 F20101206_AAASJA baldwin_j_Page_075.QC.jpg dc2fb26a095a8c5d3f2021d0c595f959 934a22e588322afdc14f3ea2f3312da9ce8761de 3743 F20101206_AAASIL baldwin_j_Page_033thm.jpg 2dcd1beee1fb2dbf0920429a0b7bd21c d8482bb8c5e3b01688ac93ae683ef9bfab94f0a9 6900 F20101206_AAASHX baldwin_j_Page_063thm.jpg 89fcf17c84fde0fd2026bdaa56835b4f 65885baa2459930458905ce882f1d29e51853d8a 15869 F20101206_AAAQZP baldwin_j_Page_054.QC.jpg 6d533a053240184abe6693e2635eb633 99670c7d122506ccacbf3334851974033f8ce74d 89670 F20101206_AAARFJ baldwin_j_Page_009.jpg 82f3e601d77a9449ae3d6218a32bc82a 320f9890bbdc2a4d8f5d4f517b4a2bf27739ad01 4691 F20101206_AAAREV baldwin_j_Page_108thm.jpg 4208b7a7ffb095b25f675d63200c781c 16f402444df72e4fe9ce4cb9522224c9fda4ac7e 15928 F20101206_AAASJB baldwin_j_Page_032.QC.jpg 4d8033fbc26589dcda927ffb37b5c9ec fae2c25387a8363e1582f0e68607e04d6a29b34d 11991 F20101206_AAASIM baldwin_j_Page_033.QC.jpg 002aaae6d0ed00f880c8191695243d73 ad402de924fb00579bbb05b09fad7a2b2cbd682d 7423 F20101206_AAASHY baldwin_j_Page_160thm.jpg 66d8981769221da231843833e18090fa 521a2b1af9e82f219b6ccb8e668861c540145b9e 37321 F20101206_AAAQZQ baldwin_j_Page_074.pro 8b4bb08ee18bc346090ea4454dd87e41 53ac7b318bbef1129e852e77173733922719eb2a 37021 F20101206_AAARFK baldwin_j_Page_011.jpg 586978b7c5f055fafa2ca68ff5962df4 0cc42a9d0ae3934156e3d3a9415b2c8ca043ebcd 3551 F20101206_AAAREW baldwin_j_Page_039thm.jpg 64956ca5be05426987a4b91d5c4ce668 7bb9d1e44816814e7085fd4b97486583a21b3f6a 5731 F20101206_AAASIN baldwin_j_Page_125.QC.jpg 10a4e217723eb5b78fa08c836e0783e3 35ab437030d91ea23fcb5a23efaaca5d0ffd0b2e 5420 F20101206_AAASHZ baldwin_j_Page_087thm.jpg 107eede0f5ef223cfa48e75629ef73cc 491eb988816867ca3f998c366c03b0adbf6f31c6 F20101206_AAAQZR baldwin_j_Page_143.tif 92201b980b6c629079962d32a5ae7175 48cc0fd92fbba4c9ab11940033b18e5ff1f46e59 77745 F20101206_AAARFL baldwin_j_Page_012.jpg eaaab931fc0eca15ca736dcf56ceffa6 fecc2da153595fcae5247108afbdf37f3f68797c 13339 F20101206_AAAREX baldwin_j_Page_052.QC.jpg b071f340e41ee1aed08df4b7e9bb30df 8e235e00c77117f680b29b5ee9e7b07ac756f4bc 26246 F20101206_AAASJC baldwin_j_Page_148.QC.jpg 9fefa87ea2c6e5bc99194e9ed4621105 bfd670881b6b536feed1cf306d65607005bbd958 15577 F20101206_AAASIO baldwin_j_Page_076.QC.jpg 7121aba8ac6ce6abf609c567ab830882 eaa530f4ce5df2ccd080c24807232e5b847b727b 37206 F20101206_AAAQZS baldwin_j_Page_023.pro 54baf245fa24a5afe2e5d6e5971c9d77 b8a35c8e6887eb32ede6cb60a270e1393724fd4e 53716 F20101206_AAARGA baldwin_j_Page_028.jpg 07263dcd2a3c7e3fd057fbc300128f3c e29bce9d577f67d4b221709712a9dc0590a2a9c1 12070 F20101206_AAARFM baldwin_j_Page_013.jpg d9b10a142849aa3b475642d1ebf3c6f3 5198bd05354ea4fb2d068541cef144efb847f4d5 F20101206_AAAREY baldwin_j_Page_078.tif a0f081faff09789224664639a5bb4af9 0d3569286bf2b47ccd2c059a5e02cfa027743941 12739 F20101206_AAASJD baldwin_j_Page_089.QC.jpg 91b663c63bb4fa2ba33672ea096cc05a edc1369c091e6cc1b3f6a78d13be51a4dabe70c9 5079 F20101206_AAASIP baldwin_j_Page_008thm.jpg 3614f296281797be653d750827dff8ef 0ae6c573aa29872daffeea0de19d93c29ce463e9 F20101206_AAAQZT baldwin_j_Page_038.tif d5fe2216e5c9fd68096e8ea645a101ca 4b48358cb390602c34d12144ccb9fb9e08ace8d9 75863 F20101206_AAARGB baldwin_j_Page_029.jpg 044e408ac655ee03f70da0a2e6d5b157 58f80ce262500d9849a5015dde917ba9ee43f101 83217 F20101206_AAARFN baldwin_j_Page_014.jpg 2354f91ce192d48e2888ccb4dccd2200 d645e25fc7f30619b3d42b617b1fc02accc806a8 187749 F20101206_AAAREZ UFE0021619_00001.mets FULL 9d3ee766e3583f8de3dfccd5292506e6 d1a6a337caedef8e208ba5e0cb52674f1020dce8 2702 F20101206_AAASJE baldwin_j_Page_112thm.jpg c0bc2fc9ede4c6406a026a54c16a23e5 cc789ba50e1709c451d4bd54ae2aef4f1dffe561 11429 F20101206_AAASIQ baldwin_j_Page_056.QC.jpg 059569eca24cbcfff7b84f9a97f8f1ca 82f3d0eff753060d62640b1b18bf94f7279665cf 63071 F20101206_AAARGC baldwin_j_Page_030.jpg c90944df8a2e4dab35f7b4b0215b384d 76bfcdd00d90907701786d4b82b4186747e873a4 77882 F20101206_AAARFO baldwin_j_Page_015.jpg c5d5ff09f8cc6100adaff91296561870 4419fdae7e72e0c18fb285fdea9b92f0a287ac9f 7814 F20101206_AAASJF baldwin_j_Page_127.QC.jpg 8428bed20a6a720f26b3895c9f0f5f78 459d152dab641b6931824f8190d05679bdd10698 7549 F20101206_AAASIR baldwin_j_Page_106thm.jpg 4e8dd5bc07d74e1d322c82db75a193dd cf962356af98bdbba06a015784e5cb30e2f676c0 86003 F20101206_AAAQZU baldwin_j_Page_150.jpg 4041137d59c62b26c9a064d4dc027ce5 c8182369cf133484e0eaa6d0e16acb24207e6b03 37980 F20101206_AAARGD baldwin_j_Page_031.jpg ae7b310cca548f4996d06e0172e80a8d ecbbf3ccb30f0774d80e32aa2922991812a7abe2 71824 F20101206_AAARFP baldwin_j_Page_016.jpg d4deceddca580ccac017317cc995432b 4a6ac2559684292aa1706a10fdcf1dd210223695 7540 F20101206_AAASJG baldwin_j_Page_120.QC.jpg a24ff0f3878ebdf693ea4a181444d834 4fcf693ac77467c36c3751a1248d0883b046b77d 18730 F20101206_AAASIS baldwin_j_Page_045.QC.jpg 07438824ef104bf7c2ab5763d5efc5d9 6292616889cd3eb2263940de38f667137a1413e8 8109 F20101206_AAAQZV baldwin_j_Page_135.QC.jpg c45e694a8469ffe89b96e9991c28fef9 a217f26febf71ea807801eac05c1aeed5d2f45e7 51652 F20101206_AAARGE baldwin_j_Page_032.jpg 16acbc8511337cb0c3cce7240264f094 4a36a2ad954c772ede1c87620a1a893fe33888c0 74913 F20101206_AAARFQ baldwin_j_Page_017.jpg c96ffc44ad141858e0f08e807b13a26c fed9530f20ad186d70611576d2ce933c82376690 2597 F20101206_AAASJH baldwin_j_Page_118thm.jpg 1ea515f2faac7d0d8e7f5c359567c32c 60b7b27161ee4b5db4428c00691f4d953632b840 7475 F20101206_AAASIT baldwin_j_Page_132.QC.jpg f138277be2b511dac9ec5377579f33ff 8ad4acabb8d8eb2e71e8755c3da6e30373165082 1553 F20101206_AAAQZW baldwin_j_Page_092.txt 4ec170aa33265e2fd44532ffa57ed9ee dd70d7ac1767ce741a84b9027041e076784ff6fa 74625 F20101206_AAARGF baldwin_j_Page_035.jpg 9c4f1cd0047077d57d2a673463a2277b 6f7552d3af9e9e8be3d8592c1c15fb5b0022e849 80149 F20101206_AAARFR baldwin_j_Page_018.jpg 1e46f738a6dd21d40cdd5cc2601a9284 60e797d0ce8c7856d223e16e7064fd9f91f1bd6f 5614 F20101206_AAASJI baldwin_j_Page_141thm.jpg d38745fc3be7772616c6a481d03438c7 7bfd7c37320eb822d67596a8732c6f15d153e444 28927 F20101206_AAASIU baldwin_j_Page_038.QC.jpg e05720080945c7d69a8e64e032824bd8 2f5ea311f511afc84d114c1ceb31c11e5b2fc620 F20101206_AAAQZX baldwin_j_Page_125.tif 57e515df0d3c346b20c47c55b0c98f5d 87846ce4fd8ef37ba5ff3f24f923fb32b0dd849d 51829 F20101206_AAARGG baldwin_j_Page_036.jpg 97215c6003dc5acd1c982e376b961c20 74e34ab3447a332f4926a7bfe3b0df7bc6b29222 56634 F20101206_AAARFS baldwin_j_Page_019.jpg 34d8ab513426dcba87d82c8f6e2d4f9f 1675c13fb492fa9acdd47ef0670e4e670ae4da4f 5226 F20101206_AAASJJ baldwin_j_Page_144thm.jpg d1861f134857d533b38ef93b517819cd aebf30fd889339608a8c8261f0afa29a8977bcd8 F20101206_AAASIV baldwin_j_Page_040thm.jpg baf84900b13e3c11f348f1c23e1ade0e c5aafa057f2e3c7222ba8e3ef3b69f1a4348f04e 32153 F20101206_AAAQZY baldwin_j_Page_058.pro 28cc1d84c7e5dfda67dcf7c07638c8a6 f58758021979156aa6fda73b2cc127c250f0e78f 67862 F20101206_AAARGH baldwin_j_Page_037.jpg 338235e61c98957f4232940a3dfb975d 36c993b8c4b55ffe1340f22d46df785fb387f3f8 70423 F20101206_AAARFT baldwin_j_Page_020.jpg dd8f75450c6656aa4b7d953105b6befa b2c7391538809a95f8046c043895d316b599cbe8 21107 F20101206_AAASJK baldwin_j_Page_021.QC.jpg b150932eb5ab3fea5e554f7b127af19c c6dfcee11864a26ac2fc681771c36455879b7eb6 4295 F20101206_AAASIW baldwin_j_Page_066thm.jpg 64606ad574a76345fd044960d8958310 961a81c262512d2f0c3702d2d9862da46ce6829f F20101206_AAAQZZ baldwin_j_Page_153.tif 762e7291d721835b93b08869b8c0646f d2fb7aeabf77610530e9d4b8c229b3587f5edbee 89580 F20101206_AAARGI baldwin_j_Page_038.jpg a8400929734bd3a349e65d1db3c655d2 0091ac350b0d4b205dc365b646fef61f22eda8a8 63977 F20101206_AAARFU baldwin_j_Page_021.jpg 880d822dff7b6de7e524bbf760459847 33ac4eef304593b4c1c013fc524fe0809f9b6e3e 4153 F20101206_AAASKA baldwin_j_Page_089thm.jpg 13144ad1b09d82aa698054f75def3c2c bf100c4b1351963b4324df02855f9616f8d454b7 6997 F20101206_AAASJL baldwin_j_Page_118.QC.jpg 2f68247d73e03299adb8372625f02998 53db6a0736723bd266f553b5202f7ba29fc4f2e8 7446 F20101206_AAASIX baldwin_j_Page_148thm.jpg b8c4d9d52d21d3ae3952163dac10629b abe23e2307cf703710e720889a15734939c3dfc0 32315 F20101206_AAARGJ baldwin_j_Page_040.jpg 01755a93066b7c83f05157f73ceeaa01 55f902c195d956fe9661377f7d3f8bcd3782050c 67005 F20101206_AAARFV baldwin_j_Page_023.jpg ef08e702c8cb665102679e0f84649b39 c2f576619aeb77b0c4b01c8156479832e811de25 6456 F20101206_AAASKB baldwin_j_Page_016thm.jpg 40550d0904bf5c8a050a5a7c0a73b769 b530dc8e5deaf0c9f7fe3fc1a71487b49ead2cb8 6215 F20101206_AAASJM baldwin_j_Page_084thm.jpg 9fc5e45cdf201747abac6ba8330e7aca 8153b8c61b6a6d95b37e2b96fe9475f1bc506688 4661 F20101206_AAASIY baldwin_j_Page_080thm.jpg c30b6c7d41b164e733849d6c273d53bd a4a723eeeaf73d711217cea50fe90d1ec0a566d6 25736 F20101206_AAARGK baldwin_j_Page_041.jpg 645cf5c9e0ec7bf015fb336fbacfffde b0bf7fa88df6b2a0f00d1e07b7b76278861f6445 69236 F20101206_AAARFW baldwin_j_Page_024.jpg 5143f51162d0bb54a854e9d85ce16e3a d20b34a193698a52b0c050f33f60c13cf7f46b59 24329 F20101206_AAASKC baldwin_j_Page_029.QC.jpg 0d9d81ecde4c960ebdf0c05dc848f10e e49e76e8e19ac0bca4f3a250f5ae61e30353664f F20101206_AAASJN baldwin_j_Page_116.QC.jpg 8d5886bd0a9e085fee5178cdc7f331b5 c2b95f1b9d89bc6339f6ff536aedb8154bcb34a4 20291 F20101206_AAASIZ baldwin_j_Page_071.QC.jpg 5cd8343d056abf8c5af6cf754af7d34e 6d6ace24d40910659dc88b0f718c157f146c1a0c 71151 F20101206_AAARGL baldwin_j_Page_042.jpg 6f47b77e73239ab6ee814f58e79e2865 a5961d0c2e569a53d030e7b4bf6d1641d808e72b 59519 F20101206_AAARFX baldwin_j_Page_025.jpg cd7d8623bad7b279c6c237cefd532d82 b0bbfdfa45ad1bbffd71d87a86b82947d60950a5 58000 F20101206_AAARHA baldwin_j_Page_058.jpg 7916bc280187ed4806867056fb968412 307119dbecaad44d758055fd11f8f09bc2a9b195 16530 F20101206_AAASJO baldwin_j_Page_094.QC.jpg 3c5a811eaa64e6cac052af1d3f14b874 b859ca265175c86eb371d035662e09445ff1a82d 58573 F20101206_AAARGM baldwin_j_Page_043.jpg ccb37eb69a53a1fa4384dcd45f827688 855c1303473c2bc5613b44ed52106c8c11ce2b02 61345 F20101206_AAARFY baldwin_j_Page_026.jpg 76d3a1637955fdb5ebb45ea94b0676b1 edb9ae92e3cb7afacb4f746f73bdb4cfeaf9279b 243164 F20101206_AAASKD UFE0021619_00001.xml acf0c947350d9c58ad650ecab7383c17 6669bf1f0130696f73a5aea56c93e0a228155be2 6679 F20101206_AAASJP baldwin_j_Page_095thm.jpg d5722b781cc468d2ef9b542b76c2a9cb fa3a9e3257694c34f3c13be4278111f37f4a1568 56440 F20101206_AAARGN baldwin_j_Page_044.jpg cab2237854bcb9d76b616e2e7a38f889 ee3690417057bc8906dffe962855e17683fd0a7b 64339 F20101206_AAARFZ baldwin_j_Page_027.jpg 65779476d9bd278247d6c618c1384128 e697ab5dacd2df2227594738bac1b788415ab037 59819 F20101206_AAARHB baldwin_j_Page_059.jpg 5f97ce46033f63acff79fa91e9cf1f1b da4d1d9c6f22657eb1f20b834d2a053f925d0192 7243 F20101206_AAASKE baldwin_j_Page_004thm.jpg c8986ea252acfd7ffcd7bacfebc883c3 2135c8f0b6dd397c422c5f35f0f8bae7a1aa57f4 15767 F20101206_AAASJQ baldwin_j_Page_077.QC.jpg 1c1e7a6d8099d30738939fa07a1e643c e2c49ea5fe0d97eb7270d300d677c101bc9f3c8d 56978 F20101206_AAARGO baldwin_j_Page_045.jpg 0b0fd71bd1bd4fda6b73fdbbd45b6c83 75e52a074a306469cd1b1372014eadc2748934d3 71291 F20101206_AAARHC baldwin_j_Page_060.jpg 7c80acdb428b1b95030588bffe36e440 393dae928c4fce77c5697fe2b673c2f9d968bc58 26042 F20101206_AAASKF baldwin_j_Page_005.QC.jpg 55b6efa5bc3f75fde6c9076441aa523d bcb88972277ccbd23d6d8b4c555abbf3b7e9f1d1 7016 F20101206_AAASJR baldwin_j_Page_015thm.jpg 7f5dc62863e89f017a7bf21bfad77444 93b09c35cb2d3e1ad16df0352bab473062ea25a4 39219 F20101206_AAARGP baldwin_j_Page_046.jpg 9cd64e6c9637fb49dd0b5a6b2b2ba7ff 63329eef969a895f1605000c5890907355d42b75 74079 F20101206_AAARHD baldwin_j_Page_061.jpg 1b9205977b0c6113fc0dce5e928b540a 1a29963b3588015cba195c618f8bf943e16d2ee7 25230 F20101206_AAASKG baldwin_j_Page_006.QC.jpg 407c099bc8701c824785f05cf4830b54 ac577de4718ccab00fe8138e86047bd61ac38511 14168 F20101206_AAASJS baldwin_j_Page_050.QC.jpg 59611db883c6ed888a6452d20a80b975 3d8ef5cd47bca930578c2e7768239af5d3981043 51249 F20101206_AAARGQ baldwin_j_Page_047.jpg b7d456fa3dfd1b715bdca25c8540b5ed 1ec308475a893ae436278e96690ea9214a89d463 16091 F20101206_AAARHE baldwin_j_Page_062.jpg 46fc952bbe311cc73dd96aef65d703dd 7ba9df60908e4f89ae227ba714a0eaf1e27cf305 22828 F20101206_AAASKH baldwin_j_Page_007.QC.jpg 57969ca5c1e74c13b7fe5d3319f93847 129bb48be15ff5f13b449981ddc898b117eed23a 20149 F20101206_AAASJT baldwin_j_Page_051.QC.jpg 31f042fe26c23685ffb4aea1cf9bc858 f6e77f651d125c58e9975e67289eb44c08ec7152 40032 F20101206_AAARGR baldwin_j_Page_048.jpg b38ecd3bff3ceb2e89496873627381fa 6d96e5add40aa5a4e644b1e704c11605c28a7a74 78268 F20101206_AAARHF baldwin_j_Page_063.jpg d7364374d4a4f184a5ddce05482e75d4 2efdee18d558096213e3c0a4f1da58dece474233 22908 F20101206_AAASKI baldwin_j_Page_010.QC.jpg b19cda811571b5ad3e4c3ffef34571e1 33c39446ceffc7b7a501fe1974c0cca7c995f15f 6095 F20101206_AAASJU baldwin_j_Page_027thm.jpg 13278430fb2659927f0027743a1d9e96 a0838fd46975bdfc2fb0b29fa5346d41b209d14a 67821 F20101206_AAARGS baldwin_j_Page_049.jpg 9be396014a5959b9e5401d0085ddb578 aba981ea0d510b3e89b17f1685d68fcd7e6c3491 58010 F20101206_AAARHG baldwin_j_Page_064.jpg 9e8da7629d4979c2cbb4e8f9b0f6dd34 6553707d951f3bb130677715a550b6f99070a853 6646 F20101206_AAASKJ baldwin_j_Page_010thm.jpg 30f66c822cd68be9b769e939b3e612d2 24e0044aa4a2cbcdc88e9360c9e584bfd84e5abb 19337 F20101206_AAASJV baldwin_j_Page_069.QC.jpg adbd229c9b95c3f2b531d2c75d1c0704 791316122edde46d2fd958daafddc88364737c03 44638 F20101206_AAARGT baldwin_j_Page_050.jpg acd4e5f606dc515cfdab082fde49949b cac9b2d3327b07851e729b972b062db4cbf09668 40024 F20101206_AAARHH baldwin_j_Page_065.jpg e870b3e4c9533bd338713e9b695868a1 c99bb2392e96a7275901c9765ed962c58b090bc2 3245 F20101206_AAASKK baldwin_j_Page_011thm.jpg a9a7b6475fca41d83da5e86617b52d89 b083a4a5b97c1fef51277e00f0d489498c634f6c 5986 F20101206_AAASJW baldwin_j_Page_030thm.jpg 93b39307f94077413f26af3c2368fa74 72154328fce9c92136ac491c6a1ca7c36504b325 63651 F20101206_AAARGU baldwin_j_Page_051.jpg 657cecc9558578c9c76496f86ac4061d f9cc643b05966526b8fa63ca62457133cdd8b53d 40397 F20101206_AAARHI baldwin_j_Page_066.jpg 1c5c66828d3ad14f557ab99623676ccc e096ad87fa97b0f6ac6f064122231e7c31dc32f8 21301 F20101206_AAASLA baldwin_j_Page_037.QC.jpg 89d44fb8c9dbe7560bca283c8bfca3b7 5120cdf23d30b878a755241bbcbc329d0e6c1dde 3963 F20101206_AAASKL baldwin_j_Page_013.QC.jpg 09e0d11a692ba6880acc3e247d2113a1 35dcc5fe6efe215a0152d82bc9d97632c219306b 6930 F20101206_AAASJX baldwin_j_Page_009thm.jpg 3af53a8ce063589a448471e21a434e5f 8f3564c00c7f674708f7f7ef8be2858cb3e8184b 41843 F20101206_AAARGV baldwin_j_Page_052.jpg f3647ffbbf3e1cb0d7e454b4e8f8036e 71ba3722ee5df3cfa22ad49cf299224ceb4d4da9 39120 F20101206_AAARHJ baldwin_j_Page_067.jpg bf385221fef841e6ec01dd9990b7c90f 3e1aad2e543af72ad06e8294e99fb993ebf1cccd 6288 F20101206_AAASLB baldwin_j_Page_037thm.jpg a8d76132552426f619283de10af082e8 b5d8f4aceede6e97d2b1a10aabb7b66266bde8ac 7444 F20101206_AAASKM baldwin_j_Page_014thm.jpg 63cc10ecd219b815d295d43e0e524555 8e12dbc2c41552900b74079e9be44bff691c46cd 12698 F20101206_AAASJY baldwin_j_Page_031.QC.jpg 13073c584b21e84875bd39c8e010e897 4f023b3ef72185c3ff9620ac7dcafe28285321f5 51263 F20101206_AAARGW baldwin_j_Page_053.jpg 91940b82efaee6a4a80f3034f4c07bd2 2c921f36842c2281ad34362797c7fbb93d66b91b 67124 F20101206_AAARHK baldwin_j_Page_068.jpg dfb4a920fc62add92571fc169091ad29 b7e32a179a65d69383d2b22de2c1c04aaeb42295 2860 F20101206_AAASLC baldwin_j_Page_041thm.jpg fbdd76298ab106a6c228fc53d818b95b 20c246993a6f5a75113165b839dd73b11468f313 6721 F20101206_AAASKN baldwin_j_Page_018thm.jpg b27cb599f496aaa61e9421fd7f16f8a8 ed82be84ddfbdc49244ca0c99a3991773a224c6c 4394 F20101206_AAASJZ baldwin_j_Page_155thm.jpg 129879a7b74ce84c7ed8dd9cb7038998 6a6d3598b8298a674c496422c3c84ae293fd7d09 49535 F20101206_AAARGX baldwin_j_Page_054.jpg 7ab37f1470508c3944bf6749133eb763 a25b0399a9b0cfe1793bb50fd20bc327ec6b1c6a 67674 F20101206_AAARIA baldwin_j_Page_090.jpg 36201bcf9b829a6d300db9718c14f929 fab7f890798276d71e6425484a0f279b544c07f5 65789 F20101206_AAARHL baldwin_j_Page_071.jpg 43a96f66934d44e52caf31d2a8ad11fa ecefe54fb1e093c2953085233cbf2d800c092785 17893 F20101206_AAASLD baldwin_j_Page_044.QC.jpg 2c6e7f941cb65f4d576a0ad248e84c49 5a4c0eabaa9bccb8b0cbf2d992c1f3f6ed5f204f 5636 F20101206_AAASKO baldwin_j_Page_019thm.jpg 3ae77ea8feb868377173002259d80761 3931f80c2e5c94705cc496d48f267223bd48d4fc 53994 F20101206_AAARGY baldwin_j_Page_055.jpg ee69ed9ed22b98e047fb08439b6cdb65 1e7cca112ce0f58ab66e463d7439e8437c6966e3 67812 F20101206_AAARIB baldwin_j_Page_091.jpg b1a1d26e976d5883c9e0ae8444adec1a 511b8a74ecf4a0eb7d9b6a59f44c01b82a8391b3 50978 F20101206_AAARHM baldwin_j_Page_072.jpg c259a9df42f3dafdba566ac53775b2c6 2f486bbeb75c1f2aae5211fca424e573ff18ae97 6490 F20101206_AAASKP baldwin_j_Page_020thm.jpg 573e3fc3e71105ef51427af8b4d5e3eb 16e2cbda97f29bcd9defcb39880c333b6c65d9d8 35612 F20101206_AAARGZ baldwin_j_Page_056.jpg 02bdae68c1046fc6fc18829238ca00dd c4240c4b62b2949000a55fb92908f6b9d5d892a3 56515 F20101206_AAARHN baldwin_j_Page_073.jpg b9ce09197497a2831caad627cb216f8e 24ba227ccc919c2a52ceb0ba8d3ac29e7d254c0e 4352 F20101206_AAASLE baldwin_j_Page_048thm.jpg b290b5e28a43c570f83d1ecfcb5eb226 296049157b55962f3af537ee3f094c8a0824ee25 6044 F20101206_AAASKQ baldwin_j_Page_021thm.jpg 3ad3f6fee45a4e8e2c15c756e6ac4a7b 5f9db64093c229aed80b0951c61b729cc21e3ae0 68989 F20101206_AAARIC baldwin_j_Page_092.jpg 633b8b5ad841858e1379132e77a71eeb 12f781505c458150d0924974cd158a8b104e170d 68431 F20101206_AAARHO baldwin_j_Page_075.jpg 19ef6cd29b5293a2b463fe5a826e4d9d 0d0a4adfae6b86303664bd1daf00966d4bb680ca 16411 F20101206_AAASLF baldwin_j_Page_053.QC.jpg df24596985f2142d47389738f597943c 664c45d38c4d764a7e5550b943baa0a0318715c4 7269 F20101206_AAASKR baldwin_j_Page_022thm.jpg 65c0326049731c26a9d88ad77cd644f5 d5a11b6e75f85bcdc8b27cb46735908d30a0eb56 54580 F20101206_AAARID baldwin_j_Page_093.jpg 531aefe82a85ec25b09c6df4285bc4eb 70d70e63d11bd18ba0d1269be43bb44d22284c0e 46023 F20101206_AAARHP baldwin_j_Page_076.jpg d27e11c1af6b9a05f54afca38715478b 98f6a1c074241dee685d135df2c37f7769d18d49 4892 F20101206_AAASLG baldwin_j_Page_053thm.jpg 5a557090b75e54caf3fd6837205cb081 8affcd39b34490d928b84dd4b4c342db256da592 21927 F20101206_AAASKS baldwin_j_Page_023.QC.jpg 810437cb6b235985f73f6b1334db366e c934dca0c8ad87b2bcf6db5c4a77f543aeb5a37a 50272 F20101206_AAARIE baldwin_j_Page_094.jpg d61b3ad5c545e86137b20882bde8abff 13b1e7da40d88ad88842211f7ecca02f34ad806c 49352 F20101206_AAARHQ baldwin_j_Page_078.jpg 3aa9aa1153d37bc25f729c7b86e15b6d f2b72202a3de1aa867c895c1e2326917a330701a 4713 F20101206_AAASLH baldwin_j_Page_054thm.jpg 7c35692bf048501eb1463e773a0403a2 bbe50767715647f158255deb54258404617b7dde 6498 F20101206_AAASKT baldwin_j_Page_024thm.jpg 4eaf0a999674ede67e1610b84e338f88 5ba8a7b0560bc3b6df9ce44b9a78c671f717a7eb 69456 F20101206_AAARIF baldwin_j_Page_095.jpg 8497e29acb1f21545a7ea87a392f8372 312ad6a773532285b8d12d543ffe136ee07b339b 51293 F20101206_AAARHR baldwin_j_Page_079.jpg fa46fc38b66d9219f5f18b7d3762ba41 a0f954554466d8ca3add88cb4a1bf4110a163c4f 3636 F20101206_AAASLI baldwin_j_Page_056thm.jpg 6b5466d96c5e097d22852775819b8a5e bcb9dfff3cb4d5fb43ef33f38d7cee95db7027a1 18347 F20101206_AAASKU baldwin_j_Page_025.QC.jpg dc40a7da48bdf0c322f6aee175c407b8 d61db9eb226834d3b2fe7180028f5dda9cdce935 62062 F20101206_AAARIG baldwin_j_Page_097.jpg 26619ceb85300e26bb96cc2e07dd5d62 1528581fa7f91ec5db65a4d6753c84c1f4cc910d 49110 F20101206_AAARHS baldwin_j_Page_080.jpg 4fa68af6b89d9b46591e47104f339b20 1994c3c263c5bc4df472b8881c6672e4eccb65f6 5066 F20101206_AAASLJ baldwin_j_Page_057thm.jpg cec3b4efbc70c5a71657bd047e593015 8c00121fd59ccddf6cb2311192909ce414fe9a05 19455 F20101206_AAASKV baldwin_j_Page_026.QC.jpg 8515c1079451dae124fb6712dde6ab83 fef6eb6382f46fe92ba4d4f518c59b43b96f9fe7 61910 F20101206_AAARIH baldwin_j_Page_098.jpg c430e9f1adc19c9400a18d9232ac9197 3997f2d3651c7e44765f3ba0d289650697a9f8e1 68591 F20101206_AAARHT baldwin_j_Page_081.jpg ccb40e2d23cc649dc0d74abbbfe46b24 4a9d7c8687202bc106dcd43f31860cac0a4b17bc 18815 F20101206_AAASLK baldwin_j_Page_058.QC.jpg c29e35c53398850f28d1940133340880 52fc96c8d2ad220a3d6fb85ad940f5aaa5446931 5732 F20101206_AAASKW baldwin_j_Page_026thm.jpg 9a5915d0670576fb14c3cf353c051426 4266fed265365d5dc6b63cd3ca56ce20a54c012b 73526 F20101206_AAARII baldwin_j_Page_100.jpg aaf257ff3f572543c4cf6ebbe5ca3fc1 b29a8a8ae19c476e59d8776a6f554502d95e7a1e 76989 F20101206_AAARHU baldwin_j_Page_083.jpg 34b7854e00a4af51a6bd15c9d4d8a9ff 898fcac7e84bbb9a5e07a40fa81dcae54ab8a250 6350 F20101206_AAASMA baldwin_j_Page_075thm.jpg dd61ba0c087cbe8a0d6bef244d17b827 22a60939abad15e098156ae4e814cf0867d24f2a 5707 F20101206_AAASLL baldwin_j_Page_059thm.jpg 9abbc32b354fe96d726284997359f185 0e5d553b46fa5ce17c438db8e0de67d40b011fbc 16409 F20101206_AAASKX baldwin_j_Page_028.QC.jpg a3e0795edb1f0a0e628f9aa2fbe936c9 29a392102419bfb53f236278006f5af791ef8e04 73297 F20101206_AAARIJ baldwin_j_Page_101.jpg f6be2e7a6db47d39a1aa3508123fe28a 9f633ba162e95ef0a8f3fbfb608799bf68235a03 63986 F20101206_AAARHV baldwin_j_Page_084.jpg cd9f4e8170b46284e0a7d477b18be50b 70e893428809b037da90b14bc707c589146d0e9d 16327 F20101206_AAASMB baldwin_j_Page_080.QC.jpg 5923b78672e3611b9f13d18a8a7c98ff 25ea5dec47cc6cb76afe6a9029864f27b469318f 21443 F20101206_AAASLM baldwin_j_Page_060.QC.jpg 791e027b2d81c9b2600da310a02dbbe0 a3fa706fd98b7ddd6d56e737257554853e4ce2dd 5437 F20101206_AAASKY baldwin_j_Page_028thm.jpg 573e8c82c87c206db1dac9d8fb987663 8cde5083ad20b35c0cf3684255cf4d87e04ad947 62145 F20101206_AAARIK baldwin_j_Page_102.jpg c5ed16b4e17d768da2f31ac2f4a9e187 71b1f9eca278b8dc3332680e2a8891a92e9a10d6 61473 F20101206_AAARHW baldwin_j_Page_085.jpg 5e92df07eb0eb886ec5422677a83355c 75a2aacd98a59258a407ccda194366bc45e9a259 22058 F20101206_AAASMC baldwin_j_Page_081.QC.jpg adbd85a81bcc0ebaa175ad26f9e1f5e4 e9438e34f6362a9c1abec26aff54d513b60d5065 6690 F20101206_AAASLN baldwin_j_Page_060thm.jpg 6fc7d880a9bf5f687866977e04aaf68a f1ba7ac7512b6c1608ca423c0615202e25597536 21293 F20101206_AAASKZ baldwin_j_Page_030.QC.jpg 2111e6f79754cc77efb53bac11ee50fe dc5dad56916acd5b94e635788716bf7bcfc9f480 14970 F20101206_AAARJA baldwin_j_Page_121.jpg 86c29f22cd087ccfd41a6a793ae2815f 1fe3bb0c086950664b30d9ab9b0db6a29c9d999a 78759 F20101206_AAARIL baldwin_j_Page_103.jpg 775811488fbc3fd6feeb3473d1b89c50 9c555e5bd11f3ee5d5cf74c347aa9f6f060300d1 38156 F20101206_AAARHX baldwin_j_Page_086.jpg 8da3ef3241d4cd019b73fbf93ca7cbb2 d330164986d96ee6fa4e329e358da4b3053cc468 21062 F20101206_AAASMD baldwin_j_Page_082.QC.jpg 3a4e29c87be66e1c78286505d202380c 0a7df0e6efec1fa47c6cbeba35f7d68610ed92df 23015 F20101206_AAASLO baldwin_j_Page_061.QC.jpg 46833a8a743e26803eed67040f793be6 9efa51a931895088432dc9311d80f43b1c1aa74f 23125 F20101206_AAARJB baldwin_j_Page_122.jpg 131fb04fde3d6864189136aafe07a716 1bf4cf3ebc0e9d8a0c494e903dc9033e451dd421 73031 F20101206_AAARIM baldwin_j_Page_104.jpg 71ec659fe3ac329f4818fa2ff5a5b772 f9d775a9eb2d8849eb99b23fa4d00eaf46d72198 54927 F20101206_AAARHY baldwin_j_Page_087.jpg 3b95ef205742a45db79062ba60104883 a12d0d6f8d21e31946f85cf32b68b01bf06ed4ae 6894 F20101206_AAASME baldwin_j_Page_082thm.jpg 372af765fc81537fac7c8867ff7cdf83 86e2ca5cb66f21149fe0a3cb241a8ac9077f3426 5377 F20101206_AAASLP baldwin_j_Page_062.QC.jpg 4f9411cd55c55ed3329d56f2c92f511c f8f6a933164d11854298695cb98d879adef73130 24942 F20101206_AAARJC baldwin_j_Page_123.jpg a57eee914ebfb205fc2318256b50c1cc 7f23fa85b9d623dfb87fd435d238e69d0cc6c461 80121 F20101206_AAARIN baldwin_j_Page_105.jpg 739db6113c4dc46a41757ac5351bec5f 3ef09b16e9f06db7e7dea5dc6332e9672c0e61cc 48433 F20101206_AAARHZ baldwin_j_Page_088.jpg d3867e5b8737cde4cbc71b28bbfd2ce6 ea3bc7936c44bb9729b1c5c562c8e134f437659d 1915 F20101206_AAASLQ baldwin_j_Page_062thm.jpg 7d41bd78c867775ce40dea4745f88ac8 bcf7a104e2126bffb77d593b22b802a3ec8390c1 80524 F20101206_AAARIO baldwin_j_Page_106.jpg 34bd9edd4d03e949eb7c0d86fc1f0dea d644a701eba424f783e875f4f6146298df8e48b1 23754 F20101206_AAASMF baldwin_j_Page_083.QC.jpg 3e1e08e2cb61dc015ff1d43f50bf1e51 469a9b04e945291bd052ed08da61e18c7cafa9a1 25081 F20101206_AAASLR baldwin_j_Page_063.QC.jpg b63c33b11720ca1eab981a7d33dc6299 f18e20f769f540cb61c1def1040007eb299d71a5 21102 F20101206_AAARJD baldwin_j_Page_124.jpg 97eeac5f78cf6fb5910efe061dae7d14 4d3616a69cd27e6dc58556f5394b5daa4a5641db 48902 F20101206_AAARIP baldwin_j_Page_108.jpg 1a8fd798c0ef22806d4f7b5891625b86 ada77b876ca6122fb1a30d083e6ab60ed0e70922 7187 F20101206_AAASMG baldwin_j_Page_083thm.jpg e8f7b60be1e05ce80726fc2a8e0cfd8a 77bde5c5e79a2ca4d5e7e71441f81f3792bb0a83 4139 F20101206_AAASLS baldwin_j_Page_065thm.jpg 7986f1bd63ef9a37fdc6bccb840f007a 6c3eb1a493e01f1614175c5da9b0384de5cfef15 23615 F20101206_AAARJE baldwin_j_Page_127.jpg 86d19c664431c6b7ee52f89cb3faa1ce 5f0ca084c12330c6496c0b523dbef3177abbb2c0 20689 F20101206_AAARIQ baldwin_j_Page_109.jpg 82934c91b18a9102943889f4a0ef5215 8290d47fabaa7a2200b86df351ae94931692deb8 19790 F20101206_AAASMH baldwin_j_Page_085.QC.jpg 2d21b8d4896fed9076c19d85ee24ce4e 7eb7c1fd1bd5512655b43185c210976cb134aac4 13195 F20101206_AAASLT baldwin_j_Page_066.QC.jpg 11cd86d75bc955e1268fdfb1d02835ac 487412ca29595f577c7f887124ec352986d366f4 F20101206_AAARJF baldwin_j_Page_128.jpg df6f4e7010ba8138b12ba0c56b1d02ab 3f4051998cd9ff4a98c340f7d36ae8343bd02ead 21581 F20101206_AAARIR baldwin_j_Page_110.jpg 6249dcea0aa636f4512ab8a400b675c8 c7b918e8829a6e20fb4e1750a83d5c4e308a9c90 12324 F20101206_AAASMI baldwin_j_Page_086.QC.jpg 7c90a341c9158c25bd612f334ccf6780 3ec1520c279e6f7c4248075e039672728fc59d41 4040 F20101206_AAASLU baldwin_j_Page_067thm.jpg 56b512b956d0b92272f178e9924a8831 e699783e9b89b96d02c420b553c06ef1a930bd05 28480 F20101206_AAARJG baldwin_j_Page_129.jpg e2e33d8744579cbd8e4d74d1ee0b1f49 8639ed4493945c023ec2c4b5fa1ada4fcdd64163 23315 F20101206_AAARIS baldwin_j_Page_111.jpg 925e4c9010ea1cbf585c8ada4a781fc2 ea37026fd41f99fbe11f7fc3dff52c13387dd5af 4184 F20101206_AAASMJ baldwin_j_Page_086thm.jpg 8a0e8ae15c540555d8d439024ab19422 99553cf138b5ac30825d00daf82fa2152abade1f 20820 F20101206_AAASLV baldwin_j_Page_068.QC.jpg f14f6613bd63629b8e6f196149bf6c41 deedb8f086c3d102085365fbcd53863949c3b93a 24814 F20101206_AAARJH baldwin_j_Page_130.jpg 723f78631767a2fe9e0264c6285e1914 b55e6c71e3256245b5cdd5a4b9a7f6849d325ca8 23500 F20101206_AAARIT baldwin_j_Page_112.jpg af4cdf93491be8ee4fd4faaaeb7beda0 52ff94674b1e10e18a3d32c8d877ffaf3e86e79c 14391 F20101206_AAASMK baldwin_j_Page_088.QC.jpg a9d765e373cb3467c4879f7d7b85315a a4a94d62a4261191a5151e95a78f446399f90ec2 6186 F20101206_AAASLW baldwin_j_Page_068thm.jpg 99b29ad89cd6df3212b646938b1935e7 1bfc07b0cdc34ac50c6ab00f6bddfbe9f886378a 20905 F20101206_AAARIU baldwin_j_Page_113.jpg 64c6cf2cb72ae1b3ca9ccd0b12d94980 c4207a450db350be04971cc7685dc1672cd8d1af 34854 F20101206_AAARJI baldwin_j_Page_131.jpg 5ed299bc84b4e2d90a65a2b217fe7fc0 93630d5571f281bf71d901caf9540429ee53c3a2 2387 F20101206_AAASNA baldwin_j_Page_109thm.jpg d4b05fcdfdfe7b2af2dff6cb9d8c081f 2d29ad21d459df3836043e037122474e5de84b4d 21527 F20101206_AAASML baldwin_j_Page_090.QC.jpg aee9707213884e7aaff1fa7d6415edba 0ca079f921e68c2d7342437c4e8253437978050c 6437 F20101206_AAASLX baldwin_j_Page_071thm.jpg f8e4f44b0baeccaf885eacc1fd41c611 2a1cab438bc239fc71839344e18097eff399c21f 21639 F20101206_AAARIV baldwin_j_Page_116.jpg add303c26e7cd805bf2e518fb973b147 0a872440faa64c94fed24c31998a2b398af7ccce 20725 F20101206_AAARJJ baldwin_j_Page_132.jpg 39dfb9027abd55ce7af285bbe2e5052c 074afbcecab13e82a1c55abfd3e70a5621533c5a 2413 F20101206_AAASNB baldwin_j_Page_113thm.jpg 5c6aa4acd9da8bb4c435d5e52772529d a78ec6e03895efa58e331639a09e8559d8284d75 20805 F20101206_AAASMM baldwin_j_Page_091.QC.jpg 27b38f7d8be8d9be81e448ec6709e9da d06825a6e421b5860f104ef5a55ddc8204a71fba 5049 F20101206_AAASLY baldwin_j_Page_073thm.jpg 2d70347667227a39167e7c18ff196050 240f64d1634da800545be09dc46e18d4d2928310 22628 F20101206_AAARIW baldwin_j_Page_117.jpg 8fa36a97031e73ff9641cd5fb516f5c9 3fbd68502f4e232170ddae9ae0f20c86eec50cdb 23645 F20101206_AAARJK baldwin_j_Page_133.jpg 205abf58080d4ed9d0bb2a959a486e9a e20f2839862241dcf9ab09e3791c7c7fbf9d665f 2671 F20101206_AAASNC baldwin_j_Page_114thm.jpg a61574c00bec1dc72d687f34e81cfd07 66857d4532051d47ff3c83fc53807e99ee08beef 17651 F20101206_AAASMN baldwin_j_Page_093.QC.jpg 49f5434cfb13da3521f3c20d9e4512c5 6c6b47bde2b884cb50d8125eede0798076a700f3 21917 F20101206_AAASLZ baldwin_j_Page_074.QC.jpg a97e16dece11edb6a8a9d1c83562215d 89b8e57fe49189b8f271b6a586cad0b4da9a7f42 20312 F20101206_AAARIX baldwin_j_Page_118.jpg 718a8c38a686258ca556f6f77c0c3859 e3101a03f4b40be31d2a50c79691cc39df0242d0 56526 F20101206_AAARKA baldwin_j_Page_152.jpg 8da12a506c7b870aa9d91b5c527661e7 b79cc25bdf043a918e3c9d1ecc69a5f71b726fca 24757 F20101206_AAARJL baldwin_j_Page_134.jpg 34e3780bd579c0572be374edc06fa1a4 f15a0c3247d7451a9b77aed4ed0a7741f17ca0fd 7745 F20101206_AAASND baldwin_j_Page_117.QC.jpg 00ab84d7a73d2b0cc036703d893ae55b fd93184c63537acbde3c531e71f020340ebb25a6 21831 F20101206_AAASMO baldwin_j_Page_095.QC.jpg a0cc99cedac96f845ce233c6b2b7a2f5 8f5ef6bb3b03531cbd184c14852a849db564238f 23720 F20101206_AAARIY baldwin_j_Page_119.jpg f2ee20f9013953a2e7f7a66f6b2df839 77ba0a54fbe184ff66369271cb61da8228b937e8 51384 F20101206_AAARKB baldwin_j_Page_153.jpg b95a208f19307f0b7eb09fa92f3ead36 1570ec7fbf60b334f1750ed5d6959fe93f0a1d57 24357 F20101206_AAARJM baldwin_j_Page_135.jpg 060519234b27541cf2f4953d55abdf0f 5db389240134d0f0379b74cf064ddb83f9319e9d 5319 F20101206_AAASNE baldwin_j_Page_121.QC.jpg caa08fd79309091d1fbd1c12395a0453 edab0e1cbafce809635ae118f9c58082d54b044c 19540 F20101206_AAASMP baldwin_j_Page_097.QC.jpg 6ffaf5292b146ebc57a83646706fdc9c 6710b2dc652c3163315e0edf2808fcd5133d9fc4 21320 F20101206_AAARIZ baldwin_j_Page_120.jpg d8ee44d73e20a1d2a4b5af964184cb3a c35105008d03cc0af5e1a04fcf28c0f22e422a56 44955 F20101206_AAARKC baldwin_j_Page_154.jpg bd3a0260d2e89e3e386a772e7196dc38 6c02490185ef4486efa2ca9d3b36f19fab6a6721 23279 F20101206_AAARJN baldwin_j_Page_136.jpg 74eb19613cc5088fe6426a6e5896faf7 07e4a2d0c4671f9fa3a0bcbec1a20c044e23c79e 8296 F20101206_AAASNF baldwin_j_Page_123.QC.jpg 6cba8329a6709332ac2bb7ae57ccf72c 9daee4634d8c976f5d1eef698739f0662a4aab01 5787 F20101206_AAASMQ baldwin_j_Page_097thm.jpg a319b43d657786038ab2ba46829b109e 082ad2512167d1808cc8ffeea0494bf25b9fbc5c 55113 F20101206_AAARKD baldwin_j_Page_155.jpg 4f4a78a4344c566df919801bb23eb377 9d9c90931e486c86c2bb2e1d4b7e2b451ff297d6 86868 F20101206_AAARJO baldwin_j_Page_137.jpg 21dd1c0372f57396006d97d8bf43002d f6e334a82bf4e2455633e2dd3a80438fc8e713a9 17568 F20101206_AAASMR baldwin_j_Page_098.QC.jpg 0f9f8ae2c113d3c19fbbb2fe42ed04b2 df930f3c7ee84eaa759e0a0753cc7933d9c5f996 69527 F20101206_AAARJP baldwin_j_Page_138.jpg b2db5a386daa4b7305fbcdce89fb821f 3d74b51c2263423b9fad52009b039808b945b869 7203 F20101206_AAASNG baldwin_j_Page_124.QC.jpg 08f6fa7290b9ddfca1771b7cd58f26ea 123f40654ef3a0a13ef2d16468e31d7ed11f37a8 5679 F20101206_AAASMS baldwin_j_Page_098thm.jpg 0035a7e5631b44893a0bac5d08269071 941e49d52f5c4072870ad316bc8298d2767f7a48 39754 F20101206_AAARKE baldwin_j_Page_156.jpg d8aadca92fe018ee9bf0bfb2a6dc3078 b7deab6af6c9e168f6e30e37d844b29003ca7881 63199 F20101206_AAARJQ baldwin_j_Page_139.jpg 9f3f2ad1cd053c2e872ec3d83d98cf3b 1542aa27585d4df677f58d6b0aeb8f5494570352 7896 F20101206_AAASNH baldwin_j_Page_126.QC.jpg 8349a26a73bb26a794b5db0c615bb6a8 fb1ae26b6f434dc9cb5c72384a053da63a23372a 21384 F20101206_AAASMT baldwin_j_Page_099.QC.jpg 21593b6fe1261cc35e1365b88f8aedc1 d52be68912ff64d055f6bb12ea5864554b3576d7 62285 F20101206_AAARKF baldwin_j_Page_157.jpg 659ccbec495b5931282ee68a1a787230 0355171bce3c5e331caa2248166b621671285d47 85640 F20101206_AAARJR baldwin_j_Page_140.jpg 83943c3dc7c29d9962e80ffc41653a4c 7ed201477eb9ed6902e8416e1aa60fd147692c3a 2684 F20101206_AAASNI baldwin_j_Page_127thm.jpg 81ed7c8eb91fbefed58b8e1ed7f7ac56 833d28de6def00f1996caaa6219f7a7310aacdfc 24957 F20101206_AAASMU baldwin_j_Page_100.QC.jpg 6df8aa7e80f696997aea079b5c4927a6 1b9e7925954d5453c07792e45feb730c95426205 68396 F20101206_AAARKG baldwin_j_Page_158.jpg 091315d832fe666d5a7081e58c244f1b 4ec574b7cc1318161545cd2ec943623c6462fc5d 58667 F20101206_AAARJS baldwin_j_Page_141.jpg 4b23b1f2945e166dcb36705be12e6e38 93e79d48a7b8b22e0abffa5efe0f039f683cc79d 2843 F20101206_AAASNJ baldwin_j_Page_130thm.jpg 60d2ff08062d3c38437dfe6586e31c98 f59108cec51a9eafd9753c65afef66ee60c1935b 25405 F20101206_AAASMV baldwin_j_Page_104.QC.jpg 48318fc561078555a1df07cf40b6abb5 1033a3b7641acb7cce172d80eb3c19cec44d33cc 102142 F20101206_AAARKH baldwin_j_Page_159.jpg 9a2336a94680d1fc5b279817374d0db7 268b377532ffa00023f41268b0e2c093108befb3 64578 F20101206_AAARJT baldwin_j_Page_142.jpg f09acb6dee29b468e5fc294bdf7118e5 03dc805bf0ea3cd9dceae5565b173e6bd3b8876c 3036 F20101206_AAASNK baldwin_j_Page_134thm.jpg 52c6ccc6f04787e5e3b52dd84ec0c6f5 aeb89fec99ed98ff910c254612862ca2f8a0e2be 25477 F20101206_AAASMW baldwin_j_Page_105.QC.jpg c192fbed3fa307fdb0d213b5953a066d ad72a3c4776efa54c14357b8c00c26685e4392ad 93242 F20101206_AAARKI baldwin_j_Page_160.jpg b6f4e3b3d6c2da3b36d0bbcb72713744 c71d04955f04f04d26ca73e90190d509ef263243 37079 F20101206_AAARJU baldwin_j_Page_143.jpg 76da889f00a504841a14836c69341c7b b4312b3a26a22fcf7bbc5addf992bbd00bc0f283 28167 F20101206_AAASOA baldwin_j_Page_159.QC.jpg 90a5aa9a59510de8b38c3ef70b7ae337 fad692a634940734ae1a3df98218a96db97171b2 2808 F20101206_AAASNL baldwin_j_Page_136thm.jpg cb76296b17660223fd3cd83d86720807 069909406b5ce32e16795922d8295e1b10c181b9 7031 F20101206_AAASMX baldwin_j_Page_105thm.jpg 6a951fc55356a46f6ebf63e115d2869d fca05a54cc240d353c00a9016e3cf445a1299da0 74201 F20101206_AAARKJ baldwin_j_Page_161.jpg c7e03e2329909d0a78aab1254dc86d1d 42b321ea7f94e36218d39566e8922ec4f1338db8 56075 F20101206_AAARJV baldwin_j_Page_144.jpg 8de8905b7f7a2ff0b61f130368564fb2 003c5c17fe933260ccf74bca114c45d472e979df 27360 F20101206_AAASOB baldwin_j_Page_160.QC.jpg dfd7d23dc92b1ea30e10c8a80bc3cf6e a8cfb2acb7e12059c817dbd2202cb67286fd0d76 22257 F20101206_AAASNM baldwin_j_Page_138.QC.jpg db272c8581b68a85e4fb36b5461e4f34 f6058e3923b017d548dc3574e38fe2b564ad1744 7156 F20101206_AAASMY baldwin_j_Page_107thm.jpg d31a906382f918a7327faaca4090313f 4f17c2d6cc983b14f6e5dd78fe93e00ce5e9ce15 30639 F20101206_AAARKK baldwin_j_Page_002.jp2 137c3ae634e77d611f8c949df80c4bbb f11e7a6b45af818586dd4497032aef3d7297e627 55991 F20101206_AAARJW baldwin_j_Page_145.jpg 3e093d83bbd0fc59fb8aca9667f568a2 c68499139f3d68b9c75164a5725a3c83c4b7c4ac 21229 F20101206_AAASOC baldwin_j_Page_161.QC.jpg 662f5decd347322499cad7d1fe98bf98 e38c7ad12382146d2f7e87a73a125ab9a5b77481 19245 F20101206_AAASNN baldwin_j_Page_139.QC.jpg d2b4d7a3ec1e50084908b61f5dfb24c1 2016d50388e9fc75fc13c40d4b51a4a43a5d9fa2 15074 F20101206_AAASMZ baldwin_j_Page_108.QC.jpg 2f6f1e092e680a942d409ceae6c308ad d6b557188a5be0254d54905f7f373f26254b67e0 1051935 F20101206_AAARLA baldwin_j_Page_022.jp2 dee522842fb992f22b25889398e9a5cc db298fbfa366755fc9cbd0d89175b3fb5e24a0cf 57319 F20101206_AAARKL baldwin_j_Page_003.jp2 3ebdbfdfdcbf4e55306bdf26132c6324 fb1371632409b45f4c9e28314117fb0928079f8e 60857 F20101206_AAARJX baldwin_j_Page_146.jpg 80cd9bcf29e4a0ec01d10e62f850ac43 62ff967430361f18b0d9498594ff921fa4ca496f 5590 F20101206_AAASNO baldwin_j_Page_139thm.jpg df705ab03686f6553b1804fa4c5489d5 95ad32f5068784553793557c9118ec797f80d3a1 867929 F20101206_AAARLB baldwin_j_Page_023.jp2 de2475ef58a3ff71426d3f7cf1684ae1 037a6435e17dc7d78d7e5f9d487a02629d5ca399 1051966 F20101206_AAARKM baldwin_j_Page_004.jp2 9056e4dd9ff915d59c31f245fb35c7fc fa0d143d004a372130c28fb2ef29a3a761109c28 82615 F20101206_AAARJY baldwin_j_Page_148.jpg 865249a5717a3e8be41ad00211178585 6f36f7afb6870947530979599150007934da9512 7523 F20101206_AAASNP baldwin_j_Page_140thm.jpg 91410e123b2731391d36d6486f24b093 3572ba02ab1fb65a2ff9cebb82d1e374bb5872bf 925681 F20101206_AAARLC baldwin_j_Page_024.jp2 4dca46cc06f56db4697d551f53e1b9fd 8a341336d8413b35ac5c021f22253c0d983835cc 1051972 F20101206_AAARKN baldwin_j_Page_006.jp2 7d88a42e00ad532f5e61f0e1eca84211 87ce8290a973ea0625ac7d56bbe3b9560069ff41 88044 F20101206_AAARJZ baldwin_j_Page_149.jpg 630f90d5f690e83e635bc7fa522ac24c 4d677914dc932435df9c3caf22926263efbe382b 19861 F20101206_AAASNQ baldwin_j_Page_142.QC.jpg 7549694a99928618ef00aa9697fe9276 fe284744fd884982c56b127c758bdb23040686ef 774984 F20101206_AAARLD baldwin_j_Page_025.jp2 16e676d73334e558faaba112740bac1c 98dda4dbc75fce38fedf8a9cdb405d18a1c1256b F20101206_AAARKO baldwin_j_Page_007.jp2 71a975793b668e4a97f22ed2965a7ed4 cabacce0bc97826cb1222dd3a8e4771a4b8a3314 3944 F20101206_AAASNR baldwin_j_Page_143thm.jpg af4fee9ca05992e5d47e85a118dc80b9 8e64b068761a615c9c9cba76015ce42377bf234a 819894 F20101206_AAARLE baldwin_j_Page_026.jp2 11e411e8700a7f42f315dd92e9631d44 259c51a26de9d9bb8d3f03716f1f4ac65cf6f900 1051978 F20101206_AAARKP baldwin_j_Page_010.jp2 fa88dad2a67c98bc9fd4715831d1f551 f57e5eb00627658af6bdaa3fbdf87937ab9d26fb 5931 F20101206_AAASNS baldwin_j_Page_146thm.jpg 3df092412c13f897ec0fa322a6e771bb b41d29ff6db947f8a13f2479b526942a761b9a0c 908973 F20101206_AAARKQ baldwin_j_Page_011.jp2 1e1f509a18f3dc92a1ad4f1113b33ef2 ebeff8881d7532597159897d25517607d1b2cb04 6055 F20101206_AAASNT baldwin_j_Page_147thm.jpg 7165a1a9bde8d0e3fcc372cf00379ba0 1109d563c408a4315c03f41c59fa46a47ad8c8c8 702670 F20101206_AAARLF baldwin_j_Page_028.jp2 d5638a63e402eb3a73fc36c4cdc9e186 0a315d36eca393fd24f4ab28b47cc8084a3935dc 70020 F20101206_AAARKR baldwin_j_Page_013.jp2 b1e5d5c34b7c73c289901c80860d5789 49d98752a3e18f1b9b34a9bb34771e58cd1bc221 1864 F20101206_AAASNU baldwin_j_Page_151thm.jpg d8aa42dd5f5af541b2f2ca5ca71b23cb bbc0c098583b881438a34dd22f5b8ad35a771b16 837259 F20101206_AAARLG baldwin_j_Page_030.jp2 8155a6fa5a27f597ca4c4de5eef74ef4 ae5abe88d9141b425eec26da9167a0cc744e983d F20101206_AAARKS baldwin_j_Page_014.jp2 25e1dc502f7ffd8ba4ad54f67f65e4d7 151c5c74a03f9dc02de20abb64a99aa462cfe612 16731 F20101206_AAASNV baldwin_j_Page_155.QC.jpg c69077054cb5d470b48b722b1f7ad925 44be1618da45a82c0a5eb252f8643c5a85b0020b 444358 F20101206_AAARLH baldwin_j_Page_031.jp2 8dc44154b0efda6310475c6219c17e0b dfb7b73eb4d515790007ab3d6dfd5dedde7a99cd F20101206_AAARKT baldwin_j_Page_015.jp2 d360f6f3b8e9afc04c70cd3ba1c14439 1b22c46014de901e7391154a619ad5d1ba2a821c 11274 F20101206_AAASNW baldwin_j_Page_156.QC.jpg 42351c21f61bf687e3767ae22efbafa2 b5983f9d279229f8c91a182745fb81b906ebd06d 651475 F20101206_AAARLI baldwin_j_Page_032.jp2 1756aa375e1a9b699c194b93c5c4a300 4c0e4be9d350a24aa2ad2b12c7b2cdedc0bc4a17 956130 F20101206_AAARKU baldwin_j_Page_016.jp2 083238d0fab77697f3200971908558b6 60d9d997cda15cf8821fd68c9c780864a3ad9cd7 3385 F20101206_AAASNX baldwin_j_Page_156thm.jpg 99fce67b7a7d9f2ec600b28f28f0f8e0 8f64604fbf8dba04fac63bf577e4b764526cbcf6 402568 F20101206_AAARLJ baldwin_j_Page_033.jp2 67628713b3204c73362864f2d8460c63 e826914b6045926c13b692a681018884f3c2ad1e 1012614 F20101206_AAARKV baldwin_j_Page_017.jp2 cb00fb7b60d3d100e25c50a9bb8fbe60 b99b3cf86ba5e06e6a4e6460ba4b8e185602061d 4903 F20101206_AAASNY baldwin_j_Page_157thm.jpg a06982d90d0cd7350a775e93cb80b752 c3016be506b27329c042d9868d0d74d8a178ea8a 585164 F20101206_AAARLK baldwin_j_Page_034.jp2 f1359e6dc442a8e9bc5ccecb73592041 c6f81397e2310580ba666a0169d16ae652adea9d F20101206_AAARKW baldwin_j_Page_018.jp2 6bac984e777351906ad0c39297300174 1a4103592999e242c6c7657440390c570e38289f 5388 F20101206_AAASNZ baldwin_j_Page_158thm.jpg 3e916f5001ea6f8be235a99566c1aa3d 28a4b0b4c17e609f5a3583073e34f26970c88e58 1000524 F20101206_AAARLL baldwin_j_Page_035.jp2 2214d385fd7f14d71cac3ce72fa50c79 89d5bcc6e5492542fd9d08cdc0b88ac28e7bfd85 762500 F20101206_AAARKX baldwin_j_Page_019.jp2 82f7de7c447cddbee08cd86699131a4c c41916671cefc7cdeae3348e10e3b334ef6d1ead 648950 F20101206_AAARMA baldwin_j_Page_053.jp2 217ace935cc2372ec3cfeaaf791f8737 21ef92de8b455db9a21d801ac1fd064f75ae65ae 680485 F20101206_AAARLM baldwin_j_Page_036.jp2 4f86fdf0d611eb7737586f537c89f885 2f1f8990914fdf40755b2aca773124b12d38d09a 924887 F20101206_AAARKY baldwin_j_Page_020.jp2 375ccafa3fc91ebd21526f301d206e79 95e1c006c482ea63b4de840647b39d7c5f68561d 592026 F20101206_AAARMB baldwin_j_Page_054.jp2 a0615346cc29988fbf04a57e530c55ad d0a490aa2d58cf40dbae669d98b2f2c252ac387d 921679 F20101206_AAARLN baldwin_j_Page_037.jp2 0364c59abbd281cac6d9b803602074a0 f58c9096109a404d74681e3307da7d1cad00cada 844191 F20101206_AAARKZ baldwin_j_Page_021.jp2 409bd9a74499785459d188e64e28ac92 179ab287e270ef8561b7801a1966662ef98b3f35 391047 F20101206_AAARMC baldwin_j_Page_056.jp2 ddebd48b65e4d0d2008225c7310f87e5 dfdba07d4448a5c58888e26d07f7ba9efa44274d 1051984 F20101206_AAARLO baldwin_j_Page_038.jp2 24e8e90e889cfb75b36cc2a70ecdffe9 4bec2d279ec9b66a7de9aed8e49a7e017fef49d6 633130 F20101206_AAARMD baldwin_j_Page_057.jp2 c86195a91e0def961d28c9b4b53fc71f 04f8e1ce75f0ad0d1fb3f278d4e0dccd276f77bb 454284 F20101206_AAARLP baldwin_j_Page_039.jp2 bcfe31c32226978c641b022f827254ae e63e1e08980f4368d9ae78ff7b433eec5851fd31 722951 F20101206_AAARME baldwin_j_Page_058.jp2 0c1b9c785d4e835faa0814f48821ae6d 9449e34d43f40c66ba2d1b6706a7c224896da363 603635 F20101206_AAARLQ baldwin_j_Page_040.jp2 f567611840572415fbf7391b698f75d5 df264fd4fd8c7d34276905913f2283a1a641958f 777409 F20101206_AAARMF baldwin_j_Page_059.jp2 d5b166de2be3444af0167a7043260d59 1b3aab2f7bb36cdb588ae9d2e1be3e7b9ba1c61f 481654 F20101206_AAARLR baldwin_j_Page_041.jp2 5a6dd7bfa4f8d046172327e7fa3dff83 ca5a04871f1f9901d14571a29f72fb1b424a94e3 973108 F20101206_AAARLS baldwin_j_Page_042.jp2 ce47c9594ad806d9a36e426ce9f513e6 59dac9feb934cd4a8c0e4c7073dff36fcbc59361 991175 F20101206_AAARMG baldwin_j_Page_060.jp2 878dd4a0d55233ad5b439ef6c201e935 d98d04d46045b521f00422635cd0b0a45b966983 744379 F20101206_AAARLT baldwin_j_Page_043.jp2 ccba9adee78d1c411d29a3c8328d3195 758c5d68fbb0429ee7a675fdda05eaa41b034e73 1026716 F20101206_AAARMH baldwin_j_Page_061.jp2 43930f62ef205f5aaf6e3d562a768df7 39bc83d7f72027877afc789d28574c038c929e7d 716684 F20101206_AAARLU baldwin_j_Page_044.jp2 1f16365ebdeb493c559785eecd8f5051 c2c2cc732e4fed4462834b1c988520cc90785a88 123384 F20101206_AAARMI baldwin_j_Page_062.jp2 2dd48453c39bcd962fa7f497750d650e db171dff17118d5212dde39622a1cdb7f6955524 459557 F20101206_AAARLV baldwin_j_Page_046.jp2 dad5d0bfcee882663d9ea5077cf91a89 e2963b765d6ceebb157a44b967604bb3335defa4 1051965 F20101206_AAARMJ baldwin_j_Page_063.jp2 aeefd8afa4bf30e38406a55d1983fd8e 57926c5eb4bcaf0444aac8e335c459e71994a971 449152 F20101206_AAARLW baldwin_j_Page_048.jp2 f5c49ee2c8125dd88e4f98c5d66cffbd 71745a623ec04f74b7f9b7cf3f08216a8697790f 768870 F20101206_AAARMK baldwin_j_Page_064.jp2 29c9cfbd2be360ca8d457f1828dac7eb 0920ffea56fed6d73a178d7a254af5d7ac2d2db0 901510 F20101206_AAARLX baldwin_j_Page_049.jp2 070d9b26f6d32e4d8d8833e3bf5bab31 92c156e9ed789a9902aa98089e2dcba0bb2535fe 554457 F20101206_AAARNA baldwin_j_Page_088.jp2 ce024d7e30ba7aec1865af1b0f771401 380ff70d4901dc0d920368c21bd1d931a9ad911b 486996 F20101206_AAARML baldwin_j_Page_066.jp2 f4095906b2d94f47c4d589137ad5192d 8bf030b707ce6bc96e8485ed19a1ab1816c18eb7 522048 F20101206_AAARLY baldwin_j_Page_050.jp2 50dcded4ebfc04afee03c05d8e82b1cf a689b5f9568ad94e5e887d356aecd3fcb32c4242 488096 F20101206_AAARNB baldwin_j_Page_089.jp2 5bd9c423d8b5134cac501f962fe4515a bb54ea034eda326972dc0937127009c622c4d5ae 805432 F20101206_AAARMM baldwin_j_Page_069.jp2 62a78fb0609f2ed7863a0c97ac1d4453 b056f47536388342baac9d0229909b98d6778ad5 834405 F20101206_AAARLZ baldwin_j_Page_051.jp2 3ee622f9798f03d3c53fc57392332bc0 175db8335b69226657e06854d10f970b19903f47 866632 F20101206_AAARNC baldwin_j_Page_090.jp2 cbb4686b0e0532b7824168d93d762810 fc41a20242496fec1a3f5f3cfaaa02ffd9c980ae 958195 F20101206_AAARMN baldwin_j_Page_070.jp2 22e0fe15a50d5b4c65a52436207cd9a9 06e35dddc36f6d46b989a2d7128849d559e4c5e2 951375 F20101206_AAARND baldwin_j_Page_091.jp2 5b91f585393d8c54efac6ae584164c69 73b543c708709cd43dc13f7d245cbfdf77e7b084 876819 F20101206_AAARMO baldwin_j_Page_071.jp2 d31fb87cc859223c093358a31ceec6d8 edf7d58d7173a2cf743a5431ad7454ff7a6116c4 879284 F20101206_AAARNE baldwin_j_Page_092.jp2 2b8263cb51ce5bd39ee6c0dd1b08beee 7c2a95bd23d26eb0e77bc6a5c70f3a2b78f4a3a2 633991 F20101206_AAARMP baldwin_j_Page_072.jp2 046bff28f5df792b9005f9f070e46a06 521e8ba8759c33dbc93460224d8d6dfc07adbd09 704728 F20101206_AAARNF baldwin_j_Page_093.jp2 9a098036bbb876ed912d8ca1b893fc7e 3e75bf058c8a9292df41a0bb0b9ac9e1ea844642 955054 F20101206_AAARMQ baldwin_j_Page_075.jp2 fb3517efdc6532e32c77c1b8814d5206 2def08cee574b3904858d0ba62aa0707049930ee 697235 F20101206_AAARNG baldwin_j_Page_094.jp2 f6aa5df14301ab6189ecc46a3cf587c7 e684ac800aac8845c0d831911f521d3bfb2ef8f2 551062 F20101206_AAARMR baldwin_j_Page_076.jp2 cbaef45b86b8660b5f55061760c4c80b 5c15975fcbcbfd507ded922a2ca05d0f6e1015be 619043 F20101206_AAARMS baldwin_j_Page_077.jp2 d92e51ce3939518f0ef400b5d331a12e 702333259e9e7a11de034ae9ee63ae5046050afb 922415 F20101206_AAARNH baldwin_j_Page_095.jp2 233f2b8d315aaf1a9450451dfee8dd9b 5d448b457426bde42b5049c9f4cf9e1d991c51fd 614458 F20101206_AAARMT baldwin_j_Page_078.jp2 c76df4e44bf86a69166407d6c77329c6 6e32af3d552ad6f5db2983d91f17c787273d0f32 836114 F20101206_AAARNI baldwin_j_Page_096.jp2 b1a17618f645895584c7f0660814969a fab2b689da4968e0a8cb019f982ec4f999d87637 646345 F20101206_AAARMU baldwin_j_Page_079.jp2 50faacb4119ea3eb9f892f4cbbc2fa82 99c5f0286441517a9249537b0618124df394c666 800028 F20101206_AAARNJ baldwin_j_Page_097.jp2 87f28fb85777073a9565c1aa7c50cbf2 44146c0424913d9d5660f1336a9db51d733a2aca 928261 F20101206_AAARMV baldwin_j_Page_081.jp2 4c1cb8b29fce8f70ba6790b1fdce040b 160172bf512e4df32d2ce9175e3af7fdc3c7b586 802379 F20101206_AAARNK baldwin_j_Page_098.jp2 27549d2b50a3736559a492459b3bb5c5 89eff6aea3f5cf7788bc16db61ef624487411efd 1013982 F20101206_AAARMW baldwin_j_Page_082.jp2 f7ac71f5f61b568a9615ab49860e5dda e231102a89a1d51a8135c021e8394957cc4e8fb2 410127 F20101206_AAAROA baldwin_j_Page_118.jp2 fb875e263b1f3151e27246a7eae156ef ca6d5f437df84143c6b5259afbcf873bce6f0554 955508 F20101206_AAARNL baldwin_j_Page_099.jp2 a74bcad10711138cc7d1a986b53d0fc6 fbf2d5f8136e56f6f7a73b5b1d300244801d6460 837150 F20101206_AAARMX baldwin_j_Page_084.jp2 7d107866ab5a1ed810b02a6d20562e53 bdcccc08fe38494f9c8ef65f64bbfa878defb447 466582 F20101206_AAAROB baldwin_j_Page_119.jp2 2f87ebd612204dde2c3ce7e07624dd25 15e96ee58d590b297daf2eb2833d34a47b2b2e0a 991689 F20101206_AAARNM baldwin_j_Page_101.jp2 9916dc3811f65f7b238937ba3600a167 6dfe24ad40f71a99125ace5742d259f7bd5c4fa3 874368 F20101206_AAARMY baldwin_j_Page_085.jp2 8b52b8c64a6eeb4faf128fbb7f4a22ea 55820f442c83056a4e12686d7305ce5ada5a2fa1 248394 F20101206_AAAROC baldwin_j_Page_121.jp2 da244db14a32da42443544a63d5597ac ebbfc4bc6bc40e264574c412a3af2b72119a967f 812029 F20101206_AAARNN baldwin_j_Page_102.jp2 9a01f735bfba0df8df4b130b43e4f635 d89064030c9615df6d3b5c6b798fb94345dc03dd 445358 F20101206_AAARMZ baldwin_j_Page_086.jp2 f16eebddb0379637865dc045d1df6644 50dba430fbb59784b37fee5412ab1dcbe3cdba9c 350472 F20101206_AAAROD baldwin_j_Page_122.jp2 a03839fe3419332bfbc7fd858d3b9e4d 5e0f0d4de524b76a1a79c05ff8357cf7ed14b74f F20101206_AAARNO baldwin_j_Page_103.jp2 36db86658d858d83d523097dcc22a76e 0c1f1bdd144f8f6668e84bd218216ba75e997fbc 491113 F20101206_AAAROE baldwin_j_Page_123.jp2 6db20418c6f46ea5c5d16d8dfc985062 1bbf9c417560e5324bbce7592ec9da31a3f69c39 1024776 F20101206_AAARNP baldwin_j_Page_104.jp2 0c1a5c0fa5b0c9aaeff7e536148e9ad5 6f62d3398c9185ccf49ef765a9c48336f965c08f 340362 F20101206_AAAROF baldwin_j_Page_124.jp2 41fd443cf0c08fc6c1ba9acedc1e0b1b 6b15dba960b3364e7f60b44eaa3ad956daa096ff 1051960 F20101206_AAARNQ baldwin_j_Page_107.jp2 0eca9956d3a28ee10ca0f40baba30923 992027e433983ca1859f2a554510c66701744ef1 627486 F20101206_AAARNR baldwin_j_Page_108.jp2 35bea666486464128c8c23819bcb1a60 50077a1dc889931e85ba7d32ee6a397476d8fc6a 285013 F20101206_AAAROG baldwin_j_Page_125.jp2 933cdbe205d54894b513f73f8fc7eb5e 5abc23ff21fc4547f19a814865f486b484f6e4fb 371775 F20101206_AAARNS baldwin_j_Page_109.jp2 841e63c90f6f8ba631605594ef275042 a8aa655b918e359bb577b6146e9569d9617b57d4 384147 F20101206_AAAROH baldwin_j_Page_126.jp2 b57395d883e448c896d1647dca73460f a7bc0dca53b399b176c7b4c68d2e56895e56cd28 406529 F20101206_AAARNT baldwin_j_Page_110.jp2 bb3ba9435c20202ea2abcabdb8c46c72 6693d8af67c3e0769026b519c7d0897c3ef22e3b 415189 F20101206_AAARNU baldwin_j_Page_111.jp2 e0df2cf0d01dbc6893c8e2321e802bd9 ac530a6265c85180f77b095e22760d505eab3ace 466753 F20101206_AAAROI baldwin_j_Page_127.jp2 e6b6f1cf416f9dc7785106bf08b12034 f5b6dded2ca556636524afabc7d8de871f199712 416569 F20101206_AAARNV baldwin_j_Page_112.jp2 43a5221bf3ebe84e924545b8b81a502e 69c0b9288568f426b48214a23936163220c4150f 391990 F20101206_AAAROJ baldwin_j_Page_128.jp2 02f91074d300be1d9b021028fe42e9c3 39ae2ae29bb2da3d94e4f3fc7c0d3744fd6d987d 374488 F20101206_AAARNW baldwin_j_Page_113.jp2 7dafebd29b835126d442efdcbad88e5f d752286f52246805f225a114f384e4fab2c71e8c 559567 F20101206_AAAROK baldwin_j_Page_129.jp2 a7bd898692f76bf962bc072571fc220c f75b2a14983eaafbd38d2bad3f9448b1aeb213c4 393841 F20101206_AAARNX baldwin_j_Page_115.jp2 ede414e887f55f95f1ac1d0244fd0589 6960d94dc594c3c53b19251d92f71a22f4b6eb5e 1051948 F20101206_AAARPA baldwin_j_Page_150.jp2 aa0f501074997731a4da2ec3e455e820 407d9705581557af74f94eee577dca13f3254102 473062 F20101206_AAAROL baldwin_j_Page_130.jp2 babaafb2fdcf4d2de17613c38282d2e9 5de75ff5667dab8d7e2a303aee2b4a7823ce552e 380959 F20101206_AAARNY baldwin_j_Page_116.jp2 638b984169f9e2029b89ea7f4829b886 bb6760576d5d70444ef61436c0ede570b0be9046 721266 F20101206_AAARPB baldwin_j_Page_152.jp2 3e891c5992b4a828e47a17ee3a7c8c76 b96d09122fc18044a7b86292a0b69f72df26d5cd 737599 F20101206_AAAROM baldwin_j_Page_131.jp2 c96dcc9487e6272196a91d3c097bfef5 ef4e70d8c7ec3519d566c80289302e20d5ffc8a2 395847 F20101206_AAARNZ baldwin_j_Page_117.jp2 ef930fe65fdfc416f767b398ad09e1f2 4105ad12ede3612d2b92cea4ae032e6220d06bd9 658237 F20101206_AAARPC baldwin_j_Page_153.jp2 bef07b27e3bc71391a9ae873a96d2704 0da31f89f293c94022a7d164de8ddd533136eaa7 481790 F20101206_AAARON baldwin_j_Page_133.jp2 289b524cbbe701e48a1f729144d384a1 3bcc3b634d2e28ef86afd7e7221cf1b8484e1aa5 516139 F20101206_AAARPD baldwin_j_Page_154.jp2 61f4e0d72951eb7c19823af95e04d05e 664a096e0fb95e0d96d5f066ba7057abf2cce518 515656 F20101206_AAAROO baldwin_j_Page_134.jp2 0808c62207f9ecf4fa18195fbff3f005 264f3b986fd7b5d79e26e6e719231963d3f057dc 670672 F20101206_AAARPE baldwin_j_Page_155.jp2 476f6a30f583cfb05aace554e43fe8ac 7e76a2f83542e729fbad6c362ebb75bdc317104f 522306 F20101206_AAAROP baldwin_j_Page_135.jp2 829b2f3f24890ccdd32c0ee3e790d76e 0cca1a6a8198eb4b8d24b3582d3f76f355bf452b 468233 F20101206_AAARPF baldwin_j_Page_156.jp2 84a7bd772e4e33ffc9301393bceca9c2 e8013bb21460c6574f7f2a74ef1c359d1943e182 1051957 F20101206_AAAROQ baldwin_j_Page_137.jp2 b88589e549401e5000bf8eeb8a661d86 85eb3eaaa65557688b1f70e2679473eb1fae24cb 799836 F20101206_AAARPG baldwin_j_Page_157.jp2 492094341cb2835e6c41d4539410c62c d8a05f0bf38cd831298e6b1fa6bfec6f3a2d586a 912772 F20101206_AAAROR baldwin_j_Page_138.jp2 f5fc5419a382088d48b67d806b6f3356 e91220e56ef76dfb79d2a0900baa902a0ab631fb 887352 F20101206_AAARPH baldwin_j_Page_158.jp2 a6a79b8ce4eb83248721a1a3cc2fc29f 005cae6350c239d3f1eed252b1c0e0d9b7d855d9 803002 F20101206_AAAROS baldwin_j_Page_139.jp2 2d99d6f6eac5243b37502ed87e6827b8 a4f878661cf3646c38083a60191d65f3695af9cb 1051980 F20101206_AAARPI baldwin_j_Page_159.jp2 8e1549ae2483c8d21309cb72e30687e4 538775af3241b990d551e0ac82c6ace4a217e7db 796485 F20101206_AAAROT baldwin_j_Page_141.jp2 11ba75620019cdfcbae10b1d45db5ebb e685a7041ffc489ba7ce28fe1846a157336b222f 897541 F20101206_AAAROU baldwin_j_Page_142.jp2 4d366bbeb31f5a857781582cd0619a7b 7065a2a998bbfd0647948542be8c7d2870b26f92 1051968 F20101206_AAARPJ baldwin_j_Page_160.jp2 69d3a1686262aa245152d604026c56e8 035c3eae9a959844fab59d451f579c8ea76dd4c2 441047 F20101206_AAAROV baldwin_j_Page_143.jp2 8da156e8eeaa6a6e0376e50e2ebc5251 e310917e5692baeabad780760ce6aece1a509ede 1024463 F20101206_AAARPK baldwin_j_Page_161.jp2 1e0c0932e96b1c22fe98d974f5a4e7a8 b0852439b9ad82441c42e8626367e3214601349a 685700 F20101206_AAAROW baldwin_j_Page_144.jp2 665cc8bd1a7642e5b4bebbc499e83d96 6131d0bdf92d9802fa8c5f9c91f81de2d33f03ac F20101206_AAARQA baldwin_j_Page_018.tif b269baa1da8962c64915f1a31c1b6ad5 c02777276a65dcb68542a235185f8b47a534fe29 639079 F20101206_AAARPL baldwin_j_Page_162.jp2 84362a140ddde7505a849574715b1fb7 48f0744054bca80803f8f8b06006955bb254d5fb 739142 F20101206_AAAROX baldwin_j_Page_145.jp2 e2a0a7eafe2a46d2ca46e5ecf36bbdbd 1b5993591185e3c275eb2a100befc2cf2fae4acf F20101206_AAARQB baldwin_j_Page_019.tif ad62de4366b744f324830aa3f67081b9 01592c366784c4ec90e3c22e2555a6ee4dd85b34 F20101206_AAARPM baldwin_j_Page_002.tif c53aff2f84bdaaf378d864f2d89883d3 bae1a603fa8af027b17322381bf106a9e0dfb382 903593 F20101206_AAAROY baldwin_j_Page_147.jp2 be2064ee34c5b21e3bc6d7bf162f7693 340bb43dbb6dca0f689873823fa70a2439b9ded3 F20101206_AAARQC baldwin_j_Page_020.tif 7598d338c42cb15274a9842724abec8b 23f88f5b8672aac09699e968defee3686c9564a8 F20101206_AAARPN baldwin_j_Page_003.tif 0bec07351f40a31e8a2fab97486e753e 6f81950413e1ed059745e39749539f02fdd89f8d F20101206_AAAROZ baldwin_j_Page_149.jp2 4fe9f4296e7344db97ed3cae3be0702a bdfeaf32c536ed51f50ea9c020206a40735aa239 F20101206_AAARQD baldwin_j_Page_021.tif b37c8943a768bc7223c943ec65fd1289 521cf1ba6805d34d3659c39ae459b8ab3186a404 F20101206_AAARPO baldwin_j_Page_004.tif d0587025cf862efa3b4c2b4fdb9b566c 54be798c4ca4fc79a284377947e91f4332705e7a F20101206_AAARQE baldwin_j_Page_022.tif c10755031f6d277aba815d3c86aa4856 ab2f1260846376446c863084edf70e5845df5d12 F20101206_AAARPP baldwin_j_Page_006.tif cb6ec079311920dabc7e72180872d3a6 03f5734d0909c85929c67d5dd7d96055639691c5 F20101206_AAARQF baldwin_j_Page_023.tif e10ada441c999ed4d7a9f857cfe9dee0 7c5ffb651db02a35d8c5d69a9231f8af597d00e6 F20101206_AAARPQ baldwin_j_Page_007.tif c041e41f4bb374dced1c4c371894b256 f256e197ce6bbb633b78966c184014fce9a061f9 F20101206_AAARQG baldwin_j_Page_025.tif 4db19bca0f92f3cb520d57ca3d99505f 025b5f2f3e1efe35e08ae53214690d6913b57306 F20101206_AAARPR baldwin_j_Page_009.tif 8ce3b1d337214dec9bfc591588e87858 3657751142e9ad9dcad8d89ce39031f1ba997773 F20101206_AAARQH baldwin_j_Page_026.tif 988e64cbb4aabf1d19db72aef4a8cddf 5b5f581a65643deb8f3d6ccc0be03438fe65ba9e F20101206_AAARPS baldwin_j_Page_010.tif 8dc557cf882d248d66d2a5284bd7c9c2 5b575f56aa367d036bb365596a939138aac954bc F20101206_AAARQI baldwin_j_Page_027.tif c3262927d1fa489817d5a8bb37e69740 a1a8c90225020e73d1d927f1336e796050155565 F20101206_AAARPT baldwin_j_Page_011.tif b8c00bd8ac429304624a33e6a4a54131 7f46391c1eb4bf1d6d45a6d297ceecf088787876 F20101206_AAARQJ baldwin_j_Page_028.tif f41899c786c7675f464cdb9f779b4ad6 97ecf978d1efad29f1a725975ba4eea21e3262d1 F20101206_AAARPU baldwin_j_Page_012.tif 5abeac0663069b236317746f4340d149 f3b17a346a60cf129ab0d7468e54963939d71a66 F20101206_AAARPV baldwin_j_Page_013.tif 3dda4c9dc5d3aa8d1123e39b8aade899 8c2b82db4f51f4296cdfc996e33c07c073ea059a F20101206_AAARQK baldwin_j_Page_029.tif e19df849338ad85b0c4119b565b70469 2774c7a81462b806e8e9a8de16d91a8ebdb05041 F20101206_AAARPW baldwin_j_Page_014.tif 43d35886d4e1a276986d580197cf14d6 94e62c59c490c4fe99c43b63a584bd42a4c11cbb F20101206_AAARQL baldwin_j_Page_030.tif db6649ac0e416d9a4f65909782edc15a 220e7528b83ca4ccae5e45f477e77ad110ac972f F20101206_AAARPX baldwin_j_Page_015.tif a446bd1e15a0025d096463c340184acc f304fb691df0ea110c27045ab9541efe73df8c18 F20101206_AAARRA baldwin_j_Page_048.tif 96a456af298e9ac069d3a95f9a958062 4c2f6220355178e8ffe5482e638b25bba1fff567 F20101206_AAARQM baldwin_j_Page_031.tif 044555ecda031da48932b4af621737b4 fbc9c68b7af641438cdd1cbf7d1ae95a7b62cab0 F20101206_AAARPY baldwin_j_Page_016.tif 19b30f77facd9ec42776701f98f93a26 5f024063ba4b3962511b6d6742f431a045f3e79b F20101206_AAARRB baldwin_j_Page_049.tif f0ce65b8ef63c3a00fed984acda67672 b869681559d88f258395510dec1d07c78d876caa F20101206_AAARQN baldwin_j_Page_032.tif a5fe1635c1378834afaeb9423283f09e a6faf52542d196e9badc7a5d293d1ab57a5fae3c F20101206_AAARPZ baldwin_j_Page_017.tif d16f0665578e662e32b3df0c53e6fca2 e671f144baf26e2f2cdb6227cb797524fe941038 F20101206_AAARRC baldwin_j_Page_050.tif bb1a9736b2421e58940a193be8f79628 3b9c77601fd50b14ef57ab5a4183c5eebdd0b3a1 F20101206_AAARQO baldwin_j_Page_033.tif 712365202df807c5fe75aa72f741f431 59d90ec7de67afc353571c31afa3198f534ab1d8 F20101206_AAARRD baldwin_j_Page_051.tif 854a69e834c65d14c8b8324bee26058a 539ddd98bd342fdfb7633d93825620fa805a0c7e F20101206_AAARQP baldwin_j_Page_034.tif 7c24fa513d9a4948c5df92d629255fa8 839bfde185f89d64d281cdc851389427fc7f49f8 F20101206_AAARRE baldwin_j_Page_052.tif 1b5ed0a659a5e4655cadf00d392fec8c 31a8532d21b949e4a68beb0072c73d292b7eaf79 F20101206_AAARQQ baldwin_j_Page_035.tif 5ecd4f666631f357bcaf84a1d8206f80 ef695a9fbe43fdb057790a234c31c47aaa973913 F20101206_AAARRF baldwin_j_Page_054.tif 9eb2bfae90978b9c19bc6c1ec48bdd32 438f5cf6756ede80760c6244708703f25fd32c1c F20101206_AAARQR baldwin_j_Page_036.tif f986756475ff94b9b6e0b38ea227bf52 9f4a5354186b3ed4773f2af60932fc3e03dbb008 F20101206_AAARRG baldwin_j_Page_055.tif 728a99e986d3f2ed613d01b2428d5c1c f5e5ee0b086e53d4cf1989b3c46bacee8d479ab1 F20101206_AAARQS baldwin_j_Page_037.tif e9cb0d9f2600d51b97aff92febdc6532 e64a3877444b0f62376a776901ea490f06175cc8 F20101206_AAARRH baldwin_j_Page_056.tif 05f725b5d04204c14c2a401e9365e5fd bfc90bc5c8a4aa657fc127c8ce47ff27569a6869 F20101206_AAARQT baldwin_j_Page_040.tif 2b42465599eb890fb99050c376e2bf3e 27be5ea437b835ce1c497738f0a9cad33578d2b3 F20101206_AAARRI baldwin_j_Page_057.tif 325daa673fdc0cd841fd4c3de0ee294b f5e4136ed2975ea34b118c247141e920e2f00fd1 F20101206_AAARQU baldwin_j_Page_041.tif 67d580898b570298d7e1ccd37b494634 fc2755592bdefa41b169fbae48a1a6671cc353c6 F20101206_AAARRJ baldwin_j_Page_058.tif 90262a29091e7321debc8e8ac535e21b ed7981afd87da1a59cf9d254d6d74c3ddae374f3 F20101206_AAARQV baldwin_j_Page_042.tif dc781f024d5e439af9ac706fb98aac51 5d03b251cd011462dd718bc29410e6d532c6528c F20101206_AAARRK baldwin_j_Page_059.tif 162f699b638eddd7c469a327bd531e26 f7926fa1228aca8ea7efed7a60cb3f59bdc2e090 F20101206_AAARQW baldwin_j_Page_043.tif 1fec261aa4e0532d890ef9f380839015 147c8d799d910c6a157365a4cec1736be2a98c8e F20101206_AAARQX baldwin_j_Page_044.tif ed617f7442a53b8087a3a98aa73ebf48 b2050df8fe486c4858f8331ea417d7ea6ba49630 F20101206_AAARSA baldwin_j_Page_079.tif 4daa55abe3e5b839def4dd90ce82c445 dd6daf22fe3a24cd3360708a49981c24570c46bb F20101206_AAARRL baldwin_j_Page_060.tif 9c19401cb941b760dd35047629422273 e5174941d38e49a74c4be661a55141c9735e9b26 F20101206_AAARQY baldwin_j_Page_045.tif 041f260e05431d50b7344e73b9e89ec1 5a0bdc922ea472c7ff6d41d334959a44daf471bb F20101206_AAARSB baldwin_j_Page_080.tif b0c6cf6a930e0481ad49cac45150e5d4 2a88ff9c02313fde3cd2289c6f6ce2fdb14f6e78 F20101206_AAARRM baldwin_j_Page_061.tif 590857e0868c1bcab1bf626d1149c6ee 851bdab570257bb7e08d6869bc6866632b707973 F20101206_AAARQZ baldwin_j_Page_046.tif e13e39c9819ecdaf4901ffff93308a4d 22501552b1bc60975794a3c6219859752ae56166 F20101206_AAARSC baldwin_j_Page_081.tif da9bae2c90e9fc7ab7481f501acae706 2e3d281c25e31b6fee0e83342036b37315763f17 F20101206_AAARRN baldwin_j_Page_063.tif 89dbae9f44eddead8868e0e6f7e44460 125c93720deb935b850880b4ac611a3a34f06a0e F20101206_AAARSD baldwin_j_Page_082.tif 47782daf85b65870a2e69e208cb9fc82 0ed14fd059636ba394e21b0dd288434f3e4ebd85 F20101206_AAARRO baldwin_j_Page_064.tif 7bf1c0aa9ae923cc75e6367905b401ce 4df02115ce661f7960e19c8b3ac46ee23eda8c8d F20101206_AAARSE baldwin_j_Page_083.tif 3f4c416d223edadc90d59a9ffcb9a0b2 eea2aa166c2ea8a94606a7a39082194884a818b5 F20101206_AAARRP baldwin_j_Page_065.tif 4053f7f22ac3b6bbcd44f547b6623ea2 36d57c18f7f974aeea847226fa45fa7dcf2830c8 F20101206_AAARSF baldwin_j_Page_085.tif 892e7fa42fa951056c6c1f82ca57441b 60152cf7fe933f3003548b2b847479524a19b38f F20101206_AAARRQ baldwin_j_Page_067.tif 9bc340eb0127bfe2a2393583dde5a213 67702ff9093c35654eefa963c022b1eaca195e56 F20101206_AAARSG baldwin_j_Page_086.tif f9afc73e40f3e9f0fe1ceedb22bf2a84 8324eff9dc8aebaf28bcc9aa9f22997f3da1a0be F20101206_AAARRR baldwin_j_Page_068.tif 04d4767c2092d78ecba34ea5be7cf4c6 05370d13f0c89961ce5ef158e1205987a11b8dbf F20101206_AAARSH baldwin_j_Page_087.tif e383e6947d9fe3259a9c4b9ad180090a db7872999e08e3c67a24b9b57edf2086cb8232f7 F20101206_AAARRS baldwin_j_Page_069.tif 6f629e4295e98bf19cb56d88b74d84a3 9922833ad2e6b83bd1da4467452922a56a24a3cb F20101206_AAARSI baldwin_j_Page_088.tif bd5ce603fcbf97aeaa624421eedc1e5a 9fbe4f0f308f9ac50013c4dcfa9e5f2014d17a0d F20101206_AAARRT baldwin_j_Page_070.tif ec26b7703099d484f46686ed171782b4 d2ee43ebff57f346e417923a59c21300c0349519 F20101206_AAARSJ baldwin_j_Page_089.tif a58c31212f326531310f39320dc11804 2d56fff93395a1f9004a8d7f0775547dabc88112 F20101206_AAARRU baldwin_j_Page_071.tif 8af02fa4215ac9f3d6bae47e59d68471 48432578280f5484f13b32da4892868aa50b717d F20101206_AAARSK baldwin_j_Page_090.tif 4a85645f0375c2f8a950020577c805ca 27a8e52cf61179e73299c17e7bfab91827d276bf F20101206_AAARRV baldwin_j_Page_072.tif e3344f5a9b447b6ff9a574d9809ca90c 95839537653e21c16de388b0f801ff6aa9a7345f F20101206_AAARSL baldwin_j_Page_091.tif 8be5b57b9fce34b7b7291396795b933c e197904cae462fd3fa9b95464447397e1ee1f93f F20101206_AAARRW baldwin_j_Page_073.tif 09a356a828551b1c6d5e0645e1b5e0b3 29d452eb9b8dbc11ee7988656450cdefb8d8d6eb F20101206_AAARTA baldwin_j_Page_110.tif 04c4e7d074e1c9218287d1d3574af777 d0b51619a2b59aa867fee0bf527a2d5bbd6e3327 F20101206_AAARRX baldwin_j_Page_075.tif 9f9cfee3a48bc0eeaa1ae8554d0e60fc 07ab3fa944dbe974c099de5e0c639f9c7ba6322f F20101206_AAARTB baldwin_j_Page_111.tif d27edd6e0ad3ee4944bb4c245de7eee7 63ff53bfb2c0337a33df565dd6e9e93cf5931d9d F20101206_AAARSM baldwin_j_Page_092.tif a69aa5621ee0df614ec594b3836a74ef cb85e191d7055c1c99401803d8d5dec6e55aaedd F20101206_AAARRY baldwin_j_Page_076.tif 2f7b265203d6fb87bfce7a9cdc17f992 f2836e55489f8277fb3be30ee7e67af0749ac22c F20101206_AAARTC baldwin_j_Page_112.tif e6bc963d93b65e6df78045ebefa142ae 71353fc817b0edce138f13e51ab322cf00540e95 F20101206_AAARSN baldwin_j_Page_094.tif 6f63a5e4ca38e4514ab63ca792dda0f4 863cb193baaeca87ae8edaa4ff390631eb795ad6 F20101206_AAARRZ baldwin_j_Page_077.tif e32bc9bf4b087fa8c2e6efb68f77b969 561d6b396e7e3067761fa51bdb3e3db199b64408 F20101206_AAARSO baldwin_j_Page_095.tif 30596cfbda2b9d9671b337488f3a8996 f8254df7057ee8915dba333ee2ee402be69118ac F20101206_AAARTD baldwin_j_Page_113.tif b4622d62734be0a7b4dc88133e29f12e ae5d731c436e53be2921ae588e48b6ee3fd41d18 F20101206_AAARSP baldwin_j_Page_096.tif e21e96b9e4fc05f13bdb40139f3c3b1a 69309215da7b3c13f7a673113a925a148ab41cb5 F20101206_AAARTE baldwin_j_Page_114.tif a21187b1192fa804b8b7a52173ba95bf 05d8f45f8d07938a333b7c54c297297c4a7711ef F20101206_AAARSQ baldwin_j_Page_097.tif 6222897428146d63f5dbc4656d41398f b1b52adede0e63b461aba243d966dfad8e27be5c F20101206_AAARTF baldwin_j_Page_115.tif 809503a0b35f2b41772582e167b7bfdc 4bb921339c3e254f63f7b2ac20e11d280368a934 F20101206_AAARSR baldwin_j_Page_098.tif 1267d9abccdc276b3414c6446755b2db 3fff79e295d570fced4c6b96f7d1c3cbbcd785c5 F20101206_AAARTG baldwin_j_Page_116.tif 96362725067be1b7e257f905b90cde69 a4fa13dbcdf8e0682edb3ac755cedef4c7f5a326 F20101206_AAARSS baldwin_j_Page_100.tif 748626160b49bd5a3a4294bc31f5137a 9bf78013878ea923eecf0c639093d69a18498e47 F20101206_AAARTH baldwin_j_Page_117.tif fa2b237fb2219d37cadfcfa00228f6ac 45eadcbb7cc3629762bfd13a1209653d5e4fc9a9 F20101206_AAARST baldwin_j_Page_101.tif 04e507b64af335a351d0fc3cd4c4bb27 233e1f8c66282ba6a0748c3324cd93c5b6e5bf9d F20101206_AAARTI baldwin_j_Page_120.tif 5861780dd660d733d99cbd42f858da07 6d84f6771574e8b84f61b32caa0bdf354d73a64a F20101206_AAARSU baldwin_j_Page_102.tif da706d1ec75635d49df084cd0e78ce35 45180827f2aca0e42dbf0eedefef6a0b05af3566 F20101206_AAARTJ baldwin_j_Page_122.tif ab52ef60263c5cda85e64d7fa197507d df30ebfd9e9314994241ce00acd95d528e04bc98 F20101206_AAARSV baldwin_j_Page_103.tif 56698478eb09e5419dbc148cf8f0b669 f0969279c9cbcf4cf5a84a43382d816011f2ac29 F20101206_AAARTK baldwin_j_Page_123.tif e93d888ab4767ddd19a6ae3916a08ab2 5478ba5dfc4c4c5151cd9f3c3245ae158fde3f1d F20101206_AAARSW baldwin_j_Page_104.tif 81bf386152b2d91fa1cd9763159cd836 9dc742bc864d7ddbe3c746e193fbc997b57db8cc F20101206_AAARTL baldwin_j_Page_124.tif 502e8185032e95736493769b09045ea9 bd66e4245eeb7ca31c8b078ccdd5a06048bafd68 F20101206_AAARSX baldwin_j_Page_105.tif e17caaca5395fe9526e75361729272b5 c987676abd6ac3d564ab760eaaff6ea42d4baaab F20101206_AAARUA baldwin_j_Page_145.tif 23d5f88643e0ba9d709e2d4f4c300531 620d5721d7959e3eeb4a78cb8d4174f0226e3e60 F20101206_AAARTM baldwin_j_Page_127.tif d4b3b06ad1ef298df76627beaa102df9 e77c0890c5642579b1098c874651fdb1310cdf56 F20101206_AAARSY baldwin_j_Page_106.tif 7fd8f40af98c0002077c0c941100c88f 13de4428c105cd7761a7362ae2c548162af7e1a7 F20101206_AAARUB baldwin_j_Page_146.tif eac69273d52a674a6a4297773570511f 486b6dad2ac63f9c80cf3afea90386ceb1402d35 F20101206_AAARSZ baldwin_j_Page_108.tif 13b3c435d385bb63c8b37252cc643cf0 1364b43d5d655f2b87eb89b18bf71da7d1b01c78 F20101206_AAARUC baldwin_j_Page_147.tif 0a0f0b2082c5fbfcd36976d5319b1384 7f74b4b0cc64d056a443e5b2461764b1c82f47f4 F20101206_AAARTN baldwin_j_Page_128.tif 07a6d3f5a27f43e41937e5d0b7cd66eb 558489eead4fe90d0379ba55850dfcfcc4935d17 F20101206_AAARUD baldwin_j_Page_148.tif 499492a6c507040423cd20b887c648ff c2187524b99a0918c443552dbe6589e0a9d06f05 F20101206_AAARTO baldwin_j_Page_129.tif 2f0d02cc4e60271e0121898ab6fb6017 04a3b077de754b13d76d5b1d3eb08fce7f0f85cb F20101206_AAARUE baldwin_j_Page_149.tif b8383e07c9eb7924e15de1e9cfce270c 4151bdb140168089ce266804613091a8dd95505a F20101206_AAARTP baldwin_j_Page_130.tif ea0dd9efe5c7a363e973f91d5345c489 590cdcb2e3fc49317ef5941a17daccd468a27e3f F20101206_AAARUF baldwin_j_Page_151.tif 458078e57b8ebc325b7dc32c2ae2b7fc dcb3cdf7e38f173c97cad6929d4c8460770739f8 F20101206_AAARTQ baldwin_j_Page_131.tif 570c17db0a3e781ab9a054284cd2107e 077dcddb950062f1b2d4a8da4727e508d9b0dc6c F20101206_AAARUG baldwin_j_Page_152.tif 7d2baba87b4b26d6f457464ae5732eb3 db7c087fae5c47ab6e3c7a38e0b209b35d26e5fa F20101206_AAARTR baldwin_j_Page_132.tif 30e64e7dd3314223bf84fd104c427083 032ef90ddbe64c4dc02962716a02d72268f897a3 1898 F20101206_AAASAA baldwin_j_Page_018.txt 89b4c191a7f6d97131939984c191f941 498b6d5c68bd72f9ee26dd5d1fe50955203f1bf6 F20101206_AAARUH baldwin_j_Page_154.tif 8de58eb10da3a25697bad5fa8c764655 6ebcdb1cb7817b5d2011ac4ea8055a77a4d9145b F20101206_AAARTS baldwin_j_Page_133.tif ea1c2d275fbbeaeaac4bb08767a4998b 79706e12ec88e73b12c95040dd376ac75d338431 1523 F20101206_AAASAB baldwin_j_Page_019.txt db9f8ee495c16da3127d5f156c575db3 effca14b788fea8e975fa2f37a467a4630ed23b8 F20101206_AAARUI baldwin_j_Page_155.tif ff1e4619eb57ff54505b8a26063b828e 4784ad38b5a23ca5624b6577169e8b0f13daceaa F20101206_AAARTT baldwin_j_Page_134.tif 07154af899bc90cb73164af5155deac0 80943caa782deea4efa97ebb613397242faf5de1 1827 F20101206_AAASAC baldwin_j_Page_020.txt 8a350af2e74431322f08db1a1a29018f 999585a0e04fa699ccb4d7b9e3cf08a93539d732 F20101206_AAARUJ baldwin_j_Page_156.tif 7abf98f759d0526060177dd281d2a751 224cd3517affa555449f1ad74b70cc34573ebaf2 F20101206_AAARTU baldwin_j_Page_135.tif 904ba360590b26e9e788eca6f7e158df f00343d3b1f4d35522e3e1f84d0e89779f82be7b 1852 F20101206_AAASAD baldwin_j_Page_022.txt 1c3f045307a36c9aeb6a5f317028d037 902c5ed53ff94289a2c351f6e70875b0b2fd9e18 F20101206_AAARUK baldwin_j_Page_157.tif e95bf82833d5960735bbb8a95e3b56e9 8618366f97e3a4743b3122f06c40183d5b7734ec F20101206_AAARTV baldwin_j_Page_136.tif 7a118e038b0de52da3e8e8a275a88271 91cbfc931b29661e9f8f0e5a67e22ea2b0d75c4f 1569 F20101206_AAASAE baldwin_j_Page_023.txt 4ab160e87469c1a6fc707c3ba155d4d0 4d7fd4430aa4fbc2f30b81f6cccf2cb3fbdcc3cf F20101206_AAARUL baldwin_j_Page_158.tif 6a7b7b201d568631536ab6e576b4dc37 e15acfa0c07a03e7a13f8f84782571209fca6eb2 F20101206_AAARTW baldwin_j_Page_138.tif 9f7777e4e6283bc863a4791cb6b6c280 ff386273c5d0fb637c4082243bfcd9b97999e298 1680 F20101206_AAASAF baldwin_j_Page_024.txt 0b6e5a6e7ec8c1154e1bde05915684fe 738253f21ceabc47602d74ad19b01c44ef95aa90 48061 F20101206_AAARVA baldwin_j_Page_015.pro ecfaefc2d992473e91fd3d8a965a8109 f21273dc3a77b4dbc01f835432a2a6241137509b F20101206_AAARUM baldwin_j_Page_159.tif 26770c39843ebce7e9355eb98d0e0edb 59fb0badbb1df6f3100ed0c8f183db7478154a5d F20101206_AAARTX baldwin_j_Page_140.tif 6d078bdd20289f57e62b6341e278fb3b 47c3368b91429a810a8830660a11a79cf9e532a1 F20101206_AAASAG baldwin_j_Page_025.txt da18c6afcc44d0b926f3ce9248560744 79b1098ea80190e2b0bf9b65a3759b72f70d884d 42288 F20101206_AAARVB baldwin_j_Page_016.pro a0de6f4cc27f80d29de80bada9b7d824 1bf1172006e4460b1e330c1a519510c33b5e63a0 F20101206_AAARUN baldwin_j_Page_160.tif acf3aa5c4c2ca47b5a308404ff20edbd 0aea99c991c29d968bfccbf6b3c517bb2d3d0cff F20101206_AAARTY baldwin_j_Page_142.tif c2b0dc6f161196f4ecfdaeec6028a5c9 efb3893dd9fe7ca8ff2614af02a64aa64cb4faeb 1499 F20101206_AAASAH baldwin_j_Page_026.txt e52a8dd43ab146bc4f651f4c1482c149 34850b693c7c4c72dfce05ef1b73d22f53150319 47998 F20101206_AAARVC baldwin_j_Page_018.pro 8a3b2315331ae7832dbf6821acfda90c ee0df4c245d69e53159b431810f649df60fa597f F20101206_AAARTZ baldwin_j_Page_144.tif dead99b7d450119048b0b6c7426833c4 601da4a53ed5b28d5518db1cc60dea7379d1741f 1780 F20101206_AAASAI baldwin_j_Page_027.txt a9ea11d68e615d500ab22fdc9469f500 f69c3ed4539067360f75fa5d39afd062997027e1 32934 F20101206_AAARVD baldwin_j_Page_019.pro 793adc5f709f2b65ce10a29a3b7332ea 554a0fb1cd0653ee99eac8d3de405fd09c946a1b F20101206_AAARUO baldwin_j_Page_161.tif ded4533cc3130cc9b5c811fd4e683b92 34fb2d66799d7b92a1ee541b9649af6a1c0a8bdb 1538 F20101206_AAASAJ baldwin_j_Page_028.txt 3fa70f0fdc86372f443df957f6c1d3cc 275ef56e200e047908a4420c620718d4d4ef7557 37216 F20101206_AAARVE baldwin_j_Page_021.pro ffe0917dbc5b58960c057650891f2e76 0de8ac6f21f4973a267e6c31e06b103aaabda571 F20101206_AAARUP baldwin_j_Page_162.tif be963dcd4cb2e13b3dc3638718889d42 292cab3af83ceac9d1ebd2407c6eb1a5190e16b7 1811 F20101206_AAASAK baldwin_j_Page_029.txt 0614c1ec25e13e3b2883681aa5d23632 229685c8f79c43364cb2cb458b93cd3af101fa65 41121 F20101206_AAARVF baldwin_j_Page_024.pro db10cc59255df6563e7db3749fbe9003 a51b25829d3a59a82ea3f1e3192fb6c44ca94711 9140 F20101206_AAARUQ baldwin_j_Page_001.pro ca0c7f44c75ec973729638ed1acb310b 06f4cbe38602c992945e81fa803d4f52e468b148 1561 F20101206_AAASAL baldwin_j_Page_030.txt dd288a469949b3218a03f079645332bd 11c926204d40642d4a68ca3ce6a028e51e3c3bf4 896 F20101206_AAASBA baldwin_j_Page_048.txt f7c7a29d08ad2891925c4ea4e4780877 dbcba3eb8b6555d4ab49eda9902041bdf1709178 33046 F20101206_AAARVG baldwin_j_Page_025.pro 43361e44ed50e87ac1b822f082c9732e 7cc5b52c01f78f9d821d6abeda5ee02eb290c933 2238 F20101206_AAARUR baldwin_j_Page_003.pro 5f5cf150cabbad3b152808f917a7f07f 39f0d6394dead266f285af3ced9ce3a594c2e969 1123 F20101206_AAASAM baldwin_j_Page_031.txt 3a92c2bcc4a18942d8ca4c11af41a688 f4d0b277cc1eaa242c46565ba26655f9b1a94180 833 F20101206_AAASBB baldwin_j_Page_050.txt 0aa2f0abe4377cbb4ec0436daad7c29b 2e56112a7da7d1905e538a9b9985d0d7b07fc261 34639 F20101206_AAARVH baldwin_j_Page_026.pro dbb3275ed4490bdbd0007533b15031d7 f230af84f6a3e88a764e565add8a989b58d37e18 54759 F20101206_AAARUS baldwin_j_Page_004.pro 6b371fe3c69bb6738c3b090789c5f33e f308149e03d5a6ca43c03b39c6fd29f15c41c320 1626 F20101206_AAASAN baldwin_j_Page_032.txt 1268e6f0190e1e7c39aa7da7f24673bd a17814bd52da23feaaf4e1bfd4f199f96b25e134 1845 F20101206_AAASBC baldwin_j_Page_051.txt 82af01c93219cf46b21f7fbb14afdf28 af9c3242ef431c8d61a2939683a0eb2746d396b6 1253 F20101206_AAASAO baldwin_j_Page_033.txt 2e10b159e722f6e08cc8854fd35f527c 4c000a2fb6616214131abe569fb3504ab0bda5dc 38593 F20101206_AAARVI baldwin_j_Page_027.pro ebc8f0ee97dab42dd69d1722f383145c af0c9673ddd33fbd1281c0844ebc87b3d40881e8 51364 F20101206_AAARUT baldwin_j_Page_005.pro ad133d361a5e04ab17ea4e84dbee760c d661bb8ed2c104a495663b8c3eb849886cc54b23 847 F20101206_AAASBD baldwin_j_Page_053.txt 6270daf01b78b1b6ef5795cfa5a2198a 4d3fded8c79401d4529ab7371e667528b7cab033 1824 F20101206_AAASAP baldwin_j_Page_035.txt 8d9b28ee75337d9c03294ba625c6eddb e139ce220de8dc08ac539a5e158a18d30c1d8467 29223 F20101206_AAARVJ baldwin_j_Page_028.pro 0bb111974a32585f45830c7885f83eae e27d3f1d383a54928fed57c8a08c83cde882d05f 66567 F20101206_AAARUU baldwin_j_Page_007.pro 85b517d0d4733b0f4a994019c23c3a3e ead7eb6f7f0af052526fad584bf05347567ad56d 872 F20101206_AAASBE baldwin_j_Page_054.txt 62da07a3ee979ebd3ef5713e2c03636c a8638ba15e6b1fbed68f931d7fbb4a6456782bbb 1373 F20101206_AAASAQ baldwin_j_Page_036.txt 7e8e6f94d8cf8ea1e3ce155d4f0dc829 d8f4a0bcc1b2d7f3bff2eceef4868f61565d3224 45663 F20101206_AAARVK baldwin_j_Page_029.pro 1f8861dbaa9f2b9990a87edc68273dba 115135e3609e9054c66bcf5693cd73df0971037d 51629 F20101206_AAARUV baldwin_j_Page_008.pro bae49ecd27ad2e164ab36dd3df4aa94e 09904f9162027eee7953ea87edc63d08df7e624c 1437 F20101206_AAASBF baldwin_j_Page_055.txt 71a67c0b1c392959afa260908e75f708 44f19c77f60d99ae077746ec60d138bd2670119c 1832 F20101206_AAASAR baldwin_j_Page_037.txt 0af807fea853e7a70c338341c5c0fc0c 7f19f01e27a41180f46cfa25835752ba526de27d 27985 F20101206_AAARVL baldwin_j_Page_032.pro d912b7a878661dbf0f82ee31693a11de d6a3a2c393076a393bd3987263eba12ef72de940 54557 F20101206_AAARUW baldwin_j_Page_010.pro 2b61295f29a2617202615899f57b4c47 6ed592c9ad931cdeb25047a2578cb350e3ae796a 