<%BANNER%>

Evaluating Adjustments to the Mean Squared Error due to Estimating Variance Parameters in Linear Mixed Models

Permanent Link: http://ufdc.ufl.edu/UFE0021619/00001

Material Information

Title: Evaluating Adjustments to the Mean Squared Error due to Estimating Variance Parameters in Linear Mixed Models
Physical Description: 1 online resource (162 p.)
Language: english
Creator: Baldwin, Jamie Mcclave
Publisher: University of Florida
Place of Publication: Gainesville, Fla.
Publication Date: 2007

Subjects

Subjects / Keywords: eblup, linear, mixed, model, msep
Statistics -- Dissertations, Academic -- UF
Genre: Statistics thesis, Ph.D.
bibliography   ( marcgt )
theses   ( marcgt )
government publication (state, provincial, terriorial, dependent)   ( marcgt )
born-digital   ( sobekcm )
Electronic Thesis or Dissertation

Notes

Abstract: Several methods are commonly used to handle the propagation of variance stemming from estimating unknown variance components in Linear Mixed Models (LMM). The accuracy and consequences of using these methods, however, have not been thoroughly investigated. Empirical Best Linear Unbiased Predictors (EBLUP) for analyzing LMMs are widely used, yet the best way to evaluate the precision of the EBLUP is not generally understood. Many developments in the estimation of the Mean Squared Error of Prediction (MSEP) of the EBLUP, and the use of these estimates for hypothesis testing, have occurred during the last two decades. This dissertation begins with a thorough review of these developments. Existing methodologies for evaluating the precision of the EBLUP are generalized to apply to multiple dimension linear combinations of fixed and random effects, and the definiteness properties of these methodologies are examined. The methods for evaluating the MSEP of the EBLUP are examined thoroughly for the balanced one-way random effects model to assess the accuracy of the methods, the effect of the parameterization of the model, and the impact of variance parameter estimation techniques on the components of the MSEP estimators. The impact of negative variance component estimates on EBLUP methodologies is examined, and an alternative solution to account for negative variance estimates is developed for the one-way model.
General Note: In the series University of Florida Digital Collections.
General Note: Includes vita.
Bibliography: Includes bibliographical references.
Source of Description: Description based on online resource; title from PDF title page.
Source of Description: This bibliographic record is available under the Creative Commons CC0 public domain dedication. The University of Florida Libraries, as creator of this bibliographic record, has waived all rights to it worldwide under copyright law, including all related and neighboring rights, to the extent allowed by law.
Statement of Responsibility: by Jamie Mcclave Baldwin.
Thesis: Thesis (Ph.D.)--University of Florida, 2007.
Local: Adviser: Littell, Ramon C.

Record Information

Source Institution: UFRGP
Rights Management: Applicable rights reserved.
Classification: lcc - LD1780 2007
System ID: UFE0021619:00001

Permanent Link: http://ufdc.ufl.edu/UFE0021619/00001

Material Information

Title: Evaluating Adjustments to the Mean Squared Error due to Estimating Variance Parameters in Linear Mixed Models
Physical Description: 1 online resource (162 p.)
Language: english
Creator: Baldwin, Jamie Mcclave
Publisher: University of Florida
Place of Publication: Gainesville, Fla.
Publication Date: 2007

Subjects

Subjects / Keywords: eblup, linear, mixed, model, msep
Statistics -- Dissertations, Academic -- UF
Genre: Statistics thesis, Ph.D.
bibliography   ( marcgt )
theses   ( marcgt )
government publication (state, provincial, terriorial, dependent)   ( marcgt )
born-digital   ( sobekcm )
Electronic Thesis or Dissertation

Notes

Abstract: Several methods are commonly used to handle the propagation of variance stemming from estimating unknown variance components in Linear Mixed Models (LMM). The accuracy and consequences of using these methods, however, have not been thoroughly investigated. Empirical Best Linear Unbiased Predictors (EBLUP) for analyzing LMMs are widely used, yet the best way to evaluate the precision of the EBLUP is not generally understood. Many developments in the estimation of the Mean Squared Error of Prediction (MSEP) of the EBLUP, and the use of these estimates for hypothesis testing, have occurred during the last two decades. This dissertation begins with a thorough review of these developments. Existing methodologies for evaluating the precision of the EBLUP are generalized to apply to multiple dimension linear combinations of fixed and random effects, and the definiteness properties of these methodologies are examined. The methods for evaluating the MSEP of the EBLUP are examined thoroughly for the balanced one-way random effects model to assess the accuracy of the methods, the effect of the parameterization of the model, and the impact of variance parameter estimation techniques on the components of the MSEP estimators. The impact of negative variance component estimates on EBLUP methodologies is examined, and an alternative solution to account for negative variance estimates is developed for the one-way model.
General Note: In the series University of Florida Digital Collections.
General Note: Includes vita.
Bibliography: Includes bibliographical references.
Source of Description: Description based on online resource; title from PDF title page.
Source of Description: This bibliographic record is available under the Creative Commons CC0 public domain dedication. The University of Florida Libraries, as creator of this bibliographic record, has waived all rights to it worldwide under copyright law, including all related and neighboring rights, to the extent allowed by law.
Statement of Responsibility: by Jamie Mcclave Baldwin.
Thesis: Thesis (Ph.D.)--University of Florida, 2007.
Local: Adviser: Littell, Ramon C.

Record Information

Source Institution: UFRGP
Rights Management: Applicable rights reserved.
Classification: lcc - LD1780 2007
System ID: UFE0021619:00001


This item has the following downloads:


Full Text





EVALUATING ADJUSTMENTS TO THE MEAN SQUARED ERROR DUE TO
ESTIMATING VARIANCE PARAMETERS IN LINEAR MIXED MODELS




















By

JAMIE MCCLAVE BALDWIN


A DISSERTATION PRESENTED TO THE GRADUATE SCHOOL
OF THE UNIVERSITY OF FLORIDA IN PARTIAL FULFILLMENT
OF THE REQUIREMENTS FOR THE DEGREE OF
DOCTOR OF PHILOSOPHY

UNIVERSITY OF FLORIDA

2007

































2007 Jamie McClave Baldwin




























To my husband, Ian, and to my children, Caryss, Zachary, and "Baby" Baldwin









ACKNOWLEDGMENTS

First and most importantly, I thank my husband, Ian Baldwin, for providing endless

patience over the last 10 years while I have pursued this degree. I would not have been able to

endure this process without his unwavering support and constant belief in me, even when I

couldn't believe in myself. I thank him for maintaining both the house and my sanity when

times were hard and for making me smile when days were too long to manage. I express my

eternal love and gratitude to my children, Caryss, Zach, and Baby Number 3, for their infinite

supply of laughter and hugs. They have inspired me to push beyond any limitation and live life

to the fullest. They are the lights of my life and the most beautiful souls I have ever encountered

and I give thanks for them everyday. My family is a daily demonstration of God's Grace; there

is nothing I could ever do to deserve their unconditional love and support.

I thank my parents, Jim and Mary Jay McClave, for their hard work, love, patience, and

constant support. To my father, I am so grateful to have such a wonderful role model, both

professionally and personally. I am forever indebted to him for continual editing and

mathematical advice. I am so fortunate to have the opportunity to work with the smartest and

bravest man that I know, not to mention the best consultant in the industry. To my mom, I am so

thankful for her wise parenting advice, too many hours of childcare to even count, and girl time

when it was most needed. I literally could not have finished this monstrous endeavor without her

emergency child care services. I also thank them for all of the generosity they have shown me

and my family in so many ways.

I would also like to thank my brother, Will McClave, his wife, Amber, and my awesome

nephews, Jake and Luke. They have been a constant support for my family and I look forward

to the next chapter of our lives together. I am so blessed to have such a wonderful, supportive

family that lives so close by. It is a rarity in this world which I truly appreciate.









To the best friends anyone could ask for, I am thankful for their friendship, love, and

encouragement through this entire process. I thank Holly Holly for always helping me see the

bright side, laughing at my stories, listening to my endless complaints, and allowing me to learn

so much about motherhood and balance from her wonderful example. I thank Heather Bristol for

being our pioneer through the statistics program. She has shown me what true bravery is and

taught me how to deal with the most unexpected of situations with grace and strength. I thank

Dr. Patches Johnson for encouraging my continuation when I thought I would go no further, and

for being my mentor in so many ways. I thank her for being a fantastic and fun traveling

companion and for her contagious laugh. I thank Meghan Medlock Abraham, my college

roommate, for a friendship that picks up right where it was left off no matter how much time has

passed and for having a mouth big enough to eat her own fist. I thank Brooke Bloomberg, my

"oldest" and dearest friend, for her sisterly bond and for beating me to the finish line; it was all

the motivation I needed! It is amazing that we have both accomplished our childhood ambitions.

We rock!!

I am grateful for other friends that have helped me along the way with their support and

laughter. My sincerest appreciation goes to Andy and Tracy Bachmann, Eric and Ann

Bloomberg Beshore, Nicole Provost and Kathy Carroll, Debbie Hagan, Lisa Jamba, Ron and

Cindy Marks, Larry and Sandy Reimer, and Yvette and Scott Silvey. I consider each one of

them family and I am so lucky to have them in my life.

Finally, I thank Dr. Ramon Littell for his patience and guidance during this seemingly

endless project. Without his loyal support and enduring belief, this dissertation could not have

been completed. This degree would not have been possible without his wealth of knowledge,

generous spirit, and encouragement.









TABLE OF CONTENTS

page

A CK N O W LED G M EN T S ................................................................. ........... ............. .....

L IST O F T A B L E S ...................................................................................................... . 8

LIST OF FIGURES .................................. .. .. .... ..... ................. .9

A B S T R A C T ............ ................... ............................................................ 12

CHAPTER

1 INTRODUCTION AND BACKGROUND ........................................ ....................... 14

1.1 Introduction to M ixed M odels ............................................. ............................. 14
1.2 T he L near M ixed M odel ................................ .. ........... ................................. ...... 15
1.3 Best Linear Unbiased Predictors (BLUPs) When 0 is Known: Mixed Model
E q u atio n s ........................................................................ .. 1 5
1.4 Empirical Best Linear Unbiased Estimation (EBLUP)...............................................16
1.5 Two-Stage Estim action and Unbiasedness ............................ .....................17

2 L IT E R A TU R E R E V IE W ........................................................................ ... ...................... 19

2.1 Precision of EBLU Ps ............................. ...... ................... ... .......... ...... .....19
2.1.1 Mean Squared Error of Prediction (MSEP) of EBLUPs...............................19
2.1.2 Kackar-Harville Approximation to MSEP.............. .... .................22
2.1.3 Estim eating C ( ) .................... .................. ........................ ............... 24
2.1.4 Estim eating M ( ) .................... ......................................................... 25
2.1.5 Estimators of M (0) Based on the Kackar-Harville Approximation ..............26
2.2 Hypothesis Testing for Single Dimension Linear Combinations ..............................27
2.3 Hypothesis Testing Methods for Multiple Dimension Linear Combinations ..............29
2.3.1 Kenward-Roger Multiple Dimension Hypothesis Testing Method ...............29
2.3.2 Fai-Cornelius Multiple Dimension Hypothesis Testing Methods ....................35
2.4 Negative Variance Component Estimates ............... ............................................. 38

3 ESTIMATES OF MSEP FOR MULTIPLE DIMENSION LINEAR COMBINATIONS ....42

3.1 Reconciling Kenward-Roger (1997) and Fai-Cornelius (1996) Definitions of the
Correction Term Approximation...................... ......... .......................... 42
3.1.1 Linear Combinations of Fixed Effects................ ............................................42
3.1.2 Linear Combinations of Fixed and Random Effects.......................................44
3.2 M multiple Dimension Bias Correction ................... ... ....................................... 49
3.2.1 Bias Correction Estimator for Multiple Dimension Linear Combinations of
F ix ed E effects ................................................... ................ 4 9









3.2.2 Bias Correction Estimator for Multiple Dimension Linear Combinations of
Fixed and R andom E effects ........................................................... .................. 51
3.2.3 Definiteness Properties of the Estimator for MSEP for Multiple-
Dimension Linear Combinations of Fixed and Random Effects ...................59
3.3 Transform Invariance of M SEP Estim ator......................................... .....................61

4 BALANCED ONE-WAY RANDOM EFFECTS MODEL.............. ............... 63

4.1 Kackar-Harville Approximation Based on REML Estimation Method......................64
4.2 Kackar-Harville Approximation Based on ANOVA Estimation Methods .................75
4.3 Bias Correction Term for the Balanced One-Way Random Effects Model..................84
4.3.1 Bias Correction Term Approximation under 0 Parameterization .....................84
4.3.2 Expected Value of BIAS" (0) with REML Covariance Parameter
E stim action ................................. ........................................... ........ 87
4.3.3 Expected Value of BIAS" (0) with ANOVA Covariance Parameter
E stim action .................................... .... ... ... ....... .. ...... ............... 92
4.3.4 Impact of Negative Values of Bias Correction Term Approximation..............95
4.4 Performance of Overall Estimators for the MSEP....................................................98
4.5 Comparison of Prediction Interval Methods .......... .................. ..... ...........101

5 THE EFFECT OF NEGATIVE VARIANCE COMPONENTS ON EBLUP ......................137

5.1 Negative Variance Component Estimates and the Balanced One Way Random
Effects M odel ...................... ............................... ......... 137
5.2 Considering the Variance Parameter as a Covariance ............................................. 141
5.3 BLUP derivation for Random Effects Model with Correlated Errors ......................143

6 CON CLU SION AND FUTURE W ORK .................................................. .....................148

APPENDIX DETAILS FOR MONTE CARLO SIMULATION STUDY ON PREDICTION
IN T E R V A L S ............................................................................ 152

L IST O F R EFE R EN C E S ..................................................................... .......................... 159

B IO G R A PH IC A L SK E T C H ......................................................................... ... ..................... 162









LIST OF TABLES


Table page

2-1 Summary of developments for MSEP estimation and hypothesis testing ....................40

4-1 Accuracy of Kackar-Harville estimator (4-17) for CR (0) ............................................71

4-2 Relative bias (% ) of estimators of correction term .................................... ..................84

4-3 Relative bias (%) of MSEP estimators under REML and ANOVA estimation ..............100

4-4 Summary of prediction interval procedures.................. ................... ...................102

4-5 Effect of prediction interval procedures for k = 6, n = 6, !2 = 1 and ^2 = .5 ................108

5-1 Generated data that produces negative variance component estimate .............................138

5-2 REML variance parameter estimates...............................................138

5-3 Solution for random effects with REML variance parameter estimates ..........................138

5-4 REML NOBOUND variance parameter estimates ........... .............. ...............139

5-5 Solution for random effects with REML NOBOUND variance parameter estimates.....139

5-6 LSM EAN S from PRO C GLM ............................................... .............................. 139

5-7 Data generated from distribution in M odel (5-1).................................. ............... 145

5-8 Covariance parameter estimates under Model (5-1).................... ............................. 146

5-9 Solution for random effects under M odel (5-1)................................... ............... 146

5-10 LSM EAN S from PROC GLM ............................................... .............................. 146

5-11 Variance parameter estimates under random effects model ........................................147

5-12 Solution for random effects under random effects model ............................................147








LIST OF FIGURES


Figure page

4-1 Equation (4-13) as a function of v for several values of k, holding n = 6. Equation
(4-14) holds when functions cross zero line. .......................................................109

4-2 Accuracy of approximations tr [A (0) B (0)] and tr [A (0) (0)] for the
correction term, CR (0), under REML variance component estimation .....................110

4-3 Accuracy of approximation tr [A(0)BR (0) and estimator E tr A(0)BR (O)] for
the correction term, CR (0), under REML variance component estimation..................111

4-4 Accuracy of approximation tr [A(0)I1 (0) and estimator E{tr A( )I1 (o)]) for
the correction term, CR (0), under REML variance component estimation............... 112

4-5 Comparison of accuracy of estimators for the correction term, CR (0), under REML
variance com ponent estim ation......... ............................ ............... ...............1.13

4-6 Comparison of accuracy of approximations tr [A () 11 (0)], tr [A(0) B (0)], and
tr [A(0)BA (0)] for the correction term, CA (0), under ANOVA variance
com ponent estim action. ...... ..... ......... ..................... ............114

4-7 Accuracy of approximation tr A (0)BA (0) and estimator E tr A(0)BA (O)] for
the correction term, CA (), under ANOVA variance component estimation .............115

4-8 Accuracy of approximation tr[A(0)B* (0)1 and estimator E tr A(0)B* (O)]) for
the correction term, CA (), under ANOVA variance component estimation .............116

4-9 Accuracy of approximation tr [A (0)11(0)] and estimator E tr A (o)1 ()]} for
the correction term, CA (), under ANOVA variance component estimation .............117

4-10 Comparison of accuracy of estimators for the correction term, CA (), under
ANOVA variance component estimation. ............. .................................................... 118

4-11 Accuracy of estimators for the correction term under REML, 0, and ANOVA, 0,
variance com ponent estim ation ......................................................... ....................... 119








4-12 Relative bias of estimators for the correction term under REML, 0, or ANOVA, 0,
variance com ponent estim ation ....................................................................................... 120

4-13 Bias produced by estimating M, (0) with M, (). ................................ ........... 121

4-14 Accuracy of approximation trA (0) I (0)1 and estimator

E --tr A(o)I-1 (o)]} for the true bias correction term, M, (0) E[M (-)1,
under REM L variance component estimation. .................................... .....................122

4-15 Accuracy of approximation tr[A(0)l1 (0)] and estimator E tr A(O)I 1)l) for
the true bias correction term, M, (0) -E M (O), under REML variance component
estim ation............... .... ........................... .............................................123

4-16 Comparing the accuracy of E tr[A(O)I1 (6)]} and E{tr A (0)-1(o)l for
the true bias correction term, M (0) -E[M ( )] ................................. ..................124

4-17 Bias produced by estimating M, (0) with M (). .................................................125

4-18 Accuracy of approximation tr A(0)I1 (0) and estimator

E tr A(6)I-1 (e)1} for true bias correction term, M, (0) -E[M, (O), under
ANOVA variance component estimation. ..........................................................126

4-19 Accuracy of approximation, tr[A(0)I 1 (0) and estimator E{tr A( )I'1 ()l} for
true bias correction term, M, (0) -EMI (0)], under ANOVA variance component
e stim a tio n ................................................................................................................1 2 7

4-20 Comparing the accuracy of E -ltr[A( ()I ( )1 and E{tr A(o) (9)1} for
the true bias correction term, M (0)- E M ( )] ............................................................128

4-21 Comparison of estimators of the MSEP for the EBLUP of a utilizing REML
variance com ponent estim ation....................................................... ........................ 129









4-22 Comparison of estimators of the MSEP for the EBLUP of a utilizing ANOVA
variance com ponent estim ation............................................................. ..................... 130

4-23 Relative bias of possible estimators for MSEP of EBLUP for .............. ...............131

4-24 Simulated coverage rates of prediction interval procedures for t = a, with k = 3...........132

4-25 Simulated coverage rates of prediction interval procedures for t = a, with k= 6...........133

4-26 Simulated coverage rates of prediction interval procedures for t = a, with k = 15.........134

4-27 Simulated coverage rates of prediction interval procedures for t = a, with k = 30.........135

4-28 Simulated coverage rates for the Kenward-Roger prediction interval procedure............136









Abstract of Dissertation Presented to the Graduate School
of the University of Florida in Partial Fulfillment of the
Requirements for the Degree of Doctor of Philosophy

EVALUATING ADJUSTMENTS TO THE MEAN SQUARED ERROR DUE TO
ESTIMATING VARIANCE PARAMETERS IN LINEAR MIXED MODELS

By

Jamie McClave Baldwin

December 2007

Chair: Ramon Littell
Major: Statistics

Several methods are commonly used to handle the propagation of variance stemming from

estimating unknown variance components in Linear Mixed Models (LMM). The accuracy and

consequences of using these methods, however, have not been thoroughly investigated.

Empirical Best Linear Unbiased Predictors (EBLUP) for analyzing LMMs are widely used, yet

the best way to evaluate the precision of the EBLUP is not generally understood. Many

developments in the estimation of the Mean Squared Error of Prediction (MSEP) of the EBLUP,

and the use of these estimates for hypothesis testing, have occurred during the last two decades.

This dissertation begins with a thorough review of these developments.

Existing methodologies for evaluating the precision of the EBLUP are generalized to apply

to multiple dimension linear combinations of fixed and random effects, and the definiteness

properties of these methodologies are examined. The methods for evaluating the MSEP of the

EBLUP are examined thoroughly for the balanced one-way random effects model to assess the

accuracy of the methods, the effect of the parameterization of the model, and the impact of

variance parameter estimation techniques on the components of the MSEP estimators. The

impact of negative variance component estimates on EBLUP methodologies is examined, and an










alternative solution to account for negative variance estimates is developed for the one-way


model.









CHAPTER 1
INTRODUCTION AND BACKGROUND

1.1 Introduction to Mixed Models

In 1984, Kackar and Harville introduced a new method to handle the propagation of

variance stemming from estimating unknown variance parameters in linear mixed models. In the

nearly quarter century since this development, there have been many adjustments, improvements

and new techniques resulting from their idea. Linear mixed models and the procedures used to

analyze them are full of complexities and nuances that can confuse even the most experienced

analyst. For example, complex covariance structures coupled with unbalanced data and variance

parameter estimates on or near the boundary of the parameter space can produce confounding

results with many mixed models procedures. This paper begins to unwind and isolate these

complexities using simple cases to explore the best options available and determine when

procedures for estimating the realized values of the random effects and evaluating the variability

of these estimates succeed and fail.

Mixed models methodologies have developed over time to accommodate situations

where prediction of the realized values of the random effects is required. These situations arise,

for example, in agriculture, education, and clinical trials. A typical agriculture example is the

need to assess the genetic contribution of a particular animal for breeding, which is considered to

be drawn from a population of animals. An example from education is the desire to assess the

effect of a particular teacher on the test scores of his or her pupils. A clinical trial with multiple

clinics and varying patient dropout rates gives another arena where mixed models are frequently

used to analyze data. Many of these analyses involve complex covariance structures and almost

all analyses involve unknown covariance parameters. It is important not only to develop

accurate methods for evaluating the realized value of the random effect, but also to evaluate the









associated mean squared error to determine the precision of the estimate. We begin by

describing the framework of the model.

1.2 The Linear Mixed Model

The general form of the linear mixed model is

y= Xp +Zu+e (1-1)

where X is a known (n xp) design matrix for the fixed effects, P is a (p x 1) vector of fixed

effect parameters, Z is a known (n x q) design matrix for the random effects, u is a (q x 1)

vector of random effects, and e is an (n x 1) vector containing the random error terms. We will

assume that E(u)= 0, V(u)= G, and E(e) =0, V(e)= R, cov(u,e)= 0. From the model in

Equation (1-1), it follows that E(y) = Xp and V(y)= V = ZGZ'+ R. The matrices G and R

are taken to have known structures but depend on an unknown dispersion parameter, 0 e 1l, for

some known parameter space 2 Multivariate normal distributions on u and e are often

assumed and many of the results are derived under normality assumptions; the validity of these

results under other distributional assumptions is questionable. For simplicity, all matrices are

assumed to be of full rank.

1.3 Best Linear Unbiased Predictors (BLUPs) When 0 is Known: Mixed Model
Equations

Best Linear Unbiased Predictors (BLUPs) originated from the desire to estimate the

realized values of the random effects along with the fixed effects, or predictable linear

combinations of both types of effects. This method was originally developed by Henderson

(1950) for estimating genetic merits in animal breeding. Henderson approached the problem

using maximum likelihood methodology, although he later acknowledged that his method cannot

be called "maximum likelihood" since the "function being maximized is not a likelihood"

(Robinson 1991, p.18). Assuming that u and e are normally distributed, Henderson maximized









the joint density of y and u with respect to P and u. This method results in Henderson's "mixed

models equations" (MMEs):

X'R-1Xp + X'R-Zii = X'R-ly
(1-2)
Z'R-1Xo+(Z'R-1Z+ G-)ii =Z'R-y

Note here that G and R are assumed fully known. The solutions to these equations,

P = X'V-'X) X'V-1y
(1-3)
ii = GZ'V-1 (Y- X),

are the Best Linear Unbiased Estimate (BLUE) and Best Linear Unbiased Predictor (BLUP),

respectively: they are linear in the data, unbiased, and achieve minimum variance or mean

squared error (MSE), respectively, among all linear, unbiased estimators or predictors (see, for

example, Harville 1990 and Robinson 1991). Note that the solution for p in Equation (1-3)

gives the GLS estimate. Robinson summarized several other justifications for the BLUP

estimates, including a Bayesian method with a flat prior on P and a N(0,G) prior on u yielding

the same result as the MMEs, as well as justifications that do not require normality. A linear

combination of fixed and random effects is predictable when the fixed effect portion is

estimable. That is, for vectors X and 6, X'P + 6'u is predictable if 'p1 = a'Xp for some a (Littell,

Milliken, Stroup, Wolfinger, and Schabenberger 2006, p. 211). When estimation of a predictable

linear combination of fixed and random effects is desired, X'p + 6'ii gives the BLUP for the

predictable function.

1.4 Empirical Best Linear Unbiased Estimation (EBLUP)

The estimates in Equation (1-3) assume G and R (and thus V) are fully known; this

assumption, however, is rarely satisfied. A common alternative is to assume G and R are known









up to a dispersion parameter 0 = (0,...,0 ,), thus G =G(0) and R= R(0). The BLUP

solutions depend on G and R, and therefore 0, which is typically unknown. The standard way of

resolving this issue is to employ "two-stage estimation;" that is, first estimate the dispersion

parameter 0 by, say, 0 then use these estimates in the equations for the BLUP of the

predictable function, yielding X'p + 6'ui, where P and u are the solutions to Equations (1-2) with

0 estimating 0:

G=G(0) p=(X-iX)1 X'V Y

R=R( ) = GZ'Vjy- Xp)

These predictors are often referred to as Empirical BLUPs (EBLUP), although they are no longer

linear functions of the data, they are not necessarily unbiased, and they no longer necessarily

have minimum MSE.

There are many ways to estimate 0, including ANOVA, MINQUE, MIVQUE, maximum

likelihood (ML), and restricted maximum likelihood (REML). The REML methods tend to be

favored from a theoretical perspective (Searle, Casella, and McCulloch, 1992). However, from

a data analytic perspective it is unclear which method is most advantageous. The technicalities

of these methods will not be reviewed here. For an overview of the technical details of these

estimation methods see, for example, Khuri and Sahai (1985) and Robinson (1987). We will

review the advantages and disadvantages of the methods as necessary in the context of

evaluating the precision of the EBLUP methodology.

1.5 Two-Stage Estimation and Unbiasedness

The consequences of two-stage estimation on the estimates of the linear combination,

'p + 6'u, are discussed in Kackar and Harville (1981). They show that when the dispersion









parameter estimates are translation invariant, even functions of the data, and the data are

symmetrically distributed, the two-stage estimation procedure resulting in the EBLUPs provides

unbiased predictors of linear combinations of effects; i.e., E [' + 6'ii = 1.'. Khatri and Shah

(1981) provide similar results. Kackar and Harville (1981) show that these conditions are

satisfied by assuming normality of the random effects and error and using REML, ML, ANOVA,

MINQUE or MIVQUE variance component estimation methods. Note that this result can be

extended to the broader class of location equivariant estimators, as shown in Kackar and Harville

(1984), citing a theorem by Wolfe (1973).

In evaluating the precision of EBLUPs, we must account for sampling variability

contributed by the estimation of 0 When this added variability is ignored, the true precision is

often grossly underestimated, as demonstrated by Kackar and Harville (1984), Harville and Jeske

(1992), and Tuchscherer, Herrendorfer, and Tuchscherer (1998). Goals of this dissertation

include clarifying and comparing existing methods of estimating the precision of the EBLUP,

assessing the accuracy of these methods and evaluating the methods of hypothesis testing which

utilize the various methods of estimating the precision. After reviewing the related literature in

Chapter 2, we generalize results on evaluating the precision of the EBLUP to include multiple

dimensional cases and random effects in Chapter 3. A case study of the balanced one-way

random effects model in Chapter 4 sheds light on the problems with existing methods and

suggests possibilities for improvement. Chapter 5 tackles the issue of negative variance

component estimates and the impact on EBLUP methodology. Conclusions and future work are

summarized in Chapter 6.









CHAPTER 2
LITERATURE REVIEW

2.1 Precision of EBLUPs

2.1.1 Mean Squared Error of Prediction (MSEP) of EBLUPs

Let t = X'p + 6'u and let f = f (0) = X'p + 6'i denote the estimate of t when 0 (and hence

V) is known. When 0 is unknown, denote the estimate of t, the EBLUP, by

S= f () = x,' + 6'f. The prediction error for the EBLUP is -t and the mean squared error


of prediction (MSEP) is M(0)= E[ -t]2. By decomposing the prediction error into four

components, each assuming a different, successive state of knowledge, Harville (1985) shows

how each component adds to the variability.

The first state of knowledge supposes full knowledge of the joint distribution of (y, t).

The unbiased minimum MSE predictor of t in this case is clearly

E[t I y]= x'p + 6'E [u I y] (Harville 1985, sec. 2.1). This is known as the Best Predictor.

The second state of knowledge supposes the first and second moments are known, i.e., P

and 0 are known but the functional form of the joint distribution of (y, t) is unknown. In this

case the predictor with minimum MSE among all linear predictors is

z = 'P + 'GZ'(ZGZ'+ R)-1 (y Xp)
= /+6'GZ'(ZGZ'+ R)-y y
where
7 = 'p -6'GZ'(ZGZ'+ R)-I X

(Harville 1985, sec. 2.2). This is known as the Best Linear Predictor.








The third state of knowledge assumes only the variances and covariances are known, i.e.,

o is known but p and the functional form of the joint distribution of (y, t) are unknown. Since rq

depends on P we must replace P with an estimate. Choosing p, the BLUE for P, results in

f = B (y)+6'GZ'(ZGZ'+ R) y
where (2-2)
B (y) = k'p 6'GZ'(ZGZ' + R)-1 Xp

(Harville 1985, sec. 2.3). This is the Best Linear Unbiased Predictor (BLUP).

The fourth state of knowledge represents the typical situation in data analysis: all

moments and the form of the joint distribution of (y, t) are unknown. That is, now 0 is also

unknown. Because the BLUP in Equation (2-2) depends on 0 we cannot directly use it without

further adjustment. Let z be maximal invariant, that is, let z = L'y where L is an n x (n -p)

matrix with rank(L) = n -p and L'X = 0. A function g(y) is translation invariant if and only if

g(y) = h(L'y). Thus let 0(z) = 0 be an arbitrary translation invariant estimate of 0. Then

iB (Y)= IB (y;W)dP(o; z), where P(o; z) is an arbitrary probability distribution on 0, gives a

feasible estimate oft. The Empirical Best Linear Unbiased Predictor, EBLUP, corresponds

to TB (y; ), the value of ZB (y) when P(*; z) is the degenerate probability distribution

assigning all mass to 0 (Harville 1985, sec. 2.4). Note that ,B (y) is translation invariant, thus

preserving the unbiasedness of the predictor.

With this notation, we now have the four components of the prediction error:

-t =Et(t y) t+r E(ty)+ B-q+ B(y;)- TB. (2-3)
(1) (2) (3) (4)








Each successive component is associated with moving from one state of knowledge to the

next. This decomposition allows analysis of the role of each component in the MSEP.

Component (1) of Equation (2-3) is uncorrelated with all other components by the

following result (Harville 1985, p. 135):

cov(h(y), E(t y) t)) = E [h(y) (E(t I y) -t)]
=E {E[h(y)(E(t y)- t)] y (2-4)
=0.

Components (2) and (3) of Equation (2-3) are also uncorrelated since

cov(y,r -E(t I y)) = 0 and r, is linear in y (Harville 1985, p. 135). Thus the Mean Squared

Error ofPrediction (MSEP), M(0), can be written as the following six components:

M (0) =E[ t]2

=E[E(t y) t]2 +E[r-E(t y)]2 +E[B, +]2 +E [,y; ) f, (0) (2-5)
(1) (2) (3) (4)
+E[ E(t Iy))(i, (y; ,) (0)) + E[(7-7)(B (Y; ( )B ())].
(5) (6)


In the development of estimators for M(0), it is often assumed that the data follow a

normal distribution. When the joint distributions of (z,r- t) and (z, B (y)) are multivariate

normal, as is the case when u and e are normally distributed, then components (5) and (6) of

Equation (2-5) are 0. However, without normality, there is no guarantee that these cross-product

terms disappear.

The remainder of this chapter will review the developments over the last two decades for

evaluating the MSEP of the EBLUP and conducting tests of hypotheses. Table 2-1 contains a









summary of the developments reviewed herein and stands as a helpful reference for these

developments and the notation used throughout.

2.1.2 Kackar-Harville Approximation to MSEP

Kackar and Harville (1984) and Harville and Jeske (1992) assume normality in developing

an approximation and estimator for M () Thus, only components (1) (4) of Equation (2-5)

need to be assessed for the approximation. Components (1), (2), and (3) of Equation (2-5)

comprise the MSEP when 0 is known. This is referred to as the naive MSEP and will be denoted

by M (0) = E [, -t]2 In the mixed model case with V(u) = G and V(e) = R, this is


M, (0) ='G6 -- 'GZ'V -ZG, + y'- v'GZV-iX) (X'V-'X)1 XV-'ZG5).

When normality is assumed, the cross-product terms are zero, and the only contribution to the

MSEP of estimating 0 is component (4) of Equation (2-5). We will refer to this component as

the correction term and denote it by C(0). M1 (0) is often used as an approximation for the

MSEP. When the data are normally distributed, this can seriously underestimate the MSEP since

it ignores the contribution of estimating 0 captured by C(0), as shown by Kackar and Harville

(1984), Harville and Jeske (1992), and Tuchscherer et al. (1998). Peixoto and Harville (1986)

expand on results in Kackar and Harville (1984) to derive exact, closed-form expressions of the

MSEP for certain predictors for the special case of the balanced one-way random effects model.

We use similar methods in Chapter 4 to enable comparisons of approximations and estimators to

the exact MSEP in the balanced one-way random effects model.

Assuming normality, Kackar and Harville (1984) provide an approximation to C(0) using

Taylor series methods. Using a Taylor series expansion of i = f () about 0








yields () = (0)+d(y; 0)' (0-0), where d(y;0)= Of(y;0)/a0. This leads to the

approximation,

[-]2 = [i()-i(0)] 2 [d(y;0)'( )]2. (2-6)

Note that when f is linear in 0 this Taylor-series approximation is exact. The validity of

the Taylor-series approximation is contingent on two conditions: First, the function being

expanded must have derivatives of all orders defined in a neighborhood of the point about which

the function is being expanded; and second, the remainder term must tend quickly to zero (Khuri

1993, p. 111). Thus if the parameter space, 0, is restricted (i.e., bounded below by zero) and the

true value of 0 falls on the boundary of the parameter space, the Taylor series expansion is not

valid. Assessing the effect of the parameterization and parameter estimates on the estimates of

the MSEP is one goal of this dissertation.

When 0 is unbiased for 0 and cov d (y; 0) d (y; 0)', (- 0) ) = 0 we have


E[i-i]2 =E d(y;0)'(0-0)] = Ed (y;0) d(y; 0)' E [(0-0)(-0). Noting that


E[d(y;0)] = 0, Kackar and Harville (1984) propose the following approximation to C () :

C(0) tr [A(0)B(0)] (2-7)

where, A(0) = var[d(y;0)] =E[d(y; )]2 and B(0) is either an approximation to or the exact

value of the MSE matrix, E 0 -0 0 -0 The inverse of the observed information matrix,

I 1 (0), is commonly used for B(0). Equation (2-7) will be referred to as the Kackar-Harville








approximation for the correction term. Thus the approximation of the MSEP of i proposed by

Kackar and Harville (1984) is

M, (0) +tr [A(0)B(0)]. (2-8)

The accuracy of the Kackar-Harville approximation of the correction term in Equation (2-7)

depends on the following conditions:

1. f linear in 0;
2. the validity of Taylor series expansion of T about 0;
3. 0 unbiased for 0;

4. cov d(y;0) d(y;0)',( 0-)( 0) =0;

5. the tractability of the exact MSE of 0 .

If all of these criteria are met, the Kackar-Harville approximation exactly equals the correction

term it is approximating.

2.1.3 Estimating C(0)

Because the calculation of the Kackar-Harville approximation to the MSEP in Equation

(2-8) requires knowing 0, it is not directly functional. An estimator based on the Kackar-

Harville approximation was developed by Harville and Jeske (1990) and Prasad and Rao (1990)

and is now implemented in the MIXED procedure of SAS. Following developments from

Prasad and Rao (1990), an estimator for tr [A(O)B(O)] is given by tr A( )B(0)1 ; where 0 is

substituted for 0 in the Kackar-Harville approximation for the correction term. The justification

for this substitution is not directly evident. Prasad and Rao (1990) show that in the special case

of the Fay-Herriot small area model, in which V(e) = R is known, the order of approximation is

o(q 1) where q is the number of small areas, or in general the number of levels of the random

effect. This result has not been generalized for the typical case when R is unknown. The








validity of this substitution and the consequences on the accuracy of the MSEP estimate will be
examined for the balanced one-way random effects model in Chapter 4.

2.1.4 Estimating M1 (0)

Historically, M, (0) has often been estimated by M, (6), that is, by substituting 6 for 0

in M, (0); however, M, (6) may have bias of the same or higher order as the correction term

(Prasad and Rao 1990; Booth and Hobert 1998). Consequently, it is just as important to account

for the bias introduced by estimating M1 (0) with M1 (6) as it is to account for C(0). Prasad

and Rao (1990), Harville and Jeske (1992), and Kenward and Roger (1997) all approximate the

bias of M1 (6) by taking the expectation of a Taylor series of M1 (6) about 0 (assuming 0 is

unbiased for 0). Harville and Jeske (1992) developed the following expression for the bias of




E M, ( M, (0) + tr[A (0) B (0)] (2-9)



82, B (9)
where A( (0)- They show that when cov(y,t), var(t) and var(y) are all linear in
802

0, A (0) = -2A (0) leading to

E[M, (6)] _M, (0)- tr[A()B()] (2-10)

Prasad and Rao (1990) and Kenward and Roger (1997) developed expressions similar to

Equation (2-10).

Again by substituting 6 for 0, possible estimates for M1 (0) are












or
M, ()+tr[A (0)B()]. (2-12)

Notice again that there is bias introduced by substituting 0 for 0 in Equations (2-11) and (2-12).

This bias is often assumed to be negligible, although the only noted justification, previously

mentioned in Section 2.1.3, comes from Prasad and Rao (1990) and only applies to Equation

(2-12) in the Fay-Herriot model. We will study this issue further through a special case in

Chapter 4.

2.1.5 Estimators of M (0) Based on the Kackar-Harville Approximation

Based on Equations (2-11) and (2-12), respectively, the following estimators of the

MSEP follow from the Kackar-Harville approximation and the bias correction approximations:

M () tr [A()B ()] + tr [A ()B() (2-13)

and

M, (e)+2tr [A ()B() (2-14)


The DDFM = KR option of the MIXED procedure in SAS utilizes Equation (2-13) which

reduces to Equation (2-14) when the model covariance structure is linear in 0.

Chapter 3 explores the Kenward and Roger (1997) and Fai and Cornelius (1996)

expansion of Equation (2-13) into multiple dimensions. Therein we establish the equality of the

Kenward-Roger and Fai-Comelius approximations to the correction term and expand the term to

include linear combinations containing random effects. We also provide a proof that the most








general form of the correction term approximation is non-negative definite and discuss the

definiteness properties of the overall MSEP estimator.

2.2 Hypothesis Testing for Single Dimension Linear Combinations

Several methods of approximating the distributions for test statistics have been developed

for single and multiple dimension linear combinations. The accuracy of these methods has been

largely untested. Following is a summary of the available methods for approximating the

distributions for single dimension linear combinations.

Giesbrecht and Burns (1985) and Jeske and Harville (1988) provide approximations for t-

type test statistics based on the Satterthwaite (1941) method. Giesbrecht and Burns (1985)

present a test statistic for fixed effects with the "naive" variance estimate:


tGB 1 G t (2-15)
x' (x'V x) )

When V is known and used in place of V in Equation (2-15), the test statistic has a normal

distribution. Thus, it is assumed that Equation (2-15) has a t-distribution with the degrees of

freedom derived from the approximate distribution of the denominator. To estimate the degrees

of freedom, a "Satterthwaite" method is used. The variance of the denominator is matched with

that of the approximating 7 distribution as follows:

x' X'Vix) 1 df 7 1
var dfGB --'X) = 2dfGB ( VGB ) var L] XVX' x ) = 2dfGB

k,(,'2 k'(X'V-'X) -

SdfGB 2 -( ) _
var (X'V-iX)








The term var [X'X'V iX) k is evaluated using a Taylor series approximation and the

asymptotic covariance matrix of 0. In the Giesbrecht-Burns case, this yields

var[ '(' X-'x)) ] cov 1 'V-z V Z'V-x(xv11 XV 1
(2-17)
'X'V-1Z V z'V-x(x V-12-1

The underlying assumption in developing this procedure is that the distribution of

'X'(XViX)) k is a multiple of a X2 -distribution.

Jeske and Harville (1988) also use a Satterthwaite method to approximate the degrees of
freedom of a t-statistic; however, they include fixed and random effects in their test statistic with

a corrected version of the MSEP estimate. The approximate t-test is given by

tm = ~- t (2-18)
M 9 + tr [A (6) B(

where
2\M, (Q)+,)tA(Q),(Q)1}}2
dfJH L] (2-19)
var M, (0)+ tr A(0)jB( )l

As in the Giesbrecht-Burns t-test, the denominator of Equation (2-19) is approximated by a
Taylor series expansion yielding

var M (0)+ tr A(0)B() h ()' (B(0)h (), (2-20)

where


h(0) = [M ()+ tr A(0)B()] and B(0) is an approximation ofvar().
90 (6)









Again, an underlying assumption is that the distribution of M, o)+ tr A () B (6)] follows a

multiple of a _2 -distribution. An extension of this method would be to incorporate the bias

correction into the MSEP as seen in Equations (2-13) and (2-14). This extension is considered

in Chapter 4.

2.3 Hypothesis Testing Methods for Multiple Dimension Linear Combinations

Kenward and Roger (1997) and Fai and Cornelius (1996) offer F-tests to handle multiple

dimension as well as single dimension cases; however, the methods for determining the

distribution of the test statistics differ. One of the Fai and Cornelius methods (DDFM =

SATTERTH option) and the Kenward-Roger method (DDFM = KR option) are available in the

SAS MIXED procedure. A goal of this dissertation is to compare these methods with other

modified methods from Fai and Cornelius (1996) for the balanced one-way random effects

model. We will also assess the effect of sample size, parameterization, and parameter value on

these hypothesis testing methods.

2.3.1 Kenward-Roger Multiple Dimension Hypothesis Testing Method

The test statistic developed by Kenward and Roger (1997) utilizes an MSEP estimate for

multiple dimension linear combinations that is adjusted for the correction term and the bias of

M, (o). This MSEP estimate is thoroughly discussed in Chapter 3, but for now will be denoted

by MK (). Note that these methods are only developed for tests of fixed effects; thus the

MSEP can be thought of as a variance. Theoretically these methods could easily extend to

include random effects. We include a simulation study for the one-way random effects

illustration to determine the validity of including random effects in these procedures.








Consider the hypothesis


Ho:LP=0 vs. Ha:Lp#0,

where L is an /-dimension matrix of constants representing the linear combination of interest.

Kenward and Roger (1997) propose the following Wald-type test statistic:

F KR 0)'L'(LMKR (6)L) 1 L(-).

An F-distribution is approximated for the test statistic to take into account the random structure

of MK (O) and to match known cases where the test statistic has an exact F-distribution. We

will briefly review the procedure to allow comparisons to alternative methods and to examine

any shortcomings.

Kenward and Roger (1997) use a Taylor series expansion to approximate

(LMK () L') ; however, it is unclear about which point the series is expanded and which

higher order terms are dropped. It appears that the series is expanded about M, (0) rather than

the true value, M (0). This may be justified by dropping higher order terms; however, this step

is the only place where the corrected MSEP is taken into account in the distribution of the test

statistic. Using this Taylor series approximation, Kenward and Roger (1997) use the identities

E[FK] = E[E[FK M ()]]

V [F] = E[4FKF MR() + V[E[FK MKR()]]

to approximate the moments of the test statistic. A two-moment-matching method is then used

to obtain estimates for a coefficient, K and degrees of freedom, m, so that

c KR l,dfIR








To make the approximating distribution match the exact cases where the distribution is known,

selected higher degree terms are added to the Taylor series expansion. The following

summarizes the results of the Kenward-Roger procedure:

1+2 df,
df = 4 + and K-=
p 1 E[FKR](dfR-2)

where
V[F KR]
2E[FKR]2

and

E[F} ] 1- ] and V[FKR]= C- B(1 B)]
( 1 (1- ) (I-B 3B)

where

S=1 J=1 L(V 00 (x '
(2-21)
*tr L' L(X'V-IX)-L ]L(X'V-1X) 1X' V X(XV- X)-



A2 =Y Btr L VLX'V-X L L(X'V-X) X' X(XV-X )1
1=1 ]=1 -1

*LLL(X'V-XX)L L L(XV-1X)- Xav-x(x'V-'x)-1,
L 00j J B










B (A + 6A2),
21
g
31+ 2(1-g)'

1-g
231+2(1-g)'
l+2-g
33/+2(1-g)'
(l +1)A -(1+4)A2
g= (1+2)A2
and B, is the ijh element of the matrix B(0).

The Kenward-Roger method is available in the PROC MIXED procedure in SAS as the

DDFM = KR option in the model statement. This option performs the adjustment to the MSEP

estimate and the hypothesis test as described above.

We now demonstrate that in the single-dimension case, the Kenward-Roger degrees of

freedom approximation reduces to the Giesbrecht-Burns degrees of freedom approximation

reviewed in Section 2.2.1 (Kenward and Roger 1997, p. 988). Recall that the Giesbrecht-Burns

test statistic did not use an adjustment for unknown 0 in the MSEP. Thus, while the Kenward-

Roger and Giesbrecht-Burns test statistics differ in the estimate of standard error, the

approximated distributions are identical.

Consider again a hypothesis test for the single dimension linear combination Ho : ,' = 0,

under the linear mixed model. The Kenward-Roger test statistic is


t = (2-22)
VA KR

where










VARKR = (Xv-lX)i-1+ 21x (XV-lX) B X'v- V V-
80, 800

-X OV-1 XX'V-1X X 8V-1 X X'V-14 2V V-XIX V-1X =
90 / 0 4 86 (v v



In this case


A, = cov( )tr X'V-)-'1 XV-Z Z'V-x 'Vx


*tr -?. 1' X'V-1X) 1 'X'V-iZ ZV -X (X'xV-X-1


XV*trX cov,) xvzX('v x)-V 'iixfvz 'V zvx(i)1
1=1 J=1 9

x1'X'V-Z Vz'v-lx x'v-lx)-1

=A

since the quantities inside the trace functions are scalars. Note also that

1 7
B= (A,+6A)= A2,
2 2
2A2 5A2
g= = 1,
3A2
-1 1
cl 3(1)+2(1+1) 7'
2
C2 --
7

and
4
C3 =
7








Thus,
E[F] = (1- A2)1,

1+ (1)( )A 22
V[FKR]=2 7 C2( 2A2


and

P l 2- 2A (1-A2)2 2-A,
(1 -A, )1-2A 2, 2 2(1-2A2 )


The degrees of freedom for the Kenward-Roger hypothesis test are then
3 2
df = 2-A 2 1
2(1- 2A)

=2 ('(X'V-X)-l j 1 2/ cov( ) I'XV-1Z' VZ'V-1X(X'V-1X)-1 (2-23)
1 [=1 0 1= 00,

*I 'X'V-vZ 0V X'V ) J{


Note that the denominator of Equation (2-23) is the Taylor series expression for

var[ 'X'VX X)x This is the same as the degrees of freedom estimate from the Giesbrecht-

Burns test in Equations (2-16) and (2-17); however, recall that the Giesbrecht-Burns test statistic
is

tGB
x'(x'vix) x

Thus df = dfGB but tK< ltGB Often, the discrepancy in these test statistics may not be

enough to make a significant difference in the test results; however, there are cases when the








discrepancy changes the conclusion of the test. The question becomes, which test is better? The

accuracy of the estimating distribution must also be questioned in either test. Note that because

the estimated distributions are equal, we can isolate the impact of modifying the estimated

variance of the EBLUP on the hypothesis test. We explore that situation further though a

simulation study in Chapter 4.

2.3.2 Fai-Cornelius Multiple Dimension Hypothesis Testing Methods

Fai and Cornelius (1996) also use a moment matching method to determine the

distribution for their multiple dimension test statistics. The differences between the Kenward-

Roger method and the Fai-Cornelius method come in the particular MSEP estimate used in the

test statistic, in the number of moments evaluated, and in the techniques used to evaluate the

moments of the test statistic.

Fai and Cornelius (1996) offer two alternatives for the test statistic: one uses the naive

MSEP estimate in matrix form, M1 (0), and the other uses the naive estimate plus a correction

term denoted by, MFC () = M1 () + CF (6), which will be discussed thoroughly in Chapter 3.

Note that neither of these takes into account the bias of M1 (0) for M, (0), as in the Kenward-

Roger procedure. The test statistics are


FFCl =(-)'L'(LM()L) 1L(p- P) (2-24)

and

FFC 0-)'L'(LMFC 6 L') L p-P). (2-25)

For each of these test statistics, both one and two moment-matching methods are offered for

estimating an F-distribution, resulting in four alternatives for conducting a multiple dimension

hypothesis test. The method of finding the moments is the same for both test statistics, so for








simplicity let MSEP denote the estimator of the MSEP in either Fai-Cornelius test statistic. To

evaluate the moments of the test statistics, first a spectral decomposition is performed so that

P'L(MSEP)L'P = diag() where P is an orthogonal matrix of eigenvectors and 4,

k = 1, 2,...1, are the eigenvalues of L(MSEP)L'. The test statistic is rewritten so that


IF, =( P)'PP'L' (L (MSEP)L' PP'L p)

=(P'L P'Lp) {P'L (MSEP)L'P1 (P'L P'LP)

= (P'L P'Lp) diag ( )}1 (P'L P'LP)
S(P'P L2 (2-26)

k=l k

= 1 t2k2
k=1
=Q.


They then note that tk would have an independent standard normal distribution if the MSEP

were known. It is reasonable then to approximate the distribution of tv with a t-distribution and

estimate the degrees of freedom, vk, with a Satterthwaite procedure, i.e.,


k L(P'L'L-P'LP) k 2 [(MSEP)
V[(V(P'Lp-P'Lp)) V (MSEP)

As in the Kenward-Roger procedure, we have the underlying assumption that MSEP has a

X2 distribution. If v > 2 for all k= 1, 2, ., 1, then the moments of Q can be approximated as

follows:










E(Q)= iE(t2 = lVk
k=1 k=l Vk 2

and

(Q)i 2v (vk 1)
k( 1 1 (Vk 2)(v -4)

From Equation (2-26) we see that the moments of the Fai-Comelius test statistics are

E(FF)= E(Q)


and
1
V (FFC)= V(Q).


One or both of these moments are matched to the moments of the approximating F-distribution,

as in the Kenward-Roger procedure, to produce an estimate of degrees of freedom (and an

estimated coefficient in the two-moment case). The one-moment method with the test statistic in

Equation (2-24) is available in the PROC MIXED procedure of SAS as the DDFM =

SATTERTH option in the model statement. Note that in the single dimension case, the Fai-

Cornelius test statistic in Equation (2-24) reduces to the Giesbrecht-Burns test statistic in

Equation (2-15) and that the Fai-Comelius one-moment method of estimating the distribution

also produces the same results as the Giesbrecht-Burns method. Thus we are avoiding the

discrepancies that occur between the Kenward-Roger method and the Giesbrecht-Bums method

described in Section 2.3.1.

We are interested in comparing the available hypothesis testing methods to determine

which one provides the most accurate Type-I error rate. Some work has already been done to

this end by Schaalje, McBride, and Fellingham (2002). Schaalje et al. (2002) compare the one

moment Fai-Cornelius method with the test statistic in Equation (2-24) (using the DDFM =

SATTERTH option in SAS) with the Kenward-Roger method (using the DDFM = KR option in









SAS). They performed simulations for four split-plot designs, two of which were balanced and

two of which were unbalanced. The simulation is repeated for five covariance structures to study

the effect of the complexity of the covariance structure. Hypothesis tests were computed for

whole plot effect and sub-plot effect using the DDFM = KR and DDFM = SATTERTH methods.

Schaalje et al. (2002) calculated simulated Type I error rates and lack-of-fit tests to determine if

the simulated p-values followed a uniform (0,1) distribution. Both methods performed equally

well for the simple Compound Symmetric covariance structure, regardless of the design. For the

other four covariance structures the Kenward-Roger method outperforms the Fai-Cornelius

method in the lack-of-fit test and the type I error rates; however, as the sample size becomes

smaller or the covariance structure becomes more complex, neither method performs well. Type

I error rates are severely inflated under both methods when complex covariance structures are

coupled with small sample sizes.

Note that when random effects are considered, it is more appropriate to focus the

investigation on prediction limits. We will shift the focus to prediction intervals and expand the

comparison to include modified Fai-Cornelius methods with adjusted MSEP estimates in

Chapter 4. Although we confine the comparison to a single dimension linear combination in the

balanced one-way random effects model, this comparison will paint a more complete picture of

the methods considered and show the need for further investigation into the proposed methods.

2.4 Negative Variance Component Estimates

Very little research has focused on the impact of negative variance parameter estimates

on BLUP procedures, the MSEP estimation methods, and hypothesis testing. Stroup and Littell

(2002) begin to explore the impact of the method of variance component estimation and negative

variance parameter estimates on tests of fixed effects. Their exploration reveals that REML may

not always be the best choice in terms of power and true Type I error values for tests of fixed









effects when negative variance parameter estimates are prevalent. The impact of negative

variance parameter estimates on the EBLUP and MSEP estimation for linear combinations

involving random effects has not been thoroughly investigated. In Chapter 5, we delve into the

complexities of negative variance component estimates and the impact on EBLUP and the

associated MSEP estimates. Smith and Murray (1984) develop a method for handling negative

covariance estimates. They propose that the variance parameter in question be considered as a

covariance parameter, thereby allowing negative estimates. The impact of this strategy on

EBLUP methods is considered in Chapter 5, leading to an alternative model and analysis

procedure.










Table 2-1. Summary of developments for MSEP estimation and hypothesis testing
Fixed or
Paper Random Correction Term Bias Term
Effects


Kackar and
Harville
1984
Giesbrecht
and Burns
1985
Jeske and
Harville
1988


Fixed and
random

Fixed


tr[A(0)B( )]


None


Fixed and tr[A(0)B(0)]
random


MSEP Estimate


M, (0) +tr [A (0)B (0)]


None


None


None


k,(XV1x)" k


Prasad and Rao
1990


Fixed and
Random


Etr[A(O)B(O)]j

tr [A (0)B()] +o (q-1)
For q = # small areas, in the Fey-
Herriot model


E[EM(6)]
= M (0) tr[A(O)B(O) +o(q ) For q = # small areas, in
the Fey-Herriot model


Harville and
o Jeske
1992


Fai and
Cornelius
1996



Kenward and
Roger
1997


Fixe ad and tr[A()B()]
random


Fixed





Fixed


tr[A(0)B(0)] defined for
multiple dimension linear
combination, denoted cr ();
shown to be nnd

tr[A(0)B(0)] defined for
multiple dimension linear
combination; c" (e)


M,(e )-tr [A(6)B6)+tr [A()B(6)B
or
M,() 2tr[A (6B

M,(e)


None


E [M,()1 M,(0)+tr[A(O)B(O)]
M,(0)-tr [A(0)B(0)]
Defined for multiple dimension linear combination,
BIAS- (e)


M '()=M(, )+C(6e)




= M ()+CR (e)+BIAS" (e


E [l (e)]M, (0)+-tr[A(e)B(0)
SM,(0)-tr[A(0)B(0)]


M(6) + tr[A(6) B()



I (6) +i2tr [A ( B










Table 2-1 cont.
Paper
Kackar and Harville
1984


Giesbrecht and Burns
1985


Jeske and Harville
1988


Hypothesis


H,: X'= 0


Ho: 'p + 6'u = 0


Test Stat


tGB
'(xv iX)V 1


Al= (6)+tr[A(6)B(6)]


Distribution of Test Stat


t where lLx ix)W]
var '(X'V X) k

Evaluated with Satterthwaite-type method



tdf H 2[M1 (9)Itr[A(9)B(9)]]
2 Mi a (e)+tr[A (e)B ()]

Evaluated with Satterthwaite-type method


Prasad and Rao
1990

Harville and Jeske
1992


Fai and Cornelius
1996


H : L= 0


)1 ^ L'(LM ^e)L) L( -p

F2 1(p-P)L'(LM ( eL')L L(-p)


One and Two moment matching; Spectral decomposition, Taylor
series and Satterthwaite techniques used to determine moments.
Note that this yields two choices of distributions for each test
statistic.


Kenward and Roger
1997


Ho: Lp=0


F 1 L'(LM (6^L') 1 ( Two moment matching; Taylor series and conditional expectations
KR L used to determine moments.








CHAPTER 3
ESTIMATES OF MSEP FOR MULTIPLE DIMENSION LINEAR COMBINATIONS

3.1 Reconciling Kenward-Roger (1997) and Fai-Cornelius (1996) Definitions of the
Correction Term Approximation

3.1.1 Linear Combinations of Fixed Effects

Kenward and Roger (1997) and Fai and Cornelius (1996) extend the methods for

estimating the MSEP summarized in Section 2.1 to multiple dimension linear combinations.

Both methods require expanding the definition of the Kackar-Harville approximation of the

correction term, and in the Kenward-Roger case, the bias correction term, into a matrix form.

Recall that the Kackar-Harville approximation of the correction term for single dimension linear

combinations is C(0) tr[A(0)B(0)] where A () var[d(y;0)] =Ed(y;0)d(y;0)',


d(y;0) = 0f(y; )/80, and B(0) is either an approximation to or the exact MSE matrix,


E 0 0) Extending this definition to multiple dimension linear combinations is not


necessarily unique.

In both papers, only linear combinations of the vector of fixed parameters are considered.

In Section 3.1.2 we extend their method to include linear combinations involving both fixed and


random effects. Noting that tr [A (0) B (0) = b cov -, where b,, is the it


element of B (), Kenward and Roger expand this definition to the multiple dimension case by


C((cov -), where L is an /-dimension matrix of constants
representing the linear combinations of interest. We will denote this approximation as C (
representing the linear combinations of interest. We will denote this approximation as CKR (0) .










Note that,


COVf Lp a
80, 0J


co L,p aL,p a
cov L L "o


COV BL2,p OLap
cov -




cov 0J
[80, 80


rLip aL2p (aLp aLp
c cov L cov,
co o' ) J Io, oj
aL2p aL2p aLp aLp
cov --2 L cov L2 OL




c aL, aL, p cov O L,P L
Sa0, a0o, 0, a 00 o


where L, is the ith row of L. Thus we can write the kmth element of C ( (0) as


{Ck (O)}= bl cov aL +b2 aCOV Lp b +*..+b cov Lk ,Lm
00, 80, 80, 802 qq 8q 80' q

Fai and Cornelius (1996) define the multiple dimension approximation to the correction

term differently and restrict their definition to a compound symmetric covariance structure with

two covariance parameters. We now expand that definition to include any covariance structure

with r parameters and show its equivalence to CR (0). They also establish the non-negative

definiteness of their correction term definition, which also applies in this more general setting.

Let CFC (0) denote the Fai and Cornelius multiple dimension approximation to the

correction term. Then,


Fc () =


(3-1)


tr {A (0) B (0))} tr {A (0) B (0)}

tr({Al(0)B(O)} ... tr{All,()B(O)}


where










COV ,Lk *L* COV ,O ,

Ak, (0)= cov : "0 :
km{ [ 8Q 8Q

Cov Lk 0L0 .P CO Lk d Lmp
0cov o,- cov 0, 0


and B(O) is as before. Then the kmnth element of CFC (0) is

{CFC (O)}k =tr{Ak( ()B(O)}= b, cov ,k O + bacov a aL +


1+ b ,, cov ,
O8q Odq

Thus, clearly Ct (0) = CFC (0).

Fai and Cornelius (1996) have shown that this definition of the correction term

approximation is non-negative definite. This becomes an important factor when the MSEP

estimate is to be used in a test statistic or confidence interval. We define the most general form

of this approximation and prove that it is non-negative definite in the Section 3.1.2.

3.1.2 Linear Combinations of Fixed and Random Effects
We can now generalize the definition of the multiple-dimension correction term

approximation to include random effects and establish that it is non-negative definite. Let

K = [L I M] denote an /-dimension matrix of constants. We now consider the linear


combination, K = LB + Mu. To expand the definition to random effects, we need to


redefine Ak (0) as








Aa(Lk + Mki) (Lp+M i)
Akm (0) = cova

A 0 o (LP + Mu) I(LO+M ,u) a(Lp+Mu) (LP+MU)
80 80

cov O(Lk+ Mku) O(L0P+Miu) cov (L + Mu ) c(L0P+M iu)

0V' 00 J, 0o 2
cov O(Lk kii) o(L O+o i) ... cov '(Lk ki) O(L.0 + Mji)

where Lk is the kth row of the fixed effects portion of the matrix of constants, L, and Mk is the

kth row of the random effects portion of the matrix of constants, M.
We now follow a similar procedure as used in Fai and Cornelius (1996) to establish that

this generalized definition of CR (0) is non-negative definite and thus the MSEP estimator

utilizing this correction term estimator is nonsingular. First, however, we need to broaden the

definition of C' (0) to incorporate linear combinations of fixed and random effects.

First, using the result that = -A A', we have
ax ax

OLkP OLk (X'V-X) 1 X'V-y LkXV-1 (X'V-X) 1
cP^_,_- --- )_ --- = Lk(X'V-1X) X' --y+Lk c X'V ly
80, 80, 00, V-
=Lk (X'V-X) X' y Lk (X'V-1X) X'fV1 X (X'V-X) X'V1y

=Lk (X'V-1X) X' (I-P)y

-Lk(XVX) X'V1 V V- I-P)y
= V-1(I-P)y
CX1V ( P)y
where C' =-Lk(X'VX) X'V1 and P= X('V- X) X1 -1

and








8V1 V/?
= Mk ZVl(I- P)y+MkGZ' (I- P)y- MkGZ'V- X

=MG Z'V -(I-P)y-MkGZ'V1 V1 (I Py
Mk 0, ~V ,X(X(
+MkGZ'V-1X(X'V-1X)1 XV-1 v V-1 (-P)y
v -/rBy


D'(
D V- (I -P) y + E V-1 (I P)y + FV1 (I
(D + E + F)V-1 (I P)y


P)y


D' = M Z',

E' = -MkGZ'V1


F' = MkGZ'V X(X'V 1X) 0X'V1 V
fa~~ 6,V


Thus,


a(Lk, +Mku)
80,


We can now assess the ijth element of Ak (0) as


{Akn(zj) (0)}


Scov(Lkp+ Mki) a(L + Mii)
a6/ o0J

cov[(C', +D' +,E' +F') V-1 (I-P)y, (C D' +E' +F)V1(I-P)y]

(C' + D + E' + F',)V-1 (I-P)V(I -P)'V1 (Cm + Dm + Emj +F,)

(Ck, + D, +E, + F,) V (I P)(Cm + D + Ej + F,)


since


aMkfl
S0,


where


(C' +D' +E' +F'VI(- P) y.
b bk i ) II PY









(I P)V(I-P) v1 =(v x(x'V-x)' ') (1 -1X(X 'Vx) X''1)
=(I-P).


We now have

(Ck + Dk + Ekl+ Fk )
Akm (0) (3-2)
(3-2)
(Ckr +Dkr +Ekr +Fkr)
xV1(I-P) (C, +D, +E, +Fm ) .- (C,1 +Dm +Em +F')]

We can now generalize the definition of the Kackar-Harville correction term approximation as

follows.

Definition 3.1. The multiple-dimension Kackar-Harville approximation for a linear

combination including fixed and random effects is

CK (0)= (trAkm (0)B(0)]}, (3-3)

where Ak (0) is defined in Equation (3-2).

We now prove that C' (0) is non-negative definite.

Theorem 3.1. The multiple-dimension Kackar-Harville approximation for the correction term,

as defined in Equation (3-3), is non-negative definite. Thus, the MSEP approximation utilizing

this correction, M1 (0) + C' (0), is nonsingular.

Proof.

We begin by examining the kmth term of C"K () We can write V1 (I- P) = Q1/2Q1/2 and

B(0)= R1/2R1/2 where Q12 and R12 both exist since V1 (I- P) is non-negative definite and

B(0) is positive definite (e.g., Schott 1997, p.16).












Consider


{(c (o)}
= rA,, (0)B(0)}

(Ckl + Dkl + Ekl + Fkl)'
= tr

(C, +D, +E ,, +F )I

=tr(C + Dk + Ek+ Fkl)

(C,+D, +E, +F ,)'


V-1 (I- P)[(C,,D + D, +E., +lF1) ... (Cmr + Dmr + Emr + F,, )]B(0)



Q Q [(Cm, + Dml m tE F ) ... (Cml, Dml, Em, + F ml) R2R1 R


+ F1,) (Cmr + Dmr + Emr + Fr R1/2R/


(Ckl + Dkl +Ekl +Fk )
Q(CD+E
(C,h + D, + E + F ,)'


Str (Q 12[(C,, +Dml +Em, + F1) ... (C,,, +Dr +Emr +F r)]R12)

*(Q (Ckl+Dl +Ek +F) (C Fk+D +E, +F,)R12)'}

= vec[(Q1/2 [(Cm + Dm1 + Eml + ) .. (Cm, +D, +E m +F,,)]R1/2)]'
*{vec[(Q1/2 [(Ckl+ E ) ( +D +El +Fk) .(C +D +E, +F,)]R1 2)]



where =vec[(Q1/2 [(C, + D, + E, +F,) ... (C, +D,, +E,+ +F,J R1/2)]

Thus we can write CKR (0) as


cK (0) = A'A
where
A=(s,,...,,)


(3-4)










Writing C' (0) in this form proves that it is non-negative definite (e.g., Schott 1997, p. 16).


Since M, (0) = E((Lp + Mi) -(LP + Mu))((LI + Mu) -(LP + Mu)) is positive definite, we


have now established that the general approximation of MSEP,

M,(0) + C (0) (3-5)

is positive definite, and hence nonsingular, and thus is a valid measure of prediction error. o

3.2 Multiple Dimension Bias Correction

Recall that it is often necessary to correct for the bias introduced into the estimate of

MSEP by using M1 (6) to estimate M1 (0). Kenward and Roger (1997) define a bias correction

term for multiple dimension linear combinations akin to the single dimension bias correction

offered by Harville and Jeske (1992) found in Equation (2-11). We now rewrite this term to

resemble the correction term in Equation (3-1), in order to maintain consistency in our notation

and enable expansion to include linear combinations containing random effects. The

definiteness properties of either the bias correction term or the estimate of MSEP containing the

bias correction term has not previously been addressed. We will address that issue in Section

3.2.3. We first consider linear combinations containing only fixed effects.

3.2.1 Bias Correction Estimator for Multiple Dimension Linear Combinations of Fixed
Effects

We let b, again be the IJh element of B(0). Kenward and Roger (1997) state the bias

correction term as

1 2Cov(LBLLP)
BIASKrr 0 2 Cv (LO',L) (3-6)
BIASKR (0) 1 Y b (3-6)
2 where

where











SCcov(LO,LO)
00,80o


a2 var(Lp)




2 'cov (L, Lp)
o0, o,


a2 cov(Lia, L2)




2 cov(LJ, LP)
00,00JUp


02 cov (L,, LI,)
So80,80,



a2 var (L,)
o0,o0,


Thus, the knth element of BIASK (0) is


i l cov(Lk,LmIJ)

S0 cov (L OL,0p)
COV(,L
00,0oJ


a2 cov LkL ,J)


a 0 0,2
co lJ)]


B^2(LPLm~~ J-


We now rewrite BIASR (0) as


BIASKR (0)


where


tr {Ai (0) B (O)}

2
tr{A, (0)B(0)} ...


02 cov(LkO, Lmp)


2 cov(LkO, L.m)


02 cov(Lk,,Lm)
a 0ao,


2 cov (LkO, Lm)


2 cov(Lk ,Lm)
a0,


To confirm the equality of Equations (3-8) and (3-6) note that the kmth element of Equation (3-8)
is also


{BIASk (0)}


(3-7)


tr {Al,

tr {A1


(3-8)









BaK1 b 2 cov Lk,Lm)
{BASk (O) = tr Akm (O)B(O))= 2 bi, '0 )

'2 cov (LkO, Lm) b2 cov (Lk, L,)
+ b12 +- ..+b (3-9)

aC2 Cov(Lk,, Ln)_
+---+b j
C 802


which is identical to Equation (3-7). It has been shown by Kenward and Roger (1997) that

Equation (3-6) reduces to the multiple dimension correction term (C' (0)) when the covariance

matrix, V is linear in 0, which gives MR (o) = M1 (o) + 2CR (6) as an estimator of the MSEP.

In the next section, we use Equations (3-8) and (3-9) to define the estimator for bias correction

for a multiple-dimension linear combination involving fixed and random effects and confirm that

the resulting bias correction term continues to reduce to the correction term in linear covariance

cases.

3.2.2 Bias Correction Estimator for Multiple Dimension Linear Combinations of Fixed
and Random Effects

The extension to include random effects is a straightforward but tedious task. However,

it is important to explore fully the functions involved to determine when the term simplifies to

the correction term approximation in Definition 3.1, and also to assess the effect on the

definiteness properties of the MSEP estimator. The generalized bias correction term has the

same form as in the fixed effect case:

1tr {A 1(0)B(0)} ... tr {AI, (O)B(0)})
BIAS 9(0) = (3-10)
2 Atr (A, (0) B(o)} tr {Ah,, () B(O)}

but now,








A covO(LO + Mki, L + M u)
Ak,(0) = 0'
a' cov(LP + Mi, Lj + M u) 02 cov(Lkp +M Ljm + M iu)
a02 0 0,0, (3-11)

a2 cov(Lk +Mki, LmP + Mm ) a2 cov(Lkp+Mki, LmP + Mmi)
8600 06,2

Now considering a single element of this matrix we see that

C2 COV LkO +Mki, Lm +Msfu)
A-(Aj) (0)} 1--0--,0---

a2 [cov (Lk, L mP)+cov(Mii, Lm) + cov(Lk, Mrii) + cov (Mkii, Mii)]
-

c2 a)] 2 c)1 (3-12)
co (LO2, L]P + COV(Mku, Lm
0 J 0 J
1 2
+- a v(LkpM i+--) [o(Mk,M)]
3 4

We now investigate each piece of Equation (3-12) using Littell et al. (2006) as a reference for

the covariances involved. Consider Part (1) of Equation (3-12):








(1) o= c L cov(y L,,) =, cov ,XV XVVy, L,,XV XVIl

-_ [L (X'VlX)1 X'VX x)Vx1 L', = 'VX L'

8= L (XV-1X) 1t X 1v VX(XX 1X)Lm

=L (XV-iX) XV-1 W VIx(xv-x)- XV-1 NV-x(x'VX)-1 L'X



-L (xVlx)' xVv1 VVI x(xVx) L't+

eL( (XV-lX)- XrV-1 NGV, NV-xxv-x( e4
+Le (XVX)-1 XV-1 N XV-x1 XV(X1 VxX L'
X ( vl(vl)-V L
=CqVI1PC, -C V-ICC -CVPC,, +C V-PC,
+L, (XV-X)- XV-i a Xv -X(XV-X)-1L' (3-13)

= v'(I-P),, -Cqv(i-P)c ,,+4(xV-x)-V' Vl -v lx(xy-'x)-l'

where C' is as defined in Section 3.1.2.
Now moving on to part (2) of Equation (3-12), we have
-O MzV-1xv-x1 e
(2) = cov( j i,LB) N= GZVR j(X -l1x

=-- M zrvx(x'Vx)-i L' +MGZV- aV-x(xV-1X)I L' (3-14)

+MGZV-X(XV XVXVj a X XVxX)-i1 Lm










= ,Z'V~1 -V1XXVXL
k aO0 ao


-M k zV -1 V V OV V 'X(X v x Lf

+MkGZ'V- O VV VX~XVX) Lm
a o ao,
MkGZ'V- 1 V V'X(XfV'X) LL
aoj ao0

MkGZ'V l Vtx (xv tx)1XfV1avvx(xfv- x)1L
aoj00

1av 1X Lf
+MkG~ V -1 v v -lX (xlr x)-l x fv d v -lx(x fv -lx) mb

1M GZ'V1 v
-Mk zvV'X(Xfv-x)- l Xfv l aoj v-x(xfv 'x)- f 0 Lb ~f -l )


MkG'V '11'~ ao, o~VX 1 V1(XVX)
a~ ao

-M kG~ fV 'X(X fv lx )-X f la v -lvx v )- fv l v -lx~xlvlx ) m
+MkGZ'V 1(XVlixv VXV1v



0, aoV V
M~z' lxxv l)- x l1 ov ov 1 L frl'3 VXX IIX~ 1L
MkGZ'V 1X(X'V 1X)1 X'V'?V V1 aV V1X(X'V1X) Lb

+MkGZfVIX(XY1X)- XfV1 x0)1L0,x1 x -l xvv


aok ao ao xv-x
a~av


1MGZ'V1 V

SMkGZ'V 1X(XfV iX)- a 2V V X(XfV'X)








D'V -1Cm,,, + D' V-1PCm,


-E'V-C, +E'V-VPC E'V-C', + E'V-PCm,
-F'V-1CmI + F'V-1PCm, F'V-1C, + F' V-1PCm
^h ~ ~ ~ A m' k^i ^i' ^^ h*i Ct ^ mi


2 V Xt'XV _1 -1 L'
MkGZ'V-1 V-XV-X'VX L'
aoa 9019(X V1X iL


-MkGZ'V1X(X'V-lX) XV-1 2V V-(xv-lx) L'
8o, 8o0


-D V-1 (I- P) C,
-EV-1 (I-P) C, -


- D V-1(I-P)Cm,
E V-1 (I- P)Cm,


k2V 0(xlx) -1 L'
MkGZ'V-1 V- X V X'V1XV) L
aoao 90(,90^iL


-MkGZ'V-1X(X'V-1X)-1 XV-1 d V VI(X'V-X 1 L' .
w' an 90,9, v"

where D E,, and F' are as defined in Section 3.1.2. Due to the symmetry of parts (2) and (3)

of Equation (3-12), we combine (2) and (3) as follows:

(2)+ (3) -(D, +E + F+ ,) V-(I- P)CC,

-(D, +E ++ F) V-1 (I-P)C,,


(3-17)


-CV-1(I- P)(Dk +EE, +Fj)
-CV-I(I-P)(Dk+Ek +Fk)
( a2G V(
aO^GO}_ { ao o\


and g, --, are linear functions of
00,00OJ


02G
0oo o,0


02v
and respectively.
aoo ao


Part (4) of Equation (3-12) tediously completes {Akm(,j) (0) as follows:


(3-15)


(3-16)


where
where /f -
80,80J


-D'V-C, + D'V -PC
bV CMi h CM


-M 2G Z'V-X (X'V-IX)-I L'
M 0,0, v


-F,'V-1 (I-P) Cmj F V-1(I-P) Cm,
-Mk z0~J'-xx'VxX" L' -
M 9,9 v / i









(4) = cov (M,u, M )] M [G GZ'V- (- P) ZG] M'

82G 82G
= M, G M' Mk Z'V-1 (I- P)ZGM,
k 60 ] m k80 80 m
02G 02p
-MkGZ'V-1(I-P)Z M' +MkGZ'V-1 ZGM'
aao, ao o, a
a2V-1
-MkGZ' -(I-P)ZGM'

-D' V-(I-P) Dj- D V-(I- P) Dm,
-EV-1(I- P) Dmj -E' V-1(I- P) Dm, (3-18)
-F',V-1(I-P) D,- F V-1 (I- P) Dm,
-D' V-1 (I- P) Ej D V-1 (I- P) Em
-DV-I-P)F -D' V-1(I-P)F
+Ef'V-1PEj + E' V-1PEm,
+E V-1PF + E' V-PF.

Now note that

CP C X Lx(Xv-lx)- xv-1'
80, 80,
= x(x'v-lx)- x'v- Ov V-x(xv-lx)-l x'v- x(xv-ix) -1 xv-1 v v-1

which leads to








02p = x(xv-ix1 xv-1 Ov V-Ix(xv-ix)1 xv-1 v -imx(xv-ix) 1 x,v-1
-X(XVl-X)- X ) V-1 OV V-1 V V-(X'V)-lx- X1 XVX-1
+x(Xfv-lxx-1 -XVX 02V viX(XV1xX)-XV1
ao a9
J



+X(X'V-lx)-1 XV-1 V V-1x xv-ix-1 XV-1 V-1 xfxv-x-1 xV-1
ao, a00

X XV-iX 1 X'V-1 V-iX XV-iXy1 x'V-1 V V-i1
+x(x'V-lx) x'v-1 Ov V-i vlx (x lx)-lx' 1 v -1 l
S, 0 9
XX'V-lx)-l X'V-1 V V XX'V-lx -l X'V-1 OV V-1
a, a e,

+X (X'V- X)-1 X'V-1 V V-1 V V-1 +x+ X(X'V- X)-1 X'V-1 OV V-1 V V-1
a0 00, 00, a00
-x(xv-lx)-lx'v-1 2V -1

Now we have
02p
MkGZ'V-1 ZGM' =-F'V1(I-P)F F'V-i (I-P)Fm, -FV-1 (I-P)E

-Fi'V-1(I-P)E, +MkX(X'V-lX) X'V-1 2 V-ixfxV-ix)1 XV-iM'

0o-o v (2V x'
-MkX (XV-lX)' XV-1 V-iM' ,

a2V 1
To complete Equation (3-18) we also must investigate- :
90,90

a2V-1 1 'V V -1 VV-1 V 1 1 VV-1 V V-1 V V-1 _V-1 2V V-1
ao0,ao ao0 a0, ao, ao0 ao0 ao, ao0,a








This gives us
82V-1 BV 9V
MkGZ' (I-P)ZGM' = MkGZ'V-1 V-1 V-1(I-P)ZGM'

GV 1V 1 2V V 1
+MkGZ'V-1 VV-1 V V-1 (I-P)ZGM' -MkGZ'V-1 V (I-P)ZGM'
0oj 00, o 00,08J
k71 1 -1 1 -1 -1
=E'V-1Em +EV-1E +EV V-F +EV-F -MkGZ'V1 V (I-P)ZGM'.
E2V+VE Fo

a2v-1 2p
Utilizing these expressions for and -- in Equation (3-18) results in

a2G a2G
(4) = Mk M' Mk Z'V-1 (I- P) ZGM'
S90,90 m k 00,0 m
M2G 02p
-MkGZ'V-1(I P)Z MG +MkGZV-1 2 ZGM'
ao, a~a ao, aoo
a2V-1
-MkGZ' (I-P)ZGM'

-D V-1 (I-P)Dmj DV-1 (I-P)Dm, EV-1 (I- P) Dmj EV1 (I- P) Dm,
-F' V-1(I- P)Dmj -F' V-1 (I-P) Dm, D'fV-1(I-P)Emj DfV-1 (I-P)Emz
-D',V-1(I-P)Fmv- -D V-1(I-P)F, -E',V-1(I P) Emj -E'kV-1(I P)Em,
-E',V-1(I-P)Fm -E'kV-1(I-P) F, F' V-1(I-P) Ea- -F' V-1(I-P)Em,
82V
-F' V-1(I-P)FM + F' V- 1(I-P)F + MkGZ'V-1 00V-1(I-P)ZGM'

+MkGZ'X(X'V-'Xxl X'V1 x 2V V XXVX1 X'V-ZGM'
( l2Vm (3-19)
-MkGZ'X(X'V-IX)- XV-1 2 V-'ZGM' (3-19)


which reduces to
(4) -(Dk,+E +F) V-(I-P)(D +Emj + Fmj)
-(D, +E, Fk' V-1(I-P)(D, +Ek, +Fk,) (3-20)
82G ( 82V
+f2 2 9








Piecing together (1) (4) in Equation (3-12) results in:
Ak( cov (LP + Mkii, L m + Mmiu)
(A y) (0)) = ---

S(Ck, + Dh + Eb + F, ) ) (I- P)(C, + D,, + Emi + FmJ)
(3-21)
-(C, +D +Ek +F )'V-1(I-P)(Cm +Dm +E +F,)
+f 02G 02V
+f a +g av .

S2G 2V
When G and V are linear in 0 the functions of f and g --- will be zero. In that

case

(Ckl + Dkl +Ekl +Fk)
Akn (0) =2 -2
F (3-22)
(Ckr +Dkr +Ekr+ Fkr)
xV 1(I-P)[(Cm +Dm +E, +F F) -. (Cr +Dm +Em +F, )].

Comparing Equation (3-22) to Equation (3-2) gives the desired result that in the most general

case involving multiple dimension linear combinations of fixed and random effects

BIASKR (0) = C' (0) when G and V are linear in 0. Thus, just as in the single dimension

fixed effects linear combinations, when G and V are linear in 0, MK (o) = M, (0)+ 2C"K () is

an estimator of the MSEP of the linear combination of fixed and random effects that accounts for

both the correction term and the bias correction term.

3.2.3 Definiteness Properties of the Estimator for MSEP for Multiple-Dimension Linear
Combinations of Fixed and Random Effects
Now that all pieces of the estimator have been defined in the multiple dimension setting an

estimator for the MSEP can be defined as









MK' (0) = M ()+C (C ) + BIASK (6) (3-23)

It has not been proven that this MSEP estimator, either in single or multiple dimensions,

is non-negative definite. Just as in the single dimension case, we have shown above that when

the variance matrix for the model is linear in 0, the bias correction estimator, BIASR (0),

reduces to the correction term, C (0) which we have shown is non-negative definite in its

most general form. Hence the simplified MSEP estimator,

M M (0) = M () +2C () (3-24)

is nonsingular and therefore a valid measure of prediction error. However, when the covariance

matrix, V, is not linear in 0, neither the bias correction term nor the MSEP estimator including

the bias correction term has been previously shown to be non-negative definite. In Chapter 4, by

exploring the special case of the balanced one-way random effects model, we provide a counter

example to prove that the bias correction is, in fact, indefinite. Indeed, we show that in a single

dimension case the estimate of MSEP given in Equation (3-23) can be smaller than the naive

estimate, M (9). From Equation (3-21) it is clear that the indefiniteness of BIAS' (0) is

S2G ( 2V
caused by f and g Note that it is not necessary for the bias correction


term to be non-negative definite for the overall estimator of the MSEP to be non-negative

definite. In fact, it is reasonable for the bias correction term to be indefinite since it is estimating

the bias of M, (6), which is also indefinite. In Chapter 4, we provide a single dimension case

where the negativity of the bias correction term causes the entire estimate of M1 (0)to be








negative, proving that M (o)+ BIAS" (o), the overall MSEP estimator for M1 (0), is

indefinite and a poor choice for estimating the MSEP.

3.3 Transform Invariance of MSEP Estimator
Another important issue concerning these estimators is transform invariance. Consider

two parameterizations of a model, 0 and D = g() Kackar and Harville (1984) showed that

the approximation for the correction term in the single dimension case, tr A(0)B(0)], is

transform invariant under the conditions that 4 = g( ) and that


B* (0) =[J () B(g(0)) J(0) where B*and B are as in tr[A(0)B(0)] and J(0) is the

Jacobian of the transformation. Both of these conditions are satisfied by using either REML or

ANOVA variance component estimation and with B(0)= I 1 (), the inverse of the observed

information matrix. This proof applies to the multiple dimension case, as we have shown that

the multiple dimension correction term approximation is simply a matrix composed of the single

dimension approximations as the components. Since each element of the matrix is

transformation invariant, the entirety is transform invariant as well.

It is also easy to show that the naive estimator of the MSEP, M1 (0), is also transform

invariant under the condition that F = g( ) The invariance of the naive estimator indicates

that the true bias of M, (6), E[M ()] M1 (0), will also be transform invariant (again with

the condition, 4 = g ()). Thus a desirable property for the estimator of the bias correction term

is transformation invariance. Clearly, BIAS" (0) is not transform invariant. In Chapter 4, we

demonstrate the lack of transform invariance for a single dimension linear combination in the









one way random effects model. This is an undesirable quality of the bias correction term

estimator. We will investigate its effects on the balanced one-way random effects model in

Chapter 4.









CHAPTER 4
BALANCED ONE-WAY RANDOM EFFECTS MODEL

Many of the methods summarized in Chapter 2 were developed only for estimating and

testing fixed effects (e.g., Giesbrecht and Bums 1985; Kenward and Roger 1997); however, the

methods are often extended to include random effects without documented justification (SAS

2003). In Chapter 3, the elements involved in extending these methods to the most general cases

were derived. By looking at the simplest model including random effects, namely the balanced

one-way random-effects model, we can examine if the use of these methods is as straightforward

as previously claimed or if additional issues arise. The simplicity of this model often allows

closed form expressions to be obtained for the approximations and estimators of the MSEP and

their expected values, in addition to the true value of the MSEP being estimated. Peixoto and

Harville (1986) derived closed form expressions for the true MSEP for a class of models

including the balanced one-way random effects model. Similar methods are used here to enable

direct comparisons of various MSEP estimators. By using this simple, balanced model, we can

measure the result of including random effects on the MSEP estimates, as well as identify issues

that may be more difficult to handle in more complex models.

Consider the model:

y, = + + e,, i = 1,...,k, j= 1,...,n
e N( 0,o2), a- N(0, 2)


where -oc < / < o, o2 > 0, o > 0. We use the parameterization 0 = (o2, ) where

2
v = 2 and consider the problem of predicting the mean of the ith level of the random
a2 + a

effect. Note that the variance matrix, V, is not linear in this parameterization, nor is this the

parameterization used by SAS". This parameterization is chosen to demonstrate the impact of a








non-linear parameterization on the bias correction term and for consistency with past work on

this topic. Comparisons will be made to results under the alternative parameterization,

D = ("2, ) for which V is linear. The BLUP and the naive MSEP, M, (0), for t = a are

straightforward to derive, as demonstrated in Kackar and Harville (1984, p. 857):


(0)= (-v)(y,-), where =, andy =(4-2)
n k


M,(0) = 1 [l +(k -1)] (4-3)
nkv


Noting that (0) is linear in 0, the accuracy of the Kackar-Harville approximation of the

correction term depends on four issues:

1) using the exact MSE of 6 versus an approximation, such as 1V (0), for B(0);
2) the appropriateness of the Taylor series approximation, depending in part on how close 0
is to the boundary of 0;
3) whether 6 is unbiased for 0;
4) the proximity of cov d (y; 0) d (y; 0)', 0) to zero.


We can evaluate these four criteria to determine their effects on the accuracy of the

Kackar-Harville approximation. We shall examine each of these criterion using both REML and

ANOVA methods of variance component estimation.

4.1 Kackar-Harville Approximation Based on REML Estimation Method

Using REML methods to estimate the variance components results in 0 = (i2, i3) where









2 = m, ifma > m
(k -1)m +k(n-1)me im <
nk -1
Smin(me / m, 1) (4-4)
where


me = -1 and ma =
k(n 1) a k-1

(e.g., Kackar and Harville 1984; Searle, Casella and McCulloch 1992).

Note that these estimators remain within the parameter space, 0, by definition. The EBLUP for

t=a is

= )= (1- -)(y, y) (4-5)

and the naive estimate of the MSEP of the EBLUP is


M(2() = [1+(k -1),6]. (4-6)
nk6


The true correction term in the REML case, as derived by Kackar and Harville (1984, p.

731), is

/ c v (nk-1)
C (0)= E[- ]2 = (k 1) (4-7)
nk

where

Y' (- ) 2 (V 1) 1
SB(ve,v)u v ve (v (-1) v(v-1) {

S i. v Iv (ve +1) (1- 2)+ (1- )2f v 1
+ 1 ( I, + ve + 1) 22 2 + ( 2 +
V +V Ve a 1) V 2 V + V








and


B(,b)= F(a)F(b) (a,b)=[B (a,b)]1 t (1-t)b-1dt
F(a + b) 0
k(n 1) k-1 V
Ve V e
2 2 Ve +VV

This results in a true MSEP, MR (0), where

M (0) (0) + CR (0)
M (0)+ c2v(nk- 1) (4-8)
nk

(Kackar and Harville 1984, p. 857).

In this model, the exact MSE matrix of 6 is tractable when using REML estimates for

the variance parameters. Thus assuming that 0 is sufficiently far from the boundary of 1 and

choosing BR (0) to be the exact MSE matrix of 6, we can assess the accuracy of

tr [A(0)BR (0) for CR (0) by simply assessing the proximity of

cov d (y; 0)d (y; 0)',(-0)(-0) to zero.

To evaluate tr [A(0)BR (0)] first note that

A (0) = 22 (4-9)
0 a22

and

BR (0) E (4-10)
E(- ^(2 _2)] E(-3 v)2









var = var(y_ 2 (k-1). As shown by Kackar and Harville (1984, p.
var nkv

v)2 = v2 where

'Ve (1- f)v 2 Va(Ve+1) 1
Bo B(v,,va)v (v -1) v (v -1)(v-2) v(v -2)

+ (2 (V, + 1) 2v, (1- 2) (1 )2
+I (v,) (vo-,2)(v2)(v-1) v2 v 2


tr[A(0)BR (0)] = a22vc = 2k -1)
nk

A Kackar-Harville approximation to the MSEP is then

o2v(k -1)^
M (0) + (k 1)
nk


(4-11)


(4-12)


(Kackar and Harville 1984, p. 857).

Now note that the only non-zero component of cov d (y;0) d (y;0)', ( -)-)' is


cov[(y- y)2,( u)2 =E(y,- y)(-v)2 ]-E[(y,-y)2[E(O-v)

Kackar and Harville (1984, p. 731) show that

E[(y- y)2(v-u) =-] CR(0)
k = cR (o)


where a 22


731), E(3(


As a result,









It is also easy to verify that E (y y)2 = k-(2 +n 2), since ~ Thus
nk a +no-,






o- v (nk -1) oC ug (k -1)
nk nk

Notice that this is simply the difference between the true correction term in the REML

case, CR (0), and the approximation, tr [A(O)BR (0)]. This is only true when the exact MSE of

6 is used for BR () Thus the condition for CR (0) = tr [A(0)B (0)], i.e., the condition for


cov d(y;0)dty;0 ),(- 0 0-0 0)'=0 is


S(nk-1)= k (k- 1). (4-14)

Figure 4-1 shows Equation (4-13) as a function ofu for several values of k, holding n = 6,

with the condition in Equation (4-14) met when the functions cross the zero line. Note that

throughout, without loss of generality, we hold C2 = 1. As k increases, the functions move

closer to zero, indicating that the Kackar-Harville approximation for CR (0) improves as k

increases. However, as k increases, the amount of information about the random effect increases,

reducing the uncertainty about the covariance parameter estimates. Hence CR (0) tends toward

zero and accounting for the correction term becomes less important, rendering the correction

term approximation irrelevant. In the remainder of this chapter, the analysis will focus on a

model with k = 6 levels of the random effect to demonstrate the impact of the different estimators

for the MSEP.








In most models, the exact MSE of 0 is not tractable and an approximation of the MSE of

0 must be used. A common choice is I'1 (0), the inverse of the observed information matrix,

with 0 replaced by 0 in practice. Consider the Kackar-Harville approximation of CR (0) where

B(O) is the asymptotic MSE of 0 IV1 (0). Kackar and Harville (1984, p. 857) show that


tr [A(0) ()0)] 22(kn 1) (4-15)
k2n(n 1)

where

2( 2)2 2a"2
k(n-1) k(n-l1)
T1( (k) (4-16)
2U20 2v2(k -1)
k(n-1) k(n-1)(k-1)


Figure 4-2 compares the two Kackar-Harville approximations in Equations (4-11) and (4-15)

with the correction term, CR (0), for k = 6 and n = 6. The success of the approximation is greatly

influenced by the value of v. For small values of v, which correspond to large values of a2

relative to "2, tr [A (0)11 (0)], is superior to tr [A (0) B (0)] for approximating the true

correction term, CR (0). However, as v moves toward 1, which corresponds to the value of 02

growing relatively smaller, the roles reverse and tr[A(0)BR (0)] becomes the better

approximation. In some cases, the choice of how to measure the variability of 0 may cause

severe over-estimation of the correction term, in turn causing a greatly inflated value for the

MSEP estimate. SAS PROC MIXED uses tr [A(0) 1 (0)] which overestimates the true









correction term by over 100% for half of the parameter space. Using an inflated estimate for the

MSEP could lead to overly conservative interpretation.

Because the correction term approximations in Equations (4-11) and (4-15) make use of

the unknown variance parameters, estimates of the variance parameters must be substituted into

these approximations. Note that there are now two levels of error being introduced into the

correction term estimators: first, the true correction term is approximated by the Taylor series

methods to yield the Kackar-Harville approximation; second, the variance parameters involved

in the approximations are replaced by their estimators. In the case of tr [A (0)I1 (O) there is a


third contribution to the error in that V1 (0) is an approximation of the MSE of 0. Substituting

the REML estimates of the variance components into Equations (4-11) and (4-15) gives two

possible estimators for the correction term:


tr [A () BR (6= k- (4-17)
nk


and

[r A()-1 ()1 2&2 (kn 1)8
Lr A (o)I() k 1)(4-18)
k 2n(n 1)

It is suggested by Prasad and Rao (1990) that the bias introduced by substituting variance

parameters in these approximations is o(k ). To assess this assertion, we investigate the

accuracy of Equations (4-17) and (4-18) by considering the expected values of both estimators.

The expected value of Equation (4-17) is not available in closed form, so a simple Monte Carlo

study was performed to evaluate the accuracy of the estimator for the one-way random effects

model in Equation (4-1) with k = 6 and n = 6. Using the RANNOR function in SAS, 10,000









independent sets of e, and a, were generated from N(0,o2 = 1) and N(0, ua) for several values

of a respectively. The values of a2 correspond to v = {0.05, 0.1, 0.15..., 1.0}. Setting

i = 0 10,000 sets of {y, : i = 1,...6, j = 1... 6 were created from the model y, =a + ej The


Monte Carlo value of E tr[A (0BR (Oj=) E 2 (k 1)o and the Monte Carlo standard


error were obtained from the 10,000 replicates. Selected results of the study are contained in

Table 4-1 (standard errors contained in parentheses) along with the values of CR (0) (Equation

(4-7)) and tr[A(0)B (0)] (Equation (4-11)).

Table 4-1. Accuracy of Kackar-Harville estimator (4-17) for CR (0)
V

0.2 0.4 0.6 0.8 1.0

CR (0) .01537 .01890 .01758 .01660 .01850

tr A(0)BR (0)] .04447 .03384 .01950 .01097 .00954


E AtrA(0)BR (0} .03567 .02652 .01951 .01513 .00954
(.00013) (.00013) (.00011) (.00009) (.00002)
**Monte Carlo value. Standard error of simulation in parentheses.


Figure 4-3 shows the results of the Monte Carlo study graphically. The accuracy of the

estimator again largely depends on the value of v. For small values of v, which correspond to

values of a2 that are large relative to 2, the estimator for the correction term is not accurate

with expected values more than twice the function it is estimating. However, as v approaches 1,

the estimator for the correction term significantly underestimates the true correction term.









The expected value of Equation (4-18) can be expressed in closed form if we assume

normal distributions. First define

k(n )me (k )ma Ue
-e 2 + 2 W = +2 e +U S = Ue + Ua



It is easy to show that ue and u, are independent chi-square random variables with k(n-1) and (k-

1) degrees of freedom, respectively (e.g., Searle 1971, p. 410). Also, w and s are independently

distributed where w has a chi-square distribution with (kn-1) degrees of freedom, and s has a

Beta(ve, Va) distribution (e.g., Johnson and Kotz 1970, Sec. 24.2). Now note that if m, > me then

me <1 so that = min -e, = -e but when ma < me, >1 so that = min -,1 = 1.
ma m ma ma )


0 ifma >me
[0ifm, < m,


E tr A V kx k22 n---) E^u}



k^7 -1 K ma-me) +E (- 3(mo,me) .
k2 n-) a



Noting that if m> me then s < so that 3(s, )= < =(ma,me)
0 if s > 20ifm
m2 (2Va ws 2
and that = 2 we see that
ma k(n- 1)v, (1- s) we see that









/ e 2 V 2 [ WS2
(1) E (n,,e) = E 9(s, )
( nmz k(n 1)v, 1-s


k(n l)ve ) I -S I


(4-20)


K k(kn -
k(n l)v,

a~ -- (kn -
k(n -1J)Ve


J s2 1 ( s) 1
0 l-s B(,ve, V)
B(v + 2, v 1)
1) .. I,(ve + 2, va


Now note that ue


a sw
sw so = and ma
k(n 1)


(a2 +na2)(1- s)w
= -which leads to
(k 1)


E (k 1)n M'(1-g M IM
))= E .-m 1-3(m -n( ,n)))

+E k(n-)Me (1-(M, me))
\ kn-1


1)ma + k(n -1)me1 (m,me
kn-1


(4-21)


C-2 + -2 E(w(l1-s)(1- (s, )))+ 2 E ws(1- (s, )))
kn 1 kn-I1

(2 +n2) (1- S)S d a ds
SB(ve, v) B(ve,va)
) B(ve, v +1) + 2 B(ve +1, v) +
+n ) [-Iv, +)] + [1I,(V, + ,V))]
B(v,,v,) B(vev )


( 2+na ) Va
Ve + Va


IT(Ve,V +1)] +2 e [1-I (Ve+1,Va)].
Ve + Va


Substituting Equations (4-20) and (4-21) for (1) and (2) in Equation (4-19), respectively, we have


2(kn 1)
k2n(n -1)


2(kn -1) { 2vu(kn- 1)vB(e + 2,
k n(-n 1) k(n -)vB(


-1) ,(ve + 2,va
Ve, va)


(4-22)


2V V
+a e-+ [-I,(v, +1,v,)]+ -v [1-I,(ve,v+l)] .
Ve [1 (Ve + 1,V)] +0-1 V [
Ve +Vo Ve + Va


(2)=E (k


tr (A(6)1r (6))]









Figure 4-4 shows the accuracy of the Kackar-Harville estimator (Equation (4-22)) for the

Kackar-Harville approximation (Equation (4-15)) and the correction term (Equation (4-7))

holding k= 6 and n= 6. Just as for the approximation, the accuracy of E r (A ()) ())


depends largely on the value of v. While tr [A (0) ()] is a fairly accurate estimator for

tr [A(0) -1 (0)1 over the entire parameter space, it may severely over-estimate CR (0) for larger

values of v. We wish to assess the assertion of Prasad and Rao (1990) that

E[tr(A(O),- ())]= ,A(0) -1 (0)+o(kk ) forthis model. Notingthat lim Y = yieldsthe

result that lim I, = 1 so that
k->

E[tr (A() 1 ())]tr [A( 0)-1 (0)]

2(kn -1) (kn )vB(ve + 2, v -1)I, (v +2, v -1) 1
k2n(n 1) k(n )vB(v, va)

+a2U-1 v [l_ (V +l ]+a20-2 [1-I (V",v+)]
Ve+Va Ve+a J
=(kl)

confirming, for this example, the assertion of Prasad and Rao (1990).

These results also show the importance of studying the effect of substituting variance

parameter estimates for the unknown values in the approximations for the correction term, the

impact of which has been typically overlooked. In the case of tr [A(0)BR (0)1 substituting

variance parameter estimates improves the accuracy for CR (0) (in the long run). However, in

the case of tr [A(0)1-1 (0) the substitution creates more inaccuracy for CR (0) for most values

ofu.









Figure 4-5 compares the accuracy of Equations (4-17) and (4-18) for the correction term.

The plot shows that tr A (0)BR (0) is more accurate (in the long-run) than tr LA (0) 11 (0)

for ) > 0.30; however, for u < 0.30, the reverse is true. This is the same phenomenon that we

noticed with the approximations in Figure 4-2. Note that SAS PROC MIXED by default uses

1-1 (0) with REML estimates for 0 in the calculation of the MSEP estimator.

4.2 Kackar-Harville Approximation Based on ANOVA Estimation Methods

The best choice of method for estimating 0 may depend on the model of interest, the level

of balance in the design, and the goals of the analysis. Stroup and Littell (2002) demonstrate the

effect of different estimation methods on the power and control of type I error for hypothesis

tests on the fixed effects in analyzing an unbalanced, multi-location experiment. Their study

indicates that REML and ML are not always the best choices when negative estimates of

variance components are likely, and may lead to inflated Type I error rates and low power. The

impact of the choice of variance component estimation method and negative variance component

estimates on the Kackar-Harville approximation has not been investigated thoroughly. In

Chapter 5 we elaborate on the impact of negative variance component estimates. To begin,

however, we develop the correction term, several approximations, and the expected values of the

estimators under ANOVA estimation for the balanced one-way random effects model.

The ANOVA estimators of 0 = (2, ) are = (2, j) where









k n
yZc, -)2
=1- 1 =1
me
k(n --1)


(4-23)


(2 + n(,2 m
a a


ma -me
n


Sma


k

=1 -l
k-1


These estimates are obtained under the METHOD = TYPE 3 option in SAS PROC MIXED.

The true correction term, CA(0), is derived under normality assumptions as follows:

k
CA (0)= E (y,-y)2 )2 (kn)-E nE (y t)2i )2

c2 +n2 (k -)ma + o [2 [ )2]
a2 E = -E 2u (-)
kIn a2 kn I


2n -
S --aE [w(1-s)(_-t )
kn '1


72 2
av+ 2E w(1
kn


.2 2
c2+n a v2E w(1
kn


s V(1) -s)
Ve (1


s) 5 1)2
<"<
^ y


S2 2E(w)E (1
kn


SC2E(kn 1) E 1 asS(1- 1 2,
kn Sve (1-s)




by independence of w and s, and noting that w -~ k Recalling that s Beta(v, ,) we find

that


vs V 2S2
E ( l-s) -s 1 =E-
Ve(-s) ve (1-S)


vs
2 -+ (1- s)


vB (ve +2,v-
v2B (ve, v )
VB(ve +2,v
2B ( v
qB(v v)


1) + V V + Va
-2 2 +V

1) Va
Ve + Va


VaS
ve(- s)








Thus the correction term is


a 2v(kn- 1) vB[(v, +2,a -1) V,
kn L vB (v,vo) +vo

f^ (4 -24)
Skn v, (v (4-24)

2(kn-1)(k 1)-C 2
k n(n-1)(k 3)

This gives the true MSEP for i&,

M (0) = M (0) + CA (0)
S2(kn 1)(k 1) 2 (4-25)
= M, (Q--+ u.
k n(n-1)(k-3)


To evaluate the Kackar-Harville approximation for the correction term, we need to

determine how to evaluate B(0). We consider three alternatives for B(0): the exact MSE


matrix E o 0o) ( o), the variance-covariance matrix of 0, and 1 (0). Recall from


Equation (4-9) that due to the form of A (0) in this model we are only concerned with assessing

the variability of 5 in order to evaluate the Kackar-Harville approximation to the correction

term.

We begin with the exact MSE of u, E [? v] Noting that m =- Fkn1),k- it is
ma

E k- v(1)\ 2 (k -1) (kn -3)
easily shown that E(?) = V (u) = (k 2k and
Sk-3 k(n-1)(k- 3) (k- 5) and

U (k-1)2 (k(n-1)+2)
E v (k 1)2 (k(n-1)+ 2) After algebraic simplification this leads to
Sk(n 1)(k 3)(k -5)








E[ v]2 = E ( 2)-2vE ()+2

k k(kn+3n 5)+ ~1 (4-26)
k(n-l)(k-3)(k-5)

Letting BA (0) = E (-)( 0-) ] we have the following approximation for C (0):


S( B 2(k-1)(k(kn+3n-5)+1)
trLA )B )] k n(n- 1)(k-3)(k 5) (4-27)

Substituting the ANOVA estimates in Equation (4-23) for 0 yields this estimator for CA (0):

S2(k-l) (k(kn + 3n- 5)+1)(
S[A(6)n k(n- 1) (k -3)(k -5)

To assess the accuracy of Equation (4-28) for estimating Equation (4-27) and CA (0), we

2 2 2
VCUU Va WS
evaluate the expected value. Recalling the relationship = k(-Tv where w and s
ma k(n- 1)v, (1-s)

are independently distributed with ~ 1_, and s ~ Beta(ve, va), we have


E \~ \=E E[w]E
Im a I= k(n- l)v, (1-s)

( 2 v B (v, + 2, va- 1)
=k(n )v (kn 1 B(v,, va (4-29)

(k-1)(k(n-1)+2)C
k(n-1)(k-3)


Thus,


r A(F)B (u )= 2(k -1)2 (k(kn + 3n- 5)+ 1)(k (n-1)+2) 2
E trAn(n 2 (kA 3)2 (k_ 5)v.
k n(n -1)(k -3)(k -5)


(4-30)








Again, we confirm the asymptotic assertion by Prasad and Rao (1990), noting that

E tr [A()B,()] tr [A(0)B, (0)]
2(k- 1)2(k(kn+3n-5)+1)(k(n- 1)+2) 2 2(k-l)(k(kn+3n-5)+1) 2
k3(n- 1)2(k- 3)2(k- 5) k2n(n-1)(k-3)(k-5)
2(k-1)(k(kkn+3n-5)+1) (k-1)(k(n-1)+2) 1
k2n(n- 1)(k- 3)(k- 5) k(n-1)(k-3) J
o(k 1)

The second alternative for B(0) is the exact variance-covariance matrix of 0, B* (0).

Because 5 is not unbiased for v, this is not the same as the MSE matrix for 0. As noted above,

2t2 (k- 1)2 (kn 3)
V () 2 = )2(k -3) which leads to the following approximation for CA (0):
k(n 1)(k -3)2(k -5)

2(k 1)3 (kn 3)
tr A(0)B* (0)= C2V. (4-31)
Skv 2 n(n-1)(k-3)2(k -5)

Again substituting the ANOVA estimates for 0 in Equation (4-31) yields an estimator for

CA(0):

tr[A()B (62 (k-1)3 (kn 3)
tr A()B ()1= ~2)_3) z. (4-32)
A u M k2 n(n-1)(k-3)2(k-5)


Using Equation (4-29) we have

Etr[A()B ()} 2(k-1)4 (kn-3)(k(n-1)+2)
Etr A 0) B 0)) =- v (4-33)
Sk3 n(nu l)2 (k -3)3 (k- 5)

Again we find that








tr [A () B ()] tr [A (0)B ()]
2(k- )4(kn-3)(k(n-1)+2) 2(k- )3 (kn- 3)
k3n(n-1)2 (k-3)3 (k -5) k2n(n-1)(k-3)2 (k5)
2(k -)3 (kn-3) 2 (k- 1)(k(n-1)+2) 1
k n(n 1)(k -3)2 (k 5) k(n -1)(k-3)
=o(k ).


The third alternative we consider for B(0) is to use the inverse of the observed

information matrix, IV' (0), as an approximation. Note that I1 (0) is the same as the asymptotic

variance of the REML estimators of 0, in Equation (4-16). Thus the Kackar-Harville

approximation with ANOVA variance component estimates using V1 (0) is the same as in the

REML case in Equation (4-15), i.e.,

tr [A(0) (0) = 2(kn 1) a2 (4-34)
k k2n(n 1)

and the estimator formed by substituting ANOVA estimates for 0 is

tr A( )T1' 2(kn 1) (4-35)
L ] k2n(n 1)

Using Equation (4-29) we have


k2n(n-1)
2(kn 1)(k 1) (k (n- 1)+ 2) (4-36)
= 0.2/-- ~)-- -- o
k3 n(n 1)2 (k- 3)

which again yields








E{tr [A )] -t[A(o)I-1()]
2(kn-1)(k-1)(k(n-1)+2) 2(kn- 1) 2
k3n(n )2 (k 3) k2n(n 1)
2(kn- 1) (k-1)(k(n-1)+2) 1
k2n(n-1) L k(n-1)(k-3)
o (k- )


Figure 4-6 compares the three Kackar-Harville approximations in Equations (4-27),

(4-31) and (4-34) to CA (0) for k = 6 and n = 6. Clearly tr [A()I 1 (0)] more closely

approximates CA (0) than the other two approximations, for all values of v. Thus, although we

would expect 1-1 (0) to be less accurate than BA (0) or B* (0) in evaluating the MSE of 0,

especially for small values of k, the approximation of CA (o) utilizing 1-1 (0) is more accurate

than the either of the other choices across the entire parameter space. In fact, the Kackar-

Harville approximation utilizing the exact MSE of 0 is the least accurate of all three

approximations. Note that this is in contrast to the results from REML variance component

estimates in Figure 4-2, where the best approximation depended on the value of u.

Figure 4-7 compares E{tr[A())BA (6)) (Equation (4-30)) to tr[A(0)BA ()] and the

correction term, CA (o) The bias for the correction term introduced by replacing the unknown

variance components with ANOVA estimates in the correction term approximation is significant

and increases as v moves toward 1. Figure 4-8 demonstrates a similar result for the Kackar-

Harville approximation and estimator utilizing the variance of 0. E tr [A () B* (0)]

(Equation (4-33)) is again inflated over the approximation and they both overestimate the true








correction term increasingly in v. Figure 4-9 shows a much different result than the previous

two figures since the accuracy of the estimator utilizing I' (8) (Equation (4-36)) is improved

over that of the approximation (Equation (4-34)). While tr [A( 0) (0) underestimates CA (0)

by up to 50%, E trA(0) ( ) is much more accurate and consistently overestimates

CA (0) by only 6.67% across the entire parameter space.

Figure 4-10 compares all three estimators (Equations (4-30), (4-33), and (4-36)) relative

to CA (0). Clearly the Kackar-Harville estimator utilizing IV' (8) is far more accurate than

either of the other two choices. This is counter-intuitive: the Kackar-Harville estimator using an

approximation of the MSE of 0 outperforms the other estimators using exact measures of the

variation of 0. Note again that this is a different result than seen with REML variance

component estimates where the results were dependent on the value of u. There, we saw that

the Kackar-Harville estimator utilizing the exact MSE of 0, BR (0), often performed better than

the one using an approximation of the MSE of 6, I1 (0) (see Figure 4-5). The choice of which

Kackar-Harville estimator for the correction term is best, therefore, depends, in part, on what

method is used for variance component estimation.

The final step in this assessment is to compare the ANOVA results to the REML results.

Figure 4-11 compares the accuracy of tr [A(O)I' ()1 (Equation (4-36)) for CA (0) with the

accuracy of the two REML estimators, tr A (0)BR ()] (Monte Carlo expected value) and

tr IA(0)I' (0)1 (Equation (4-19)) for CR (0), with k= 6 and n = 6. The ANOVA estimator









utilizing the information matrix, tr A () 1 (o)1, appears to perform better than either REML

based estimator, although the added variation to the MSEP of i is larger for the EBLUP with

ANOVA variance components, i.e., CA (0) > CR (0). We can assess the accuracy relative to the

appropriate correction term to better compare the performance of the estimators. The relative

bias of the two REML based estimators of the correction term and the ANOVA based estimator

with 1-1 (8) is tabulated in Table 4-2 and graphically depicted in Figure 4-12. Relative bias was

calculated as

E t[AB]) C(0)
C(0)

The ANOVA estimator of the correction term has consistently lower relative bias than either of

the REML estimators. Thus, although the variation added to the MSEP by using ANOVA

variance component estimates is larger than when using REML variance component estimates, a

more consistently accurate estimate of the added variation is available in the ANOVA case.

While the accuracy of both REML estimators depends largely on the value of u, the ANOVA

estimator is significantly more accurate across all values ofu.

This special case has demonstrated the importance of the choice of variance component

estimation method, the choice of measure of variation of 0, and the added variability of

substituting parameter estimates for unknown variance components in the correction term

estimates. While ANOVA methods add more variability to the EBLUP than REML estimation,

the estimator for the MSEP of the EBLUP is more accurate with ANOVA methods than with

REML methods. The best choice of how to measure the variability of 6 was not consistent

across variance component estimation methods. For ANOVA methods, a better choice for B(0)









was I1 (0), while for REML estimation the better choice depended on the value of v. We also

discovered that the effect of substituting parameter estimates depends, in part, on the choice for

B(90)and the variance component estimation method. The impact of this substitution is often

considerable and could cause the MSEP estimate to be significantly inflated.



Table 4-2. Relative bias (%) of estimators of correction term


0.2 0.4 0.6 0.8 1.0

REML**
EtrE A( )BR( 132.0 40.3 11.0 -8.8 -48.5
E tr [A( )B,()]
REML
E Itr A()I1 ()l 33.7 85.0 152.3 202.8 220.8
ANOVA
E tr A() l()] 6.7 6.7 6.7 6.7 6.7
**Monte Carlo values used to compute relative bias.



4.3 Bias Correction Term for the Balanced One-Way Random Effects Model

4.3.1 Bias Correction Term Approximation under 0 Parameterization

We now address the bias correction term approximation for the one-way random effects

model. We will consider the case where B(0) = V1 (0) so that the calculations for the

approximation will be the same for the REML and ANOVA, and so that closed-form expressions

are attainable. The difference comes in the expected values of the estimators of the bias

correction term due to the truncation of the REML estimates. We will show that in either case

the bias correction estimator can be negative, thus proving by counter-example that the bias








correction estimator is indefinite. We begin however, with demonstrating the lack of transform

invariance.

Recall that the parameterization we are working with is 0 = (o2, ). If we were

considering the parameterization ( = (C2, ) wherein the variance matrix, V, is linear in the

parameters, the bias correction term would be identical to the correction term approximation,

which is transform invariant, i.e., -tr (A ()B(())= tr (A ()B( ))= tr (A(0)B(0)), and

non-negativity would not be an issue since tr (A (0)B (0))is non-negative definite in its most

general definition as proven in Chapter 3. This is not the case with 0 as the parameterization, as

will be demonstrated in this chapter, since -tr (A (0) B (0)) # tr (A(0) B (0)).

We now give a simple demonstration that the bias correction term approximation is not

transform invariant, as discussed in Section 3.3, through the continuing example of the balanced

one-way random effects model. The true bias of the naive MSEP, EM1 () M1 (0), is

invariant to transformation as long as r ()= r( ). This follows from the definition of


M, (0) = E [(0) t]2 Thus the lack of transform invariance of the bias estimate is a concern.

To calculate the bias correction term approximation under the 0 parameterization, we

first compute the second derivatives of the naive MSEP,M1 (0), with respect to the variance

parameters:

M, (0) 1 -2v+kv ku2 +2
dO2 nkv
-2M (0)
= 0,
(aoU2)2








and


M, (0) -_ (k- 2 + 2v(1-k)) a2 (1 2u+kkv kv +2)
uv nkv nkv2
2(v2(1-k)-1)
nkv2

a2M1 (0) 202
(aU)2 nkv3


2M (0) _(1
B02av


-k)v -1
nkV2


So,
0
(0) (1-k)2 -1

nkv2

Utilizing I11 (0) from Equation (4-16), we have
202 {(1-k)v2- 1
nk2 (n 1)
A(0)IT( (0)
S20 C
f (C v) nk


(1- k) 2
nku2
k2
202
nku3


f( v)


(1-k)v -1) 402 (kn- 1)
2 (n-1) nk2 (n-1)(k-l)u


where f(o, v) is a function of 0 = ( v) that is inconsequential in this calculation.

This gives


(1-k)v -1 kn-1
tr [A(0) 1 (0)] = 4C2 F + 21 kn 1
L nk (n-1)v nk (n-1)(k- 1)

Thus the bias correction term for the naive estimator of the MSEP under the 0

parameterization is


(4-39)


(-,2v)


(4-37)






(4-38)








1F(k-l)v2 +1 kn-1
BA2 (0)nk (n-1)v nk (n-1)(k-l)u (44

which is obviously different from the bias correction term approximation used under the (

parameterization, which would be identical to Equation (4-15). Clearly the lack of transform

invariance is attributable to the higher order derivatives in A (0). We will examine the impact of

the parameterization by comparing each bias correction estimate to the true bias.

4.3.2 Expected Value of BIASR (0) with REML Covariance Parameter Estimation

We begin by calculating the expected value of the bias correction term estimator once the

REML estimates are substituted for the unknown variance components.

Recall from Equations (4-19), (4-20), and (4-21) that

E (j6 c 2 o2v(kn-1)vB(v, + 2, v -1)I, (v + 2, v -1)
Sk(n -1)vB(v, v,))
(4-41)
+2 [1 _I(V+1,V)]+U-2v-1 [1-I J(Ve,( +)]V .
Ve +Va Ve +Va

We also need

EK2 (m(k 1) m + k(n -1)mn (i (mm)) (4-42)
E = E (nMana IMe +E (1 -'(n^,Ma ))e (4-42)
v kn 1


where

)=2 + E(w -s)(s,2))

=2 1 /-l-, 1(l-s)l- -
k-1l- B(ve,v)
= 02 ll y(Ve,Va +1)

and from Equation (4-21),








(2)=(o2 +na) V' [I_,(v,",+l)]+C2 V, [_1-I(v +1,v)].
Ve + Va Ve + Va

We can utilize Equations (4-41) and (4-42) to compute the expected value of BIAS (0) as

follows:

E[BIAS (0)= E -ltr A(0I) 0)(

=2Ej2 (k -1)i +1 kn-1
[nk 2(n-l ) nk (n-1)(k 1
=2 k(- I -E (/2\ 2 E /2-)
nk2 (n- 1) v nk(k- 1)
=2 k-1 C o(kn 1)vB(v, + 2, v, -1) (Ve + 2, v -1)
nk (n1) k(n )veB(ve, va)




nk (k 1) Ve + Va
+*C2 Ve [1- (Ve+ l,)] 4 4)
Ve + V- aI (4-43)

To evaluate the accuracy of the bias correction term estimator, the true bias correction

term (which is the negative true bias) of M1 (0) for M1 (0) can also be calculated as

M, (0) -EM1 (0)], for REML variance component estimates of 0 Because M () = 0 when

ma








E[M, E I 1+(k-1)"]}

=E ( ) +(k -l1)6 l) 3(],,(m,) (4-44)

E na (k- 2)m, (k -1)n
= E +- 3(Mn) .
nk nk nkmn

We establish that

M (,VI) = (ve, +1), (4-45)
nk nk

(k 2)me M (k 2)2 E(sw)(s,)
nk nk (n 1)
(k- 2)(kn 1) )]
nk2 (n 1) (4-46)
(k-2)(kn -1)2 B(V, +1,v )
nk (n-1) B(v ,,va)
(k 2) 2(
-I,(ve +1, Va)
nk

and
7( 1 nk l( \)77 (kn 1)
Snkma nk k(n )v)
,(B(ve +2, v,) (ve +2, vo-1)2t (4-47)
B(ve,va)
(k 1) (kn-k +2)ac v
2 . I, (v + 2, v 1).
nk (n-1)(k-3)

Combining Equations (4-45), (4-46), and (4-47) into Equation (4-44) gives us the expected value

for M, (9) with REML variance component estimates:








0 21 (k-2)o-2
E M, = I(e,Va + 1+ (,(ve ,+1, a)
nk nk
(4-48)
(k- 1)2 (kn-k + 2)-2v
nk2 (n-1)(k 3)

Thus the true bias correction term of M (0) for M, (0) is


M,(0)-E[M,(0)]= 1+k[l -l)]

(k-1)2 (kn-k+2)v
+ -IT,(ve +72,v, 1) (4-49)
k(n 1)(k -3)
VlI,(v, +1) -(k 2) I, (ve +1,va)}.

We can now make comparisons among the proposed bias correction term estimators of

the naive MSEP, M1 (0). Figure 4-13 pictorially demonstrates the bias of M1 (o) for M, (0)

under REML methods. From this figure it is easy to see that M, (0) has both positive and

negative bias depending on the value of v. Figure 4-14 depicts the true bias correction term,

M1 (0)- E M1 (0), from Equation (4-49), along with the bias correction term approximation,

--tr[A(0)I-1 (0) (Equation (4-40)) and its expected value under REML estimation (Equation

(4-43)). First note that the effect of substituting variance parameter estimates in the bias

correction approximation does not have much of an impact, unlike the results of the correction

term approximation. Although --tr A (0) (0) and E --tr1(A 1 0 1 ) arenearly

equal over the entire span of v, both significantly underestimate the true bias correction term

over most of the parameter space. In fact, the true bias correction term is positive for v < 0.75

reflecting that M, (0) is underestimating M1 (0). However, the bias approximation and








expected value are negative. Thus by using this bias correction term estimator, we would

actually exacerbate the problem and significantly increase the amount by which M, (9) is

already underestimating M, (0). That is, while M, (9) tends to underestimate M, (0) for

v < 0.75, the "correction" by the bias correction term actually causes further underestimation of

M1 (0), especially for small values of v. The use of this form of the bias correction term would

actually produce a poorer estimate for M, (0) than using M, (9) alone. As v increases, we see

that the true bias correction term becomes negative, reflecting that M, (9) is overestimating

M1 (0); thus it is appropriate for the bias correction term estimator to be negative. As shown in

Figure 4-14, the bias correction term approximation performs significantly better in this upper
range of the parameter space than for v <.75.
Now we compare this result to the bias adjustment under the linear parameterization,

S=(o"2, C- ), recalling that

-tr(A () I (0))= tr(A (D)I ())= tr(A (0)1(0)),

where tr (A(0) 11(0)) is in Equation (4-15).

Figure 4-15 shows M, (0) EM (o)1 along with tr [A (0)1 (0)] (Equation (4-15)) and

E tr A () I1 (O)J (Equation (4-22)). We see a much different result for this bias correction


term than we did for tr[A(0)T1 (0)1. Now we see the bias correction term approximation

and estimator significantly overestimating the true bias correction term, especially for large
values of v. For small values of v, the estimator is fairly accurate. This is the exact opposite of








the previous result in Figure 4-14. Figure 4-16 compares the accuracy of --tr A(0)11 ()]

and tr A(0)I (E)] for M, () -E[M (O). For v<.75, M, (0)- EM (O) is positive. In

this range, tr A (6) I1 (0)] is a significantly more accurate estimator because


E I-tr A(O)I(o)]l} is negative. For v>.75, M, (0)-EM, (o)1 is negative, and


Strip A(0)I1 (e ) is significantly more accurate than tr IA(0)I1 ()1. Thus the choice of

which estimator is better depends on the value of v and also on whether it is better to over- or

under-estimate M1 (0). Note that SAS uses the 0 parameterization in the DDFM = KR option

of PROC MIXED. Hence the results from Figure 4-15 are applicable to SAS procedures.

4.3.3 Expected Value of BIAS"' (0) with ANOVA Covariance Parameter Estimation

We can do this same comparison for ANOVA variance parameter estimates. Recall that

the bias correction approximation in Equation (4-40) is the same under ANOVA and REML

variance parameters due to the choice of 1-1 (0). However, because the ANOVA variance

component estimates are not truncated, the expected values of the bias correction term estimator,

E tr [A(0)1 (e)1 and the naive MSEP, ELM, (8)1 will differ from the REML case.

First, recall from Equation (4-29) that

S(k-1)(k(n-1)+2) (
k (n -1)(k-3)

and note that
E[J2z-13= E(m)= E k-l a 2 U-1 (4-51)







Using Equations (4-50) and (4-51), the expected value for the bias correction term estimator is

E[BIAS(6) = E -tr[A(6) -16)]

=2E (2 -(kl) 132+ kn-1 i
Snk2(n-1)u nk2 (n-l)(k-l)0
= 2 k-1( E ( 2)- 2 )E( < )}) (4-52)
[[nk2 (n-1)) nk (k-1)
2f k-1 (k 1)(k(n 1)+2)
nk (n-1) k(n-1)(k-3)

nk(k ) 1)

Note that by allowing the variance parameter estimates to take values outside of the parameter

space, M, (8) does not maintain the non-negative definite property. That is, in taking the

expected value of M1 (8) with ANOVA variance parameter estimates, we are including negative

values of M1 (8). Allowing variance parameter estimates outside of the parameter space in

expected value calculations does not affect the definiteness properties of the bias correction term,

- triA()I1 ()] since it is indefinite as will be discussed in Section 4.3.4. The calculation

for the expected value of M1 (8) proceeds as follows:


E[M ()] = E 1+(k-1)
nkt3
m (k-2) m (k- 1) m2
= E { ( 2)e m (4-53)
nk nk nkm,
C2V1 (k-2) (k- 1) (kn- k+2)aCv
nk nk nk (n-l)(k-3)
nk nk nk (n-1)(k-3)








Thus the true bias correction term (negative bias) of M1 (8) for M, (0) is

M,(0)- [M,(0)] =2 ) 1[+(k-1)u]
nkv
2 -1 (k-2)a2 (k-1)2(kn-k+2)a2u
nk nk nk2 (n-1)(k -3)
2 (k 1)(kn -1)a2
nk (n-1)(k- 3)


We can now compare this to the bias correction approximation,


2tr [A(0)-1(0)] and the


expected value of the bias correction estimator, E tr A() ()-1 ) ,as well as

tr [A(0)I-1 (0)]and E{tr[A( 0)I-()) .

Figure 4-17 shows M, (0) and E[M, (6)]. We see that E[M1 (0)] consistently

underestimates M1 (0) over the entire parameter space, growing more severe as v approaches 1.

Figure 4-18 compares the true bias correction term, M, (0) -[M, ()] to tr[A(0) 1 (0)]


and E--tr A(0)I (eo)]}. We see a very similar result as in Figure 4-14 for REML


methods. Again, while MI () underestimates M, (0), adding tr[A(0)1-1 (0)] as a bias

correction would increase the amount by which M1 (0) is underestimated. The bias correction

term improves as v increases but never becomes positive, and therefore never improves the

unadjusted estimator, M1 ().


(4-54)







Again we want to compare this to the bias correction under the linear parameterization

( = (2,2) Figure 4-19 shows M, (0) ELM, (o)] along with tr[A(0) 1 (0)] (Equation

(4-15)) and E{tr (A )1I (6)]) (Equation (4-36)). The result is much improved in this

parameterization. We can see that tr A( 0) ()] is a highly accurate estimator for the true

bias correction term, M, (0) E[M (6)] and using this estimator would improve the estimator

for M, (0) over the entire parameter space. That is, M, () )+tr A () V ()] is a more accurate

estimator of M1 (0) than M, (0). Figure 4-20 compares the accuracy of tr [A(0)I1 (0) and

- tr A(0)I J0 for M (0) -EiM, .)1. Because M,(0)- E M, (O) is positive over the

entire parameter space, -tr [A( )11 ()] is an unacceptable estimator of M (0) EM ()].

However, tr A(0)I'1 ()] is a highly accurate estimator for M, (0)- ELM (0)] with nearly

constant relative bias of 6.67%.
4.3.4 Impact of Negative Values of Bias Correction Term Approximation
As mentioned in Section 3.2.3 the bias correction approximation given by Equation
(3-10) in indefinite. The balanced one-way random effects model under the

0 = (o2,v) parameterization provides a sufficient example. Figure 4-15, shows that

- tr A(0) 1 (0)1 takes negative values over the entire range of v. We can determine from

Equation (4-40) when the bias correction term estimator will be negative in terms of k and n.
First rewrite Equation (4-40) as









-ltr A(0)I (0)]= 2C2(k21)02+1 2 k-1
2 nk2 (n-1)v nk2(n-1)(k-1)v
2r2 [(k-1)2 v2 -k(n-1)]
nk2 (n -1) (k 1)v

Noting that the denominator is always positive when v is in its parameter space, we determine

that the bias correction term estimator in Equation (4-40) is negative when

k(n-1)
u< \ -.
(k -1)2

For k = 6 and n = 6, as in our example, this value is v < 1.09 which means that the bias

correction term estimator is negative over the entire (0, 1] parameter space for u. This

counterexample suffices to prove that the bias correction term approximation in its most general

form,

1tr (A (0) B (0)) tr {AI, (0) )B (0))
BIAS KR(0) 1
2 tr {(A, (0) B(0)} tr {A,, (0) B(O)}


from Equations (3-10) and (3-11), is indefinite. This in itself is not necessarily problematic since

the true bias correction term, M, (0) -E M ( )], is also indefinite. However, if the magnitude

of this piece is large enough to cause the entire MSEP estimator to be indefinite, then problems

arise. It is beyond the scope of this dissertation to determine if the MSEP estimator utilizing the

correction term estimator and the bias correction term estimator, M (0) + CK (0) + BIASK (6),

is non-negative definite. However, it can be shown that as an estimator for M1 (0),

M, (0)+BIASf (0) (4-55)

is indefinite by looking at the same special case of the balanced one-way random effects model.








The form of Equation (4-55) for this model is

M,( + BIAS' (6)= t ) [1+(k -1) 0]

+2'2 (k 1)v2+1 kn 1 (4-56)
nk2(n-)v nk2(n-1)(k-1)
C 2 ((k -1)2 (2- k(n -1))2 + k(n 1)(k- 1)(k- 2)v + k2(n -1)- 3k(n -1))
nk2(n 1)(k 1)

To determine when Equation (4-56) is negative, we find the roots of the numerator (noting that

the denominator is greater than zero for k, n > 1) by solving the quadratic equation in v. This

yields the following result:

M,) + BIAS (6) < 0

where

3k(n-1)- k2(n- 1) (k(n 1)(k-1)(k1))2 k(n-1)(k -1)(k -2) 57)
(k-1)2(2-k(n-1)) 4((k 1)2(2 -k(n 1)))2 2(k- 1)2(2-k(n- 1))

or

S3k(n -1) k2 (n -1) (k(n -1)(k -1)(k -1))2 k(n 1)(k 1)(k 2)
i <- +
(k-1)2(2-k(n-1)) 4((k 1)2(2-k(n-1)))2 2(k 1)2(2-k(n- 1))

For k = 6 and n = 6 this gives 63 > 0.987 and 6 < -0.130. While the lower bound on 6 is outside

the parameter space, the upper bound is within the parameter space and a valid result for the

parameter estimate. In the case ofk = 3 and n = 3 the result is 6 > 0.75 and 6 < 0. Again, the

lower bound is outside of the parameter space but the upper bound is well within the parameter

space. This demonstrates the danger in using M (6) + BIASK (6) as an estimator for M1 (0) or

M (0) and serves as a general warning of the possible effect of the indefiniteness of









BIASR (0). The simplicity of this example raises even more cause for concern in more

complex models.

4.4 Performance of Overall Estimators for the MSEP

Now that all the pieces of the MSEP estimators have been evaluated, we can combine

these results to get an overall picture for how accurate each MSEP estimator is for the true

MSEP. Recall that the true MSEP values will differ for REML and ANOVA variance

components, since each variance component estimation method adds a different amount of

uncertainty to the MSEP of the EBLUP and a different amount of bias to M1 (0). For REML

methods, we will look at MSEP estimators utilizing both BR (0) and I1 (0) in the correction

term estimator since there was no clear preference (see Figure 4-5). However, for ANOVA

methods, the estimators using I1 (0) were clearly superior to other choices of B(0) and thus we

will limit our comparison to V1 (0) for ANOVA methods. Thus the following equations list the

possible estimators for the MSEP of the EBLUP of t = a For REML variance parameter

estimates we have three options:

M, ()+2tr[A(0)BR ()], (4-58)




M, (6) +2tr [A (6) ()], (4-59)

and


M, ) + tr [A (0)I ] tr A f0)rI 1]. (4-60)



For ANOVA variance parameter estimates we have 2 choices for MSEP estimator:










M, ()+2tr [A ()()], (4-61)

and
M(0)+ tr[A (0)(0)] tr[A (0)I(0) (4-62)



Note that while Equations (4-58), (4-59), and (4-61) are derived under the D = (2, )

parameterization, due to the translation invariance of the elements involved, for simplicity we

will use 0 notation.

We begin with the REML comparison. Figure 4-21 shows the true MSEP for the EBLUP

of t = a using REML variance parameter estimates, MR (0), along with the expected values of


the three options in equations (4-58), (4-59), and (4-60). For completeness, E[M (6)] is

included as well. As we have seen throughout with REML variance parameter estimates, the

results are dependent on the values of u. For small values of v, Equation (4-59) is most

accurate, almost matching the true MSEP exactly. As u grows, though, both Equation (4-58)

and (4-60) become more accurate than Equation (4-59). Note that SAS uses Equation (4-59) as

the estimator of the MSEP in the DDFM = KR option of PROC MIXED, which is not the best

choice over a large range of u. It is especially problematic for u close to one, which again

corresponds to values of crO small relative to 2.

Now looking at ANOVA comparisons in Figure 4-22, we see that Equation (4-61) is

most accurate for the true MSEP, MA (0), over the entire range of u. It is clearly the best

choice for MSEP estimator when ANOVA methods are used.

We also want to determine if REML or ANOVA methods are preferred when comparing

MSEP estimators. To do so, we look at the relative bias of each of the five MSEP estimators









listed above for their respective true MSEP values. Table 4-3 summarizes the results for select

values of v and Figure 4-23 illustrates this comparison. The table and figure clearly shows that

the MSEP estimator with the smallest relative bias over the entire range of v is

M, (0)+ 2tr [A (0)1 (6)] which uses ANOVA methods. Thus for accuracy of the MSEP

estimator, ANOVA variance component estimation under the 0 parameterization with MSEP

estimators utilizing B(D) = I 1 () are the best choice among the ones compared here.

Table 4-3. Relative bias (%) of MSEP estimators under REML and ANOVA estimation

0.2 0.4 0.6 0.8 1

M(O) -13.28 -26.27 -30.88 -22.76 37.38

M, )+2tr A )BR 16.75 10.59 11.71 36.21 140.50

M, ()+22tr [A ( )V () 4.03 22.33 65.91 173.14 620.57

Mi, )+tr A (6) ( -tr A 0)' -26.84 -18.26 2.72 57.93 294.72


M (8) -17.72 -51.38 -93.33 -142.68 -200.00

M,1 ()+2tr[A( )1(e)] 1.19 3.42 6.24 9.51 13.35

M, ()+tr [A (0)(0) -tr [A(0)(0) -29.70 -36.58 -49.77 -67.18 -88.37

**Monte Carlo values used to compute relative bias.

It is important to note that while this is the MSEP estimator used by SAS in the DDFM

= KR option when ANOVA methods are chosen with this model, the MSEP estimates are

truncated at zero which changes the expected value of the MSEP estimator from the result given

here. The disadvantage of using ANOVA methods is the likelihood of negative variance

parameter estimates. However, by truncating the MSEP estimates, SAS negates the advantage









of increased accuracy associated with ANOVA methods. The implication of allowing

untruncated MSEP values is that negative estimates of the MSEP are possible. The impact of

negative variance parameter estimates on EBLUP procedures in the balanced one-way random

effects model are further explored in Chapter 5.

4.5 Comparison of Prediction Interval Methods

Several methods for conducting hypothesis tests in linear mixed models are reviewed in

Chapter 2. Table 2-1 contains a summary of the methods, including how the distribution is

determined and which standard error is used in the test statistic. We now want to examine these

methods in terms of the balanced one-way random effects model. Because we are estimating the

realized value of a random effect, it is more appropriate to examine the methods in terms of

prediction intervals. The goal of this examination is to determine the accuracy of the prediction

interval estimation methods and determine where further investigation is necessary.

Four prediction intervals will be compared in a Monte Carlo simulation study to determine

which method has the most accurate coverage rate. All of the prediction intervals have a similar

structure. The estimate of the realized value of the random effect, a,, is i ,the EBLUP for a,, as

in Equation (4-2). This is consistent in all four methods. The structure for the prediction

intervals is

(MSEP)t 2,-. (4-63)


The estimate of MSEP and the degrees of freedom for the t-distribution depend on the

method used to calculate the prediction interval. The methods used to determine the MSEP

estimate and the degrees of freedom are summarized in Table 4-4.








Table 4-4. Summary of prediction interval procedures.
GB KR


MSEP


MI(4'D)


FC(0)


M (6) M (')
+2tr A() B (c)] +2tr [A () B ()]


dfG


~J4C J4


dfFC()
F 2


FC(0)

M, ()
+tr[A(0)B()]

tr A(o)B()]
22

dfFc(o)
4 Zvi,


LFr 9FA2ST,-- 2 "fJ UJGB z IV FFC(D) ] MLIV PF FC(O)
var MSEPFC(S) var LMSEPFc(F )
var L I ] I

S[MSEP(.)]
Note: var [MSEP h(.) B(.)h(. ) where h (,) = and B(.) = 1 (*).

Note that all of the methods use a "Satterthwaite-type" method for estimating the degrees

of freedom. The degrees of freedom are calculated using

2 MSEP~
df= L (4-64)
var [MSEP

0 [MSEP(.)
where var[MSEP] h(.)'B(.)h(.) for h(.) = a. and B(.)=I (.). The disparity

between the methods comes in the parameterization, and the estimate of the MSEP (or standard

error) which subsequently impacts the degrees of freedom. We again study results under two

parameterizations, = (2, v) and = (D2, C ). Note that due to the translation invariance of

M1 (.) and tr [A(.)B(.)], the figures and results demonstrated earlier in this chapter pertaining

to M, (o) and M, () + 2tr [A()B(O)] still apply to the standard error estimates used in the

Giesbrecht-Burns, Kenward-Roger, and FC (0) prediction interval procedures.


]









Two of the methods for formulating the prediction interval are used in SAS PROC

MIXED. The first, from the DDFM = SATTERTH option, is similar to the Giesbrecht-Burns

(1985) procedure outlined in Chapter 2; however, in this application we are calculating

prediction limits for the realized value of a random effect rather than a fixed effect, as in that

paper. The method is also developed in Jeske and Harville (1988) and Fai and Cornelius (1996).

We will refer to it as the Giesbrecht-Burns prediction interval method, since it uses the naive

MSEP estimate. The standard error is estimated by the naive MSEP, (M 1 ())/2 where M, (0)

is in Equation (4-3).

The other method available in PROC MIXED through the DDFM = KR option is the

Kenward-Roger (1997) approach. The standard error estimate includes a correction term and a

bias correction term, as in Equation (4-59). Note that the Kenward-Roger prediction interval is

determined under the D parameterization. Recall that under the parameterization, = (o2, a),

the bias correction term estimate is equal to the correction term estimate, thus the MSEP estimate

used in the Kenward-Roger prediction interval is

M, (o) +2tr A (i)B o)] =M () +2tr A(6)B(O)]. It is important to keep the

parameterization in mind throughout, as the MSEP estimate and the degrees of freedom

calculations depend on which parameterization is used. It is demonstrated in Chapter 2 that in a

single-dimension linear combination, the Kenward-Roger method for determining the degrees of

freedom reduces to the Giesbrecht-Burns method. Thus, although the Kenward-Roger and

Giesbrecht-Burns methods for determining the prediction limits use different estimates for the

standard error, the degrees of freedom for the t-distribution will be the same. Kenward-Roger








prediction limits adjust the MSEP estimate for estimating unknown variance components but do

not adjust the t-distribution for the change in standard error.

The other two procedures for creating prediction intervals, FC (0) and FC (0), are based

on the Fai-Cornelius (1996) hypothesis testing methods. For the FC intervals, both the MSEP

estimate and the degrees of freedom of the t-distribution account for the correction term and the

bias correction term. Note that this is actually an extension of the methods developed by Fai and

Cornelius (1996). The bias correction term was not considered as an adjustment to the MSEP

estimate in the methods they proposed. The bias correction term is added here to be consistent

with MSEP estimator used in the Kenward-Roger method. The FC methods differ in the

parameterization. Under the c = (o2, a) parameterization, the MSEP estimate is the same as

in the Kenward-Roger parameterization,

M1 () + 2tr [A () B () =M, () + 2tr A (o) B (). Unlike the Kenward-Roger prediction

interval, the FC (D) procedure adjusts the degrees of freedom of the t-distribution for the

adjustments to the MSEP estimate. The degrees of freedom are still calculated using a

"Satterthwaite" method; however, now the corrected MSEP estimate,

M, () + 2tr [A (0)B( ) is used in the calculation of the degrees of freedom, rather than the

naive MSEP estimate, M1 (D).

The final prediction limit method considered, FC (0), differs from the other three in the

parameterization, 0 = (o2, v). Recall that the bias correction term estimate is not translation

invariant, and thus by changing the parameterization, we also change the MSEP estimate. The









MSEP estimate for the FC(O) procedure is M,(O)+t [Atr ) B ()1 -tr A()B(O). This is


used in both the standard error estimate and the degrees of freedom calculation.

A Monte Carlo simulation study was conducted to compare the four prediction interval

methods. The study is similar to the one conducted in Section 4.1 to evaluate the correction term

estimators. Using the RANNOR function in SAS, 10,000 independent sets of e, and a, were

generated from N(0, '2 =1) and N(0,o,) for several values of O, respectively. The values of

-a correspond to v= {0.05, 0.1, 0.15..., 1.0}. Setting = 0, 10,000 sets of


y, : i = 1,...k, j = 1...6) were created from the model y, = a + e The number of levels of the

random effect is varied to study the impact on the performance of the prediction interval

procedures. The values for k are k = 3, 6, 15, or 30, while n is held constant at n = 6. Simulating

the data in this way allows the realized values generated for the a, 's to be tracked. These are

the realized values of the random effect for which the prediction intervals are constructed. By

tracking the realized values of the random effect, we can determine how often the prediction

intervals contain the true value of the realized random effect, providing the actual (simulated)

coverage rate for a nominal 95% prediction interval.

The prediction limits for a, for the Giesbrecht-Burns and Kenward-Roger procedures are

produced by the CL option in the ESTIMATE statement in PROC MIXED in SAS, with the

DDFM = SATTERTH or DDFM = KR options, respectively. To calculate the FC prediction

limits for either parameterization, the degrees of freedom must be calculated directly and then

used to determine the critical value from the t-distribution. For the FC (0) procedure, the

standard error from the Kenward-Roger procedure is used to calculate the prediction limits. For









the FC (0) procedure, the standard error also has to be calculated directly in addition to the

degrees of freedom because of the change in parameterization. The SAS programming

statements used to perform the simulation study are contained in Appendix A along with the

derivatives needed to compute the degrees of freedom for the FC procedures.

The results of the simulation study are best demonstrated pictorially. Figures 4-24 through

4-27 contain the results of the simulation study for k = 3, 6, 15, and 30, respectively. It is clear

that the Kenward-Roger procedure produces the most inflated coverage rates consistently across

values of k and for the entire parameter space of v. The prediction limits for the Kenward-

Roger procedure will always be larger than the Giesbrecht-Burns procedure since

M, () + 2tr [A (O) B () M, (). Adjusting the degrees of freedom in the FC (D) procedure

for the adjusted MSEP estimate generally increases the degrees of freedom, causing the critical

value from the t-distribution to shrink and thus reducing the width of the prediction interval. The

coverage rates for the FC (D) procedure are generally closer to 95% than the coverage rates for

the Kenward-Roger procedure.

The performance of the FC(0) prediction interval is more dependent on the value for k

and the parameter v than the other intervals. For k = 3, the coverage rate for small values of v

is exceedingly small. As v increases, the performance improves dramatically. This is due to the

inclusion of the bias correction term under the 0 = (C2, v) parameterization. Small values ofk

combined with small values of v produce small MSEP estimates. Recall from Figure 4-23 that

the relative bias of M ()+ tr[A(0)B()] [-tr[A(0)B(6)] approached -50% for small

values of v. This is caused by the significant underestimation of the true bias correction term by









- tr A ()B ( (see Figure 4-14). For larger values of k, the FC(0) prediction interval

procedure compares favorably with the other procedures and consistently improves on the

performance of the Kenward-Roger prediction interval for larger values of v. For larger values

of v recall that M ()+ 2tr [A ()B (O) overestimated the true MSEP more so than


M, ()+tr [A ()B() tr [A ()B )l Thus we would expect a more accurate prediction

interval with the FC(0) method in this range of the parameter space.

To demonstrate the relationship between the prediction intervals more concretely, consider

the hypothetical case where k = 6, n = 6, with data producing variance parameter estimates of

a2 = 1 and (,2 = .5. Table 4-5 contains the values for the standard error, degrees of freedom,

critical t-value and the width of the prediction interval produced by each of the four procedures

in this scenario. The width of the Kenward-Roger prediction interval increases over the width of

the Giesbrecht-Burns interval due to the increase in the standard error estimate. The width of the

FC (0) prediction interval is smaller than both the Giesbrecht-Bums and the Kenward-Roger

intervals because of the increased degrees of freedom and subsequent decrease in the critical t-

value. The width of the FC(0) interval is by far the smallest due to the decrease in the standard

error estimate and the critical t-value.

Finally, to demonstrate the effect of k on the performance of the prediction interval

procedures, Figure 4-28 shows the Kenward-Roger coverage rates for all values of k considered

in the simulation study. This is demonstrative of all the procedures in that as k increases, the

procedures perform better and the coverage rates move closer to the nominal value of 95%.









This study shows the importance of investigating the Fai-Comelius methods for hypothesis

tests and prediction intervals in more complex situations. These preliminary results indicate that

the Fai-Comelius methods often perform better than the Kenward-Roger and the Giesbrecht-

Burns methods used in SAS. The Kenward-Roger procedure produces an overly-conservative

prediction interval because it does not adjust the degrees of freedom for the inflated MSEP

estimate. The study also demonstrates the impact of the parameterization and the lack of

transform invariance of the bias correction term on accuracy of the prediction interval methods.

In more complex covariance structures, the parameterization may be vital to an accurate analysis.

Table 4-5. Effect of prediction interval procedures for k = 6, n = 6, (-2 = 1 and ^ 2 = .5
GB KR FC(0) FC(0)

Standard Error estimate 0.43301 0.46894 0.46894 0.40196

Degrees of Freedom 8.15 8.15 16.39 16.52

t025,df 2.298 2.298 2.116 2.114

Width of CI 1.99 2.16 1.98 1.70


















0



-1
1 ~~~~~~~~ ~ ~- - ----------------------------------------------









-2



-3



-4



-5



-6



-7

0 0.2 0.4 0.6 0.8 1 1.2




k=6 k=9 k=12 k=15 k=18




Figure 4-1. Equation (4-13) as a function of v for several values of k, holding n = 6. Equation (4-14) holds when functions cross
zero line.











0.07


0.06


0.05


0.04


0.03


0.02


0.01


0


tr [A(0)I-1 ()]














S tr A( (0)


0.2 0.4 0.6 0.8


Figure 4-2. Accuracy of approximations tr [A(0)BR (0)] and tr [A(0)11 (0)] for the correction term, CR (0), under REML
variance component estimation.













0.045

0.04

0.035

0.03

0.025

0.02

0.015

0.01

0.005


Accuracy of approximation tr [A(0)BR (0) and estimator E tr [A ()BR) () for the correction term, CR (), under


REML variance component estimation.


E{tr[A(0)BR()]}




tr[A(9)BR ()]


Figure 4-3.














0.06


0.05


0.04


0.03


0.02


0.01


S[A (O)r ()]


0.2 0.4 0.6 0.8


u


Figure 4-4. Accuracy of approximation tr [A(0)-' (0)] and estimator E tr A(o)I-1 (0) for the correction term, CR (0), under

REML variance component estimation.


I I I


E tr A(6) (6))


IC (8,1


























E Atr(A (6) B,


0.2 0.4


Figure 4-5. Comparison of accuracy of estimators for the correction term, CR (0), under REML variance component estimation.


0.06


0.05



0.04



0.03



0.02



0.01



0













0.9 tr [A(0)BA (0)]


0.8 tr[A(0)B (0)]

0.7

0.6

0.5

0.4

0.3

0.2 (

0.1
0 tr[ A(0)I1 (0)
0 a
0 0.2 0.4 0.6 0.8 1 1.2



Figure 4-6. Comparison of accuracy of approximations tr[A(0)I1 (0)], tr A(o)B* (0)], and tr A(0)BA (o)] for the correction

term, CA (0), under ANOVA variance component estimation.













1.6

1.4 E tr [A 0)BA ) )]

1.2

1 tr [A ()B, ()]

0.8

0.6

0.4

0.2



0 0.2 0.4 0.6 0.8 1 1.2



Figure 4-7. Accuracy of approximation tr A(0)BA (0)] and estimator E tr [A()BA ()} for the correction term, CA (0), under

ANOVA variance component estimation.











1.6


1.4 -E tr[ A)B (B)]6

1.2


1
tr [A ()BA (0)]
0.8


0.6


0.4


0.2


0
0 0.2 0.4 0.6 0.8 1 1.2
U


Figure 4-8. Accuracy of approximation tr A(0)B* (0)1 and estimator E tr [A()B* (6)]) for the correction term, CA (0), under

ANOVA variance component estimation.














0.12 TrLAU MUj


0.1 -


0.08


0.06 tr[A (0)-1 ()]


0.04


0.02



0 0.2 0.4 0.6 0.8 1 1.2
U


Figure 4-9. Accuracy of approximation tr [A(0)I1 (0)] and estimator E tr [A () -1 (8)] for the correction term, CA (0), under

ANOVA variance component estimation.












1.6 E tr[A(0)B, ()]}

1.4 E tr [A 0) B ()]

1.2



0.8

0.6

0.4

0.2 E tr[A(e) (e)]


0 0.2 0.4 0.6 0.8 1 1.2


Figure 4-10. Comparison of accuracy of estimators for the correction term, CA (0), under ANOVA variance component estimation.











0.14

Etr[A ())I-1()]}
0.12


0.1 c


0.08


0.06


0.04


0.02

E tr A () B, (
0 j
0 0.2 0.4 0.6 0.8 1 1.2

Figure 4-11. Accuracy of estimators for the correction term under REML, and ANOVA, variance component estimation.

Figure 4-11. Accuracy of estimators for the correction term under REML, 0, and ANOVA, 6, variance component estimation.













500.00%


400.00%




300.00%




200.00%




100.00%




0.00%




-100.00%


Figure 4-12. Relative bias of estimators for the correction term under REML, 0, or ANOVA, 0, variance component estimation.

















0.8


0.6


0.4


0.2 7l

IE[Mi (0)]]
0
0 0.2 0.4 0.6 0.8 1 1.2


Figure 4-13. Bias produced by estimating M1 (0) with M1 ().










0.1

M,(6)-EM:6)]
0 1 _
0.2 0.4 6 0.8 1 12

-0.E1 t[A (1)I-1 ()


-0.2 1r A


-0.3


-0.4


-0.5


-0.6



Figure 4-14. Accuracy of approximation tr [A(0)I1 (0)] and estimator E -tr A )I1 () for the true bias correction term,
M ()- 2, under REML variance component estimation.
M, (0) E[MI ()], under REML variance component estimation.












0.06 -

0.05

0.04










00.2 0.4 0.6 't 1
-0.01 M (0)-[M()]

-0.02

-0.03



Figure 4-15. Accuracy of approximation tr [A(0)I-(0)] and estimator E {tr A A()I )l for the true bias correction term,

M, (0) E[M ()] under REML variance component estimation.











IjEtrFA(0)I 1


E I tr [A()I-1()


U


Figure 4-16. Comparing the accuracy of E tr [A ()I 1 ()] and E tr A (0)I1 ()) for the true bias correction term,

M, (0) E[M ()].


-0.1


-0.2


-0.3


-0.4


-0.5


-0.6


M, () EM()













0.75


0.65


0.55


0.45


0.35


0.25


0.15
ul

0.05


-0.05


-0.15


Figure 4-17. Bias produced by estimating M1 (0) with M1 (6).


|M (e)|











M, (0)-E[M,()]
0.1



0.2 0.4 0.6 0.8 1 12



-0.2 E{ ttr [A(A)1-I()]}


-0.3


-0.4


-0.5


-0.6



Figure 4-18. Accuracy of approximation -tr[A(0)I-1 (0) and estimator E tr A()I-1 () for true bias correction term,
S()- M ( under ANOVA variance component estimation.
M, (0) E[MM (6)], under ANOVA variance component estimation.













0.12 "\" L7\ I \j)J


0.1
IM,()-E[M(0)]I
0.08


0.06

Itr[A(e )I1
0.04
-4-

0.02


0
0 0.2 0.4 0.6 0.8 1 1.2


Figure 4-19. Accuracy of approximation, tr [A (0) 1'(0)] and estimator E tr A () I-1 ())} for true bias correction term,

M, (0)- E [M () under ANOVA variance component estimation.
















01


0


-01


-02


-03


-04


-05


-06




Figure 4-20.
Figure 4-20.


Comparing the accuracy of E -


tr A(6)Ie) )] and E tr A ()I ()] for the true bias correction term,
2LIJL1


M, (0 E[M I (6) ]












































0.2 0.4 0.6 0.8 1
u


Figure 4-21. Comparison of estimators of the MSEP for the EBLUP of a utilizing REML variance component estimation.








































Figure 4-22. Comparison of estimators of the MSEP for the EBLUP of ac utilizing ANOVA variance component estimation.










300.00%



250.00%



200.00%



150.00%



100.00%



50.00%



0.00%



-50.00%



-100.00%


Figure 4-23. Relative bias of possible estimators for MSEP of EBLUP for a
















1



0.98



80.96
0


0.94



0.92



0.9


0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9


Simulated coverage rates of prediction interval procedures for t = a, with k


Figure 4-24.

















1



0.98



0.96 FC (D)



o0.94



0.92



0.9



0.88
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9


Figure 4-25. Simulated coverage rates of prediction interval procedures for t = a, with k
















1



0.99



0.98



0.97
0


U
0.96



0.95



0.94



0.93


0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
1)


Simulated coverage rates of prediction interval procedures for t = a, with k = 15.


Figure 4-26.

















1



0.99



s 0.98
go


o 0.97
U


0.96



0.95



0.94


0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
u

Figure 4-27. Simulated coverage rates of prediction interval procedures for t = a with k = 30.















1.U1



1



0.99



S 0.98



0.97 -



0.96



0.95


0.945 ----------------------- ---------------------------------------------------------------------------------------------------------------------------------------
0.94 1 1 1


Figure 4-28. Simulated coverage rates for the Kenward-Roger prediction interval procedure.









CHAPTER 5
THE EFFECT OF NEGATIVE VARIANCE COMPONENTS ON EBLUP

The effect of negative variance component estimates on EBLUP procedures is a topic that

has been largely neglected (Stroup and Littell 2002). While the use of REML estimates, which

are truncated at zero, would seem to negate this issue, there are still many circumstances when

other variance parameter estimation methods are chosen and negative variance components may

arise. For example, as shown in Stroup and Littell (2002) and in the results on the accuracy of

the correction term and bias correction term estimates in Chapter 4, REML may not always be

the preferred choice for variance component estimation. One strategy for handling negative

variance component estimation is to allow that the component may be a covariance rather than a

variance, thus alleviating the problem of truncating at zero. We begin to look at the impact of

that strategy by again turning to the special case of the balanced one way random effects model.

5.1 Negative Variance Component Estimates and the Balanced One Way Random
Effects Model

To demonstrate the current issues with negative variance component estimation, we first

look at the current SAS 9.1 output for PROC MIXED, both when the variance component

estimates are truncated at zero and when they are not. Using the REML NOBOUND statement

in the procedure statement allows the variance component estimates to take any value on the real

line. Note that because we are in a balanced situation, the ANOVA variance component

estimation would produce the same results as REML NOBOUND. We compare the effect of

truncating using simulated data, generated from a normal distribution with o2 =.01 and cO2 =1.

This distribution produces a high likelihood that the data generated will have larger variation

within a block than between the blocks, hence often producing a negative estimate for Ca2. For

demonstration purposes, the number of levels of the random effect is k = 5 and the number of










observations at each level is n = 6. A dataset that meets the above condition is given in Table 5-

1. It is easy to spot that more variation occurs within the block levels than between them.

Table 5-1. Generated data that produces negative variance component estimate
Replicate 1 2 3 4 5 6
Block 1 2.34754 0.24672 -0.5585 0.60564 -0.9062 1.37627
Block 2 1.01248 -0.0493 0.49583 -1.3952 -0.8644 1.93825
Block 3 0.18826 0.91668 0.68925 0.96593 0.95929 -0.7838
Block 4 -0.4741 1.31312 -1.5036 0.06752 1.2383 -0.7888
Block 5 -1.2176 0.64999 0.32013 1.4081 -0.1774 1.77956

The results from the REML option in the MIXED procedure of SAS are contained in Tables

5-2 and 5-3 and the results from the REML NOBOUND options in the MIXED procedure are

contained in Tables 5-4 and 5-5. Also, for comparison, least square means (LSMEANS)

generated by PROC GLM are given in Table 5-6. These give a starting point from which to

compare the EBLUPS resulting from the REML and REML NOBOUND options. The following

SAS code generates the output in Tables 5-2 and 5-3:

proc mixed data = &sample method = reml ;
class blk trt;
model y = /ddfm = kenwardroger ;
random blk / s ;

Table 5-2. REML variance parameter estimates
Covariance parameter Estimate
Block 0
Residual 1.0643

Table 5-3. Solution for random effects with REML variance parameter estimates
Effect Block Estimate Std Err Pred DF t Value Pr > It
Block 1 0
Block 2 0
Block 3 0
Block 4 0
Block 5 0









The NOBOUND option in the following SAS code is the only difference from the previous

code and generates the SAS output in Tables 5-4 and 5-5:

proc mixed data = &sample method = reml nobound ;
class blk trt;
model y = /ddfm = kenwardroger ;
random blk / s ;

Table 5-4. REML NOBOUND variance parameter estimates
Covariance Parameter Estimate
Blk -0.1410
Residual 1.1810


Table 5-5. Solution for random effects with REML NOBOUND variance parameter estimates
Effect Block Estimate Std Err Pred DF t Value Pr > It
Block 1 -0.4846 0 29 -Infty <.0001
Block 2 0.3461 0 29 Infty <.0001
Block 3 -0.4106 0 29 -Infty <.0001
Block 4 0.8870 0 29 Infty <.0001
Block 5 -0.3379 0 29 -Infty <.0001


Table 5-6. LSMEANS from PROC GLM
Block LSMEAN

1 0.51858487

2 0.18960919
3 0.48927203
4 -0.02458600
5 0.46046652

The results from the REML option, which truncates the variance component estimate for block

effect at zero, are straightforward. The solution statement produces zero estimates for all blocks

and missing values for the standard error, t-values, degrees of freedom, and p-values. The

REML NOBOUND option is not quite so straightforward. When variance component estimates

are positive, we can think of the BLUP as a "shrinkage" predictor. In other words, using the









variance component estimate as a weight, the block means are shrunk toward the overall mean.

In addition, in a balanced model, the BLUPS maintain the rank order of the Ismeans. This can be

easily seen in the structure of the BLUP for the balanced one-way random effects model in

Equation (4-2).

Now, looking at the EBLUP estimates in Table 5-5, the first indication of a problem is

that the standard errors are listed as zero and hence the t-values are listed as infinity. The

calculations for standard error in this case are negative, causing SAS to truncate the standard

error at zero. Comparing the estimates in Table 5-5 to the LSMEANS from PROC GLM in

Table 5-6 shows that the negative variance component estimate causes the EBLUP estimates to

invert their rank order and expands the range of the estimates. The block with the largest

LSMEAN (0.5186) now has the smallest EBLUP (-0.4846) and, likewise, the block with the

smallest LSMEAN (-0.0246) now has the largest EBLUP (.8870). Also, the range of the

LSMEANS is 0.5432 while the range of the EBLUP estimates is 1.3716. We normally expect

the BLUP procedure to "shrink" the block means toward the overall mean according to the

variation in the model; however, in this case, the estimates are inverted about the mean and

expanded away from the overall mean. While the standard errors and t-values listed with the

EBLUPs are a clear red flag, the unexpected results of the EBLUPS require further investigation.

Note also that the standard errors may be positive even when <2a < 0; this would eliminate the

red flag and leave more room for error in the interpretation of the results.

If the data analyst believes that a one-way random effects model is the true model for the

research situation, then the recommendation from an EBLUP standpoint is to truncate at zero

rather than proceed with a negative variance component estimate. Any use of EBLUPS while

allowing the variance component to give a negative estimate is likely to yield misleading results.









The LSMEANS likely yield more reliable estimates than the EBLUPS from the REML

NOBOUND option. Keep in mind that it is not always the best choice to truncate at zero when

variance component estimates are negative if the analyst is interested in fixed effects (Stroup and

Littell 2002).

5.2 Considering the Variance Parameter as a Covariance

One idea that surfaces when faced with negative variance component estimates is

considering the parameter as a covariance rather than a variance, thus allowing the parameter

estimate to take on values over the entire real line. Smith and Murray (1984) consider this option

and the impact on estimation and hypothesis testing of the covariance component. For the

balanced one way random effects model discussed in Chapter 4, the covariance structure would

be modified from


2 .
cov(yj ,) = 0-- +-2, i = iU j = j,
2a, i j',jjf
= 0, i i'

to

cov(y, jy,),) = 0 + 2, i ', j =
= 0, i=i', ij'
=0, i i'

where

0_o 1_0a" : U2 + 0a
0<00 <&0,

One problem overlooked by Smith and Murray (1984) is that in a one way model, if the

variance structure is modified in this way, the model is no longer a typical random effects model.

The motivating example used in Smith and Murray (1984) is weaning rates of twin calves for 40

cows. In the original model, the cow effect is considered random, with the cows regarded as a









random sample from a population of cows, and c,, the variance associated with this population.

When the estimate for o2 is negative, the definition of the parameter is changed from the

population variance of the cows to the covariance between twins born to the same cow. Thus the

population variance for the cows is ignored all together and the negative covariance estimate is

interpreted as reflecting competition for nutrition between the twin calves. The negative value of

the variance parameter estimate is likely reflecting the combination of the positive cow

population variance and the negative correlation between twins born to the same cow. That is,

there is more negative intraclass variability than positive interclass variability resulting in a

negative parameter estimate.

The inversion of the LSMEANs that we see with the EBLUPs in PROC MIXED when

the variance component estimate is negative occurs because the model is effectively changed to

this Smith and Murray (1984) model. The EBLUP calculated by PROC MIXED for a in this

case is



a +n0o-


giving a negative coefficient when --- < 6 < 0. Furthermore, may be sufficiently large
n

nO
so that n < -1. Thus rather than "shrinking" the means, as we expect EBLUP procedures
2 +na

to do, the EBLUP inverts and expands the range of the means. Clearly this is an unacceptable

result. The question remains: what is an analyst to do when faced with this result. One possible

solution is derived in the next section.









5.3 BLUP derivation for Random Effects Model with Correlated Errors

We now present a model that would account for both the population variance of the

random effect and the correlation of replicates within a level of the random effect. Consider a

model with a random effect, as well as correlated errors. The model and variance structure in

Equation (4-1) would be modified to:

ye, = +a + e,i =l,..., k, j=l ,...,n
a ~N( 0,o

where e are normally distributed with
E(e) = 0

-2 + 0, i=i',j=j' (
(5-1)
cov(e,,), i = i', j=I + j
0, i # i'


In mixed models notation, this equates to

y = Xp + Zu + e

where
x =1', p =
Z=Ik 01

u=[ala2 ...ak N(0,G)

where
G = OIk

and

e=[elel2...ek '~ N(0,R)

where
R=Ik ,(o2I + J)

so that










V=ZGZ'+ R =Ik {2n ( + )J



From general matrix theory (e.g., Searle 1982), we have


v --iC 2 \ + 08
V -1= 1 2 'n 2 (C2 + \"" r
{2 2 +nj


which gives the BLUP for u as




2
0-a
ii= GZV-1 (y- -)= Ik (



or the BLUP for ac as


a2
anu, (3 y) (5-2)



Note that for the model structure given in Equation (5-1) to be valid, V must be positive definite.

Therefore, the parameter space for which the model is valid is
0-2
2
-2>0, a2>0, a2 + < -
n

In this parameter space the coefficient for the BLUP, given in Equation (5-2), is non-negative.

Thus the order of the LSMEANS for the blocks is preserved in the BLUP. One key

consideration for this model is the problem of overparameterization. The parameters C02 and 08

are confounded and cannot be simultaneously estimated. One solution is to use an estimate for

0-2 from previous experience and consider the parameter as known. This leaves only 0 and -2

as unknown and the EBLUP analysis can be performed where the unknown variance parameters

are estimated by an unbounded method, such as REML NOBOUND in PROC MIXED. There










are many situations, such as agriculture or education, where using a previous estimate as the

parameter is realistic. Data contained in Table 5-5 are generated from a distribution as


described in Equation (5-1) with o-2 =.9, 0O


.13 and o-C =.07.


Table 5-7. Data generated from distribution in Model (5-1)
Replicate 1 2 3 4
Block 1 1.77754 -0.72357 -0.79317 -0.79674
Block 2 -0.62131 -1.15758 -1.21619 -0.24776
Block 3 -0.3412 -0.7861 0.41517 0.75414
Block 4 -0.05152 -0.9336 0.31271 -0.3097
Block 5 0.75952 -0.27854 0.28914 1.03234


5
1.31455
0.5867
1.34176
1.04458
-0.88724


6
-0.98605
0.12472
0.75806
1.07845
-2.24931


The appropriate SAS statements to analyze this data in the manner described in Equation (5-2)

are

data gmat ;
input row col value;
cards;
1 1 .07
2 2 .07
3 3 .07
4 4 .07
5 5 .07


proc mixed data = &sample method = reml ;
class blk trt
by rn ;
model y = /ddfm = kenwardroger ;
random blk / gdata=gmat g s ;
repeated trt / sub = blk type=cs ;
run ;


The gdata option after the random statement provides the known G matrix which in this case is

G =.07I,


where In is an n-dimensional identity matrix.


The output from this analysis of the data in Table 5-7 are contained in Tables 5-8 and 5-9.

The EBLUPS for the levels of the random effect are generated by the solutions or s option









in the random statement of the SAS code, and are given in Table 5-9 under "Estimate."

Comparison of these values to the LSMEANS generated by GLM in Table 5-10 show that the

function worked as expected, maintaining the rank order of the means and "shrinking" the

predictors toward the overall mean.

Table 5-8. Covariance parameter estimates under Model (5-1)
Covariance Parameter Subject Estimate
Variance Block 0.9497
CS Block -0.1311


Table 5-9. Solution for random effects under Model (5-1)
Effect Block Estimate Std Err Pred DF t Value Pr > Itl
Block 1 -0.00593 0.2646 2.16 -0.02 0.9840
Block 2 -0.2849 0.2646 2.16 -1.08 0.3868
Block 3 0.2761 0.2646 2.16 1.04 0.3991
Block 4 0.1560 0.2646 2.16 0.59 0.6113
Block 5 -0.1412 0.2646 2.16 -0.53 0.6433


Table 5-10. LSMEANS from PROC GLM
Block y LSMEAN
1 -0.03457327
2 -0.42190439
3 0.35697247
4 0.19015370
5 -0.22234738


Note that the DDFM = KR method was used here which uses an adjusted standard error. If the

DDFM = SATTERTH option is used, the estimates and degrees of freedom remain the same, but

the standard error is reduced to .1647 which would impact the prediction limits as shown in

Section 4.5. Given the conclusion in Chapter 4 regarding the inaccuracy of the adjustment under









certain circumstances, the accuracy of either standard error estimate and their effect on

prediction intervals for the realized values of the random effect needs further study. This

remains as future research.

For comparison, Tables 5-11 and 5-12 contain the results of the analysis of the data in

Table 5-7 if the correlation were ignored and the model were run as a random effects model with

the REML NOBOUND option. We see that the variance parameter estimate is negative and the

EBLUPs are inverted and expanded as shown in Section 5.1.

Table 5-11. Variance parameter estimates under random effects model
Covariance Parameter Estimate
Block -0.06110
Residual 0.9497


Table 5-12. Solution for random effects under random effects model
Effect Block Estimate Std Err Pred DF t Value Pr > It
Block 1 0.005176 0 29 Infty <.0001
Block 2 0.2487 0 29 Infty <.0001
Block 3 -0.2410 0 29 -Infty <.0001
Block 4 -0.1361 0 29 -Infty <.0001
Block 5 0.1232 0 29 Infty <.0001


One issue that occurs with analyzing data in this way is that it is prone to convergence

failure. Of 10,000 simulated data sets from the distribution described in Equation (5-1) and

analyzed with this procedure, 21.5% failed to converge when wither REML or REML

NOBOUND procedures were used. Even with this complication, this parameterization remains

one of the only viable options if predictors of the realized values of the random effects are

desired in a case with negative variance component estimation.









CHAPTER 6
CONCLUSION AND FUTURE WORK

The primary goal of this dissertation was to examine the impact of estimating variance

components on the estimators of the MSEP for the EBLUP. This was accomplished by

investigating properties of generalized components of the MSEP estimator, examining the

behavior of various procedures in the balanced one-way random effects model, and determining

the impact of negative variance components on EBLUP procedures.

Following the work by Kackar and Haville (1984), Kenward and Roger (1997), Jeske and

Harville (1988), and Fai and Cornelius (1996), estimators for the correction term and the bias

correction term were generalized to accommodate multiple dimension linear combinations of

fixed and random effects. Properties of these generalized estimators were investigated. We

showed that while for linear covariance structures the bias correction term estimator,

BIAS( (0), continues to reduce to the correction term estimator, C" (0), which is non-

negative definite, BIASR (0), in general, is indefinite. The indefiniteness of BIASK (0)

causes the estimator, M1 (0)+ BIASR (0), of the naive MSEP, M1 (0), to be indefinite. Future

work includes the investigation of definiteness properties of the MSEP estimator

M1 (0) + C (0) + BIAS' (0), which includes both the correction term and the bias correction

term. The magnitude of M1 (0) +C' (0) may outweigh that of BIAS" (0) to produce a

positive definite estimator. We also demonstrated that BIAS (0) is not transform invariant,

and that the choice of parameterization can have a significant impact on the bias correction term,

the MSEP estimate, and the coverage rates of prediction intervals.

The balanced one-way random effects model provided a simple setting in which the

various components of MSEP estimators could be thoroughly explored. Closed form









expressions allowed for direct examination of the correction term and bias correction term

estimators. Through this example, the impact of variance component estimation techniques, the

values of the variance components, and the parameterization was demonstrated. We determined

that the MSEP estimator may significantly over- or under-estimate the true MSEP depending on

these factors. We showed that truncating variance parameter estimates under REML estimation

can cause significant (long-run) overestimation of both the correction term and the bias

correction term, over most of the parameter space. While these procedures have been considered

suspect near the boundary of the parameter space, this study showed that the accuracy of the

methods is questionable over a much greater range of the parameter space.

We also discovered that the choice of how to measure the MSE of the variance parameter

estimates can significantly impact the accuracy of the correction term estimator. In REML

situations, using the exact MSE of the variance parameter estimates in the correction term

estimator resulted in a more accurate correction term estimator than using the asymptotic MSE

over two-thirds of the parameter space. However, if variance parameter estimates are left

untruncated, as in ANOVA variance component estimates, the asymptotic MSE provided a more

accurate correction term estimator over the entire parameter space.

The importance of the parameterization of the model is apparent in the bias correction term

estimator since it is not transform invariant. Under a linear parameterization, BIAS (0)

slightly overestimates the true bias correction term; while under a nonlinear parameterization

BIAS( (0) significantly underestimates the true bias correction term for a large portion of the

parameter space when using REML variance component estimates. Under ANOVA variance

parameter estimation, the linear parameterization provides a significantly more accurate bias

correction term estimator. Even with this simple model, we see how the interaction of a few









factors complicates the results. The question of which MSEP estimator is best depends on the

parameterization, the true value of the variance parameter, and the choice of variance parameter

estimation method. Each of these factors deserves investigation in more complex models.

These same factors impact the coverage rates of prediction intervals for a realized value of

the random effect. We introduced a modification of the Fai-Cornelius methods for

approximating the degrees of freedom that includes a bias correction term in the MSEP

estimator. The results of a Monte Carlo simulation study showed that the Fai-Cornelius based

methods compare favorably with the Giesbrecht-Burns and Kenward-Roger methods currently

available in SAS. The simulation study also demonstrated a shortcoming of the Kenward-

Roger method in that the degrees of freedom are not adjusted for the increased MSEP estimate

for a single dimension linear combination. In fact, of the four prediction interval procedures

included in the simulation study, the Kenward-Roger method consistently performed the poorest,

producing the most inflated coverage rates. Because the modified Fai-Cornelius method adjusts

both the MSEP and the distribution for the variance component estimates, this method deserves

more attention and may significantly improve the coverage rates or Type I error for hypothesis

testing in many situations. Future work includes further investigation of these methods in more

complex settings.

The impact of negative variance component estimates on the EBLUP and the MSEP

estimators was studied. We show that the EBLUP is no longer a reliable or valid estimator of the

random effect when negative variance component estimates are present. The proposed solution

by Smith and Murray (1984) of regarding the variance parameter as a covariance is shown to be

invalid for EBLUP procedures. An alternative solution of a random effects model with

correlated errors was presented. Further study of this proposed method is indicated. Future









work also includes the investigation of the impact of negative variance parameter estimates on

fixed effects estimators, their standard errors, and distribution estimation techniques.








APPENDIX
DETAILS FOR MONTE CARLO SIMULATION STUDY ON PREDICTION INTERVALS

The calculations necessary for the FC (0) and FC (0) prediction intervals are detailed

below, followed by an example of the SAS code used for the Monte Carlo simulation study to

determine coverage rates for the four prediction interval procedures in Section 4.5.

Recall from Equation (4-64) that the inverse of the observed information matrix,

B() = I' (*) and the vectors of derivatives of the MSEP, h(.), are necessary for the

"Satterthwaite"-method of calculating degrees of freedom. First we look at the FC (D) method.

The inverse of the observed information matrix for the D = (2, 2) is

B (0) = I1- (0)

2(2 )2 2( 2 )2
k(n-1) kn(n- 1) (A-l)

-2(C2) 22 2 ( 2
kn(n-l) n2 k-1 k(n-1)



L82 2a,

where, due to translation invariance, M1 (0) is as in Equation (4-3) and tr[A ()B(()] is as

in Equation (4-15). Thus,

a8M, () + 2tr [A(D)B(()]} (1- u)[ +(k -1) ]
O82 nkv
2(kn 1)a2[2 +2na2 (A-2)
k2 2 2
k2n(n-l)(c2 +na)2

and








a{M, (4)+2tr[A(4V)B(04)]} a'(2nu- +ka2) +n 2(a)2
0: k(oZ+. o )2
S (A-3)
2n(kn 1)(C2)
k n(n-( 1)2 n)2

The degrees of freedom for the FC (0) method are calculated as

2 [M () + 2tr [A () B (Q)]]2
dc(I) = var [M, ()+ 2tr [A(() B()

where

varM [M (4)+2tr A ()B ()1] h ) ()h (4))

with the elements of h (D) from Equations (A-2) and (A-3) and T' (D) from Equation (A-l).

For the FC (0) method, we turn to the 0 = (o, v) parameterization. The inverse of the

observed information matrix is now

B(0) = -1 (0)
2(2)2 2(a2)v
k(n-1) k(n-1) (A-4)
2(C )v 2vt(kn-1)
k(n-1) k(k-1)(n-1)

We use the MSEP estimator utilizing both the correction term and the bias correction term

elements to be consistent with the Kenward-Roger and the FC (D) methods. Because the bias

correction term estimator is not transform invariant, the MSEP estimate is different than the one

used in the D parameterization. The MSEP estimator in the 0 = (2, v) parameterization is

given in Equation (4-60). Thus









S (0)+ tr [A(0)B(0)]-ltr [A(0) B(0)]1
h(O) = a-2
00


( I1
C j (0) +tr [A(0) B(0)]- r [A(0) B(0)]
2


where


S{M, (0) +tr[A(0)B(0)]
8a2


trA()B(0) (1- v k -1
nkv
2v(kn-1) 4[k(n-1)+(k-1)v2
k2n(n -1) nk2 (n -1)(k 1)2


{ M, (0)+ trA(0) B(0)] trA(0) B(0)] 2 [E(1- k) v2-
gv nkv2 (A-6)
2o(kn-)_ 42[ 2(k-1)2 -k(n-l)+(k-l)22]
Sk2n(n-1) nk2(n-1)(k-1)v2

The degrees of freedom for the FC(0) method are calculated as

2
2f() [M () + tr [A r )B( ) tr A()B )

var MI (6) + tr [A ()B()]- tr [A ()B(


where


var[M, ()+tr[A(0)B() tr A)B(0) h)B(0)h ()


with the elements of h () from Equations (A-5) and (A-6), and IV1 () from Equation (A-4).


(A-5)











An example of the SAS code used for the Monte Carlo simulation study to determine


coverage rates for the four prediction interval procedures, utilizing the above derivations for the


FC(D) and FC(O) methods, follows.


/*Sample program for Monte Carlo simulation study of prediction intervals*/



/*Generating Data for random effect and error from normal distributions and
creating y ij, observations*/

/*k=3,n=6,sigma(a) squared=10*/

data normal ;

do mc = 1 to 10000;
do sigma a = (10)**.5 ;
do i = 1 to 3 ;
stda = rannor(17)
do j = 1 to 6 ;
e ij = rannor (18)
a = sigma a*stda
y ij = a +eij ;
output ;
end
end ;
end
end ;

run

/*Creates Satterthwaite prediction limits on al*/

proc mixed data = normal method=REML noclprint noinfo noitprint;
class i ;
model y ij = / ddfm=Satterth ;
random i ;
estimate 'effect 1' | i 1 0 0 /cl;
by mc ;
ods listing exclude estimates
ods listing exclude fitstatistics ;
ods listing exclude covparms
ods output estimates=estl ;
run ;

/*Creates Kenward-Roger prediction limits on al*/
/*Outputs Kenward-Roger Standard Error for use in FC(() prediction
intervals*/
/*Outputs Covariance Parameter Estimates to use in FC(G) calculations*/

proc mixed data = normal method=REML noclprint noinfo noitprint;
class i ;
model y ij = / ddfm=kenwardroger ;












random i ;
estimate 'effect 1' | i 1 0 0 /cl;
by mc ;
ods listing exclude estimates ;
ods listing exclude fitstatistics
ods listing exclude covparms
ods output estimates=est2 ;
ods output covparms=cp ;
run ;

data est ;
set estl ;
drop Label
rename tValue=Satt tval ;
rename DF=Satt df ;
rename Probt = Satt pvl ;
rename StdErr=seSATT ;
rename upper = satt uppercl ;
rename lower = satt lowercl ;
run ;

data est2 ;
set est2
drop Label estimate;
rename StdErr=seKR ;
rename tValue=KR tval
rename DF=KR df ;
rename Probt = KR pval ;
rename upper = KR uppercl ;
rename lower = KR lowercl ;
run ;

proc sort data=cp ;
by covparm ;

data siga resid out
set cp ;
if covparm='i' then output siga ;
else if covparm='Residual' then output resid ;
run ;

data siga ;
set siga ;
siga = estimate ;
drop covparm estimate ;
run ;

data resid ;
set resid
resid = estimate ;
drop covparm estimate
run ;

/*Isolates al for each data set*/

data a ;
set normal ;











where i = 1 and j = 1 ;
run ;

data est ;
merge estl est2 a siga resid;
by mc ;
k=3;
n=6;
upshat = resid/(resid + n*siga)
ml = resid*(l-upshat)*(1+((k-l)*upshat))/(n*k*upshat)

/*elements of inverse information matrix for <*/

bll = 2*resid*resid/(k*(n-1)) ;
bsigsiga = -2*resid*resid/(k*n*(n-1)) ;
bsiga = 2*resid*resid*(i/(n*n))*((1/(upshat*upshat*(k-1)))+(1/(k*(n-1))))

/*derivatives of Ml, ( parameterization*/

dl = n*(k-l)*siga*siga/(k*(resid+n*siga)*(resid+n*siga)) ;
d2 = (resid*(2*n*siga + k*resid) +
n*n*siga*siga)/(k*(resid+n*siga)*(resid+n*siga)) ;

/*derivatives of 2*trab, ( parameterization*/

dl FC2 = dl + 2*((2*(k*n-l)*resid*resid + 4*(k*n-
l)*n*resid*siga)/(k*k*n*(n-l)*(resid+n*siga)*(resid+n*siga))) ;
d2 FC2 = d2 2*(2*n*(k*n-l)*upshat*upshat/(k*k*n*(n-1)))

/*dfFC2 = FC(() degrees of freedom*/

dbdFC2 = dl FC2*(dl FC2*bll +d2 FC2*bsigsiga) + d2 FC2*(dl FC2*bsigsiga
+d2 FC2*bsiga) ;
dfFC2 = 2*(seKR**2)*(seKR**2)/dbdFC2 ;

/*critical t-value for alpha=.05, FC(() degrees of freedom*/

tFC025 = quantile('T',.975,dfFC2);

/*FC(() prediction limits*/

FC phi uppercl= estimate +tFC025*seKR ;
FC Phi lowercl = estimate tFC025*seKR ;

/*MSEP estimate for 8 parameterization*/

msekrbias = ml + 2*resid*upshat*(k*n-l)/(k*k*n*(n-1))
2*resid*((k*(n-1))-((k-l)*(k-1)*upshat*upshat))/(n*k*k*(n-
l)*(k-l)*upshat) ;
sqmsekrbias = msekrbias**(1/2) ;

/*elements of inverse information matrix for 8 (along with bll)*/

b12 = 2*resid*upshat/(k*(n-1)) ;
b22 = 2*upshat*upshat*(k*n 1)/(k*(n-1)*(k-1))

/*derivatives of Ml, 8 parameterization*/












dlsatt = (1-upshat)*(l+(k-l)*upshat)/(n*k*upshat) ;
d2satt = resid*((1-k)*upshat*upshat -1)/(n*k*upshat*upshat)

/*derivative of trab + bias term, 8 parameterization*/

dl FC3 = dlsatt + 2*upshat*(k*n-l)/(k*k*n*(n-1)) (2*k*(n-1) 2*(k-1)*(k
1)*upshat*upshat)/(n*k*k*(n-1)*(k-1)*upshat) ;
d2 FC3 = d2satt + 2*resid*(k*n-l)/(k*k*n*(n-1))
+ 2*resid*(2*(k-1)*(k-1)*upshat*upshat + k*(n-1) (k-l)*(k-
1)*upshat*upshat)/(n*k*k*(n-1)*(k-1)*upshat*upshat) ;

/*dfFC3 = FC(G) degrees of freedom*/

dbdFC3 = dl FC3*(dl FC3*bll +d2 FC3*bl2) + d2 FC3*(dl FC3*bl2 +d2 FC3*b22)

dfFC3 = 2*msekrbias*msekrbias/dbdFC3 ;


/*critical t-value, alpha


.05, FC(G) degrees of freedom*/


tFC025bias = quantile('T',.975,dfFC3);

/*FC(8) prediction limits*/


FC theta uppercl
FC theta lowercl


estimate +tFC025bias*sqmsekrbias ;
estimate tFC025bias*sqmsekrbias ;


/*Flags each prediction interval containing al*/


if satt uppercl = or satt lowercl = then satt cover = ;
else if satt uppercl > a and satt lowercl < a then satt cover = 1
else satt cover=0 ;

if KR uppercl = or KR lowercl = then KR cover = ;
else if KR uppercl > a and KR lowercl < a then KR cover = 1 ;
else KR cover=0 ;

if FC phi uppercl = or FC phi lowercl = then FC phi cover = ;
else if FC phi uppercl > a and FC phi lowercl < a then FC phi cover = 1 ;
else FC phi cover=0 ;

if FC theta uppercl = or FC theta lowercl = then FC theta cover = ;
else if FC theta uppercl > a and FC theta lowercl < a then FC theta cover


else FC theta cover
run ;


/*Calculates True Coverage Rates for Each Prediction Interval Method*/

proc means data = est n mean stderr 1clm uclm;
var satt cover KR cover FC phi cover FC theta cover ;
title 'Sigmasq a=10 k=3 n=6' ;
title 'Simulated true coverage rates' ;
title 'BLUP al'
run ;









LIST OF REFERENCES


Booth, J. G. and Hobert, J. P. (1998), "Standard Errors of Prediction in Generalized Linear
Mixed Models," Journal of the American Statistical Association, 93, 262-272.

Corbeil, R. R. and Searle, S. R. (1976), "Restricted Maximum Likelihood Estimation of Variance
Components in the Mixed Model," Technometrics, 18, 31-38.

Fai, H. T. and Cornelius, P. L. (1996), "Approximate F-tests for Multiple Degree of Freedom
Hypotheses in Generalized Least Squares Analyses of Unbalanced Split-Plot Experiments,"
Journal of Statistical Computation and Simulation, 54, 363-378.

Ferguson, T. S. (1967), Mathematical Statistics, New York: Academic Press.

Giesbrecht, F. G. and Burns, J. C. (1985), "Two-Stage Analysis Based on a Mixed Model:
Large-Sample Asymptotic Theory and Small Sample Simulation Results," Biometrics, 41, 477-
486.

Harville, D. A. (1976), "Extension of the Gauss-Markov Theorem to Include the Estimation of
Random Effects," The Annals of Statistics, 4, 384-395.

Harville, D. A. (1977), "Maximum Likelihood Approaches to Variance Component Estimation
and to Related Problems," Journal of the American Statistical Association, 72, 320-338.

Harville, D. A. (1985), "Decomposition of Prediction error," Journal of the American Statistical
Association, 80, 132-138.

Harville, D. A. (1990), "BLUP and Beyond," In Advances in Statistical Methodsfor Genetic
Improvement ofLivestock, eds. D. Gianola and K. Hammond, 239-276.

Harville, D. A. and Carriquiry, A. L. (1992), "Classical and Bayesian Prediction as Applied to an
Unbalanced Mixed Linear Model," Biometrics, 48 ,987-1003.

Harville, D. A. and Jeske, D. R. (1992), "Mean Squared Error of Estimation or Prediction Under
General Linear Model," Journal of the American Statistical Association, 87, 724-731.

Henderson, C. R. (1950), "Estimation of Genetic Parameters," Annals ofMathematical Statistics,
21, 309-310.

Hulting, F. L. and Harville, D. A. (1991), "Some Bayesian and Non-Bayesian Procedures for the
Analysis of Comparative Experiments and for Small-Area Estimation: Computational Aspects,
Frequentist Properties, and Relationships," Journal of the American Statistical Association, 86,
557-568.

Jeske, D. R. and Harville, D. A. (1988), "Prediction-Interval Procedures and (Fixed-Effects)
Confidence-Interval Procedures for Mixed Linear Models," Communications in Statistics A.
Theory and Methods, 17, 1053-1087.










Johnson, Norman L. and Kotz, Samuel (1970), Distributions in Statistics: Continuous Univariate
Distributions (Vol. 2), New York: John Wiley.

Kackar, R. N. and Harville, D. A. (1981), "Unbiasedness of Two-Stage Estimation and
Prediction Procedures for Mixed Linear Models," Communications in Statistics A. Theory and
Methods, 10, 1249-1261.

Kackar, R. N. and Harville, D. A. (1984), "Approximations for Standard Errors of Estimators of
Fixed and Random Effects in Mixed Linear Models," Journal of the American Statistical
Association, 79, 853-862.

Kenward, M. G., and Roger, J. H. (1997), "Small Sample Inference for Fixed Effects from
Restricted Maximum Likelihood," Biometrics, 53, 983-997.

Khatri, C. G. and Shah, K. R. (1981), "On the Unbiased Estimation of Fixed Effects in a Mixed
Model for Growth Curves," Communications in Statistics A. Theory and Methods, 10, 401-406.

Khuri, A. I. (1993), Advanced Calculus in ith Applications in Statistics, New York: John Wiley &
Sons.

Khuri, A. I. and Sahai, H. (1985), "Variance Component Analysis: A Selective Literature
Survey," International Statistical Review, 53, 279-300.

Lahiri, P. and Rao, J. N. K. (1995), "Robust Estimation of Mean Squared Error of Small Area
Estimators," Journal of the American Statistical Association, 90, 758-766.

Littell, R. C., Milliken, G. A., Stroup, W. W., Wolfinger, R. D., and Schabenberger, O. (1996),
SAS for Mixed Models (2nd ed.), Cary, NC, SAS Institute Inc.

Peixoto, J. L., and Harville, D. A. (1985), "Comparison of Alternative Predictors Under the
Balanced One-Way Random Model," Journal of the American Statistical Association, 81, 431-
436.

Prasad, N. G. N. and Rao, J. N. K. (1990), "The Estimation of Mean Square Error of Small Area
Estimators," Journal of the American Statistical Association, 85, 163-171.

Puntanen, S. and Styan, G. P. H. (1989), "The Equality of the Ordinary Least Squares Estimator
and the Best Linear Unbiased Estimator," The American Statistician, 3, 153-164.

Robinson, D. L. (1987), "Estimation and Use of Variance Components," The Statistician, 36, 3-
14.

Robinson, G. K. (1991), "That BLUP is a Good Thing: The Estimation of Random Effects,"
Statistical Science, 6, 15-51.









SAS Institute, Inc. (2003), SAS/STAT User 's Guide in SAS OnlineDoc, Version 9.1.3, Cary,
NC.

Satterthwaite, F. E. (1941), "Synthesis of Variance," Psychometrika, 6, 309-316.

Satterthwaite, F. E. (1946), "An Approximate Distribution of Estimates of Variance
Components," Biometrics Bulletin, 2, 110-114.

Searle, S. R. (1971), Linear Models, New York: John Wiley & Sons.

Searle, S. R. (1987), Linear Models for Unbalanced Data, New York: John Wiley & Sons.

Searle, S. R., Casella, G. and McCulloch, C. E. (1992), Variance Components, New York: John
Wiley & Sons.

Schaalje, G.B., McBride, J.J. and Fellingham, G.W. (2002), "Adequacy of approximations to
distributions of test statistics in complex mixed linear models," Journal ofAgricultural,
Biological and Environmental Statistics, 7, 512-524.

Smith, David W., and Murray, Leigh W. (1984), "An Alternative to Eisenhart's Model II and
Mixed Model in the Case of Negative Variance Estimates," Journal of the American Statistical
Association, 79, 145-151.

Stroup, W. and Littell, R. C. (2002), "Impact of Variance Component Estimates on Fixed Effect
Inference in Unbalanced Mixed Models," Proceedings of the 14 Annual Kansas State
University Conference in Applied Statistics in Agriculture, 32-48.

Tuchscherer A., Herrendorfer, G., and Tuchscherer, M. (1998), "Evaluation of the Best Linear
Unbiased Prediction in Mixed Linear Models with Estimated Variance Components by Means of
MSE of Prediction and the Genetic Selection Differential," Biometrical Journal, 40, 949-962.

Wolfe, D. A. (1973), "Some General Results About Uncorrelated Statistics," Journal of the
American Statistical Association, 68, 1013-1018.









BIOGRAPHICAL SKETCH

Jamie McClave Baldwin is a native of Gainesville, Florida. She is the daughter of Dr.

James T. McClave and Mary Jay McClave, both of whom hold graduate degrees from the

University of Florida. She is married to Ian Baldwin and mother to Caryss and Zach Baldwin.

She will deliver their third child in January, 2008. She is a graduate of Gainesville High School,

Vanderbilt University (B. A., mathematics and economics, magna cum laude, 1997) in

Nashville, Tennessee, and the University of Florida (M. Stat., statistics, 1999; Ph.D., December

2007).

Jamie began her career in statistics while still in high school, as a data entry assistant at

Info Tech, Inc. She has held several internships with Info Tech, Inc., allowing her to gain

understanding of each step of the statistical analysis process. While at U. F., she worked as a

graduate teaching assistant in the Department of Statistics and as a graduate research assistant in

IFAS, Statistics. After graduation, Jamie will remain in Gainesville as a Statistical Consultant at

Info Tech, Inc.





PAGE 1

1 EVALUATING ADJUSTMENTS TO THE MEAN SQUARED ERROR DUE TO ESTIMATING VARIANCE PARAMETER S IN LINEAR MIXED MODELS By JAMIE MCCLAVE BALDWIN A DISSERTATION PRESENTED TO THE GRADUATE SCHOOL OF THE UNIVERSITY OF FLOR IDA IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY UNIVERSITY OF FLORIDA 2007

PAGE 2

2 2007 Jamie McClave Baldwin

PAGE 3

3 To my husband, Ian, and to my children, Caryss, Zachary, and Baby Baldwin

PAGE 4

4 ACKNOWLEDGMENTS First and m ost importantly, I thank my husband, Ian Baldwin, for providing endless patience over the last 10 years whil e I have pursued this degree. I would not have been able to endure this process without his unwavering supp ort and constant belief in me, even when I couldnt believe in myself. I thank him for ma intaining both the house and my sanity when times were hard and for making me smile when days were too long to manage. I express my eternal love and gratitude to my children, Cary ss, Zach, and Baby Number 3, for their infinite supply of laughter and hugs. They have insp ired me to push beyond a ny limitation and live life to the fullest. They are the lights of my life a nd the most beautiful souls I have ever encountered and I give thanks for them everyday. My family is a daily demonstrati on of Gods Grace; there is nothing I could ever do to deserve their unconditional love and support. I thank my parents, Jim and Mary Jay McClav e, for their hard wor k, love, patience, and constant support. To my father, I am so grat eful to have such a wonderful role model, both professionally and personally. I am forever indebted to him for continual editing and mathematical advice. I am so fortunate to ha ve the opportunity to work with the smartest and bravest man that I know, not to mention the best cons ultant in the industry. To my mom, I am so thankful for her wise parenting advice, too many hours of childcare to even count, and girl time when it was most needed. I literally could not ha ve finished this monstrous endeavor without her emergency child care services. I also thank them for all of the generosity they have shown me and my family in so many ways. I would also like to thank my brother, Will McClave, his wife, Amber, and my awesome nephews, Jake and Luke. They have been a cons tant support for my family and I look forward to the next chapter of our lives together. I am so blessed to have such a wonderful, supportive family that lives so close by. It is a rarity in this world which I truly appreciate.

PAGE 5

5 To the best friends anyone could ask for, I am thankful for their friendship, love, and encouragement through this entire process. I th ank Holly Holly for always helping me see the bright side, laughing at my stories, listening to my endless complaints, and allowing me to learn so much about motherhood and balance from her w onderful example. I thank Heather Bristol for being our pioneer through the statistics program. She has shown me what true bravery is and taught me how to deal with the most unexpected of situations with grac e and strength. I thank Dr. Patches Johnson for encouraging my continua tion when I thought I would go no further, and for being my mentor in so many ways. I tha nk her for being a fantas tic and fun traveling companion and for her contagious laugh. I thank Meghan Medlock Abraham, my college roommate, for a friendship that picks up right wh ere it was left off no matter how much time has passed and for having a mouth big enough to eat her own fist. I thank Brooke Bloomberg, my oldest and dearest friend, for he r sisterly bond and for beating me to the finish line; it was all the motivation I needed! It is amazing that we have both accomplished our childhood ambitions. We rock!! I am grateful for other friends that have he lped me along the way w ith their support and laughter. My sincerest appreciation goes to Andy and Tracy Bachmann, Eric and Ann Bloomberg Beshore, Nicole Provost and Kathy Carroll, Debbie Hagan, Lisa Jamba, Ron and Cindy Marks, Larry and Sandy Reimer, and Yvette and Scott Silvey. I co nsider each one of them family and I am so lucky to have them in my life. Finally, I thank Dr. Ramon Littell for his patience and guidance during this seemingly endless project. Without his loyal support and en during belief, this disse rtation could not have been completed. This degree would not have be en possible without his wealth of knowledge, generous spirit, and encouragement.

PAGE 6

6 TABLE OF CONTENTS page ACKNOWLEDGMENTS...............................................................................................................4 LIST OF TABLES................................................................................................................. ..........8 LIST OF FIGURES.........................................................................................................................9 ABSTRACT...................................................................................................................................12 CHAP TER 1 INTRODUCTION AND BACKGROUND........................................................................... 14 1.1 Introduction to Mixed Models ......................................................................................14 1.2 The Linear Mixed Model .............................................................................................. 15 1.3 Best Linear Unbiased Predictors (BL UPs) When is Known: Mixed Model Equations.......................................................................................................................15 1.4 Empirical Best Linear Unbiased Estimation (EBLUP)................................................. 16 1.5 Two-Stage Estim ation and Unbiasedness..................................................................... 17 2 LITERATURE REVIEW.......................................................................................................19 2.1 Precision of EBLUPs .................................................................................................... 19 2.1.1 Mean Squared Error of Pred iction (MSEP) of EBLUPs ...................................19 2.1.2 Kackar-Harville Approxim ation to MSEP........................................................ 22 2.1.3 Estim ating C ...............................................................................................24 2.1.4 Estim ating 1M .............................................................................................25 2.1.5 Estim ators of M Based on the Kackar-Harville Approximation............... 26 2.2 Hypothesis Testing for Single Di m ension Linear Combinations................................. 27 2.3 Hypothesis Testing M ethods for Multiple Dimension Linear Combinations............... 29 2.3.1 Kenward-Roger Multiple Dim ens ion Hypothesis Testing Method.................. 29 2.3.2 Fai-Cornelius Multiple D imension Hypothesis Testing Methods.................... 35 2.4 Negative Variance Component Estim ates.....................................................................38 3 ESTIMATES OF MSEP FOR MULTIPLE DIMENSION LI NEAR COMBINATIONS.... 42 3.1 Reconciling Kenward-Roger (1997) and Fa i-Cornelius (1996 ) Definitions of the Correction Term Approximation................................................................................... 42 3.1.1 Linear Combinations of Fixed Effects ..............................................................42 3.1.2 Linear Combinations of Fixed and Random Effects.........................................44 3.2 Multip le Dimension Bias Correction............................................................................ 49 3.2.1 Bias Correc tion Estimator for Multiple Dimension Linear Combinations of Fixed Effects.....................................................................................................49

PAGE 7

7 3.2.2 Bias Correc tion Estimator for Multiple Dimension Linear Combinations of Fixed and Random Effects................................................................................ 51 3.2.3 Definiteness Properties of the Estimator for MSEP for MultipleDimension Linear Combinations of Fixed and Random Effects...................... 59 3.3 Transform Invariance of MSEP Estimator.................................................................... 61 4 BALANCED ONE-WAY RANDOM EFFECTS MODEL................................................... 63 4.1 Kackar-Harville Approxim ation Ba sed on REML Estimation Method........................ 64 4.2 Kackar-Harville Approxim ation Based on ANOVA Estimation Methods...................75 4.3 Bias Correction Term for the Balanced One-Way Random Effects Model.................. 84 4.3.1 Bias Correction Term Approximation under Parameterization..................... 84 4.3.2 Expected V alue of KRBIAS with REML Covariance Parameter Estimation......................................................................................................... 87 4.3.3 Expected V alue of KRBIAS with ANOVA Covariance Parameter Estimation......................................................................................................... 92 4.3.4 Impact of Negative Values of Bias Correction Term Approximation.............. 95 4.4 Perform ance of Overall Estimators for the MSEP........................................................ 98 4.5 Comparison of Prediction Interval Methods............................................................... 101 5 THE EFFECT OF NEGATIVE VAR IANCE COMPONENTS ON EBLUP...................... 137 5.1 Negative Variance Component Estim ates and the Balanced One Way Random Effects Model..............................................................................................................137 5.2 Considering the Variance Param eter as a Covariance................................................ 141 5.3 BLUP derivation for Random Effect s Model with Correlated Errors........................ 143 6 CONCLUSION AND FUTURE WORK............................................................................. 148 APPENDIX DETAILS FOR MONTE CARLO SI MULATION ST UDY ON PREDICTION INTERVALS........................................................................................................................152 LIST OF REFERENCES.............................................................................................................159 BIOGRAPHICAL SKETCH.......................................................................................................162

PAGE 8

8 LIST OF TABLES Table page 2-1 Summary of developments for MS EP estim ation and hypothesis testing......................... 40 4-1 Accuracy of Kackar-Harville estimator (4-17) for RC ................................................71 4-2 Relative bias (%) of estimators of correction term............................................................ 84 4-3 Relative bias (%) of MSEP estimators under REML and ANOVA estimation.............. 100 4-4 Summary of prediction interval procedures. .................................................................... 102 4-5 Effect of prediction interval procedures for k = 6, n = 6, 2 1 and 2 .5a ................ 108 5-1 Generated data that produces negative variance component estimate............................. 138 5-2 REML variance parameter estimates............................................................................... 138 5-3 Solution for random effects with RE ML variance parameter estimates.......................... 138 5-4 REML NOBOUND variance parameter estimates.......................................................... 139 5-5 Solution for random effects with REML NOBOUND variance param eter estimates.....139 5-6 LSMEANS from PROC GLM......................................................................................... 139 5-7 Data generated from distribution in Model (5-1)............................................................. 145 5-8 Covariance parameter es tim ates under Mode1 (5-1)....................................................... 146 5-9 Solution for random effects under Model (5-1)............................................................... 146 5-10 LSMEANS from PROC GLM......................................................................................... 146 5-11 Variance parameter estimates under random effects model............................................147 5-12 Solution for random effect s under random effects model............................................... 147

PAGE 9

9 LIST OF FIGURES Figure page 4-1 Equation (4-13) as a function of for several values of k, holding n = 6. Equation (4-14) holds when functions cross zero line.................................................................... 109 4-2 Accuracy of approximations Rtr A B and tr -1A I for the correction term, RC under REML variance component estimation......................... 110 4-3 Accuracy of approximation Rtr A B and estimator REtr A B for the correction term, RC under REML variance component estimation.................... 111 4-4 Accuracy of approximation tr -1A I and estimator Etr -1A I for the correction term, RC under REML variance component estimation.................... 112 4-5 Comparison of accuracy of estimators for the correction term, RC under REML variance component estimation........................................................................................ 113 4-6 Comparison of accuracy of approximations tr -1A I Atr A B and Atr A B for the correction term, AC under ANOVA variance component estimation...................................................................................................... 114 4-7 Accuracy of approximation Atr A B and estimator AEtr A B for the correction term, AC under ANOVA variance component estimation............... 115 4-8 Accuracy of approximation Atr A B and estimator AEtr A B for the correction term, AC under ANOVA variance component estimation................ 116 4-9 Accuracy of approximation tr -1A I and estimator Etr -1A I for the correction term, AC under ANOVA variance component estimation................ 117 4-10 Comparison of accuracy of estimators for the correction term, AC under ANOVA variance component estimation........................................................................ 118 4-11 Accuracy of estimators for the correction term under REML, and ANOVA, variance component estimation........................................................................................ 119

PAGE 10

10 4-12 Relative bias of estimators fo r the correction term under REML, or ANOVA, variance component estimation........................................................................................ 120 4-13 Bias produced by estimating 1M with 1M.......................................................... 121 4-14 Accuracy of approximation 1 2 tr -1 I and estimator 1 2 Etr -1 I for the true bias correction term, 11 MEM under REML variance component estimation................................................................. 122 4-15 Accuracy of approximation tr -1A I and estimator Etr -1A I for the true bias correction term, 11 MEM under REML variance component estimation..................................................................................................................... ....123 4-16 Comparing the accuracy of 1 2 Etr -1 I and Etr -1A I for the true bias correction term, 11 MEM .......................................................... 124 4-17 Bias produced by estimating 1M with 1M .......................................................... 125 4-18 Accuracy of approximation 1 2 tr -1 I and estimator 1 2 Etr -1 I for true bias correction term, 11MEM under ANOVA variance component estimation........................................................................ 126 4-19 Accuracy of approximation, tr -1A I and estimator Etr -1A I for true bias correction term, 11MEM under ANOVA variance component estimation..................................................................................................................... ....127 4-20 Comparing the accuracy of 1 2 Etr -1 I and Etr -1A I for the true bias correction term, 11MEM .......................................................... 128 4-21 Comparison of estimators of the MSEP for the EBLUP o f ia utilizing REML variance component estimation........................................................................................ 129

PAGE 11

11 4-22 Comparison of estimators of the MSEP for the EBLUP o f ia utilizing ANOVA variance component estimation........................................................................................ 130 4-23 Relative bias of possible es tim ators for MSEP of EBLUP for ia ................................... 131 4-24 Simulated coverage rates of pr ediction interval p rocedures for 1ta with k = 3...........132 4-25 Simulated coverage rates of pr ediction interval p rocedures for 1ta with k = 6...........133 4-26 Simulated coverage rates of pr ediction interval p rocedures for 1ta with k = 15.........134 4-27 Simulated coverage rates of pr ediction interval p rocedures for 1ta with k = 30.........135 4-28 Simulated coverage rates for the Kenw ard-Roger prediction in terval p rocedure............ 136

PAGE 12

12 Abstract of Dissertation Pres ented to the Graduate School of the University of Florida in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy EVALUATING ADJUSTMENTS TO THE MEAN SQUARED ERROR DUE TO ESTIMATING VARIANCE PARAMETERS IN LINEAR MIXED MODELS By Jamie McClave Baldwin December 2007 Chair: Ramon Littell Major: Statistics Several methods are commonly used to handl e the propagation of va riance stemming from estimating unknown variance componen ts in Linear Mixed Models (LMM). The accuracy and consequences of using these methods, however have not been thoroughly investigated. Empirical Best Linear Unbiased Predictors (E BLUP) for analyzing LMMs are widely used, yet the best way to evaluate the precision of the EBLUP is not generally understood. Many developments in the estimation of the Mean Squa red Error of Prediction (MSEP) of the EBLUP, and the use of these estimates for hypothesis testi ng, have occurred during the last two decades. This dissertation begins with a thor ough review of these developments. Existing methodologies for evaluating the precis ion of the EBLUP are generalized to apply to multiple dimension linear combinations of fixed and random effects, and the definiteness properties of these methodologies are examined. The methods for evaluating the MSEP of the EBLUP are examined thoroughly for the balanced one-way random effects model to assess the accuracy of the methods, the effect of the para meterization of the model, and the impact of variance parameter estimation techniques on the components of the MSEP estimators. The impact of negative variance component estimates on EBLUP methodologies is examined, and an

PAGE 13

13 alternative solution to account for negative variance estimates is developed for the one-way model.

PAGE 14

14 CHAPTER 1 INTRODUCTION AND BACKGROUND 1.1 Introduction to Mixed Models In 1984, Kackar and Harville introduced a new m ethod to handle the propagation of variance stemming from estimating unknown variance parameters in linear mixed models. In the nearly quarter century since this development, there have been many adjustments, improvements and new techniques resulting from their idea. Lin ear mixed models and th e procedures used to analyze them are full of complexities and nuances that can confuse even the most experienced analyst. For example, complex covariance struct ures coupled with unbalanced data and variance parameter estimates on or near the boundary of the parameter space can produce confounding results with many mixed models procedures. Th is paper begins to unwind and isolate these complexities using simple cases to explore th e best options available and determine when procedures for estimating the realized values of the random effects and evaluating the variability of these estimates succeed and fail. Mixed models methodologies have develope d over time to accommodate situations where prediction of the realized va lues of the random effects is requ ired. These situations arise, for example, in agriculture, education, and clinic al trials. A typical ag riculture example is the need to assess the genetic contribut ion of a particular animal for br eeding, which is considered to be drawn from a population of animals. An exam ple from education is th e desire to assess the effect of a particular teacher on the test scores of his or her pupils. A clinical trial with multiple clinics and varying patient dropout rates gives another arena where mixed models are frequently used to analyze data. Many of these analyses involve complex c ovariance structures and almost all analyses involve unknown cova riance parameters. It is im portant not only to develop accurate methods for evaluating the realized value of the random effect, but also to evaluate the

PAGE 15

15 associated mean squared error to determine the precision of the estimate. We begin by describing the framework of the model. 1.2 The Linear Mixed Model The general for m of the linear mixed model is yX Zue (1-1) where X is a known (n x p ) design matrix for the fixed effects, is a ( p x 1) vector of fixed effect parameters, Z is a known (n x q ) design matrix for the random effects, u is a ( q x 1) vector of random effects, and e is an ( n x 1) vector containing the ra ndom error terms. We will assume that Eu0, VuG, and E e0, V eR, cov u,e0. From the model in Equation (1-1), it follows that EyX and V y VZGZR. The matrices G and R are taken to have known structures but depend on an unknown dispersion parameter, for some known parameter space Multivariate normal distributions on u and e are often assumed and many of the results are derived under normality assumptions; the validity of these results under other distributional assumptions is questionable. For simplicity, all matrices are assumed to be of full rank. 1.3 Best Linear Unbiased Predictors (BLUPs) When is Known: Mixed Model Equations Best Linear Unbiased Predictors (BLUPs) originated from the desire to estimate the realized values of the random effects along with the fixed eff ects, or predictable linear combinations of both types of effects. Th is method was originally developed by Henderson (1950) for estimating genetic merits in animal breeding. Henderson approached the problem using maximum likelihood methodology, although he later acknowledged that his method cannot be called maximum likelihood since the func tion being maximized is not a likelihood (Robinson 1991, p.18). Assuming that u and e are normally distributed, Henderson maximized

PAGE 16

16 the joint density of y and u with respect to and u. This method results in Hendersons mixed models equations (MMEs): -1 -1 -1 -1 -1-1 -1XRX XRZuXRy ZRX ZRZGuZR y (1-2) Note here that G and R are assumed fully known. The solutions to these equations, -1 -1 -1 -1 =XVXXV y u=GZVy-X (1-3) are the Best Linear Unbiased Estimate (BLUE) and Best Linear Unbi ased Predictor (BLUP), respectively: they are linear in the data, unbiased, and achieve minimum variance or mean squared error (MSE), respectivel y, among all linear, unbiased estimat ors or predictors (see, for example, Harville 1990 and Robinson 1991). Note that the solution for in Equation (1-3) gives th e GLS estimate. Robinson summarized several other justifications for the BLUP estimates, including a Bayesian method with a flat prior on and a N0,G prior on u yielding the same result as the MMEs, as well as justif ications that do not require normality. A linear combination of fixed and random effects is pr edictable when the fixed effect portion is estimable. That is, for vectors and + u is predictable if =aX for some a (Littell, Milliken, Stroup, Wolfinger, and Schabenberger 2006, p. 211). When estimation of a predictable linear combination of fixed and random effects is desired, + u gives the BLUP for the predictable function. 1.4 Empirical Best Linear U nbiased Estimation (EBLUP) The estimates in Equation (1-3) assume G and R (and thus V) are fully known; this assumption, however, is rarely satisfied. A common alternative is to assume G and R are known

PAGE 17

17 up to a dispersion parameter 1,,r thus G=G and R=R The BLUP solutions depend on G and R, and therefore which is typically unknown. The standard way of resolving this issue is to employ two-stage es timation; that is, first estimate the dispersion parameter by, say, then use these estimates in the equations for the BLUP of the predictable function, yielding + u, where and u are the solutions to Equations (1-2) with estimating : G=G R=R V=V -1 -1 -1 -1 =XVXXV y u=GZVy-X These predictors are often referred to as Em pirical BLUPs (EBLUP), although they are no longer linear functions of the data, th ey are not necessarily unbiased, and they no longer necessarily have minimum MSE. There are many ways to estimate including ANOVA, MINQUE, MIVQUE, maximum likelihood (ML), and restricted maximum likeli hood (REML). The REML methods tend to be favored from a theoretical perspective (Searle, Casella, and McCulloch, 1992). However, from a data analytic perspect ive it is unclear which method is most advantageous. The technicalities of these methods will not be reviewed here. Fo r an overview of the technical details of these estimation methods see, for example, Khuri and Sahai (1985) and Robinson (1987). We will review the advantages and disadvantages of the methods as necessary in the context of evaluating the precision of the EBLUP methodology. 1.5 Two-Stage Estimation and Unbiasedness The consequ ences of two-stage estimation on the estimates of the linear combination, + u, are discussed in Kackar and Harville ( 1981). They show that when the dispersion

PAGE 18

18 parameter estimates are translation invariant, even functions of the data, and the data are symmetrically distributed, the two-stage estima tion procedure resulting in the EBLUPs provides unbiased predictors of linear co mbinations of effects; i.e., E + u Khatri and Shah (1981) provide similar results. Kackar and Ha rville (1981) show that these conditions are satisfied by assuming normality of the random effects and error and using REML, ML, ANOVA, MINQUE or MIVQUE variance component estimation methods. Note that this result can be extended to the broader class of location equivariant estimators, as shown in Kackar and Harville (1984), citing a theorem by Wolfe (1973). In evaluating the precision of EBLUPs, we must account for sampling variability contributed by the estimation of When this added variability is ignored, the true precision is often grossly underestimated, as demonstrated by Kackar and Harville (1984), Harville and Jeske (1992), and Tuchscherer, Herrendorfer, and Tuchsc herer (1998). Goals of this dissertation include clarifying and compari ng existing methods of estimating the precision of the EBLUP, assessing the accuracy of these methods and ev aluating the methods of hypothesis testing which utilize the various methods of estimating the precisi on. After reviewing th e related literature in Chapter 2, we generalize results on evaluating the precision of the EBLUP to include multiple dimensional cases and random effects in Chapte r 3. A case study of the balanced one-way random effects model in Chapter 4 sheds light on the problems with existing methods and suggests possibilities for improvement. Chapte r 5 tackles the issue of negative variance component estimates and the impact on EBLUP me thodology. Conclusions and future work are summarized in Chapter 6.

PAGE 19

19 CHAPTER 2 LITERATURE REVIEW 2.1 Precision of EBLUPs 2.1.1 Mean Squared Error of P rediction (MSEP) of EBLUPs Let t + u and let + u denote the estimate of t when (and hence V ) is known. When is unknown, denote the estimate of t, the EBLUP, by u. The prediction error for the EBLUP is t and the mean squared error of prediction (MSEP) is 2 M Et. By decomposing the prediction error into four components, each assuming a different, successive state of knowledge, Harville (1985) shows how each component adds to the variability. The first state of knowledge supposes full knowledge of the joint distribution of (y, t). The unbiased minimum MSE predictor of t in this case is clearly |EtE y + u|y(Harville 1985, sec. 2.1). This is known as the Best Predictor. The second state of knowledge supposes th e first and second mome nts are known, i.e., and are known but the functional form of the joint distribution of (y, t) is unknown. In this case the predictor with minimum MS E among all linear predictors is where -1 -1 -1 + GZZGZ+R y -X GZZGZ+Ry GZZGZ+RX (2-1) (Harville 1985, sec. 2.2). This is known as the Best Linear Predictor.

PAGE 20

20 The third state of knowledge assumes only th e variances and covari ances are known, i.e., is known but and the functional form of the joint distribution of (y, t) are unknown. Since depends on we must replace with an estimate. Choosing the BLUE for results in where B B -1 -1y GZZGZ+Ry y GZZGZ+RX (2-2) (Harville 1985, sec. 2.3). This is the Best Linear Unbiased Predictor (BLUP). The fourth state of knowledge represents th e typical situation in data analysis: all moments and the form of the joint distribution of (y, t) are unknown. That is, now is also unknown. Because the BLUP in Equation (2-2) depends on we cannot direc tly use it without further adjustment. Let z be maximal invariant, that is, let z L y where L is an n x (n p) matrix with rank(L) = n p and 0 LX. A function g y is translation invariant if and only if gh y L y Thus let z be an arbitrary translation invariant estimate of Then BBdPzyy; ; where ; Pz is an arbitrary probability distribution on gives a feasible estimate of t. The Empirical Best Linear Unbiased Predictor, EBLUP, corresponds to ;By the value of B y when ;Pz is the degenerate probability distribution assigning all mass to (Harville 1985, sec. 2.4). Note that B y is translation invariant, thus preserving the unbiasedness of the predictor. With this notation, we now have the f our components of th e prediction error: 3 12 4 ||BBBtEttEt yyy; (2-3)

PAGE 21

21 Each successive component is associated with moving from one state of knowledge to the next. This decomposition allows analysis of the role of each component in the MSEP. Component (1) of Equation (2-3) is uncorrelated with all other com ponents by the following result (Harville 1985, p. 135): cov((),(|)))()(|) ()((|))| 0. hEttEhEtt EEhEtt yyyy yyy (2-4) Components (2) and (3) of Equation (2-3) are also uncorrelated since cov(,(|))0Et yy and B is linear in y (Harville 1985, p. 135). Thus the Mean Squared Error of Prediction (MSEP), M can be written as the following six components: 2 2 222 (1) (2) (3) (4) (5) (6) (|) (|) ; (|); ;.BBB BBBBBMEt EEtytEEtyEEy EEtyyEy (2-5) In the development of estimators for M it is often assumed that the data follow a normal distribution. When the joint distributions of zt and ,Bz y are multivariate normal, as is the case when u and e are normally distributed, then components (5) and (6) of Equation (2-5) are 0. However, without normality, th ere is no guarantee that these cross-product term s disappear. The remainder of this chapter will review the developments over the last two decades for evaluating the MSEP of the EBLUP and conducting tests of hypotheses. Table 2-1 contains a

PAGE 22

22 summary of the developments reviewed herein and stands as a helpful reference for these developments and the notation used throughout. 2.1.2 Kackar-Harville Approximation to MSEP Kackar and Harville (1984) and Harville a nd Jeske (1992) assume normality in developing an approximation and estimator for M Thus, only components (1) (4) of Equation (2-5) need to be assessed for the approxim ation. Components (1), (2), and (3) of Equation (2-5) com prise the MSEP when is known. This is referred to as the nave MSEP and will be denoted by 2 1 B M Et In the mixed model case with V(u) = G and V(e) = R, this is 1M -1 -1 -1 -1 -1 G GZVZG + GZVXXVX -XVZG When normality is assumed, the cross-product te rms are zero, and the only contribution to the MSEP of estimating is component (4) of Equation (2-5). We will refer to this component as the correction term and denote it by C. 1M is often used as an approximation for the MSEP. When the data are normally distributed, this can seriously underestimate the MSEP since it ignores the contri bution of estimating captured by C, as shown by Kackar and Harville (1984), Harville and Jeske (1992), and Tuchscherer et al. (1998). Peixoto and Harville (1986) expand on results in Kackar and Harville (1984) to derive exact, closed-form expressions of the MSEP for certain predictors for the special case of the balanced one-way random effects model. We use similar methods in Chapter 4 to enable comparisons of approximations and estimators to the exact MSEP in the balanced one-way random effects model. Assuming normality, Kackar and Harville (1984) provide an approximation to C using Taylor series methods. Using a Taylor series expansion of about

PAGE 23

23 yields () d y ; where ()() dy; y; This leads to the approximation, 2 2 2 = d( y ; ) (2-6) Note that when is linear in this Taylor-series approximati on is exact. The validity of the Taylor-series approximation is contingent on two conditi ons: First, the function being expanded must have derivatives of all orders defined in a neighborhood of the point about which the function is being expanded; and second, the remainder term mu st tend quickly to zero (Khuri 1993, p. 111). Thus if the parameter space, is restricted (i.e., bounded below by zero) and the true value of falls on the boundary of the parameter space, the Taylor series expansion is not valid. Assessing the effect of the parameterizat ion and parameter estimates on the estimates of the MSEP is one goal of this dissertation. When is unbiased for and cov,0 dy; dy; we have 2 EEEE 2 d( y ; ) d y ; d y ; Noting that 0E dy; Kackar and Harville (1984) propose the following approximation to C: Ctr A B (2-7) where, 2var() = E A dy; d(y; ), and B is either an approximation to or the exact value of the MSE matrix, E The inverse of the observed information matrix, 1I is commonly used for B Equation (2-7) will be referred to as the Kackar-Harville

PAGE 24

24 approximation for the correction term. Thus th e approximation of the MSEP of proposed by Kackar and Harville (1984) is 1Mtr A B (2-8) The accuracy of the Kackar-Harville approximation of the correcti on term in Equation (2-7) depends on the following conditions: 1. linear in ; 2. the validity of Taylor series expansion of about ; 3. unbiased for ; 4. cov 0 dy; dy; ; 5. the tractability of the exact MSE of If all of these criteria are me t, the Kackar-Harville approximation exactly equals the correction term it is approximating. 2.1.3 Estimating C Because the calculation of th e Kackar-Harville approximati on to the MSEP in Equation (2-8) requires knowing it is not directly functional. An estimator based on the KackarHarville approximation was developed by Harville and Jeske (1990) and Prasad and Rao (1990) and is now implemented in the MIXED procedure of SAS. Following developments from Prasad and Rao (1990), an estimator for tr A B is given by tr A B ; where is substituted for in the Kackar-Harville approximation for the correction term. The justification for this substitution is not directly evident. Pr asad and Rao (1990) show that in the special case of the Fay-Herriot small area model, in which V(e) = R is known, the order of approximation is 1()oq where q is the number of small areas, or in ge neral the number of levels of the random effect. This result has not been generalized for the typical case when R is unknown. The

PAGE 25

25 validity of this substitution and the consequences on the accuracy of the MSEP estimate will be examined for the balanced one-way random effects model in Chapter 4. 2.1.4 Estimating 1M Historically, 1M has often been estimated by 1M that is, by substituting for in 1M; however, 1M may have bias of the same or hi gher order as the correction term (Prasad and Rao 1990; Booth and Hobert 1998). Consequently, it is just as important to account for the bias introduced by estimating 1M with 1M as it is to account for C Prasad and Rao (1990), Harville and Jeske (1992), and Kenward and Roger (1997) all approximate the bias of 1M by taking the expectatio n of a Taylor series of 1M about (assuming is unbiased for ). Harville and Jeske (1992) developed the following expression for the bias of 1M : 111 2EMMtr B (2-9) where 2 1 2M They show that when cov,t y vart and var y are all linear in 2 =-A leading to 11EMMtr A B (2-10) Prasad and Rao (1990) and Kenward and Roge r (1997) developed expressions similar to Equation (2-10). Again by substituting for possible estimates for 1M are

PAGE 26

26 11 2Mtr B (2-11) or 1Mtr A B (2-12) Notice again that there is bias introduced by substituting for in Equations (2-11) and (2-12). This bias is often assumed to be negligible, although the only noted justif ication, previously m entioned in Section 2.1.3, comes from Prasad and Rao (1990) and only applies to Equation (2-12) in the Fay-Herriot model. We will stud y this issu e further through a special case in Chapter 4. 2.1.5 Estimators of M Based on the Kackar-Harville Approximation Based on Equations (2-11) and (2-12), respectively, the following estimators of the MSEP follow from the Kackar-Harville approxim ation and the bias correction approximations: 11 2Mtrtr B A B (2-13) and 1 2Mtr A B (2-14) The DDFM = KR option of the MIXED procedure in SAS utilizes Equation (2-13) which reduces to E quation (2-14) when the model covariance structure is linear in Chapter 3 explores the Kenward and Roge r (1997) and Fai and Cornelius (1996) expansion of Equation (2-13) into multiple dimensions. Therein we establish the equality of the Kenward-Roger and Fai-Cornelius approxim ations to the correction term and expand the term to include linear combinations containing random eff ects. We also provide a proof that the most

PAGE 27

27 general form of the correction term approximation is non-nega tive definite and discuss the definiteness properties of the overall MSEP estimator. 2.2 Hypothesis Testing for Single Dimension Linear Combinations Several methods of approximating the distributions for test statistics have been developed for single and multiple dimension linear combinatio ns. The accuracy of these methods has been largely untested. Following is a summary of the available methods for approximating the distributions for single dimension linear combinations. Giesbrecht and Burns (1985) and Jeske and Ha rville (1988) provide approximations for ttype test statistics based on the Satterthwaite (1941) method. Giesbrecht and Burns (1985) present a test statistic for fixed effect s with the nave variance estimate: GBGB dftt -1 -1 XVX (2-15) When V is known and used in place of Vin Equation (2-15), the test statistic has a normal distribution. Thus, it is assum ed that Equation (2-15) has a t-distribution with the degrees of freedom derived from the approximate distribution of the denominator. To estimate the degrees of freedom, a Satterthwaite method is used. Th e variance of the denominator is matched with that of the approximating 2 distribution as follows: 2 2 2 var 2 var 2 2 varGB GB GB GB GBdf df df df df -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 XVX XVX XVX XVX XVX XVX (2-16)

PAGE 28

28 The term var -1 -1 XVX is evaluated using a Taylor series approximation and the asymptotic covariance matrix of In the Giesbrecht-Burns case, this yields 11 var cov, .rr ij ij i j -1 -1 -1-1-1-1 -1 -1 -1 -1V XVX XVZZVXXVX V XVZZVXXVX (2-17) The underlying assumption in developing th is procedure is that the distribution of -1 -1 XVX is a multiple of a 2 -distribution. Jeske and Harville (1988) also use a Satterthwaite method to approximate the degrees of freedom of a t-statistic; however, they include fixed and random effects in their test statistic with a corrected version of the MSEP estimate. The approximate t-test is given by 1 J H J Hd ftt Mtr A B (2-18) where 2 1 1 2 varJHMtr df Mtr A B A B (2-19) As in the Giesbrecht-Burns t -test, the denominator of Equation (2-19) is approximated by a Taylor series expansion yielding 1 var,Mtr A B h B h (2-20) where 1 and is an approximation of var. Mtr A B h B

PAGE 29

29 Again, an underlying assumption is that the distribution of 1 Mtr A B follows a multiple of a 2 -distribution. An extension of this method would be to incorporate the bias correction into the MSEP as seen in Equations (2-13) and (2-14). This extension is considered in Chapter 4 2.3 Hypothesis Testing Methods for Mult iple Dim ension Linear Combinations Kenward and Roger (1997) and Fai and Cornelius (1996) offer F -tests to handle multiple dimension as well as single dimension cases; however, the methods for determining the distribution of the test statis tics differ. One of the Fai and Cornelius methods (DDFM = SATTERTH option) and the Kenwar d-Roger method (DDFM = KR op tion) are available in the SAS MIXED procedure. A goal of this dissertati on is to compare these methods with other modified methods from Fai and Cornelius ( 1996) for the balanced one-way random effects model. We will also assess the effect of sample size, parameterization, and parameter value on these hypothesis testing methods. 2.3.1 Kenward-Roger Multiple Dime nsion Hypothesis Testing Method The test statistic develope d by Kenward and Roger (1997) utilizes an MSEP estimate for multiple dimension linear combinations that is ad justed for the correction term and the bias of 1 M This MSEP estimate is thoroughly discussed in Chapter 3, but for now will be denoted by KRM Note that these methods are only devel oped for tests of fixed effects; thus the MSEP can be thought of as a variance. Theoretically these methods could easily extend to include random effects. We include a si mulation study for the one-way random effects illustration to determine the validity of in cluding random effects in these procedures.

PAGE 30

30 Consider the hypothesis vs. 0aH:L =0H:L 0, where L is an l -dimension matrix of constants representing the linear combination of interest. Kenward and Roger (1997) propose the following Wald-type test statistic: 11 KR KRF l LLM LL An F -distribution is approximated for the test statistic to take in to account the random structure of KRM and to match known cases where the test statistic has an exact F -distribution. We will briefly review the procedure to allow comp arisons to alternative methods and to examine any shortcomings. Kenward and Roger (1997) use a Taylor series expansion to approximate 1KR LM L; however, it is unclear about which point the series is expanded and which higher order terms are dropped. It app ears that the series is expanded about 1M rather than the true value, M This may be justified by dropping higher order terms; however, this step is the only place where the corrected MSEP is take n into account in the distribution of the test statistic. Using this Taylor series approximati on, Kenward and Roger (1997) use the identities | ||KR KRKR KR KR KR KR KREFEEF VFEVF VEF M M M to approximate the moments of th e test statistic. A two-moment -matching method is then used to obtain estimates for a coefficient, and degrees of freedom, m so that K R K RldfFF

PAGE 31

31 To make the approximating distribution match th e exact cases where the distribution is known, selected higher degree terms are added to th e Taylor series expans ion. The following summarizes the results of th e Kenward-Roger procedure: 2 4 and = 12KR KR KRKRdf l df lE Fd f where 2, 2KR KRVF EF and 1 21 2 2312 1 and 11KR KRAc B EF VF ll cBcB where 1 11 i j ,rr ij ijABtr tr 1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1V LLXVXLLXVXXXXVX V LLXVXLLXVXXXXVX (2-21) 2 11 i j ,rr ij ijABtr -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1V LLXVXLLXVXXXXVX V LLXVXLLXVXXXXVX

PAGE 32

32 12 1 2 3 12 21 6, 2 321 321 2 321 14 2 BAA l g c lg lg c lg lg c lg lAlA g lA and Bij is the ijth element of the matrix B The Kenward-Roger method is available in the PROC MIXED procedure in SAS as the DDFM = KR option in the model statement. This option performs the adjustment to the MSEP estimate and the hypothesis test as described above. We now demonstrate that in the single-dim ension case, the Kenwar d-Roger degrees of freedom approximation reduces to the Giesbrec ht-Burns degrees of freedom approximation reviewed in Section 2.2.1 (Kenward and Roger 1997, p. 988). Recall that the Giesbrecht-Burns test statistic did not use an adjustment for unknown in the MSEP. Thus, while the KenwardRoger and Giesbrecht-Burns test statistics di ffer in the estimate of standard error, the approximated distributions are identical. Consider again a hypothesis test for th e single dimension linear combination :0oH under the linear mixed model. The Ke nward-Roger test statistic is KR K Rt VAR (2-22) where

PAGE 33

33 1 11 2 11 2 1 4KR ij ij ijijVAR B -1 -1-1 -1-1 -1-1-1-1 V=VVV XVX XVXXVX VVV XXXVXXXXVVXXVX In this case 1 11 1 cov, *rr ij ij i j r ijAt r tr -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -2 -1 -1V XVX XVZZVXXVX V XVX XVZZVXXVX XVX 1 2 cov, *r ij i jA -1 -1-1-1 -1 -1-1-1V XVZZVXXVX V XVZZVXXVX since the quantities inside the trace func tions are scalars. Note also that 122 22 2 1 217 6, 22 25 1, 3 11 3(1)2(11)7 2 7 B AAA AA g A c c and 34 7c

PAGE 34

34 Thus, 1 2 2 2 22 22 221, 17 1 2 72 2, 112 2747 11 7272KR KREFA A A VF AA AA and 2 2 22 2 2 221 22 2212 112 A AA A AA The degrees of freedom for the Kenw ard-Roger hypothesis test are then 2 2 2 1132 4 2 1 212 2c o v KR rr ij ij i jdf A A A 2 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1V XVX XVZZVXXVX V XVZZVXXVX (2-23) Note that the denominator of Equation (2-23) is the Taylor series expression for var -1 XVX This is the same as the degrees of freedom estimate from the GiesbrechtBurns test in Equations (2-16) and (2-17); however, recall that the Giesbrecht-Burns test statistic is GBt -1 -1 XVX Thus K RGBdfdf but K RGBtt Often, the discrepancy in these test statistics may not be enough to make a significant diffe rence in the test results; how ever, there are cases when the

PAGE 35

35 discrepancy changes the conclusion of the test. The question becomes, which test is better? The accuracy of the estimating distribution must also be questioned in either test. Note that because the estimated distributions are equal, we can isolate the impact of modifying the estimated variance of the EBLUP on the hypothesis test. We explore that situ ation further though a simulation study in Chapter 4. 2.3.2 Fai-Cornelius Multiple Dime nsion Hypothesis Testing Methods Fai and Cornelius (1996) also use a moment matching method to determine the distribution for their multiple dimension test st atistics. The differences between the KenwardRoger method and the Fai-Cornelius method come in the particular MSEP estimate used in the test statistic, in the number of moments evaluated, and in the techniques used to evaluate the moments of the test statistic. Fai and Cornelius (1996) offer two alternatives for the test statistic: one uses the nave MSEP estimate in matrix form, 1M and the other uses the nave estimate plus a correction term denoted by, FC FC1M M C which will be discussed thoroughly in Chapter 3. Note that neither of these ta kes into account the bias of 1M for 1M as in the KenwardRoger procedure. The test statistics are 1 111 FCF l LLM LL (2-24) and 1 21 FC FCF l LLM LL (2-25) For each of these test statistics, both one and two moment-matching methods are offered for estimating an F -distribution, resulting in four alternatives for conducting a multiple dimension hypothesis test. The method of finding the moments is the same for both test statistics, so for

PAGE 36

36 simplicity let MSEP denote the estimator of the MSEP in eith er Fai-Cornelius test statistic. To evaluate the moments of the test statistics, fi rst a spectral decomposition is performed so that kdiag PLMSEPLP where P is an orthogonal matrix of eigenvectors and k 1,2,, kl are the eigenvalues of ) L(MSEPL. The test statistic is rewritten so that 1 1 1 2 1 2 1 .kFC k l k k k l klF diag t Q PPLLMSEPLPPL PL -PL PLMSEPLPPL -PL PL -PL PL -PL PL -PL (2-26) They then note that kt would have an independe nt standard normal distribution if the MSEP were known. It is reasonable then to approximate the distribution of kt with a t-distribution and estimate the degrees of freedom, k with a Satterthwaite procedure, i.e., 2 2 kk k k kV V VV MSEP PL -PL MSEP PL -PL As in the Kenward-Roger procedure, we have the underlying assumption that MSEP has a 2 distribution. If 2k for all k= 1, 2, ., l, then the moments of Q can be approximated as follows:

PAGE 37

37 2 112kll k kk kEQEt and 2 2 2 1121 24kll kk kk kkVQVt From Equation (2-26) we see that the moments of the Fai-Co rnelius test statistics are 1FC E FEQ l and 21FCVFVQ l. One or both of these moments are matche d to the moments of the approximating F-distribution, as in the Kenward-Roger procedure, to produce an estimate of degrees of freedom (and an estimated coefficient in the two-moment case). The one-moment method with the test statistic in Equation (2-24) is available in the P ROC MIXED procedure of SAS as the DDFM = SATTERTH option in the model statement. Note that in the single dimension case, the FaiCornelius test statistic in Equation (2-24) reduces to the Giesbrecht-Burns test statistic in Equation (2-15) and that the Fai-Cornelius one-mo m ent method of estimating the distribution also produces the same results as the Giesbrecht-Burns method. Thus we are avoiding the discrepancies that occur betw een the Kenward-Roger method a nd the Giesbrecht-Burns method described in Section 2.3.1. We are interested in comparing the available hypothesis testing methods to determine which one provides the most accurate Type-I error rate. Some work has already been done to this end by Schaalje, McBride, and Fellingham ( 2002). Schaalje et al. (2002) compare the one moment Fai-Cornelius method with the test statistic in Equation (2-24) (using the DDFM = SATTERTH option in S AS) with the Kenward-Roger method (using the DDFM = KR option in

PAGE 38

38 SAS). They performed simulations for four split-p lot designs, two of which were balanced and two of which were unbalanced. The simulation is re peated for five covariance structures to study the effect of the complexity of the covariance st ructure. Hypothesis tests were computed for whole plot effect and sub-plot effect using the DDFM = KR and DDFM = SATTERTH methods. Schaalje et al. (2002) calculated simulated Type I error rates and l ack-of-fit tests to determine if the simulated p-values followed a uniform (0,1) distribution. Both me thods performed equally well for the simple Compound Symmet ric covariance structure, regardless of the design. For the other four covariance structur es the Kenward-Roger method out performs the Fai-Cornelius method in the lack-of-fit test a nd the type I error rates; howev er, as the sample size becomes smaller or the covariance structure becomes more complex, neither method performs well. Type I error rates are severe ly inflated under both methods when complex covariance structures are coupled with small sample sizes. Note that when random effects are consider ed, it is more appropriate to focus the investigation on prediction limits. We will shift the focus to prediction intervals and expand the comparison to include modified Fai-Cornelius methods with adjusted MSEP estimates in Chapter 4. Although we confine the comparison to a single dimension linear combination in the balanced one-way random effects model, this comparison will paint a more complete picture of the methods considered and show the need for further investigation in to the proposed methods. 2.4 Negative Variance Component Estimates Very little research has focused on the imp act of negative variance parameter estimates on BLUP procedures, the MSEP estimation met hods, and hypothesis testing. Stroup and Littell (2002) begin to explore the impact of the met hod of variance component estimation and negative variance parameter estimates on tests of fixed eff ects. Their exploration reveals that REML may not always be the best choice in terms of power and true Type I error va lues for tests of fixed

PAGE 39

39 effects when negative variance parameter estimat es are prevalent. The impact of negative variance parameter estimates on the EBLUP an d MSEP estimation for linear combinations involving random effects has not been thoroughly investigated. In Chapter 5, we delve into the complexities of negative variance component estimates and the impact on EBLUP and the associated MSEP estimates. Smith and Murray (1984) develop a method for handling negative covariance estimates. They propose that the vari ance parameter in question be considered as a covariance parameter, thereby a llowing negative estimates. The impact of this strategy on EBLUP methods is considered in Chapter 5, l eading to an alternative model and analysis procedure.

PAGE 40

40Table 2-1. Summary of de velopments for MSEP estimation and hypothesis testing Paper Fixed or Random Effects Correction Term Bias Term MSEP Estimate Kackar and Harville 1984 Fixed and random tr A B None 1Mtr A B Giesbrecht and Burns 1985 Fixed None None -1 -1 XVX Jeske and Harville 1988 Fixed and random tr A B None 1 Mtr A B Prasad and Rao 1990 Fixed and Random 1 Etr tr oq A B A B For q = # small areas, in the FeyHerriot model 1 1 1 = () EM M tr oq A B For q = # small areas, in the Fey-Herriot model 1 2 Mtr A B Harville and Jeske 1992 Fixed and random tr A B 11 11 2 EMMtr Mtr B A B 11 2 Mtrtr B A B or 1 2 Mtr A B Fai and Cornelius 1996 Fixed tr A B defined for multiple dimension linear combination, denoted FCC ; shown to be nnd None 1M or FC FC1M M C Kenward and Roger 1997 Fixed tr A B defined for multiple dimension linear combination; KRC 1 2 Etr tr 11 1M M B M A B Defined for multiple dimension linear combination, KRBIAS KR KRKR1M M C BIAS

PAGE 41

41Table 2-1 cont. Paper Hypothesis Test Stat Distribution of Test Stat Kackar and Harville 1984 N/A N/A N/A Giesbrecht and Burns 1985 0: 0 H GBt -1 -1 XVX GBdft where 2 2 varGBdf -1 -1 -1 -1 XVX XVX Evaluated with Satte rthwaite-type method Jeske and Harville 1988 0: 0 H u 1 JHt Mtr A B J Hdft where 2 1 1 2 varJHMtr df Mtr A B A B Evaluated with Satte rthwaite-type method Prasad and Rao 1990 N/A N/A N/A Harville and Jeske 1992 N/A N/A N/A Fai and Cornelius 1996 0: H L 0 1 111 FCF l LLM LL 1 21 FC FCF l LLM LL One and Two moment matching; Spectral decomposition, Taylor series and Satterthwaite techniques used to determine moments. Note that this yields two choices of distributions for each test statistic. Kenward and Roger 1997 0: H L 0 11 KR KRF l LLM LL Two moment matching; Taylor seri es and conditional expectations used to determine moments.

PAGE 42

42 CHAPTER 3 ESTIMATES OF MSEP FOR MULTIPLE DI MENSION LINEAR COMBINATIONS 3.1 Reconciling Kenward-Roger (1997) and Fai-Cornelius (1996) Definitions of the Correction Term Approximation 3.1.1 Linear Combinatio ns of Fixed Effects Kenward and Roger (1997) and Fai and Cornelius (1996) extend the methods for estimating the MSEP summarized in Section 2.1 to multiple dimension linear combinations. Both methods require expanding the definition of the Kackar-H arville approximation of the correction term, and in the Kenward-Roger case, the bias correction term, into a matrix form. Recall that the Kackar-Harville approximation of the correction term for single dimension linear combinations is Ctr A B where var() = dE A y; dy; dy; ()() dy; y; and B is either an approximation to or the exact MSE matrix, E Extending this definition to multiple dimension linear combinations is not necessarily unique. In both papers, only linear combinations of the vector of fixed parameters are considered. In Section 3.1.2 we extend their method to in clude linear combinations involving both fixed and random effects. Noting that 11cov,rr ij ji ijtr b A B where bij is the ijth element of B Kenward and Roger expand this definition to the multiple dimension case by 11cov,rr KR ij ji ijb L L C C where L is an l-dimension matrix of constants representing the linear combinations of interest. We will denote this approximation as KRC

PAGE 43

43 Note that, 11 12 1 21 22 21cov,cov, cov, cov,cov, cov, cov,l ij ij ij ijij ij ij L L L L L L L L L L L L L L 12cov,cov, cov,lll l ij ij ij L L L L L L where Li is the ith row of L. Thus we can write the kmth element of KRC as 11 12 11 12cov,cov, cov,.KR kmkmkm km qq qqbbb L L L L L L C Fai and Cornelius (1996) define the multiple dimension approximation to the correction term differently and restrict their definition to a compound symmetric covariance structure with two covariance parameters. We now expand that definition to include a ny covariance structure with r parameters and show its equivalence to KRC They also establish the non-negative definiteness of their correction term definition, wh ich also applies in this more general setting. Let FCC denote the Fai and Cornelius multiple dimension approximation to the correction term. Then, 11 1 1 l FC ll ltr tr tr tr A B A B C A B A B (3-1) where

PAGE 44

44 11 1 1cov,cov, cov, cov,cov,km km r km km km km rr r L L L L L L A L L L L and B is as before. Then the kmth element of FCC is 11 12 11 12cov, cov, cov,.FC km km km km km qq qqtr b b b L L L L C A B L L Thus, clearly KRC = FCC Fai and Cornelius (1996) have shown that this definition of the correction term approximation is non-negative definite. This b ecomes an important factor when the MSEP estimate is to be used in a test statistic or conf idence interval. We define the most general form of this approximation and prove that it is non -negative definite in the Section 3.1.2. 3.1.2 Linear Combinations of Fixed and Random Effects We can now generalize the definition of the multiple-dimension correction term approximation to include random effects and esta blish that it is non-negative definite. Let | KLM denote an l-dimension matrix of constants. We now consider the linear combination, KL Mu u. To expand the definition to random effects, we need to redefine kmA as

PAGE 45

45 11 1 1()() cov ()()()() cov cov ()()()() cov cov ,kkmm km kkmmkkmm r kkmm kkmm rr L MuL Mu A L MuL Mu L MuL Mu L MuL Mu L MuL Mu r where kL is the kth row of the fixed effects portion of the matrix of constants, L, and kM is the kth row of the random effects portion of the matrix of constants, M. We now follow a similar procedure as used in Fai and Cornelius (1996) to establish that this generalized definition of KRC is non-negative definite and thus the MSEP estimator utilizing this correction term es timator is nonsingular. First, how ever, we need to broaden the definition of KRC to incorporate linear combinati ons of fixed and random effects. First, using the result that 1 11xx AA AA, we have 1 1 1 111 1 1 where k k kk iiii kk ii k i k i ki 11 1 1 11 11 111 1 1 1 111 1LXVXXVy XVX L V LXVXX y LX V y VV LXVXXyLXVXXXXVXXVy V LXVXXIPy V LXVXXVVIPy CVIPy 11 and kik i 11 11V CLXVXXVPXXVXXV and

PAGE 46

46 1 k kkk ii i i kk ii k i ki ki ki kikiki 1 11 111 1111 111 1Mu GV MZVIPyMGZIPyMGZVX GV MZVIPyMGZVVIPy V MGZVXXVXXVVIPy DVIPyEVIPyFVIPy DEFVIPy where ,kik i kik i 1G DMZ V EMGZV and 1.kik i 111V FMGZVXXVXXV Thus, ()kk kikikiki i 1L Mu CDEFVIP y We can now assess the ijth element of kmA as ()()() cov cov kkmm kmij ij kikikiki mjmjmjmj kikikiki mjmjmjmj kikikiki mjmjmjmj 11 11 1L MuL Mu A CDEFVIP y CDEFVIP y CDEFVIPVIPVCDEF CDEFVIPCDEF since

PAGE 47

47 11. 111 1 11IPVIPVVXXVXXVVXXVXXV IP We now have 1111 1111 kkkk km krkrkrkr mmmm mrmrmrmr 1CDEF A CDEF VIPCDEFCDEF (3-2) We can now generalize the definition of the K ackar-Harville correction term approximation as follows. Definition 3.1. The multiple-dimension Kackar-Harville approximation for a linear combination including fixed and random effects is ,KR kmtr C A B (3-3) where kmA is defined in Equation (3-2). We now prove that KRC is non-negative definite. Theorem 3.1. The multiple-dimension Kackar-Harville approximation for the correction term, as defined in Equation (3-3), is non-negative definite. Thus, the MSEP approximation utilizing this correction, KR1M C is nonsingular. Proof. We begin by examining the kmth term of KRC We can write //11 2 1 2VIPQQ and //1212B RR where Q1/2 and R1/2 both exist since 1VIP is non-negative definite and B is positive definite (e.g., Schott 1997, p.16).

PAGE 48

48 Consider 1111 1111 1111 1/21/2 1111 111 km kkkk mmmm mrmrmrmr krkrkrkr kkkk mmmm mmm krkrkrkrKR kmtr tr tr C A B CDEF 1 VIPCDEFCDEFB CDEF CDEF QQCDEFCDE CDEF 1/21/2 1 1111 1/2 1/21/2 1/2 1111 1/2 1/2 1111 m kkkk mmmmmrmrmrmr krkrkrkr mmmmmrmrmrmrtr tr FRR CDEF QCDEFCDEFRR Q CDEF QCDEFCDEFR 1/2 1/2 1111 1/2 1/2 1111 1/2 1/2 1111 *kkkk krkrkrkr mmmm mrmrmrmr kkkk krkrkrkr mkkvec vec QCDEFCDEFR QCDEFCDEFR QCDEFCDEFR m where 1/2 1/2 1111 iiii iririririvec QCDEFCDEFR Thus we can write KRC as 1where ,,.KR l C (3-4)

PAGE 49

49 Writing KRC in this form proves that it is non-nega tive definite (e.g., Schott 1997, p.16). Since E 1M L +MuL +MuL +MuL +Mu is positive definite, we have now established that the general approximation of MSEP, KR1M C (3-5) is positive definite, and hence nonsingular, and t hus is a valid measure of prediction error. 3.2 Multiple Dimension Bias Correction Recall that it is often nece ssary to correct for the bias introduced into the estimate of MSEP by using 1M to estimate 1M Kenward and Roger (1997) define a bias correction term for multiple dimension linear combinations akin to the single dimension bias correction offered by Harville and Jeske (1992) found in Equation (2-11). We now rewrite this term to resem ble the correction term in Equation (3-1), in order to maintain consistency in our notation and enable expansion to include linear com b inations containing random effects. The definiteness properties of either the bias correction term or the estimate of MSEP containing the bias correction term has not previously been addr essed. We will address that issue in Section 3.2.3. We first consider linear combinations containing only fixed effects. 3.2.1 Bias Correction Estimator for Multiple Dimen sion Linear Combinations of Fixed Effects We let bij again be the ijth element of B Kenward and Roger (1997) state the bias correction term as 2 11cov 1 2rr KR ij ji ijb L ,L BIAS (3-6) where

PAGE 50

50 222 1121 2 222 12var cov, cov, cov cov,cov, varl ijijij ij lll ijijij L L L L L L ,L L L L L L Thus, the kmth element of KRBIAS is 22 11 12 2 11 2 22 2cov cov 1 2 cov cov .km km KR km km km ij rr ij rbb bb L ,L L ,L BIAS L ,L L ,L (3-7) We now rewrite KRBIAS as 1 11 2l KR ll ltr tr tr tr 11 B B BIAS B B (3-8) where 22 2 2 11 22 2 1cov, cov, cov, cov, cov,km km r km km km km rr L L L L L L L L L L To confirm the equality of Equations (3-8) and (3-6) note that the kmth element of Equation (3-8) is also

PAGE 51

51 2 11 2 1 22 12 12cov, 1 2 cov,cov, km km km ij ijKR km kmb bbtrB L L L L L L BIAS 2 2cov,km rr rb L L (3-9) which is identical to Equation (3-7). It has been shown by Ke nward and Roger (1997) that Equation (3-6) reduces to the multiple dimension correction term ( KRC ) when the covariance matrix, V is linear in which gives KR KR 1M =M +2C as an estimator of the MSEP. In the next section, we use Equations (3-8) and (3-9) to define the estimator for bias correction for a m ultiple-dimension linear combination involvi ng fixed and random effects and confirm that the resulting bias correction term continues to reduce to the correction te rm in linear covariance cases. 3.2.2 Bias Correction Estimator for Multiple Dimen sion Linear Combinations of Fixed and Random Effects The extension to include random effects is a straightforward but tedi ous task. However, it is important to explore fully the functions invo lved to determine when the term simplifies to the correction term approximation in Definition 3.1, and also to assess the effect on the definiteness properties of the MSEP estimator. The generalized bias co rrection term has the same form as in the fixed effect case: 11 1 11 2l KR llltr tr tr tr B B BIAS B B (3-10) but now,

PAGE 52

52 2 22 2 11 22 2 1cov, cov,cov, cov, cov,kkmm km kkmmkkmm r kkmm kkmm rr L MuL Mu L MuL Mu L MuL Mu L MuL Mu L MuL Mu (3-11) Now considering a single element of this matrix we see that 2 () 2 22 12cov, cov,cov,cov,cov, cov,cov, kkmm kmij ij kmkmkmkm ij kmkm ijij L MuL Mu L L MuL L MuMuMu L L MuL 22 34 cov, cov,.km km ij ij L MuMuMu (3-12) We now investigate each piece of Equation (3-12) using Littell et al. (2006) as a reference for the covarian ces involved. C onsider Part (1) of Equation (3-12):

PAGE 53

53 22 221cov, cov ,km k m ij ij km k m ij ij km ji k -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1-1 -1 -1 -1 -1 -1-1-1 -1 -1 -1L L LXVXXVyLXVXXVy LXVXXVXXVXLLXVXL V LXVXXVVXXVXL LXVXXV 2 m ji km ji km ij km ij k j -1 -1 -1-1 -1-1-1 -1 -1 -1 -1-1-1-1 -1 -1 -1 -1 -1-1 -1 -1 -1 -1-1-1-1 -1 -1 -1-1-1VV VXXVXXVVXXVXL VV LXVXXVVVXXVXL V LXVXXVVXXVXL VV LXVXXVVVXXVXL V LXVXXVVXXVX 2 2,m i kjmikjmikimjkimj km ij kimjkjmikm ij -1 -1 -1-1-1 -1 -1 -1 -1 -1 -1 -1 -1 -1-1 -1 -1 -1 -1 -1 -1 -1-1V XVVXXVXL CVPCCVCCVPCCVPC V LXVXXVVXXVXL V CVIPCCVIPCLXVXXVVXXVXL (3-13) where ki C is as defined in Section 3.1.2. Now moving on to part (2) of Equation (3-12), we have 222cov km k m ij ij km km ji i k i -1 -1-1 -1 -1 -1-1 -1-1-1 -1 -1 -1-1 -1-1-1Mu,L MGZVXXVXL GV MZVXXVXLMGZVVXXVXL V MGZVXXVXXVVXXVX m L (3-14)

PAGE 54

54 km ij k m ij km ji km ji km ij k -1 -1-1 -1 -1 -1 -1 -1 -1-1 -1 -1 -1-1 -1 -1 -1-1-1 -1 -1 -1-1-1 -1 -1GV MZVVXXVXL GV MZVXXVXXVVXXVXL GV MZVVXXVXL VV MGZVVVXXVXL VV MGZVVVXXVXL V MGZV m ji k m ji k m ij k ji -1 -1 -1 -1 -1-1 -1 -1 -1 -1 -1 -1-1 -1 -1 -1 -1-1 -1 -1-1 -1 -1 -1 -1 -1 -1 -1-1 -1 -1-1 -1V VXXVXXVVXXVXL GV MZVXXVXXVVXXVXL VV MGZVVXXVXXVVXXVXL VV MGZVXXVXXVVXXVXXVVXXVXL 2 m k m ji k m ij k m ij km ij k -1 -1 -1 -1 -1-1-1 -1 -1 -1 -1 -1 -1 -1-1 -1 -1-1 -1 -1 -1 -1 -1 -1-1-1 -1 -1 -1 -1VV MGZVXXVXXVVVXXVXL VV MGZVXXVXXVVXXVXXVVXXVXL VV MGZVXXVXXVVVXXVXL G MZVXXVXL M 2 2 m ij k m ij -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1V GZVVXXVXL V MGZVXXVXXVVXXVXL

PAGE 55

55 22 kimjkimjkjmikjmi kimjkimjkjmikjmi kimjkimjkjmikjmi km k ij ij -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1DVC+DVPCDVC+DVPC EVC+EVPCEVC+EVPC FVC+FVPCFVC+FVPC GV MZVXXVXLMGZVVXXVXL 2 m k m ij -1 -1 -1 -1 -1 -1 -1V MGZVXXVXXVVXXVXL (3-15) 22 2.ki mjkj mi ki mjkj mi ki mjkj mi km km ij ij k m ij -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1DVI-PCDVI-PC EVI-PCEVI-PC FVI-PCFVI-PC GV MZVXXVXLMGZVVXXVXL V MGZVXXVXXVVXXVXL (3-16) where ki D ki E and ki F are as defined in Section 3.1.2. Du e to the symmetry of parts (2) and (3) of Equation (3-12), we combine (2) and (3) as follows: 22 11(2)(3) ,kikiki mj kjkjkj mi ki kjkjkj kj kikiki ij ijfg -1 -1 -1 -1DE+FVI-PC DE+FVI-PC CVI-PD+E+F CVI-PD+E+F GV (3-17) where 2 1 ijf G and 2 1 ijg V are linear functions of 2 ij G and 2 ij V, respectively. Part (4) of Equation (3-12) tediously completes ()kmij as follows:

PAGE 56

56 22 22 22 24cov,km k m ij ij kmk m ij ij km km ij ij km ij ki mjkj mi -1 -1 -1 -1 -1 -1 -1MuMuMG-GZVI-PZGM GG MMMZVI-PZGM GP MGZVI-PZMMGZVZGM V MGZI-PZGM -DVI-PD-DVI-PD .ki mjkj mi ki mjkj mi ki mjkj mi ki mjkj mi kimjkjmi kimjkjmi -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1EVI-PD-EVI-PD -FVI-PD-FVI-PD -DVI-PE-DVI-PE -DVI-PF-DVI-PF +EVPE+EVPE +EVPF+EVPF (3-18) Now note that ii i i -1 -1-1 -1 -1 -1 -1 -1-1 -1 -1 -1 -1-1P XXVXXV VV XXVXXVVXXVXXVXXVXXVV which leads to

PAGE 57

57 2 2ij j i ji ij ij -1 -1 -1 -1 -1-1 -1 -1-1 -1 -1 -1 -1 -1 -1-1-1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1-1-1 -1 -1PVV XXVXXVVXXVXXVVXXVXXV VV XXVXXVVVXXVXXV V XXVXXVVXXVXXV VV XXVXXVVVXXVXXV ij ij ji ji i -1 -1 -1 -1 -1-1 -1 -1-1 -1 -1 -1 -1 -1 -1-1 -1 -1-1 -1 -1 -1 -1-1 -1 -1-1 -1 -1 -1 -1-1-1 -1 -1VV XXVXXVVXXVXXVVXXVXXV VV XXVXXVVXXVXXVV VV XXVXXVVXXVXXVV VV V XXVXXVVVXXVXXVV 2.j ij -1-1 -1 -1 -1 -1V V V XXVXXVV Now we have 2 2 2.k mkj miki mjkj mi ij ki mjk m ij km ij -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1P MGZVZGMFVI-PFFVI-PFFVI-PE V FVI-PEMXXVXXVVXXVXXVM V MXXVXXVVM To complete Equation (3-18) we also must investigate 21 ij V: 2 2 ijji ij ji ij -1 -1-1-1-1-1-1-1-1-1-1VVVVVVV VVVVVVVVVV.

PAGE 58

58 This gives us 2 2 2.km k m ij ij km k m ji ij kimjkjmikimjkjmik m ij -1 -1-1-1 -1-1-1-1-1-1 -1 -1 -1 -1 -1 -1VV V MGZI-PZGMMGZVVVI-PZGM VV V MGZVVVVI-PZGMMGZVVI-PZGM V EVE+EVE+EVF+EVFMGZVVI-PZGM Utilizing these expressions for 21 ij V and 2 ij Pin Equation (3-18) results in 22 22 2(4)kmk m ij ij km km ij ij km ij ki mjkj miki mjkj mi ki mjkj mi -1 -1-1 -1 -1 -1 -1 -1 -1 -1GG MMMZVI-PZGM GP MGZVI-PZMMGZVZGM V MGZI-PZGM -DVI-PD-DVI-PD-EVI-PD-EVI-PD -FVI-PD-FVI-PD 2 kimjkjmi ki mjkj miki mjkj mi ki mjkj miki mjkj mi ki mjkj mik m ij k -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1-DVI-PE-DVI-PE -DVI-PF-DVI-PFEVI-PEEVI-PE EVI-PFEVI-PFFVI-PEFVI-PE V FVI-PF+FVI-PFMGZVVI-PZGM M 2 2 m ij km ij -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1V GZXXVXXVVXXVXXVZGM V MGZXXVXXVVZGM(3-19) which reduces to 22 22(4) .kikiki mjmjmj kjkjkj kikiki ij ijfg -1 -1DE+FVI-PD+E+F D+E+FVI-PD+E+F GV (3-20)

PAGE 59

59 Piecing together (1) (4) in Equation (3-12) results in: 2 () 22cov, .kkmm kmij ij kikikiki mjmjmjmj kjkjkjkj mimimimi ijijfg -1 -1L MuL Mu CDE+FVI-PCD+E+F CD+E+FVI-PCD+E+F GV (3-21) When G and V are linear in the functions of 2 ijf Gand 2 ijg Vwill be zero. In that case 1111 11112 .kkkk km krkrkrkr mmmm mrmrmrmr 1CDEF CDEF VIPCDEFCDEF (3-22) Comparing Equation (3-22) to Equation (3-2) gives the desired result that in the most general case invo lving multiple dimension linear combinations of fixed and random effects KR BIAS KRC when G and V are linear in Thus, just as in the single dimension fixed effects linear combinations, when G and V are linear in KR KR 1M =M +2C is an estimator of the MSEP of the linear combinat ion of fixed and random effects that accounts for both the correction term and th e bias correction term. 3.2.3 Definiteness Properties of the Estima tor for MSEP for Mult iple-Dimension Linear Combinations of Fixed and Random Effects Now that all pieces of the estimator have been defined in the multiple dimension setting an estimator for the MSEP can be defined as

PAGE 60

60 KR KR KR1M M C BIAS (3-23) It has not been proven that th is MSEP estimator, either in single or multiple dimensions, is non-negative definite. Just as in the single dimension case, we have shown above that when the variance matrix for the model is linear in the bias correction estimator, KRBIAS reduces to the correction term, KRC which we have shown is non-negative definite in its most general form. Hence the simplified MSEP estimator, 2KR KR1M M C (3-24) is nonsingular and therefore a valid measure of prediction error. However, when the covariance matrix, V, is not linear in neither the bias correction term nor the MSEP estimator including the bias correction term has been previously show n to be non-negative definite. In Chapter 4, by exploring the special case of th e balanced one-way random effects model, we provide a counter example to prove that the bias correction is, in fa ct, indefinite. Indeed, we show that in a single dimension case the estimate of MSEP given in Equation (3-23) can be smaller than the nave estim ate, 1M From Equation (3-21) it is clear that the indefiniteness of KRBIAS is caused by 2ijf G and 2ijg V. Note that it is not nece ssary for the bias correction term to be non-negative definite for the overa ll estimator of the MSEP to be non-negative definite. In fact, it is reasonable for the bias correction term to be indefinite since it is estimating the bias of 1M which is also indefinite. In Chapte r 4, we provide a single dimension case where the negativity of the bias correct ion term causes the entire estimate of 1M to be

PAGE 61

61 negative, proving that KR1M BIAS the overall MSEP estimator for 1M is indefinite and a poor choice for estimating the MSEP. 3.3 Transform Invariance of MSEP Estimator Another important issue concerning these esti mators is transform invariance. Consider two parameterizations of a model, and g Kackar and Harville (1984) showed that the approximation for the correction term in the single dimension case, tr A B is transform invariant under the conditions that g and that 1 1g B J B J where B*and B are as in tr A B and J is the Jacobian of the transformation. Both of these co nditions are satisfied by using either REML or ANOVA variance component estimation and with 1 B I the inverse of the observed information matrix. This proof applies to the multiple dimension case, as we have shown that the multiple dimension correction term approximation is simply a matrix composed of the single dimension approximations as the components. Since each element of the matrix is transformation invariant, the entirety is transform invariant as well. It is also easy to show that the nave estimator of the MSEP, 1M is also transform invariant under the condition that g The invariance of the nave estimator indicates that the true bias of 1M E 11M M will also be transfor m invariant (again with the condition, g ). Thus a desirable property for the estimator of the bias correction term is transformation invariance. Clearly, KRBIAS is not transform invariant. In Chapter 4, we demonstrate the lack of transform invariance fo r a single dimension linear combination in the

PAGE 62

62 one way random effects model. This is an undesirable quality of th e bias correction term estimator. We will investigate its effects on the balanced one-way random effects model in Chapter 4.

PAGE 63

63 CHAPTER 4 BALANCED ONE-WAY RANDOM EFFECTS MODEL Many of the methods summarized in Chapter 2 were developed only for estimating and testing fixed effects (e.g., Gi esbrecht and Burns 1985; Kenward and Roger 1997); however, the methods are often extended to include random effects without documented justification (SAS 2003). In Chapter 3, the elements involved in ex tending these methods to the most general cases were derived. By looking at the simplest mode l including random effects, namely the balanced one-way random-effects model, we can examine if the use of these methods is as straightforward as previously claimed or if additional issues ar ise. The simplicity of this model often allows closed form expressions to be obtained for the approximations and estimators of the MSEP and their expected values, in addition to the true value of the MSEP being estimated. Peixoto and Harville (1986) derived closed form expressi ons for the true MSEP for a class of models including the balanced one-way ra ndom effects model. Similar methods are used here to enable direct comparisons of various MSEP estimators. By using this simple, balanced model, we can measure the result of including ra ndom effects on the MSEP estimates, as well as identify issues that may be more difficult to handle in more complex models. Consider the model: 22, 1,,, 1,, 0,, 0,ij iij ij i a y aeikjn eNaN (4-1) where 22, 0, 0a. We use the parameterization 2, where 2 22an and consider the problem of predicting the mean of the ith level of the random effect. Note that the variance matrix, V, is not linear in this para meterization, nor is this the parameterization used by SAS. This parameterization is chosen to demonstrate the impact of a

PAGE 64

64 non-linear parameterization on the bias correction term and for consistency with past work on this topic. Comparisons will be made to results under the altern ative parameterization, 22,a for which V is linear. The BLUP and the nave MSEP, 1M, for ita are straightforward to derive, as demonstrat ed in Kackar and Harville (1984, p. 857): 1 11 where and n k ij i j i iiy y yyy y nk (4-2) 2 11 11Mk nk (4-3) Noting that is linear in the accuracy of the Kackar-Harville approximation of the correction term depe nds on four issues: 1) using the exact MSE of versus an approximation, such as -1I for B ; 2) the appropriateness of the Taylor series approximation, depending in part on how close is to the boundary of ; 3) whether is unbiased for ; 4) the proximity of cov, dy; d y ; to zero. We can evaluate these four criteria to de termine their effects on the accuracy of the Kackar-Harville approximation. We shall examin e each of these criterion using both REML and ANOVA methods of variance component estimation. 4.1 Kackar-Harville Appr oximation Based on REML Estimation Method Using REML methods to estimate the variance components results in 2 where

PAGE 65

65 2 2 2 11 1 if (1)(1) if 1 min(/,1) where and (1)1eae ae ae ea kn k iji i ij i eammm kmknm mm nk mm yy nyy mm kn k (4-4) (e.g., Kackar and Harville 1984; Searle, Casella and McCulloch 1992). Note that these estimators remain within the parameter space, by definition. The EBLUP for ita is 1i y y (4-5) and the nave estimate of the MSEP of the EBLUP is 2 1 1 11 Mk nk (4-6) The true correction term in the REML case, as derived by Kackar and Harville (1984, p. 731), is 2 2 11 Rnk CE nk (4-7) where 1 1 1 2 2211 21 ,11 1121 I,1 2 1a ee eaaea a ae aa ea eaea eaB

PAGE 66

66 and 1 11 0()() ,, I,,( 1) () (1)1 22x ab x e ea eaab B ab abBabttdt ab knk This results in a true MSEP, RM where 1 2 1 11RRMMC nk M nk (4-8) (Kackar and Harville 1984, p. 857). In this model, the exact MSE matrix of is tractable when using REML estimates for the variance parameters. Thus assuming that is sufficiently far from the boundary of and choosing RB to be the exact MSE matrix of we can assess the accuracy of Rtr A B for RC by simply assessing the proximity of cov, dy; d y ; to zero. To evaluate Rtr A B first note that 2200 0 a A (4-9) and 2 22 22 2 22 REE EE B (4-10)

PAGE 67

67 where 2 221 varvarik ay y nk As shown by Kackar and Harville (1984, p. 731), 2 2 0E where 1 0 2 2 2211 21 ,1122 11 21 2 I, 121a eae eaaeaa a ae a ea eaaaB As a result, 2 2 0 220(1)Rk tr a nk A B (4-11) A Kackar-Harville approximation to the MSEP is then 2 0 1(1)k M nk (4-12) (Kackar and Harville 1984, p. 857). Now note that the only non-zero component of cov, dy; d y ; is 222222 cov,iiiyyEyyEyyE Kackar and Harville (1 984, p. 731) show that 2 22 11 iRnk EyyC nk

PAGE 68

68 It is also easy to verify that 2 221iak Eyy n nk since 2 1 221a k akm n Thus recalling that 2 2 0E and combining results, we have 22 22 10 22 22 1011 cov, 11 .i ankk yy n nk nk nk k nk nk (4-13) Notice that this is simply the difference be tween the true correcti on term in the REML case,RC and the approximation, Rtr A B This is only true when the exact MSE of is used for RB Thus the condition for RRCtr A B i.e., the condition for cov,0 dy; dy; is 1011 nkk (4-14) Figure 4-1 shows Equation (4-13) as a function of for several values of k, holding n = 6, with the condition in Equation (4-14) met when the functions cr oss th e zero line. Note that throughout, without loss of generality, we hold 21 As k increases, the functions move closer to zero, indica ting that the Kackar-Harville approximation for RC improves as k increases. However, as k increases, the amount of information about the random effect increases, reducing the uncertainty about the cova riance parameter estimates. Hence RC tends toward zero and accounting for the correction term b ecomes less important, rendering the correction term approximation irrelevant. In the remainder of this chapter, the analysis will focus on a model with k = 6 levels of the random effect to demonstr ate the impact of the different estimators for the MSEP.

PAGE 69

69 In most models, the exact MSE of is not tractable and an approximation of the MSE of must be used. A common choice is -1I the inverse of the observed information matrix, with replaced by in practice. Consider the Kackar-Harville approximation of RC where B is the asymptotic MSE of -1I Kackar and Harville (1984, p. 857) show that 2 22(1) (1) kn tr knn -1A I (4-15) where 2 2 2 2 22 2 11 21 2 111knkn kn knknk -1I (4-16) Figure 4-2 compares the two Kackar-Harville approximations in Equations (4-11) and (4-15) with the co rrection term,RC for k = 6 and n = 6. The success of the approximation is greatly influenced by the value of For small values of which correspond to large values of 2a relative to 2 tr -1A I is superior to Rtr A B for approximating the true correction term, RC However, as moves toward 1, which corresponds to the value of 2 a growing relatively smaller, the roles reverse and Rtr A B becomes the better approximation. In some cases, the choice of how to measure the variability of may cause severe over-estimation of the correction term, in turn causing a greatly inflated value for the MSEP estimate. SAS PROC MIXED uses tr -1A I which overestimates the true

PAGE 70

70 correction term by over 100% for half of the parameter space. Using an inflated estimate for the MSEP could lead to overly conservative interpretation. Because the correction term approximations in Equations (4-11) and (4-15) make use of the unknown variance param eters, estimates of the va riance parameters must be substituted into these approximations. Note that there are now two levels of error being introduced into the correction term estimators: first, the true corr ection term is approximated by the Taylor series methods to yield the Kackar-Harville approximati on; second, the variance parameters involved in the approximations are replaced by their estimators. In the case of tr -1A I there is a third contribution to the error in that -1I is an approximation of the MSE of Substituting the REML estimates of the variance components into Equations (4-11) and (4-15) gives two possible estim ators for the correction term: 2 0 (1) Rk tr nk A B (4-17) and 2 2 2(1) (1) kn tr knn -1A I (4-18) It is suggested by Prasad and Rao (1990) that the bias introduced by substituting variance parameters in these approximations is 1ok. To assess this assertion, we investigate the accuracy of Equations (4-17) and (4-18) by considering the expected values of both estim ators. The expected value of Equation (4-17) is not available in closed for m, so a simple Monte Carlo study was performed to evaluate the accuracy of the estimator for the one-way random effects model in Equation (4-1) with k = 6 and n = 6. Using the RANNOR function in SAS, 10,000

PAGE 71

71 independent sets of eij and ai were generated from 2(0,1)N and 20,aN for several values of 2a respectively. The values of 2a correspond to 0.05, 0.1, 0.15, 1.0. Setting 0 10,000 sets of :1,6,16ijyij were created from the model ijiij y ae. The Monte Carlo value of 2 0 (1) Rk Etr E nk A B and the Monte Carlo standard error were obtained from the 10,000 replicates. Selected results of the study are contained in Table 4-1 (standard errors contained in parentheses) along with the values of RC (Equation (4-7)) and Rtr A B (Equation (4-11)). Table 4-1. Accuracy of Kackar-Harville estim ator (4-17) for RC 0.2 0.4 0.6 0.8 1.0 RC .01537 .01890 .01758 .01660 .01850 Rtr A B .04447 .03384 .01950 .01097 .00954 **REtr A B .03567 (.00013) .02652 (.00013) .01951 (.00011) .01513 (.00009) .00954 (.00002) **Monte Carlo value. Standard error of simulation in parentheses. Figure 4-3 shows the results of the Monte Ca rlo study graphically. The accuracy of the estimator again largely depends on the value of For small values of which correspond to values of 2a that are large relative to2 the estimator for the correction term is not accurate with expected values more than twice th e function it is estimating. However, as approaches 1, the estimator for the correcti on term significantly underestimat es the true correction term.

PAGE 72

72 The expected value of Equation (4-18) can be expressed in closed form if we assume norm al distributions. First define 222(1)(1) eae eae a ae aknmkm u uuw u u s nu u It is easy to show that ue and ua are independent chi-squa re random variables with k(n-1) and (k1) degrees of freedom, respectivel y (e.g., Searle 1971, p. 410). Also, w and s are independently distributed where w has a chi-square distribution with (kn-1) degrees of freedom, and s has a Beta(e, a) distribution (e.g., Johnson and Kotz 1970, Sec. 24.2). Now note that if ma > me then 1e am m so that min,1ee aamm mm but when ma < me, 1e am m so that min,11e am m Defining 1 if 0 if ae ae aemm mm mm gives 2 2 2 2 1 221 (1) 21 1(1) ,1 (1) 1ae e ae ae akn Etr E knn kn kmknm m EmmE mm knnm kn -1A I (4-19) Noting that if ma > me then s < so that 1 if 1 if ,, 0 if 0 if ae ae e aemm s sm m s mm and that 22 2(1)(1)ea aem ws mkns we see that

PAGE 73

73 22 2 2 2 1 1 2 2 0 21, (1)1 (1)1 1 (1) (1)1(,) (2,1) (1) ( (1) (,)a eea ae ae a e a ee a ae a ee am ws EmmEs mk ns s EwEs kn s ss s kn ds kn sB B kn I kn B 2,1).ea (4-20) Now note that 2 so (1)eesw uswm kn and 221 (1)a answ m k which leads to 22 21(1) 1 21 ,1 11 (1) 1, 1 11, 11ae a ae ae e ae akmknm km Em m Em m kn kn knm Em m kn n Ewss Ews kn kn 11 11 11 22 2 22 2 22 21, 11 1 (,)(,) (,1)(1,) 1(,1)1(1,) (,)(,) 1(,1)1(1,).aa eea eaea eaea aeaea eaea ae aeaea eaeas ssss nsdssds BB BB nII BB nII (4-21) Substituting Equations (4-20) and (4-21) for (1) and (2) in Equation (4-19), respectively, we have 2 2 2 2 22 121 (1) 21 (1)(2,1)(2,1) (1) (1)(,) 1(1,) 1(,1).aeaea eea ea ea ea ea eakn Etr E knn kn knB I knn knB II -1A I (4-22)

PAGE 74

74 Figure 4-4 shows the accuracy of th e Kackar-Harville estimator (Equation (4-22)) for the Kackar-Harville approxim ation (Equation (4-15)) and the correction term (Equation (4-7)) holding k = 6 and n = 6. Just as for the approxim ation, the accuracy of Etr -1A I depends largely on the value of While tr -1A I is a fairly accurate estimator for tr -1A I over the entire parameter space, it may severely over-estimate RC for larger values of We wish to assess the asserti on of Prasad and Rao (1990) that 1 E tr tr ok -1 -1A I A I for this model. Noting that lim1k yields the result that lim1kI so that 2 2 2122 1 21 (1)(2,1)(2,1) 1 (1) (1)(,) 1(1,) 1(,1)aeaea eea ea ea ea ea eaEtr tr kn knB I knn knB II ok -1 -1A I A I confirming, for this example, the assertion of Prasad and Rao (1990). These results also show the importance of studying the effect of substituting variance parameter estimates for the unknown values in th e approximations for th e correction term, the impact of which has been typically overlooked. In the case of Rtr A B substituting variance parameter estimates improves the accuracy for RC (in the long run). However, in the case of tr -1A I the substitution creates more inaccuracy for RC for most values of

PAGE 75

75 Figure 4-5 compares the accuracy of Equations (4-17) and (4-18) for the correction term. The plot shows that Rtr A B is more accurate (in the long-run) than tr -1A I for > 0.30; however, for < 0.30, the reverse is true. This is the same phenomenon that we noticed with the approximations in Figure 4-2. Note that SAS PROC MIXED by default uses -1I with REML estimates for in the calculation of the MSEP estimator. 4.2 Kackar-Harville Approximation Ba sed on ANOVA Estimation Methods The best cho ice of method for estimating may depend on the model of interest, the level of balance in the design, and the goals of the an alysis. Stroup and Littell (2002) demonstrate the effect of different estimation methods on the pow er and control of type I error for hypothesis tests on the fixed effects in analyzing an unba lanced, multi-location experiment. Their study indicates that REML and ML ar e not always the best choices when negative estimates of variance components are likely, and may lead to inflated Type I error rates and low power. The impact of the choice of variance component estimation method and negative variance component estimates on the Kackar-Harville approximation has not been investig ated thoroughly. In Chapter 5 we elaborate on the impact of nega tive variance component estimates. To begin, however, we develop the correction term, several approximations, and the expected values of the estimators under ANOVA estimation for the balanced one-way random effects model. The ANOVA estimators of 2, are 2, where

PAGE 76

76 2 11 2 2 22 2 2 1, (1) = 1kn iji ij e e aa k i aei aayy m kn m nm nyy mm m nk (4-23) These estimates are obtained under the METHOD = TYPE 3 option in SAS PROC MIXED. The true correction term, CA( ), is derived under normality assumptions as follows: 22 22 1 1 22 22 22 22 2 22 22 2 2 22 2() (1) (1) (1)1 (1) (1k Ai i i aa a a a aa aa eCEyyknEnyy nkmn EEu knnkn nn EwsEws knkn ns Ews kn s 22 22 2 2 21( 1 ) 1 )( 1 ) (1) (1)1, (1)aa e a ens EwEs kn s s kn Es kn s by independence of w and s and noting that 2 1knw Recalling that ,easBeta we find that 2 22 2 2 2 2 2(1)1 2(1) (1) (1) 2,1 2 2,1 ,aa a eee aea aa eeaeaea aea a eeaeasss Es E s ss B B B B

PAGE 77

77 Thus the correction term is 2 2 2 2 2 22,1 (1) (1) 1 211 13aea a A eeaea a eaB kn C knB kn kn knk knnk (4-24) This gives the true MSEP for 1 2 1 2211 13AAMMC knk M knnk (4-25) To evaluate the Kackar-Harville approxima tion for the correction term, we need to determine how to evaluate B We consider thre e alternatives for B : the exact MSE matrix E the variance-covariance matrix of and -1I Recall from Equation (4-9) that due to the form of A in this model we are only concerned with assessing the variability of in order to evaluate the Kackar-H arville approximation to the correction term. We begin with the exact MSE of 2E Noting that (1),1 e knk am F m it is easily shown that 1 3 k E k 22 22(1)(3) (1)(3)(5) kkn V knkk and 2 2 2112 135 kkn E knkk After algebraic simplification this leads to

PAGE 78

78 2 22 22 351 2. 135EEE kknn knkk (4-26) Letting AE B we have the following approximation forAC: 2 221351 135Akkknn tr knnkk A B (4-27) Substituting the ANOVA estimates in Equation (4-23) for yields this estimator for AC: 2 221351 135Akkknn tr knnkk A B (4-28) To assess the accuracy of Equation (4-28) for estimating Equation (4-27) and AC, we evaluate the expected value. Recalling the relationship 22 2(1)(1)ea aem ws mkns where w and s are independently distributed with2 1knw and ,easBeta we have 22 2 2 2 2(1)(1) 2,1 1 (1), 112 13ea ae ea a ee am s EEEwE mkn s B kn kn B kkn knk (4-29) Thus, 2 2 22 32135112 135Akkknnkn Etr knnkk A B (4-30)

PAGE 79

79 Again, we confirm the asymptotic assert ion by Prasad and Rao (1990), noting that 2 22 22 2 3 2 2 1213511221351 135 135 21351112 1 13513 .AAEtr tr kkknnknkkknn knnkk knnkk kkknnkkn knnkkknk ok A B A B The second alternative for B is the exact variance-covariance matrix of *AB Because is not unbiased for this is not the same as the MSE matrix for As noted above, 22 22(1)(3) (1)(3)(5) kkn V knkk which leads to the following approximation for AC: 3 *2 2 2213 135Akkn tr knnkk A B (4-31) Again substituting the ANOVA estimates for in Equation (4-31) yields an estimator for AC: 3 *2 2 2213 135Akkn tr knnkk A B (4-32) Using Equation (4-29) we have 4 *2 23 321312 135Akknkn Etr knnkk A B (4-33) Again we find that

PAGE 80

80 ** 4 3 22 23 2 32 3 2 2 2 121312 213 135135 112 213 1 13 135 .AAEtr tr kknkn kkn knnkk knnkk kkn kkn knk knnkk ok A B A B The third alternative we consider for B is to use the inverse of the observed information matrix, -1I as an approximation. Note that -1I is the same as the asymptotic variance of the REML estimators of in Equation (4-16). Thus the Kackar-Harville approxim ation with ANOVA variance component estimates using -1I is the same as in the REML case in Equation (4-15), i.e., 2 22(1) (1)kn tr knn -1A I (4-34) and the estimator formed by substituting ANOVA estimates for is 2 22(1) (1)kn tr knn -1A I (4-35) Using Equation (4-29) we have 2 2 2 322(1) (1) 2(1)112 (1)3 kn Etr E knn knkkn knnk -1A I (4-36) which again yields

PAGE 81

81 22 322 2 2 12(1)112 2(1) (1)3 (1) 112 2(1) 1 (1)13 Etr tr knkkn kn knnk knn kkn kn knnknk ok -1 -1A I A I Figure 4-6 compares the three Kacka r-Harville approximations in Equations (4-27), (4-31) and (4-34) to AC for k = 6 and n = 6. Clearly tr -1A I more closely approximates AC than the other two approxi mations, for all values of Thus, although we would expect -1I to be less accurate than AB or *AB in evaluating the MSE of especially for small values of k, the approximation of AC utilizing -1I is more accurate than the either of the other choices across the entire parameter space. In fact, the KackarHarville approximation utilizing the exact MSE of is the least accurate of all three approximations. Note that this is in contrast to the results from REML variance component estimates in Figure 4-2, where the best approximation depended on the value of Figure 4-7 compares AEtr A B (Equation (4-30)) to Atr A B and the correction term, AC The bias for the correction te rm introduced by replacing the unknown variance components with ANOVA es timates in the correction term approximation is significant and increases as moves toward 1. Figure 4-8 demonstr ates a similar result for the KackarHarville approximation and estimat or utilizing the variance of AEtr A B (Equation (4-33)) is again inflated over the approx im ation and they both overestimate the true

PAGE 82

82 correction term increasingly in Figure 4-9 shows a much diffe rent result than the previous two figures since the accuracy of the estimator utilizing -1I (Equation (4-36)) is improved over that of the approxim ation (Equation (4-34)). While tr -1A I underestimates AC by up to 50%, Etr -1A I is much more accurate and consistently overestimates AC by only 6.67% across the entire parameter space. Figure 4-10 compares all three estimators (Equations (4-30), (4-33), and (4-36)) relative to AC Clearly the Kackar-Harville estimator utilizing -1I is far more accurate than either of the other two choices. This is counter -intuitive: the Kackar-Harville estimator using an approximation of the MSE of outperforms the other estimators using exact measures of the variation of Note again that this is a different result than seen with REML variance component estimates where the results were dependent on the value of There, we saw that the Kackar-Harville estimato r utilizing the exact MSE of RB often performed better than the one using an approxi mation of the MSE of -1I (see Figure 4-5). The choice of which Kackar-Harville estimator for th e correction term is best, theref ore, depends, in part, on what method is used for variance component estimation. The final step in this assessment is to co mpare the ANOVA results to the REML results. Figure 4-11 compares the accuracy of tr -1A I (Equation (4-36)) for AC with the accuracy of the two REML estimators, Rtr A B (Monte Carlo expected value) and tr -1A I (Equation (4-19)) for RC with k = 6 and n = 6. The ANOVA estimator

PAGE 83

83 utilizing the information matrix, tr -1A I appears to perform better than either REML based estimator, although the a dded variation to the MSEP of is larger for the EBLUP with ANOVA variance components, i.e., AC > RC We can assess the accu racy relative to the appropriate correction term to better compare th e performance of the estimators. The relative bias of the two REML based estimators of the correction term and the ANOVA based estimator with -1I is tabulated in Table 4-2 and graphically depicted in Figu re 4-12. Relative bias was calculated as EtrC C AB The ANOVA estimator of the correction term has consistently lower relative bias than either of the REML estimators. Thus, although the variation added to the MSEP by using ANOVA variance component estimates is la rger than when using REML variance component estimates, a more consistently accurate es timate of the added variation is available in the ANOVA case. While the accuracy of both REML estim ators depends largely on the value of the ANOVA estimator is significantly more accurate across all values of This special case has demonstrated the im portance of the choice of variance component estimation method, the choice of measure of variation of and the added variability of substituting parameter estimates for unknown va riance components in the correction term estimates. While ANOVA methods add more variability to the EBLUP than REML estimation, the estimator for the MSEP of the EBLUP is more accurate with ANOVA methods than with REML methods. The best choice of how to measure the variability of was not consistent across variance component estimation methods. For ANOVA methods, a better choice for B

PAGE 84

84 was -1I while for REML estimation the bette r choice depended on the value of We also discovered that the effect of substituting parame ter estimates depends, in part, on the choice for B and the variance component estimation method. The impact of this substitution is often considerable and could cause the MSEP es timate to be significantly inflated. Table 4-2. Relative bias (%) of estimators of correction term 0.2 0.4 0.6 0.8 1.0 REML** REtr A B 132.0 40.3 11.0 -8.8 -48.5 REML Etr -1A I 33.7 85.0 152.3 202.8 220.8 ANOVA Etr -1A I 6.7 6.7 6.7 6.7 6.7 **Monte Carlo values used to compute relative bias. 4.3 Bias Correction Term for the Ba la nced One-Way Random Effects Model 4.3.1 Bias Correction Term Approximation under Parameteriz ation We now address the bias co rrection term approximation for the one-way random effects model. We will consider the case where -1B I so that the calculations for the approximation will be the same for the REML a nd ANOVA, and so that closed-form expressions are attainable. The difference comes in the ex pected values of the estimators of the bias correction term due to the truncation of the REML estimates. We will show that in either case the bias correction estimator ca n be negative, thus proving by counter-example that the bias

PAGE 85

85 correction estimator is indefinite We begin however, with demons trating the lack of transform invariance. Recall that the parameteriza tion we are working with is 2, If we were considering the parameterization 22,a wherein the variance matrix, V, is linear in the parameters, the bias correction term would be identical to the correction term approximation, which is transform invariant, i.e., 1 2tr tr tr B A B A B and non-negativity would not be an issue since tr A B is non-negative definite in its most general definition as proven in Chapte r 3. This is not the case with as the parameterization, as will be demonstrated in this chapter, since 1 2tr tr B A B We now give a simple demonstration that the bias correction term approximation is not transform invariant, as discusse d in Section 3.3, through the con tinuing example of the balanced one-way random effects model. The true bias of the nave MSEP, E 11M M is invariant to transformation as long as This follows from the definition of 2Et 1M Thus the lack of transform invariance of the bias estimate is a concern. To calculate the bias correction term approximation under the parameterization, we first compute the second deri vatives of the nave MSEP, 1M with respect to the variance parameters: 22 1 2 2 1 2 212 0,M kk nk M

PAGE 86

86 and 22 2 2 1 2 22 212 221 11 kk kk M nk nk k nk 2 2 1 2 32 ,M nk and 22 1 2211Mk nk So, 2 2 2 2 2311 0 11 2k nk k nknk (4-37) Utilizing -1I from Equation (4-16), we have 22 2 2 22 2 2 22211 1 211 41 111k f nkn k kn f nknnknk -1 I (4-38) where 2,f is a function of 2, that is inconsequential in this calculation. This gives 2 2 2211 1 4. 111k kn tr nknnknk -1 I (4-39) Thus the bias correction term for the nave estimator of the MSEP under the 2, parameterization is

PAGE 87

87 2 2 2211 11 2 21 1 1KRk kn tr nknnknk -1BIAS I (4-40) which is obviously different from the bias correction term approximation used under the parameterization, which would be identical to Equation (4-15). Clearly the lack of transform invariance is attributable to the higher order derivatives in We will examine the impact of the parameterization by comparing each bias co rrection estimate to the true bias. 4.3.2 Expected Value of KRBIAS with REML Covariance Parameter Estimation We begin by calculating the expected value of the bias correction term estimator once the REML estimates are substituted for the unknown variance components. Recall from Equations (4-19), (4-20), and (4-21) that 2 2 221(1)(2,1)(2,1) (1)(,) 1(1,) 1(,1).aeaea eea ea ea ea ea eaknB I E knB II (4-41) We also need 2 1 21(1) ,1 1ae aae aekmknm EEmmmE mm kn (4-42) where 22 1 1 21 0 2111, 1 1 1 1 1(,) (,1)a ea ea ean Ewss k ss kn sd s kB I and from Equation (4-21),

PAGE 88

88 22 221 ( 1 )1 ( 1 )ae ae a e a ea eanI I We can utilize Equations (4-41) and (4-42) to compute th e expected valu e of KRBIAS as follows: 2 2 22 22 1 2 2 21 2 11 1 2 111 12 2 11 (1)(2,1)(2,1) 1 2 1( 1 )KR aeaea eEEt r k kn E nknnknk k EE nkn nkk knB I k nkn kn -1BIAS I 22 1 21 22 2(,) 1(1,) 1(,1) 1 (,1) 1(,1) 1 1(1,).ea ea ea ea ea ea a ea a ea ea e ea eaB II InI nkk I (4-43) To evaluate the accuracy of the bias corre ction term estimator, the true bias correction term (which is the negative true bias) of 1M for 1M can also be calculated as 11 MEM for REML variance co mponent estimates of Because 1 0M when aemm the expected value of 1M is

PAGE 89

89 2 1 2 2 1 11 1 11, 21 ,.ae ee a ae aEME k nk Ekm m nk kmkm m Em m nknknkm (4-44) We establish that 21,( ,1 ) ,a ae eam EmmI nk nk (4-45) 2 2 2 2 2 2 222 ,, 1 21 1 21 (1,) (1,) 1(,) 2 (1,),e ae ea ea ea eakmk Em mEs ws nk nkn kkn Ess nkn kkn B I nknB k I nk (4-46) and 2 2 2 2 21 1 ,( 1 ) (1) (2,1) (2,1) (,) 12 (2,1). 13e a ae ae ea ea ea eakm k Em mk n nkm nkkn B I B kknk I nknk (4-47) Combining Equations (4-45), (4-46), and (4-47) into Equation (4-44) gives us the expected value for 1M with REML variance component estimates:

PAGE 90

90 2 21 1 2 2 22 (,1) (1,) 12 (2,1). 13ea ea eak EMII nknk kknk I nknk (4-48) Thus the true bias correction term of 1M for 1M is 2 11 2 11 11 12 (2,1) 13 (,1)2(1,).ea eaeaMEM k nk kknk I knk IkI (4-49) We can now make comparisons among the proposed bias correcti on term estimators of the nave MSEP, 1M Figure 4-13 pictorially demonstrates the bias of 1M for 1M under REML methods. From this figure it is easy to see that 1M has both positive and negative bias depending on the value of Figure 4-14 depicts the tr ue bias correction term, 11 MEM from Equation (4-49), along with the bias correction term approximation, 1 2tr -1 I (Equation (4-40)) and its expected valu e under REML estim ation (Equation (4-43)). First note that the effect of substi tuting varian ce parameter estimates in the bias correction approximation does not have much of an impact, unlike the results of the correction term approximation. Although 1 2tr -1 I and 1 2Etr -1 I are nearly equal over the entire span of both significantly underestimate the true bias correction term over most of the parameter space. In fact, the true bias correction term is positive for 0.75 reflecting that 1M is underestimating 1M However, the bias approximation and

PAGE 91

91 expected value are negative. Thus by using th is bias correction term estimator, we would actually exacerbate the problem and si gnificantly increase the amount by which 1M is already underestimating 1M That is, while 1M tends to underestimate 1M for 0.75 the correction by the bias correction term actually causes further underestimation of 1M especially for small values of The use of this form of the bias correction term would actually produce a poorer estimate for 1M than using 1M alone. As increases, we see that the true bias correction term becomes negative, reflecting that 1M is overestimating 1M ; thus it is appropriate for the bias correction term estimator to be negative. As shown in Figure 4-14, the bias correction term approxima tion performs significantly better in this upper range of the parameter space than for .75 Now we compare this result to the bias adjustment under the lin ear parameterization, 22,a recalling that 1 2tr tr tr -1 -1 -1 I A I A I where tr-1A I is in Equation (4-15). Figure 4-15 shows 11 MEM along with tr -1A I (Equation (4-15)) and Etr -1A I (Equation (4-22)). We see a much different result for this bias correction term than we did for 1 2tr -1 I Now we see the bias correction term approximation and estimator significantly overestimating the true bias correction term, especially for large values of For small values of the estimator is fairly accurate. This is the exact opposite of

PAGE 92

92 the previous result in Figure 4-14. Figure 4-16 compares the accuracy of 1 2tr -1 I and tr -1A I for 11 MEM For .75 11 MEM is positive. In this range, tr -1A I is a significantly more accurate estimator because 1 2Etr -1 I is negative. For .75 11 MEM is negative, and 1 2tr -1 I is significantly more accurate than tr -1A I Thus the choice of which estimator is better depends on the value of and also on whether it is better to overor under-estimate 1M Note that SAS uses the parameterization in the DDFM = KR option of PROC MIXED. Hence the results from Figure 4-15 are a pplicable to SAS procedures. 4.3.3 Expected Value of KRBIAS with ANOVA Covariance Parameter Estimation We can do this same comparison for ANOVA variance parameter estim ates. Recall that the bias correction approximation in Equation (4-40) is the same under ANOVA and REML varian ce parameters due to the choice of -1I However, because the ANOVA variance component estimates are not truncated, the expected values of the bias correction term estimator, 1 2Etr -1 I and the nave MSEP, 1EM will differ from the REML case. First, recall from Equation (4-29) that 22112 13kkn E knk (4-50) and note that 21 21211a au EEmE k (4-51)

PAGE 93

93 Using Equations (4-50) and (4-51), the expected value for the bias correction term estimator is 2 2 22 22 1 21 2 11 1 2 111 12 2 11 2KREEt r k kn E nknnknk k EE nkn nkk k -1BIAS I 2 2 21112 1 113 2 1 kkn nknknk nkk (4-52) Note that by allowing the variance parameter estimates to take values outside of the parameter space, 1M does not maintain the non-negative defini te property. That is, in taking the expected value of 1M with ANOVA variance parameter estimates, we are including negative values of 1M Allowing variance parameter estimates outside of the parameter space in expected value calculations does not affect the definiteness properties of th e bias correction term, 1 2tr -1 I since it is indefinite as will be discu ssed in Section 4.3.4. The calculation for the expected value of 1M proceeds as follows: 2 1 2 2 22 21 2 1 11 21 212 13ee a aEME k nk kmkm m E nknknkm kkknk nknknknk (4-53)

PAGE 94

94 Thus the true bias correction term (negative bias) of 1M for 1M is 2 11 2 22 21 2 2 21 11 212 13 211 13 MEM k nk kkknk nknknknk kkn nknk (4-54) We can now compare this to the bias correction approximation, 1 2tr -1 I and the expected value of the bias correction estimator, 1 2Etr -1 I ,as well as tr -1A I and Etr -1A I Figure 4-17 shows 1M and 1EM We see that 1EM consistently underestimates 1M over the entire parameter space, growing more severe as approaches 1. Figure 4-18 compares the tr ue bias correction term, 11MEM to 1 2tr -1 I and 1 2Etr -1 I We see a very similar resu lt as in Figure 4-14 for REML methods. Again, while 1M underestimates 1M adding 1 2tr -1 I as a bias correction would increase the amount by which 1M is underestimated. The bias correction term improves as increases but never becomes positive, and therefore never improves the unadjusted estimator, 1M

PAGE 95

95 Again we want to compare this to the bi as correction under the linear parameterization 22,a Figure 4-19 shows 11MEM along with tr -1A I (Equation (4-15)) and Etr -1A I (Equation (4-36)). The result is much improved in this param eterization. We can see that tr -1A I is a highly accurate estimator for the true bias correction term, 11MEM and using this estimator would improve the estimator for 1M over the entire parameter space. That is, 1Mtr -1 A I is a more accurate estimator of 1M than 1M Figure 4-20 compares the accuracy of tr -1A I and 1 2tr -1 I for 11MEM Because 11MEM is positive over the entire parameter space, 1 2tr -1 I is an unacceptable estimator of 11MEM However, tr -1A I is a highly accurate estimator for 11MEM with nearly constant relative bias of 6.67%. 4.3.4 Impact of Negative Values of Bias Correction Term Approximation As mentioned in Section 3.2.3 the bias correction approximation given by Equation (3-10) in indefinite. The balanced one-way random eff ects model under the 2, parameterization provides a sufficient example. Figure 4-15, shows that 1 2tr -1 I takes negative values ov er the entire range of We can determine from Equation (4-40) when the bias correction term estimator will be negative in terms of k and n. First rewrite Equation (4-40) as

PAGE 96

96 2 2 22 2 22 211 11 2 2111 211 11k kn tr nknnknk kkn nknk -1 I Noting that the denominator is always positive when is in its parameter space, we determine that the bias correction term estimator in Equation (4-40) is negative when 21 1kn k For k = 6 and n = 6, as in our example, this value is 1.09 which means that the bias correction term estimator is negative over the entire (0, 1] parameter space for This counterexample suffices to prove that the bias correction term approximati on in its most general form, 11 1 11 2l KR ll ltr tr tr tr B B BIAS B B from Equations (3-10) and (3-11), is indefinite. This in itself is not necessarily prob lematic since the true bias correction term, 11 MEM is also indefinite. However, if the magnitude of this piece is large enough to cause the entire MSEP estimator to be indefinite, then problems arise. It is beyond the scope of this dissertation to determine if the MSEP estimator utilizing the correction term estimator and the bias correction term estimator, KR KR1M C BIAS is non-negative definite. However, it can be shown that as an estimator for 1M KR1M BIAS (4-55) is indefinite by looking at the sa me special case of the balanced one-way random effects model.

PAGE 97

97 The form of Equation (4-55) for this model is 2 1 2 2 22 2 222 2 1 11 11 1 2 111 12(1)(1)(1)(2)(1)3(1) (1)(1)KRMk nk k kn nknnknk kknknkkknkn nknk BIAS (4-56) To determine when Equation (4-56) is negative, we find the r oots of the num era tor (noting that the denominator is greater than zero for k, n > 1) by solving the quadratic equation in This yields the following result: 1 0KRM BIAS where 2 2 2 2 2 2(1)(1)(1) 3(1)(1)(1)(1)(2) (1)(2(1)) 2(1)(2(1)) 4(1)(2(1))knkk knknknkk kkn kkn kkn (4-57) or 2 2 2 2 2 2(1)(1)(1) 3(1)(1)(1)(1)(2) (1)(2(1)) 2(1)(2(1)) 4(1)(2(1))knkk knknknkk kkn kkn kkn For k = 6 and n = 6 this gives 0.987 and 0.130 While the lower bound on is outside the parameter space, the upper bound is within the parameter space and a valid result for the parameter estimate. In the case of k = 3 and n = 3 the result is 0.75 and 0 Again, the lower bound is outside of the parameter space but the upper bound is well within the parameter space. This demonstrates the danger in using 1KRM BIAS as an estimator for 1M or M and serves as a general warning of the possible effect of the indefiniteness of

PAGE 98

98 KRBIAS The simplicity of this example raises even more cause for concern in more complex models. 4.4 Performance of Overall Estimators for the MSEP Now that all the pieces of the MSEP estimators have been evaluated, we can combine these results to get an overall picture for how accurate each MSEP estim ator is for the true MSEP. Recall that the true MSEP values will differ for REML and ANOVA variance components, since each varian ce component estimation method a dds a different amount of uncertainty to the MSEP of the EBLUP and a different amount of bias to 1M For REML methods, we will look at MSEP estimators utilizing both RB and -1I in the correction term estimator since there was no clear pref erence (see Figure 4-5). However, for ANOVA methods, the estimators using -1I were clearly superior to other choices of B and thus we will limit our comparison to -1I for ANOVA methods. Thus th e following equations list the possible estimators for the MSEP of the EBLUP of ita For REML variance parameter estimates we have three options: 1 2,RMtr A B (4-58) 1 2, Mtr -1 A I (4-59) and 11 2Mtrtr -1 -1 A I I (4-60) For ANOVA variance parameter estimates we have 2 choices for MSEP estimator:

PAGE 99

99 12, Mtr -1 A I (4-61) and 11 2Mtrtr -1 -1 A I I (4-62) Note that while Equations (4-58), (4-59), and (4-61) are derived under the 22,a parameterization, due to the translation invariance of the elements involved, for simplicity we will use notation. We begin with the REML comparison. Figure 4-21 shows the true MSEP for the EBLUP of ita using REML variance parameter estimates, RM along with the expected values of the three options in equations (4-58), (4-59), and (4-60). For completeness, 1 EM is included as well. As we ha ve seen throughout with REML va riance parameter estimates, the results are dependent on the values of For small values of Equation (4-59) is most accurate, almost m atching the true MSEP exactly. As grows, though, both Equation (4-58) and (4-60) become more accurate than Equation (4-59). Note that SAS uses Equation (4-59) as the es timator of the MSEP in the DDFM = KR op tion of PROC MIXED, which is not the best choice over a large range of It is especially problematic for close to one, which again corresponds to values of 2a small relative to 2 Now looking at ANOVA comparisons in Figure 4-22, we see that Equation (4-61) is most accurate for the tru e MSEP, AM over the entire range of It is clearly the best choice for MSEP estimator when ANOVA methods are used. We also want to determine if REML or ANOVA methods are preferred when comparing MSEP estimators. To do so, we look at the rela tive bias of each of the five MSEP estimators

PAGE 100

100 listed above for their respective true MSEP values Table 4-3 summarizes the results for select values of and Figure 4-23 illustrates this comparison. The table and figure clearly shows that the MSEP estimator with the smallest relative bias over the entire range of is 12 Mtr -1 A I which uses ANOVA methods. Thus for accuracy of the MSEP estimator, ANOVA variance component estimation under the parameterization with MSEP estimators utilizing -1B =I are the best choice among the ones compared here. Table 4-3. Relative bias (%) of MSEP estimators under REML and ANOVA estimation 0.2 0.4 0.6 0.8 1 1M -13.28 -26.27 -30.88 -22.76 37.38 1 2* *RMtr A B 16.75 10.59 11.71 36.21 140.50 1 2 Mtr -1 A I 4.03 22.33 65.91 173.14 620.57 11 2Mtrtr -1 -1 A I I -26.84 -18.26 2.72 57.93 294.72 1M -17.72 -51.38 -93.33 -142.68 -200.0012 Mtr -1 A I 1.19 3.42 6.24 9.51 13.35 11 2Mtrtr -1 -1 A I I -29.70 -36.58 -49.77 -67.18 -88.37 **Monte Carlo values used to compute relative bias. It is important to note that while th is is the MSEP estimator used by SAS in the DDFM = KR option when ANOVA methods are chosen w ith this model, the MSEP estimates are truncated at zero which changes the expected value of the MSEP estimator from the result given here. The disadvantage of using ANOVA methods is the likelihood of negative variance parameter estimates. However, by truncating the MSEP estimates, SAS negates the advantage

PAGE 101

101 of increased accuracy associated with ANOVA methods. The implication of allowing untruncated MSEP values is that negative estimates of the MSEP are possible. The impact of negative variance parameter estimates on EBLUP procedures in the balanced one-way random effects model are further explored in Chapter 5. 4.5 Comparison of Prediction Interval Methods Several methods for conducting hypothesis tests in linear mixed models are reviewed in Chapter 2. Table 2-1 contains a summary of the methods, including how the distribution is determined and which standard error is used in th e test statistic. We now want to examine these methods in terms of the balanced one-way random effects model. Because we are estimating the realized value of a random eff ect, it is more appropriate to ex amine the methods in terms of prediction intervals. The goal of this examination is to determin e the accuracy of the prediction interval estimation methods and determine where further investigation is necessary. Four prediction intervals will be compared in a Monte Carlo simulation study to determine which method has the most accurate coverage rate. All of the prediction in tervals have a similar structure. The estimate of the realized value of the random effect, 1a, is ,the EBLUP for 1a, as in Equation (4-2). This is consistent in all four methods. The structure for the prediction inte rvals is 1/2, .dfMSEPt (4-63) The estimate of M SEP and the degrees of freedom for the t-distribution depend on the method used to calculate the pr ediction interval. The method s used to determine the MSEP estimate and the degrees of freedom are summarized in Table 4-4.

PAGE 102

102 Table 4-4. Summary of pred iction interval procedures. GB KR FC FC MSEP 1M 1 2 M tr A B 1 2 M tr A B 1 1 2 M tr tr A B B DF 22 varGB GB GBdf MSEP MSEP K RGBdfdf 22 varFC FC FCdf MSEP MSEP 22 varFC FC FCdf MSEP MSEP Note: var B MSEP hh where MSEP h and 1BI. Note that all of the methods use a Sattert hwaite-type method for estimating the degrees of freedom. The degrees of fr eedom are calculated using 22 var MSEP df M SEP (4-64) where var B MSEP hh for MSEP h and 1BI. The disparity between the methods comes in the parameterizati on, and the estimate of the MSEP (or standard error) which subsequently impacts the degrees of freedom. We again study results under two parameterizations, 2, and 22,a Note that due to the translation invariance of 1M and tr AB, the figures and results demonstrated earlier in this chapter pertaining to 1 M and 1 2 Mtr A B still apply to the standard error estimates used in the Giesbrecht-Burns, Kenward-Roger, and FC prediction interval procedures.

PAGE 103

103 Two of the methods for formulating the prediction interval are used in SAS PROC MIXED. The first, from the DDFM = SATTERTH option, is similar to the Giesbrecht-Burns (1985) procedure outlined in Ch apter 2; however, in this a pplication we are calculating prediction limits for the realized value of a random effect rather than a fixed effect, as in that paper. The method is also developed in Jeske a nd Harville (1988) and Fai and Cornelius (1996). We will refer to it as the Gies brecht-Burns prediction interval method, since it uses the nave MSEP estimate. The standard erro r is estimated by the nave MSEP, 1/2 1 M where 1 M is in Equation (4-3). The other m ethod available in PROC MI XED through the DDFM = KR option is the Kenward-Roger (1997) approach. The standard error estimate includes a correc tion term and a bias correction term, as in Equation (4-59). Note that the Kenward-Roger prediction interval is determ ined under the parameterization. Recall that under the parameterization, 22,a the bias correction term estimate is equal to the correction term estimate, thus the MSEP estimate used in the Kenward-Roger prediction interval is 11 22 MtrMtr A B A B It is important to keep the parameterization in mind throughout, as the MSEP estimate and the degrees of freedom calculations depend on which parameteri zation is used. It is demons trated in Chapter 2 that in a single-dimension linear combination, the Kenwar d-Roger method for determining the degrees of freedom reduces to the Giesbrecht-Burns me thod. Thus, although the Kenward-Roger and Giesbrecht-Burns methods for determining the prediction limits use different estimates for the standard error, the degrees of freedom for the t -distribution will be the same. Kenward-Roger

PAGE 104

104 prediction limits adjust the MSEP estimate for estimating unknown variance components but do not adjust the t -distribution for the cha nge in standard error. The other two procedures for creating prediction intervals, FC and FC are based on the Fai-Cornelius (1996) hypothe sis testing methods. For the FC intervals, both the MSEP estimate and the degrees of freedom of the t -distribution account for th e correction term and the bias correction term. Note that this is actually an ex tension of the methods developed by Fai and Cornelius (1996). The bias correc tion term was not considered as an adjustment to the MSEP estimate in the methods they proposed. The bias co rrection term is added here to be consistent with MSEP estimator used in the Kenward-R oger method. The FC methods differ in the parameterization. Under the 22,a parameterization, the MSEP estimate is the same as in the Kenward-Roger parameterization, 11 22 MtrMtr A B A B Unlike the Kenward-Roger prediction interval, the FC procedure adjusts the de grees of freedom of the t -distribution for the adjustments to the MSEP estimate. The degr ees of freedom are still calculated using a Satterthwaite method; however, now the corrected MSEP estimate, 1 2 Mtr A B is used in the calculation of the degrees of freedom, rather than the nave MSEP estimate, 1 M. The final prediction limit method considered, FC differs from the other three in the parameterization, 2, Recall that the bias correction term estimate is not translation invariant, and thus by changing the parameterizati on, we also change the MSEP estimate. The

PAGE 105

105 MSEP estimate for the FC procedure is 11 2 Mtrtr A B B This is used in both the standard error estimate and the degrees of freedom calculation. A Monte Carlo simulation study was conducted to compare the four prediction interval methods. The study is similar to the one conducted in Section 4.1 to evaluate the correction term estimators. Using the RANNOR function in SAS, 10,000 independent sets of eij and ai were generated from 2(0,1) N and 20,aN for several values of2 a respectively. The values of 2 a correspond to 0.05, 0.1, 0.15, 1.0. Setting 0 10,000 sets of :1,,16ijyikj were created from the model ijiij y ae The number of levels of the random effect is varied to study the impact on the performance of the prediction interval procedures. The values for k are k = 3, 6, 15, or 30, while n is held constant at n = 6. Simulating the data in this way allows the realized values generated for the 'ias to be tracked. These are the realized values of the random effect for whic h the prediction interval s are constructed. By tracking the realized values of the random effect, we can dete rmine how often the prediction intervals contain the true valu e of the realized ra ndom effect, providing th e actual (simulated) coverage rate for a nominal 95% prediction interval. The prediction limits for 1a for the Giesbrecht-Burns and Kenward-Roger procedures are produced by the CL option in the ESTIMATE statement in PROC MIXED in SAS, with the DDFM = SATTERTH or DDFM = KR options, respec tively. To calculate the FC prediction limits for either parameterization, the degrees of freedom must be calculat ed directly and then used to determine the critical value from the t -distribution. For the FC procedure, the standard error from the KenwardRoger procedure is used to calculate the prediction limits. For

PAGE 106

106 the FC procedure, the standard error also has to be calculated directly in addition to the degrees of freedom because of the ch ange in parameterization. The SAS programming statements used to perform the simulation st udy are contained in Appendix A along with the derivatives needed to compute the degrees of freedom for the FC procedures. The results of the simulation study are best de monstrated pictorially. Figures 4-24 through 4-27 contain the results of the simulation study for k = 3, 6, 15, and 30, respectively. It is clear that the Kenward-Roger procedure produces the most inflated cove rage rates consistently across values of k and for the entire parameter space of The prediction limits for the KenwardRoger procedure will always be larger than the Giesbrecht-Burns procedure since 11 2 MtrM A B Adjusting the degrees of freedom in the FC procedure for the adjusted MSEP estimate generally increas es the degrees of freedom, causing the critical value from the t -distribution to shrink and thus reducing the width of the prediction interval. The coverage rates for the FC procedure are generally closer to 95% than the coverage rates for the Kenward-Roger procedure. The performance of the FC prediction interval is more dependent on the value for k and the parameter than the other intervals. For k = 3, the coverage rate for small values of is exceedingly small. As increases, the performance improves dramatically. This is due to the inclusion of the bias correction term under the 2, parameterization. Small values of k combined with small values of produce small MSEP estimates. Recall from Figure 4-23 that the relative bias of 11 2 Mtrtr A B B approached -50% for small values of This is caused by the significant underestimation of the true bias correction term by

PAGE 107

107 1 2 tr B (see Figure 4-14). For larger values of k, the FC prediction interval procedure compares favorably with the other procedures and consistently improves on the performance of the Kenward-Roger predic tion interval for larger values of For larger values of recall that 1 2 Mtr A B overestimated the true MSEP more so than 11 2 Mtrtr A B B Thus we would expect a more accurate prediction interval with the FC method in this range of the parameter space. To demonstrate the relationship between the pr ediction intervals more concretely, consider the hypothetical case where k = 6, n = 6, with data producing vari ance parameter estimates of 2 1 and 2 .5a Table 4-5 contains the values for the standard error, degrees of freedom, critical t -value and the width of the prediction interv al produced by each of the four procedures in this scenario. The width of the Kenward-Roge r prediction interval in creases over the width of the Giesbrecht-Burns interv al due to the increase in the standard error estimate. The width of the FC prediction interval is smaller than both the Giesbrecht-Burns and the Kenward-Roger intervals because of the increased degrees of freedom and subsequent de crease in the critical t value. The width of the FC interval is by far the smallest due to the decrease in the standard error estimate and the critical t -value. Finally, to demonstrate the effect of k on the performance of th e prediction interval procedures, Figure 4-28 shows the Kenward-R oger coverage rates for all values of k considered in the simulation study. This is demonstr ative of all the procedures in that as k increases, the procedures perform better and th e coverage rates move closer to the nominal value of 95%.

PAGE 108

108 This study shows the importance of investigating the Fai-Cornelius methods for hypothesis tests and prediction intervals in more complex situ ations. These preliminary results indicate that the Fai-Cornelius methods often perform bette r than the Kenward-Roger and the GiesbrechtBurns methods used in SAS. The Kenward-Roger procedure produces an overly-conservative prediction interval because it does not adjust the degrees of freedom for the inflated MSEP estimate. The study also demonstrates the imp act of the parameterization and the lack of transform invariance of the bias correction term on accuracy of the predic tion interval methods. In more complex covariance structures, the paramete rization may be vital to an accurate analysis. Table 4-5. Effect of predic tion interval procedures for k = 6, n = 6, 2 1 and 2 .5a GB KR FC FC Standard Error estimate 0.43301 0.46894 0.46894 0.40196 Degrees of Freedom 8.15 8.15 16.39 16.52 .025,dft 2.298 2.298 2.116 2.114 Width of CI 1.99 2.16 1.98 1.70

PAGE 109

109 -7 -6 -5 -4 -3 -2 -1 0 1 0 0.2 0.4 0.6 0.8 1 1.2 k=6 k=9 k=12 k=15 k=18 Figure 4-1. Equation (4-13) as a function of for several values of k, holding n = 6. Equation (4-14) hol ds when functions cross zero line.

PAGE 110

110 0 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0 0.2 0.4 0.6 0.8 1 1.2 tr -1A I RC Rtr A B Figure 4-2. Accuracy of approximations Rtr A B and tr -1A I for the correction term, RC under REML variance component estimation.

PAGE 111

111. 0 0.005 0.01 0.015 0.02 0.025 0.03 0.035 0.04 0.045 0.05 0 0.2 0.4 0.6 0.8 1 1.2 REtr A B RC Rtr A B Figure 4-3. Accuracy of approximation Rtr A B and estimator REtr A B for the correction term, RC under REML variance component estimation.

PAGE 112

112 0 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0 0.2 0.4 0.6 0.8 1 1.2 Etr -1A I tr -1A I RC Figure 4-4. Accuracy of approximation tr -1A I and estimator Etr -1A I for the correction term, RC under REML variance component estimation.

PAGE 113

113 0 0.01 0.02 0.03 0.04 0.05 0.06 0 0.2 0.4 0.6 0.8 1 1.2 REtr A B RC Etr -1A I Figure 4-5. Comparison of accuracy of estimators for the correction term, RC under REML variance component estimation.

PAGE 114

114 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 00.20.40.60.811.2 Atr A B Atr A B AC tr -1A I Figure 4-6. Comparison of accuracy of approximations tr -1A I Atr A B and Atr A B for the correction term, AC under ANOVA variance component estimation.

PAGE 115

115 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 00.20.40.60.811.2 AC Atr A B AEtr A B Figure 4-7. Accuracy of approximation Atr A B and estimator AEtr A B for the correction term, AC under ANOVA variance component estimation.

PAGE 116

116 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 00.20.40.60.811.2 AC Atr A B AEtr A B Figure 4-8. Accuracy of approximation Atr A B and estimator AEtr A B for the correction term, AC under ANOVA variance component estimation.

PAGE 117

117 0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0 0.2 0.4 0.6 0.8 1 1.2 AC Etr -1A I tr -1A I Figure 4-9. Accuracy of approximation tr -1A I and estimator Etr -1A I for the correction term, AC under ANOVA variance component estimation.

PAGE 118

118 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 0 0.2 0.4 0.6 0.8 1 1.2 AC AEtr A B AEtr A B Etr -1A I Figure 4-10. Comparison of accuracy of estimators for the correction term, AC under ANOVA variance component estimation.

PAGE 119

119 0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0 0.2 0.4 0.6 0.8 1 1.2 AC RC REtr A B Etr -1A I Etr -1A I Figure 4-11. Accuracy of estimator s for the correction term under REML, and ANOVA, variance component estimation.

PAGE 120

120 -100.00% 0.00% 100.00% 200.00% 300.00% 400.00% 500.00% 0 0.2 0.4 0.6 0.8 1 1.2Relative Bias REtr A B Etr -1A I Etr -1A I Figure 4-12. Relative bias of estimat ors for the correction term under REML, or ANOVA, variance component estimation.

PAGE 121

121 0 0.2 0.4 0.6 0.8 1 1.2 0 0.2 0.4 0.6 0.8 1 1.2 1M 1 EM Figure 4-13. Bias produced by estimating 1M with 1 M.

PAGE 122

122 -0.6 -0.5 -0.4 -0.3 -0.2 -0.1 0 0.1 0 0.2 0.4 0.6 0.8 1 1.2 1 2 tr -1 I 1 2 Etr -1 I 11 MEM Figure 4-14. Accuracy of approximation 1 2 tr -1 I and estimator 1 2 Etr -1 I for the true bias correction term, 11 MEM under REML variance component estimation.

PAGE 123

123 -0.03 -0.02 -0.01 0 0.01 0.02 0.03 0.04 0.05 0.06 0.07 00.20.40.60.811.2 tr -1A I Etr -1A I 11 MEM Figure 4-15. Accuracy of approximation tr -1A I and estimator Etr -1A I for the true bias correction term, 11 MEM under REML variance component estimation.

PAGE 124

124 -0.6 -0.5 -0.4 -0.3 -0.2 -0.1 0 0.1 0 0.2 0.4 0.6 0.8 1 1.2 11 MEM 1 2 Etr -1 I Etr -1A I Figure 4-16. Comparing the accuracy of 1 2 Etr -1 I and Etr -1A I for the true bias correction term, 11 MEM

PAGE 125

125 -0.15 -0.05 0.05 0.15 0.25 0.35 0.45 0.55 0.65 0.75 00.20.40.60.811.2 1M 1EM Figure 4-17. Bias produced by estimating 1M with 1M.

PAGE 126

126 -0.6 -0.5 -0.4 -0.3 -0.2 -0.1 0 0.1 0.2 00.20.40.60.811.2 1 2 Etr -1 I 1 2 tr -1 I 11MEM Figure 4-18. Accuracy of approximation 1 2 tr -1 I and estimator 1 2 Etr -1 I for true bias correction term, 11MEM under ANOVA variance component estimation.

PAGE 127

127 0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0 0.2 0.4 0.6 0.8 1 1.2 Etr -1A I tr -1A I 11MEM Figure 4-19. Accuracy of approximation, tr -1A I and estimator Etr -1A I for true bias correction term, 11MEM under ANOVA variance component estimation.

PAGE 128

128 -0.6 -0.5 -0.4 -0.3 -0.2 -0.1 0 0.1 0.2 0 0.2 0.4 0.6 0.8 1 1.2 11MEM 1 2 Etr -1 I Etr -1A I Figure 4-20. Comparing the accuracy of 1 2 Etr -1 I and Etr -1A I for the true bias correction term, 11MEM

PAGE 129

129 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 0.2 0.4 0.6 0.8 1 1.2 1 2 Mtr -1 A I 11 2 Mtrtr -1 -1 A I I 1 RRMMC 1 M 1 2RMtr A B Figure 4-21. Comparison of estimator s of the MSEP for the EBLUP of ia utilizing REML variance component estimation.

PAGE 130

130 -0.2 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 1.2 1 AAMMC 12 Mtr -1 A I 11 2 Mtrtr -1 -1 A I I 1M Figure 4-22. Comparison of estimator s of the MSEP for the EBLUP of ia utilizing ANOVA variance component estimation.

PAGE 131

131 -100.00% -50.00% 0.00% 50.00% 100.00% 150.00% 200.00% 250.00% 300.00% 0 0.2 0.4 0.6 0.8 1 1.2% Relative Bias 11 2 Mtrtr -1 -1 A I I 12 Mtr -1 A I 1 2 Mtr -1 A I 11 2 Mtrtr -1 -1 A I I 1 2RMtr A B 1 M 1M Figure 4-23. Relative bias of possibl e estimators for MSEP of EBLUP for ia

PAGE 132

132 0.9 0.92 0.94 0.96 0.98 1 1.02 00.10.20.30.40.50.60.70.80.91Coverage Rate FC FC K R GB Figure 4-24. Simulated coverage rates of prediction interval procedures for 1ta with k = 3.

PAGE 133

133 0.88 0.9 0.92 0.94 0.96 0.98 1 1.02 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1Coverage Rate FC FC KR GB Figure 4-25. Simulated coverage rates of prediction interval procedures for 1ta with k = 6.

PAGE 134

134 0.93 0.94 0.95 0.96 0.97 0.98 0.99 1 1.01 00.10.20.30.40.50.60.70.80.91Coverage Rate FC FC KR GB Figure 4-26. Simulated coverage rates of prediction interval procedures for 1ta with k = 15.

PAGE 135

135 0.94 0.95 0.96 0.97 0.98 0.99 1 1.01 00.10.20.30.40.50.60.70.80.91Coverage Rate FC FC KR GB Figure 4-27. Simulated coverage rates of prediction interval procedures for 1ta with k = 30.

PAGE 136

136 0.94 0.95 0.96 0.97 0.98 0.99 1 1.01 00.10.20.30.40.50.60.70.80.91Coverage Rate k = 30 k = 15 k = 6 k = 3 Figure 4-28. Simulated coverage rates for the Kenward-Roger prediction interval procedure.

PAGE 137

137 CHAPTER 5 THE EFFECT OF NEGATIVE VARIANCE COMPONENTS ON EBLUP The effect of negative varian ce component estim ates on EBLUP procedures is a topic that has been largely neglected (Stroup and Littell 20 02). While the use of REML estimates, which are truncated at zero, would seem to negate this issue, there are still many circumstances when other variance parameter estimation methods ar e chosen and negative variance components may arise. For example, as shown in Stroup and Litt ell (2002) and in the resu lts on the accuracy of the correction term and bias correction term estim ates in Chapter 4, REML may not always be the preferred choice for varian ce component estimation. One strategy for handling negative variance component estimation is to allow that th e component may be a covariance rather than a variance, thus alleviatin g the problem of truncating at zero. We begin to look at the impact of that strategy by again turning to the special case of the balanced one way random effects model. 5.1 Negative Variance Component Estimates and the Balanced One Way Random Effects Model To demonstrate the current issues with ne gative variance component estimation, we first look at the current SAS 9.1 output for PROC MIXED, both when the variance component estimates are truncated at zero and when they are not. Using the REML NOBOUND statement in the procedure statement allows the variance co mponent estimates to take any value on the real line. Note that because we are in a bala nced situation, the ANOVA variance component estimation would produce the same results as REML NOBOUND. We compare the effect of truncating using simulated data, gene rated from a normal distribution with 2.01a and 21 This distribution produces a high likelihood that the data generate d will have larger variation within a block than between the blocks, he nce often producing a negative estimate for 2a For demonstration purposes, the number of levels of the random effect is k = 5 and the number of

PAGE 138

138 observations at each level is n = 6. A dataset that meets the a bove condition is given in Table 51. It is easy to spot that more variation occurs within the block le vels than between them. Table 5-1. Generated data that produ ces negative variance component estimate Replicate 1 2 3 4 5 6 Block 1 2.34754 0.24672 -0.5585 0.60564 -0.9062 1.37627 Block 2 1.01248 -0.0493 0.49583 -1.3952 -0.8644 1.93825 Block 3 0.18826 0.91668 0.68925 0.96593 0.95929 -0.7838 Block 4 -0.4741 1.31312 -1.5036 0.06752 1.2383 -0.7888 Block 5 -1.2176 0.64999 0.32013 1.4081 -0.1774 1.77956 The results from the REML option in the MIXED procedure of SAS are contained in Tables 5-2 and 5-3 and the results from the REML NOBOUND options in th e MIXED procedure are contained in Tables 5-4 and 5-5. Also, for comparison, least square means (LSMEANS) generated by PROC GLM are given in Table 5-6. These give a starting point from which to compare the EBLUPS resulting from the REML and REML NOBOUND options. The following SAS code generates the output in Tables 5-2 and 5-3: proc mixed data = &sample method = reml ; class blk trt; model y = /ddfm = kenwardroger ; random blk / s ; Table 5-2. REML variance parameter estimates Covariance parameter Estimate Block 0 Residual 1.0643 Table 5-3. Solution for random effects w ith REML variance parameter estimates Effect Block Estimate Std Err PredDF t Value Pr > |t| Block 1 0 . Block 2 0 . Block 3 0 . Block 4 0 . Block 5 0 .

PAGE 139

139 The NOBOUND option in the following SAS code is the only diffe rence from the previous code and generates the SAS output in Tables 5-4 and 5-5: proc mixed data = &sample method = reml nobound ; class blk trt; model y = /ddfm = kenwardroger ; random blk / s ; Table 5-4. REML NOBOUND variance parameter estimates Covariance Parameter Estimate Blk -0.1410 Residual 1.1810 Table 5-5. Solution for random effects with REML NOBOUND variance parameter estimates Effect Block Estimate Std Err PredDF t Value Pr > |t| Block 1 -0.4846 0 29 -Infty <.0001 Block 2 0.3461 0 29 Infty <.0001 Block 3 -0.4106 0 29 -Infty <.0001 Block 4 0.8870 0 29 Infty <.0001 Block 5 -0.3379 0 29 -Infty <.0001 Table 5-6. LSMEANS from PROC GLM Block LSMEAN 1 0.51858487 2 0.18960919 3 0.48927203 4 -0.02458600 5 0.46046652 The results from the REML option, which trunc ates the variance component estimate for block effect at zero, are straightforward. The solution statement produces zero estimates for all blocks and missing values for the standard error, t-va lues, degrees of freedom, and p-values. The REML NOBOUND option is not quite so straightforward. When variance component estimates are positive, we can think of the BLUP as a shr inkage predictor. In other words, using the

PAGE 140

140 variance component estimate as a weight, the bl ock means are shrunk towa rd the overall mean. In addition, in a balanced model, the BLUPS mainta in the rank order of the lsmeans. This can be easily seen in the structure of the BLUP for the balanced one-way random effects model in Equation (4-2). Now, looking at the EBLUP estimates in Tabl e 5-5, the first indication of a problem is that the standard errors are listed as zero and hence the t-values are listed as infinity. The calculations for standard error in this case are negative, causing SAS to truncate the standard error at zero. Comparing the estimates in Table 5-5 to the LSMEANS from PROC GLM in Table 5-6 shows that the nega tive variance component estimate causes the EBLUP estimates to invert their rank order and expands the range of the estimates. The block with the largest LSMEAN (0.5186) now has the smallest EBLUP (-0 .4846) and, likewise, the block with the smallest LSMEAN (-0.0246) now has the larges t EBLUP (.8870). Also, the range of the LSMEANS is 0.5432 while the range of the EBLU P estimates is 1.3716. We normally expect the BLUP procedure to shrink the block m eans toward the overall mean according to the variation in the model; however, in this case, the estimates are inverted about the mean and expanded away from the overall mean. While the standard errors and t-values listed with the EBLUPs are a clear red flag, the unexpected results of the EBLUPS require further investigation. Note also that the standard e rrors may be positive even when 20a ; this would eliminate the red flag and leave more room for error in the interpretation of the results. If the data analyst believes that a one-way random effects model is the true model for the research situation, then the r ecommendation from an EBLUP sta ndpoint is to truncate at zero rather than proceed with a negative variance component estimate. A ny use of EBLUPS while allowing the variance component to gi ve a negative estimate is likely to yield misleading results.

PAGE 141

141 The LSMEANS likely yield more reliable estim ates than the EBLUPS from the REML NOBOUND option. Keep in mind that it is not always the best choice to truncate at zero when variance component estimates are ne gative if the analyst is interested in fi xed effects (Stroup and Littell 2002). 5.2 Considering the Variance Parameter as a Covariance One idea that surfaces when faced with negative variance component estimates is considering the parameter as a covariance rather than a variance, thus allowing the parameter estimate to take on values over the entire real line Smith and Murray (1984) consider this option and the impact on estimation and hypothesis te sting of the covarian ce component. For the balanced one way random effects model discussed in Chapter 4, the covariance structure would be modified from 22 2cov(,), 0, ijija a y yiijj iijj ii to 2cov(,), 0, ijija a y yiijj iijj ii where 20aa One problem overlooked by Smith and Murray (1984) is that in a one way model, if the variance structure is modified in this way, the model is no longer a typical random effects model. The motivating example used in Smith and Murray (1984) is weaning rates of twin calves for 40 cows. In the original model, the cow effect is considered random, with the cows regarded as a

PAGE 142

142 random sample from a population of cows, and 2 a the variance associated with this population. When the estimate for 2a is negative, the definition of the parameter is changed from the population variance of the cows to the covariance between twins born to the same cow. Thus the population variance for the cows is ignored all to gether and the negative covariance estimate is interpreted as reflecting competition for nutrition between the twin calves. The negative value of the variance parameter estimate is likely reflecting the combination of the positive cow population variance and the negative correlation between twins born to the same cow. That is, there is more negative intraclass variability th an positive interclass variability resulting in a negative parameter estimate. The inversion of the LSMEANs that we s ee with the EBLUPs in PROC MIXED when the variance component estimate is negative occurs because the model is effectively changed to this Smith and Murray (1984) model. The EBLUP calculated by PROC MIXED for ia in this case is ... 2 a i an y y n giving a negative coefficient when 2 0an Furthermore, a may be sufficiently large so that 2 1 a an n Thus rather than shrinking the means, as we expect EBLUP procedures to do, the EBLUP inverts and expands the range of the means. Clearly this is an unacceptable result. The question remains: what is an analyst to do when faced with this result. One possible solution is derived in the next section.

PAGE 143

143 5.3 BLUP derivation for Random Effects Mo del with Correlated Errors We now present a model that would account for both the population variance of the random effect and the correlation of replicates with in a level of the random effect. Consider a model with a random effect, as we ll as correlated errors. The model and variance structure in Equation (4-1) would be modified to: 2,1,,,1,, 0,ij iij ia y aeikjn aN where are normally distributed withije 20 cov(,), 0, ij a ijijaEe iijj eeiijj ii (5-1) In mixed models notation, this equates to y=X +Zu+e where 12, ,,N kn kaaa X1 ZI1 uN0,G where 2 akGI and 11,12,kneee eN 0 R where 2()knanRIIJ so that

PAGE 144

144 22 knaan V=ZGZ+RIIJ. From general matrix theory (e.g., Searle 1982), we have 2 1 2 221aa kn n aan VII J which gives the BLUP for u as 2 .. 22 1a kn aa y n -1uGZVyI y or the BLUP for ia as 2 ... 22 a ii aan ay y n (5-2) Note that for the model structure given in Equation (5-1) to be valid, V m ust be positive definite. Therefore, the parameter space for which the model is valid is 2 2220, 0, aaan In this parameter space the coefficient for the BLUP, given in Equation (5-2), is non-negative. Thus the order of the LSMEANS for the bloc ks is preserved in the BLUP. One key consideration for this model is the problem of overparameterization. The parameters 2a and a are confounded and cannot be simu ltaneously estimated. One solution is to use an estimate for 2 a from previous experience and consider the parameter as known. This leaves only a and 2 as unknown and the EBLUP analysis can be pe rformed where the unknown variance parameters are estimated by an unbounded method, such as REML NOBOUND in PROC MIXED. There

PAGE 145

145 are many situations, such as agriculture or educ ation, where using a previous estimate as the parameter is realistic. Data contained in Table 5-5 are generated from a distribution as described in Equation (5-1) with 2.9, .13a and 2.07a Table 5-7. Data generated from distribution in Model (5-1) Replicate 1 2 3 4 5 6 Block 1 1.77754 -0.72357 -0.79317 -0.79674 1.31455 -0.98605 Block 2 -0.62131 -1.15758 -1.21619 -0.24776 0.5867 0.12472 Block 3 -0.3412 -0.7861 0.41517 0.75414 1.34176 0.75806 Block 4 -0.05152 -0.9336 0.31271 -0.3097 1.04458 1.07845 Block 5 0.75952 -0.27854 0.28914 1.03234 -0.88724 -2.24931 The appropriate SAS statements to analyze this data in the manner described in Equation (5-2) are data gmat ; input row col value; cards; 1 1 .07 2 2 .07 3 3 .07 4 4 .07 5 5 .07 ; proc mixed data = &sample method = reml ; class blk trt ; by rn ; model y = /ddfm = kenwardroger ; random blk / gdata=gmat g s ; repeated trt / sub = blk type=cs ; run ; The gdata option after the random st atement provides the known G matrix which in this case is .07 5GI where nI is an n-dimensional identity matrix. The output from this analysis of the data in Table 5-7 are contained in Tables 5-8 and 5-9. The EBLUPS for the levels of the random effect are generated by the solutions or s option

PAGE 146

146 in the random statement of the SAS code, and are given in Table 5-9 under Estimate. Comparison of these values to the LSMEANS ge nerated by GLM in Table 5-10 show that the function worked as expected, maintaining the rank order of the means and shrinking the predictors toward the overall mean. Table 5-8. Covariance parame ter estimates under Mode1 (5-1) Covariance Parameter Subject Estimate Variance Block 0.9497 CS Block -0.1311 Table 5-9. Solution for ra ndom effects under Model (5-1) Effect Block Estimate Std Err PredDF t Value Pr > |t| Block 1 -0.00593 0.2646 2.16 -0.02 0.9840 Block 2 -0.2849 0.2646 2.16 -1.08 0.3868 Block 3 0.2761 0.2646 2.16 1.04 0.3991 Block 4 0.1560 0.2646 2.16 0.59 0.6113 Block 5 -0.1412 0.2646 2.16 -0.53 0.6433 Table 5-10. LSMEANS from PROC GLM Block y LSMEAN 1 -0.03457327 2 -0.42190439 3 0.35697247 4 0.19015370 5 -0.22234738 Note that the DDFM = KR method wa s used here which uses an adju sted standard error. If the DDFM = SATTERTH option is used, the estimates a nd degrees of freedom remain the same, but the standard error is reduced to .1647 which would impact the prediction limits as shown in Section 4.5. Given the conclusion in Chapter 4 regarding the inaccuracy of the adjustment under

PAGE 147

147 certain circumstances, the accuracy of either standard error estimate and their effect on prediction intervals for the rea lized values of the random effect needs further study. This remains as future research. For comparison, Tables 5-11 and 5-12 contain the results of the analysis of the data in Table 5-7 if the correlation were ignored and the model were run as a random effects model with the REML NOBOUND option. We see that the va riance parameter estimate is negative and the EBLUPs are inverted and expanded as shown in Section 5.1. Table 5-11. Variance parameter estimates under random effects model Covariance Parameter Estimate Block -0.06110 Residual 0.9497 Table 5-12. Solution for random effects under random effects model Effect Block Estimate Std Err PredDF t Value Pr > |t| Block 1 0.005176 0 29 Infty <.0001 Block 2 0.2487 0 29 Infty <.0001 Block 3 -0.2410 0 29 -Infty <.0001 Block 4 -0.1361 0 29 -Infty <.0001 Block 5 0.1232 0 29 Infty <.0001 One issue that occurs with an alyzing data in this way is th at it is prone to convergence failure. Of 10,000 simulated data sets fr om the distribution described in Equation (5-1) and analyzed with this procedure, 21.5% failed to con verge when wither REML or REML NOBOUND procedures were used. Even with th is complication, this parameterization remains one of the only viable options if predictors of the realized values of the random effects are desired in a case with negative variance component estimation.

PAGE 148

148 CHAPTER 6 CONCLUSION AND FUTURE WORK The prim ary goal of this dissertation was to examine the impact of estimating variance components on the estimators of the MSEP for the EBLUP. This was accomplished by investigating propertie s of generalized components of the MSEP estimator, examining the behavior of various procedures in the balanced one-way random effects model, and determining the impact of negative variance components on EBLUP procedures. Following the work by Kackar and Haville (1984), Kenward and Roger (1997), Jeske and Harville (1988), and Fai and Cornelius (1996), estimators for the correction term and the bias correction term were generalized to accommodate multiple dimension linear combinations of fixed and random effects. Properties of these ge neralized estimators were investigated. We showed that while for linear covariance stru ctures the bias correction term estimator, KRBIAS continues to reduce to th e correction term estimator, KRC which is nonnegative definite, KRBIAS in general, is indefin ite. The indefiniteness of KRBIAS causes the estimator, KR1M BIAS of the nave MSEP, 1M to be indefinite. Future work includes the investigation of defin iteness properties of the MSEP estimator KR KR1M C BIAS which includes both the correcti on term and the bias correction term. The magnitude of KR1M C may outweigh that of KRBIAS to produce a positive definite estimator. We also demonstrated that KRBIAS is not transform invariant, and that the choice of parameteri zation can have a significant imp act on the bias correction term, the MSEP estimate, and the coverage rates of prediction intervals. The balanced one-way random effects model provided a simple setting in which the various components of MSEP estimators coul d be thoroughly explored. Closed form

PAGE 149

149 expressions allowed for direct examination of the correction term and bias correction term estimators. Through this example, the impact of variance component estimation techniques, the values of the variance components, and the para meterization was demonstrated. We determined that the MSEP estimator may significantly ove ror under-estimate the true MSEP depending on these factors. We showed that truncating variance parameter estimates under REML estimation can cause significant (long-run) overestimation of both the correction term and the bias correction term, over most of the parameter space. While these procedures have been considered suspect near the boundary of the parameter space, this study showed that the accuracy of the methods is questionable over a much greater range of the parameter space. We also discovered that the choice of how to measure the MSE of the variance parameter estimates can significantly impact the accuracy of the correction term estimator. In REML situations, using the exact MSE of the varian ce parameter estimates in the correction term estimator resulted in a more accu rate correction term estimator than using the asymptotic MSE over two-thirds of the parameter space. Howeve r, if variance parameter estimates are left untruncated, as in ANOVA variance component estimates, the asymptotic MSE provided a more accurate correction term estimator over the entire parameter space. The importance of the parameterization of the m odel is apparent in th e bias correction term estimator since it is not transform invari ant. Under a linear parameterization, KRBIAS slightly overestimates the true bias correcti on term; while under a nonl inear parameterization KRBIAS significantly underestimates the true bias correction term for a large portion of the parameter space when using REML variance component estimates. Under ANOVA variance parameter estimation, the linear parameterizatio n provides a significantly more accurate bias correction term estimator. Even with this simp le model, we see how the interaction of a few

PAGE 150

150 factors complicates the results. The question of which MSEP estimator is best depends on the parameterization, the true value of the variance parameter, and the choice of variance parameter estimation method. Each of these factors deserves investigation in more complex models. These same factors impact the coverage rates of prediction intervals fo r a realized value of the random effect. We introduced a modifi cation of the Fai-Cornelius methods for approximating the degrees of freedom that in cludes a bias correction term in the MSEP estimator. The results of a Monte Carlo simula tion study showed that the Fai-Cornelius based methods compare favorably with the Giesbrecht -Burns and Kenward-Roger methods currently available in SAS. The simulation study also demonstr ated a shortcoming of the KenwardRoger method in that the degrees of freedom ar e not adjusted for the increased MSEP estimate for a single dimension linear combination. In f act, of the four predicti on interval procedures included in the simulation study, the Kenward-R oger method consistently performed the poorest, producing the most inflated covera ge rates. Because the modified Fai-Cornelius method adjusts both the MSEP and the distribution for the variance component estimates, this method deserves more attention and may significantly improve the coverage rates or Type I error for hypothesis testing in many situations. Futu re work includes further investig ation of these methods in more complex settings. The impact of negative va riance component estimates on the EBLUP and the MSEP estimators was studied. We show that the EBLUP is no longer a reliable or valid estimator of the random effect when negative variance component estimates are present. The proposed solution by Smith and Murray (1984) of rega rding the variance parameter as a covariance is shown to be invalid for EBLUP procedures. An alterna tive solution of a rando m effects model with correlated errors was presented. Further study of this proposed method is indicated. Future

PAGE 151

151 work also includes the investigation of the impa ct of negative varian ce parameter estimates on fixed effects estimators, their standard errors, and distribution estimation techniques.

PAGE 152

152 APPENDIX DETAILS FOR MONTE CARLO SIMULATION STUDY ON PREDICTION INTERVALS The calcu lations necessary for the FC and FCprediction intervals are detailed below, followed by an example of the SAS code used for the Monte Carlo simulation study to determine coverage rates for the four prediction interval procedures in Section 4.5. Recall from Equation (4-64) that the inverse of th e observed infor mation matrix, 1BI and the vectors of derivatives of the MSEP, h, are necessary for the Satterthwaite-method of calculating degrees of freedom. First we look at the FC method. The inverse of the observed information matrix for the 22,a is 22 22 22 22 2 222 11 22 1 111 kn knn knnnkkn -1B I (A-1) We also need 11 2222aMtr Mtr A B A B h where, due to translation invariance, 1M is as in Equation (4-3) and tr A B is as in Equation (4-15). Thus, 1 2 222 2 2222 111 212 1a aMtr k nk kn n knnn A B (A-2) and

PAGE 153

153 2 22222 1 2 2 22 2 2 2 2222 2 21 1aa a a aMtr nkn kn nkn knnn A B (A-3) The degrees of freedom for the FC method are calculated as 2 1 1 22 var2FCMtr df Mtr A B A B where 1 var2 Mtr -1 A B h I h with the elements of h from Equations (A-2) and (A-3) and -1I from Equation (A-1). For the FC method, we turn to the 2, parameterization. The inverse of the observed information matrix is now 2 22 2 222 11 2 21 111 knkn kn knkkn -1B I (A-4) We use the MSEP estimator utilizing both the correction term and the bias correction term elements to be consistent w ith the Kenward-Roger and the FC methods. Because the bias correction term estimator is not transform invariant, the MSEP estimate is different than the one used in the parameterization. The MSEP estimator in the 2, parameterization is given in Equation (4-60). Thus

PAGE 154

154 1 1 211 22 Mtr tr Mtr tr A B B A B B h where 1 2 2 2 2221 111 2 411 21 111Mtrtr k nk knk kn knnnknk A B B (A-5) and 22 1 2 22 22 2 22 21 11 2 42111 21 11 1Mtrtr k nk kknk kn knnnknk A B B (A-6) The degrees of freedom for the FCmethod are calculated as 2 1 11 2 2 1 var 2FCMtrtr df Mtrtr A B B A B B where 11 var 2 Mtrtr A B B h B h with the elements of h from Equations (A-5) and (A-6), and -1I from Equation (A-4).

PAGE 155

155 An example of the SAS code used for the Monte Carlo simulation study to determine coverage rates for the four prediction interval procedures, utilizing the above derivations for the FC and FC methods, follows. /*Sample program for Monte Carlo simulation study of prediction intervals*/ /*Generating Data for random effect and error from normal distributions and creating y_ij, observations*/ /*k=3,n=6,sigma(a)_squared=10*/ data normal ; do mc = 1 to 10000; do sigma_a = (10)**.5 ; do i = 1 to 3 ; stda = rannor(17) ; do j = 1 to 6 ; e_ij = rannor (18) ; a = sigma_a*stda ; y_ij = a +e_ij ; output ; end ; end ; end ; end ; run ; /*Creates Satterthwaite prediction limits on a1*/ proc mixed data = normal method=REML noclprint noinfo noitprint; class i ; model y_ij = / ddfm=Satterth ; random i ; estimate 'effect 1' | i 1 0 0 /cl; by mc ; ods listing exclude estimates ; ods listing exclude fitstatistics ; ods listing exclude covparms ; ods output estimates=est1 ; run ; /*Creates Kenward-Roger prediction limits on a1*/ /*Outputs Kenward-Roger Standard Error for use in FC( ) prediction intervals*/ /*Outputs Covariance Parameter Estimates to use in FC( ) calculations*/ proc mixed data = normal method=REML noclprint noinfo noitprint; class i ; model y_ij = / ddfm=kenwardroger ;

PAGE 156

156 random i ; estimate 'effect 1' | i 1 0 0 /cl; by mc ; ods listing exclude estimates ; ods listing exclude fitstatistics ; ods listing exclude covparms ; ods output estimates=est2 ; ods output covparms=cp ; run ; data est1 ; set est1 ; drop Label ; rename tValue=Satt_tval ; rename DF=Satt_df ; rename Probt = Satt_pvl ; rename StdErr=seSATT ; rename upper = satt_uppercl ; rename lower = satt_lowercl ; run ; data est2 ; set est2 ; drop Label estimate; rename StdErr=seKR ; rename tValue=KR_tval ; rename DF=KR_df ; rename Probt = KR_pval ; rename upper = KR_uppercl ; rename lower = KR_lowercl ; run ; proc sort data=cp ; by covparm ; data siga resid out ; set cp ; if covparm='i' then output siga ; else if covparm='Residual' then output resid ; run ; data siga ; set siga ; siga = estimate ; drop covparm estimate ; run ; data resid ; set resid ; resid = estimate ; drop covparm estimate ; run ; /*Isolates a1 for each data set*/ data a ; set normal ;

PAGE 157

157 where i = 1 and j = 1 ; run ; data est ; merge est1 est2 a siga resid; by mc ; k=3 ; n=6 ; upshat = resid/(resid + n*siga) ; m1 = resid*(1-upshat)*(1+((k-1)*upshat))/(n*k*upshat) ; /*elements of inverse information matrix for */ b11 = 2*resid*resid/(k*(n-1)) ; bsigsiga = -2*resid*resid/(k*n*(n-1)) ; bsiga = 2*resid*resid*(1/(n*n))*((1/(upshat*upshat*(k-1)))+(1/(k*(n-1)))) ; /*derivatives of M1, parameterization*/ d1 = n*(k-1)*siga*siga/(k*(resid+n*siga)*(resid+n*siga)) ; d2 = (resid*(2*n*siga + k*resid) + n*n*siga*siga)/(k*(resid+n*siga)*(resid+n*siga)) ; /*derivatives of 2*trab, parameterization*/ d1_FC2 = d1 + 2*((2*(k*n-1)*resid*resid + 4*(k*n1)*n*resid*siga)/(k*k*n*(n-1)*(resid+n*siga)*(resid+n*siga))) ; d2_FC2 = d2 2*(2*n*(k*n-1)*upshat*upshat/(k*k*n*(n-1))) ; /*dfFC2 = FC( ) degrees of freedom*/ dbdFC2 = d1_FC2*(d1_FC2*b11 +d2_FC2*bsigsiga) + d2_FC2*(d1_FC2*bsigsiga +d2_FC2*bsiga) ; dfFC2 = 2*(seKR**2)*(seKR**2)/dbdFC2 ; /*critical t-value for alpha=.05, FC( ) degrees of freedom*/ tFC025 = quantile('T',.975,dfFC2); /*FC( ) prediction limits*/ FC_phi_uppercl= estimate +tFC025*seKR ; FC_Phi_lowercl = estimate tFC025*seKR ; /*MSEP estimate for parameterization*/ msekrbias = m1 + 2*resid*upshat*(k*n-1)/(k*k*n*(n-1)) 2*resid*((k*(n-1))-((k-1)*(k-1)*upshat*upshat))/(n*k*k*(n1)*(k-1)*upshat) ; sqmsekrbias = msekrbias**(1/2) ; /*elements of inverse information matrix for (along with b11)*/ b12 = 2*resid*upshat/(k*(n-1)) ; b22 = 2*upshat*upshat*(k*n 1)/(k*(n-1)*(k-1)) ; /*derivatives of M1, parameterization*/

PAGE 158

158 d1satt = (1-upshat)*(1+(k-1)*upshat)/(n*k*upshat) ; d2satt = resid*((1-k)*upshat*upshat -1)/(n*k*upshat*upshat) ; /*derivative of trab + bias term, parameterization*/ d1_FC3 = d1satt + 2*upshat*(k*n-1)/(k*k*n*(n-1)) (2*k*(n-1) 2*(k-1)*(k1)*upshat*upshat)/(n*k*k*(n-1)*(k-1)*upshat) ; d2_FC3 = d2satt + 2*resid*(k*n-1)/(k*k*n*(n-1)) + 2*resid*(2*(k-1)*(k-1)*upshat*upshat + k*(n-1) (k-1)*(k1)*upshat*upshat)/(n*k*k*(n-1)*(k-1)*upshat*upshat) ; /*dfFC3 = FC( ) degrees of freedom*/ dbdFC3 = d1_FC3*(d1_FC3*b11 +d2_FC3*b12) + d2_FC3*(d1_FC3*b12 +d2_FC3*b22) ; dfFC3 = 2*msekrbias*msekrbias/dbdFC3 ; /*critical t-value, alpha = .05, FC( ) degrees of freedom*/ tFC025bias = quantile('T',.975,dfFC3); /*FC( ) prediction limits*/ FC_theta_uppercl = estimate +tFC025bias*sqmsekrbias ; FC_theta_lowercl = estimate tFC025bias*sqmsekrbias ; /*Flags each prediction interval containing a1*/ if satt_uppercl = or satt_lowercl = then satt_cover = ; else if satt_uppercl > a and satt_lowercl < a then satt_cover = 1 ; else satt_cover=0 ; if KR_uppercl = or KR_lowercl = then KR_cover = ; else if KR_uppercl > a and KR_lowercl < a then KR_cover = 1 ; else KR_cover=0 ; if FC_phi_uppercl = or FC_phi_lowercl = then FC_phi_cover = ; else if FC_phi_uppercl > a and FC_phi_lowercl < a then FC_phi_cover = 1 ; else FC_phi_cover=0 ; if FC_theta_uppercl = or FC_theta_lowercl = then FC_theta_cover = ; else if FC_theta_uppercl > a and FC_theta_lowercl < a then FC_theta_cover = 1 ; else FC_theta_cover = 0 ; run ; /*Calculates True Coverage Rates for Each Prediction Interval Method*/ proc means data = est n mean stderr lclm uclm; var satt_cover KR_cover FC_phi_cover FC_theta_cover ; title 'Sigmasq_a=10 k=3 n=6' ; title2 'Simulated true coverage rates' ; title3 'BLUP a1' ; run ;

PAGE 159

159 LIST OF REFERENCES Booth, J. G. and Hobert, J. P. (1998), Standard E rrors of Prediction in Generalized Linear Mixed Models, Journal of the American Statistical Association 93, 262-272. Corbeil, R. R. and Searle, S. R. (1976), Restr icted Maximum Likelihood Estimation of Variance Components in the Mixed Model, Technometrics 18, 31-38. Fai, H. T. and Cornelius, P. L. (1996), Appr oximate F-tests for Multiple Degree of Freedom Hypotheses in Generalized Least Squares Analyses of Unbalan ced Split-Plot Experiments, Journal of Statistical Co mputation and Simulation 54, 363-378. Ferguson, T. S. (1967), Mathematical Statistics New York: Academic Press. Giesbrecht, F. G. and Burns, J. C. (1985), Two-Stage Analysis Based on a Mixed Model: Large-Sample Asymptotic Theory and Small Sample Simulation Results, Biometrics 41, 477486. Harville, D. A. (1976), Extension of the Gaus s-Markov Theorem to Include the Estimation of Random Effects, The Annals of Statistics 4, 384-395. Harville, D. A. (1977), Maximum Likelihood A pproaches to Variance Component Estimation and to Related Problems, Journal of the American Statistical Association 72, 320-338. Harville, D. A. (1985), Decom position of Prediction error, Journal of the American Statistical Association 80, 132-138. Harville, D. A. (1990), BLUP and Beyond, In Advances in Statistica l Methods for Genetic Improvement of Livestock eds. D. Gianola and K. Hammond, 239-276. Harville, D. A. and Carriquiry, A. L. (1992), Cla ssical and Bayesian Predic tion as Applied to an Unbalanced Mixed Linear Model, Biometrics 48 ,987-1003. Harville, D. A. and Jeske, D. R. (1992), Mean Squared Error of Estimation or Prediction Under General Linear Model, Journal of the American Statistical Association 87, 724-731. Henderson, C. R. (1950), Estimation of Genetic Parameters, Annals of Mathematical Statistics 21, 309-310. Hulting, F. L. and Harville, D. A. (1991), Some Bayesian and Non-Bayesian Procedures for the Analysis of Comparative Experiments and for Small-Area Estimation: Computational Aspects, Frequentist Properties, and Relationships, Journal of the American Statistical Association 86, 557-568. Jeske, D. R. and Harville, D. A. (1988), Predi ction-Interval Procedures and (Fixed-Effects) Confidence-Interval Procedures for Mixed Linear Models, Communications in Statistics A. Theory and Methods 17, 1053-1087.

PAGE 160

160 Johnson, Norman L. and Kotz, Samuel (1970), Distributions in Statisti cs: Continuous Univariate Distributions (Vol. 2), New York: John Wiley. Kackar, R. N. and Harville, D. A. (1981) Unbiasedness of Two-Stage Estimation and Prediction Procedures for Mixed Linear Models, Communications in Statistics A. Theory and Methods, 10, 1249-1261. Kackar, R. N. and Harville, D. A. (1984), Approximations for Standard Errors of Estimators of Fixed and Random Effects in Mixed Linear Models, Journal of the Am erican Statistical Association 79, 853-862. Kenward, M. G., and Roger, J. H. (1997), S mall Sample Inference for Fixed Effects from Restricted Maximum Likelihood, Biometrics 53, 983-997. Khatri, C. G. and Shah, K. R. (1981), On the U nbiased Estimation of Fixed Effects in a Mixed Model for Growth Curves, Communications in Statistics A. Theory and Methods 10, 401-406. Khuri, A. I. (1993), Advanced Calculus with Applications in Statistics New York: John Wiley & Sons. Khuri, A. I. and Sahai, H. (1985), Varian ce Component Analysis: A Selective Literature Survey, International Statistical Review 53, 279-300. Lahiri, P. and Rao, J. N. K. (1995), Robust Estimation of Mean Squared Error of Small Area Estimators, Journal of the American Statistical Association 90, 758-766. Littell, R. C., Milliken, G. A., Stroup, W. W., Wolfinger, R. D., and Schabenberger, O. (1996), SAS for Mixed Models (2nd ed.), Cary, NC, SAS Institute Inc. Peixoto, J. L., and Harville, D. A. (1985), C omparison of Alternative Predictors Under the Balanced One-Way Random Model, Journal of the American Statistical Association 81, 431436. Prasad, N. G. N. and Rao, J. N. K. (1990), The Estimation of Mean Squa re Error of Small Area Estimators, Journal of the American Statistical Association 85, 163-171. Puntanen, S. and Styan, G. P. H. (1989), The Equality of the Ordinary Least Squares Estimator and the Best Linear Unbiased Estimator, The American Statistician 3, 153-164. Robinson, D. L. (1987), Estimati on and Use of Variance Components, The Statistician 36, 314. Robinson, G. K. (1991), That BLUP is a G ood Thing: The Estimation of Random Effects, Statistical Science 6, 15-51.

PAGE 161

161 SAS Institute, Inc. (2003), SAS/STAT Users Guide in SAS OnlineDoc Version 9.1.3, Cary, NC. Satterthwaite, F. E. (1941), Synthesis of Variance, Psychometrika 6, 309-316. Satterthwaite, F. E. (1946), An Approximate Distribution of Estimates of Variance Components, Biometrics Bulletin 2, 110-114. Searle, S. R. (1971), Linear Models New York: John Wiley & Sons. Searle, S. R. (1987), Linear Models for Unbalanced Data New York: John Wiley & Sons. Searle, S. R., Casella, G. and McCulloch, C. E. (1992), Variance Components New York: John Wiley & Sons. Schaalje, G.B., McBride, J.J. and Fellingham, G.W. (2002), Adequacy of approximations to distributions of test statistics in complex mixed linear models, Journal of Agricultural, Biological and Environmental Statistics 7, 512-524. Smith, David W., and Murray, Leigh W. (1984), An Alternative to Ei senharts Model II and Mixed Model in the Case of Negative Variance Estimates, Journal of the American Statistical Association 79, 145-151. Stroup, W. and Littell, R. C. (2002), Impact of Variance Component Es timates on Fixed Effect Inference in Unbalanced Mixed Models, Proceedings of the 14th Annual Kansas State University Conference in Applied Statistics in Agriculture 32-48. Tuchscherer A., Herrendorfer, G., and Tuchscherer, M. (1998), Evaluatio n of the Best Linear Unbiased Prediction in Mixed Linear Models with Estimated Variance Components by Means of MSE of Prediction and the Genetic Selection Differential, Biometrical Journal 40, 949-962. Wolfe, D. A. (1973), Some General Re sults About Uncorrelated Statistics, Journal of the American Statistical Association 68, 1013-1018.

PAGE 162

162 BIOGRAPHICAL SKETCH Jam ie McClave Baldwin is a native of Gainesv ille, Florida. She is the daughter of Dr. James T. McClave and Mary Jay McClave, both of whom hold graduate degrees from the University of Florida. She is married to Ian Ba ldwin and mother to Caryss and Zach Baldwin. She will deliver their thir d child in January, 2008. She is a graduate of Gainesville High School, Vanderbilt University (B. A., mathematics and economics, magna cum laude 1997) in Nashville, Tennessee, and the University of Flor ida (M. Stat., statistics, 1999; Ph.D., December 2007). Jamie began her career in statistics while stil l in high school, as a data entry assistant at Info Tech, Inc. She has held several internsh ips with Info Tech, Inc., allowing her to gain understanding of each step of the statistical analysis process. While at U. F., she worked as a graduate teaching assistant in the Department of St atistics and as a graduate research assistant in IFAS, Statistics. After graduation, Jamie will remain in Gainesville as a Statistical Consultant at Info Tech, Inc.


xml version 1.0 encoding UTF-8
REPORT xmlns http:www.fcla.edudlsmddaitss xmlns:xsi http:www.w3.org2001XMLSchema-instance xsi:schemaLocation http:www.fcla.edudlsmddaitssdaitssReport.xsd
INGEST IEID E20101206_AAAABW INGEST_TIME 2010-12-06T09:44:59Z PACKAGE UFE0021619_00001
AGREEMENT_INFO ACCOUNT UF PROJECT UFDC
FILES
FILE SIZE 842 DFID F20101206_AAASBG ORIGIN DEPOSITOR PATH baldwin_j_Page_056.txt GLOBAL false PRESERVATION BIT MESSAGE_DIGEST ALGORITHM MD5
b162db0c06ba6490790063d2edd75d62
SHA-1
2e565837565819f8a7c7d8fdf66be9ec6c8aa5a8
768 F20101206_AAASAS baldwin_j_Page_039.txt
2ad1694fc5b13d1af8efed9c1cce837d
1ebed3db2a56216d86d2e574c31d80bd0b0a28f4
15692 F20101206_AAARVM baldwin_j_Page_033.pro
157ab832f209e1f2abb788c15edc467b
da50f64323405f0961e750c97ba230e004e048c0
21093 F20101206_AAARUX baldwin_j_Page_011.pro
27ce79ac04a7e096268028f9c6699381
60bff6f3a412909311484e5c249a7db56d5a0307
41058 F20101206_AAARWA baldwin_j_Page_049.pro
ab8289b3234afbb5cceff090176e10b1
1cae5f827f690427f9d193118954696bdf3c4f03
1141 F20101206_AAASBH baldwin_j_Page_057.txt
b731f2bf21edce9c634c083d47844d57
2b292fab37256c8ba32b4ffec1ecbb3b85fa7c92
1544 F20101206_AAASAT baldwin_j_Page_040.txt
f581c027944a7daecac37678eb8277e5
29f326480e8a3228b6d45e3fc108a9c07cf41b8d
23497 F20101206_AAARVN baldwin_j_Page_034.pro
b2a0d99fbab9f94de1e6756d1f7d8d76
ada3dce51f4c646ab3ba2b7f248873b2890cf4ff
44953 F20101206_AAARUY baldwin_j_Page_012.pro
aa9761c0356fb2a4f0ce9e3b57773e40
c308ded01b42987ab5fc88375e6e6edf8a6e64c6
38033 F20101206_AAARWB baldwin_j_Page_051.pro
e745e489fdfc7199c30c3f59233ae74d
59875da6440ca5d4d5037d4b8ef5f9f7038b17bf
1668 F20101206_AAASBI baldwin_j_Page_058.txt
fee01679de6102c327f8fac40d202a01
87ab8604e5dd5e378150a57681352f2c58753194
1101 F20101206_AAASAU baldwin_j_Page_041.txt
6a0b7a0864432005caabac8347ea33f1
82a7b54c54ddb05d84e2b94e4142ab9924a17451
44328 F20101206_AAARVO baldwin_j_Page_035.pro
417472c75df517ce3a1047c695f3224d
78a5041a80a01729e802e44e8bf77462940be9c6
52661 F20101206_AAARUZ baldwin_j_Page_014.pro
3dcf82903b6c875d171616dd28f37af0
1b2bdff25e0a9b2e00478b377b3c24ce24617f25
19602 F20101206_AAARWC baldwin_j_Page_052.pro
1b6a143eeeed017ace4ccd91921d6976
8e81375dca260fb13ebe22d41a6ae0b514e9d9fd
1849 F20101206_AAASBJ baldwin_j_Page_060.txt
8b47f8a97da072d926fbcb39cdd5ceff
313a0d411147130e08fe54a12c63df6755246769
1749 F20101206_AAASAV baldwin_j_Page_042.txt
de9cd6a60cb8606c15b7df2adc8a6d04
a20adcd0b509f690e0099494693edc3ac51f8b60
19537 F20101206_AAARWD baldwin_j_Page_053.pro
4b2c8c146bead844a22d51b824b078f2
9480bcb36392b903bc0e6b09046dfbfaf36020c5
1784 F20101206_AAASBK baldwin_j_Page_061.txt
fbc8e4a33a89ee87ffedbc72297bb2ba
d9709f2f8079ca2bf0dc5f61901cfd78d8de990a
1485 F20101206_AAASAW baldwin_j_Page_044.txt
d200a852383df42e89dc0abf9529c8df
59e2bc06745d78d1caeba696266d89eea7375f22
28457 F20101206_AAARVP baldwin_j_Page_036.pro
09bb1e72585515988acd8f67032ee9ca
9154f85fe5ba575b7f05a3c7ee063aafeaff768f
18793 F20101206_AAARWE baldwin_j_Page_054.pro
e7741d984d4960874be1b4abdf63c08c
572e0004641a2396ea81343a5386f0c10ace1fbe
213 F20101206_AAASBL baldwin_j_Page_062.txt
2753e6d0a7eac398b4a21a24b1af10d0
93ec50f7321453e14d9c85b1cd7616311fa51cb2
1498 F20101206_AAASAX baldwin_j_Page_045.txt
a153dd6b8e3febede6543c0e8f90cf42
b4aba47b20c83045c0867120c9414670db7e4040
40414 F20101206_AAARVQ baldwin_j_Page_037.pro
f3d425f73651908eb6b8d2333dd2a8e7
30d6d62c40c2da28580f0597c07c64274b1a29fc
32026 F20101206_AAARWF baldwin_j_Page_055.pro
7b15e8071c5b8803bbcfbc5ea51499e1
8b9127bb4a3cf1ceed15809ba454fa75b90161eb
1504 F20101206_AAASCA baldwin_j_Page_079.txt
7cb33d07da889af6a81c7847f48ae4ea
7a190f1c6f1d876260a54718e4f973fbea8273d1
1989 F20101206_AAASBM baldwin_j_Page_063.txt
1582f0fe44f2b174fa562ac05a107f97
33cb86d1024608956f72b34c51c69e0fbfeecfdc
811 F20101206_AAASAY baldwin_j_Page_046.txt
60a06f81acc678f9917a2109bdc48c3a
741310f00ec04f278a594bd993b2c584fe5a5d0b
56641 F20101206_AAARVR baldwin_j_Page_038.pro
f114d176d26294c3793bc54cd310cce2
1f237c6747887385cefaf10a8ce6a4bbda1baaa7
15806 F20101206_AAARWG baldwin_j_Page_056.pro
580ce2b6f80ade0a47b3decefa90213f
a025d5487b10374905170a7f41bc625b71e9e909
1729 F20101206_AAASCB baldwin_j_Page_081.txt
5520470016c8959713c2e8c786c4a582
4aad226215c6c9dc8ffef6a4fb15c8b18597062d
1422 F20101206_AAASBN baldwin_j_Page_065.txt
c01c2f9c5c1eafed3310dd0fde018355
fdb22bf0b0372716dc48f4b572f625011f6f83e9
19310 F20101206_AAARVS baldwin_j_Page_039.pro
691a03515a6505c5ed6649c5a3738a24
fd7862bb0884a192d0f643393f3ccb4cd74fcfe9
24323 F20101206_AAARWH baldwin_j_Page_057.pro
99f2f83edf4519a3c5b62db5feba8c60
dc2426ed7861166fb14c69f1ba3d921a9fb3cdf6
1742 F20101206_AAASCC baldwin_j_Page_082.txt
3c5fe2e66db4a1a41acecbc383c6bde3
7eb67f579dcef58a3ed2e2f28d4e0e0794ef9d93
1301 F20101206_AAASBO baldwin_j_Page_066.txt
c1c193c24ba8c133eb5a2d5b324ca897
07683b7bc9d453320ab3a77cdac28e3572c57ec3
1374 F20101206_AAASAZ baldwin_j_Page_047.txt
98ed8ea7a85aa02c3031ed34665f65a3
42199bcac5405a8ca1d690d57a869e35c9885023
32555 F20101206_AAARVT baldwin_j_Page_040.pro
6de9d81b1266182f1f615a26273e7eca
79b38abaee4bae9832f3dc2226adcb538bf04752
34006 F20101206_AAARWI baldwin_j_Page_059.pro
a1068b4fef8aa2234748097581e10ad2
dd262b4a761e8af6037a7a8d9be6686d502d0845
1874 F20101206_AAASCD baldwin_j_Page_083.txt
a220da7977c29166878c16dae65b6ef5
fed59a8febf14f64dc10ead652ff41962f9ce999
1078 F20101206_AAASBP baldwin_j_Page_067.txt
582a064c8630c9f8f41200b9e6187540
a4b3ab9d28b89346bae08d7c5e123461dd1e9693
41048 F20101206_AAARVU baldwin_j_Page_042.pro
4cc214372eb78903b2bfe3d85ae2216c
8501483f928e07d55bbb0ecf293ced546c3cccaa
44133 F20101206_AAARWJ baldwin_j_Page_060.pro
196dc11a668de567df9a8c505de8324d
0a092fd95d7f6e729e29926ee22cf9615ec93e35
1688 F20101206_AAASCE baldwin_j_Page_085.txt
b02c00322e71494a5ee4a2081e2f3973
95ea42e8dd1550401ded601599b430ae5ab147ce
1692 F20101206_AAASBQ baldwin_j_Page_069.txt
c63bb8994850cfb45c778593638f2869
09ed44061bdb2e671adf974353ed326424127d3a
29145 F20101206_AAARVV baldwin_j_Page_044.pro
8ee3e5464aac5e7ec353ee31bf555516
d16690e2fea2d56df2bf222ed9a5d7292004f0ca
5258 F20101206_AAARWK baldwin_j_Page_062.pro
d0405840ce451675e46d6a42e7848f73
a063023c623ef74b7dc0d42afbea8ab7863999bf
1647 F20101206_AAASCF baldwin_j_Page_087.txt
bc098c8c75867d1e79387c466e9a88f5
795fe39615a6c12e58be37f6ed1deaba742e3431
1912 F20101206_AAASBR baldwin_j_Page_070.txt
495e4728728fdc2b177b65262ab45b4b
ec37def725aa453c93a397ad89725c43d0246e26
32669 F20101206_AAARVW baldwin_j_Page_045.pro
7580e0a08a7203535eea3b0baf452cf7
319465bdd803ede66631ac0b233c930f9830bec9
47086 F20101206_AAARWL baldwin_j_Page_063.pro
e9de6f55945a97e89b95b896a85896be
b45dc52011e65a9e81bd76cdddf6619e978012ab
1867 F20101206_AAASCG baldwin_j_Page_090.txt
a3cfc8c55fe2f723a77373b6d1a72e15
102c86a615ca5e3a148971bd16bfb48bdcbbba7b
1770 F20101206_AAASBS baldwin_j_Page_071.txt
217a2770c4d9fa487c85b7a5b7606290
53d677804e87234e27514a174ccfb3316b519fa5
19312 F20101206_AAARVX baldwin_j_Page_046.pro
03d848801b1a2d916284ae6cdb3ff31a
9fc59954a00e13db5fc513a579f93684e7bff876
24904 F20101206_AAARXA baldwin_j_Page_080.pro
fb5c132a520570f536486d8295b09381
3a68e241bd6ab2db2fca8cd976efe91330e84c7f
33112 F20101206_AAARWM baldwin_j_Page_064.pro
bb62facfa8fe01e2d53f32c1d0dd401a
ba62c2021ce885c3c62aac57102ddcba9e57ce80
F20101206_AAASCH baldwin_j_Page_093.txt
2be1cdcac28fb008e27796e26ca18371
170c036c48d8a07c09cc8c2da50b7f61e294fe5e
1328 F20101206_AAASBT baldwin_j_Page_072.txt
15077f56d98ac02d72f2501342a06b18
6654d89e3fed94b44d64ff0aa19f345468b8723b
29374 F20101206_AAARVY baldwin_j_Page_047.pro
0c62c7043d36d2c4b2de37d4c55d208e
8c22218c8f2d880e8c0f20e8126dda089e016a0e
40348 F20101206_AAARXB baldwin_j_Page_081.pro
927c8229425524595d060a0d607ca4ee
cc5a5c101c9094b78126337d8c004b7105226f90
21508 F20101206_AAARWN baldwin_j_Page_065.pro
618497fac8b178ac985b7b9ff7aa11a1
8fcb93c42e1447cd2c0924f135e9f6dece3448b8
1320 F20101206_AAASCI baldwin_j_Page_094.txt
a1605e5b9d7c5ae69caf910bda398f31
1d905b7d9ce9e0fbaad91efba78ab288ebb57838
1508 F20101206_AAASBU baldwin_j_Page_073.txt
9168e0a0f52ff0c914687c50749e875d
01c600fade7a971f2b624c25ba589c14d6aa1e21
20097 F20101206_AAARVZ baldwin_j_Page_048.pro
c6c477a645ddc4ca5ce8009426d99e11
88954f3dfc64d9859e01d3092322b7691351807f
44008 F20101206_AAARXC baldwin_j_Page_082.pro
edd7959bac28e944346f1a1a69885b91
628233c4adeb733a1ee03f1953adbc72ea99c185
21674 F20101206_AAARWO baldwin_j_Page_066.pro
7be9fead9d80c883d35c0fea210b955d
491198e17deb8bbfc6db86e56fab970bc5283acf
1515 F20101206_AAASCJ baldwin_j_Page_095.txt
1070bcbe42c5a045e769ce681e0eb905
584e00cf57b93c42b6fed58d852e94e4e48e3276
1596 F20101206_AAASBV baldwin_j_Page_074.txt
bfd603435ff98589694298d53b9afc14
3daef551f86a3dee58bf4c55157913819fcad20f
46061 F20101206_AAARXD baldwin_j_Page_083.pro
cd91217e3d0edfc2d079da95a62d7cb7
e17c59eea7ec94bc06a7b8071f6d044da6e87ea3
18824 F20101206_AAARWP baldwin_j_Page_067.pro
57f2592dd43ee724cf52333da29f4ae0
7bdabd27b26c9044a5848fb7847fa08fad57723e
1840 F20101206_AAASCK baldwin_j_Page_096.txt
bca1365ff6ed1ec168ab26fbbc2e2a04
8d8869e336af3e63a7d46783f39edeb63f671b2e
F20101206_AAASBW baldwin_j_Page_075.txt
ab224831b5a754197af71b0c36c297f9
b9ca89dc748c1ec989964f7322a874443873dce8
37830 F20101206_AAARXE baldwin_j_Page_084.pro
f13bb5300caac439ff6722a317d5d534
9f4405f3947efb2f7d8fdd3e009e44a828778f10
1614 F20101206_AAASCL baldwin_j_Page_097.txt
160ecfd3fbf03bdc4561392a4ed8899c
250c6ee3562528bcde7f70460a6314dad3c777e9
1220 F20101206_AAASBX baldwin_j_Page_076.txt
d8679128a2cc14cdd0f25d07da12cd28
e0fc2c868282544dd1ed2aa079b67fea85565064
38017 F20101206_AAARXF baldwin_j_Page_085.pro
edf447638de8babe12fcc7c0d45be28e
d459fe1b893f5f539f81c74dc5d4429dda43c186
36062 F20101206_AAARWQ baldwin_j_Page_068.pro
62265fdabb94dd53c6a3d9c1b8b8459e
2ca2f3ceb626188a54b699a685f10d2264b5c91e
599 F20101206_AAASDA baldwin_j_Page_116.txt
8ba3cb24917390f566c751f1a2570c98
3382dd73c0e5fe4edd2de48d7e8b75ce6a45c850
F20101206_AAASCM baldwin_j_Page_098.txt
771159e512c30cfc4402644ac06db547
f590157575f280bd9bf75a6603bc3fa9de385c3f
1699 F20101206_AAASBY baldwin_j_Page_077.txt
e9186cf62b8daed6c80bff37cc595b95
d21b93d42cea7410c339d816023d9131ecba6034
19915 F20101206_AAARXG baldwin_j_Page_086.pro
d82230b8b8432c06ef2a71719930c795
c0eb17ca48eff648c7ad5e6c7377b6a040f1f070
36052 F20101206_AAARWR baldwin_j_Page_069.pro
037b8396b2b7d70af56676af99844ef9
489a723ae37ab8ff0a3d36b0bba5da5fe4bead30
666 F20101206_AAASDB baldwin_j_Page_118.txt
fab0274490ac08af5b0fc98000a14004
1809e4616fba400b80d67ec276b976c8f8ae7ffa
1831 F20101206_AAASCN baldwin_j_Page_101.txt
f6d1fbbd58af9bfd344e5fc59350e6ea
d214fdee9dfa624967a4f0afae0f3befa5cfb148
1579 F20101206_AAASBZ baldwin_j_Page_078.txt
e210d314d16cdd4a1c51b7608b86f765
e01d93d850b14b2548e77837b609aeba65872179
37977 F20101206_AAARXH baldwin_j_Page_090.pro
ef7bf21acdc0b13e6e43402f057fb7c9
7fe0047bcfa1492eaa309f13c878cea781e6d5eb
43270 F20101206_AAARWS baldwin_j_Page_070.pro
abbe378169ff17ee7028d842d08de668
34e88d887a839f0414fa04ab81332df42fc68327
856 F20101206_AAASDC baldwin_j_Page_119.txt
b25342cd79cb2d08d2bba505a58e5e62
028c83d54cd255572fae482d72dcf5c12995eeb8
1746 F20101206_AAASCO baldwin_j_Page_104.txt
3dff4fe603fa12fbc65c828c16c766d3
d795de606727d45a65b82595e303ff91b18fc546
41756 F20101206_AAARXI baldwin_j_Page_091.pro
1f2180cd69ca56cb97f40a76ca6f98f3
2a9bb874ff5b24ae26e34d37529f0f0b319e9410
26623 F20101206_AAARWT baldwin_j_Page_072.pro
a1e28c9c4fbba685e2856005b92cc14c
767eb04a1ecec2b6041556eea98ecc70f2d0ee69
4806 F20101206_AAARAA baldwin_j_Page_078thm.jpg
8c0ad9a51f8461a470fdd5613465fece
72f8f17600660356ff4d6e7c3260af8b380298f0
260 F20101206_AAASDD baldwin_j_Page_120.txt
2078e93cc3f01bcaf17ebb7593504385
9bc7455662cff287ec9bd23d952a3d5cc4e38449
1960 F20101206_AAASCP baldwin_j_Page_105.txt
fc5c305325cec52d7135c12e2b7b400c
8edc73518cffb470ecd90f4505936a5158428144
31001 F20101206_AAARXJ baldwin_j_Page_093.pro
b09c0b69f598f926ea3d8f9b02857453
2f993a1bab89a29dd324a304dc1fbac2b3cf6b81
31019 F20101206_AAARWU baldwin_j_Page_073.pro
73f5db68234e1ec80f2397743ab5282c
aab5fec8320c0f9d1fe495419d2d56f66f74d92b
63340 F20101206_AAARAB baldwin_j_Page_074.jpg
df2ffd5c3961c12b3e25e05a5c49ed5a
751b776a5d6833e001720203dbfa047d3949ebe6
457 F20101206_AAASDE baldwin_j_Page_121.txt
7276db36f394bd899151f92b3da32baf
3d44655d2767878260e1f811340d173a037007a1
1950 F20101206_AAASCQ baldwin_j_Page_106.txt
93177f95215ee6c1ade5e769c4e0eddf
237040fd78fdcda6d6a2efdce052a0537890aecd
38455 F20101206_AAARXK baldwin_j_Page_095.pro
f669488ec7bbead185e6dfc18bd9a6e2
20ed6f7265b14ed908121a124a7c2310c670de91
41329 F20101206_AAARWV baldwin_j_Page_075.pro
2a5cf762a04d61facbea08b14c9229a9
90d24d91fbe01262c2fe9af5dfa96343633b1a6e
1248 F20101206_AAARAC baldwin_j_Page_034.txt
2c15f85be4d5f091a93fde7f9aaac047
e034d67549723c3f7a4946bea80be63a0b3bc6d3
681 F20101206_AAASDF baldwin_j_Page_122.txt
97862457ccf15faf326813e732298f6c
179e8395018a5d2dfd979ccc7e45653dc2fa1e38
F20101206_AAASCR baldwin_j_Page_107.txt
e3907de47595433424725b85f2020600
a81a78dbe8db545b1d9a1722280969dd2880b204
37727 F20101206_AAARXL baldwin_j_Page_096.pro
ef94cc4a6f6a495a39f4b4770a4a5889
f20061ebf4abdab7898476bebd3e357e63811c63
25837 F20101206_AAARWW baldwin_j_Page_076.pro
c1d7b560c39c5cf28b6d4ec79fdc86fe
3aa06a36e68f1a5276bfe9412414451eb5812554
5209 F20101206_AAARAD baldwin_j_Page_079thm.jpg
f710f0e39f9c8f77cfa62e85d6437de0
fb0093be346b4f4dceaa6e3ae1d91fd8035a486b
540 F20101206_AAASDG baldwin_j_Page_123.txt
3a8f0e318c668491c878fb482ede58b8
71b6bdc5ac7c049f9792b516871ccf449686a54d
1321 F20101206_AAASCS baldwin_j_Page_108.txt
65b3175d79be04378f7e0bb33c1d1593
5f88f3d72fa1c86961376204f956c40377ac82dc
35218 F20101206_AAARXM baldwin_j_Page_097.pro
5b66380f60df7d934a23d12e1ac2c047
2b6fbaf42059cb370aacaa52786a62601a430e3d
30596 F20101206_AAARWX baldwin_j_Page_077.pro
7a3beff6f0e0d55b00c26bd720e13a36
ac424dfdca925a79aa6c1333909b079c056eadc2
819 F20101206_AAARAE baldwin_j_Page_127.txt
f962c8cc27b4f29b40ece1f47ba5b12d
5fbd8d532a8e213daaeaeaba765c5762e29dd57a
8662 F20101206_AAARYA baldwin_j_Page_112.pro
91a0adfd301d96c858931f0f0ddfb55e
8646602123feb7f8e5b4da7aefb2b2318477d147
359 F20101206_AAASDH baldwin_j_Page_124.txt
c79d3ebc00c9c0716821d1171d8f0f9f
34d51af7dee069193feb70bd2a042b0889016674
673 F20101206_AAASCT baldwin_j_Page_109.txt
3944a44a39b76df3de31e29742f7f44e
05de4688e2c66e6baa11b38b20c084ef51f90e58
41505 F20101206_AAARXN baldwin_j_Page_099.pro
66fabdd18d5d3ceaad266cb9f02ed2cb
d8214a124ce5b1a5e5d719f0fa3267189e3c51a3
26407 F20101206_AAARWY baldwin_j_Page_078.pro
35d36f9b6fa4c1bc652a0dfc5ef7e8b9
666b43cf7a6071a95163996223d21fb59b4230f7
6410 F20101206_AAARAF baldwin_j_Page_081thm.jpg
fa48c2a6fa145bea56a56c31974d5659
65245770cf9b798f959749929f01b6e6ddd901a9
14933 F20101206_AAARYB baldwin_j_Page_114.pro
3ffd916ccafdabf464e0b5b7e8547ac5
dff837631f9808800f62ed7465e80b81dfc98838
317 F20101206_AAASDI baldwin_j_Page_129.txt
25e01ec106a4da56345add919a0f2a12
57c1ff7f4c7ccc12f30fa8ae5803abec28cf18f8
390 F20101206_AAASCU baldwin_j_Page_110.txt
f4b9fa6d5ebebf00291d875d36540a0c
338ef6719296bd1b37cff66c58552f54e00994e4
43095 F20101206_AAARXO baldwin_j_Page_100.pro
bf40eacb586ea29a490c37dd4366b7b7
8327b264708527721b74cae02f409096b4b29553
29100 F20101206_AAARWZ baldwin_j_Page_079.pro
651d7fec1bc650908404bc48dd439f1f
3f77340dcaa759ec716dcca59b0a372e18af6b71
1800 F20101206_AAARAG baldwin_j_Page_017.txt
c67d782301302158493a39fb5aa3aca1
d8065bb9fd10bd74878076800867b4c40b430104
10800 F20101206_AAARYC baldwin_j_Page_115.pro
2db18329e58afad42fd7d0809e4c4606
682b7d455d7496eb5c4e668f489c33603ab3c54a
195 F20101206_AAASDJ baldwin_j_Page_130.txt
528451591761d43dd286bf0101c40c4e
aa745f985ad4b3cdcfdad22ce56b8c8fef323b57
321 F20101206_AAASCV baldwin_j_Page_111.txt
026e51e129d9881ba11b43dae58dda1d
b6147234b544d010ebf9b1cc04bafcad715a9249
44238 F20101206_AAARXP baldwin_j_Page_101.pro
c41bcaec5a7f8cd10aaa9830bbe3da48
cc21af9cb936f778bf61c46b7e866d4e2dc0948c
85205 F20101206_AAARAH baldwin_j_Page_007.jpg
3d60b72f675e648eb1dc1de81a9d9129
11875d451e20554ef0e99b96b689f6df66b179af
10910 F20101206_AAARYD baldwin_j_Page_117.pro
88e47ba1a01ec70966ff846ffaefdbe4
a8b92d5ffe70074da7f08f5aee9349a23cfb27fa
229 F20101206_AAASDK baldwin_j_Page_131.txt
0eecc2ee25671e78cde6102ca2e7b441
2ef1bf790cd4062c8f08ef89d4d2285d2c863909
468 F20101206_AAASCW baldwin_j_Page_112.txt
6c51030b0769518089c91426b7f9f4f0
fd1601c40bcaa9d618ba85b51b3b9937e0af8149
35258 F20101206_AAARXQ baldwin_j_Page_102.pro
31c0bf12cde39d79dc0787af2cfa4b66
3a887181ba82ff1a54eb843c3a976abe297fbcc0
1115 F20101206_AAARAI baldwin_j_Page_088.txt
135062fd9853859db01e0462ae414ad3
7d894594627ed9c39aa2a85f7ddfc969a0d40a95
12179 F20101206_AAARYE baldwin_j_Page_118.pro
d7c744a449e7b229d15486f9e2714949
4a2795b9be146f03f05598fbbb59de6e63f73ab2
313 F20101206_AAASDL baldwin_j_Page_132.txt
25eff9783dd9cce9764424a1b586a151
2166e4acee6ffc7857ad25a095e665c15b30306e
300 F20101206_AAASCX baldwin_j_Page_113.txt
d700a09334a6d31c88c14133af038ee2
0f2c637e2a2eebc2175f7c90369afe384d43f15b
25271604 F20101206_AAARAJ baldwin_j_Page_039.tif
4123def7d1ef8f1cecba46e2cdf83b5d
e70b945a1312967a215c9564b821ed445fdd87c7
13031 F20101206_AAARYF baldwin_j_Page_119.pro
6593e6d57c4fbb8b071d1da1da8a4df1
dca3ed404ea093901743d04562a369bf3c25b835
203 F20101206_AAASEA baldwin_j_Page_151.txt
f55942cc431c048847830cf4f9b7cfdd
f79ef3eb4e6535935e87af05dd1d72a0aea9a47e
368 F20101206_AAASDM baldwin_j_Page_133.txt
8d184da07fab243becc2026154f4b477
67b1de2f6afb34fd5735f2b2eae8d07bbd0b1e96
964 F20101206_AAASCY baldwin_j_Page_114.txt
5479b4aeb70c744744d13a48f3044c1e
2c472c51b0ded9471cf59965204181c720c15f19
47555 F20101206_AAARXR baldwin_j_Page_103.pro
28b3e7da979dd6f897a92d5f34f409fb
e87039f8ffc0e7aa986df2b3cba89a45c45b4634
2934 F20101206_AAARAK baldwin_j_Page_123thm.jpg
b9ef8a0c3cfd24e371b6c0f4fc9192b0
17040a8810f9ed412f7237de594de9c5b51c910c
5150 F20101206_AAARYG baldwin_j_Page_120.pro
1531c30f8699d489a2ad98c3511129bf
64a80ce3d34fc3eca805f9d69f69176631091fa0
F20101206_AAASEB baldwin_j_Page_152.txt
047156126a0ebff1a832c0221498c5a0
bcf404c7120d6f58d76312816d707258ab2ccb10
411 F20101206_AAASDN baldwin_j_Page_134.txt
13881bb270f306e2906ca7c15f15b38a
9768957313c491319c28abcd73ffa340834b554c
610 F20101206_AAASCZ baldwin_j_Page_115.txt
a9d6044d11b5949e312719fdf8c2e96a
3f37379e598a1125751dec889f2f363dde189cf2
44343 F20101206_AAARXS baldwin_j_Page_104.pro
4384e44f62396b9da70ed39a3fe8896e
cfcbdddf1f7ed5688b2f4de89f8272dabdc5886e
22567 F20101206_AAARAL baldwin_j_Page_042.QC.jpg
03c29fc4d947fae1064e52426e2df01b
fc4ad0b831c9928781da3827aff5a0f5b84c1c34
5531 F20101206_AAARYH baldwin_j_Page_121.pro
1674324a5165f0a362a8cfcd88bbedcc
5126be43cc87caa9e3f5d2c493e37f5fa1fcf338
1555 F20101206_AAASEC baldwin_j_Page_153.txt
31c8d4c68a447fb9cb1d65a1eb92cf16
8719e5d2448d5ca2d0b86144215bb22c31ae096b
F20101206_AAASDO baldwin_j_Page_135.txt
a4dc871b94e2334f786ab954af66b39c
f05289ed66fccf972600455f26500317f647d89f
49699 F20101206_AAARXT baldwin_j_Page_105.pro
7989bbbcedf413db1e0df4d77372d402
5506d65343e9f0cba54c1dceb513a51be642fa01
F20101206_AAARBA baldwin_j_Page_137.tif
ffff27c19663527d41d6148fdf9d8ad0
d99b7a3acc22ca26ab7fe5e6ba91ef52607ec8f6
877481 F20101206_AAARAM baldwin_j_Page_027.jp2
67ce761182bbc7bb18fd8c020951b840
eca5ee6ee31d6c60470eb95a12d46775161388f9
12326 F20101206_AAARYI baldwin_j_Page_122.pro
43da52dfa75d738873d49b896ee0bc58
16a28d782a67a283be9ce7d65217154af3704a78
1117 F20101206_AAASED baldwin_j_Page_154.txt
233a66a6a2b40e86212316aa9d9438cb
a05e6a7d6bd007073cdb1875113c17af7b94c33e
2178 F20101206_AAASDP baldwin_j_Page_137.txt
26fdc93f02f0a24aeae734072c8397b2
5937ceba4c652ecbd5db7092a3745dd7033e18e3
49430 F20101206_AAARXU baldwin_j_Page_106.pro
0331f0f9c18c2c183bc602be3b69a6ed
6c6d21353422f7603b1e6d46647b88beecd666db
292 F20101206_AAARBB baldwin_j_Page_125.txt
bc0963e0c33eaf005d5232c18e2ef877
a35e91d5bc9e3a04361fdb8d4e3e6a6f9b8de306
629 F20101206_AAARAN baldwin_j_Page_126.txt
841e9848d7f07016e5543a4018f95bc8
3c8ecec9c534c1b73504de86a099c97cce71e553
9921 F20101206_AAARYJ baldwin_j_Page_123.pro
129362bcc68251414f2179ee8e1f1e5b
41786b3abae05174b8cb6a2c250524622e0a7f55
1095 F20101206_AAASEE baldwin_j_Page_156.txt
58132221d7beff9cc8774e420a3356aa
66b294375aa38a4c714bd5f6f1cb756b801468cd
1952 F20101206_AAASDQ baldwin_j_Page_138.txt
a03f095a2c95f42fed24d908ff9ddef7
1141d0be7e1dfd75780a30e40b39568663240a6b
47244 F20101206_AAARXV baldwin_j_Page_107.pro
7db2e409530340a95901d369f6679816
7e7dd8dd68ad55267f6a8a92246f9d7e18676e86
3700 F20101206_AAARBC baldwin_j_Page_003.QC.jpg
c345751b7c0f4abe196a87119f9ee28a
8a45643d49e729cfeb6f0f36012d5d0cd9e74a48
48164 F20101206_AAARAO baldwin_j_Page_077.jpg
557648cc83ed0d82a0ca6b188d632387
c41a1d485d920c8d4d86f0aec2e09dd4c0da9713
6403 F20101206_AAARYK baldwin_j_Page_124.pro
423c052c6d739a48f2fe082ff15d9e8b
10bd7058ffd9752da5060b01af3b43ed5e5b1704
1938 F20101206_AAASEF baldwin_j_Page_158.txt
038d7696c3b4123add386f446ad74bd0
21bdbd2d20cd0b213b7aab37ebad2f2c40d0a5c0
1967 F20101206_AAASDR baldwin_j_Page_139.txt
b6c0448b5bfeb3bd0284ad58fb99a96d
95c653a995d29dfaddc94db83d3bf37688c1975a
29385 F20101206_AAARXW baldwin_j_Page_108.pro
95e2abc615791d37d66e8f2e23143fb5
1fcb571d33463bb3f915c3cdffce0e7f72e1ca5d
20035 F20101206_AAARBD baldwin_j_Page_084.QC.jpg
d624c132ab4fe0b231b39bfbfbb18de6
9bc0388a28dfe89fcfbfad5273245af983fb6cc3
43901 F20101206_AAARAP baldwin_j_Page_017.pro
18b211f06d3f017493bf7ab729a75a01
b39b93e2dce8f1da0a4ddbe9b461a78941948f14
3822 F20101206_AAARYL baldwin_j_Page_125.pro
59d3d9cb3be48969e591c03696f77df1
b3f7a63783870441500e9e536f66a0e87e042081
2472 F20101206_AAASEG baldwin_j_Page_159.txt
8252b4a85afa811ef300133213ab6cb9
c6076b62dd0428e3a2446c43c1563527fc5e5b66
2110 F20101206_AAASDS baldwin_j_Page_140.txt
9dda48fbbc6845ef34f6f3f2b0445f8f
d1e150506b60e6e2f258cd5d785e9ded3f9d0e5c
10576 F20101206_AAARXX baldwin_j_Page_109.pro
b24dfdc0aa9de9ea5f2df07f62def6e2
d0b23f6a0e9bddd295365f188671eee49a3c387c
1663 F20101206_AAARBE baldwin_j_Page_144.txt
6c0b88312ca60719e33527e9d4b7e11c
9e3743947e456d628ba5b34aa7e2348702ae2353
6404 F20101206_AAARAQ baldwin_j_Page_109.QC.jpg
174485c90daf9543425035e4002e2278
1e82d806d75b21ab75ce2f603c8cc7c36c186ed0
36607 F20101206_AAARZA baldwin_j_Page_141.pro
32c32d286c1453bcc0a030559f9a864e
7ca1c0dd6c4a3f98f7463cc7e416afc6c59d6f49
10060 F20101206_AAARYM baldwin_j_Page_126.pro
1996ee7ef2ce4e39cbe7ea7af9bc1c1e
6c56d01306885ff6def92cc7d2d6802bd193e6a5
2275 F20101206_AAASEH baldwin_j_Page_160.txt
e9381517abee8cddb9dc7f07b022a9e7
5db9ae6131164662c2af8abbc8d9f20960f0b14a
1774 F20101206_AAASDT baldwin_j_Page_141.txt
387273351c1aa79e0fbb8ee3d6ee4e8e
2adb5939ef689f02d63caf22ed85fd72f715608b
7685 F20101206_AAARXY baldwin_j_Page_110.pro
3fccf14becb40adda53d4cca4f5fd5cd
e36ee9fb4c6783e28fd547cd09741b83f8c9fffc
4680 F20101206_AAARBF baldwin_j_Page_088thm.jpg
907b97f9f8dba204bea4d3424b91b6cb
85e9e21475d441ca2748cc42aeb15ffa47106f7e
1880 F20101206_AAARAR baldwin_j_Page_103.txt
c5eecf6fff8c7a4eb36c651e52438eff
63387ff2c4c70aa09ab6d8c67976987f89bf248d
19663 F20101206_AAARZB baldwin_j_Page_143.pro
61781f76cef419f8189bba660eaa359d
0b74295f2a18c4be233debc35004685fbb71b9f0
10127 F20101206_AAARYN baldwin_j_Page_127.pro
a1b116faa441d63ce2beb17b7ec42e2f
2f5a26f4402c2580067881180523fca47184b8ca
1724 F20101206_AAASEI baldwin_j_Page_161.txt
0ed633a6a784155be89084a19b5ab48d
a506856f63fa95da53b33094fc357abbd8a4c254
1653 F20101206_AAASDU baldwin_j_Page_142.txt
b0f47e1ef7d55255705a8afd9961dbff
7fa84252f8695a071ac65bfee3c247b871f33d69
7085 F20101206_AAARXZ baldwin_j_Page_111.pro
d4041aa8db548018aaea81959cc5d4ad
829e99f0fbce68ec59a0818f6bfdfca14f06a7e5
1947 F20101206_AAARBG baldwin_j_Page_016.txt
2fdf44cb343af5027fb6168b6da6a835
97f67abb73433ac915343dddf792eb9b66289a69
25078 F20101206_AAARAS baldwin_j_Page_106.QC.jpg
e6545a8b1d1804b7fb7eb6ff59888831
4fb35050de92df0a243031c8f78ba431861f0476
36794 F20101206_AAARZC baldwin_j_Page_145.pro
543ecb435ab2ee006319ec730b5a468e
7375eb9d99a180feb7c0d3153b7205f931d21d9f
5441 F20101206_AAARYO baldwin_j_Page_128.pro
fb1a82b71d8cbbd7037f6f7424b4a213
75ceba72a8132217cc0246465f1db6fbe22c9948
1164 F20101206_AAASEJ baldwin_j_Page_162.txt
3e2869cea54d647cbadccc2c0ab7e051
fd667cf609b70575cd7fde6936760fedd1983a09
1319 F20101206_AAASDV baldwin_j_Page_143.txt
26dead20cc28fdb6cdc5763f4b348f7b
b306d3a78cb793a7a996562daa27769dae973afa
1072 F20101206_AAARBH baldwin_j_Page_052.txt
26d01225a169f342fa9e4336dba13448
163d975253efd820dbb6f14dbd93095fc95f8972
460555 F20101206_AAARAT baldwin_j_Page_065.jp2
bd162bab3f57e2a2bcadf29b97568f30
2d49b840abc0fa29f4ad199e472d5fda56dca4c9
54990 F20101206_AAARZD baldwin_j_Page_149.pro
7f9bdae5cc9c405bfb33ffa5f7da7f08
d8b76d136a376cf384be297ac4539d79b3ad5d96
4539 F20101206_AAARYP baldwin_j_Page_129.pro
252ea2078cbcc3b57aca1ed71d808570
f255f6fb290b9d3efed5f9c597ac96e403c62656
2383 F20101206_AAASEK baldwin_j_Page_001thm.jpg
ed9dc699f7371744550ef8967805a426
8c89d7cf1274325512681cc536cdb8429431e555
1891 F20101206_AAASDW baldwin_j_Page_146.txt
d7c760cb88e81064ce71f4f590fc97b0
0e216d624411595a25d620e34dca9b91d719b16c
1406 F20101206_AAARBI baldwin_j_Page_080.txt
7daf010d8e20f558434d87eb02eaefe2
afa48724f02f54804192479647fa037dde62b3e0
978877 F20101206_AAARAU baldwin_j_Page_100.jp2
648ae380cb899bd5847ecd5315432923
49093ae4ecefefc25a607aeefc9b4210c68739b9
53959 F20101206_AAARZE baldwin_j_Page_150.pro
86d4b1ad690d6c48eb926d4de20fb408
59e8e8ab320124fe87df1696305df441b6253de8
3337 F20101206_AAARYQ baldwin_j_Page_130.pro
5884b27485bdea9111445dc2b46ed159
1f54c2b267e33154121f0f46a931fc0b37e04e7a
1074325 F20101206_AAASEL baldwin_j.pdf
ddc25e68d20c537a789bbe1ffbd13e9b
a338b30f944bfa3985344ec9b7f83b435c1e2515
1934 F20101206_AAASDX baldwin_j_Page_147.txt
685bd5c90cc34f5b85b1fe518225f535
de43393674230830af90c0c951a33d32f738af26
118856 F20101206_AAARBJ baldwin_j_Page_151.jp2
ac187a69778b3a9e1f5cc70b26c85682
8a909b14dabfd55c38f4574ad1f0976df49b1b55
437740 F20101206_AAARAV baldwin_j_Page_114.jp2
ad6d03235d0b97aa383a63762af63873
8a335e019314ac0c183b29fe751457f870e13131
5061 F20101206_AAARZF baldwin_j_Page_151.pro
8fe31727d932a4487fc90fbcdd426c77
7e3812fb03a86dd35428535fe9f3a4ff87509a10
5407 F20101206_AAARYR baldwin_j_Page_132.pro
bc48c0fb94a1f356a8dbd5134164fbe2
1b0a2e3c35b679eb7b42769b0e9e251e2d04d361
26870 F20101206_AAASFA baldwin_j_Page_140.QC.jpg
f256414d4e9302dd7bd03b965cadb01a
50d31affbdf0d61eeff302c64deb718bc6bad79e
2564 F20101206_AAASEM baldwin_j_Page_124thm.jpg
b6e0a940bcc6cc20abd022bdbeb74884
2d42f96a60c616bc1fa45782df9e07d2a4e2b778
2008 F20101206_AAASDY baldwin_j_Page_148.txt
8b14c0dc37b365aad5da4c0b4403d610
c685999c296baa35fedfc9a9ea574c27c8e8db6f
F20101206_AAARBK baldwin_j_Page_093.tif
03bf20940527fc1850a322b6adab68f6
9c5932ff2dc1efa99755c2a45e50facfe74ffbfc
28587 F20101206_AAARAW baldwin_j_Page_094.pro
b0876922f6368d249a3eddd51eb16103
5a353140ca555c94f9d972081fa49ef761ff5030
28424 F20101206_AAARZG baldwin_j_Page_152.pro
b822983ac3ced8a87a89aeda2bf7ec18
0efa3fabfcd2d18b59e19618c8b3082c5846ff52
19073 F20101206_AAASFB baldwin_j_Page_064.QC.jpg
2b0c6b28ddf17cd458e280cf86fd6bea
b7a20ff75c9752f82de67374e25e1e16d99e447c
6947 F20101206_AAASEN baldwin_j_Page_029thm.jpg
2c740ba605d8480c4c1ba6f38b36dd83
5b57c99deccac0b6e186ea9a5fbf14047aa0cd5d
2159 F20101206_AAASDZ baldwin_j_Page_149.txt
68cc619fb4c8fc2b7bbc5ebced5c077c
a8b0d02a86f89367a4cadcdfde093ffbed7ddc13
46430 F20101206_AAARBL baldwin_j_Page_034.jpg
e99cd9945bfb3e69566eebe50ff3ab28
76f1403e17c1b6e60c36fa68c75dcbdbf5520ab3
22569 F20101206_AAARAX baldwin_j_Page_088.pro
10664b29134c49e42f88b4fbff1500ff
76ee6990ca250c1f7e75baf770f2f0004825b737
27361 F20101206_AAARZH baldwin_j_Page_153.pro
40368b2b2aa5ff7eb79e5171fe547bd6
81a3261391a95245b6664e0291223333be7b509f
6345 F20101206_AAARYS baldwin_j_Page_133.pro
db9f6e1f6329877cb563abfeb60d23e1
d37beef14e19c41c3b2630d5cb1f578d428a9d5f
8074 F20101206_AAASFC baldwin_j_Page_134.QC.jpg
3a29cf5f72d5c10a6d88c515d8cd3c4c
eedc4ef00eb9e1aac4d8496d81730ede997c651c
5361 F20101206_AAASEO baldwin_j_Page_153thm.jpg
7cbf0f2685c1322e0dc1d2869e1eeed0
58747897ecdbebbf88c9c2c907ec7e680adceef0
65996 F20101206_AAARBM baldwin_j_Page_147.jpg
362aa2e5066e3e2d28734764056dd5ac
9e349232df2701018aed36ada78d8d8a85ebcec6
48519 F20101206_AAARAY baldwin_j_Page_162.jpg
301c86b7f0defc815a93f95e90a37536
4eb2d8033b647e13ee4c6d0a47af7f3e9b0b0012
19612 F20101206_AAARZI baldwin_j_Page_154.pro
67b11c3f4f21c305c6607af0a3b3734a
f84090eddeefc5bb12abba474948a1e32350fdb5
6192 F20101206_AAARYT baldwin_j_Page_134.pro
bd412255166658a200262666c5488f09
73699f8a88e8cb1328c034e003a484cdf868c9b2
19693 F20101206_AAARCA baldwin_j_Page_158.QC.jpg
e602d97d2f150321c0178433aa28f549
deddebc387981308fa0fb0b344bc13697ccd4299
8122 F20101206_AAASFD baldwin_j_Page_001.QC.jpg
9d2a2cc9bdb44269e6d82c2c0f166cf8
459cf5e47d992e5833a9db07ea9563cdd35905e1
8184 F20101206_AAASEP baldwin_j_Page_133.QC.jpg
a17b195120f988716e442a9131127c6e
ac0b6c03e44e8a1f0d91be2648611b223ca9c403
F20101206_AAARBN baldwin_j_Page_066.tif
677792c6381f9b298bfbf81c9e73e2cf
c4090bb8987433091411a99edd602aa2841d7325
1610 F20101206_AAARAZ baldwin_j_Page_068.txt
f265db645627f9e86a61df5a86792f0b
e2041921ae409bca1bdd184a0e10abfecaa416dd
33598 F20101206_AAARZJ baldwin_j_Page_155.pro
e20c8a6212aa0a88979cc5bd5cc722fe
fd4ea67d29a038ff17d8a24f9a77e36100cfb35d
5820 F20101206_AAARYU baldwin_j_Page_135.pro
994bcaa2d47fe30eb2a033b9563a07cc
af60629f5e283fc3c6fb1d596b892ebbc5535248
60503 F20101206_AAARCB baldwin_j_Page_069.jpg
2c4e6fc67a9d1c3f34ad60d5212b7cd1
6c82c78c1bc4c3a1de391960f2181ff3e540d297
5982 F20101206_AAASFE baldwin_j_Page_007thm.jpg
498e936fdceb489d8bcbf9346bc89761
15788d7fa4906164f0e83c26c2ea62578ff6821e
6080 F20101206_AAASEQ baldwin_j_Page_102thm.jpg
9ba7d940b021898769f44e3d7b4e6e29
54af1e9207e94654e02b30f04b0819d52e6591b9
39373 F20101206_AAARBO baldwin_j_Page_142.pro
143fa8b4d49d86c816ec1f6c8bd8575e
f0a463d60fbc6c714688737688ab7e7e4cd22191
1578 F20101206_AAAQWI baldwin_j_Page_145.txt
ab388b4c46c8abe244ef080c615f5a8b
d5bf1134f5cbe8353e6f74da10badcfc3cb6d1e1
24725 F20101206_AAARZK baldwin_j_Page_156.pro
885fa85bf9a399c01e3eca286b30ddf9
dba9fc211b39eccaddbb847d0f5108b7a9b74686
10908 F20101206_AAARYV baldwin_j_Page_136.pro
a2261d9579d997e04d7d96fb0b64caf1
aab29d422df952f3fee08d85c298891254316634
7559 F20101206_AAARCC baldwin_j_Page_122.QC.jpg
0647e4838960dd795b60489200dfd66f
361b20ed9fd77903b31e9ede8827cc6f50c21d7d
26482 F20101206_AAASFF baldwin_j_Page_137.QC.jpg
824ef1b7928a28aaadeaa9c92bbf5dbe
ea31e3e726b0a494c20019c5c555c4bf254b615d
2510 F20101206_AAASER baldwin_j_Page_110thm.jpg
b015ac3a030a9c06198acb3c32e2d8c1
f0c31255387f14a192f0772441c0bf617ddc8e72
25265604 F20101206_AAARBP baldwin_j_Page_109.tif
4755f0c195a0a09884bc6dd423658438
5cd18656ce08c7aa845261bf15171b6d0b08a67e
18648 F20101206_AAAQWJ baldwin_j_Page_019.QC.jpg
6925269aac5a23afec132e859b7e9178
272388747a408ebff01fabb2b2432535497f7e0a
40122 F20101206_AAARZL baldwin_j_Page_157.pro
802b50c3339440049945b79b1e93bf26
a2a7c1d3811faac9ed0d7fd626ae98fab0914581
53147 F20101206_AAARYW baldwin_j_Page_137.pro
791c8938ece6043dd6276c5c75cd641b
3e3fefa14ddbdf2ea295ac59fb295e9b64deb45c
7645 F20101206_AAARCD baldwin_j_Page_149thm.jpg
7d01ed3585073ef5ff4de7a6dec172fc
0b490fdddc7a76e297cfda1a9275ed901908f707
6023 F20101206_AAASFG baldwin_j_Page_069thm.jpg
cd484f3b1491fb6d0ca1e61b930a5774
6658317b7f0c378fb3f38f4e0c59bd4d68c3684a
24257 F20101206_AAASES baldwin_j_Page_101.QC.jpg
5c0305fe7299a0f302fff59fcab9cc16
f8fa5f41ab5948216023c3cfc158647657d02d8a
669496 F20101206_AAARBQ baldwin_j_Page_047.jp2
67dae05f52f70314cbfe0829d67ff5a6
cd5663f34c6763d847168089c0a31aacee4cc015
63259 F20101206_AAAQWK baldwin_j_Page_096.jpg
1c34f6813043abf8582ce90f823507fe
de3f32eccbb57911e959144e7f05b5ceb98c8d35
46135 F20101206_AAARZM baldwin_j_Page_158.pro
9c163014361521f4a1cfa7f8ad1e941e
5e0007f7b544a85d7a9b27d2e51af667c199ce04
42778 F20101206_AAARYX baldwin_j_Page_138.pro
93d575137c2396a8909847431c299538
448927abdaf32bb93e908251fa35c91b1779a11a
669 F20101206_AAARCE baldwin_j_Page_117.txt
febbb12cf53bdbc91b7b15f9e5ab4a38
fab532dec4d5fd381400e1a057dea85d46a5d282
21892 F20101206_AAASFH baldwin_j_Page_017.QC.jpg
b939844985e0a902b8a48d99eefd896c
f96e79d939775f259bb5c60eea6209aecb0e77b5
2698 F20101206_AAASET baldwin_j_Page_135thm.jpg
d8619a2bb0bd18086ac1736599c55b8c
f032e8ff4ad7a2e03e43e6b64cbba236efefc921
7336 F20101206_AAARBR baldwin_j_Page_114.QC.jpg
6270d5e24bc5ed3b5b99c57f899e1803
aa887366f0e44de8c42c13e7ff136feaddfd2973
95 F20101206_AAAQWL baldwin_j_Page_002.txt
016a4da100df461fb9cf5f1f70713b69
322afaeedb2fab2e08b2b2dedd829738c5f8b992
64065 F20101206_AAARZN baldwin_j_Page_159.pro
b7cbc81a82b22dcd012df1c8499a6e06
2bfe2030d6c79f13d6eb9202ee724db1fe2810d8
41558 F20101206_AAARYY baldwin_j_Page_139.pro
83fafd85b4ca8a7b553a58abfaef9bf2
11456ca2d14c2f15bc0b67fa9e531de317b83233
1369 F20101206_AAARCF baldwin_j_Page_043.txt
b7e1aa3ffa49e420aad91756d8c66f93
9e47c7569be0fe4c90d3183537a1a46ca8fa2059
18061 F20101206_AAASFI baldwin_j_Page_043.QC.jpg
63edd2b159371582acdf5df0a07ace22
66b315d8b60150f330188171256b027932b689a4
2958 F20101206_AAASEU baldwin_j_Page_133thm.jpg
6c2da7ca8bb60867b8f327dd5fb6468a
283bcc256d4edb561ec7f1bbb5785306a45ec8de
1503 F20101206_AAAQXA baldwin_j_Page_064.txt
9423ef2ef7b00622ae9e69b0dd9645f0
335369d119696f40877b6baa241c55563fec4f04
1051944 F20101206_AAARBS baldwin_j_Page_106.jp2
b597caba21705ee2a32e1094e9eab778
af5412e4f907ded00cbcd43255b5c7d3ab162a79
2875 F20101206_AAAQWM baldwin_j_Page_120thm.jpg
f7e07e4749ad5d2c3b0d454ec467e56b
75dc2cb43b3c207e24eac100bac2b91de656206e
58356 F20101206_AAARZO baldwin_j_Page_160.pro
50d8f8259e54966d26cb7fea117a9fb1
7691d11ef49e1a2bee8d6e2ed8b9a3b3a047b37e
53649 F20101206_AAARYZ baldwin_j_Page_140.pro
f80939a43b1a3006ceb0b52ccaa96639
e840b48be0cbf109b157f60d3df61a42052cfaeb
40963 F20101206_AAARCG baldwin_j_Page_146.pro
ec050089a35f00b6502b96a1ba695bd8
7f54df6a2d9707f70913b8a59e10894c44f81cbd
7762 F20101206_AAASFJ baldwin_j_Page_041.QC.jpg
1c7b9f678f8bdd50123a4b79e717b670
8a91222213a18d69dc22cc5a3f2115dca6c5ed6e
23516 F20101206_AAASEV baldwin_j_Page_035.QC.jpg
5a2fc4bb4ff7dad518efa7281d0a2c31
69c3defb206286d00ae4760ef7ee0254c051a8d6
36690 F20101206_AAAQXB baldwin_j_Page_033.jpg
6eb318ce39a9d5bb0c1dbbf845218057
81b8f5eae6cb12a12d931bcea7ef2a342418b914
F20101206_AAARBT baldwin_j_Page_053.tif
b811ad5bd30e2c9fbb0b5b4e95b80349
7b2f1d1fbc81d49c5ad3cd2c2bb2eaf6e7a893ec
12244 F20101206_AAAQWN baldwin_j_Page_067.QC.jpg
edac5b2a57a5f27151ae15483902d75a
10d980a71eab6103d5439b3c8e258bba41a520fb
43859 F20101206_AAARZP baldwin_j_Page_161.pro
b991f6bb4c5d913e58547ab16204f96a
104752fdbaa351770c2a4abb93432ff5f8c11dac
7464 F20101206_AAARCH baldwin_j_Page_112.QC.jpg
48b1faf2f4ea090627d909a351b3c303
de1ef8262952e4ad3ddc698761a60f6b572fdd5b
3597 F20101206_AAASFK baldwin_j_Page_131thm.jpg
02806ac6fde5e6f2ba8c2a709a00556c
734af6f480f77c6b06103e0ad9feac72f0f97995
12916 F20101206_AAASEW baldwin_j_Page_046.QC.jpg
cbcc5f64720e031be61f609cee878c3c
ba267c4af55a93cdd377cc2a8d098f977df01419
1613 F20101206_AAAQXC baldwin_j_Page_084.txt
5101235ceffa1b64807620100da2c1cf
14a1e51bf759e9856f8efc1c9ad6130c8f4003e1
44774 F20101206_AAARBU baldwin_j_Page_061.pro
7b734b919d704f247413aa074a931ee7
e859af84db36911eb2f57c6d05f4ca22c7ba5ef3
F20101206_AAAQWO baldwin_j_Page_141.tif
a5766ad498e87632f0e5d32475600802
248bb585ec483cbc614c56b89cf7a13e0b2a260b
490 F20101206_AAARZQ baldwin_j_Page_001.txt
cb39e225babbc41a1a81d232582698c1
ae31e52db828eed92fae214a5afcec2be48720bf
F20101206_AAARCI baldwin_j_Page_107.tif
07b2147ed37f679395d28f742dc553d4
833084dcce4bf2beed452a80d5bba3de2743658b
6833 F20101206_AAASFL baldwin_j_Page_113.QC.jpg
de4de1e2e1775bd7b590e744cc5d54cd
48468fc254e0cea7fb5394c8e320bb562061eaa2
16135 F20101206_AAASEX baldwin_j_Page_036.QC.jpg
7e89fd6cbdc95169f7fd57fdd1964d2e
d5782c9fe75566a8f7c83ce41e681b17b3ec8931
15912 F20101206_AAAQXD baldwin_j_Page_078.QC.jpg
716a53b3a3c2882df7ddaecc0b716153
75e12cf129c4840aa3c57f25db686a95f6bbd127
F20101206_AAARBV baldwin_j_Page_024.tif
3671150f8ff50fbddf7677dbb15e2db9
288bbb73a821913e475e796d9deb1e7416d805d2
527 F20101206_AAAQWP baldwin_j_Page_136.txt
091de8a88de2b5497a77f8037cb2cbbe
f68781f1722c4f8a24ccdf03d4923cde32be82e6
130 F20101206_AAARZR baldwin_j_Page_003.txt
39c64c38d16fed841947dca094f1bd04
db87f62e29885b7116f75b9d959159e3010dcc55
46770 F20101206_AAARCJ baldwin_j_Page_022.pro
326b98566e3154d5c4d7245bd0389dc2
269e30e0627e28a6901fbc71e3095c9d70808571
16282 F20101206_AAASGA baldwin_j_Page_079.QC.jpg
8284af1b020371ad6ebf3da8dc83c4c3
c1ac52873fe2cc56f32bfdc112da25ac5ec917b3
5393 F20101206_AAASFM baldwin_j_Page_043thm.jpg
b98690a475c720812bc44527bddb1db0
2d25f434eba40bb4b4beb05272cd28794eb0ac17
5704 F20101206_AAASEY baldwin_j_Page_058thm.jpg
ea87344198c9044f650d0466ae65a9b5
f45288e312770f86075702ba30c25ec6e3c18ae4
6259 F20101206_AAAQXE baldwin_j_Page_090thm.jpg
0670e4c636d9de50585877ee79d201d2
c44a7f671fa1f79376db9a5af7dd965110039b9b
2742 F20101206_AAARBW baldwin_j_Page_117thm.jpg
4cd9678aedb58100aab7ab92ed22b56c
2086c40b5b7663d35902ba52024c9440e64c0a29
6225 F20101206_AAAQWQ baldwin_j_Page_070thm.jpg
fc544b1c320c5b5fe7328bbbf2d7c9c3
b93aea80fc9501cb07c71007bf123e98579759e7
2185 F20101206_AAARZS baldwin_j_Page_004.txt
e74234373135c2f84d95c2998b8be656
bbe56471a469e204043a590b6b860361e2ccfaa4
15631 F20101206_AAARCK baldwin_j_Page_151.jpg
4a3f2be15ea5ad89b7ba93bd0dc8f8b9
121553f564f816d1239145c6d5700fd20538fb37
5225 F20101206_AAASGB baldwin_j_Page_151.QC.jpg
a50e4e8cd21d2f5525c78af45e7d2e3c
78c8678e866fbac0f42b6b8c2f0565efe50fd2ee
1592 F20101206_AAASFN baldwin_j_Page_013thm.jpg
6e768389c38fb8e3f6e45a3f9eac646e
972d74de217b4e2f9e6e37fb1a904900bfb93db3
F20101206_AAASEZ baldwin_j_Page_096.QC.jpg
445e0e26725f821284795c9efe938fd4
9bdbff155361132fb7947f28374211fd65fdef61
84505 F20101206_AAAQXF baldwin_j_Page_006.pro
1c3de055dc792bb34a81d7cde414a9b5
0e08123927f605ba82b8ac51d58b780b0d4096c7
11593 F20101206_AAARBX baldwin_j_Page_143.QC.jpg
eb5c932600fb842ea71ddbd86d8eb8a5
db339c8f7ce6e8df57d39d1f5ec47e58062fb308
30381 F20101206_AAARCL baldwin_j_Page_043.pro
165effa936d487567aeadcb8ec14237f
810efda924ca0d36748f5fb1e9832909191efbce
4937 F20101206_AAASGC baldwin_j_Page_055thm.jpg
686ba043400871efb9ff397e2e8efe87
826e6daa80ac4630013caec7f04914174031c2c5
7554 F20101206_AAASFO baldwin_j_Page_150thm.jpg
2476612b9cafe35b18bf751f28aa25e4
5a26c0eae1ccfe9d30ddc3190b38c08721142dfc
49241 F20101206_AAAQXG baldwin_j_Page_148.pro
753ffea0f74f657899773452ec082534
9292189b63cd1a6dd58b4c4fcb15dd335cd25e36
1571 F20101206_AAARBY baldwin_j_Page_003thm.jpg
b262f3eaed810bbd151c306e38eeb0e3
c7e0bd861a9478834dd70ed2a3977142d2136de9
385766 F20101206_AAAQWR baldwin_j_Page_120.jp2
50ca4a2e7157c5eb29f991be664f983c
6dfdbe1cd07bf32d98d807847e74b3423aef8441
3527 F20101206_AAARZT baldwin_j_Page_006.txt
241f9ed10aa661dfbce7a1cda00048b1
dd76dba30115c25529534fffc71e43f699ead8d3
5528 F20101206_AAARDA baldwin_j_Page_152thm.jpg
2f475edeeb01d260e432c8d8b0d8980a
f106b40a9c443f712887fe2bf96111b2cf8d0da1
3167 F20101206_AAARCM baldwin_j_Page_002.QC.jpg
8edc02cbf3f54196802646bdc70f85b4
1474ffd02194cbe1e47eaa129c108d5f88f89b36
11094 F20101206_AAASGD baldwin_j_Page_131.QC.jpg
5687c9252dcf6fc5a58ad19b2ee8858d
725cdef892c78fae9e33cb425bcb9f9e0996a530
27120 F20101206_AAASFP baldwin_j_Page_149.QC.jpg
2e75cfb3d9437cf80ca841152f0468b0
a0dd0438a98c616ee60ff76c51b449bce89b6cce
19791 F20101206_AAAQXH baldwin_j_Page_008.QC.jpg
7c9680e24a644312ad84e4852a5b479b
75efcef98ab3b0d8846b015e452e07e6296ee38f
F20101206_AAARBZ baldwin_j_Page_001.tif
44f8112e911275ade77ffa390be0cc2e
b04d96d5a36e65760ee518bba57fd9c237f00655
475065 F20101206_AAAQWS baldwin_j_Page_136.jp2
4fdc2098225a2937e6712bb192a345c6
ea9bf4e82bb2eca68384b30ab67814b1ef0110ae
2090 F20101206_AAARZU baldwin_j_Page_008.txt
65a5a9dc8c3daf1ca0cb95981a202ed5
bbf83a9b308ed669f937e08ecbd23c92dd99fdcb
5298 F20101206_AAARDB baldwin_j_Page_047thm.jpg
8fa97bff4b742077929631ee11c96c6f
847c9be26d8ecb97adb7dcf1fecfb17a612fa20d
1331 F20101206_AAARCN baldwin_j_Page_089.txt
424881092e595b503625167366da2bfc
795b2a9e60a3aedea02a8df8f4247f04fd512f69
17671 F20101206_AAASGE baldwin_j_Page_087.QC.jpg
211fc0659c405a1c11593ca029b12cc8
6efb672ae0d33532d5350c08f9ef170d93508239
7046 F20101206_AAASFQ baldwin_j_Page_100thm.jpg
99ceb41cbd0ae0faa1424a77bf08f810
21644988dbd46cd513663c2f52505bb92c0d68da
F20101206_AAAQXI baldwin_j_Page_126.tif
3349d49dbc54a42962237cf7bd69fb1c
fef79baac83f84fad0bd02ef5b5c5cd851865585
2674 F20101206_AAAQWT baldwin_j_Page_122thm.jpg
5d0d7a0a51947251a9a87a5a92bf8e98
d56196e6d7aa97998d1d164cf26370bc3073149c
2310 F20101206_AAARZV baldwin_j_Page_009.txt
e413ea131cfd4489e9d86ae3f95556ec
d64ad2d983f73cb42b7a3e4a3381fe575232a78a
16097 F20101206_AAARDC baldwin_j_Page_125.jpg
6414aba089fe5dbf3d5b2f0363fdb535
93ebbb22fc14a280ec2e4dcd07bb2aa8a9693be2
24617 F20101206_AAARCO baldwin_j_Page_015.QC.jpg
577ac8e8485e3182dabf3397b02c2eab
9839cd84b3f14573c493e1b07b84d9f7f678a373
18356 F20101206_AAASGF baldwin_j_Page_152.QC.jpg
05af9c2faa9d18e3cfb041666a7888ee
8977a08b5ebbf92d585555743ca9c650f0f8c724
4219 F20101206_AAASFR baldwin_j_Page_046thm.jpg
aaf68c18222a3244e43fcd7bbf303282
e5442f782f7ba0bd2a56a5a88d2de570985de136
309 F20101206_AAAQXJ baldwin_j_Page_128.txt
49a0da0b52b25724ac9093cf1d377d9a
c09f36697dc9df5a1510975b154e221e23fb5287
6832 F20101206_AAAQWU baldwin_j_Page_042thm.jpg
59f702fe4583ec7c8e12be5f695ce044
30274c785fb6b15ae3d1aecf525b53ce33413fb7
2203 F20101206_AAARZW baldwin_j_Page_010.txt
6d12ffce496a6bd5b61291467a64e50e
9f978cfcabcc7d0bf18e2189da51d62023e42615
19588 F20101206_AAARDD baldwin_j_Page_141.QC.jpg
a0a33d19b06ae9aad9ba8b45e8101417
635a9a5e984271a7aeaa83a3d799d6e8999d40a5
29648 F20101206_AAARCP baldwin_j_Page_144.pro
270bc5c3ab32e88f39f938e1f77e0a26
af3407c9098d808bfbda830a7b724788d56923c2
7303 F20101206_AAASGG baldwin_j_Page_110.QC.jpg
7439834a6c20a1506ee6b3bf4fe42b5e
af9801775a19e3ea6258ad4bfe6064d241c64386
16404 F20101206_AAASFS baldwin_j_Page_072.QC.jpg
f5dab0a1014d3c4d7b1451e9894fe147
dd62eb79e501661bf6b853e251fd9ca1c6877d48
F20101206_AAAQXK baldwin_j_Page_047.tif
658424f0c7760d305e3546c4e258962e
0711419cdbfbc0f793ab97a7217fb368bf5c83b8
1051981 F20101206_AAAQWV baldwin_j_Page_148.jp2
f51d181e88f2c5501611657ec3a870c1
e30567618a1593e2e0f573ffe06aebe48c0df61b
123 F20101206_AAARZX baldwin_j_Page_013.txt
26166f03c40f06afc236ddeec3a6b283
e6c97f5911326371f4ca752543057b047453d9ec
663904 F20101206_AAARDE baldwin_j_Page_055.jp2
b25586ae75ba2496fe3ca2d458673377
b73842e7ec5d3f8b107687cc0246a8f869f1bdc7
36775 F20101206_AAARCQ baldwin_j_Page_039.jpg
d51542308ed2e3487a1df5e08b63f5ee
6858be7739097c6b1ae12d78a63d76f87997679b
6905 F20101206_AAASGH baldwin_j_Page_091thm.jpg
6e9ba88db70544592f6ab10f6fe03714
5973aaf588c3d9d710365ea58cbda04803ed08ca
5291 F20101206_AAASFT baldwin_j_Page_044thm.jpg
ac44e1f485cdd44bab9c7d3ba5130a92
70979d5b6e9a7bb703714946c3e537b23e48f501
25773 F20101206_AAAQXL baldwin_j_Page_018.QC.jpg
e19d21e168d40343ee806876108a9a3a
bc5e7a8a33b20b949afdfa65fc562f889927e063
F20101206_AAAQWW baldwin_j_Page_008.tif
7ca74389043e5383240e61b3ba8bc84f
9b159f6e742983b36be90023436f50374fd2c45a
2162 F20101206_AAARZY baldwin_j_Page_014.txt
9d8d6b6b0373c2705e613760e21db40b
b21367d56ee2ba0a1c162a1768644a16b1a524ca
882908 F20101206_AAARDF baldwin_j_Page_068.jp2
ee60fd287384949ebb7de9c3089bd887
c8a8e781133bba8079471e7cbc94abe4bb718235
2121 F20101206_AAARCR baldwin_j_Page_150.txt
823f30aa4233114a2948e8331ee255dd
df27f9ba1f88a4c3630e840e04876fb1577e777b
7849 F20101206_AAASGI baldwin_j_Page_159thm.jpg
a0927929c4b65785f9c2afb664edac57
14d72fdb1d16a1d961eadc956a62931a31dee9de
7875 F20101206_AAASFU baldwin_j_Page_111.QC.jpg
f3e842c97736b9630dc5e1212f6c2438
34b945dfa92445200db14ed8e436b9a328b351c3
1695 F20101206_AAAQXM baldwin_j_Page_157.txt
aa6fa360378d1830a8f592d34ce8c566
90b6385f000dc682c296cd680f4c5c2a01074fa8
17882 F20101206_AAAQWX baldwin_j_Page_050.pro
6dbdc3028974e0a5880fcca28aa57033
0470099f0949111ccac3b31f7e9796d91ab638cf
F20101206_AAARZZ baldwin_j_Page_015.txt
0e4237d5b8b3461355a62e2ae59626b6
ea95f2c7a48a02ee9185131e2b487b38b6765a0d
1051951 F20101206_AAARDG baldwin_j_Page_029.jp2
3637d0f94467ed65663ee7297a10dbca
bb0fc7ce05a68eabc422c3948aeecf6811fce0d9
23554 F20101206_AAAQYA baldwin_j_Page_114.jpg
45d5ce2e71aa9487b93755ef3f82b67c
a0d9c19b41da3eb9b54405f60871c26661bd33c9
4705 F20101206_AAARCS baldwin_j_Page_032thm.jpg
d02a95cec6644fa2a5676106c6332ff9
1009d96c442821535df594b5f4dd288ebb42854d
1339 F20101206_AAASGJ baldwin_j_Page_002thm.jpg
d95d8dca284fcb7c392d51deba05b8c3
7308fffa782147edd0cfa27790b61669e647c75f
21404 F20101206_AAASFV baldwin_j_Page_147.QC.jpg
ee27ff16b5ca1ec4c95a790a2a6d68ca
fd68471cd1bbe26c1dcecd69e990487eaef203d6
44095 F20101206_AAAQXN baldwin_j_Page_147.pro
b4f66ed65e8f44b374d78d1e7972bbc2
eac0ed278be65260e345d5a533122f22e49856a2
600022 F20101206_AAAQWY baldwin_j_Page_080.jp2
117908b57c303f2e0cada4e06a6a86eb
dbc6e516cab7afb35da9b74d36ebee53d2e66479
1948 F20101206_AAARDH baldwin_j_Page_012.txt
61f77ba4b6764fa283cd6d4611bdf225
d5e73a4f02c1ce9bb713e699ef63397ddaa24bfa
20697 F20101206_AAAQYB baldwin_j_Page_089.pro
f85d865ece3c5ced747a8270a5723d33
5bb86be73cd056743e2953677a7e4b1b9878bf51
6135 F20101206_AAARCT baldwin_j_Page_051thm.jpg
91339999492c1fd711fe8dab46355509
b3b3fefe43cd962e95069be9139b43d6c4259999
2997 F20101206_AAASGK baldwin_j_Page_129thm.jpg
cb04dfb3ae4dce2e4de6579467649f43
2121b10cf28a04297f442ffb3fba58f6b520f7fc
15960 F20101206_AAASFW baldwin_j_Page_153.QC.jpg
20e33dbb3ec2631ec47f37bba69691ab
292367fdf4f2bb63572f7b9006df9cad336394e4
69996 F20101206_AAAQWZ baldwin_j_Page_099.jpg
fb2efd2706e2192e6763774869aa7351
cae8039b6e8a382b6f19c39c4162ecfb5de6e14f
2940 F20101206_AAARDI baldwin_j_Page_013.pro
33165318675303e22a6ef041f0aa260a
77de381027411312b04c8a59f747f459eb9beeba
1051982 F20101206_AAAQYC baldwin_j_Page_009.jp2
09fdf80b965ab83136ba371a358d4fe7
93f9ce878798117313aa562539ffff83c5dda73a
F20101206_AAARCU baldwin_j_Page_105.jp2
9cce67c865b116c021a57292863d0062
34ddb98e2935f526e76b156beef239526d7ca20e
3926 F20101206_AAAQXO baldwin_j_Page_052thm.jpg
c48040307fdd62c9ea1098ff0f400f9a
f0abe2164dc0ccfe006508b48e39a7699e65dbfb
11553 F20101206_AAASGL baldwin_j_Page_011.QC.jpg
3f327940a50e7dc39075964d20624069
575d9b2f933af8640de2e8554a2ef2dcf44a03ae
4334 F20101206_AAASFX baldwin_j_Page_154thm.jpg
33ef1965163b6104d48a45cce6df6fa5
c3e6b5acf959ba593df2b730b0d98af525dd1591
1051979 F20101206_AAARDJ baldwin_j_Page_005.jp2
c14abafe6e422086cd8e90ed889d3f52
19e9221103cfc04f91c903078c161ac037e0e775
1689 F20101206_AAAQYD baldwin_j_Page_059.txt
4617861ae955dd0e631fd7eebf919565
01e7316ffbd89cd607f49d6439d713ad2bef046c
803054 F20101206_AAARCV baldwin_j_Page_146.jp2
ccaa4d9d3b2f3a8eee18d265652a8faf
b374cd3dc36e5c91bc9b1edf147ccea0fffc39e1
1051974 F20101206_AAAQXP baldwin_j_Page_083.jp2
4d1e3f3de0b5a80ea90521e6f8435aae
2f5e3a9152bb3c3a978cbd088e5135ffc654c24f
27891 F20101206_AAASHA baldwin_j_Page_150.QC.jpg
4823e4f2261af2ed6f6624e583ed1b89
837d6aa2be884ce3abf5d214fea335761e4eeb09
8132 F20101206_AAASGM baldwin_j_Page_119.QC.jpg
7e71f2643553b5e2710b54749ef7f12d
ad3ee85a4364a4f04df02e71e27e285651090fe8
13777 F20101206_AAASFY baldwin_j_Page_154.QC.jpg
ce052a1a81152b04a0ed6ec7cc4e5d48
59c31ab4c19d4ba54a0e99f14b159d563fbbce8d
5561 F20101206_AAARDK baldwin_j_Page_094thm.jpg
6fa2a7cdd17621e0d690fc23e8b1facf
2d3e07f0b4c0bdf392e2cecc5774b5f5ed7e213f
2244 F20101206_AAAQYE baldwin_j_Page_038.txt
6ca107f97227f21f257e7d488d0f7aac
e44c9a0c230dcb81b850a8f091bb766cb448297c
F20101206_AAARCW baldwin_j_Page_099.tif
63ffc3eacf216e011ff3f4c76553203c
3e6049e241d1953ced92924e8cb8982ccc326436
23724 F20101206_AAAQXQ baldwin_j_Page_126.jpg
4fef8a3c18c10ae761f7e515b084e97a
45eeeab470c0d0092dad3c0a0617974fe907155e
24676 F20101206_AAASHB baldwin_j_Page_107.QC.jpg
cb6b9be8626bdc51571b907bba3cacda
9e3d4315c91a9a1e81caca162e66077b9ca1610b
9953 F20101206_AAASGN baldwin_j_Page_040.QC.jpg
4cdbf0f00ea1eee28de5482f3c74b847
3a9694cf6059444914ca97053c6635e0af8d5567
2727 F20101206_AAASFZ baldwin_j_Page_111thm.jpg
8de71ba4bfcf2843725542c40574ca95
83fc7d08d20d49321d777a56567b788554ea3dca
F20101206_AAARDL baldwin_j_Page_119.tif
0b0b867d50edcca1674952539521c0f0
b95d66443815db9d222e6ac8acc6ef56792d24f5
21715 F20101206_AAAQYF baldwin_j_Page_115.jpg
900f66ebee21607872a11910a88e4618
1ec228ea22b4a2bd0e64bb8112750a5adf536498
4503 F20101206_AAARCX baldwin_j_Page_050thm.jpg
674c0d1005f5725c9f44739dfb155dd3
25d84f7eaced27462bd1fb5dfde7fb0453477061
7065 F20101206_AAAQXR baldwin_j_Page_005thm.jpg
6fa33635c2420160122245590c3643d5
2e3c99649f1318fd884745ef7283829bf7d5e243
2632 F20101206_AAASHC baldwin_j_Page_116thm.jpg
f65beb39823bde412b723f8b8bd096eb
2144deb62ac503fc947504cae3164f7ad9984f91
23896 F20101206_AAASGO baldwin_j_Page_020.QC.jpg
9cf8d0860742dfbac29beadeaaf1595e
02ee96aa10406253aa71d728db8b6327a524b7a9
F20101206_AAAREA baldwin_j_Page_062.tif
805ee84ec16668ce3e334798db9b896b
f2439e3699f16a71fede85da1399efa81c032a0b
68107 F20101206_AAARDM baldwin_j_Page_070.jpg
55ef2e5b25ba024d6efd1f923748400c
71f20bcbbc80da56cc73513de2a72ff0e6535cd1
671809 F20101206_AAAQYG baldwin_j_Page_073.jp2
248c294f88def1bea5c4639a60fc7f17
e767162a8eea0e26188b9d59619d9d835f1f3230
1018 F20101206_AAARCY baldwin_j_Page_002.pro
751e8a4964a71c24927009aff0f8e55b
0afb4547e92c0e4ab418b95531e1e57d17831f3d
6481 F20101206_AAASHD baldwin_j_Page_101thm.jpg
e973d6154090ff2590839887fbaadd45
307353fc8c2bc1a88f816b95d37f3b5813963408
17172 F20101206_AAASGP baldwin_j_Page_073.QC.jpg
52a75c27380d5d234b47c73c84234dff
04a3f4fd28b540945233f1050221e6c9c572c944
34185 F20101206_AAAREB baldwin_j_Page_098.pro
0b0d06df7f4ff962400982bd8dd6dc53
8683b7e1e2ee0a1f335738721aebb37df52e7fe9
41937 F20101206_AAARDN baldwin_j_Page_020.pro
ee980eecca09e000ca74404acab4e167
7422c7f1772414fa3fbd15c1866a08922ee1217b
1419 F20101206_AAAQYH baldwin_j_Page_155.txt
c15d581980e245312b87ce38448d2078
e6c75a9c2699861d4ed329aab8c80de44e7b6349
25933 F20101206_AAARCZ baldwin_j_Page_103.QC.jpg
fcd60bfe08d8bce237732874990d35ee
402fc7b4ca0abd0bbdf486dd7e18c7234b431e90
1051986 F20101206_AAAQXS baldwin_j_Page_012.jp2
953489cbefe4f9f2a82a52a697461853
e7a5e84782653b7eb8f36487d6db88e2ce28347c
6216 F20101206_AAASHE baldwin_j_Page_023thm.jpg
65924eeff99f372d2e43c0fedd3ceeac
705a714cbf8dd2ee4d77825f85e8f3bd483637dc
2762 F20101206_AAASGQ baldwin_j_Page_132thm.jpg
141929cefa16b0becdd18844593dc8f8
ea3a7c4f9ad2cdc14dd4d22135d6fa258268a19b
849450 F20101206_AAAREC baldwin_j_Page_074.jp2
c97f2ecb2b50f4052ecc76a31a1cf269
488fde1a9825b083492302884dc7f1231dd1942b
72537 F20101206_AAARDO baldwin_j_Page_082.jpg
b7254d818decc1c6163dea32e410ee31
29340dbae08d9fc61180350943edc72d1f0f837a
7639 F20101206_AAAQYI baldwin_j_Page_137thm.jpg
c936c92e613bf3a5f7dc1a53ee28a104
e9b99f853f9d3ca40b016a199685beb7bde105c1
39957 F20101206_AAAQXT baldwin_j_Page_071.pro
c2881397d109a1bc8e1800ec08025742
b5475cde35ff28acc1b7a9c9a3eba6deb24a5f58
2165 F20101206_AAASHF baldwin_j_Page_121thm.jpg
c2e5571f20a3a78924ef21c9a124d9b5
a84062d20ddb840fc1aff7409e21f798144a0adc
2591 F20101206_AAASGR baldwin_j_Page_115thm.jpg
6fe1517075800f1160f5c936e50a180d
1bcf0bf2dc847ee1789239a8d81fbd0e2c8cbef4
12928 F20101206_AAARED baldwin_j_Page_065.QC.jpg
65d9855474df3d1349e5de3ce069111f
42d82622238df2bc3be00a765fbf9968967cee9d
19243 F20101206_AAARDP baldwin_j_Page_145.QC.jpg
46013e1eda0e718807f72c17a5a5df69
e7496e83a03501ac58ba1b2dd2c988cff41a4ba4
F20101206_AAAQYJ baldwin_j_Page_074.tif
af67d5d58732a6226acf5f1c91032d1b
28c0ef7d3d276729617d367fbc6c3b84255fb301
56529 F20101206_AAAQXU baldwin_j_Page_009.pro
05e710cfa55eaf748cba6df389d59c30
79971299541af35b0db8c1aad5c03d94da61d073
3862 F20101206_AAASHG baldwin_j_Page_031thm.jpg
bfa37ea52da282b83f9442f295feecd4
8409cc7a4032131062179dfaf3c9adcbc83cdeaf
5278 F20101206_AAASGS baldwin_j_Page_036thm.jpg
1f7aa458fb41edad1846c8b78ed2048f
60f522612b7bead32cf79da56de67290b6654d94
31065 F20101206_AAAREE baldwin_j_Page_087.pro
bc6864019c46f6abbdd8ae88aba95a57
8ded851301fceab76db80dd4261c19c46f7d79dd
1869 F20101206_AAARDQ baldwin_j_Page_049.txt
0b4a4d6aba030dc7d8ca9e7d3c031ef8
8491a8487ff233892e0f4f777f0ae1940cfc5046
1657 F20101206_AAAQYK baldwin_j_Page_091.txt
dcde74ccbdd77847392b24018c6666b0
e4cd310e3cf5e1b5db6927d45474cbe2eb655381
F20101206_AAAQXV baldwin_j_Page_150.tif
6edde70e091a026d08dba6bc5e96373d
927f41c5bc2c8014da9dc61e6328a44b29f3c52d
7165 F20101206_AAASHH baldwin_j_Page_103thm.jpg
5f77485828d3429e4a6694e7eab7e29d
cdeb5c4e4954b27c277e5e6ee51e673a89ac7c86
6937 F20101206_AAASGT baldwin_j_Page_035thm.jpg
1c1e3060c7b93f4414484cbe82a19381
ef60a83822268d980e4a22eb426e97e506180c00
455560 F20101206_AAAREF baldwin_j_Page_067.jp2
07eb024d52404eef2984d0665913ba20
847eac1d9349dd8b6c336344a2ea0a52b6172686
F20101206_AAARDR baldwin_j_Page_005.tif
e7ae462e2072efacc7f06823645efce8
5525937c1f24ab076bcad8cccd5addefeaeb791e
F20101206_AAAQYL baldwin_j_Page_084.tif
daee4708c394eece27532f552d7e90fc
a1aaa386736cb4cc3c9b4c9cf97fd56c97bb73f0
2836 F20101206_AAAQXW baldwin_j_Page_128thm.jpg
5f0141239f836acde444186a5a2d6b5b
83df68b78284bb5f3b937cf759faa929f0079c3c
16014 F20101206_AAASHI baldwin_j_Page_047.QC.jpg
024b254558106b3ee57b4090ea9f1b2d
b11587ab415fca35fa96c19772a9a33035cc01c9
4963 F20101206_AAASGU baldwin_j_Page_072thm.jpg
a4dea0309bc11439307b34b1573177ca
c45313b48f6a0b6309d50acda91ce1da9d492ee7
5989 F20101206_AAAREG baldwin_j_Page_138thm.jpg
94d6808a7cc27d1fec8cae7cd0a89d97
a449e6c455b855456da004877a908ceae477f944
7262 F20101206_AAAQZA baldwin_j_Page_115.QC.jpg
38b5ae1d2a8776345993ec38bd8909a5
95d922ccfcb061e80baf90eff65470f963ec82fe
1051985 F20101206_AAARDS baldwin_j_Page_140.jp2
ac02699281f7bb84929a5b84ffb21778
61db8b483f53223ed698952e8c6863e9921c6c53
27000 F20101206_AAAQYM baldwin_j_Page_004.QC.jpg
9d6ab11e9451548eb8e8509ca08ad25a
d147610e7d01a952a9660e5b826fe0588a3295f4
76293 F20101206_AAAQXX baldwin_j_Page_107.jpg
5cb5f6130ab4a3b4576b6aa8b03faaa3
3461f78d9a9df9920a44c16d2826c6917f8b22b4
2746 F20101206_AAASHJ baldwin_j_Page_126thm.jpg
04598c50ba7d4b7e753842f50e538374
0ddb1acdcf350de084d0fcea55aebd45b9797e61
26117 F20101206_AAASGV baldwin_j_Page_014.QC.jpg
9c512aa9f96221b9bc4e2d195f31a819
373b52fbcf5e79039f24fedce3dd9c73906bb533
F20101206_AAAREH baldwin_j_Page_118.tif
73e84064757713b3c30dd1dea75e866e
e501a046b9cd3ab9373b1313cb16ee5da2c0ce90
697424 F20101206_AAAQZB baldwin_j_Page_087.jp2
cb7c0b12189f68c0004683bc9424094d
ab471523307586baaf382abc36033df07a188b88
F20101206_AAARDT baldwin_j_Page_121.tif
f5e45637257ca650094abdaf1b3eb664
bc9932f8fd1bbe6f8b5413e469e7643ba2cf0c75
40420 F20101206_AAAQYN baldwin_j_Page_089.jpg
d7bcfb2fec1fd70210ddcad7270cb737
9d071528d8c4eb5e1a5984bf3e4b72f9417ebfd9
4204 F20101206_AAAQXY baldwin_j_Page_131.pro
d4adcf6b5f3f79fb2464585642e23196
7056aef148d4457e9838cdaa5f492bd57fa14061
20088 F20101206_AAASHK baldwin_j_Page_059.QC.jpg
28a6e4c45f63c65023848fe037a96952
43a7abd35d6671b9f794034aec9e636257a048ca
7055 F20101206_AAASGW baldwin_j_Page_061thm.jpg
53a816c14ce4960cdc5fb14996d16288
3494e2355f3783b67ac3f47a14c7fc9d434e7ae2
20062 F20101206_AAAREI baldwin_j_Page_146.QC.jpg
8352faaf6bb22878d12e7baed8f0845c
c6657ce7d1156f367ea78e4d5f0abdd0cd19753e
6319 F20101206_AAAQZC baldwin_j_Page_006thm.jpg
1e111bf5345f33f4cfc7f124bf3b95b0
662bae570a1feebe400066a619fc294535898f76
24665 F20101206_AAARDU baldwin_j_Page_041.pro
c3d8db95d6f47edaedf920b58ced277b
4347cd65bba8af9f9707d5d268785986981d1d56
1886 F20101206_AAAQYO baldwin_j_Page_100.txt
926b4091cb02c5ebae3f1463fa7fd320
32266b82b976a3124ae94c02212581ee5cd9631b
17543 F20101206_AAAQXZ baldwin_j_Page_144.QC.jpg
3bc13d779094325f233e202f227c319c
81e587e5175b487dd0ae29b470a2771693f4ff43
19447 F20101206_AAASIA baldwin_j_Page_102.QC.jpg
0cbb5f23618797df4646fff4778819bd
030512140b136228cfb97e871ce64ecb0c626715
5115 F20101206_AAASHL baldwin_j_Page_145thm.jpg
17683abbbe232c1624241d4323542fb7
c83ad537568f0a1867150a345312a859c58feae7
6654 F20101206_AAASGX baldwin_j_Page_012thm.jpg
25f38942e1a753f2e62dd12f8492e07b
cdded35b4c44b92c01ceae7a19d2b2a4783c2632
28148 F20101206_AAAREJ baldwin_j_Page_162.pro
ce2dcdff5d3a36bb21a5a7a5295f87ba
c7fa9da3f95b8312bf8b51730bb0accad03576df
2987 F20101206_AAAQZD baldwin_j_Page_119thm.jpg
52c7ef1fe889be364b7fdf8e1dff844c
dff07b2459b6af8ad9832576267ad95bbe7b68d6
75698 F20101206_AAARDV baldwin_j_Page_010.jpg
6c3cba59c67befd3498190989279e252
975674c9dd5d1a2c2f9f562c36c8e9718a7ddc99
6798 F20101206_AAAQYP baldwin_j_Page_104thm.jpg
d52dffa284b8d7062bbe76e23a45b7d9
f3eaee3f4ecf04cf248ab296371ea8427fd05605
6665 F20101206_AAASHM baldwin_j_Page_099thm.jpg
c25c0adeb28c1c809204c68f219010a7
d9cd0c46ab619026ea52aaf7496f6f6224bc45d4
9331 F20101206_AAASGY baldwin_j_Page_129.QC.jpg
45894f2b955d382cd2af4e6ee0328f02
896e2d2f8ebc7800ed02f635da895452625c1f53
21245 F20101206_AAAREK baldwin_j_Page_027.QC.jpg
69d8d5a5525ed0d36b230cf21474a0b8
33353abf90b3c7d5b73c74eab04e5606c0e8895c
7672 F20101206_AAAQZE baldwin_j_Page_038thm.jpg
d4d0b1c525cb87502ae6de58fffb8ab6
034c820ac5026ebc113ce52dcceb44640486a734
17704 F20101206_AAARDW baldwin_j_Page_031.pro
735bc364a472d77fb38a75051faf433c
338c045c1a7c856211dce15a993d5f4222db743b
844 F20101206_AAAQYQ baldwin_j_Page_011.txt
46615488324d99832b445ce134a0ad72
0a942feb44c053b14b573979b8f0aa3af2cc2d4a
17836 F20101206_AAASIB baldwin_j_Page_055.QC.jpg
97d3c9aa93a43e43dc605065775738a5
c234b78f98f613306a9182572bfeaf0a8271e07f
18564 F20101206_AAASHN baldwin_j_Page_157.QC.jpg
0665884ac0b6336370315ffcab814c9a
c5041d78f2137f258c0d091bfb6fe17b123ad5f9
24475 F20101206_AAASGZ baldwin_j_Page_009.QC.jpg
9b49d4c8c3800e64565b2ab4cd5da077
3639d53f3e60abf1727772917ce510a27fff6d4b
402685 F20101206_AAAQZF baldwin_j_Page_132.jp2
8d3c715118896d2e2b22bb0fea41f99f
dd2eee6ce3abcbf0f3fdd893b4d01978421f9508
1051977 F20101206_AAARDX baldwin_j_Page_008.jp2
617e6488efc57b50939f136d2d459a4f
fc5380404b133c0fa40fa80d5271106b127e8ff7
8318 F20101206_AAAQYR baldwin_j_Page_116.pro
9f8361453c4513b44d4608f2bbb6f533
4f7493d5f88f4e04efd293d8d0f50414acfbfdcd
7924 F20101206_AAAREL baldwin_j_Page_130.QC.jpg
a92db07fc86c1e5bcae9f61df45a6680
4abe92c19043337484ee7085a28e4faff6f5e382
6077 F20101206_AAASIC baldwin_j_Page_074thm.jpg
b5caa2e27b55d8973f39b77fd704905d
6f3316a633d94d5971386acf45c9df923db19336
23782 F20101206_AAASHO baldwin_j_Page_016.QC.jpg
43bc4d956e4d606baacef7a0c4832bb3
89a89252a1cee1f2c0a653415b7cf6b34d3c0c87
728753 F20101206_AAAQZG baldwin_j_Page_045.jp2
e36443c98d746b8116b624404a4aa3dd
f3c12a959ef7cf782b60c5fb5a4373665e26e899
51227 F20101206_AAARDY baldwin_j_Page_057.jpg
abb9dd7bb99c8a9eef774fa76d86dfe0
fa7cb5ff90fde64c20699f0c38f071d5c61f1332
11571 F20101206_AAAQYS baldwin_j_Page_039.QC.jpg
5f3ce5aa5f89108c51d91afd701315e7
b58bbf50e29ed4c22705255079b710db5c796f97
4455 F20101206_AAAREM baldwin_j_Page_162thm.jpg
216e54f937334161c84cc28203a7f9e2
e30dd0e138f3d417f4f2cc5bf89ef629e6af91d0
6068 F20101206_AAASID baldwin_j_Page_096thm.jpg
2e8e0463894f44c228896ace9d25c749
a57ffde3c80bdb306348b6032b8ca56e143be44b
5580 F20101206_AAASHP baldwin_j_Page_093thm.jpg
bf0d8e5d152fcf531cf2dc4bc2b1523e
d5d11074021d5e2c69ec0ce8da20ebad21c0b06f
2757 F20101206_AAAQZH baldwin_j_Page_007.txt
76115b9801e940a72c9a7a26444c7468
89cbd4f1952c4047230474ae4f98f3d8c4eead63
76824 F20101206_AAARDZ baldwin_j_Page_022.jpg
b6662dd46b431f8c1e3f069159cc9004
b062767ff63d9e79715485f26ef697647dbed69a
6227 F20101206_AAAREN baldwin_j_Page_142thm.jpg
303243b76b4fb160ca6afaf307ebb225
efbeb225d1d3316f1519b5b658b3547bd8e89726
6507 F20101206_AAASIE baldwin_j_Page_092thm.jpg
8338be076eba5e7b208b991c4afc4c51
bcdc78fcbc4eef09e65b59ca8f75d6804fb62905
22773 F20101206_AAASHQ baldwin_j_Page_092.QC.jpg
46116b20d9c80ed687543cd0dc726995
5aa789eb691fa68af39ab18d67fb4b6dacd3868e
5173 F20101206_AAAQZI baldwin_j_Page_077thm.jpg
f2f52577bfa1a06ea3ead14e55437d7d
66238d972fcb933993748365595a97b2b1c71c92
5626 F20101206_AAAQYT baldwin_j_Page_113.pro
d339566878457c9b83f8f025d3b32ed4
a4f7e3de7ff0a98de923c23bac2f3adcf760b7a3
27152 F20101206_AAARFC baldwin_j_Page_001.jpg
fe2e4246d6280699ec902e464187adb6
859a09465b8b6747d711d6e8f4be158f0d9ba9a0
467096 F20101206_AAAREO baldwin_j_Page_052.jp2
f6dcc0f5089e8feb2a9a0d7005cd1292
a40561f2bc8386fd2b39744dbc434c238e6629f7
12637 F20101206_AAASIF baldwin_j_Page_048.QC.jpg
7b1f3d3b879eaad359d70c1f29562afe
e8814090f616f1e879b4f3c81b9012a07e8a0ad1
23308 F20101206_AAASHR baldwin_j_Page_012.QC.jpg
6fd13e8d1d0bf4e2ec9d5980410f1902
f7ad5a7c8420efa55691a259f6aea216b49075ad
1802 F20101206_AAAQZJ baldwin_j_Page_102.txt
ade5251a8642b60c2b3f250694527339
5ece3a2dcff87b65eb97560267767fc3892828ba
1199 F20101206_AAAQYU baldwin_j_Page_086.txt
5033d3eae298718317d9f9793a08d892
ad25bb016fef44cf9d212f9b7bddf2dc8467f90c
10413 F20101206_AAARFD baldwin_j_Page_002.jpg
b0b9764911719a0a8e90f184b3befcd0
c5aebbc51e747b52345127ea8eddea7da5eebdf7
6886 F20101206_AAAREP baldwin_j_Page_017thm.jpg
cae3833022927caa84a73df29f6aa257
c05047258b9a44393f1d817a164c7f22ae99ae2e
7577 F20101206_AAASIG baldwin_j_Page_128.QC.jpg
2f42622d5377b14977aca9915ed9b1e4
28c7e61c26ae951f5a65d3f1d7d7d32060b9f077
8351 F20101206_AAASHS baldwin_j_Page_136.QC.jpg
8e1a4d5465edd4d80c8e9b04af949888
81019678bbd91161103a563253dd7ac17cc8a321
4547 F20101206_AAAQZK baldwin_j_Page_076thm.jpg
177111cf2fda8048308e66b92baf8a7a
f2a9017a61b92e1b7df73f3633236c264ca97d11
35937 F20101206_AAAQYV baldwin_j_Page_030.pro
e0f1a8eb40af1d04df8b77b2b864d6ab
d56e194ea2f913ad4813be37fa8f3e776133a1a0
11228 F20101206_AAARFE baldwin_j_Page_003.jpg
27ef09cc5c3f520130042233818618bb
85f101931798a08f6900b3fe23e85e1081490aef
23918 F20101206_AAAREQ baldwin_j_Page_022.QC.jpg
38179855b947989f3909a5a92337ac5b
a7ee24a905ccd333a7582e722b4b6d9a7b24a0a8
2305 F20101206_AAASIH baldwin_j_Page_125thm.jpg
d70db7975361bc65b3db788825354788
b0fb17a60b2b51cde6b7b12e5aa135e5bbc138a5
4830 F20101206_AAASHT baldwin_j_Page_034thm.jpg
e1d4215e2b6a9f62288216a6b47e8d5a
90119979fa25899d3cef7ad64308663c006927c0
5728 F20101206_AAAQZL baldwin_j_Page_064thm.jpg
4e581bf5105205fe4fe871e0d87160bd
38dc9353cef706f448c4f68087f4c24d2c25ecd7
2028 F20101206_AAAQYW baldwin_j_Page_005.txt
171bda878780c17d8cd9f4cf86cb3e3f
173d36e931bf75d0b796c4a96b859687267e316d
86601 F20101206_AAARFF baldwin_j_Page_004.jpg
640e72de25f27cddc1b50f82ae16c6e8
123c921e8a71f9b126042603986b72d06a3c7335
1705 F20101206_AAARER baldwin_j_Page_021.txt
156eb653772b510691052ce8f6cf4e97
5feff97a176d40aa82e12536fcbb311394ec70aa
22233 F20101206_AAASII baldwin_j_Page_049.QC.jpg
25b616d1bef1a151b7055ac7958b8eda
b87a0527ee66666ff788b47caefc118efdfdca6d
21782 F20101206_AAASHU baldwin_j_Page_024.QC.jpg
7da525df2f22829672cbcd38e13aa3fd
c46cad8a8df9d0c18b73eaa17c8e23401526751b
6146 F20101206_AAAQZM baldwin_j_Page_085thm.jpg
a17047093544ebb1cfa710091a20b3b8
edbaea1849da0780f8c49681879c4b0a2b63f138
278607 F20101206_AAAQYX baldwin_j_Page_001.jp2
123b7b61b1ff2412fe3249ad6e408055
7b0b5738bc49eb95d4a991895b84f34c05d2e8d9
82619 F20101206_AAARFG baldwin_j_Page_005.jpg
7cd4de7476000c4895c072ded0fae151
74c4fe4b5f53ad19b88fed2e3f04b0769339c430
21465 F20101206_AAARES baldwin_j_Page_070.QC.jpg
2c1b670f444f5a659539c63382bebf2c
66e9ab12b1a9d8867330354ff87d237def73f7df
5568 F20101206_AAASIJ baldwin_j_Page_045thm.jpg
0c7d590c60b486b2b125346fcb636dda
0054bb0f30a011ee7c19f5d17b673de6dc42792d
5793 F20101206_AAASHV baldwin_j_Page_025thm.jpg
f1e9303dba041df0db89adf29bcc21bb
9de7a72e3197fefe5abe89e46ed608bbf5bffd9c
6275 F20101206_AAAQZN baldwin_j_Page_049thm.jpg
bd021c6f7b2234a4396973c2f3a8d116
b771814a75d3944555dc17d1618604fb4da3dfa4
36095 F20101206_AAAQYY baldwin_j_Page_092.pro
4be2a84fd1ab0544d7f3504244e8de31
ee9428b32ecd9d358103b8ac3f8972464d49fe5c
92674 F20101206_AAARFH baldwin_j_Page_006.jpg
ca0ccb1f7b6b4d038f67fc52dcaa16b9
bbe9d9b57462963037c7d9eeee2f7721442451b2
F20101206_AAARET baldwin_j_Page_139.tif
88ca6fca08cb6a69ffbf8f3ad3db4446
033fef0afd17ab64d1af2314aabd2d8d05d73879
16668 F20101206_AAASIK baldwin_j_Page_057.QC.jpg
12328ef5e4799a4bd175f3263ddc91cb
f0b2c06413aca36e89d89a8d4320f8e836b97f79
5720 F20101206_AAASHW baldwin_j_Page_161thm.jpg
db84b01ee0e1e4ddbf2ceff270107dc3
186ede8ef8f89937b8647d3d03f73e48cdc58b2a
15399 F20101206_AAAQZO baldwin_j_Page_162.QC.jpg
93ef40ea2eda9cef12bbc4734c25ef15
a8568f75937539875950f3778c90bf3ba5b05a9e
1727 F20101206_AAAQYZ baldwin_j_Page_099.txt
b2502e0f220d9bcead2a447428126fd3
7e40daa6468b9cfbd185dbf1d428d1d930a7b0a9
68657 F20101206_AAARFI baldwin_j_Page_008.jpg
1f8f91ae8e6726963c06d6a1aef2963d
122748fc706510e25052a909dfe8021fd9369a31
15368 F20101206_AAAREU baldwin_j_Page_034.QC.jpg
bd069d862e74168673b8205c7fba5fb3
442bc3ce5b594811507c89b60ed15e615491d6db
22529 F20101206_AAASJA baldwin_j_Page_075.QC.jpg
dc2fb26a095a8c5d3f2021d0c595f959
934a22e588322afdc14f3ea2f3312da9ce8761de
3743 F20101206_AAASIL baldwin_j_Page_033thm.jpg
2dcd1beee1fb2dbf0920429a0b7bd21c
d8482bb8c5e3b01688ac93ae683ef9bfab94f0a9
6900 F20101206_AAASHX baldwin_j_Page_063thm.jpg
89fcf17c84fde0fd2026bdaa56835b4f
65885baa2459930458905ce882f1d29e51853d8a
15869 F20101206_AAAQZP baldwin_j_Page_054.QC.jpg
6d533a053240184abe6693e2635eb633
99670c7d122506ccacbf3334851974033f8ce74d
89670 F20101206_AAARFJ baldwin_j_Page_009.jpg
82f3e601d77a9449ae3d6218a32bc82a
320f9890bbdc2a4d8f5d4f517b4a2bf27739ad01
4691 F20101206_AAAREV baldwin_j_Page_108thm.jpg
4208b7a7ffb095b25f675d63200c781c
16f402444df72e4fe9ce4cb9522224c9fda4ac7e
15928 F20101206_AAASJB baldwin_j_Page_032.QC.jpg
4d8033fbc26589dcda927ffb37b5c9ec
fae2c25387a8363e1582f0e68607e04d6a29b34d
11991 F20101206_AAASIM baldwin_j_Page_033.QC.jpg
002aaae6d0ed00f880c8191695243d73
ad402de924fb00579bbb05b09fad7a2b2cbd682d
7423 F20101206_AAASHY baldwin_j_Page_160thm.jpg
66d8981769221da231843833e18090fa
521a2b1af9e82f219b6ccb8e668861c540145b9e
37321 F20101206_AAAQZQ baldwin_j_Page_074.pro
8b4bb08ee18bc346090ea4454dd87e41
53ac7b318bbef1129e852e77173733922719eb2a
37021 F20101206_AAARFK baldwin_j_Page_011.jpg
586978b7c5f055fafa2ca68ff5962df4
0cc42a9d0ae3934156e3d3a9415b2c8ca043ebcd
3551 F20101206_AAAREW baldwin_j_Page_039thm.jpg
64956ca5be05426987a4b91d5c4ce668
7bb9d1e44816814e7085fd4b97486583a21b3f6a
5731 F20101206_AAASIN baldwin_j_Page_125.QC.jpg
10a4e217723eb5b78fa08c836e0783e3
35ab437030d91ea23fcb5a23efaaca5d0ffd0b2e
5420 F20101206_AAASHZ baldwin_j_Page_087thm.jpg
107eede0f5ef223cfa48e75629ef73cc
491eb988816867ca3f998c366c03b0adbf6f31c6
F20101206_AAAQZR baldwin_j_Page_143.tif
92201b980b6c629079962d32a5ae7175
48cc0fd92fbba4c9ab11940033b18e5ff1f46e59
77745 F20101206_AAARFL baldwin_j_Page_012.jpg
eaaab931fc0eca15ca736dcf56ceffa6
fecc2da153595fcae5247108afbdf37f3f68797c
13339 F20101206_AAAREX baldwin_j_Page_052.QC.jpg
b071f340e41ee1aed08df4b7e9bb30df
8e235e00c77117f680b29b5ee9e7b07ac756f4bc
26246 F20101206_AAASJC baldwin_j_Page_148.QC.jpg
9fefa87ea2c6e5bc99194e9ed4621105
bfd670881b6b536feed1cf306d65607005bbd958
15577 F20101206_AAASIO baldwin_j_Page_076.QC.jpg
7121aba8ac6ce6abf609c567ab830882
eaa530f4ce5df2ccd080c24807232e5b847b727b
37206 F20101206_AAAQZS baldwin_j_Page_023.pro
54baf245fa24a5afe2e5d6e5971c9d77
b8a35c8e6887eb32ede6cb60a270e1393724fd4e
53716 F20101206_AAARGA baldwin_j_Page_028.jpg
07263dcd2a3c7e3fd057fbc300128f3c
e29bce9d577f67d4b221709712a9dc0590a2a9c1
12070 F20101206_AAARFM baldwin_j_Page_013.jpg
d9b10a142849aa3b475642d1ebf3c6f3
5198bd05354ea4fb2d068541cef144efb847f4d5
F20101206_AAAREY baldwin_j_Page_078.tif
a0f081faff09789224664639a5bb4af9
0d3569286bf2b47ccd2c059a5e02cfa027743941
12739 F20101206_AAASJD baldwin_j_Page_089.QC.jpg
91b663c63bb4fa2ba33672ea096cc05a
edc1369c091e6cc1b3f6a78d13be51a4dabe70c9
5079 F20101206_AAASIP baldwin_j_Page_008thm.jpg
3614f296281797be653d750827dff8ef
0ae6c573aa29872daffeea0de19d93c29ce463e9
F20101206_AAAQZT baldwin_j_Page_038.tif
d5fe2216e5c9fd68096e8ea645a101ca
4b48358cb390602c34d12144ccb9fb9e08ace8d9
75863 F20101206_AAARGB baldwin_j_Page_029.jpg
044e408ac655ee03f70da0a2e6d5b157
58f80ce262500d9849a5015dde917ba9ee43f101
83217 F20101206_AAARFN baldwin_j_Page_014.jpg
2354f91ce192d48e2888ccb4dccd2200
d645e25fc7f30619b3d42b617b1fc02accc806a8
187749 F20101206_AAAREZ UFE0021619_00001.mets FULL
9d3ee766e3583f8de3dfccd5292506e6
d1a6a337caedef8e208ba5e0cb52674f1020dce8
2702 F20101206_AAASJE baldwin_j_Page_112thm.jpg
c0bc2fc9ede4c6406a026a54c16a23e5
cc789ba50e1709c451d4bd54ae2aef4f1dffe561
11429 F20101206_AAASIQ baldwin_j_Page_056.QC.jpg
059569eca24cbcfff7b84f9a97f8f1ca
82f3d0eff753060d62640b1b18bf94f7279665cf
63071 F20101206_AAARGC baldwin_j_Page_030.jpg
c90944df8a2e4dab35f7b4b0215b384d
76bfcdd00d90907701786d4b82b4186747e873a4
77882 F20101206_AAARFO baldwin_j_Page_015.jpg
c5d5ff09f8cc6100adaff91296561870
4419fdae7e72e0c18fb285fdea9b92f0a287ac9f
7814 F20101206_AAASJF baldwin_j_Page_127.QC.jpg
8428bed20a6a720f26b3895c9f0f5f78
459d152dab641b6931824f8190d05679bdd10698
7549 F20101206_AAASIR baldwin_j_Page_106thm.jpg
4e8dd5bc07d74e1d322c82db75a193dd
cf962356af98bdbba06a015784e5cb30e2f676c0
86003 F20101206_AAAQZU baldwin_j_Page_150.jpg
4041137d59c62b26c9a064d4dc027ce5
c8182369cf133484e0eaa6d0e16acb24207e6b03
37980 F20101206_AAARGD baldwin_j_Page_031.jpg
ae7b310cca548f4996d06e0172e80a8d
ecbbf3ccb30f0774d80e32aa2922991812a7abe2
71824 F20101206_AAARFP baldwin_j_Page_016.jpg
d4deceddca580ccac017317cc995432b
4a6ac2559684292aa1706a10fdcf1dd210223695
7540 F20101206_AAASJG baldwin_j_Page_120.QC.jpg
a24ff0f3878ebdf693ea4a181444d834
4fcf693ac77467c36c3751a1248d0883b046b77d
18730 F20101206_AAASIS baldwin_j_Page_045.QC.jpg
07438824ef104bf7c2ab5763d5efc5d9
6292616889cd3eb2263940de38f667137a1413e8
8109 F20101206_AAAQZV baldwin_j_Page_135.QC.jpg
c45e694a8469ffe89b96e9991c28fef9
a217f26febf71ea807801eac05c1aeed5d2f45e7
51652 F20101206_AAARGE baldwin_j_Page_032.jpg
16acbc8511337cb0c3cce7240264f094
4a36a2ad954c772ede1c87620a1a893fe33888c0
74913 F20101206_AAARFQ baldwin_j_Page_017.jpg
c96ffc44ad141858e0f08e807b13a26c
fed9530f20ad186d70611576d2ce933c82376690
2597 F20101206_AAASJH baldwin_j_Page_118thm.jpg
1ea515f2faac7d0d8e7f5c359567c32c
60b7b27161ee4b5db4428c00691f4d953632b840
7475 F20101206_AAASIT baldwin_j_Page_132.QC.jpg
f138277be2b511dac9ec5377579f33ff
8ad4acabb8d8eb2e71e8755c3da6e30373165082
1553 F20101206_AAAQZW baldwin_j_Page_092.txt
4ec170aa33265e2fd44532ffa57ed9ee
dd70d7ac1767ce741a84b9027041e076784ff6fa
74625 F20101206_AAARGF baldwin_j_Page_035.jpg
9c4f1cd0047077d57d2a673463a2277b
6f7552d3af9e9e8be3d8592c1c15fb5b0022e849
80149 F20101206_AAARFR baldwin_j_Page_018.jpg
1e46f738a6dd21d40cdd5cc2601a9284
60e797d0ce8c7856d223e16e7064fd9f91f1bd6f
5614 F20101206_AAASJI baldwin_j_Page_141thm.jpg
d38745fc3be7772616c6a481d03438c7
7bfd7c37320eb822d67596a8732c6f15d153e444
28927 F20101206_AAASIU baldwin_j_Page_038.QC.jpg
e05720080945c7d69a8e64e032824bd8
2f5ea311f511afc84d114c1ceb31c11e5b2fc620
F20101206_AAAQZX baldwin_j_Page_125.tif
57e515df0d3c346b20c47c55b0c98f5d
87846ce4fd8ef37ba5ff3f24f923fb32b0dd849d
51829 F20101206_AAARGG baldwin_j_Page_036.jpg
97215c6003dc5acd1c982e376b961c20
74e34ab3447a332f4926a7bfe3b0df7bc6b29222
56634 F20101206_AAARFS baldwin_j_Page_019.jpg
34d8ab513426dcba87d82c8f6e2d4f9f
1675c13fb492fa9acdd47ef0670e4e670ae4da4f
5226 F20101206_AAASJJ baldwin_j_Page_144thm.jpg
d1861f134857d533b38ef93b517819cd
aebf30fd889339608a8c8261f0afa29a8977bcd8
F20101206_AAASIV baldwin_j_Page_040thm.jpg
baf84900b13e3c11f348f1c23e1ade0e
c5aafa057f2e3c7222ba8e3ef3b69f1a4348f04e
32153 F20101206_AAAQZY baldwin_j_Page_058.pro
28cc1d84c7e5dfda67dcf7c07638c8a6
f58758021979156aa6fda73b2cc127c250f0e78f
67862 F20101206_AAARGH baldwin_j_Page_037.jpg
338235e61c98957f4232940a3dfb975d
36c993b8c4b55ffe1340f22d46df785fb387f3f8
70423 F20101206_AAARFT baldwin_j_Page_020.jpg
dd8f75450c6656aa4b7d953105b6befa
b2c7391538809a95f8046c043895d316b599cbe8
21107 F20101206_AAASJK baldwin_j_Page_021.QC.jpg
b150932eb5ab3fea5e554f7b127af19c
c6dfcee11864a26ac2fc681771c36455879b7eb6
4295 F20101206_AAASIW baldwin_j_Page_066thm.jpg
64606ad574a76345fd044960d8958310
961a81c262512d2f0c3702d2d9862da46ce6829f
F20101206_AAAQZZ baldwin_j_Page_153.tif
762e7291d721835b93b08869b8c0646f
d2fb7aeabf77610530e9d4b8c229b3587f5edbee
89580 F20101206_AAARGI baldwin_j_Page_038.jpg
a8400929734bd3a349e65d1db3c655d2
0091ac350b0d4b205dc365b646fef61f22eda8a8
63977 F20101206_AAARFU baldwin_j_Page_021.jpg
880d822dff7b6de7e524bbf760459847
33ac4eef304593b4c1c013fc524fe0809f9b6e3e
4153 F20101206_AAASKA baldwin_j_Page_089thm.jpg
13144ad1b09d82aa698054f75def3c2c
bf100c4b1351963b4324df02855f9616f8d454b7
6997 F20101206_AAASJL baldwin_j_Page_118.QC.jpg
2f68247d73e03299adb8372625f02998
53db6a0736723bd266f553b5202f7ba29fc4f2e8
7446 F20101206_AAASIX baldwin_j_Page_148thm.jpg
b8c4d9d52d21d3ae3952163dac10629b
abe23e2307cf703710e720889a15734939c3dfc0
32315 F20101206_AAARGJ baldwin_j_Page_040.jpg
01755a93066b7c83f05157f73ceeaa01
55f902c195d956fe9661377f7d3f8bcd3782050c
67005 F20101206_AAARFV baldwin_j_Page_023.jpg
ef08e702c8cb665102679e0f84649b39
c2f576619aeb77b0c4b01c8156479832e811de25
6456 F20101206_AAASKB baldwin_j_Page_016thm.jpg
40550d0904bf5c8a050a5a7c0a73b769
b530dc8e5deaf0c9f7fe3fc1a71487b49ead2cb8
6215 F20101206_AAASJM baldwin_j_Page_084thm.jpg
9fc5e45cdf201747abac6ba8330e7aca
8153b8c61b6a6d95b37e2b96fe9475f1bc506688
4661 F20101206_AAASIY baldwin_j_Page_080thm.jpg
c30b6c7d41b164e733849d6c273d53bd
a4a723eeeaf73d711217cea50fe90d1ec0a566d6
25736 F20101206_AAARGK baldwin_j_Page_041.jpg
645cf5c9e0ec7bf015fb336fbacfffde
b0bf7fa88df6b2a0f00d1e07b7b76278861f6445
69236 F20101206_AAARFW baldwin_j_Page_024.jpg
5143f51162d0bb54a854e9d85ce16e3a
d20b34a193698a52b0c050f33f60c13cf7f46b59
24329 F20101206_AAASKC baldwin_j_Page_029.QC.jpg
0d9d81ecde4c960ebdf0c05dc848f10e
e49e76e8e19ac0bca4f3a250f5ae61e30353664f
F20101206_AAASJN baldwin_j_Page_116.QC.jpg
8d5886bd0a9e085fee5178cdc7f331b5
c2b95f1b9d89bc6339f6ff536aedb8154bcb34a4
20291 F20101206_AAASIZ baldwin_j_Page_071.QC.jpg
5cd8343d056abf8c5af6cf754af7d34e
6d6ace24d40910659dc88b0f718c157f146c1a0c
71151 F20101206_AAARGL baldwin_j_Page_042.jpg
6f47b77e73239ab6ee814f58e79e2865
a5961d0c2e569a53d030e7b4bf6d1641d808e72b
59519 F20101206_AAARFX baldwin_j_Page_025.jpg
cd7d8623bad7b279c6c237cefd532d82
b0bbfdfa45ad1bbffd71d87a86b82947d60950a5
58000 F20101206_AAARHA baldwin_j_Page_058.jpg
7916bc280187ed4806867056fb968412
307119dbecaad44d758055fd11f8f09bc2a9b195
16530 F20101206_AAASJO baldwin_j_Page_094.QC.jpg
3c5a811eaa64e6cac052af1d3f14b874
b859ca265175c86eb371d035662e09445ff1a82d
58573 F20101206_AAARGM baldwin_j_Page_043.jpg
ccb37eb69a53a1fa4384dcd45f827688
855c1303473c2bc5613b44ed52106c8c11ce2b02
61345 F20101206_AAARFY baldwin_j_Page_026.jpg
76d3a1637955fdb5ebb45ea94b0676b1
edb9ae92e3cb7afacb4f746f73bdb4cfeaf9279b
243164 F20101206_AAASKD UFE0021619_00001.xml
acf0c947350d9c58ad650ecab7383c17
6669bf1f0130696f73a5aea56c93e0a228155be2
6679 F20101206_AAASJP baldwin_j_Page_095thm.jpg
d5722b781cc468d2ef9b542b76c2a9cb
fa3a9e3257694c34f3c13be4278111f37f4a1568
56440 F20101206_AAARGN baldwin_j_Page_044.jpg
cab2237854bcb9d76b616e2e7a38f889
ee3690417057bc8906dffe962855e17683fd0a7b
64339 F20101206_AAARFZ baldwin_j_Page_027.jpg
65779476d9bd278247d6c618c1384128
e697ab5dacd2df2227594738bac1b788415ab037
59819 F20101206_AAARHB baldwin_j_Page_059.jpg
5f97ce46033f63acff79fa91e9cf1f1b
da4d1d9c6f22657eb1f20b834d2a053f925d0192
7243 F20101206_AAASKE baldwin_j_Page_004thm.jpg
c8986ea252acfd7ffcd7bacfebc883c3
2135c8f0b6dd397c422c5f35f0f8bae7a1aa57f4
15767 F20101206_AAASJQ baldwin_j_Page_077.QC.jpg
1c1e7a6d8099d30738939fa07a1e643c
e2c49ea5fe0d97eb7270d300d677c101bc9f3c8d
56978 F20101206_AAARGO baldwin_j_Page_045.jpg
0b0fd71bd1bd4fda6b73fdbbd45b6c83
75e52a074a306469cd1b1372014eadc2748934d3
71291 F20101206_AAARHC baldwin_j_Page_060.jpg
7c80acdb428b1b95030588bffe36e440
393dae928c4fce77c5697fe2b673c2f9d968bc58
26042 F20101206_AAASKF baldwin_j_Page_005.QC.jpg
55b6efa5bc3f75fde6c9076441aa523d
bcb88972277ccbd23d6d8b4c555abbf3b7e9f1d1
7016 F20101206_AAASJR baldwin_j_Page_015thm.jpg
7f5dc62863e89f017a7bf21bfad77444
93b09c35cb2d3e1ad16df0352bab473062ea25a4
39219 F20101206_AAARGP baldwin_j_Page_046.jpg
9cd64e6c9637fb49dd0b5a6b2b2ba7ff
63329eef969a895f1605000c5890907355d42b75
74079 F20101206_AAARHD baldwin_j_Page_061.jpg
1b9205977b0c6113fc0dce5e928b540a
1a29963b3588015cba195c618f8bf943e16d2ee7
25230 F20101206_AAASKG baldwin_j_Page_006.QC.jpg
407c099bc8701c824785f05cf4830b54
ac577de4718ccab00fe8138e86047bd61ac38511
14168 F20101206_AAASJS baldwin_j_Page_050.QC.jpg
59611db883c6ed888a6452d20a80b975
3d8ef5cd47bca930578c2e7768239af5d3981043
51249 F20101206_AAARGQ baldwin_j_Page_047.jpg
b7d456fa3dfd1b715bdca25c8540b5ed
1ec308475a893ae436278e96690ea9214a89d463
16091 F20101206_AAARHE baldwin_j_Page_062.jpg
46fc952bbe311cc73dd96aef65d703dd
7ba9df60908e4f89ae227ba714a0eaf1e27cf305
22828 F20101206_AAASKH baldwin_j_Page_007.QC.jpg
57969ca5c1e74c13b7fe5d3319f93847
129bb48be15ff5f13b449981ddc898b117eed23a
20149 F20101206_AAASJT baldwin_j_Page_051.QC.jpg
31f042fe26c23685ffb4aea1cf9bc858
f6e77f651d125c58e9975e67289eb44c08ec7152
40032 F20101206_AAARGR baldwin_j_Page_048.jpg
b38ecd3bff3ceb2e89496873627381fa
6d96e5add40aa5a4e644b1e704c11605c28a7a74
78268 F20101206_AAARHF baldwin_j_Page_063.jpg
d7364374d4a4f184a5ddce05482e75d4
2efdee18d558096213e3c0a4f1da58dece474233
22908 F20101206_AAASKI baldwin_j_Page_010.QC.jpg
b19cda811571b5ad3e4c3ffef34571e1
33c39446ceffc7b7a501fe1974c0cca7c995f15f
6095 F20101206_AAASJU baldwin_j_Page_027thm.jpg
13278430fb2659927f0027743a1d9e96
a0838fd46975bdfc2fb0b29fa5346d41b209d14a
67821 F20101206_AAARGS baldwin_j_Page_049.jpg
9be396014a5959b9e5401d0085ddb578
aba981ea0d510b3e89b17f1685d68fcd7e6c3491
58010 F20101206_AAARHG baldwin_j_Page_064.jpg
9e8da7629d4979c2cbb4e8f9b0f6dd34
6553707d951f3bb130677715a550b6f99070a853
6646 F20101206_AAASKJ baldwin_j_Page_010thm.jpg
30f66c822cd68be9b769e939b3e612d2
24e0044aa4a2cbcdc88e9360c9e584bfd84e5abb
19337 F20101206_AAASJV baldwin_j_Page_069.QC.jpg
adbd229c9b95c3f2b531d2c75d1c0704
791316122edde46d2fd958daafddc88364737c03
44638 F20101206_AAARGT baldwin_j_Page_050.jpg
acd4e5f606dc515cfdab082fde49949b
cac9b2d3327b07851e729b972b062db4cbf09668
40024 F20101206_AAARHH baldwin_j_Page_065.jpg
e870b3e4c9533bd338713e9b695868a1
c99bb2392e96a7275901c9765ed962c58b090bc2
3245 F20101206_AAASKK baldwin_j_Page_011thm.jpg
a9a7b6475fca41d83da5e86617b52d89
b083a4a5b97c1fef51277e00f0d489498c634f6c
5986 F20101206_AAASJW baldwin_j_Page_030thm.jpg
93b39307f94077413f26af3c2368fa74
72154328fce9c92136ac491c6a1ca7c36504b325
63651 F20101206_AAARGU baldwin_j_Page_051.jpg
657cecc9558578c9c76496f86ac4061d
f9cc643b05966526b8fa63ca62457133cdd8b53d
40397 F20101206_AAARHI baldwin_j_Page_066.jpg
1c5c66828d3ad14f557ab99623676ccc
e096ad87fa97b0f6ac6f064122231e7c31dc32f8
21301 F20101206_AAASLA baldwin_j_Page_037.QC.jpg
89d44fb8c9dbe7560bca283c8bfca3b7
5120cdf23d30b878a755241bbcbc329d0e6c1dde
3963 F20101206_AAASKL baldwin_j_Page_013.QC.jpg
09e0d11a692ba6880acc3e247d2113a1
35dcc5fe6efe215a0152d82bc9d97632c219306b
6930 F20101206_AAASJX baldwin_j_Page_009thm.jpg
3af53a8ce063589a448471e21a434e5f
8f3564c00c7f674708f7f7ef8be2858cb3e8184b
41843 F20101206_AAARGV baldwin_j_Page_052.jpg
f3647ffbbf3e1cb0d7e454b4e8f8036e
71ba3722ee5df3cfa22ad49cf299224ceb4d4da9
39120 F20101206_AAARHJ baldwin_j_Page_067.jpg
bf385221fef841e6ec01dd9990b7c90f
3e1aad2e543af72ad06e8294e99fb993ebf1cccd
6288 F20101206_AAASLB baldwin_j_Page_037thm.jpg
a8d76132552426f619283de10af082e8
b5d8f4aceede6e97d2b1a10aabb7b66266bde8ac
7444 F20101206_AAASKM baldwin_j_Page_014thm.jpg
63cc10ecd219b815d295d43e0e524555
8e12dbc2c41552900b74079e9be44bff691c46cd
12698 F20101206_AAASJY baldwin_j_Page_031.QC.jpg
13073c584b21e84875bd39c8e010e897
4f023b3ef72185c3ff9620ac7dcafe28285321f5
51263 F20101206_AAARGW baldwin_j_Page_053.jpg
91940b82efaee6a4a80f3034f4c07bd2
2c921f36842c2281ad34362797c7fbb93d66b91b
67124 F20101206_AAARHK baldwin_j_Page_068.jpg
dfb4a920fc62add92571fc169091ad29
b7e32a179a65d69383d2b22de2c1c04aaeb42295
2860 F20101206_AAASLC baldwin_j_Page_041thm.jpg
fbdd76298ab106a6c228fc53d818b95b
20c246993a6f5a75113165b839dd73b11468f313
6721 F20101206_AAASKN baldwin_j_Page_018thm.jpg
b27cb599f496aaa61e9421fd7f16f8a8
ed82be84ddfbdc49244ca0c99a3991773a224c6c
4394 F20101206_AAASJZ baldwin_j_Page_155thm.jpg
129879a7b74ce84c7ed8dd9cb7038998
6a6d3598b8298a674c496422c3c84ae293fd7d09
49535 F20101206_AAARGX baldwin_j_Page_054.jpg
7ab37f1470508c3944bf6749133eb763
a25b0399a9b0cfe1793bb50fd20bc327ec6b1c6a
67674 F20101206_AAARIA baldwin_j_Page_090.jpg
36201bcf9b829a6d300db9718c14f929
fab7f890798276d71e6425484a0f279b544c07f5
65789 F20101206_AAARHL baldwin_j_Page_071.jpg
43a96f66934d44e52caf31d2a8ad11fa
ecefe54fb1e093c2953085233cbf2d800c092785
17893 F20101206_AAASLD baldwin_j_Page_044.QC.jpg
2c6e7f941cb65f4d576a0ad248e84c49
5a4c0eabaa9bccb8b0cbf2d992c1f3f6ed5f204f
5636 F20101206_AAASKO baldwin_j_Page_019thm.jpg
3ae77ea8feb868377173002259d80761
3931f80c2e5c94705cc496d48f267223bd48d4fc
53994 F20101206_AAARGY baldwin_j_Page_055.jpg
ee69ed9ed22b98e047fb08439b6cdb65
1e7cca112ce0f58ab66e463d7439e8437c6966e3
67812 F20101206_AAARIB baldwin_j_Page_091.jpg
b1a1d26e976d5883c9e0ae8444adec1a
511b8a74ecf4a0eb7d9b6a59f44c01b82a8391b3
50978 F20101206_AAARHM baldwin_j_Page_072.jpg
c259a9df42f3dafdba566ac53775b2c6
2f486bbeb75c1f2aae5211fca424e573ff18ae97
6490 F20101206_AAASKP baldwin_j_Page_020thm.jpg
573e3fc3e71105ef51427af8b4d5e3eb
16e2cbda97f29bcd9defcb39880c333b6c65d9d8
35612 F20101206_AAARGZ baldwin_j_Page_056.jpg
02bdae68c1046fc6fc18829238ca00dd
c4240c4b62b2949000a55fb92908f6b9d5d892a3
56515 F20101206_AAARHN baldwin_j_Page_073.jpg
b9ce09197497a2831caad627cb216f8e
24ba227ccc919c2a52ceb0ba8d3ac29e7d254c0e
4352 F20101206_AAASLE baldwin_j_Page_048thm.jpg
b290b5e28a43c570f83d1ecfcb5eb226
296049157b55962f3af537ee3f094c8a0824ee25
6044 F20101206_AAASKQ baldwin_j_Page_021thm.jpg
3ad3f6fee45a4e8e2c15c756e6ac4a7b
5f9db64093c229aed80b0951c61b729cc21e3ae0
68989 F20101206_AAARIC baldwin_j_Page_092.jpg
633b8b5ad841858e1379132e77a71eeb
12f781505c458150d0924974cd158a8b104e170d
68431 F20101206_AAARHO baldwin_j_Page_075.jpg
19ef6cd29b5293a2b463fe5a826e4d9d
0d0a4adfae6b86303664bd1daf00966d4bb680ca
16411 F20101206_AAASLF baldwin_j_Page_053.QC.jpg
df24596985f2142d47389738f597943c
664c45d38c4d764a7e5550b943baa0a0318715c4
7269 F20101206_AAASKR baldwin_j_Page_022thm.jpg
65c0326049731c26a9d88ad77cd644f5
d5a11b6e75f85bcdc8b27cb46735908d30a0eb56
54580 F20101206_AAARID baldwin_j_Page_093.jpg
531aefe82a85ec25b09c6df4285bc4eb
70d70e63d11bd18ba0d1269be43bb44d22284c0e
46023 F20101206_AAARHP baldwin_j_Page_076.jpg
d27e11c1af6b9a05f54afca38715478b
98f6a1c074241dee685d135df2c37f7769d18d49
4892 F20101206_AAASLG baldwin_j_Page_053thm.jpg
5a557090b75e54caf3fd6837205cb081
8affcd39b34490d928b84dd4b4c342db256da592
21927 F20101206_AAASKS baldwin_j_Page_023.QC.jpg
810437cb6b235985f73f6b1334db366e
c934dca0c8ad87b2bcf6db5c4a77f543aeb5a37a
50272 F20101206_AAARIE baldwin_j_Page_094.jpg
d61b3ad5c545e86137b20882bde8abff
13b1e7da40d88ad88842211f7ecca02f34ad806c
49352 F20101206_AAARHQ baldwin_j_Page_078.jpg
3aa9aa1153d37bc25f729c7b86e15b6d
f2b72202a3de1aa867c895c1e2326917a330701a
4713 F20101206_AAASLH baldwin_j_Page_054thm.jpg
7c35692bf048501eb1463e773a0403a2
bbe50767715647f158255deb54258404617b7dde
6498 F20101206_AAASKT baldwin_j_Page_024thm.jpg
4eaf0a999674ede67e1610b84e338f88
5ba8a7b0560bc3b6df9ce44b9a78c671f717a7eb
69456 F20101206_AAARIF baldwin_j_Page_095.jpg
8497e29acb1f21545a7ea87a392f8372
312ad6a773532285b8d12d543ffe136ee07b339b
51293 F20101206_AAARHR baldwin_j_Page_079.jpg
fa46fc38b66d9219f5f18b7d3762ba41
a0f954554466d8ca3add88cb4a1bf4110a163c4f
3636 F20101206_AAASLI baldwin_j_Page_056thm.jpg
6b5466d96c5e097d22852775819b8a5e
bcb9dfff3cb4d5fb43ef33f38d7cee95db7027a1
18347 F20101206_AAASKU baldwin_j_Page_025.QC.jpg
dc40a7da48bdf0c322f6aee175c407b8
d61db9eb226834d3b2fe7180028f5dda9cdce935
62062 F20101206_AAARIG baldwin_j_Page_097.jpg
26619ceb85300e26bb96cc2e07dd5d62
1528581fa7f91ec5db65a4d6753c84c1f4cc910d
49110 F20101206_AAARHS baldwin_j_Page_080.jpg
4fa68af6b89d9b46591e47104f339b20
1994c3c263c5bc4df472b8881c6672e4eccb65f6
5066 F20101206_AAASLJ baldwin_j_Page_057thm.jpg
cec3b4efbc70c5a71657bd047e593015
8c00121fd59ccddf6cb2311192909ce414fe9a05
19455 F20101206_AAASKV baldwin_j_Page_026.QC.jpg
8515c1079451dae124fb6712dde6ab83
fef6eb6382f46fe92ba4d4f518c59b43b96f9fe7
61910 F20101206_AAARIH baldwin_j_Page_098.jpg
c430e9f1adc19c9400a18d9232ac9197
3997f2d3651c7e44765f3ba0d289650697a9f8e1
68591 F20101206_AAARHT baldwin_j_Page_081.jpg
ccb40e2d23cc649dc0d74abbbfe46b24
4a9d7c8687202bc106dcd43f31860cac0a4b17bc
18815 F20101206_AAASLK baldwin_j_Page_058.QC.jpg
c29e35c53398850f28d1940133340880
52fc96c8d2ad220a3d6fb85ad940f5aaa5446931
5732 F20101206_AAASKW baldwin_j_Page_026thm.jpg
9a5915d0670576fb14c3cf353c051426
4266fed265365d5dc6b63cd3ca56ce20a54c012b
73526 F20101206_AAARII baldwin_j_Page_100.jpg
aaf257ff3f572543c4cf6ebbe5ca3fc1
b29a8a8ae19c476e59d8776a6f554502d95e7a1e
76989 F20101206_AAARHU baldwin_j_Page_083.jpg
34b7854e00a4af51a6bd15c9d4d8a9ff
898fcac7e84bbb9a5e07a40fa81dcae54ab8a250
6350 F20101206_AAASMA baldwin_j_Page_075thm.jpg
dd61ba0c087cbe8a0d6bef244d17b827
22a60939abad15e098156ae4e814cf0867d24f2a
5707 F20101206_AAASLL baldwin_j_Page_059thm.jpg
9abbc32b354fe96d726284997359f185
0e5d553b46fa5ce17c438db8e0de67d40b011fbc
16409 F20101206_AAASKX baldwin_j_Page_028.QC.jpg
a3e0795edb1f0a0e628f9aa2fbe936c9
29a392102419bfb53f236278006f5af791ef8e04
73297 F20101206_AAARIJ baldwin_j_Page_101.jpg
f6be2e7a6db47d39a1aa3508123fe28a
9f633ba162e95ef0a8f3fbfb608799bf68235a03
63986 F20101206_AAARHV baldwin_j_Page_084.jpg
cd9f4e8170b46284e0a7d477b18be50b
70e893428809b037da90b14bc707c589146d0e9d
16327 F20101206_AAASMB baldwin_j_Page_080.QC.jpg
5923b78672e3611b9f13d18a8a7c98ff
25ea5dec47cc6cb76afe6a9029864f27b469318f
21443 F20101206_AAASLM baldwin_j_Page_060.QC.jpg
791e027b2d81c9b2600da310a02dbbe0
a3fa706fd98b7ddd6d56e737257554853e4ce2dd
5437 F20101206_AAASKY baldwin_j_Page_028thm.jpg
573e8c82c87c206db1dac9d8fb987663
8cde5083ad20b35c0cf3684255cf4d87e04ad947
62145 F20101206_AAARIK baldwin_j_Page_102.jpg
c5ed16b4e17d768da2f31ac2f4a9e187
71b1f9eca278b8dc3332680e2a8891a92e9a10d6
61473 F20101206_AAARHW baldwin_j_Page_085.jpg
5e92df07eb0eb886ec5422677a83355c
75a2aacd98a59258a407ccda194366bc45e9a259
22058 F20101206_AAASMC baldwin_j_Page_081.QC.jpg
adbd85a81bcc0ebaa175ad26f9e1f5e4
e9438e34f6362a9c1abec26aff54d513b60d5065
6690 F20101206_AAASLN baldwin_j_Page_060thm.jpg
6fc7d880a9bf5f687866977e04aaf68a
f1ba7ac7512b6c1608ca423c0615202e25597536
21293 F20101206_AAASKZ baldwin_j_Page_030.QC.jpg
2111e6f79754cc77efb53bac11ee50fe
dc5dad56916acd5b94e635788716bf7bcfc9f480
14970 F20101206_AAARJA baldwin_j_Page_121.jpg
86c29f22cd087ccfd41a6a793ae2815f
1fe3bb0c086950664b30d9ab9b0db6a29c9d999a
78759 F20101206_AAARIL baldwin_j_Page_103.jpg
775811488fbc3fd6feeb3473d1b89c50
9c555e5bd11f3ee5d5cf74c347aa9f6f060300d1
38156 F20101206_AAARHX baldwin_j_Page_086.jpg
8da3ef3241d4cd019b73fbf93ca7cbb2
d330164986d96ee6fa4e329e358da4b3053cc468
21062 F20101206_AAASMD baldwin_j_Page_082.QC.jpg
3a4e29c87be66e1c78286505d202380c
0a7df0e6efec1fa47c6cbeba35f7d68610ed92df
23015 F20101206_AAASLO baldwin_j_Page_061.QC.jpg
46833a8a743e26803eed67040f793be6
9efa51a931895088432dc9311d80f43b1c1aa74f
23125 F20101206_AAARJB baldwin_j_Page_122.jpg
131fb04fde3d6864189136aafe07a716
1bf4cf3ebc0e9d8a0c494e903dc9033e451dd421
73031 F20101206_AAARIM baldwin_j_Page_104.jpg
71ec659fe3ac329f4818fa2ff5a5b772
f9d775a9eb2d8849eb99b23fa4d00eaf46d72198
54927 F20101206_AAARHY baldwin_j_Page_087.jpg
3b95ef205742a45db79062ba60104883
a12d0d6f8d21e31946f85cf32b68b01bf06ed4ae
6894 F20101206_AAASME baldwin_j_Page_082thm.jpg
372af765fc81537fac7c8867ff7cdf83
86e2ca5cb66f21149fe0a3cb241a8ac9077f3426
5377 F20101206_AAASLP baldwin_j_Page_062.QC.jpg
4f9411cd55c55ed3329d56f2c92f511c
f8f6a933164d11854298695cb98d879adef73130
24942 F20101206_AAARJC baldwin_j_Page_123.jpg
a57eee914ebfb205fc2318256b50c1cc
7f23fa85b9d623dfb87fd435d238e69d0cc6c461
80121 F20101206_AAARIN baldwin_j_Page_105.jpg
739db6113c4dc46a41757ac5351bec5f
3ef09b16e9f06db7e7dea5dc6332e9672c0e61cc
48433 F20101206_AAARHZ baldwin_j_Page_088.jpg
d3867e5b8737cde4cbc71b28bbfd2ce6
ea3bc7936c44bb9729b1c5c562c8e134f437659d
1915 F20101206_AAASLQ baldwin_j_Page_062thm.jpg
7d41bd78c867775ce40dea4745f88ac8
bcf7a104e2126bffb77d593b22b802a3ec8390c1
80524 F20101206_AAARIO baldwin_j_Page_106.jpg
34bd9edd4d03e949eb7c0d86fc1f0dea
d644a701eba424f783e875f4f6146298df8e48b1
23754 F20101206_AAASMF baldwin_j_Page_083.QC.jpg
3e1e08e2cb61dc015ff1d43f50bf1e51
469a9b04e945291bd052ed08da61e18c7cafa9a1
25081 F20101206_AAASLR baldwin_j_Page_063.QC.jpg
b63c33b11720ca1eab981a7d33dc6299
f18e20f769f540cb61c1def1040007eb299d71a5
21102 F20101206_AAARJD baldwin_j_Page_124.jpg
97eeac5f78cf6fb5910efe061dae7d14
4d3616a69cd27e6dc58556f5394b5daa4a5641db
48902 F20101206_AAARIP baldwin_j_Page_108.jpg
1a8fd798c0ef22806d4f7b5891625b86
ada77b876ca6122fb1a30d083e6ab60ed0e70922
7187 F20101206_AAASMG baldwin_j_Page_083thm.jpg
e8f7b60be1e05ce80726fc2a8e0cfd8a
77bde5c5e79a2ca4d5e7e71441f81f3792bb0a83
4139 F20101206_AAASLS baldwin_j_Page_065thm.jpg
7986f1bd63ef9a37fdc6bccb840f007a
6c3eb1a493e01f1614175c5da9b0384de5cfef15
23615 F20101206_AAARJE baldwin_j_Page_127.jpg
86d19c664431c6b7ee52f89cb3faa1ce
5f0ca084c12330c6496c0b523dbef3177abbb2c0
20689 F20101206_AAARIQ baldwin_j_Page_109.jpg
82934c91b18a9102943889f4a0ef5215
8290d47fabaa7a2200b86df351ae94931692deb8
19790 F20101206_AAASMH baldwin_j_Page_085.QC.jpg
2d21b8d4896fed9076c19d85ee24ce4e
7eb7c1fd1bd5512655b43185c210976cb134aac4
13195 F20101206_AAASLT baldwin_j_Page_066.QC.jpg
11cd86d75bc955e1268fdfb1d02835ac
487412ca29595f577c7f887124ec352986d366f4
F20101206_AAARJF baldwin_j_Page_128.jpg
df6f4e7010ba8138b12ba0c56b1d02ab
3f4051998cd9ff4a98c340f7d36ae8343bd02ead
21581 F20101206_AAARIR baldwin_j_Page_110.jpg
6249dcea0aa636f4512ab8a400b675c8
c7b918e8829a6e20fb4e1750a83d5c4e308a9c90
12324 F20101206_AAASMI baldwin_j_Page_086.QC.jpg
7c90a341c9158c25bd612f334ccf6780
3ec1520c279e6f7c4248075e039672728fc59d41
4040 F20101206_AAASLU baldwin_j_Page_067thm.jpg
56b512b956d0b92272f178e9924a8831
e699783e9b89b96d02c420b553c06ef1a930bd05
28480 F20101206_AAARJG baldwin_j_Page_129.jpg
e2e33d8744579cbd8e4d74d1ee0b1f49
8639ed4493945c023ec2c4b5fa1ada4fcdd64163
23315 F20101206_AAARIS baldwin_j_Page_111.jpg
925e4c9010ea1cbf585c8ada4a781fc2
ea37026fd41f99fbe11f7fc3dff52c13387dd5af
4184 F20101206_AAASMJ baldwin_j_Page_086thm.jpg
8a0e8ae15c540555d8d439024ab19422
99553cf138b5ac30825d00daf82fa2152abade1f
20820 F20101206_AAASLV baldwin_j_Page_068.QC.jpg
f14f6613bd63629b8e6f196149bf6c41
deedb8f086c3d102085365fbcd53863949c3b93a
24814 F20101206_AAARJH baldwin_j_Page_130.jpg
723f78631767a2fe9e0264c6285e1914
b55e6c71e3256245b5cdd5a4b9a7f6849d325ca8
23500 F20101206_AAARIT baldwin_j_Page_112.jpg
af4cdf93491be8ee4fd4faaaeb7beda0
52ff94674b1e10e18a3d32c8d877ffaf3e86e79c
14391 F20101206_AAASMK baldwin_j_Page_088.QC.jpg
a9d765e373cb3467c4879f7d7b85315a
a4a94d62a4261191a5151e95a78f446399f90ec2
6186 F20101206_AAASLW baldwin_j_Page_068thm.jpg
99b29ad89cd6df3212b646938b1935e7
1bfc07b0cdc34ac50c6ab00f6bddfbe9f886378a
20905 F20101206_AAARIU baldwin_j_Page_113.jpg
64c6cf2cb72ae1b3ca9ccd0b12d94980
c4207a450db350be04971cc7685dc1672cd8d1af
34854 F20101206_AAARJI baldwin_j_Page_131.jpg
5ed299bc84b4e2d90a65a2b217fe7fc0
93630d5571f281bf71d901caf9540429ee53c3a2
2387 F20101206_AAASNA baldwin_j_Page_109thm.jpg
d4b05fcdfdfe7b2af2dff6cb9d8c081f
2d29ad21d459df3836043e037122474e5de84b4d
21527 F20101206_AAASML baldwin_j_Page_090.QC.jpg
aee9707213884e7aaff1fa7d6415edba
0ca079f921e68c2d7342437c4e8253437978050c
6437 F20101206_AAASLX baldwin_j_Page_071thm.jpg
f8e4f44b0baeccaf885eacc1fd41c611
2a1cab438bc239fc71839344e18097eff399c21f
21639 F20101206_AAARIV baldwin_j_Page_116.jpg
add303c26e7cd805bf2e518fb973b147
0a872440faa64c94fed24c31998a2b398af7ccce
20725 F20101206_AAARJJ baldwin_j_Page_132.jpg
39dfb9027abd55ce7af285bbe2e5052c
074afbcecab13e82a1c55abfd3e70a5621533c5a
2413 F20101206_AAASNB baldwin_j_Page_113thm.jpg
5c6aa4acd9da8bb4c435d5e52772529d
a78ec6e03895efa58e331639a09e8559d8284d75
20805 F20101206_AAASMM baldwin_j_Page_091.QC.jpg
27b38f7d8be8d9be81e448ec6709e9da
d06825a6e421b5860f104ef5a55ddc8204a71fba
5049 F20101206_AAASLY baldwin_j_Page_073thm.jpg
2d70347667227a39167e7c18ff196050
240f64d1634da800545be09dc46e18d4d2928310
22628 F20101206_AAARIW baldwin_j_Page_117.jpg
8fa36a97031e73ff9641cd5fb516f5c9
3fbd68502f4e232170ddae9ae0f20c86eec50cdb
23645 F20101206_AAARJK baldwin_j_Page_133.jpg
205abf58080d4ed9d0bb2a959a486e9a
e20f2839862241dcf9ab09e3791c7c7fbf9d665f
2671 F20101206_AAASNC baldwin_j_Page_114thm.jpg
a61574c00bec1dc72d687f34e81cfd07
66857d4532051d47ff3c83fc53807e99ee08beef
17651 F20101206_AAASMN baldwin_j_Page_093.QC.jpg
49f5434cfb13da3521f3c20d9e4512c5
6c6b47bde2b884cb50d8125eede0798076a700f3
21917 F20101206_AAASLZ baldwin_j_Page_074.QC.jpg
a97e16dece11edb6a8a9d1c83562215d
89b8e57fe49189b8f271b6a586cad0b4da9a7f42
20312 F20101206_AAARIX baldwin_j_Page_118.jpg
718a8c38a686258ca556f6f77c0c3859
e3101a03f4b40be31d2a50c79691cc39df0242d0
56526 F20101206_AAARKA baldwin_j_Page_152.jpg
8da12a506c7b870aa9d91b5c527661e7
b79cc25bdf043a918e3c9d1ecc69a5f71b726fca
24757 F20101206_AAARJL baldwin_j_Page_134.jpg
34e3780bd579c0572be374edc06fa1a4
f15a0c3247d7451a9b77aed4ed0a7741f17ca0fd
7745 F20101206_AAASND baldwin_j_Page_117.QC.jpg
00ab84d7a73d2b0cc036703d893ae55b
fd93184c63537acbde3c531e71f020340ebb25a6
21831 F20101206_AAASMO baldwin_j_Page_095.QC.jpg
a0cc99cedac96f845ce233c6b2b7a2f5
8f5ef6bb3b03531cbd184c14852a849db564238f
23720 F20101206_AAARIY baldwin_j_Page_119.jpg
f2ee20f9013953a2e7f7a66f6b2df839
77ba0a54fbe184ff66369271cb61da8228b937e8
51384 F20101206_AAARKB baldwin_j_Page_153.jpg
b95a208f19307f0b7eb09fa92f3ead36
1570ec7fbf60b334f1750ed5d6959fe93f0a1d57
24357 F20101206_AAARJM baldwin_j_Page_135.jpg
060519234b27541cf2f4953d55abdf0f
5db389240134d0f0379b74cf064ddb83f9319e9d
5319 F20101206_AAASNE baldwin_j_Page_121.QC.jpg
caa08fd79309091d1fbd1c12395a0453
edab0e1cbafce809635ae118f9c58082d54b044c
19540 F20101206_AAASMP baldwin_j_Page_097.QC.jpg
6ffaf5292b146ebc57a83646706fdc9c
6710b2dc652c3163315e0edf2808fcd5133d9fc4
21320 F20101206_AAARIZ baldwin_j_Page_120.jpg
d8ee44d73e20a1d2a4b5af964184cb3a
c35105008d03cc0af5e1a04fcf28c0f22e422a56
44955 F20101206_AAARKC baldwin_j_Page_154.jpg
bd3a0260d2e89e3e386a772e7196dc38
6c02490185ef4486efa2ca9d3b36f19fab6a6721
23279 F20101206_AAARJN baldwin_j_Page_136.jpg
74eb19613cc5088fe6426a6e5896faf7
07e4a2d0c4671f9fa3a0bcbec1a20c044e23c79e
8296 F20101206_AAASNF baldwin_j_Page_123.QC.jpg
6cba8329a6709332ac2bb7ae57ccf72c
9daee4634d8c976f5d1eef698739f0662a4aab01
5787 F20101206_AAASMQ baldwin_j_Page_097thm.jpg
a319b43d657786038ab2ba46829b109e
082ad2512167d1808cc8ffeea0494bf25b9fbc5c
55113 F20101206_AAARKD baldwin_j_Page_155.jpg
4f4a78a4344c566df919801bb23eb377
9d9c90931e486c86c2bb2e1d4b7e2b451ff297d6
86868 F20101206_AAARJO baldwin_j_Page_137.jpg
21dd1c0372f57396006d97d8bf43002d
f6e334a82bf4e2455633e2dd3a80438fc8e713a9
17568 F20101206_AAASMR baldwin_j_Page_098.QC.jpg
0f9f8ae2c113d3c19fbbb2fe42ed04b2
df930f3c7ee84eaa759e0a0753cc7933d9c5f996
69527 F20101206_AAARJP baldwin_j_Page_138.jpg
b2db5a386daa4b7305fbcdce89fb821f
3d74b51c2263423b9fad52009b039808b945b869
7203 F20101206_AAASNG baldwin_j_Page_124.QC.jpg
08f6fa7290b9ddfca1771b7cd58f26ea
123f40654ef3a0a13ef2d16468e31d7ed11f37a8
5679 F20101206_AAASMS baldwin_j_Page_098thm.jpg
0035a7e5631b44893a0bac5d08269071
941e49d52f5c4072870ad316bc8298d2767f7a48
39754 F20101206_AAARKE baldwin_j_Page_156.jpg
d8aadca92fe018ee9bf0bfb2a6dc3078
b7deab6af6c9e168f6e30e37d844b29003ca7881
63199 F20101206_AAARJQ baldwin_j_Page_139.jpg
9f3f2ad1cd053c2e872ec3d83d98cf3b
1542aa27585d4df677f58d6b0aeb8f5494570352
7896 F20101206_AAASNH baldwin_j_Page_126.QC.jpg
8349a26a73bb26a794b5db0c615bb6a8
fb1ae26b6f434dc9cb5c72384a053da63a23372a
21384 F20101206_AAASMT baldwin_j_Page_099.QC.jpg
21593b6fe1261cc35e1365b88f8aedc1
d52be68912ff64d055f6bb12ea5864554b3576d7
62285 F20101206_AAARKF baldwin_j_Page_157.jpg
659ccbec495b5931282ee68a1a787230
0355171bce3c5e331caa2248166b621671285d47
85640 F20101206_AAARJR baldwin_j_Page_140.jpg
83943c3dc7c29d9962e80ffc41653a4c
7ed201477eb9ed6902e8416e1aa60fd147692c3a
2684 F20101206_AAASNI baldwin_j_Page_127thm.jpg
81ed7c8eb91fbefed58b8e1ed7f7ac56
833d28de6def00f1996caaa6219f7a7310aacdfc
24957 F20101206_AAASMU baldwin_j_Page_100.QC.jpg
6df8aa7e80f696997aea079b5c4927a6
1b9e7925954d5453c07792e45feb730c95426205
68396 F20101206_AAARKG baldwin_j_Page_158.jpg
091315d832fe666d5a7081e58c244f1b
4ec574b7cc1318161545cd2ec943623c6462fc5d
58667 F20101206_AAARJS baldwin_j_Page_141.jpg
4b23b1f2945e166dcb36705be12e6e38
93e79d48a7b8b22e0abffa5efe0f039f683cc79d
2843 F20101206_AAASNJ baldwin_j_Page_130thm.jpg
60d2ff08062d3c38437dfe6586e31c98
f59108cec51a9eafd9753c65afef66ee60c1935b
25405 F20101206_AAASMV baldwin_j_Page_104.QC.jpg
48318fc561078555a1df07cf40b6abb5
1033a3b7641acb7cce172d80eb3c19cec44d33cc
102142 F20101206_AAARKH baldwin_j_Page_159.jpg
9a2336a94680d1fc5b279817374d0db7
268b377532ffa00023f41268b0e2c093108befb3
64578 F20101206_AAARJT baldwin_j_Page_142.jpg
f09acb6dee29b468e5fc294bdf7118e5
03dc805bf0ea3cd9dceae5565b173e6bd3b8876c
3036 F20101206_AAASNK baldwin_j_Page_134thm.jpg
52c6ccc6f04787e5e3b52dd84ec0c6f5
aeb89fec99ed98ff910c254612862ca2f8a0e2be
25477 F20101206_AAASMW baldwin_j_Page_105.QC.jpg
c192fbed3fa307fdb0d213b5953a066d
ad72a3c4776efa54c14357b8c00c26685e4392ad
93242 F20101206_AAARKI baldwin_j_Page_160.jpg
b6f4e3b3d6c2da3b36d0bbcb72713744
c71d04955f04f04d26ca73e90190d509ef263243
37079 F20101206_AAARJU baldwin_j_Page_143.jpg
76da889f00a504841a14836c69341c7b
b4312b3a26a22fcf7bbc5addf992bbd00bc0f283
28167 F20101206_AAASOA baldwin_j_Page_159.QC.jpg
90a5aa9a59510de8b38c3ef70b7ae337
fad692a634940734ae1a3df98218a96db97171b2
2808 F20101206_AAASNL baldwin_j_Page_136thm.jpg
cb76296b17660223fd3cd83d86720807
069909406b5ce32e16795922d8295e1b10c181b9
7031 F20101206_AAASMX baldwin_j_Page_105thm.jpg
6a951fc55356a46f6ebf63e115d2869d
fca05a54cc240d353c00a9016e3cf445a1299da0
74201 F20101206_AAARKJ baldwin_j_Page_161.jpg
c7e03e2329909d0a78aab1254dc86d1d
42b321ea7f94e36218d39566e8922ec4f1338db8
56075 F20101206_AAARJV baldwin_j_Page_144.jpg
8de8905b7f7a2ff0b61f130368564fb2
003c5c17fe933260ccf74bca114c45d472e979df
27360 F20101206_AAASOB baldwin_j_Page_160.QC.jpg
dfd7d23dc92b1ea30e10c8a80bc3cf6e
a8cfb2acb7e12059c817dbd2202cb67286fd0d76
22257 F20101206_AAASNM baldwin_j_Page_138.QC.jpg
db272c8581b68a85e4fb36b5461e4f34
f6058e3923b017d548dc3574e38fe2b564ad1744
7156 F20101206_AAASMY baldwin_j_Page_107thm.jpg
d31a906382f918a7327faaca4090313f
4f17c2d6cc983b14f6e5dd78fe93e00ce5e9ce15
30639 F20101206_AAARKK baldwin_j_Page_002.jp2
137c3ae634e77d611f8c949df80c4bbb
f11e7a6b45af818586dd4497032aef3d7297e627
55991 F20101206_AAARJW baldwin_j_Page_145.jpg
3e093d83bbd0fc59fb8aca9667f568a2
c68499139f3d68b9c75164a5725a3c83c4b7c4ac
21229 F20101206_AAASOC baldwin_j_Page_161.QC.jpg
662f5decd347322499cad7d1fe98bf98
e38c7ad12382146d2f7e87a73a125ab9a5b77481
19245 F20101206_AAASNN baldwin_j_Page_139.QC.jpg
d2b4d7a3ec1e50084908b61f5dfb24c1
2016d50388e9fc75fc13c40d4b51a4a43a5d9fa2
15074 F20101206_AAASMZ baldwin_j_Page_108.QC.jpg
2f6f1e092e680a942d409ceae6c308ad
d6b557188a5be0254d54905f7f373f26254b67e0
1051935 F20101206_AAARLA baldwin_j_Page_022.jp2
dee522842fb992f22b25889398e9a5cc
db298fbfa366755fc9cbd0d89175b3fb5e24a0cf
57319 F20101206_AAARKL baldwin_j_Page_003.jp2
3ebdbfdfdcbf4e55306bdf26132c6324
fb1371632409b45f4c9e28314117fb0928079f8e
60857 F20101206_AAARJX baldwin_j_Page_146.jpg
80cd9bcf29e4a0ec01d10e62f850ac43
62ff967430361f18b0d9498594ff921fa4ca496f
5590 F20101206_AAASNO baldwin_j_Page_139thm.jpg
df705ab03686f6553b1804fa4c5489d5
95ad32f5068784553793557c9118ec797f80d3a1
867929 F20101206_AAARLB baldwin_j_Page_023.jp2
de2475ef58a3ff71426d3f7cf1684ae1
037a6435e17dc7d78d7e5f9d487a02629d5ca399
1051966 F20101206_AAARKM baldwin_j_Page_004.jp2
9056e4dd9ff915d59c31f245fb35c7fc
fa0d143d004a372130c28fb2ef29a3a761109c28
82615 F20101206_AAARJY baldwin_j_Page_148.jpg
865249a5717a3e8be41ad00211178585
6f36f7afb6870947530979599150007934da9512
7523 F20101206_AAASNP baldwin_j_Page_140thm.jpg
91410e123b2731391d36d6486f24b093
3572ba02ab1fb65a2ff9cebb82d1e374bb5872bf
925681 F20101206_AAARLC baldwin_j_Page_024.jp2
4dca46cc06f56db4697d551f53e1b9fd
8a341336d8413b35ac5c021f22253c0d983835cc
1051972 F20101206_AAARKN baldwin_j_Page_006.jp2
7d88a42e00ad532f5e61f0e1eca84211
87ce8290a973ea0625ac7d56bbe3b9560069ff41
88044 F20101206_AAARJZ baldwin_j_Page_149.jpg
630f90d5f690e83e635bc7fa522ac24c
4d677914dc932435df9c3caf22926263efbe382b
19861 F20101206_AAASNQ baldwin_j_Page_142.QC.jpg
7549694a99928618ef00aa9697fe9276
fe284744fd884982c56b127c758bdb23040686ef
774984 F20101206_AAARLD baldwin_j_Page_025.jp2
16e676d73334e558faaba112740bac1c
98dda4dbc75fce38fedf8a9cdb405d18a1c1256b
F20101206_AAARKO baldwin_j_Page_007.jp2
71a975793b668e4a97f22ed2965a7ed4
cabacce0bc97826cb1222dd3a8e4771a4b8a3314
3944 F20101206_AAASNR baldwin_j_Page_143thm.jpg
af4fee9ca05992e5d47e85a118dc80b9
8e64b068761a615c9c9cba76015ce42377bf234a
819894 F20101206_AAARLE baldwin_j_Page_026.jp2
11e411e8700a7f42f315dd92e9631d44
259c51a26de9d9bb8d3f03716f1f4ac65cf6f900
1051978 F20101206_AAARKP baldwin_j_Page_010.jp2
fa88dad2a67c98bc9fd4715831d1f551
f57e5eb00627658af6bdaa3fbdf87937ab9d26fb
5931 F20101206_AAASNS baldwin_j_Page_146thm.jpg
3df092412c13f897ec0fa322a6e771bb
b41d29ff6db947f8a13f2479b526942a761b9a0c
908973 F20101206_AAARKQ baldwin_j_Page_011.jp2
1e1f509a18f3dc92a1ad4f1113b33ef2
ebeff8881d7532597159897d25517607d1b2cb04
6055 F20101206_AAASNT baldwin_j_Page_147thm.jpg
7165a1a9bde8d0e3fcc372cf00379ba0
1109d563c408a4315c03f41c59fa46a47ad8c8c8
702670 F20101206_AAARLF baldwin_j_Page_028.jp2
d5638a63e402eb3a73fc36c4cdc9e186
0a315d36eca393fd24f4ab28b47cc8084a3935dc
70020 F20101206_AAARKR baldwin_j_Page_013.jp2
b1e5d5c34b7c73c289901c80860d5789
49d98752a3e18f1b9b34a9bb34771e58cd1bc221
1864 F20101206_AAASNU baldwin_j_Page_151thm.jpg
d8aa42dd5f5af541b2f2ca5ca71b23cb
bbc0c098583b881438a34dd22f5b8ad35a771b16
837259 F20101206_AAARLG baldwin_j_Page_030.jp2
8155a6fa5a27f597ca4c4de5eef74ef4
ae5abe88d9141b425eec26da9167a0cc744e983d
F20101206_AAARKS baldwin_j_Page_014.jp2
25e1dc502f7ffd8ba4ad54f67f65e4d7
151c5c74a03f9dc02de20abb64a99aa462cfe612
16731 F20101206_AAASNV baldwin_j_Page_155.QC.jpg
c69077054cb5d470b48b722b1f7ad925
44be1618da45a82c0a5eb252f8643c5a85b0020b
444358 F20101206_AAARLH baldwin_j_Page_031.jp2
8dc44154b0efda6310475c6219c17e0b
dfb7b73eb4d515790007ab3d6dfd5dedde7a99cd
F20101206_AAARKT baldwin_j_Page_015.jp2
d360f6f3b8e9afc04c70cd3ba1c14439
1b22c46014de901e7391154a619ad5d1ba2a821c
11274 F20101206_AAASNW baldwin_j_Page_156.QC.jpg
42351c21f61bf687e3767ae22efbafa2
b5983f9d279229f8c91a182745fb81b906ebd06d
651475 F20101206_AAARLI baldwin_j_Page_032.jp2
1756aa375e1a9b699c194b93c5c4a300
4c0e4be9d350a24aa2ad2b12c7b2cdedc0bc4a17
956130 F20101206_AAARKU baldwin_j_Page_016.jp2
083238d0fab77697f3200971908558b6
60d9d997cda15cf8821fd68c9c780864a3ad9cd7
3385 F20101206_AAASNX baldwin_j_Page_156thm.jpg
99fce67b7a7d9f2ec600b28f28f0f8e0
8f64604fbf8dba04fac63bf577e4b764526cbcf6
402568 F20101206_AAARLJ baldwin_j_Page_033.jp2
67628713b3204c73362864f2d8460c63
e826914b6045926c13b692a681018884f3c2ad1e
1012614 F20101206_AAARKV baldwin_j_Page_017.jp2
cb00fb7b60d3d100e25c50a9bb8fbe60
b99b3cf86ba5e06e6a4e6460ba4b8e185602061d
4903 F20101206_AAASNY baldwin_j_Page_157thm.jpg
a06982d90d0cd7350a775e93cb80b752
c3016be506b27329c042d9868d0d74d8a178ea8a
585164 F20101206_AAARLK baldwin_j_Page_034.jp2
f1359e6dc442a8e9bc5ccecb73592041
c6f81397e2310580ba666a0169d16ae652adea9d
F20101206_AAARKW baldwin_j_Page_018.jp2
6bac984e777351906ad0c39297300174
1a4103592999e242c6c7657440390c570e38289f
5388 F20101206_AAASNZ baldwin_j_Page_158thm.jpg
3e916f5001ea6f8be235a99566c1aa3d
28a4b0b4c17e609f5a3583073e34f26970c88e58
1000524 F20101206_AAARLL baldwin_j_Page_035.jp2
2214d385fd7f14d71cac3ce72fa50c79
89d5bcc6e5492542fd9d08cdc0b88ac28e7bfd85
762500 F20101206_AAARKX baldwin_j_Page_019.jp2
82f7de7c447cddbee08cd86699131a4c
c41916671cefc7cdeae3348e10e3b334ef6d1ead
648950 F20101206_AAARMA baldwin_j_Page_053.jp2
217ace935cc2372ec3cfeaaf791f8737
21ef92de8b455db9a21d801ac1fd064f75ae65ae
680485 F20101206_AAARLM baldwin_j_Page_036.jp2
4f86fdf0d611eb7737586f537c89f885
2f1f8990914fdf40755b2aca773124b12d38d09a
924887 F20101206_AAARKY baldwin_j_Page_020.jp2
375ccafa3fc91ebd21526f301d206e79
95e1c006c482ea63b4de840647b39d7c5f68561d
592026 F20101206_AAARMB baldwin_j_Page_054.jp2
a0615346cc29988fbf04a57e530c55ad
d0a490aa2d58cf40dbae669d98b2f2c252ac387d
921679 F20101206_AAARLN baldwin_j_Page_037.jp2
0364c59abbd281cac6d9b803602074a0
f58c9096109a404d74681e3307da7d1cad00cada
844191 F20101206_AAARKZ baldwin_j_Page_021.jp2
409bd9a74499785459d188e64e28ac92
179ab287e270ef8561b7801a1966662ef98b3f35
391047 F20101206_AAARMC baldwin_j_Page_056.jp2
ddebd48b65e4d0d2008225c7310f87e5
dfdba07d4448a5c58888e26d07f7ba9efa44274d
1051984 F20101206_AAARLO baldwin_j_Page_038.jp2
24e8e90e889cfb75b36cc2a70ecdffe9
4bec2d279ec9b66a7de9aed8e49a7e017fef49d6
633130 F20101206_AAARMD baldwin_j_Page_057.jp2
c86195a91e0def961d28c9b4b53fc71f
04f8e1ce75f0ad0d1fb3f278d4e0dccd276f77bb
454284 F20101206_AAARLP baldwin_j_Page_039.jp2
bcfe31c32226978c641b022f827254ae
e63e1e08980f4368d9ae78ff7b433eec5851fd31
722951 F20101206_AAARME baldwin_j_Page_058.jp2
0c1b9c785d4e835faa0814f48821ae6d
9449e34d43f40c66ba2d1b6706a7c224896da363
603635 F20101206_AAARLQ baldwin_j_Page_040.jp2
f567611840572415fbf7391b698f75d5
df264fd4fd8c7d34276905913f2283a1a641958f
777409 F20101206_AAARMF baldwin_j_Page_059.jp2
d5b166de2be3444af0167a7043260d59
1b3aab2f7bb36cdb588ae9d2e1be3e7b9ba1c61f
481654 F20101206_AAARLR baldwin_j_Page_041.jp2
5a6dd7bfa4f8d046172327e7fa3dff83
ca5a04871f1f9901d14571a29f72fb1b424a94e3
973108 F20101206_AAARLS baldwin_j_Page_042.jp2
ce47c9594ad806d9a36e426ce9f513e6
59dac9feb934cd4a8c0e4c7073dff36fcbc59361
991175 F20101206_AAARMG baldwin_j_Page_060.jp2
878dd4a0d55233ad5b439ef6c201e935
d98d04d46045b521f00422635cd0b0a45b966983
744379 F20101206_AAARLT baldwin_j_Page_043.jp2
ccba9adee78d1c411d29a3c8328d3195
758c5d68fbb0429ee7a675fdda05eaa41b034e73
1026716 F20101206_AAARMH baldwin_j_Page_061.jp2
43930f62ef205f5aaf6e3d562a768df7
39bc83d7f72027877afc789d28574c038c929e7d
716684 F20101206_AAARLU baldwin_j_Page_044.jp2
1f16365ebdeb493c559785eecd8f5051
c2c2cc732e4fed4462834b1c988520cc90785a88
123384 F20101206_AAARMI baldwin_j_Page_062.jp2
2dd48453c39bcd962fa7f497750d650e
db171dff17118d5212dde39622a1cdb7f6955524
459557 F20101206_AAARLV baldwin_j_Page_046.jp2
dad5d0bfcee882663d9ea5077cf91a89
e2963b765d6ceebb157a44b967604bb3335defa4
1051965 F20101206_AAARMJ baldwin_j_Page_063.jp2
aeefd8afa4bf30e38406a55d1983fd8e
57926c5eb4bcaf0444aac8e335c459e71994a971
449152 F20101206_AAARLW baldwin_j_Page_048.jp2
f5c49ee2c8125dd88e4f98c5d66cffbd
71745a623ec04f74b7f9b7cf3f08216a8697790f
768870 F20101206_AAARMK baldwin_j_Page_064.jp2
29c9cfbd2be360ca8d457f1828dac7eb
0920ffea56fed6d73a178d7a254af5d7ac2d2db0
901510 F20101206_AAARLX baldwin_j_Page_049.jp2
070d9b26f6d32e4d8d8833e3bf5bab31
92c156e9ed789a9902aa98089e2dcba0bb2535fe
554457 F20101206_AAARNA baldwin_j_Page_088.jp2
ce024d7e30ba7aec1865af1b0f771401
380ff70d4901dc0d920368c21bd1d931a9ad911b
486996 F20101206_AAARML baldwin_j_Page_066.jp2
f4095906b2d94f47c4d589137ad5192d
8bf030b707ce6bc96e8485ed19a1ab1816c18eb7
522048 F20101206_AAARLY baldwin_j_Page_050.jp2
50dcded4ebfc04afee03c05d8e82b1cf
a689b5f9568ad94e5e887d356aecd3fcb32c4242
488096 F20101206_AAARNB baldwin_j_Page_089.jp2
5bd9c423d8b5134cac501f962fe4515a
bb54ea034eda326972dc0937127009c622c4d5ae
805432 F20101206_AAARMM baldwin_j_Page_069.jp2
62a78fb0609f2ed7863a0c97ac1d4453
b056f47536388342baac9d0229909b98d6778ad5
834405 F20101206_AAARLZ baldwin_j_Page_051.jp2
3ee622f9798f03d3c53fc57392332bc0
175db8335b69226657e06854d10f970b19903f47
866632 F20101206_AAARNC baldwin_j_Page_090.jp2
cbb4686b0e0532b7824168d93d762810
fc41a20242496fec1a3f5f3cfaaa02ffd9c980ae
958195 F20101206_AAARMN baldwin_j_Page_070.jp2
22e0fe15a50d5b4c65a52436207cd9a9
06e35dddc36f6d46b989a2d7128849d559e4c5e2
951375 F20101206_AAARND baldwin_j_Page_091.jp2
5b91f585393d8c54efac6ae584164c69
73b543c708709cd43dc13f7d245cbfdf77e7b084
876819 F20101206_AAARMO baldwin_j_Page_071.jp2
d31fb87cc859223c093358a31ceec6d8
edf7d58d7173a2cf743a5431ad7454ff7a6116c4
879284 F20101206_AAARNE baldwin_j_Page_092.jp2
2b8263cb51ce5bd39ee6c0dd1b08beee
7c2a95bd23d26eb0e77bc6a5c70f3a2b78f4a3a2
633991 F20101206_AAARMP baldwin_j_Page_072.jp2
046bff28f5df792b9005f9f070e46a06
521e8ba8759c33dbc93460224d8d6dfc07adbd09
704728 F20101206_AAARNF baldwin_j_Page_093.jp2
9a098036bbb876ed912d8ca1b893fc7e
3e75bf058c8a9292df41a0bb0b9ac9e1ea844642
955054 F20101206_AAARMQ baldwin_j_Page_075.jp2
fb3517efdc6532e32c77c1b8814d5206
2def08cee574b3904858d0ba62aa0707049930ee
697235 F20101206_AAARNG baldwin_j_Page_094.jp2
f6aa5df14301ab6189ecc46a3cf587c7
e684ac800aac8845c0d831911f521d3bfb2ef8f2
551062 F20101206_AAARMR baldwin_j_Page_076.jp2
cbaef45b86b8660b5f55061760c4c80b
5c15975fcbcbfd507ded922a2ca05d0f6e1015be
619043 F20101206_AAARMS baldwin_j_Page_077.jp2
d92e51ce3939518f0ef400b5d331a12e
702333259e9e7a11de034ae9ee63ae5046050afb
922415 F20101206_AAARNH baldwin_j_Page_095.jp2
233f2b8d315aaf1a9450451dfee8dd9b
5d448b457426bde42b5049c9f4cf9e1d991c51fd
614458 F20101206_AAARMT baldwin_j_Page_078.jp2
c76df4e44bf86a69166407d6c77329c6
6e32af3d552ad6f5db2983d91f17c787273d0f32
836114 F20101206_AAARNI baldwin_j_Page_096.jp2
b1a17618f645895584c7f0660814969a
fab2b689da4968e0a8cb019f982ec4f999d87637
646345 F20101206_AAARMU baldwin_j_Page_079.jp2
50faacb4119ea3eb9f892f4cbbc2fa82
99c5f0286441517a9249537b0618124df394c666
800028 F20101206_AAARNJ baldwin_j_Page_097.jp2
87f28fb85777073a9565c1aa7c50cbf2
44146c0424913d9d5660f1336a9db51d733a2aca
928261 F20101206_AAARMV baldwin_j_Page_081.jp2
4c1cb8b29fce8f70ba6790b1fdce040b
160172bf512e4df32d2ce9175e3af7fdc3c7b586
802379 F20101206_AAARNK baldwin_j_Page_098.jp2
27549d2b50a3736559a492459b3bb5c5
89eff6aea3f5cf7788bc16db61ef624487411efd
1013982 F20101206_AAARMW baldwin_j_Page_082.jp2
f7ac71f5f61b568a9615ab49860e5dda
e231102a89a1d51a8135c021e8394957cc4e8fb2
410127 F20101206_AAAROA baldwin_j_Page_118.jp2
fb875e263b1f3151e27246a7eae156ef
ca6d5f437df84143c6b5259afbcf873bce6f0554
955508 F20101206_AAARNL baldwin_j_Page_099.jp2
a74bcad10711138cc7d1a986b53d0fc6
fbf2d5f8136e56f6f7a73b5b1d300244801d6460
837150 F20101206_AAARMX baldwin_j_Page_084.jp2
7d107866ab5a1ed810b02a6d20562e53
bdcccc08fe38494f9c8ef65f64bbfa878defb447
466582 F20101206_AAAROB baldwin_j_Page_119.jp2
2f87ebd612204dde2c3ce7e07624dd25
15e96ee58d590b297daf2eb2833d34a47b2b2e0a
991689 F20101206_AAARNM baldwin_j_Page_101.jp2
9916dc3811f65f7b238937ba3600a167
6dfe24ad40f71a99125ace5742d259f7bd5c4fa3
874368 F20101206_AAARMY baldwin_j_Page_085.jp2
8b52b8c64a6eeb4faf128fbb7f4a22ea
55820f442c83056a4e12686d7305ce5ada5a2fa1
248394 F20101206_AAAROC baldwin_j_Page_121.jp2
da244db14a32da42443544a63d5597ac
ebbfc4bc6bc40e264574c412a3af2b72119a967f
812029 F20101206_AAARNN baldwin_j_Page_102.jp2
9a01f735bfba0df8df4b130b43e4f635
d89064030c9615df6d3b5c6b798fb94345dc03dd
445358 F20101206_AAARMZ baldwin_j_Page_086.jp2
f16eebddb0379637865dc045d1df6644
50dba430fbb59784b37fee5412ab1dcbe3cdba9c
350472 F20101206_AAAROD baldwin_j_Page_122.jp2
a03839fe3419332bfbc7fd858d3b9e4d
5e0f0d4de524b76a1a79c05ff8357cf7ed14b74f
F20101206_AAARNO baldwin_j_Page_103.jp2
36db86658d858d83d523097dcc22a76e
0c1f1bdd144f8f6668e84bd218216ba75e997fbc
491113 F20101206_AAAROE baldwin_j_Page_123.jp2
6db20418c6f46ea5c5d16d8dfc985062
1bbf9c417560e5324bbce7592ec9da31a3f69c39
1024776 F20101206_AAARNP baldwin_j_Page_104.jp2
0c1a5c0fa5b0c9aaeff7e536148e9ad5
6f62d3398c9185ccf49ef765a9c48336f965c08f
340362 F20101206_AAAROF baldwin_j_Page_124.jp2
41fd443cf0c08fc6c1ba9acedc1e0b1b
6b15dba960b3364e7f60b44eaa3ad956daa096ff
1051960 F20101206_AAARNQ baldwin_j_Page_107.jp2
0eca9956d3a28ee10ca0f40baba30923
992027e433983ca1859f2a554510c66701744ef1
627486 F20101206_AAARNR baldwin_j_Page_108.jp2
35bea666486464128c8c23819bcb1a60
50077a1dc889931e85ba7d32ee6a397476d8fc6a
285013 F20101206_AAAROG baldwin_j_Page_125.jp2
933cdbe205d54894b513f73f8fc7eb5e
5abc23ff21fc4547f19a814865f486b484f6e4fb
371775 F20101206_AAARNS baldwin_j_Page_109.jp2
841e63c90f6f8ba631605594ef275042
a8aa655b918e359bb577b6146e9569d9617b57d4
384147 F20101206_AAAROH baldwin_j_Page_126.jp2
b57395d883e448c896d1647dca73460f
a7bc0dca53b399b176c7b4c68d2e56895e56cd28
406529 F20101206_AAARNT baldwin_j_Page_110.jp2
bb3ba9435c20202ea2abcabdb8c46c72
6693d8af67c3e0769026b519c7d0897c3ef22e3b
415189 F20101206_AAARNU baldwin_j_Page_111.jp2
e0df2cf0d01dbc6893c8e2321e802bd9
ac530a6265c85180f77b095e22760d505eab3ace
466753 F20101206_AAAROI baldwin_j_Page_127.jp2
e6b6f1cf416f9dc7785106bf08b12034
f5b6dded2ca556636524afabc7d8de871f199712
416569 F20101206_AAARNV baldwin_j_Page_112.jp2
43a5221bf3ebe84e924545b8b81a502e
69c0b9288568f426b48214a23936163220c4150f
391990 F20101206_AAAROJ baldwin_j_Page_128.jp2
02f91074d300be1d9b021028fe42e9c3
39ae2ae29bb2da3d94e4f3fc7c0d3744fd6d987d
374488 F20101206_AAARNW baldwin_j_Page_113.jp2
7dafebd29b835126d442efdcbad88e5f
d752286f52246805f225a114f384e4fab2c71e8c
559567 F20101206_AAAROK baldwin_j_Page_129.jp2
a7bd898692f76bf962bc072571fc220c
f75b2a14983eaafbd38d2bad3f9448b1aeb213c4
393841 F20101206_AAARNX baldwin_j_Page_115.jp2
ede414e887f55f95f1ac1d0244fd0589
6960d94dc594c3c53b19251d92f71a22f4b6eb5e
1051948 F20101206_AAARPA baldwin_j_Page_150.jp2
aa0f501074997731a4da2ec3e455e820
407d9705581557af74f94eee577dca13f3254102
473062 F20101206_AAAROL baldwin_j_Page_130.jp2
babaafb2fdcf4d2de17613c38282d2e9
5de75ff5667dab8d7e2a303aee2b4a7823ce552e
380959 F20101206_AAARNY baldwin_j_Page_116.jp2
638b984169f9e2029b89ea7f4829b886
bb6760576d5d70444ef61436c0ede570b0be9046
721266 F20101206_AAARPB baldwin_j_Page_152.jp2
3e891c5992b4a828e47a17ee3a7c8c76
b96d09122fc18044a7b86292a0b69f72df26d5cd
737599 F20101206_AAAROM baldwin_j_Page_131.jp2
c96dcc9487e6272196a91d3c097bfef5
ef4e70d8c7ec3519d566c80289302e20d5ffc8a2
395847 F20101206_AAARNZ baldwin_j_Page_117.jp2
ef930fe65fdfc416f767b398ad09e1f2
4105ad12ede3612d2b92cea4ae032e6220d06bd9
658237 F20101206_AAARPC baldwin_j_Page_153.jp2
bef07b27e3bc71391a9ae873a96d2704
0da31f89f293c94022a7d164de8ddd533136eaa7
481790 F20101206_AAARON baldwin_j_Page_133.jp2
289b524cbbe701e48a1f729144d384a1
3bcc3b634d2e28ef86afd7e7221cf1b8484e1aa5
516139 F20101206_AAARPD baldwin_j_Page_154.jp2
61f4e0d72951eb7c19823af95e04d05e
664a096e0fb95e0d96d5f066ba7057abf2cce518
515656 F20101206_AAAROO baldwin_j_Page_134.jp2
0808c62207f9ecf4fa18195fbff3f005
264f3b986fd7b5d79e26e6e719231963d3f057dc
670672 F20101206_AAARPE baldwin_j_Page_155.jp2
476f6a30f583cfb05aace554e43fe8ac
7e76a2f83542e729fbad6c362ebb75bdc317104f
522306 F20101206_AAAROP baldwin_j_Page_135.jp2
829b2f3f24890ccdd32c0ee3e790d76e
0cca1a6a8198eb4b8d24b3582d3f76f355bf452b
468233 F20101206_AAARPF baldwin_j_Page_156.jp2
84a7bd772e4e33ffc9301393bceca9c2
e8013bb21460c6574f7f2a74ef1c359d1943e182
1051957 F20101206_AAAROQ baldwin_j_Page_137.jp2
b88589e549401e5000bf8eeb8a661d86
85eb3eaaa65557688b1f70e2679473eb1fae24cb
799836 F20101206_AAARPG baldwin_j_Page_157.jp2
492094341cb2835e6c41d4539410c62c
d8a05f0bf38cd831298e6b1fa6bfec6f3a2d586a
912772 F20101206_AAAROR baldwin_j_Page_138.jp2
f5fc5419a382088d48b67d806b6f3356
e91220e56ef76dfb79d2a0900baa902a0ab631fb
887352 F20101206_AAARPH baldwin_j_Page_158.jp2
a6a79b8ce4eb83248721a1a3cc2fc29f
005cae6350c239d3f1eed252b1c0e0d9b7d855d9
803002 F20101206_AAAROS baldwin_j_Page_139.jp2
2d99d6f6eac5243b37502ed87e6827b8
a4f878661cf3646c38083a60191d65f3695af9cb
1051980 F20101206_AAARPI baldwin_j_Page_159.jp2
8e1549ae2483c8d21309cb72e30687e4
538775af3241b990d551e0ac82c6ace4a217e7db
796485 F20101206_AAAROT baldwin_j_Page_141.jp2
11ba75620019cdfcbae10b1d45db5ebb
e685a7041ffc489ba7ce28fe1846a157336b222f
897541 F20101206_AAAROU baldwin_j_Page_142.jp2
4d366bbeb31f5a857781582cd0619a7b
7065a2a998bbfd0647948542be8c7d2870b26f92
1051968 F20101206_AAARPJ baldwin_j_Page_160.jp2
69d3a1686262aa245152d604026c56e8
035c3eae9a959844fab59d451f579c8ea76dd4c2
441047 F20101206_AAAROV baldwin_j_Page_143.jp2
8da156e8eeaa6a6e0376e50e2ebc5251
e310917e5692baeabad780760ce6aece1a509ede
1024463 F20101206_AAARPK baldwin_j_Page_161.jp2
1e0c0932e96b1c22fe98d974f5a4e7a8
b0852439b9ad82441c42e8626367e3214601349a
685700 F20101206_AAAROW baldwin_j_Page_144.jp2
665cc8bd1a7642e5b4bebbc499e83d96
6131d0bdf92d9802fa8c5f9c91f81de2d33f03ac
F20101206_AAARQA baldwin_j_Page_018.tif
b269baa1da8962c64915f1a31c1b6ad5
c02777276a65dcb68542a235185f8b47a534fe29
639079 F20101206_AAARPL baldwin_j_Page_162.jp2
84362a140ddde7505a849574715b1fb7
48f0744054bca80803f8f8b06006955bb254d5fb
739142 F20101206_AAAROX baldwin_j_Page_145.jp2
e2a0a7eafe2a46d2ca46e5ecf36bbdbd
1b5993591185e3c275eb2a100befc2cf2fae4acf
F20101206_AAARQB baldwin_j_Page_019.tif
ad62de4366b744f324830aa3f67081b9
01592c366784c4ec90e3c22e2555a6ee4dd85b34
F20101206_AAARPM baldwin_j_Page_002.tif
c53aff2f84bdaaf378d864f2d89883d3
bae1a603fa8af027b17322381bf106a9e0dfb382
903593 F20101206_AAAROY baldwin_j_Page_147.jp2
be2064ee34c5b21e3bc6d7bf162f7693
340bb43dbb6dca0f689873823fa70a2439b9ded3
F20101206_AAARQC baldwin_j_Page_020.tif
7598d338c42cb15274a9842724abec8b
23f88f5b8672aac09699e968defee3686c9564a8
F20101206_AAARPN baldwin_j_Page_003.tif
0bec07351f40a31e8a2fab97486e753e
6f81950413e1ed059745e39749539f02fdd89f8d
F20101206_AAAROZ baldwin_j_Page_149.jp2
4fe9f4296e7344db97ed3cae3be0702a
bdfeaf32c536ed51f50ea9c020206a40735aa239
F20101206_AAARQD baldwin_j_Page_021.tif
b37c8943a768bc7223c943ec65fd1289
521cf1ba6805d34d3659c39ae459b8ab3186a404
F20101206_AAARPO baldwin_j_Page_004.tif
d0587025cf862efa3b4c2b4fdb9b566c
54be798c4ca4fc79a284377947e91f4332705e7a
F20101206_AAARQE baldwin_j_Page_022.tif
c10755031f6d277aba815d3c86aa4856
ab2f1260846376446c863084edf70e5845df5d12
F20101206_AAARPP baldwin_j_Page_006.tif
cb6ec079311920dabc7e72180872d3a6
03f5734d0909c85929c67d5dd7d96055639691c5
F20101206_AAARQF baldwin_j_Page_023.tif
e10ada441c999ed4d7a9f857cfe9dee0
7c5ffb651db02a35d8c5d69a9231f8af597d00e6
F20101206_AAARPQ baldwin_j_Page_007.tif
c041e41f4bb374dced1c4c371894b256
f256e197ce6bbb633b78966c184014fce9a061f9
F20101206_AAARQG baldwin_j_Page_025.tif
4db19bca0f92f3cb520d57ca3d99505f
025b5f2f3e1efe35e08ae53214690d6913b57306
F20101206_AAARPR baldwin_j_Page_009.tif
8ce3b1d337214dec9bfc591588e87858
3657751142e9ad9dcad8d89ce39031f1ba997773
F20101206_AAARQH baldwin_j_Page_026.tif
988e64cbb4aabf1d19db72aef4a8cddf
5b5f581a65643deb8f3d6ccc0be03438fe65ba9e
F20101206_AAARPS baldwin_j_Page_010.tif
8dc557cf882d248d66d2a5284bd7c9c2
5b575f56aa367d036bb365596a939138aac954bc
F20101206_AAARQI baldwin_j_Page_027.tif
c3262927d1fa489817d5a8bb37e69740
a1a8c90225020e73d1d927f1336e796050155565
F20101206_AAARPT baldwin_j_Page_011.tif
b8c00bd8ac429304624a33e6a4a54131
7f46391c1eb4bf1d6d45a6d297ceecf088787876
F20101206_AAARQJ baldwin_j_Page_028.tif
f41899c786c7675f464cdb9f779b4ad6
97ecf978d1efad29f1a725975ba4eea21e3262d1
F20101206_AAARPU baldwin_j_Page_012.tif
5abeac0663069b236317746f4340d149
f3b17a346a60cf129ab0d7468e54963939d71a66
F20101206_AAARPV baldwin_j_Page_013.tif
3dda4c9dc5d3aa8d1123e39b8aade899
8c2b82db4f51f4296cdfc996e33c07c073ea059a
F20101206_AAARQK baldwin_j_Page_029.tif
e19df849338ad85b0c4119b565b70469
2774c7a81462b806e8e9a8de16d91a8ebdb05041
F20101206_AAARPW baldwin_j_Page_014.tif
43d35886d4e1a276986d580197cf14d6
94e62c59c490c4fe99c43b63a584bd42a4c11cbb
F20101206_AAARQL baldwin_j_Page_030.tif
db6649ac0e416d9a4f65909782edc15a
220e7528b83ca4ccae5e45f477e77ad110ac972f
F20101206_AAARPX baldwin_j_Page_015.tif
a446bd1e15a0025d096463c340184acc
f304fb691df0ea110c27045ab9541efe73df8c18
F20101206_AAARRA baldwin_j_Page_048.tif
96a456af298e9ac069d3a95f9a958062
4c2f6220355178e8ffe5482e638b25bba1fff567
F20101206_AAARQM baldwin_j_Page_031.tif
044555ecda031da48932b4af621737b4
fbc9c68b7af641438cdd1cbf7d1ae95a7b62cab0
F20101206_AAARPY baldwin_j_Page_016.tif
19b30f77facd9ec42776701f98f93a26
5f024063ba4b3962511b6d6742f431a045f3e79b
F20101206_AAARRB baldwin_j_Page_049.tif
f0ce65b8ef63c3a00fed984acda67672
b869681559d88f258395510dec1d07c78d876caa
F20101206_AAARQN baldwin_j_Page_032.tif
a5fe1635c1378834afaeb9423283f09e
a6faf52542d196e9badc7a5d293d1ab57a5fae3c
F20101206_AAARPZ baldwin_j_Page_017.tif
d16f0665578e662e32b3df0c53e6fca2
e671f144baf26e2f2cdb6227cb797524fe941038
F20101206_AAARRC baldwin_j_Page_050.tif
bb1a9736b2421e58940a193be8f79628
3b9c77601fd50b14ef57ab5a4183c5eebdd0b3a1
F20101206_AAARQO baldwin_j_Page_033.tif
712365202df807c5fe75aa72f741f431
59d90ec7de67afc353571c31afa3198f534ab1d8
F20101206_AAARRD baldwin_j_Page_051.tif
854a69e834c65d14c8b8324bee26058a
539ddd98bd342fdfb7633d93825620fa805a0c7e
F20101206_AAARQP baldwin_j_Page_034.tif
7c24fa513d9a4948c5df92d629255fa8
839bfde185f89d64d281cdc851389427fc7f49f8
F20101206_AAARRE baldwin_j_Page_052.tif
1b5ed0a659a5e4655cadf00d392fec8c
31a8532d21b949e4a68beb0072c73d292b7eaf79
F20101206_AAARQQ baldwin_j_Page_035.tif
5ecd4f666631f357bcaf84a1d8206f80
ef695a9fbe43fdb057790a234c31c47aaa973913
F20101206_AAARRF baldwin_j_Page_054.tif
9eb2bfae90978b9c19bc6c1ec48bdd32
438f5cf6756ede80760c6244708703f25fd32c1c
F20101206_AAARQR baldwin_j_Page_036.tif
f986756475ff94b9b6e0b38ea227bf52
9f4a5354186b3ed4773f2af60932fc3e03dbb008
F20101206_AAARRG baldwin_j_Page_055.tif
728a99e986d3f2ed613d01b2428d5c1c
f5e5ee0b086e53d4cf1989b3c46bacee8d479ab1
F20101206_AAARQS baldwin_j_Page_037.tif
e9cb0d9f2600d51b97aff92febdc6532
e64a3877444b0f62376a776901ea490f06175cc8
F20101206_AAARRH baldwin_j_Page_056.tif
05f725b5d04204c14c2a401e9365e5fd
bfc90bc5c8a4aa657fc127c8ce47ff27569a6869
F20101206_AAARQT baldwin_j_Page_040.tif
2b42465599eb890fb99050c376e2bf3e
27be5ea437b835ce1c497738f0a9cad33578d2b3
F20101206_AAARRI baldwin_j_Page_057.tif
325daa673fdc0cd841fd4c3de0ee294b
f5e4136ed2975ea34b118c247141e920e2f00fd1
F20101206_AAARQU baldwin_j_Page_041.tif
67d580898b570298d7e1ccd37b494634
fc2755592bdefa41b169fbae48a1a6671cc353c6
F20101206_AAARRJ baldwin_j_Page_058.tif
90262a29091e7321debc8e8ac535e21b
ed7981afd87da1a59cf9d254d6d74c3ddae374f3
F20101206_AAARQV baldwin_j_Page_042.tif
dc781f024d5e439af9ac706fb98aac51
5d03b251cd011462dd718bc29410e6d532c6528c
F20101206_AAARRK baldwin_j_Page_059.tif
162f699b638eddd7c469a327bd531e26
f7926fa1228aca8ea7efed7a60cb3f59bdc2e090
F20101206_AAARQW baldwin_j_Page_043.tif
1fec261aa4e0532d890ef9f380839015
147c8d799d910c6a157365a4cec1736be2a98c8e
F20101206_AAARQX baldwin_j_Page_044.tif
ed617f7442a53b8087a3a98aa73ebf48
b2050df8fe486c4858f8331ea417d7ea6ba49630
F20101206_AAARSA baldwin_j_Page_079.tif
4daa55abe3e5b839def4dd90ce82c445
dd6daf22fe3a24cd3360708a49981c24570c46bb
F20101206_AAARRL baldwin_j_Page_060.tif
9c19401cb941b760dd35047629422273
e5174941d38e49a74c4be661a55141c9735e9b26
F20101206_AAARQY baldwin_j_Page_045.tif
041f260e05431d50b7344e73b9e89ec1
5a0bdc922ea472c7ff6d41d334959a44daf471bb
F20101206_AAARSB baldwin_j_Page_080.tif
b0c6cf6a930e0481ad49cac45150e5d4
2a88ff9c02313fde3cd2289c6f6ce2fdb14f6e78
F20101206_AAARRM baldwin_j_Page_061.tif
590857e0868c1bcab1bf626d1149c6ee
851bdab570257bb7e08d6869bc6866632b707973
F20101206_AAARQZ baldwin_j_Page_046.tif
e13e39c9819ecdaf4901ffff93308a4d
22501552b1bc60975794a3c6219859752ae56166
F20101206_AAARSC baldwin_j_Page_081.tif
da9bae2c90e9fc7ab7481f501acae706
2e3d281c25e31b6fee0e83342036b37315763f17
F20101206_AAARRN baldwin_j_Page_063.tif
89dbae9f44eddead8868e0e6f7e44460
125c93720deb935b850880b4ac611a3a34f06a0e
F20101206_AAARSD baldwin_j_Page_082.tif
47782daf85b65870a2e69e208cb9fc82
0ed14fd059636ba394e21b0dd288434f3e4ebd85
F20101206_AAARRO baldwin_j_Page_064.tif
7bf1c0aa9ae923cc75e6367905b401ce
4df02115ce661f7960e19c8b3ac46ee23eda8c8d
F20101206_AAARSE baldwin_j_Page_083.tif
3f4c416d223edadc90d59a9ffcb9a0b2
eea2aa166c2ea8a94606a7a39082194884a818b5
F20101206_AAARRP baldwin_j_Page_065.tif
4053f7f22ac3b6bbcd44f547b6623ea2
36d57c18f7f974aeea847226fa45fa7dcf2830c8
F20101206_AAARSF baldwin_j_Page_085.tif
892e7fa42fa951056c6c1f82ca57441b
60152cf7fe933f3003548b2b847479524a19b38f
F20101206_AAARRQ baldwin_j_Page_067.tif
9bc340eb0127bfe2a2393583dde5a213
67702ff9093c35654eefa963c022b1eaca195e56
F20101206_AAARSG baldwin_j_Page_086.tif
f9afc73e40f3e9f0fe1ceedb22bf2a84
8324eff9dc8aebaf28bcc9aa9f22997f3da1a0be
F20101206_AAARRR baldwin_j_Page_068.tif
04d4767c2092d78ecba34ea5be7cf4c6
05370d13f0c89961ce5ef158e1205987a11b8dbf
F20101206_AAARSH baldwin_j_Page_087.tif
e383e6947d9fe3259a9c4b9ad180090a
db7872999e08e3c67a24b9b57edf2086cb8232f7
F20101206_AAARRS baldwin_j_Page_069.tif
6f629e4295e98bf19cb56d88b74d84a3
9922833ad2e6b83bd1da4467452922a56a24a3cb
F20101206_AAARSI baldwin_j_Page_088.tif
bd5ce603fcbf97aeaa624421eedc1e5a
9fbe4f0f308f9ac50013c4dcfa9e5f2014d17a0d
F20101206_AAARRT baldwin_j_Page_070.tif
ec26b7703099d484f46686ed171782b4
d2ee43ebff57f346e417923a59c21300c0349519
F20101206_AAARSJ baldwin_j_Page_089.tif
a58c31212f326531310f39320dc11804
2d56fff93395a1f9004a8d7f0775547dabc88112
F20101206_AAARRU baldwin_j_Page_071.tif
8af02fa4215ac9f3d6bae47e59d68471
48432578280f5484f13b32da4892868aa50b717d
F20101206_AAARSK baldwin_j_Page_090.tif
4a85645f0375c2f8a950020577c805ca
27a8e52cf61179e73299c17e7bfab91827d276bf
F20101206_AAARRV baldwin_j_Page_072.tif
e3344f5a9b447b6ff9a574d9809ca90c
95839537653e21c16de388b0f801ff6aa9a7345f
F20101206_AAARSL baldwin_j_Page_091.tif
8be5b57b9fce34b7b7291396795b933c
e197904cae462fd3fa9b95464447397e1ee1f93f
F20101206_AAARRW baldwin_j_Page_073.tif
09a356a828551b1c6d5e0645e1b5e0b3
29d452eb9b8dbc11ee7988656450cdefb8d8d6eb
F20101206_AAARTA baldwin_j_Page_110.tif
04c4e7d074e1c9218287d1d3574af777
d0b51619a2b59aa867fee0bf527a2d5bbd6e3327
F20101206_AAARRX baldwin_j_Page_075.tif
9f9cfee3a48bc0eeaa1ae8554d0e60fc
07ab3fa944dbe974c099de5e0c639f9c7ba6322f
F20101206_AAARTB baldwin_j_Page_111.tif
d27edd6e0ad3ee4944bb4c245de7eee7
63ff53bfb2c0337a33df565dd6e9e93cf5931d9d
F20101206_AAARSM baldwin_j_Page_092.tif
a69aa5621ee0df614ec594b3836a74ef
cb85e191d7055c1c99401803d8d5dec6e55aaedd
F20101206_AAARRY baldwin_j_Page_076.tif
2f7b265203d6fb87bfce7a9cdc17f992
f2836e55489f8277fb3be30ee7e67af0749ac22c
F20101206_AAARTC baldwin_j_Page_112.tif
e6bc963d93b65e6df78045ebefa142ae
71353fc817b0edce138f13e51ab322cf00540e95
F20101206_AAARSN baldwin_j_Page_094.tif
6f63a5e4ca38e4514ab63ca792dda0f4
863cb193baaeca87ae8edaa4ff390631eb795ad6
F20101206_AAARRZ baldwin_j_Page_077.tif
e32bc9bf4b087fa8c2e6efb68f77b969
561d6b396e7e3067761fa51bdb3e3db199b64408
F20101206_AAARSO baldwin_j_Page_095.tif
30596cfbda2b9d9671b337488f3a8996
f8254df7057ee8915dba333ee2ee402be69118ac
F20101206_AAARTD baldwin_j_Page_113.tif
b4622d62734be0a7b4dc88133e29f12e
ae5d731c436e53be2921ae588e48b6ee3fd41d18
F20101206_AAARSP baldwin_j_Page_096.tif
e21e96b9e4fc05f13bdb40139f3c3b1a
69309215da7b3c13f7a673113a925a148ab41cb5
F20101206_AAARTE baldwin_j_Page_114.tif
a21187b1192fa804b8b7a52173ba95bf
05d8f45f8d07938a333b7c54c297297c4a7711ef
F20101206_AAARSQ baldwin_j_Page_097.tif
6222897428146d63f5dbc4656d41398f
b1b52adede0e63b461aba243d966dfad8e27be5c
F20101206_AAARTF baldwin_j_Page_115.tif
809503a0b35f2b41772582e167b7bfdc
4bb921339c3e254f63f7b2ac20e11d280368a934
F20101206_AAARSR baldwin_j_Page_098.tif
1267d9abccdc276b3414c6446755b2db
3fff79e295d570fced4c6b96f7d1c3cbbcd785c5
F20101206_AAARTG baldwin_j_Page_116.tif
96362725067be1b7e257f905b90cde69
a4fa13dbcdf8e0682edb3ac755cedef4c7f5a326
F20101206_AAARSS baldwin_j_Page_100.tif
748626160b49bd5a3a4294bc31f5137a
9bf78013878ea923eecf0c639093d69a18498e47
F20101206_AAARTH baldwin_j_Page_117.tif
fa2b237fb2219d37cadfcfa00228f6ac
45eadcbb7cc3629762bfd13a1209653d5e4fc9a9
F20101206_AAARST baldwin_j_Page_101.tif
04e507b64af335a351d0fc3cd4c4bb27
233e1f8c66282ba6a0748c3324cd93c5b6e5bf9d
F20101206_AAARTI baldwin_j_Page_120.tif
5861780dd660d733d99cbd42f858da07
6d84f6771574e8b84f61b32caa0bdf354d73a64a
F20101206_AAARSU baldwin_j_Page_102.tif
da706d1ec75635d49df084cd0e78ce35
45180827f2aca0e42dbf0eedefef6a0b05af3566
F20101206_AAARTJ baldwin_j_Page_122.tif
ab52ef60263c5cda85e64d7fa197507d
df30ebfd9e9314994241ce00acd95d528e04bc98
F20101206_AAARSV baldwin_j_Page_103.tif
56698478eb09e5419dbc148cf8f0b669
f0969279c9cbcf4cf5a84a43382d816011f2ac29
F20101206_AAARTK baldwin_j_Page_123.tif
e93d888ab4767ddd19a6ae3916a08ab2
5478ba5dfc4c4c5151cd9f3c3245ae158fde3f1d
F20101206_AAARSW baldwin_j_Page_104.tif
81bf386152b2d91fa1cd9763159cd836
9dc742bc864d7ddbe3c746e193fbc997b57db8cc
F20101206_AAARTL baldwin_j_Page_124.tif
502e8185032e95736493769b09045ea9
bd66e4245eeb7ca31c8b078ccdd5a06048bafd68
F20101206_AAARSX baldwin_j_Page_105.tif
e17caaca5395fe9526e75361729272b5
c987676abd6ac3d564ab760eaaff6ea42d4baaab
F20101206_AAARUA baldwin_j_Page_145.tif
23d5f88643e0ba9d709e2d4f4c300531
620d5721d7959e3eeb4a78cb8d4174f0226e3e60
F20101206_AAARTM baldwin_j_Page_127.tif
d4b3b06ad1ef298df76627beaa102df9
e77c0890c5642579b1098c874651fdb1310cdf56
F20101206_AAARSY baldwin_j_Page_106.tif
7fd8f40af98c0002077c0c941100c88f
13de4428c105cd7761a7362ae2c548162af7e1a7
F20101206_AAARUB baldwin_j_Page_146.tif
eac69273d52a674a6a4297773570511f
486b6dad2ac63f9c80cf3afea90386ceb1402d35
F20101206_AAARSZ baldwin_j_Page_108.tif
13b3c435d385bb63c8b37252cc643cf0
1364b43d5d655f2b87eb89b18bf71da7d1b01c78
F20101206_AAARUC baldwin_j_Page_147.tif
0a0f0b2082c5fbfcd36976d5319b1384
7f74b4b0cc64d056a443e5b2461764b1c82f47f4
F20101206_AAARTN baldwin_j_Page_128.tif
07a6d3f5a27f43e41937e5d0b7cd66eb
558489eead4fe90d0379ba55850dfcfcc4935d17
F20101206_AAARUD baldwin_j_Page_148.tif
499492a6c507040423cd20b887c648ff
c2187524b99a0918c443552dbe6589e0a9d06f05
F20101206_AAARTO baldwin_j_Page_129.tif
2f0d02cc4e60271e0121898ab6fb6017
04a3b077de754b13d76d5b1d3eb08fce7f0f85cb
F20101206_AAARUE baldwin_j_Page_149.tif
b8383e07c9eb7924e15de1e9cfce270c
4151bdb140168089ce266804613091a8dd95505a
F20101206_AAARTP baldwin_j_Page_130.tif
ea0dd9efe5c7a363e973f91d5345c489
590cdcb2e3fc49317ef5941a17daccd468a27e3f
F20101206_AAARUF baldwin_j_Page_151.tif
458078e57b8ebc325b7dc32c2ae2b7fc
dcb3cdf7e38f173c97cad6929d4c8460770739f8
F20101206_AAARTQ baldwin_j_Page_131.tif
570c17db0a3e781ab9a054284cd2107e
077dcddb950062f1b2d4a8da4727e508d9b0dc6c
F20101206_AAARUG baldwin_j_Page_152.tif
7d2baba87b4b26d6f457464ae5732eb3
db7c087fae5c47ab6e3c7a38e0b209b35d26e5fa
F20101206_AAARTR baldwin_j_Page_132.tif
30e64e7dd3314223bf84fd104c427083
032ef90ddbe64c4dc02962716a02d72268f897a3
1898 F20101206_AAASAA baldwin_j_Page_018.txt
89b4c191a7f6d97131939984c191f941
498b6d5c68bd72f9ee26dd5d1fe50955203f1bf6
F20101206_AAARUH baldwin_j_Page_154.tif
8de58eb10da3a25697bad5fa8c764655
6ebcdb1cb7817b5d2011ac4ea8055a77a4d9145b
F20101206_AAARTS baldwin_j_Page_133.tif
ea1c2d275fbbeaeaac4bb08767a4998b
79706e12ec88e73b12c95040dd376ac75d338431
1523 F20101206_AAASAB baldwin_j_Page_019.txt
db9f8ee495c16da3127d5f156c575db3
effca14b788fea8e975fa2f37a467a4630ed23b8
F20101206_AAARUI baldwin_j_Page_155.tif
ff1e4619eb57ff54505b8a26063b828e
4784ad38b5a23ca5624b6577169e8b0f13daceaa
F20101206_AAARTT baldwin_j_Page_134.tif
07154af899bc90cb73164af5155deac0
80943caa782deea4efa97ebb613397242faf5de1
1827 F20101206_AAASAC baldwin_j_Page_020.txt
8a350af2e74431322f08db1a1a29018f
999585a0e04fa699ccb4d7b9e3cf08a93539d732
F20101206_AAARUJ baldwin_j_Page_156.tif
7abf98f759d0526060177dd281d2a751
224cd3517affa555449f1ad74b70cc34573ebaf2
F20101206_AAARTU baldwin_j_Page_135.tif
904ba360590b26e9e788eca6f7e158df
f00343d3b1f4d35522e3e1f84d0e89779f82be7b
1852 F20101206_AAASAD baldwin_j_Page_022.txt
1c3f045307a36c9aeb6a5f317028d037
902c5ed53ff94289a2c351f6e70875b0b2fd9e18
F20101206_AAARUK baldwin_j_Page_157.tif
e95bf82833d5960735bbb8a95e3b56e9
8618366f97e3a4743b3122f06c40183d5b7734ec
F20101206_AAARTV baldwin_j_Page_136.tif
7a118e038b0de52da3e8e8a275a88271
91cbfc931b29661e9f8f0e5a67e22ea2b0d75c4f
1569 F20101206_AAASAE baldwin_j_Page_023.txt
4ab160e87469c1a6fc707c3ba155d4d0
4d7fd4430aa4fbc2f30b81f6cccf2cb3fbdcc3cf
F20101206_AAARUL baldwin_j_Page_158.tif
6a7b7b201d568631536ab6e576b4dc37
e15acfa0c07a03e7a13f8f84782571209fca6eb2
F20101206_AAARTW baldwin_j_Page_138.tif
9f7777e4e6283bc863a4791cb6b6c280
ff386273c5d0fb637c4082243bfcd9b97999e298
1680 F20101206_AAASAF baldwin_j_Page_024.txt
0b6e5a6e7ec8c1154e1bde05915684fe
738253f21ceabc47602d74ad19b01c44ef95aa90
48061 F20101206_AAARVA baldwin_j_Page_015.pro
ecfaefc2d992473e91fd3d8a965a8109
f21273dc3a77b4dbc01f835432a2a6241137509b
F20101206_AAARUM baldwin_j_Page_159.tif
26770c39843ebce7e9355eb98d0e0edb
59fb0badbb1df6f3100ed0c8f183db7478154a5d
F20101206_AAARTX baldwin_j_Page_140.tif
6d078bdd20289f57e62b6341e278fb3b
47c3368b91429a810a8830660a11a79cf9e532a1
F20101206_AAASAG baldwin_j_Page_025.txt
da18c6afcc44d0b926f3ce9248560744
79b1098ea80190e2b0bf9b65a3759b72f70d884d
42288 F20101206_AAARVB baldwin_j_Page_016.pro
a0de6f4cc27f80d29de80bada9b7d824
1bf1172006e4460b1e330c1a519510c33b5e63a0
F20101206_AAARUN baldwin_j_Page_160.tif
acf3aa5c4c2ca47b5a308404ff20edbd
0aea99c991c29d968bfccbf6b3c517bb2d3d0cff
F20101206_AAARTY baldwin_j_Page_142.tif
c2b0dc6f161196f4ecfdaeec6028a5c9
efb3893dd9fe7ca8ff2614af02a64aa64cb4faeb
1499 F20101206_AAASAH baldwin_j_Page_026.txt
e52a8dd43ab146bc4f651f4c1482c149
34850b693c7c4c72dfce05ef1b73d22f53150319
47998 F20101206_AAARVC baldwin_j_Page_018.pro
8a3b2315331ae7832dbf6821acfda90c
ee0df4c245d69e53159b431810f649df60fa597f
F20101206_AAARTZ baldwin_j_Page_144.tif
dead99b7d450119048b0b6c7426833c4
601da4a53ed5b28d5518db1cc60dea7379d1741f
1780 F20101206_AAASAI baldwin_j_Page_027.txt
a9ea11d68e615d500ab22fdc9469f500
f69c3ed4539067360f75fa5d39afd062997027e1
32934 F20101206_AAARVD baldwin_j_Page_019.pro
793adc5f709f2b65ce10a29a3b7332ea
554a0fb1cd0653ee99eac8d3de405fd09c946a1b
F20101206_AAARUO baldwin_j_Page_161.tif
ded4533cc3130cc9b5c811fd4e683b92
34fb2d66799d7b92a1ee541b9649af6a1c0a8bdb
1538 F20101206_AAASAJ baldwin_j_Page_028.txt
3fa70f0fdc86372f443df957f6c1d3cc
275ef56e200e047908a4420c620718d4d4ef7557
37216 F20101206_AAARVE baldwin_j_Page_021.pro
ffe0917dbc5b58960c057650891f2e76
0de8ac6f21f4973a267e6c31e06b103aaabda571
F20101206_AAARUP baldwin_j_Page_162.tif
be963dcd4cb2e13b3dc3638718889d42
292cab3af83ceac9d1ebd2407c6eb1a5190e16b7
1811 F20101206_AAASAK baldwin_j_Page_029.txt
0614c1ec25e13e3b2883681aa5d23632
229685c8f79c43364cb2cb458b93cd3af101fa65
41121 F20101206_AAARVF baldwin_j_Page_024.pro
db10cc59255df6563e7db3749fbe9003
a51b25829d3a59a82ea3f1e3192fb6c44ca94711
9140 F20101206_AAARUQ baldwin_j_Page_001.pro
ca0c7f44c75ec973729638ed1acb310b
06f4cbe38602c992945e81fa803d4f52e468b148
1561 F20101206_AAASAL baldwin_j_Page_030.txt
dd288a469949b3218a03f079645332bd
11c926204d40642d4a68ca3ce6a028e51e3c3bf4
896 F20101206_AAASBA baldwin_j_Page_048.txt
f7c7a29d08ad2891925c4ea4e4780877
dbcba3eb8b6555d4ab49eda9902041bdf1709178
33046 F20101206_AAARVG baldwin_j_Page_025.pro
43361e44ed50e87ac1b822f082c9732e
7cc5b52c01f78f9d821d6abeda5ee02eb290c933
2238 F20101206_AAARUR baldwin_j_Page_003.pro
5f5cf150cabbad3b152808f917a7f07f
39f0d6394dead266f285af3ced9ce3a594c2e969
1123 F20101206_AAASAM baldwin_j_Page_031.txt
3a92c2bcc4a18942d8ca4c11af41a688
f4d0b277cc1eaa242c46565ba26655f9b1a94180
833 F20101206_AAASBB baldwin_j_Page_050.txt
0aa2f0abe4377cbb4ec0436daad7c29b
2e56112a7da7d1905e538a9b9985d0d7b07fc261
34639 F20101206_AAARVH baldwin_j_Page_026.pro
dbb3275ed4490bdbd0007533b15031d7
f230af84f6a3e88a764e565add8a989b58d37e18
54759 F20101206_AAARUS baldwin_j_Page_004.pro
6b371fe3c69bb6738c3b090789c5f33e
f308149e03d5a6ca43c03b39c6fd29f15c41c320
1626 F20101206_AAASAN baldwin_j_Page_032.txt
1268e6f0190e1e7c39aa7da7f24673bd
a17814bd52da23feaaf4e1bfd4f199f96b25e134
1845 F20101206_AAASBC baldwin_j_Page_051.txt
82af01c93219cf46b21f7fbb14afdf28
af9c3242ef431c8d61a2939683a0eb2746d396b6
1253 F20101206_AAASAO baldwin_j_Page_033.txt
2e10b159e722f6e08cc8854fd35f527c
4c000a2fb6616214131abe569fb3504ab0bda5dc
38593 F20101206_AAARVI baldwin_j_Page_027.pro
ebc8f0ee97dab42dd69d1722f383145c
af0c9673ddd33fbd1281c0844ebc87b3d40881e8
51364 F20101206_AAARUT baldwin_j_Page_005.pro
ad133d361a5e04ab17ea4e84dbee760c
d661bb8ed2c104a495663b8c3eb849886cc54b23
847 F20101206_AAASBD baldwin_j_Page_053.txt
6270daf01b78b1b6ef5795cfa5a2198a
4d3fded8c79401d4529ab7371e667528b7cab033
1824 F20101206_AAASAP baldwin_j_Page_035.txt
8d9b28ee75337d9c03294ba625c6eddb
e139ce220de8dc08ac539a5e158a18d30c1d8467
29223 F20101206_AAARVJ baldwin_j_Page_028.pro
0bb111974a32585f45830c7885f83eae
e27d3f1d383a54928fed57c8a08c83cde882d05f
66567 F20101206_AAARUU baldwin_j_Page_007.pro
85b517d0d4733b0f4a994019c23c3a3e
ead7eb6f7f0af052526fad584bf05347567ad56d
872 F20101206_AAASBE baldwin_j_Page_054.txt
62da07a3ee979ebd3ef5713e2c03636c
a8638ba15e6b1fbed68f931d7fbb4a6456782bbb
1373 F20101206_AAASAQ baldwin_j_Page_036.txt
7e8e6f94d8cf8ea1e3ce155d4f0dc829
d8f4a0bcc1b2d7f3bff2eceef4868f61565d3224
45663 F20101206_AAARVK baldwin_j_Page_029.pro
1f8861dbaa9f2b9990a87edc68273dba
115135e3609e9054c66bcf5693cd73df0971037d
51629 F20101206_AAARUV baldwin_j_Page_008.pro
bae49ecd27ad2e164ab36dd3df4aa94e
09904f9162027eee7953ea87edc63d08df7e624c
1437 F20101206_AAASBF baldwin_j_Page_055.txt
71a67c0b1c392959afa260908e75f708
44f19c77f60d99ae077746ec60d138bd2670119c
1832 F20101206_AAASAR baldwin_j_Page_037.txt
0af807fea853e7a70c338341c5c0fc0c
7f19f01e27a41180f46cfa25835752ba526de27d
27985 F20101206_AAARVL baldwin_j_Page_032.pro
d912b7a878661dbf0f82ee31693a11de
d6a3a2c393076a393bd3987263eba12ef72de940
54557 F20101206_AAARUW baldwin_j_Page_010.pro
2b61295f29a2617202615899f57b4c47
6ed592c9ad931cdeb25047a2578cb350e3ae796a