Citation
Coaching in Parent-Child Interaction Therapy Development and Reliability of the Therapist-As-Coach Coding System (TACCS)

Material Information

Title:
Coaching in Parent-Child Interaction Therapy Development and Reliability of the Therapist-As-Coach Coding System (TACCS)
Creator:
O'BRIEN, KELLY ANN ( Author, Primary )
Copyright Date:
2008

Subjects

Subjects / Keywords:
Child psychology ( jstor )
Coaching ( jstor )
Estimate reliability ( jstor )
Gene therapy ( jstor )
Manuals ( jstor )
Medical treatment outcomes ( jstor )
Parents ( jstor )
Patient compliance ( jstor )
Social interaction ( jstor )
Verbalization ( jstor )

Record Information

Source Institution:
University of Florida
Holding Location:
University of Florida
Rights Management:
Copyright Kelly Ann O'Brien. Permission granted to the University of Florida to digitize, archive and distribute this item for non-profit research and educational purposes. Any reuse of this item in excess of fair use or other copyright exemptions requires permission of the copyright holder.
Embargo Date:
4/30/2009
Resource Identifier:
73722827 ( OCLC )

Downloads

This item is only available as the following downloads:


Full Text

PAGE 1

COACHING IN PARENT-CHILD INTERACTION THERAPY DEVELOPMENT AND RELIABILITY OF THE THERAPIST-AS-COACH CODING SYSTEM (TACCS) By KELLY ANN O’BRIEN A THESIS PRESENTED TO THE GRADUATE SCHOOL OF THE UNIVERSITY OF FLORI DA IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF SCIENCE UNIVERSITY OF FLORIDA 2004

PAGE 2

Copyright 2004 by Kelly A. O’Brien

PAGE 3

This document is dedicated to my mother, for her support and strength.

PAGE 4

ACKNOWLEDGMENTS I would like to thank Dr. Eyberg for her guidance, enthusiasm, and understanding through the ongoing process of this research. I would also like to thank Dr. Boggs for his guidance and support. iv

PAGE 5

TABLE OF CONTENTS page ACKNOWLEDGMENTS.................................................................................................iv LIST OF TABLES.............................................................................................................vi ABSTRACT......................................................................................................................vii INTRODUCTION...............................................................................................................1 Development of the Therapist-As-Coach Coding System............................................3 Rater Training...............................................................................................................5 METHODS..........................................................................................................................7 Participants...................................................................................................................7 Procedure......................................................................................................................8 Reliability.....................................................................................................................8 RESULTS..........................................................................................................................11 The Reliability of the TACCS Therapist Verbalization Categories...........................11 The Reliability of the Parent Compliance Categories................................................12 DISCUSSION....................................................................................................................17 Reliability and Frequency of Occurrence of the TACCS Categories.........................18 Refinement of the TACCS categories........................................................................19 Limitations..................................................................................................................20 Future Research..........................................................................................................21 APPENDIX: SUMMARY OF TACCS CATEGORIES..................................................23 LIST OF REFERENCES...................................................................................................25 BIOGRAPHICAL SKETCH.............................................................................................27 v

PAGE 6

LIST OF TABLES Table page 1. Kappa Coefficients and Mean Percent Agreement for the TACCS Therapist Verbalization Categories Across Segments.............................................................14 2. Kappa Coefficients and Mean Percent Agreement for the TACCS Parent Compliance Categories Across Segments....................................................................................15 3. Frequency of TACCS Therapist Verbalization Categories Across Segments..............15 4 Kappa Confusion Matrix Across All Therapist Verbalization Categories....................16 5. Kappa Confusion Matrix Across All Parent Compliance Categories...........................16 vi

PAGE 7

Abstract of Thesis Presented to the Graduate School of the University of Florida in Partial Fulfillment of the Requirements for the Degree of Master of Science COACHING IN PARENT-CHILD INTERACTION THERAPY (PCIT) DEVELOPMENT AND RELIABILITY OF THE THERAPIST-AS-COACH CODING SYSTEM (TACCS) By Kelly A. O’Brien May 2004 Chair: Sheila Eyberg Major Department: Clinical and Health Psychology Coaching parent behavior during interactions with their child is a core component of Parent-Child Interaction Therapy (PCIT) and one of the primary interpersonal processes facilitating change in the parent-child relationship during treatment. Although skillful coaching is assumed to mediate treatment outcome, there has been no empirical examination of this assumption, in part due to lack of a method for measuring this aspect of treatment. In this study, the Therapist-As-Coach Coding System (TACCS) was developed to measure therapist verbalizations and parent compliance during the PCIT coaching interactions. Seventeen therapist verbalization categories and three parent compliance categories constitute the TACCS. Two undergraduate research assistants were trained to code the therapist and parent behaviors from videotaped coaching interactions during both early and late phases of treatment. Percent agreement was calculated throughout the coding trials and used to refine the categories and definitions of vii

PAGE 8

the coding system. In the pilot testing of 21 coaching sessions coded with the current 20 TACCS categories, 13 categories were coded with “good” to “excellent” reliability and five demonstrated “fair” reliability. The TACCS categories with the lowest reliability estimates occurred with the lowest frequency in the sessions coded. We discuss implications for research on process variables during PCIT coaching that may predict treatment outcome. The TACCS will also contribute to the neglected research area of therapist training and skill acquisition. viii

PAGE 9

INTRODUCTION Parent-Child Interaction Therapy (PCIT) is an empirically supported treatment for young children with behavior problems (Brestan & Eyberg, 1998). Outcome studies have established the efficacy of PCIT for improved child behavior and parent interactional style (Eisenstadt, Eyberg, McNeil, Newcomb, & Funderburk, 1993; Eyberg & Robinson, 1982; Schuhmann, Foote, Eyberg, Boggs & Algina, 1998), with more recent literature demonstrating maintenance of treatment gains up to six years (Hood & Eyberg, 2003). Generalization of treatment effects has also been shown in school behavior (McNeil et al., 1991) and to untreated siblings (Funderburk, Eyberg, Newcomb, McNeil, Hembree-Kigin, & Capage, 1998). PCIT literature to date focuses on treatment outcome and maintenance. However, if the efficacy of the treatment and the mechanisms through which change occurs are to be better understood, specific components of the therapy need to be further examined. Although the importance of the therapist-client relationship is well documented in the individual therapy literature, it remains a neglected area of research in parent-child therapy research (Borrego & Urquiza, 1998). Addressing this lack of process research, a recent PCIT study showed that therapist verbalizations to the parent in the first clinical interview was able to predict treatment outcome (Harwood & Eyberg, 2003). More specifically, Harwood and Eyberg showed that a high rate of supportive statements and close-ended questions by the therapist in the initial interview, along with a low rate of therapist facilitation, predicted 1

PAGE 10

2 treatment drop-out. The findings of that study suggest the merit of further examination of the therapist-parent interaction. The direct conversation between the therapist and parent, measured by Harwood and Eyberg, involves a minimal amount of therapy time in PCIT, with most of the therapy session consisting of in-vivo coaching of parent skills. Coaching is generally assumed to be important in the acquisition and generalization of parent skills and is assumed to moderate treatment outcome, but there has been no empirical examination of this assumption, in part due to the lack of a reliable method for measuring this aspect of treatment. The Coach Coding System (CCS) is an existing coding system designed for use in live coding situations of PCIT coaching (Timmer & Urquiza, 2002). The reliability and validity of this system have not been reported yet, and the operational definitions and examples provided for the categories are not fully developed. In addition, the system was developed only for live coding thus, limiting the number of behaviors able to be studied due to the demands on the rater during live coding. Coaching is the therapist’s verbal communication to the parent, providing direct, immediate feedback about the parent’s use of skills taught in treatment. Live coaching in PCIT is usually done from an observation room via a “bug-in-the-ear” device and one-way mirror. This procedure allows the parent to be alone in a playroom with the child while receiving immediate feedback from the therapist. Parent mistakes are corrected in the moment, minimizing inaccurate or inconsistent use of skills. In the first phase of treatment, Child-Directed Interaction (CDI), parents are coached in the use of play therapy skills to improve the parent-child attachment of parent. These skills include differential attention and the PRIDE skills (giving the child labeled praises, reflecting

PAGE 11

3 appropriate child verbalizations, imitating their play, and describing their appropriate behavior, all with enthusiasm), as well as avoiding commands, questions and criticisms. In the second phase of treatment, Parent-Directed Interaction (PDI), parents are coached to use a specific procedure that emphasizes consistency and predictability with simple, direct commands and follow-through. Treatment continues until parents meet established mastery criteria for the skills in each phase. Until graduation from treatment, a portion of each session is spent in live coaching of the skills, with the exception of the two didactic “Teach” sessions that occur at the beginning of each phase. The length of time spent coaching in each session often varies across families and sessions, however it is recommended that twenty minutes be spent coaching for a one-parent family and ten to fifteen minutes for each parent in a two-parent family. Coaching verbalizations are brief, quick and mostly positive statements to guide the parent in the play situation in the use of the play therapy and discipline skills. Gentle corrections are given to highlight parent mistakes when needed. Therapists must be skilled in the timing and delivery of feedback to the parent, who is involved in an ongoing exchange with the child. This means staying attentive to each family’s individual interaction style in each session while working towards the treatment goals of skill mastery and improved parent-child interaction. Development of the Therapist-As-Coach Coding System The Therapist-As-Coach Coding System (TACCS) is a behavioral coding system for therapist verbalizations and parent compliance during the coaching component of PCIT. The development of the TACCS began with the identification of specific therapist verbalizations and parent behaviors during coaching that are thought to facilitate and impact the process and outcome of treatment.

PAGE 12

4 Categories needed to be adequate for all therapist-parent interactions throughout treatment. More specifically, the categories should adequately capture each individual therapist’s skill and style, each dyad’s interaction, and at each time-point of therapy. A goal in creating the TACCS was that all therapist verbalizations could be coded exhaustively, to obtain the most complete picture of the coaching component of therapy. Several resources contributed to the generation of the initial TACCS categories, including the sections of the PCIT manual pertaining to coaching, the existing coding system for live coaching situations (the CCS), PCIT training materials, and therapist input and feedback. In addition, numerous videotapes were examined to assess the adequacy of the categories in capturing the therapist-parent interaction as categories were generated. Early rater performance and feedback was immensely valuable in the early development of the coding system; rater feedback continues to be important in the refinement of the coding system. During coaching, therapists provide a range of feedback to parents that could be described broadly by behaviors such as praising, directing, teaching, correcting, and supporting. The existing 17 therapist verbalization categories have been operationally defined and exemplified to capture these skills that facilitate treatment progress (see Appendix A for a summary of TACCS categories and definitions). For example, all therapist praises serve to reinforce parent behavior, however with the TACCS categories praise could be in the form of an “Unlabeled Praise”, a “PCIT Praise”, a “Contingent Praise”, or an “Advanced Praise”. By examining the specific types of praise a therapist may or may not use, their clinical utility can be examined empirically. For example, the therapist can give a short, nonspecific praise like “Good job,” or they can praise a

PAGE 13

5 specific skill just used by the parent like “Excellent reflection.” Similarly, a therapist can direct parent behavior with varying techniques, categorized in the TACCS as “Direct Command,” “Indirect Command”, and “Direct Leading” statements. A Direct Leading statement provides the parent with words to repeat verbatim to the child, while a Direct Command indicates a specific action to be performed by the parent. Again, the utility of these directives may vary across coaching situations and remains an empirical question. After categories were operationally defined, specific examples were (and continue to be) generated to help define and differentiate categories for raters. The Dyadic Parent Child Interaction Coding System-II, (DPICS-II; Eyberg, Bessmer, Newcomb, Edwards, & Robinson, 1994) which is used to code the parent-child interaction in PCIT before and after treatment, has been influential in the development of the TACCS manual. The TACCS manual follows the DPICS-II format of definitions followed by specific guidelines and examples. Rater Training To begin, raters read PCIT literature relating to treatment, and particularly coaching, to inform them of the nature and procedures of therapy. They were also given the information parents receive in the didactic CDI and PDI teach sessions to give them a clear understanding of the goals of coaching. After reading the TACCS manual raters completed several worksheets to give them practice distinguishing categories, which is especially important for two similar categories. For example, there is sometimes a subtle difference between different types of praises or teaching. In addition, sometimes a verbalization can fall into two categories. For example, a verbalization can be both a Direct Command and Corrective Feedback,

PAGE 14

6 such as “Make sure you give the warning.” It is important for raters to learn specific rules to apply in these situations to improve rater accuracy. Transcripts of sample coaching situations and a criterion tape were done until raters reached 90% criterion. Weekly meetings were held to discuss inter-rater reliability and coding discrepancies and to answer any questions that had come up during the previous week while coding.

PAGE 15

METHODS Participants Twenty-one sessions comprised of 18 parent-child dyads (11 mother-child and seven father-child dyads) and 13 therapists were observed in the present study. Coaching sessions ranged from 3 minutes, 0 seconds to 17 minutes, 28 seconds, with a mean length of 9 minutes, 38 seconds. Nine of the sessions coded were in the CDI phase of treatment (ranging from CDI Coach One to CDI Coach Five) and 12 of the sessions were in the PDI phase (ranging from PDI Coach One to PDI Coach Eight). A sample representing a variety of parent-child dyads, therapists, and phases of treatment were included to ensure that the coding system could be applied adequately across these variables. The families were participating in a treatment outcome and two-year maintenance study of PCIT. All children were between the ages of three to seven at the start of treatment and met criteria for Oppositional Defiant Disorder (ODD) based on their parent’s responses to the DISC, a structured interview based on DSM-IV criteria. Each session was routinely videotaped and parents’ informed consent indicated that all information collected would be used for research purposes. All videotapes were kept in a locked room and each family was only identified by a number. Two undergraduate research assistants, a 19-year-old Caucasian male and an 18-year-old Caucasian female, served as the primary observers and received research credit for their work. Raters were not informed of any demographic information, treatment status (e.g., treatment completer or drop-out), or other identifying information of the 7

PAGE 16

8 families. Raters were also unaware of therapist experience (e.g., lead therapist or co-therapist). All therapists were enrolled in a clinical psychology doctorate program. Each family was assigned a lead therapist and a co-therapist. Lead therapists had been co-therapists for at least two cases prior to being assigned as lead therapist. Lead therapists and co-therapists both attended weekly group supervisions with two principal licensed clinical supervisors. Eight lead therapists and five co-therapists coached in the 21 sessions coded. Procedure Videotapes of sessions were observed by three undergraduate research assistants and the author to determine if coaching verbalizations were captured by the audio system. If coaching was audible, the beginning and ending times of each coaching segment were documented for each parent. It was also noted if there was a therapist change mid-session or between parents. The author then watched the segments to identify the therapists. Because reliability calculations have been used in the refinement of the coding system to date, each segment was coded by both primary observers in the present study. Reliability Cohen’s kappa (1960) was computed to determine initial reliability estimates of the 17 therapist verbalization and 3 parent compliance categories of the TACCS. Calculating the kappa statistic when investigating reliability has several advantages to other reliability estimates such as Pearson product-moment correlations and percent agreement. For example, kappa takes into account agreement for each unit of behavior coded by both raters, while Pearson correlations are unable to provide information about agreement on specific occurrences, which may result in inflation of reliability estimates.

PAGE 17

9 In addition, the kappa statistic controls for chance agreement between raters, while both Pearson correlations and percent agreement do not. A third advantage of calculating kappa is that all codes by both raters are considered instead of only comparing a second rater to a primary rater, which can be affected by the individual raters patterns of coding. Thus, kappa is based both on behaviors coded by both raters and on those behaviors coded by only one rater. Again, because of the use of coding performance in the development and refinement of the TACCS, the 21 coaching segments included in the present study were coded by both observers. Percent agreement was calculated regularly during training and coding so that discrepancies could be addressed in meetings and clarified in the manual when needed. The kappa statistics were computed using a computer program developed specifically for the DPICS-II (Eyberg & Celebi, 1993), the existing behavioral coding system designed for the assessment of the quality of parent-child social interactions in PCIT. This program calculated kappa by comparing a series of film pairs coded by the independent observers using a one-second window. Just as the reliability of the child and parent behaviors for the DPICS-II are calculated separately, the therapist and parent behaviors were examined independently. Analyzing therapist and parent behaviors separately controls for inflation of the kappa statistic by comparing categories that are likely to be confused with one another and not comparing non-related categories. For example, therapist Direct Commands could be confused with therapist Direct Leading, but not with parent Compliance. The Fleiss (1981) system of classification of kappa values is widely used in the literature and was used to evaluate the kappa statistics in this study. He indicated that

PAGE 18

10 kappa values greater than .75 can be considered excellent agreement beyond chance and kappa values from .60 to .75 indicate good agreement beyond chance, while values from .40 to .60 can be considered fair agreement and values that fall below .40 can be considered poor agreement.

PAGE 19

RESULTS The Reliability of the TACCS Therapist Verbalization Categories Kappa estimates (as well as percent agreement) for the individual therapist verbalization categories summed across the 21 coaching segments coded by both raters are shown in Table 1, with kappa coefficients ranging from .25 to .79. Categories are presented using Fleiss’ (1981) convention of interpreting kappa statistics as either being “excellent,” “good,” or “fair” reliability estimates, with the highest kappa estimates appearing first. Most of the therapist verbalization categories were coded with greater agreement than would be expected by chance, with the exception of Teaching Statements and Advanced Teaching statements. Using Fleiss’ classification, the therapist verbalization categories with excellent agreement beyond chance include Indirect Leading, Positive Observation, Direct Leading, and Corrective Feedback. Categories considered to have good agreement were PCIT Praise, Information Statement, Supportive Statement, Contingent Praise, Acknowledgment, Direct Command, Humor and Unlabeled Praise. The Indirect Command, PCIT Teaching, and Advanced Praise categories would be considered to have fair agreement. The Teaching Statement and Advanced Teaching categories would be considered to have poor agreement. Percent agreement was calculated each week to aid refinement of the coding system. When percent agreement was calculated for each category across the 21 sessions coded, most often this reliability estimate was consistent with the kappa coefficients 11

PAGE 20

12 calculated. However, the Indirect Command and Positive Observation categories had markedly lower percent agreement overall in comparison to the kappa estimates calculated for these categories. seconds. The Reliability of the Parent Compliance Categories The parent compliance categories were also examined for reliability using the kappa statistic and are shown in Table 2. Kappa estimates for the parent compliance categories ranged from .40 to .70. As before, the categories are organized using Fleiss’ categorization of “excellent,” “good,” or “fair” reliability estimates. The Parent Compliance category was coded with good agreement beyond chance. The Parent No Opportunity to Comply and Parent Noncompliance categories would be considered to have fair agreement. Similar to the percent agreement calculations for several of the therapist verbalization categories, the low frequency of the Parent Noncompliance and Parent No Opportunity to Comply categories in quite a few of the sessions resulted in a restricted range of reliability estimates for these categories. The range of reliability estimates can also be better understood by examining the frequency with which the categories were coded across sessions for both raters (see Table 3). The third column in Table 3, labeled # of Sessions, lists the number of sessions in which the category was coded for at least one occurrence by one of the raters. Examination of these frequencies is an indication of the number of opportunities the raters had to agree or disagree for each category. For example, Advanced Teaching statements occurred in only three therapist-parent interactions and were coded only four

PAGE 21

13 times by each rater overall. This is in contrast to PCIT Praise, which was coded over 480 times by both raters and occurred in all of the 21 sessions. Confusion Matrices Generating confusion matrices when refining a coding system is extremely helpful for identifying those categories that are most often confused with one another in coding. Shown in Tables 4 and 5 are the complete Kappa confusion matrices for the therapist verbalization and parent compliance categories, respectively. Examination of the confusion matrices for the therapist verbalization categories showed the categories most often confused with one another to be (a) PCIT Praise and Contingent Praise, (b) Direct Leading and Direct Command, (c) Advanced Praise and PCIT Praise, (d) Direct Command and Corrective Feedback, (e) Direct Leading and Indirect Command, (f) Direct Command and PCIT Teaching, (g) Direct Leading and Indirect Leading, (h) PCIT Teaching and Corrective Feedback, (i) Direct Leading and Corrective Feedback, and (j) Indirect Leading and Positive Observation. The confusion matrix for the parent compliance categories showed that Parent Compliance and Parent No Opportunity to Comply were confused most often.

PAGE 22

14 Table 1. Kappa Coefficients and Mean Percent Agreement for the TACCS Therapist Verbalization Categories Across Segments CATEGORY k % agree Excellent Positive Observation .79 .60 Indirect Leading .78 .63 Corrective Feedback .77 .71 Direct Leading .76 .78 Good Acknowledgment .74 .64 PCIT Praise .72 .81 Direct Command .72 .76 Supportive Statement .71 .72 Humor .71 .75 Unlabeled Praise .63 .79 Information Statement .63 .69 Contingent Praise .61 .52 Fair PCIT Teaching .59 .51 Indirect Command .55 .40 Advanced Praise .54 .71 Poor Teaching Statement .28 .28 Advanced Teaching .25 .42 Note. Analyses based on N =21 coaching segments with a mean length of 9 minutes, 38

PAGE 23

15 Table 2. Kappa Coefficients and Mean Percent Agreement for the TACCS Parent Compliance Categories Across Segments Category k % agree Good Parent Compliance .70 .70 Fair Parent No Opportunity to Comply .42 .38 Parent Noncompliance .40 .36 Note. CO = Parent Compliance, NC = Parent Noncompliance, NOC = Parent No Opportunity to Comply Table 3. Frequency of TACCS Therapist Verbalization Categories Across Segments Category Rater 1 Freq Rater 2 Freq # of Sessions Direct Command 175 174 19 Indirect Command 36 33 33 Direct Leading 162 152 18 Indirect Leading 63 60 14 PCIT Praise 481 491 21 Advanced Praise 23 22 10 Contingent Praise 76 73 16 Unlabeled Praise 365 357 21 Teaching Statement 9 6 4 PCIT Teaching 59 63 18 Advanced Teaching 4 4 3 Corrective Feedback 71 71 15 Supportive Statement 10 12 6 Humor 12 15 10 Acknowledgment 128 124 20 Information Statement 49 50 19 Positive Observation 45 38 13 Note. Analyses based on N=21 coaching segments with a mean length of 9 minutes, 38 seconds.

PAGE 24

16 Table 4 Kappa Confusion Matrix Across All Therapist Verbalization Categories VAR AK PO IC DC HU AT PT TS IS IL DL CF CP SS PP AP UP AK 98 0 0 1 0 0 1 0 2 2 3 1 0 0 2 0 5 PO 0 35 0 1 0 0 0 0 3 4 0 0 0 0 2 1 3 IC 0 0 23 4 0 0 1 0 0 0 4 1 0 0 0 0 0 DC 0 1 6 140 0 1 7 1 7 0 8 6 0 1 2 0 4 HU 0 0 0 1 10 0 0 0 0 0 1 0 0 0 0 0 0 AT 0 0 0 0 0 1 1 1 0 0 0 0 0 0 0 1 0 PT 0 1 3 2 0 0 44 1 0 1 1 3 0 1 0 0 3 TS 0 0 0 1 1 0 2 2 0 0 0 0 0 0 0 0 1 IS 2 3 0 0 0 0 2 0 40 0 0 0 0 0 3 0 4 IL 5 0 0 1 1 0 1 0 1 54 4 0 0 0 0 0 0 DL 1 0 7 12 0 0 1 0 2 4 150 2 2 0 5 0 10 CF 1 0 1 8 0 0 5 0 0 0 5 60 0 2 1 0 3 CP 0 0 0 0 0 0 1 0 0 0 2 0 60 0 16 0 15 SS 0 0 1 2 0 0 1 0 0 0 0 1 0 10 1 0 2 PP 2 1 0 2 1 0 1 0 3 2 2 1 16 1 455 7 75 AP 0 0 0 0 0 0 0 0 0 0 0 0 0 0 5 17 5 UP 5 2 0 3 0 1 2 1 2 0 6 1 18 1 72 5 341 Note. AK = Acknowledgment, PO = Positive Observation, IC = Indirect Command, DC = Direct Command, HU = Humor, AT = Advanced Teaching, PT = PCIT Teaching, TS = Teaching Statement, IS = Information Statement, IL = Indirect Leading, DL = Direct Leading, CF = Corrective Feedback, CP = Contingent Praise, SS = Supportive Statement, PP = PCIT Praise, AP = Advanced Praise, UP = Unlabeled Praise Table 5. Kappa Confusion Matrix Across All Parent Compliance Categories Category CO NC NOC CO 223 16 27 NC 2 17 4 NOC 8 3 35 Note. CO = Parent Compliance, NC = Parent Noncompliance, NOC = Parent No Opportunity to Comply

PAGE 25

DISCUSSION The purpose of this study was to develop and establish initial reliability estimates for a coding system for measuring the therapist-parent interaction during the coaching component of PCIT. Reliability estimates for two raters coding 21 videotaped coaching segments provide evidence of the reliability of the majority of categories of the TACCS. Possible reasons for the variance in reliability estimates across the TACCS categories include not only the low frequency of occurrence of some of the behaviors in and across sessions, but also the developmental stage of the coding system. The TACCS categories were ranked into “excellent,” “good,” “fair,” or “poor” reliability groupings according to accepted standards for kappa coefficients (Fleiss, 1981). In addition, confusion matrices, the frequencies of codes for each category for each rater, and the number of sessions in which at least one occurrence of the behavior was coded, were examined to help identify the sources of coding error and the variability of reliability estimates. The categories found to have excellent reliability were Corrective Feedback and Direct Leading. The categories with excellent to good reliability were Positive Observation, Indirect Leading, and Unlabeled Praise. The categories found to have good reliability were PCIT Praise, Direct Command, Supportive Statement, Humor, Acknowledgment, Information Statement, and Parent Compliance. The categories with fair reliability estimates were PCIT Teaching, Indirect Command, Advanced Praise, 17

PAGE 26

18 Parent Noncompliance and Parent No Opportunity to Comply. The categories with poor reliability were Teaching Statement and Advanced Teaching. Reliability and Frequency of Occurrence of the TACCS Categories The Teaching Statement and Advanced Teaching categories were the only two therapist verbalization categories that were classified as having “poor” reliability. This is not surprising given the low frequency of occurrence of these categories during the 21 interaction sessions coded. In fact, a Teaching statement was coded in only four sessions and an Advanced Teaching statement was coded in only three sessions. It is difficult to assess reliability for categories that occur infrequently because there is little margin for error. For example, if a Teaching Statement occurs once in a session and one coder misses the statement, the percent agreement is .00. On the other hand, if both agree on the statement, percent agreement is 1.00. Low reliability estimates may also be due to limited experience for coding these categories. PCIT Teaching statements, Indirect Commands, and Advanced Praise were the therapist verbalization categories coded with “fair” reliability. Parent Noncompliance and Parent No Opportunity to Comply, two of the three parent behavior categories, were also coded with “fair” reliability. All of these categories were coded in the majority of sessions but still had low frequencies, often coded only one or two occurrences per session. Categories with higher frequencies per session, like PCIT Praise, are affected less by one error compared to categories that occur infrequently within a session, which leaves little room for rater disagreement. Lower reliability estimates for the less frequently coded TACCS categories is consistent with previous findings that showed that higher frequency behaviors are observed with greater reliability than behaviors with a low frequency of occurrence (Dorsey, Nelson, & Hayes, 1986).

PAGE 27

19 Refinement of the TACCS categories The TACCS categories and manual were in the developmental stage during rater coding. Rater discrepancies were sometimes based on limitations in the thoroughness of the manual rather than inaccuracy; numerous verbalizations in a segment brought new coding challenges that needed to be addressed in the TACCS manual with additional guidelines and examples. The confusion matrices illustrated the categories that were more difficult for raters to distinguish from one another, such as Direct Leading and Direct Command. The subtle distinctions that sometimes need to be made between a Direct Leading and Direct Command verbalization has come up several times in rater meetings and as a result several guidelines have been added to the manual. For example, therapists often say sentences like “Say hand me the (brief pause), and just pick something for him to give you.” The initial guideline and examples relating to “A verbalization that is both a Direct Command and Direct Leading statement is coded Direct Leading” did not address verbalizations like those in the example. More specific guidelines were added to the Direct Leading category so that coders could decide more objectively if “enough” of the statement is present to be considered Direct Leading. The manual also had to address verbalizations that were coded inconsistently because they fell into several different categories depending on the context. In the coaching situation, the timing of a verbalization often determines how it will be coded. For example, the verbalization “You want to make sure your command is direct” could fit into the PCIT Teaching statement, Direct Command, or Corrective Feedback categories depending on what the parent has previously done and therapist intent. Clearly differentiating the same verbalization according to different contexts by examples in the manuals is important for verbalizations that can have different therapist “intentions.” The

PAGE 28

20 use of context and rater inference should be avoided in a behavioral observation system as much as possible, however it is necessary to use clear examples in the manual for situations in which context must be taken into account. Throughout the process of reliability coding, the manual has expanded to include more of the therapists’ “tricks.” For example, to encourage a parent to change an unlabeled praise to a labeled praise, the therapist will say “for?” after a parent said “Great job” or “Thank you” to the child. It was difficult for raters, less familiar with coaching, to know what categories would be considered for the verbalization “for”? Specific examples such as this were added to the manual to increase rater accuracy for these common therapist tactics. Another common therapist tactic occurred when parents started to give a command in question form and therapists interrupted with the prompt “please” to indicate to the parent to restart the command as a statement beginning with the word “please”. This verbalization was added as a specific example of the Corrective Feedback category to highlight the fact that the therapist is having the parent correct his or her indirect command. Limitations The accuracy of the raters’ coding could not be assessed in this study. Although reliability estimates give an indication of rater agreement they do not assure rater accuracy. Accuracy refers to the verbalization being coded with the correct category. Thus, if both raters coded a verbalization into the same incorrect category, they would have high reliability but low accuracy. It is important in the development of an observational coding system to establish an accuracy criterion to help ensure that the categories are applied correctly to behaviors, so that meaningful results are obtained. It will be possible to estimate accuracy and prevent coder drift in the future by comparing

PAGE 29

21 each rater’s codes to segments that have been coded to consensus by a group of trained raters. Future Research It will be important to establish the validity of the TACCS by examining discriminative validity. For example, earlier and later coaching sessions should be discriminated by the TACCS categories (e.g, session one versus session four), as well as expert versus novice coaches. In addition, predictive validity will be established by predicting treatment status (e.g., completer vs. drop-out) with the TACCS categories. The development of a reliable and valid coding system for the therapist-parent interaction during the coaching component of PCIT provides a valuable tool for future research examining therapist and parent behaviors thought to facilitate and impact the process and outcome of treatment. Valid categories measuring therapist coaching behaviors such as praise, directives, and support in parent training, as well as parent behaviors such as compliance and noncompliance to therapist directives, may help to address the variables mediating successful treatment outcome for families. In future studies, it will be important to examine the relation between the therapist-parent interaction in weekly coaching sessions and treatment completion, maintenance of treatment gains, and parent skill acquisition. In addition, the TACCS provides a useful measurement of therapists’ in-session verbal behavior and skill. The evaluation of therapist training is a neglected area in the psychotherapy literature, in part due to the lack of valid measures of therapist behavior. Using the TACCS, therapist skill acquisition can be measured by examining a therapist’s behavior over time. In addition, the comparison of expert therapist behavior and novice therapist behavior has implications for therapist training and evaluation.

PAGE 30

22 Lastly, the exhaustive categories included in the TACCS and DPICS-II manuals for coding therapist, parent, and child behaviors in Parent-Child Interaction Therapy allows for sequential analysis of therapist and parent behavior during coaching. Examining the interactive patterns and reciprocal influence of the therapist and family in treatment will allow for a better understanding of the mechanisms through which change occurs in PCIT.

PAGE 31

APPENDIX SUMMARY OF TACCS CATEGORIES A. Therapist Behaviors 1. Direct Command a declarative statement made by the therapist indicating a specific action to be performed by the parent. 2. Indirect Command a statement made by the therapist for an action to be performed by the parent that is implied or in question form. 3. Direct Leading a verbalization that the therapist makes for the purpose of having the parent repeat it verbatim to the child. It might be useful to think of this as “ feeding lines ” to the parent. These statements can include having the parents describe their own behavior, reflect what the child has said, describe or praise an appropriate behavior of the child, or state a command to be given to the child. 4. Indirect Leading an observation made by the therapist about an attribute or action of the child that serves to highlight to the parents and give attention to what they can comment on. 5. PCIT Praise a praise given by the therapist after a parent’s use of skills specifically taught in PCIT. These skills include use of PRIDE skills, strategic ignoring, appropriate following in CDI, and correct use of the PDI skills including commands, praises, and the steps of the time-out procedure. 6. Advanced Praise a verbalization of the therapist that praises more specific qualities of the parent’s use of skills or general interaction with the child. 7. Contingent Praise a therapist praise of the parent’s verbalization or action that is complying with a therapist command or direct leading statement. 8. Unlabeled Praise a nonspecific positive evaluative statement by the therapist about the parent’s verbalization(s) or action(s). 9. Information Statement therapist statement that provides information to the parent but does not serve to teach or evaluate. Information statements include general observations of the child’s neutral behavior (i.e., “He’s just checking out the time-out room”) as well as information about the toys or room. 10. Teaching Statement a therapist verbalization that provides an interpretation or explanation for the child’s behavior or about child development in general. 23

PAGE 32

24 11. PCIT Teaching Statement – a therapist statement that explains the reason to use a particular skill or explains previous coaching. 12. Advanced Teaching Statement – a therapist statement that provides information about the progress of the family in treatment. These statements are highly specific to the individual child’s or family’s needs or the generalization of behaviors outside the session or home. 13. Positive Observation – a verbalization that highlights the child’s positive response to or gains in therapy. A positive observation is often given to point out behaviors associated with “better” attachment, parent-child interaction, or compliance. 14. Corrective Feedback any verbalization of the therapist that serves either to point out or to correct a mistake made by the parent. 15. Supportive Statement a verbalization that acknowledges the parent’s emotional reaction to the child or the parent’s emotional state. Supportive statements can also serve to assure the parent of the child’s safety or emotional state. 16. Humor a statement made by the therapist that has a joking or teasing quality and is often accompanied by laughing. 17. Acknowledgment a brief vocalization of the therapist indicating attention. This includes the therapist answering a question asked by the parent with yes or no. B. Parent Behaviors 1. Parent Compliance – when the parent attempts to comply with a preceding therapist command or repeats a direct leading statement. Compliance is coded only when there are two or fewer verbalizations by the parent between the therapist command or leading statement and compliance. 2. Parent Noncompliance when the parent does not attempt to comply with a preceding therapist command or direct leading statement. If the parent has more than two verbalizations between the therapist command or leading statement and compliance, and has opportunity to comply, code NC. 3. Parent No Opportunity to Comply when the parent does not have an opportunity to comply with a therapist command or direct leading statement within his or her next two statements to the child. Parent no opportunity to comply is also coded when there is no observable behavior to assess for compliance (i.e., Get him to start having fun), if there is a change in the parent-child interaction that does not allow for compliance, or if the therapist issues a second directive without giving the parent the opportunity to comply with the first.

PAGE 33

LIST OF REFERENCES Borrego, J. J., & Urquiza, A. J. (1998). Importance of therapist use of social reinforcement with parents as a model for parent-child relationships: An example with parent-child interaction therapy. Child & Family Behavior Therapy, 20, 27-54. Brestan, E. V., & Eyberg, S. M. (1998). Effective psychosocial treatments of conduct-disordered children and adolescents: 29 years, 82 studies, and 5272 kids. Journal of Clinical Child Psychology, 27, 180-189. Brestan, E. V., Eyberg, S. M., Boggs, S., & Algina, J. (1997). Parent-child interaction therapy: Parent perceptions of untreated siblings. Child and Family Behavior Therapy, 19, 13-28. Brestan, E. V., Foote, R.C., & Eyberg, S. M. (2003). The dyadic parent-child interaction coding system II (DPICS II): Reliability and validity with father-child dyads. (Manuscript submitted for publication). Dorsey, B. L., Nelson, R. O., & Hayes, S. C. (1986). The effects of code complexity and behavioral frequency on observer accuracy and interobserver agreement. Behavioral Assessment, 8, 349-363. Eisenstadt, T. H., Eyberg, S. M., McNeil, C. B., Newcomb, K., & Funderburk, B. (1993). Parent-child interaction therapy with behavior problem children: relative effectiveness of two stages and overall treatment outcome. Journal of Clinical Child Psychology, 22, 42-51. Eyberg, S. M., & Robinson, E. (1982). Parent-child interaction training: Effects on family functioning. Journal of Clinical Child Psychology, 11, 130-137. Fleiss, J. L. (1981). Balanced incomplete block designs for interrater reliability studies. Applied Psychological Measurement, 5, 105-112. Funderburk, B. W., Eyberg, S. M., Newcomb, K., McNeil, C., Hembree-Kigin, T., & Capage, L. (1998). Parent-child interaction therapy with behavior problem children: Maintenance of treatment effects in the school setting. Child and Family Behavior Therapy, 20, 17-38. Harwood, M. D., & Eyberg, S. M. (2003). Therapist verbal behavior early in treatment: Relation to successful completion of parent-child interaction therapy. Journal of Clinical Child and Adolescent Psychology. 25

PAGE 34

26 Hood, K. K., & Eyberg, S. M. (2003). Outcomes of parent-child interaction therapy: Mothers’ reports of maintenance three to six years after treatment. Journal of Clinical Child and Adolescent Psychology, 32, 419-429. McNeil, C. B., Eyberg, S. M., Eisenstadt, T. H., Newcomb, K., & Funderburk, B. (1991). Parent-child interaction therapy with behavior problem children: Generalization of treatment effects to the school setting. Journal of Clinical Child Psychology, 20, 140-151. Schuhmann, E. M., Foote, R. C., Eyberg, S. M., Boggs, S. R., & Algina, J. (1998). Efficacy of parent-child interaction therapy: Interim report of a randomized trial with short-term maintenance. Journal of Clinical Child Psychology, 27, 34-45. Timmer, S., & Urquiza, A. (2002). Coach Coding System (CCS), unpublished.

PAGE 35

BIOGRAPHICAL SKETCH Kelly O’Brien received her bachelor’s degree in psychology at St. Mary’s College of Maryland in 2002. She plans to receive her master’s degree in May 2004 and will pursue a doctoral degree in the Department of Clinical and Health Psychology, specializing in child psychology. Her primary research interest is treatment outcome for child populations. 27