<%BANNER%>

Identifying Benchmark Competency Criteria for a Rehabilitation Counseling Clinical Supervision Instrument

Permanent Link: http://ufdc.ufl.edu/UFE0015740/00001

Material Information

Title: Identifying Benchmark Competency Criteria for a Rehabilitation Counseling Clinical Supervision Instrument A Delphi Study
Physical Description: 1 online resource (170 p.)
Language: english
Creator: Moorhouse, Michael
Publisher: University of Florida
Place of Publication: Gainesville, Fla.
Publication Date: 2008

Subjects

Subjects / Keywords: competency, counseling, supervision
Rehabilitation Science -- Dissertations, Academic -- UF
Genre: Rehabilitation Science thesis, Ph.D.
bibliography   ( marcgt )
theses   ( marcgt )
government publication (state, provincial, terriorial, dependent)   ( marcgt )
born-digital   ( sobekcm )
Electronic Thesis or Dissertation

Notes

Abstract: The purpose of this study was to identify and draw consensus on a comprehensive set of competency criteria useful for clinical supervision evaluation. An essential function of any counseling profession is to ensure that graduate students are personally and professionally competent to practice within a community setting. Although the rehabilitation counseling profession may provide general guidance for competency standards, the responsibility to ensure that students have attained the necessary knowledge and skill routinely falls to the training faculty. Consequently, training programs have a substantial amount of autonomy when developing these competency criteria. Even though some flexibility with these tasks is necessary for individual programmatic goals, considerable variability promotes inconsistently monitoring and evaluation of students. The clinical portion of the training programs is perhaps most vulnerable to these challenges due to the dual role of the supervisor, philosophical differences regarding the purpose of clinical training, supervisor preparedness, the supervisor's fear of litigation, and difficulties in measuring clinical competence. Despite these challenges, this research was intended to identify a variety of clinical skills, attributes, and behaviors to aid clinical supervisors measuring trainee?s competency during clinical supervision. Using the Delphi methodology, 21 rehabilitation counseling educators identified a variety of performance benchmarks useful for clinical supervision assessment. The study resulted in 185 items useful for clinical competency evaluation. Items represented an assortment of competency domains including general indicators of the counseling process, specific clinical process skills, case conceptualization items, treatment planning items, professional behaviors, professional development items, self care items, personal attributes, and items specific to the supervision process. Implications for education and public policy are discussed as well as future development of a clinical supervision instrument measuring trainee clinical performance.
General Note: In the series University of Florida Digital Collections.
General Note: Includes vita.
Bibliography: Includes bibliographical references.
Source of Description: Description based on online resource; title from PDF title page.
Source of Description: This bibliographic record is available under the Creative Commons CC0 public domain dedication. The University of Florida Libraries, as creator of this bibliographic record, has waived all rights to it worldwide under copyright law, including all related and neighboring rights, to the extent allowed by law.
Statement of Responsibility: by Michael Moorhouse.
Thesis: Thesis (Ph.D.)--University of Florida, 2008.
Local: Adviser: Shaw, Linda R.

Record Information

Source Institution: UFRGP
Rights Management: Applicable rights reserved.
Classification: lcc - LD1780 2008
System ID: UFE0015740:00001

Permanent Link: http://ufdc.ufl.edu/UFE0015740/00001

Material Information

Title: Identifying Benchmark Competency Criteria for a Rehabilitation Counseling Clinical Supervision Instrument A Delphi Study
Physical Description: 1 online resource (170 p.)
Language: english
Creator: Moorhouse, Michael
Publisher: University of Florida
Place of Publication: Gainesville, Fla.
Publication Date: 2008

Subjects

Subjects / Keywords: competency, counseling, supervision
Rehabilitation Science -- Dissertations, Academic -- UF
Genre: Rehabilitation Science thesis, Ph.D.
bibliography   ( marcgt )
theses   ( marcgt )
government publication (state, provincial, terriorial, dependent)   ( marcgt )
born-digital   ( sobekcm )
Electronic Thesis or Dissertation

Notes

Abstract: The purpose of this study was to identify and draw consensus on a comprehensive set of competency criteria useful for clinical supervision evaluation. An essential function of any counseling profession is to ensure that graduate students are personally and professionally competent to practice within a community setting. Although the rehabilitation counseling profession may provide general guidance for competency standards, the responsibility to ensure that students have attained the necessary knowledge and skill routinely falls to the training faculty. Consequently, training programs have a substantial amount of autonomy when developing these competency criteria. Even though some flexibility with these tasks is necessary for individual programmatic goals, considerable variability promotes inconsistently monitoring and evaluation of students. The clinical portion of the training programs is perhaps most vulnerable to these challenges due to the dual role of the supervisor, philosophical differences regarding the purpose of clinical training, supervisor preparedness, the supervisor's fear of litigation, and difficulties in measuring clinical competence. Despite these challenges, this research was intended to identify a variety of clinical skills, attributes, and behaviors to aid clinical supervisors measuring trainee?s competency during clinical supervision. Using the Delphi methodology, 21 rehabilitation counseling educators identified a variety of performance benchmarks useful for clinical supervision assessment. The study resulted in 185 items useful for clinical competency evaluation. Items represented an assortment of competency domains including general indicators of the counseling process, specific clinical process skills, case conceptualization items, treatment planning items, professional behaviors, professional development items, self care items, personal attributes, and items specific to the supervision process. Implications for education and public policy are discussed as well as future development of a clinical supervision instrument measuring trainee clinical performance.
General Note: In the series University of Florida Digital Collections.
General Note: Includes vita.
Bibliography: Includes bibliographical references.
Source of Description: Description based on online resource; title from PDF title page.
Source of Description: This bibliographic record is available under the Creative Commons CC0 public domain dedication. The University of Florida Libraries, as creator of this bibliographic record, has waived all rights to it worldwide under copyright law, including all related and neighboring rights, to the extent allowed by law.
Statement of Responsibility: by Michael Moorhouse.
Thesis: Thesis (Ph.D.)--University of Florida, 2008.
Local: Adviser: Shaw, Linda R.

Record Information

Source Institution: UFRGP
Rights Management: Applicable rights reserved.
Classification: lcc - LD1780 2008
System ID: UFE0015740:00001


This item has the following downloads:


Full Text





IDENTIFYING BENCHMARK COMPETENCY CRITERIA FOR A REHABILITATION
COUNSELING CLINICAL SUPERVISION INSTRUMENT: A DELPHI STUDY




















By

MICHAEL DEVIN MOORHOUSE


A DISSERTATION PRESENTED TO THE GRADUATE SCHOOL
OF THE UNIVERSITY OF FLORIDA IN PARTIAL FULFILLMENT
OF THE REQUIREMENTS FOR THE DEGREE OF
DOCTOR OF PHILOSOPHY

UNIVERSITY OF FLORIDA

2008


































2008 Michael Devin Moorhouse




































To my parents Ken and Rose









ACKNOWLEDGMENTS

I would first like to thank my dissertation committee for all their mentorship and assistance

through out this process. To Dr. Linda Shaw, my chair, I appreciate all your direction and

supervision during my training; your expertise has contributed immensely to this project. To Dr.

Mary Ellen Young, thank you for your generosity during this study; you always made yourself

available no matter how busy you were. To Dr. John Rosenbek, you are an exemplary educator

and professional who continually demonstrates genuine care and interest in your student's

professional development thank you. To Dr. Peter Sherrard, I appreciate all your contributions

to this project. You always helped challenged me to conceptualize my ideas holistically and with

great thought. I would also like to thank to Drs. Mary Hennessey, Tracey Barnett, and Lynn

Koch for their continued support and assistance during this study. In addition, I would like to

extend a special thanks to my friend and colleague Dr. Jamie Pomeranz. Jamie, your mentorship

has greatly impacted my professional development and I truly would not be the researcher or

teacher I am today without your guidance. I would also like to acknowledge my family's

continued support during these last five years. To my mother Rose, your unconditional love,

support, and guidance through out the years have provided me with the strength to overcome

many obstacles and made me a better person. To my father Ken, the immense work ethic,

humility, and perseverance you have shown through out the years have been not only an

example, but an inspiration to me. I love you both very much. Lastly, I would like to thank the

love of my life, Donna. I cannot begin to articulate the profound impact you've had on me.

Everyday I think about how lucky I am to have you and cannot wait for us to begin our future

together.









TABLE OF CONTENTS

page

A C K N O W L E D G M E N T S ..............................................................................................................4

L IST O F T A B L E S ...................... ............... ....................................................... . 8

LIST OF FIGU RE S ................................................................. 9

ABSTRACT ............................................ .. ......... ........... 10

CHAPTER

1 INTRODUCTION ............... ............................ .............................. 12

B a c k g ro u n d ....................................................................................................................... 12
T he C clinical Supervision R ole................................................................ ............................15
The Need for a Standardized Instrument to Evaluate Rehabilitation Counseling Trainee
Com petency during Clinical Supervision ........................................ ....................... 15
Methods for Identifying Competency Criteria ............................................ ............... 23
G oals of R research ....................................................................................................... ......23
R research Question 1 ...................................... .. .... ........ ....... ..... 23

2 L ITE R A TU R E R E V IE W ........................................................................ .. .......................24

D defining Im pairm ent .............. ......... .............. ........ ............................ .. .. .......... 24
Rehabilitation Counseling Accreditation and Code of Ethics Literature ..............................27
Prevalence of Trainee Impairment and Dismissal Rates ................................. ...............29
Identifying T rainee P problem s ............................................... ..................... .......................3 1
D u e P process P rocedu res......... ........... .............................................................. ...... .......... 3 5
Clinical Supervision Evaluation Criteria..............................................................................44
Supervision M models ..................................... ....... .......... .. ....... .... 47
Clinical Supervision Instrum ents......................................... .................. ............... 51
T h e R asch M odel .................................................................................................................... 57

3 METHODOLOGY ............................. ...................... ........63

R research Question 1 ...................................... .. .... ........ ....... ..... 63
S tu d y D e sig n ............................................................................................................. 6 3
S am p lin g ............. .. .................. ..........................................................................6 5
D elphi P procedure ...............................66.............................
D elp h i rou n d 1 ...............................................................6 7
D elp h i rou n d 2 .............................................................6 9
D elp h i rou n d 3 .............................................................7 0
P ilo t T e stin g ............................................................................................................... 7 1











4 R E SU L T S ....................................................... 74

D e lp h i S tu d y ..................................................................................................................... 7 4
Panel D em graphics ......... ..... .... ............. ......... ................... 74
D elphi R ou n d 1..............................................................7 5
D elphi R ou n d 2 .............................................................76
D elphi R ou n d 3 .............................................................77

5 D IS C U S S IO N ...... ........................ ................................................................ 8 9

Introduction ........................................................................89
O verview of Significant Findings..................................................................................... 89
Differences in Participant Response Sets ............................................ .......... 89
Item Identification ................. ......... ........ ............ 91
Item s C on sen su s ...............................92.............................
L im stations ............................... ..............................................96
Im p licatio n s .........................................................................9 8
E d u catio n ............................... .. .... ... .......................................9 8
Clinical competency dimensions and benchmarks...................... ............... 98
Supervision evaluation continuity .................... ... ............... 103
Improving communication between supervisor and trainee................................104
P ub lic P policy ................................................. ... .............. ...... .............106
Association of American Colleges and Universities / Council for Higher
Education A accreditation .............................................................. .......... 106
Council on Rehabilitation Education / Commission on Rehabilitation
Counseling Certification....................... ......... ......... 107
F u tu re R e se arch .............................................................................................................. 10 8
C o n c lu sio n .......................... ............. ... .........................................................1 1 0

APPENDIX

A D ELPH I R O U N D 1 SU R V E Y .................................................................................... 112

B ROUND 1 EMAIL TO REHABILITATION COUNSELING EDUCATORS ................... 115

C IN F O R M E D C O N SE N T ................................................................................................ 117

D ROUND 1 FRIENDLY REMINDER EMAIL ..............................................118

E A U D IT T R IA L ............................................................................................................... 12 0

F ROUND 2 EMAIL TO REHABILITATION COUNSELING EDUCATORS ................... 131

G D ELPH I R O U N D 2 SU R V EY ....................................................................................... 132

H ROUND 2 FRIENDLY REMINDER EMAIL ..............................................141




6









I ROUND 3 EMAIL TO REHABILITATION COUNSELING EDUCATORS ................... 142

J D E LPH I R O U N D 3 SU R V E Y .................................................................... ....................143

K ROUND 3 FRIENDLY REMINDER EMAIL ..................................................160

L IST O F R E F E R E N C E S .................................................................................... .................... 16 1

B IO G R A PH IC A L SK E T C H .......................................................................... ....................... 170









LIST OF TABLES


Table page

2-1 Frequencies of impaired trainee characteristics as reported by previous studies (select
non-academ ic v ariables) .......................................................................... ....................34

3-1 Pilot test findings and revisions ............................... ....... ................................ 72

4-1 P participant dem ographics.......................................................................... ................... 75

4-2 P participant credentials............................................................................. .....................76

4-3 Items with means between 4.50 and 5.00 arranged according to consensus ...................78

4-4 Items with means between 4.00 and 4.49 arranged according to consensus ................. 82

4-5 Items with means between 3.50 and 3.99 arranged according to consensus ...................86

4-6 Items with means between 3.00 and 3.49 arranged according to consensus ...................87

4-7 Items with means below 3.00 arranged according to consensus ....................................88









LIST OF FIGURES


Figure p e

2-1 Sam ple free and test free related to height.................................. .......................... 59

2-2 Hierarchy of counseling ability compared to total score ................................................61









Abstract of Dissertation Presented to the Graduate School
of the University of Florida in Partial Fulfillment of the
Requirements for the Degree of Doctor of Philosophy

IDENTIFYING BENCHMARK COMPETENCY CRITERIA FOR A REHABILITATION
COUNSELING CLINICAL SUPERVISION INSTRUMENT: A DELPHI STUDY


By

Michael Devin Moorhouse

August, 2008

Chair: Linda Shaw
Major: Rehabilitation Science

The purpose of this study was to identify and draw consensus on a comprehensive set of

competency criteria useful for clinical supervision evaluation. An essential function of any

counseling profession is to ensure that graduate students are personally and professionally

competent to practice within a community setting. Although the rehabilitation counseling

profession may provide general guidance for competency standards, the responsibility to ensure

that students have attained the necessary knowledge and skill routinely falls to the training

faculty. Consequently, training programs have a substantial amount of autonomy when

developing these competency criteria. Even though some flexibility with these tasks is necessary

for individual programmatic goals, considerable variability promotes inconsistently monitoring

and evaluation of students. The clinical portion of the training programs is perhaps most

vulnerable to these challenges due to the dual role of the supervisor, philosophical differences

regarding the purpose of clinical training, supervisor preparedness, the supervisor's fear of

litigation, and difficulties in measuring clinical competence. Despite these challenges, this

research was intended to identify a variety of clinical skills, attributes, and behaviors to aid

clinical supervisors measuring trainee's competency during clinical supervision. Using the









Delphi methodology, 21 rehabilitation counseling educators identified a variety of performance

benchmarks useful for clinical supervision assessment. The study resulted in 185 items useful

for clinical competency evaluation. Items represented an assortment of competency domains

including general indicators of the counseling process, specific clinical process skills, case

conceptualization items, treatment planning items, professional behaviors, professional

development items, self care items, personal attributes, and items specific to the supervision

process. Implications for education and public policy are discussed as well as future

development of a clinical supervision instrument measuring trainee clinical performance.









CHAPTER 1
INTRODUCTION

Background

An essential function of any counseling profession is to ensure that graduate students are

personally and professionally competent to practice within a community setting. In accordance

with this fundamental principle, counseling educators develop appropriate curriculum and

learning objectives to make certain that students receive the required training for professional

practice. Although each counseling profession identifies the general knowledge domains and

expectations for adequate trainee preparation, the responsibility to ensure that students have

attained the necessary knowledge and skill normally falls to the training faculty (Robiner,

Fuhrman, Ristvedt, 1993). The same professional standards that require students to be

professionally competent provide only general guidelines regarding evaluation criteria and

expectations of minimal student performance. Thus, faculty members typically rely on

homegrown evaluation instruments that vary from program to program.

Further complicating the evaluation problem is the fact that training programs may have

difficulty identifying and addressing their students' nonacademic limitations (Hahn & Molnar,

1991). More specifically, programs frequently struggle to identify and intervene with students

who perform well academically, but may have character or psychological issues that interfere

with their counseling ability (Bemak, Epp, & Keys, 1999). The challenges associated with

personal suitability coupled with non-specific evaluation criteria may result in inconsistent

evaluation of the student. Consequently, programs may be faced with the difficult decision of

endorsing questionable trainees for certification or licensure. By allowing impaired trainees to

graduate without remediation, counselor educators not only endanger their professional integrity,

but more importantly jeopardize their service to the public.









Similar to other clinical counseling programs, the rehabilitation counseling field relies on

professional organizations like the Commission on Rehabilitation Counselor Certification

(CRCC) and the Council of Rehabilitation Education (CORE) to help guide trainee evaluation.

However, these organizations provide only general guidelines regarding trainee competence and

therefore rehabilitation counseling programs must develop additional criteria to supplement the

existing standards. For example, CORE (2002) lists several requisite knowledge domains for the

rehabilitation counseling profession. One domain, "Counseling and Consultation," (C.5)

describes the general tasks for which a rehabilitation counseling trainee is expected to

demonstrate competency within the counseling process. More specifically, the domain contains

tasks such as: (a) conduct individual counseling sessions with consumers (C.5.1), (b) develop

and maintain a counseling relationship with the individual (C.5.2), (c) recommend strategies to

assist the consumer in solving identified problems that may impede the rehabilitation process

(C.5.6).

While this information clearly facilitates the academic process and provides a general

outline for assessing student competence, nowhere in the accreditation standards is competency

operationally defined. Faculty members must therefore determine what constitutes

"competence" and must also determine how to evaluate the student's rehabilitation counseling

ability. Rehabilitation counseling programs have a substantial amount of autonomy when

developing these criteria and procedures. Although some flexibility with these tasks is necessary

to accommodate each program's goals, considerable variability between these programs may

result in counseling programs inconsistently monitoring and evaluating students (Robiner et al.,

1993).









Research on rehabilitation counselor roles and functions (Leahy, Shapson, & Wright,

1987; Muthard & Salomone, 1969; Rubin, Matkin, et al., 1984) might be a resource in

determining the competencies upon which students should be assessed. However, these studies

focus only on the general tasks of the rehabilitation counseling professional. While this research

provides invaluable information pertaining to the professional identity of rehabilitation

counselors, the studies examining the roles and functions of rehabilitation counselors provide

limited information pertaining to the trainee's counseling ability. As Janikowski (1990) notes,

the rehabilitation counseling literature has typically focused on the functions of the profession,

rather than the competence of the professional. In this instance, Janikowski refers to function as

the specific tasks orjob duties related to occupation, whereas competency is the underlying

characteristics of the professional that lead to occupational success.

For educational programs to consistently monitor and evaluate trainees, supervisors should

define competencies that trainees should acquire and the standards that the trainees should reach,

as well as the criteria used to evaluate those competencies and the thresholds at which trainees

meet those standards (Robiner et al., 1993). Even though it is unlikely that uniform evaluation

criteria will suit every program, the rehabilitation counseling profession could, as a first step,

operationally define the concept of counseling competence and develop a comprehensive set of

behaviors (e.g. ability to empathize, ability to reflectively listen, etc.) that will aid in trainee

evaluation. By developing specific, behaviorally defined criteria, trainer feedback becomes more

concise and faculty can more consistently evaluate whether the student meets program goals

(Freeman, 1985). As Kerl, Garcia, McCullough, and Maxwell (2002) opine, objective evaluation

helps faculty members communicate with their trainees as well as helps prevent capricious









evaluator judgment. Thus, rehabilitation counseling educators will strengthen the profession by

ensuring programs produce appropriate counselors who demonstrate adequate counseling ability.

The Clinical Supervision Role

Even when trainee evaluation is continuous, many student limitations are more likely to be

identified during the clinical portion of the program (Lamb, Presser, Pfost, Baum, Jackson, &

Jarvis, 1987). Perhaps one area of counselor training most appropriate for identifying trainee

competence is during clinical supervision. Unlike licensure exams or classroom tests, supervisors

can assess how well the trainee applies classroom knowledge to a counseling session and they

therefore have the unique ability to determine the trainee's skill and judgment (Robiner et al.,

1993). Additionally, Pope and Kline (1999) recognize that within the counseling profession, the

trainee's personal characteristics play an integral role during evaluation. Therefore, clinical

supervision may offer the best opportunity for training faculty to identify and address the

student's clinical skill and personal characteristics. During clinical work, trainees are interacting

with real clients rather than in simulated classroom settings, and therefore faculty can better

assess the trainee's professional and personal shortcomings that might not have been apparent

during coursework.

The Need for a Standardized Instrument to Evaluate Rehabilitation Counseling Trainee
Competency during Clinical Supervision

Clinical supervision is a basic component of counselor training that has developed into an

independent body of theory and research (Bernard & Goodyear, 2004). However, within the

rehabilitation counseling profession, clinical supervision research continues to be understudied

(Herbert & Richardson, 1995; Herbert, Ward, & Hemlick, 1995; Maki & Delworth, 1995;

Stebnicki, 1998). More specifically, the rehabilitation counseling literature has vastly

overlooked trainee evaluation from the supervisor's perspective. Although Thielsen and Leahy









(2001) have identified "evaluation and assessment" as a critical knowledge domain for field-

based clinical supervision, there is a paucity of rehabilitation counseling research regarding

competency criteria for evaluating rehabilitation counseling trainees. In fact, when Koch,

Schultz, Hennessey, and Conyers (2005) surveyed rehabilitation counseling educators,

professionals, and students, they suggested that the "identification and assessment of clinical

supervision competencies" was an area warranting further investigation in rehabilitation

education (p.10). Despite this need, the rehabilitation counseling literature has focused on

various other aspects of clinical supervision including identifying helpful supervisor practices

(Stebnicki, Allen, & Janikowski, 1997), identifying current supervision methods use in

rehabilitation counseling (Herbert, 2002), describing ethical considerations within clinical

supervision (Blackwell, Strohmer, Belcas, & Burton, 2002; Tarvydas, 1995), describing models

to ensure proper trainee due process (Michaelson, Estrada-Hernandez, & Wadsworth, 2003),

identifying the need for masters level supervision training (Scott, Nolin & Wilburn, 2006),

determining what knowledge is essential to provide effective supervision (Thielsen & Leahy,

2001), and developing supervision models for both the educational setting (Maki & Delworth,

1995; Schultz, Copple, & Ososkie, 1999) and public setting (Schultz, Ososkie, Fried, Nelson, &

Bardos, 2002). Although this body of research greatly expands our understanding and

conceptualization of clinical supervision, it fails to fully address what Bernard and Goodyear

(2004) suggest may be the "nucleus" of clinical supervision trainee evaluation.

In 1995, Maki and Delworth recognized the need to operationally define clinical

supervision specifically for the rehabilitation counseling profession. Using Bernard and

Goodyear's (1992) definition as a foundation, Maki and Delworth stated that rehabilitation

counseling supervision is "a distinct intervention, the use of which requires the trained supervisor









to have specific knowledge and skills in multiple domains, including, but not limited to,

education, consultation, and counseling" (p.284). The authors hoped that this definition would

help supervisors better characterize the relationship between the supervisor and trainee (Maki &

Delworth, 1995).

Despite Maki and Delworth's (1995) effort to define clinical supervision relative to

rehabilitation counseling, Stebnicki (1998) suggested that their definition was too general to

encompass the unique characteristics of the profession. Therefore, Stebnicki proposed a more

elaborate clinical supervision definition that sought to better reflect rehabilitation counseling. He

defines clinical supervision as:

a specialty area that facilitates a positive interpersonal relationship with supervisees using a
diversity of supervisory styles and approaches in both an individual and group dynamic
process and is facilitated using the roles of educator, consultant, and counselor to stimulate
and increase the supervisee's (a) process skills and psychosocial rehabilitation counseling
strategies to enhance positive and ethical client outcomes, (b) efficacy as a counselor being
mindful of the supervisee's ongoing developmental and personal growth needs in the
process, and (c) conceptualization of salient features concerning the client's disability and
other critical aspects related to his or her problem issues (p.139-140).

Although each definition provides a general framework for rehabilitation counseling, both fail to

mention the evaluative nature of clinical supervision. As Bernard and Goodyear (2004) state,

"evaluation is implicit in the supervisors' mandate to safeguard clients, both those who will be

seen by the supervisee and those who would be seen in the future" (p. 11). Specifically,

supervisors evaluate students to ensure they reach a minimal level of professionally competence

and exhibit suitable personal characteristics for working with clients.

In general, trainee evaluation reveals a set of complex issues that permeate any evaluative

process. As previously stated, clinical supervisors within the academic setting aid in trainee

development, as well as limit impaired trainees from entering the counseling profession.

Robiner et al. (1993, p.4) assert that these dual roles of"nurturant teacher" and "vigilant









gatekeeper" may serve as the primary source of supervisory conflict. A rehabilitation counselor,

like any counselor or therapist, is trained to be non-judgmental and facilitate client growth.

However, the qualities that make for a good counselor may also interfere with the supervisor's

ability to evaluate a trainee. Hoffman, Hill, Holmes, and Freitas (2005) suggest that supervisors

may feel uncomfortable when providing the trainee subjective feedback, particularly when it

involves the trainee's personality or professional issues. In fact, Gizara and Forrest (2004)

found that the personal impact for supervisors when dealing with an impaired trainee was an

overwhelming concern. Specifically, the authors note that "none of the responses brought the

participant group together so dramatically as inquiries about the personal impact of dealing with

intern impairment" (p.136). As Bernard and Goodyear (2004) note, imagine a situation where

the supervisor uses the trainee's progress as the critical indicator for evaluation. The trainee has

worked hard and improved during the practicum, but continues to perform below program

expectations. Now the supervisor must weigh the student's effort and progress against the

ethical responsibility to the profession and potential clients (Bernard & Goodyear). Thus, trainee

evaluation can lead to a "disturbing role shift" for supervisors (Hahn & Molnar, 1991, p.417),

and cause them some level of personal discord if remediation and/or dismissal procedures are

employed, particularly if a relationship with the student exists (Gizara & Forrest).

Similar to the personal struggle educators may face when evaluating a trainee, Gizara and

Forrest (2004) found that supervisors may feel unprepared when dealing with problematic

trainees. Specifically, the authors note that several participants felt ill-equipped to identify and

address problematic trainees. Moreover, the authors note that supervisors may doubt their own

ability to appropriately identify problematic trainees and in fact, may feel an increased sense of

vulnerability when their own supervisory skills came to the forefront.









Another challenge to effective trainee evaluation is the fear that trainees may initiate legal

action upon the supervisor, program, and/or school. Several authors (Frame & Stevens-Smith,

1995; Tedesco, 1982; Vacha-Haase, Davenport, & Kerewsky, 2004) suggest that litigation may

account for the faculty member's reservations regarding trainee evaluation. In fact, Vacha-Haase

et al. reported that fear of litigation was highest when students were terminated for a lack of

interpersonal skills and supervision difficulties. Specifically, the authors note that, "it may be

difficult to document more subjectively determined problematic behaviors adequately, whereas

problematic behaviors with objective criteria may be easier for faculty to identify confidently

and act on with more certainty" (Vacha-Haase et al., p. 119). Despite the authors' assertion,

several authors (Frame & Smith-Stevens; Knoff & Prout, 1985; Olkin & Gaughen, 1991; Kerl et

al., 2002) note that various court decisions have upheld the program's right to dismiss students

who exhibit personal or professional deficiencies. Although these precedents may support the

counseling program's opinion, Knoff and Prout caution that these court rulings do require

programs follow proper due process procedures.

Consequently, several authors (Baldo, Softas-Nall, & Shaw, 1997; Bemak, Epp, & Keys,

1999; Frame & Stevens-Smith, 1995; Lamb, Presser, Pfost, Baum, Jackson, & Jarvis, 1987;

Lumadue & Duffey, 1999) have written specific remediation and dismissal procedures to ensure

counseling and psychology programs provide adequate due process. Although the literature and

professional organizations have recognized the importance of due process, approximately half of

the programs continue to operate without these policies and procedures in place (Olkin, &

Gaughen, 1991; Vacha-Haase et al., 2004). This may be, at least in part because programs that

have comprehensive policies and procedures in place often find that the dismissal process can be









complicated and result in additional programmatic modifications (McAdams, Foster, & Ward,

2007).

A fourth challenge associated with trainee evaluation is the level of support a supervisor

receives from the department and fellow faculty members may greatly impact the evaluation

process (Gizara & Forrest, 2004). In fact, Vacha-Haase et al. (2004) found that training directors

identified disagreement among faculty as the second biggest barrier to dismissing a student

behind fear of litigation. In their qualitative study investigating the experiences of supervisors in

dealing with impaired trainees, Gizara and Forrest found that the level of conflict within the

supervisory group can impact the evaluation process. The authors found that supervisory

disagreement can manifest in a variety of ways. For example, the way in which a supervisor

conceptualizes the clinical process can impact the level of supervisory support. Specifically, one

supervisor may view clinical supervision/internship to be a development process, while another

supervisor may perceive that supervision is primarily a gatekeeping function whereby trainees

must meet some minimal level of competence. In addition to differences in perception about the

purpose of clinical supervision, Gizara and Forrest reported that supervisors also suggested that

preexisting interpersonal conflict within the supervisory group can influence the evaluation

process to the point where some supervisors revealed that they may filter the amount of

information they provide when discussing their supervisees.

In addition to the challenges rehabilitation counseling educators face when evaluating a

problematic trainee, a program's failure to adequately address the trainee's deficits can

negatively impact other students in the training program. Oliver, Bernstein, Anderson,

Blashfield, and Roberts (2007) surveyed graduate students in clinical psychology to examine

their attitudes toward impaired peers. Among other things, the authors found that students may









not only resent problematic students for causing them additional work and lost opportunity, but

may also resent faculty members who put off addressing a problematic trainee or simply allow

him or her to proceed through the program without meeting some specified standard. Similarly,

Mearns and Allen (1991) found that students viewed faculty members as being significantly less

active than faculty viewed themselves when addressing an impaired graduate student. In

addition, the authors found that students demonstrate a variety of emotions when confronted with

an impaired peer. Specifically, students frequently reported feelings of angry, conflicted,

frustrated, apprehensive, and concerned. Interestingly, Mearns and Allen found that faculty

members tended to overestimate the number of students who sought to protect their impaired

peer. In fact, when confronted with an impaired peer, students reported that "their feelings of

ethical obligation and betrayal outweighed sentiments of loyalty to their problematic peers"

(p. 198). Additionally, Mearns and Allen found that although the overall incidence of perceived

impairment and ethical improprieties was low, the reported incidence of problematic functioning

negatively correlated with the climate of the program. Thus, the challenges associated with

properly evaluating and remediating a problematic trainee not only impacts the faculty and

trainee, but may also alienate other trainees from the program.

While the literature regarding clinical supervision may bring attention to problematic

trainees, a consistent methodology for adequately measuring the trainee's counseling

competency remains elusive. Bernard and Goodyear (2004) recognize that within clinical

supervision, "the difficulty of establishing criteria for evaluation and the equally difficult task of

measuring them is a professional reality (p.23)." According to Lamb, Cochran, & Jackson

(1991), programs generally assess general student performance in three domains: (a) knowledge

and application of professional standards, (b) competency, and (c) personal functioning.









Although the counseling and psychology literature is replete with articles and books describing

various professional competencies and personal attributes appropriate for these domains (Corey,

2004; Cormier & Cormier, 1991; Hensley-Choate, Smith, & Spruill, 2005; Hill, 2004; Johnson &

Campbell, 2002; McCarthy & Leierer, 2001; Miller & Rollnick, 2002; Nelson-Jones, 2005; Pope

& Kline, 1999; Ragg, 2000; Shebib, 2002), the specific criteria used to evaluate the trainee

among the programs is quite variable. As Robiner et al. (1993) state, until there is consensus

regarding performance standards within specific competence domains, all evaluation is

vulnerable to subjectivity.

In an effort to aid in trainee evaluation, several authors (Bernard, 1997; Eriksen, &

McAuliffe, 2003; Hackney & Cromier, 1994; Lumadue & Duffey, 1999; Myrick & Kelly, 1971;

Getting & Michaels, 1982; Robiner, Fuhrman, Ristvedt, Bobbitt, & Schirvar, 1994) have

developed instruments to measure the trainee's professional competency and character.

Although each instrument provides an excellent resource for clinical supervisors, each one was

developed for professions other than rehabilitation counseling. Additionally, no clinical

assessment instrument has been validated using Modern Test Theory (MTT). Compared to

Classical Test Theory (CTT), MTT techniques like Rasch analysis allow the researcher to

examine the psychometric properties of an instrument at the item level, rather than at the

instrument level (Bond & Fox, 2001). As a result, the Rasch analysis produces a set of unique

psychometric statistics, such as an item hierarchy, that allows supervisors to determine the

trainee's ability level based on specific performance behaviors arranged in hierarchal order from

least to most difficult. In other words, the trainee's ability level is based on the types of items

near his or her ability level, rather than simply relying on a sum of scores. Practically speaking,

the supervisor knows what types of counseling behaviors are below, at, and above the trainee's









ability level. Thus, supervisors can simultaneously measure trainee competence and target

specific interventions while documenting the trainee's professional development over the course

of his or her academic career.

Methods for Identifying Competency Criteria

One method appropriate for identifying rehabilitation counseling competency criteria is the

Delphi method. The Delphi method enables researchers to elicit anonymous expert opinion, and

then refines this information for experts to reexamine their responses) compared to the group

response (Vazquez-Ramos, Leahy, & Estrada-Hernandez, 2007). The process consists of a

minimum of three rounds with the ultimate goal of reaching consensus among the experts

without the bias drawbacks of comparable techniques (Linderman, 1981). By identifying

specific competency criteria related to professional skills and personal suitability, rehabilitation

educators can ensure trainees are prepared to counsel people with disabilities.

Goals of Research

The primary goal of this research is to implement a Delphi study in order to identify

specific competency criteria useful for the development of a rehabilitation counseling clinical

supervision measure.

Research Question 1

What professional, personal, and clinical competency items are useful for the evaluation of

rehabilitation counseling trainees?









CHAPTER 2
LITERATURE REVIEW

Defining Impairment

In order to understand the inherent difficulties associated with trainee evaluation, it is

necessary to be familiar with the terminology that is often used to describe students who exhibit

problematic professional and/or personal behaviors. Unfortunately, researchers usually do not

operationalize these types of terms (Vacha-Hasse et al., 2004) and as a result, certain terms may

be used generically or inappropriately to describe different sets of student issues (Elman &

Forrest, 2007). Gizara and Forrest (2004) caution that without accurately identifying a student

issue, the nature of the problem can be obscured. As a result, educators may not only use an

ineffective strategy to address the issue, but also be unaware of the relevant literature that could

be used to deal with the problem (Gizara & Forrest). Among all the counseling professions, only

the psychology literature (Elman & Forrest, 2004; Forrest, Elman, Gizara, & Vacha-Hasse, 1999;

Gizara & Forrest; Vacha-Haase et al.) continues to debate the appropriateness of certain terms

associated with professional and personal behaviors. Although the rehabilitation counseling

profession clearly recognizes the importance of endorsing only suitable trainees, dialogue about

the appropriate use of certain words remains largely unexplored.

Although some counseling literature has referred to problematic students as "unsuitable"

(Bernard, 1975; Biaggio, Gasparikova-Krasnec, & Bauer, 1983) or having "professional

deficiencies" (Procidano et al., 1995), the most common term used to identify these types of

students is impaired (Bemak et al., 1999; Boxley, Drew, & Rangel, 1986; Bradley & Post, 1991;

Frame & Stevens-Smith, 1995; Lamb et al., 1987, Li, 2000; Mearns & Allen, 1991; Wilkerson,

2006; Woodyear, 1997). One of the earliest definitions of impairment is offered by Lamb et al.

(p.598):









An interference in professional functioning that is reflected in one or more of the following
ways: (a) an inability and/or unwillingness to acquire and integrate professional standards
into one's repertoire of professional behavior, (b) an inability to acquire professional skills
in order to reach an acceptable level of competency, and (c) an inability to control personal
stress, psychological dysfunction, and/or excessive emotional reactions that interfere with
professional functioning.

Although this definition provides a comprehensive summary of various trainee issues (i.e.

professionalism, development of counseling skills, personal suitableness), the term impairment

was originally reserved for professionals whose performance regressed due to such issues as

mental illness, substance abuse, or emotional problems (Forrest et al., 1999). The use of the

same term in two very different contexts has prompted several authors (Elman & Forrest, 2007;

Forrest et al.; Gizara & Forrest, 2004; Kutz, 1986) to call for a clearer delineation of these types

of words. Specifically, these authors argue that the term impairment be reserved for situations

where the trainee or practitioner first establishes a baseline of adequate performance and then

reverts to an unacceptable level of performance. On the other hand, the term incompetent or not

competent should refer to an individual who has not yet reached a minimum level of acceptable

performance. In fact, the rehabilitation counselor code of professional ethics (CRCC, 2002)

appears to align with this model by reserving the word impairment to identify professionals

whose physical, mental, or emotional problems are likely to harm the client or others (D. .i).

Similarity, the code uses the term competence when referring to the professional's knowledge

(D. .a, D.l.c., D.l.h, F.5.a, G.l.c).

Ironically, Gizara and Forrest (2004) investigated, among other topics, how supervisors

defined trainee impairment, rather than incompetence. The authors report that the participants

clearly articulated three themes indicative of trainee impairment, while a fourth was fairly

inconsistent. The three identifiable themes were: (a) the intern's behavior was professionally

harmful or deficient, (b) the behavior was a clear pattern, and (c) the behavior was not resolving









(Gizara & Forrest). The ambiguous fourth theme centered on whether or not trainee's behavior

required a noticeable shift in performance. During the interview, the respondents categorized

impairment in two different groups. The first category related to the "diminished functioning" of

the trainee and the second category related to trainee's "skill deficits" (Gizara & Forrest). This

second category, "skill deficits," included a wide range of issues such as clinical skills,

interpersonal difficulties, lack of self awareness, and lack of professional responsibility. The

authors acknowledge that "the vast majority of examples of impairment offered spontaneously

by these participants did not meet their original conception of a 'noticeable shift in performance'

but rather were more illustrative of skill deficiency" (p.133). Hence, psychology intern

supervisors appeared to associate the term impairment more closely with the trainee's skill

development, personal suitability, and adherence to professional standards, rather than whether

the trainee reverted back to an unacceptable level of performance.

Although the argument for the appropriate use and understanding of certain terms to

describe different trainee issues is clearly valid, impairment appears to be the most common term

found in the literature that relates to issues associated with the student's personal suitability and

professional development. However, as Elman and Forrest (2007) illustrate, using the term

impairment "overlaps with its use in the Americans with Disabilities Act (ADA; 1990) and the

protections and guidance defined therein" (p. 502). Specifically, using the term impairment

suggests the trainee has a physical or mental impairment that limits one or more major life

activities, and therefore supervisors may be vulnerable to legal ramifications if they use the word

impairment. Consequently, for the purposes of this study, this author will use the terms

competency or competent to refer to a trainee's ability to adequately demonstrate counseling









skills, function in a personal appropriate manner, and follow the professional standards set forth

by the rehabilitation counseling profession.

Rehabilitation Counseling Accreditation and Code of Ethics Literature

During the clinical portion of a student's training, rehabilitation counseling educators may

be challenged to determine whether a trainee is qualified for professional endorsement. Unlike

traditional coursework where students demonstrate their competency through classroom

participation and assignments, clinical supervisors evaluate the trainee's ability to effectively

counsel clients. Although this task is required by every counseling education program, the

criteria used to evaluate a student during clinical supervision may vary considerably from

program to program. This inconsistency is particularly noted during clinical supervision because

trainee evaluation is largely subjective and therefore it may be more difficult to identify what

characteristics and behaviors separate suitable trainees from impaired trainees.

The importance of proper trainee evaluation is evidenced in both the 2002 Council of

Rehabilitation Education (CORE) accreditation standards and the 2002 Commission of Certified

Rehabilitation Counseling (CRCC) code of professional ethics. CORE states that clinical faculty

will review the progress of both practicum and internship students (D.1.7, D.2.4) as well as

require a written procedure addressing problematic trainees (D.1.8, D.3.4). More explicitly,

Section G.2.b of the CRCC code of professional ethics further identifies the evaluation role of

faculty members:

Rehabilitation counselor educators will clearly state, in advance of training, to students and
internship supervisees, the levels of competency expected, appraisal methods, and timing
of evaluation for both didactic and experiential components. Rehabilitation counselor
educators will provide students and internship supervisees with periodic performance
appraisal and evaluation feedback throughout the training program.

Additionally, both CORE (2002) and the CRCC (2002) recognize that trainee development

transcends the acquisition of counseling skills and extends into personal suitability. CORE

27









defines a rehabilitation counselor as someone who possesses the knowledge, skill, and attitudes

needed to collaborate with people with disabilities. Additionally, these accreditation standards

are inundated with statements regarding the trainee's ability to empower the client, be self-

aware, and understand the importance of a collaborative relationship. Likewise, Section G.3.a of

the CRCC recognizes the importance of personal suitability:

Rehabilitation counselors, through ongoing evaluation and appraisal, will be aware of the
academic and personal limitations of students and supervisees that might impede
performance. Rehabilitation counselors will assist students and supervisees in securing
remedial assistance when needed, and will dismiss students or supervisees who are unable
to provide competent service due to academic or personal limitations.

Despite the emphasis on trainee evaluation, neither CORE (2002) nor the CRCC (2002)

provide guidance relative to specific evaluation criteria for trainees and students. In particular,

the CORE accreditation standards list general competency domains composed of knowledge

areas and trainee behaviors, but do not outline clear expectations for supervisors to evaluate

trainee competency. For example, one competency domain, "Counseling and Consultation," lists

15 knowledge areas (i.e. counseling and personality theory, mental health counseling,

interviewing and counseling skill development) followed by 15 behaviors that are indicative of

that domain. Although these areas are behavioral, they are too broad to consider for evaluation

criteria during clinical supervision. For instance, the behaviors include such items as: (a)

"conduct individual counseling sessions with consumers" (C.5.1), (b) "develop and maintain a

counseling relationship with the individual" (C.5.2), and (c) "assist consumers in modifying their

lifestyles to accommodate individual functional limitations" (C.5.14). Practically speaking, how

does a faculty member determine whether a student is adequately conducting an individual

counseling session (C.5.1) without defining what behaviors reflect this process? Moreover,

when does the trainee's ability level qualify him or her for professional endorsement? Without









more specific indicators of professional performance, clinical supervisors may be unable to

determine whether a trainee is counseling effectively.

These general knowledge domains are particularly ineffective as evaluation criteria during

clinical supervision because counseling is a coalescence of behaviors, skills, and attitudes that

are continually evolving. During clinical work, the supervisor and trainee review and process

counseling sessions and address other personal issues that may materialize. Although

supervisors utilize various resources to evaluate the trainee and guide professional development,

general evaluation criteria will ultimately lead to inconsistent evaluation of trainees.

Prevalence of Trainee Impairment and Dismissal Rates

Several studies (Biaggio et al., 1983; Boxley et al., 1986; Bradley & Post, 1991;

Gallessich & Olmstead, 1987; Huprich & Rudd, 2004; Mearns & Allen, 1991; Olkin & Gaughen,

1991; Procidano et al., 1995; Schwebel & Coster, 1998; Tedesco, 1982; Vacha-Haase, 1995)

have examined trainee impairment and/or dismissal rates among their respective professions.

Although Forrest et al. (1999) believe that drawing conclusions from this literature is challenging

because the authors of these studies requested data and reported findings differently, this

literature does provide a general perspective on trainee impairment and dismissal trends among

psychology and counseling programs.

In general, studies that examined trainee impairment rates found that the vast majority of

programs regularly encounter an impaired student. In fact, Olkin and Gaughen (1991) found that

over two-thirds (76%) of master's programs in mental health (e.g. clinical and counseling

psychology, counselor education, community psychology) identified one to three problematic

students per year and 24% identified four or more problematic students per year. Similarly,

Procidano et al. (1995) found that 89% of responding psychology departments reported one or

more instances of non-academic professional deficiency among their students within the last 5

29









years. Mearns and Allen (1991) found that 93% of faculty had perceived an impaired trainee

within the last 5 years, while Schwebel and Coster (1998) reported that within the last 5 years,

psychology programs averaged 2.55 impaired students. On the other hand, Huprich and Rudd

(2004) reported that within the last 10 years, 98% of responding psychology doctoral programs

and 68% of internship sites were aware of at least one impaired trainee. Current indications of

trainee impairment suggested that 65% of doctoral programs identified one or more impaired

students and 10% of internship sites identified one impaired student. Finally, Boxley et al.

(1986) and Olkin and Gaughen found remarkably similar annual trainee impairment rates despite

surveying different populations. Boxley et al. reported an annual trainee impairment rate of

4.6%, while Olkin and Gaughen found that the mean percentage of problematic students was

4.8%.

In addition to trainee impairment studies, several studies focused on trainee termination

rates. As expected, the prevalence of trainee terminations among psychology and counselor

education programs was lower than the reported student impairment rates. Tedesco (1982)

surveyed 167 directors from APA approved internship sites and noted that within the last 5 years,

51 trainees did not complete their internship because they either left on their own (n=24) or were

terminated by the site director (n=27). Furthermore, the internship directors considered an

additional 89 trainees for termination. Biaggio et al. (1983) reported that 86% of doctoral

programs and 75% of master's programs in clinical psychology had attempted at least 1

dismissal. Regarding annual termination rates, Gallessich and Olmstead (1987) found that over

the past three years, counseling psychology programs have terminated an average of one student.

Similarly, Vacha-Haase (1995) reported that 52% of responding clinical, counseling, and school

psychology programs had at least one student dismissal within the last 3 years. On the other









hand, Bradley and Post (1991) indicated that within the last 5 years, counselor education

programs had dismissed anywhere from zero to 30 students. Of the programs reporting at least

one student dismissal in the last five years (81%), the average number of terminations was 5.7

students.

In sum, trainee impairment appears to be a common issue across psychology and

counseling programs. Although the percentage of impaired trainees may appear relatively minor

(4.6-4.8%) compared to the number of trainees in counseling programs, Biaggio et al. (1983)

note that "it is a steady trickle that requires attention (p. 19)." In particular, the amount of time

counseling programs invest in impaired trainees is considerably higher than with a non-impaired

student (Vacha-Hasse, 1995). Vacha-Hasse found that training directors may devote as much as

20 hours a month to impaired students, while faculty members may spend as much as 40 hours a

month addressing student impairment issues. In reality, impaired trainees consume a

disproportionate amount of faculty time and program resources. Although one of the goals of

counselor training is to develop competent professionals, a supervisor's first priority is to ensure

that trainees do not harm clients and others (Frame & Stevens-Smith, 1995). Therefore, it is

necessary for counseling programs to identify the types of non-academic trainee impairment

issues that may interfere with the student's development. By familiarizing themselves with these

impairment traits, counseling programs can not only ensure they identify impaired trainees, but

also fulfill their obligation to protect the client population at large.

Identifying Trainee Problems

Although identifying the rate of problematic students highlights the need for proper trainee

evaluation, outlining what specific behaviors constitute trainee impairments can better prepare

faculty members for evaluation and remediation procedures. By delineating various student

issues, educators can simultaneously strengthen their professional gatekeeping process while

31









aiding in trainee development. Specifically, identifying the various trainee problems may

improve faculty and student communication. Faculty may be more likely to effectively intervene

with problematic students if they and trainees acknowledge that problems are likely to occur

(Vacha-Haase et al., 2004).

As previously mentioned, the ability to identify and remediate a problematic trainee is a

fundamental component for any counseling profession. Researchers from various counseling

professions (Boxley et al., 1986; Huprich & Rudd, 2004; Li, 2001; Olkin & Gaughen, 1991;

Procidano et al., 1995; Vacha-Haase, 1995) have investigated how training faculty and/or

program directors classify problematic students. Collectively, these studies have identified

several areas of trainee issues beyond the traditional academic coursework. Table 2-1 illustrates

the most common non-academic trainee issues among counseling related programs. Despite this

extensive research, drawing conclusions from these studies may be difficult because of

differences in sampling, questioning, and categorization of trainee issues (Forrest et al., 1999).

As first described by Forrest et al., studies solicited respondents from a variety of positions

including chairpersons or training directors from psychology departments (Huprich & Rudd;

Procidano et al.; Vacha-Haase), psychology internship program directors (Boxley et al.),

CACREP academic units (Li), and chairpersons from clinically oriented master's programs in

mental health (Olkin & Gaughen). In addition, the manner in which the researchers phrased the

questions varies considerably in these types of studies (Forrest et al.). For example, Olkin and

Gaughen provided a finite list of trainee problems and asked participants to rank the top four,

whereas Huprich and Rudd asked an open ended question and reported only the five most

frequent impairments. Finally, despite some overlap between the studies, several student issues

(adjustment disorder, anxiety problems, intrapersonal problems, lied, marital problems, maturity









issues, misrepresented skills, and social problems) were reported in only one study. This may

make comparison between studies particularly difficult because some specific problems from

one study may be generic issues that are a part of a larger category in other studies (Forrest et

al.). For example, Olkin and Gaughen used the category "intrapersonal" to comprise substance

abuse issues, emotional problems, personality problems, rigidity, and immaturity, whereas

Vacha-Hasse reported substance abuse, emotional problems, and personality issues as separate

categories. The difference in methodologies makes direct comparison between the studies

complicated.

Despite these variations, a review of this literature suggests that some trainee issues are

more common than others. Based upon these studies, it would appear that "clinical skill

deficits," "interpersonal problems," and "supervision difficulties" are consistently identified as

the most prevalent, non-academic, trainee problems. Clinical skill deficits were cited in three of

the six studies (Olkin & Gaughen, 1991; Procidano et al., 1995; Vacha-Hasse, 1995) and were

identified as the most common trainee problem in all three studies (77%, 46%, and 65%

respectively). Similarly, interpersonal problems were identified in three studies (Li, 2001; Olkin

& Gaughen; Vacha-Hasse), and was ranked as the most prevalent trainee problem in one study

(Li; tied with supervision difficulties), the second most prevalent issue in another study (Olkin &

Gaughen,), and the third most prevalent impairment in a third study (Vacha-Hasse,). Finally,

supervision difficulties ranked as the most identified trainee problem in one study (Li; tied with

interpersonal problems) and the second most identified trainee issue in two other studies (Olkin

& Gaughen; Vacha-Hasse). Supervision difficulties were included in three of the six studies.










Table 2-1. Frequencies of impaired trainee characteristics as reported by previous studies (select non-academic variables)
Boxley et al. Olkin & Gaughen Vacha-Hasse Procidano et al. Li Huprich &
Impairment (1986)a (1991)" (1995)' (1995)' (2001)e (2004
Adjustment Disorder 14
Anxiety Problems 25
Clinical Skills Deficit 77 65 46
Depression 31 29 23
Emotional Problems 31 40 34*
Ethical 25* 8
Fatigue or Burnout 19 27
Inappropriate Boundaries 58
Intrapersonal Problems 54
Interpersonal Problems 70 42 73
Lied 42
Marital Problems 27
Maturity 23
Misrepresented Skills 36
Personality Disorders 35 15 34* 31 19
Physical Illness 27 10 40
Refused Counseling 36
Social Problems 19
Substance Disorders 10 15 4
Supervision Difficulties 58 52 73
Unprofessional Behavior 25* 27
*modified/expanded from Forrest, Elman, Gizara, & Vacha-Hasse (1999)
Note. Procidano et al. list "emotional problems" and "personality disorders" as one category. Note. Olkin and Gaughen listed "ethical" issues and rigidity, and "unprofessional behavior" as
category and used "intrapersonal problems" to include substance abuse issues, emotional problems, personality disorders, immaturity. Note. Both Vacha-Hasse and Li are partial lists of trail
impairments. aRespondents were asked to provide information on the cases they felt were representative of impaired trainees. bRespondents ranked their top four choices from a list of seven
possibilities. CRespondents were asked to identify the frequency of current student difficulties. dRespondents were asked to affirm any nonacademic deficiencies within the last five years.
eRespondents were asked to indicate how frequent an impairment indicator was used as a basis for remediation. flndicates the five most commonly reported impairments for internship.









The absence of these issues from some of the studies is not surprising, as the way in which

the authors define trainee deficiencies shaped how the participants responded to the questioning.

Two studies (Boxley et al., 1986; Huprich & Rudd, 2004) did not include any of the three trainee

problems; however, as previously mentioned, Huprich and Rudd focused only on the incidence

of psychological impairments among trainees. Similarly, Boxley et al. defined trainee

impairment as, "any physical, emotional, or educational deficiency that interferes with the

quality of the intern's professional performance, education, or family life" (p.51). Using this

specific definition, clinical skill deficits, interpersonal problems, or supervision difficulties may

not have qualified as an academic issue. Additionally, Procidano et al. (1995) only examined

general categories of trainee issues. The authors simply differentiated between clinical skills,

emotional / personal problems, and ethical issues, thereby condensing the list of trainee issues.

In sum, when clinical skills deficits, interpersonal problems and supervision difficulties

were included in a study, no other trainee issues were more prevalent. These three issues are

directly associated with clinical supervision. Clinical supervisors directly facilitate the trainee's

skill development as well as observe how the trainee interacts with their clients. This finding

only reinforces the importance of proper trainee evaluation during clinical supervision and

suggests the need for comprehensive evaluation tools.

Due Process Procedures

In 1975, Bernard suggested that counseling programs incorporate some general due

process guidelines in order to protect themselves from potential lawsuits and ensure students are

evaluated fairly. He recognized that due process procedures for addressing the trainee's personal

inadequacies were unclear and outlined four principles for proper due process. Since Bernard's

article, several authors (Knoff & Prout, 1985; Lamb et al., 1987; Miller, 1979; Miller & Rickard,

1983) have written and expanded upon the more prominent aspects of due process. The

35









following paragraph includes Bernard's original outline as well as additional contributions

provided by other due process literature.

First, programs should provide students with written guidelines that outline what

professional and personal behaviors are expected (Bernard, 1975; Lamb et al., 1987). These

guidelines should include the specific procedures for student evaluation and a statement

stipulating that students may be terminated for personal shortcomings (Bernard; Knoff and Prout,

1985). Second, faculty members should continually evaluate all students and provide them with

written evaluations (Bernard; Lamb et al.; Knoff & Proutt; Miller, 1979; Miller & Rickard,

1983). Although Bernard suggests that faculty evaluate students at least once a year, other

authors suggest more regular evaluations to ensure that programs identify and address student

impairments within a reasonable timeframe. Third, if a problematic student is identified, the

program should develop a written remediation plan that includes: (a) defining the problem

(Knoff & Proutt; Lamb et al.), (b) identifying expected behavior (Bernard; Knoff & Proutt;

Miller), (c) specifying remediation options (Knoff & Proutt), (d) identifying a reasonable

timetable for completion (Bernard; Knoff & Proutt; Lamb et al), and (e) indicating clear

consequences for failing to complete the remediation (Bernard). Finally, appeal procedures for

students to challenge remediation or dismissal decisions should be in place (Bernard; Knoff &

Proutt). As Knoff and Proutt note, although legal precedent does not require programs to have a

formal appeal process, the hearing "does represent an important due process gesture, which again

demonstrates a faculty's sincere effort to make the termination process fair and open" (p. 796).

In addition to general due process parameters, several authors (Baldo & Softas-Nall, 1997;

Bemak, Epp, & Keys, 1999; Frame & Smith-Stevens, 1995; Lamb, Cochran, & Jackson, 1991;

Lumadue & Duffey, 1999) have described student monitoring models to assist counseling









programs in identifying and remediating problematic trainees. As Vacha-Haase et al. (2005)

state, an increased specificity in evaluation guidelines would alleviate the burden an individual

program may face when deciding what to do with a problematic trainee. Although these models

vary in approach, each one seeks to improve professional gatekeeping procedures by

implementing extensive due process policies and protecting individual faculty members from the

sole responsibility of student remediation.

In 1991, Lamb et al. expanded their previous work on due process by describing a specific

monitoring model. Although originally conceptualized for counseling psychology programs, the

authors contend that this model can be adapted to many academic programs and practicum sites.

This model includes processes related to: (a) reconnaissance and identification of trainee issues,

(b) discussion and consultation, (c) implementation and review procedures, and (d) anticipating

and responding to individual and organizational responses. Within each section, Lamb et al.

describe various strategies and discuss several questions for faculty members to consider during

the remediation process. Noteworthy topics included differentiating between problematic

behaviors and impairments, the importance of early identification and on-going review, and how

a dismissed student can affect the program climate.

Lamb et al. (1991) describe problematic behaviors as, "an intern's behaviors, attitudes, or

characteristics that may require remediation, but are perceived as not unexpected or excessive for

professional training" (p.292). In other words, problematic behaviors are situational deficiencies

that trainees commonly exhibit throughout supervision (e.g. counseling performance anxiety).

Immediately after identifying a problematic behavior, the authors recommend the supervisor

discuss the concerns with other training personnel, promptly develop and implement an

intervention strategy, and allow the trainee an opportunity to address the behavior. Therefore, if









the problematic behavior does become an impairment the documentation verifies the staff s

proactive response and ensures the trainee was made aware of the problematic behaviors before

more drastic measures were taken.

On the other hand, when training personnel are unable to remediate the student's

problematic behavior, these deficiencies may develop into an impairment. According to Lamb et

al. (1991), trainee impairments develop when most of the following occur (p.292):

(a) the intern does not acknowledge, understand, or address the problematic behavior when
it is identified, (b) the problematic behavior is not merely a reflection of a skill deficit that
can be rectified by academic or didactic training, (c) the quality of service delivered by the
intern is consistently negatively affected, (d) the problematic behavior is not restricted to
one area of professional functioning, (e) the problematic behavior has potential for ethical
or legal ramifications if not addressed, (f) a disproportionate amount of attention by
training personal is required, (g) the intern's behavior does not change as a function of
feedback, remediation efforts, of time, and (h) the intern's behavior negatively affects the
public image of the agency.

In situations where initial remediation attempts are unsuccessful, the authors recommend a

systematic procedure whereby relevant personnel meet to discuss the specifics of the student's

behavior and determine whether the behavior constitutes impairment. Lamb et al. recommend a

more formal approach take place if training personnel deem the student impaired. Specifically,

the student is notified, in writing, of the parameters surrounding the impairment as well as

provided several ways in which to remediate the behaviors. The trainee then meets with all

relevant training personnel to address the seriousness of the circumstances. Once the probation

period begins, supervisors continually review the trainee's progress and inform the trainee about

his/her performance. If the program decides to terminate a student, a letter is sent to the student

outlining the probation terms, the student's actions, and the reasons for the dismissal. Finally,

the trainee is given an opportunity to appeal the decision.

Regardless of whether the program successfully remediates or dismisses the student,

Lamb et al. (1991) recognize that this process affects people beyond the trainee. Specifically, the

38









academic institution, other interns, and those specifically involved with identifying the

impairment and implementing the remediation plan, can all be negatively affected. Despite these

concerns, the authors conclude that if training personnel handle the remediation / dismissal

process humanely, this process can benefit both the intern and staff members. For example, an

intern may reexamine his/her professional development and avoid future difficulties.

Concurrently, training staff and other interns may also reexamine their professional behavior

(Lamb et al.).

Lamb et al. (1991) were one of the first to suggest a general model for student

remediation/dismissal procedures. Although this article addresses many essential issues and

raised several important questions, Frame and Stevens-Smith (1995) were one of the first to

evaluate how a specific model affects trainee evaluation. These authors described a three-step

model that was developed and implemented at the University of Colorado at Denver. Before

classes begin, students reviewed the student handbook and signed a statement indicating that

they will abide by all policies. Within the handbook, the faculty provide a policy statement that

expressed, "the faculty's belief in the essential function of 'personal characteristics' in the

development of ethical and competent counselors" (p. 124), and the Personal Characteristics

Evaluation Form (PCEF), an assessment that includes expected trainee characteristics.

Each faculty member uses the PCEF to evaluate every student at the mid-term and end of

the semester (Frame & Stevens-Smith, 1995). If a professor identifies a problematic student, he

or she first discusses the student with the entire faculty and then discusses the evaluation and

remediation (if any) with the student. If a student receives more than one evaluation in a

semester, or receives a form from the same professor in back-to-back semesters, the student must

meet with his/her graduate advisor to discuss additional remediation or possible reconsideration









in the program. Additionally, if a student receives three or more negative evaluations in one

semester, the student will be required to meet with the faculty advisor and two other faculty

members for possible program termination.

One year after implementing these procedures, faculty members and students evaluated the

effectiveness of this model (Frame & Stevens-Smith, 1995). Although only 50% of the faculty

surveyed felt that the evaluation process provided them with a concrete approach to student

evaluation (25% were neutral), counseling educators stated that they had become "more

intentional" about evaluating a student's personal characteristics (Frame & Stevens-Smith,

p.126). In relation to the students surveyed, 82% indicated that they were aware of the

evaluation process. Unfortunately, Frame and Stevens-Smith present no data indicating the

student's thoughts about the evaluation process itself. The authors conclude that by utilizing due

process procedures, programs can protect themselves from unfair student evaluation and more

importantly protect future clients from harm (Frame & Smith-Stevens).

In response to the Frame and Smith-Stevens (1995) article, Baldo et al. (1997) believe that

individual faculty members may be placing themselves into a situation of unnecessary hardship

if they directly evaluate and remediate problematic students. Specifically, faculty members who

identify and remediate problematic students may become the target of that student's aggression.

Therefore, Baldo et al. assert that programs should include the entire faculty during the

evaluation process, thereby protecting individual faculty members from any excessive duress.

In essence, this type of monitoring model allows faculty members, at any time, to discuss

student concerns during regularly scheduled faculty meetings. Once a year, program faculty

members meet to specifically review the student's progress. If the program faculty members

believe the student's progress is unsatisfactory, the student's advisor, not the concerned faculty









member, meets with the student to discuss possible remediation, voluntary resignation, or

dismissal.

If the faculty members do decide that remediation is necessary, the student, the student's

advisor, and a retention committee develop a remediation plan. Once the plan is approved (or

modified) by the program faculty, the student signs the plan and begins the necessary steps to

fulfill the obligations. Near the re-evaluation date, the student will present all necessary

documentation to the advisor and request that his/her current professors evaluate the progress.

This information is then reviewed by the entire faculty and they decide whether the student: (a)

continues in the program without remediation, (b) continues probation / remediation, (c)

voluntary resigns, or (d) is dismissed. If the faculty members decide to dismiss a student, he or

she is notified in writing and has 30 days to present a written request for re-evaluation. Upon

receipt of the request, the faculty will allow the student to present his/her case. Following the

presentation, the faculty members will re-evaluate the student and notify the student of their final

decision. The authors conclude that involving the entire faculty ensures proper due process for

both the faculty members) and student as well as helps prevent the possibility of an individual

faculty member experiencing unnecessary stress.

Similar to the previous models, Bemak, et al. (1999) offer a five-step model that aids

counselor training programs with the remediation and/or dismissal of impaired counseling

students. The authors note that the most taxing concern facing counseling programs is how to

properly evaluate and remediate students who present psychological or personality impairments,

but otherwise perform basic counseling skills competently. Therefore, Bemak et al. suggest that

counseling programs incorporate the idea of psychological adjustment throughout the training

process. Step one requires that the faculty clarify and communicate program expectations before









the admissions process. Like previous models, the authors suggest that in addition to department

standards that address traditional criteria like monitoring and dismissal procedures, an advisor

should identify "sound mental health and personal development as critical aspects of professional

suitability" (p.25). Once the policies are reviewed and agreed upon, step two requires the

students sign a contract indicating their understanding of these procedures.

The third step, identifying trainee concerns, requires that faculty members use

departmental criteria to evaluate a student's development. If a student exhibits any professional

or personal deficits faculty members are to promptly contact the student's advisor.

Subsequently, several department members, including faculty, supervisors, and the Department

Chair, may convene to address the student issuess. Once these faculty members discuss the

concerns, a remediation plan is developed with the student (step four). Finally, faculty must

monitor the student and continually provide feedback regarding his/her progress. If a student is

unable to fulfill the requirements outlined in the remediation plan he or she may appeal

according to the institution's appeal process. Bemak et al. (1999) conclude that this model offers

several advantages to student monitoring including: (a) an ongoing student assessment, (b)

increased faculty communication regarding student impairment, (c) timely feedback to students,

(d) proper student due process, and (e) a clear process that outlines a faculty member's response

to student impairment.

Lumadue and Duffey (1999) discuss the difficulties associated with student evaluation and

the importance of properly implemented due process procedures. To improve upon the

monitoring process, Lumadue and Duffey describe an evaluation instrument, the Professional

Performance Fitness Evaluation (PPFE), which was developed by faculty at Southwest Texas

State University (SWT). Unlike the PCEF (Frame & Smith-Stevens, 1995) that uses abstract









student characteristics, the PPFE utilizes specific student behaviors to evaluate trainee

development. The authors contend that a specific behavioral criterion ensures greater

consistency among evaluators (Lumadue & Duffey).

If a professor identifies an issue, the student and the professor meet to discuss the matter.

Then, if the issue remains unresolved, the professor contacts the department chairperson and the

student meets with a retention committee. The committee reviews the issues and makes one of

three recommendations: (a) the student is fit to continue, (b) the student should undergo

remediation, or (c) the student should be dismissed. If the student chooses to appeal the

committee's decision, he or she must do so by writing a letter to the Department Chair. If the

Chair upholds the committee's decision, then the student can re-appeal to the Dean of the

college. However, if the Dean also upholds the decision, then the student can initiate legal

action.

Although proper due process procedures protect the counseling program and student from

capricious evaluation, a considerable number of programs continue to operate without these

policies and procedures in place. Boxley et al. (1986) found that 66% of psychology internship

programs reported "no adequate or judicious means of assuring the right of due process to interns

who are dropped or deferred from continuing in the program" (p.51). In contrast, Procidano et

al. (1995) found that 74% of doctoral programs in professional psychology reported some policy

for addressing a student's professional deficiency; however, only two-thirds of these policies

were in writing. In 1991, Olkin and Gaughen found that 45% of counseling oriented programs

had no written policies for evaluating problematic students. Of the programs that did have a

policy, over 40% did not generally give these policies to their students. Similarly, Vacha-Hasse

et al. (2004) found that 53% of APA accredited psychology programs reported that they did not









have written policies for problematic students. Furthermore, these authors note that of the

programs that did have a policy only 46% routinely supplied this information to their students.

Finally, although Huprich and Rudd (2004) limited their questioning to psychological

impairment, the authors found that only 58% of doctoral programs reported that they had a

formal procedure to address these types of student issues. On the other hand, 84% of internship

sites reported they had formal procedures in place to address a student's psychological

impairment.

Despite some slight variability between these studies, a substantial number of programs are

reported as not having written policies and procedures in place to address problematic students.

As Vacha-Hasse et al. (2004) notes, the number of programs without these policies are,

"somewhat surprising, given that programs without such guidelines may be vulnerable on several

fronts, including compliance with accreditation requirements and explanation of due process"

(p.118). Interestingly, accreditation bodies continue to allow programs to operate without proper

due process procedures despite recognizing the need for programs to graduate personally and

professionally competent trainees.

Clinical Supervision Evaluation Criteria

Without clearly identifying what skills are essential for clinical supervision, the process

of deciding what skills are appropriate for evaluation is an imperfect process (Bernard &

Goodyear, 2004). Fortunately, educators can examine the standards for their profession, the

general counseling literature, and research to identify what specific skills they want to include

for clinical evaluation. Although accreditation and professional standards literature is limited, it

provides an excellent starting point for understanding the general types of items that should be

included during evaluation. As previously mentioned, the rehabilitation counseling code of

professional ethics suggests that faculty endorse only those students who demonstrate adequate

44









professional ability as well as are personally suitable to provide service to people with

disabilities. In addition, the code outlines the professional standards which identify and define a

rehabilitation counselor (e.g. ensuring client autonomy, A. .d; respecting cultural diversity,

A.2.a; promoting client empowerment, C. 1 .c). Given that the professional standards refer to

three types of criteria (professional skill, personal suitability, and professional standards), it is

perhaps fitting that definitions of impairment as well as clinical supervision instruments also

typically include these general criteria. Although each respective counseling profession may

address general categories of trainee impairment, individual programs are responsible for

developing their own evaluation criteria as well as reviewing the professional literature to

identify specific criteria that would aid in this process.

Perhaps the most prolific source of clinical evaluation criteria comes from reviewing the

general counseling literature. The counseling literature is inundated with various books and

theories (e.g. Chan, Berven, & Thomas, 2004; Corey; 2004; Cromier & Cromier, 1991; Hill,

2004; Miller & Rollnick, 2004) that identify basic and advanced counseling skills that help

facilitate the counseling session. Although there is an exhaustive amount of information to draw

from, books and theories can vary with regards to content and organization. For example,

Cormier and Cormier provide an excellent overview of various skills ranging from basic

listening responses to the use of advanced clinical techniques. On the other hand, Walbom

(1996) opines that the specific counseling skills are less important than the common elements of

counseling. Specifically, he suggests that four process variables are necessary for change: (a) the

therapeutic relationship, (b) cognitive insight, (c) affective experience, and (d) appropriate client

expectations. Therefore, although there are a variety of resources that identify a mixture of

counseling skills, consensus regarding what skills are necessary for counseling varies.









In addition to counseling skills, the trainee's personal attributes are oftentimes considered

necessary for clinical evaluation. Although not all homegrown or published clinical instruments

may explicitly address the trainee's personal suitability, this issue has received increased

attention within counseling training programs. More recently, supervision instruments have

included sections to address the personal suitability of the student. For example, Bernard (1997)

includes a "personalization skills" on an evaluation instrument. This section assesses the

student's self-awareness, interpersonal and intrapersonal depth and flexibility, as well as how

well the trainee works with others. Likewise, the evaluation instrument described by Kerl et al.

(2002) includes a six question section dedicated to the maturity of the student. Despite the

importance given to the trainee's personal suitability, programs may struggle identifying students

with character or psychological issues (Bemak et al., 1999). This may be due to the fact that

personal impairment is usually difficult to identify within the counseling session because it may

manifest as a skill deficiency or other professional impairment. For example, if the trainee is an

abnormally anxious person, he or she may ask only superficial questions. In this instance, the

supervisor may focus on developing the student's use of open-ended questions or other probing

techniques, rather than addressing the trainee's anxiety issue. Therefore, for the purpose of

clinical evaluation, it is important to identify specific behaviors which are indicative of suitable

personal functioning.

A third area less commonly found in clinical supervision criteria are the student's

adherence to professional standards. As mentioned earlier, each respective counseling

profession adheres to specific standards to help define and guide the profession. Within

rehabilitation counseling, the professional standards address a variety of issues including the









principles of ethical behavior, confidentiality issues, as well as underlying professional themes

such as client empowerment (CRCC, 2002).

In sum, establishing evaluation criteria may be challenging and perhaps the most labor-

intensive part of the evaluation process (Bernard & Goodyear, 2004). Typically, clinical

evaluation criteria include counseling skills (basic and advanced skills), personal suitability, and

adherence to professional standards. Although these three areas appear to encompass different

aspects of counselor training, each component can potentially relate with one another.

Supervision Models

In 1995, Maki and Delworth sought to improve a student's therapeutic competence and

enhance client monitoring by describing a clinical supervision model for the rehabilitation

counseling profession. Specifically, the authors developed the Structured Developmental Model

(SDM) by modifying Stoltenberg and Delworth's Integrated Developmental Model (IDM, 1987).

Basically, the IDM catalogs the counselor's progression through four levels of professional

development (Level 1, Level 2, Level 3, and Level 3 Integrated) while simultaneously providing

supervisors with strategies to facilitate the counselor's development. In order to assess the

student's level of development, the supervisor examines the counselor's motivation, autonomy,

and awareness relative to eight competency domains. For example, Stoltenberg and Delworth

(1987) describe a level 1 counselor as highly motivated, highly dependent, and focused strictly

on themselves rather than the client. These characteristics are indicative of a new counselor who

is relying on the supervisor to find the "right" way to counsel a client. As the counselor

develops, he or she becomes more independent and aware of both self and the client.

Additionally, after vacillating between high and low levels of motivation (level 2), the counselor

begins to show more consistent motivation and develops his/her own professional identify.









According to Stoltenberg and Delworth (1987), "vertical development, or movement ahead

to the next level, is explicitly stressed, but the model in practice also attends to issues of

horizontal development across domains" (p.36). In other words, the counselor not only

progresses to higher stages of development, but does so across the domains. Although the

authors believe that counselor development is generally irreversible, they recognize that

counselors may periodically regress to earlier stages of development when faced with unfamiliar

circumstances; however these regressions do not indicate that the counselor has reverted back to

a previous stage (Stoltenberg & Delworth).

Using the IDM as a basis, Maki and Delworth (1995) reorganized the eight competence

domains to develop the SDM. Unlike the IDM which conceptualized counselor development

through eight simultaneous domains, the SDM categorized the domains into one of two groups:

(a) primary domains, and (b) process domains. Primary domains, also known as meta-domains,

include: (a) sensitivity to individual differences, (b) theoretical orientation, and (c) professional

ethics and are continually examined during the counselor's development. The remaining

domains, known as process domains, include: (a) interpersonal assessment, (b) client assessment

within the environment, (c) case conceptualization, (d) treatment goals and plans, and (e)

intervention strategies. These five process domains mirror the conventional service delivery

sequence and are addressed within the context of the primary domains (Maki & Delworth). The

authors contend that reorganizing the competence domains to better emulate the counseling

process will aid in clinical supervision.

Despite the obvious advantages of a clinical supervision model, the debate over the

usefulness of a developmental model, like the IDM, to explain counselor development has

endured (Ellis, 1991; Holloway, 1987; Holloway, 1988; Stoltenberg & Delworth, 1988;









Stoltenberg, McNeil, & Crethar, 1995; Worthington, 1987). Many researchers (e.g. Chagnon &

Russell, 1995; Murray, Portman, & Maki, 2003; Reising & Daniels, 1983; Wiley & Ray, 1986)

began investigating the developmental supervisory process and supported the notion that trainees

do exhibit identifiable characteristics across their development; however these differences were

only evident when differentiating novice trainees from more advanced trainees. In conclusion,

Bernard and Goodyear (2004) caution that although developmental models are intuitively

appealing, they may mislead their supporters away from alternative explanations of supervision.

An alternative to the SDM (Maki & Delworth, 1995) is the Integrative Model of

Supervision in Rehabilitation (Schultz et al., 1999). Recognizing that rehabilitation counselors

work with a variety of populations in different environments, Schultz et al. suggest that any

supervision model within the rehabilitation profession should be flexible enough to

accommodate the differences. Therefore, the Integrative Model was based on rehabilitation

principles rather than a specific personality theory or psychological process. The authors state

that mastering the art of counseling before completing the clinical portion of the program is

improbable; therefore, supervisees should be trained in multiple rehabilitation concepts, but not

necessarily be expected to master every professional skill before graduation. As a result,

supervisors become responsible for not only teaching counseling skills and fostering attitudes

that allow the counselor to function independently, but also ensuring trainees understand that

counselor development is a lifelong process (Schultz et al.).

In order for supervisors to fulfill their obligation, the Integrative Model of Supervision in

Rehabilitation requires the educator be able to shift roles to or from teacher, counselor, and

consultant (Schultz et al., 1999). Specifically, the authors reference Bernard's Discrimination

Model (1997) where the supervisor adopts a different role depending on whether he or she









wishes to focus on the trainee's intervention skills (e.g. empathy, confrontation, etc.),

conceptualization skills (e.g. identify themes, discriminate information, etc.), or

"personalization" skills (e.g. personality, cultural influence, etc.). In addition, the supervisor

assesses the trainee progression through 3 phases of development. Phase I (Technical), focuses

the trainee on developing basic interpersonal skills such as empathy as well as appropriate

attitudes, self-trust, and internal evaluation processes. Once the trainee transitions to Phase II

(Integrative), he or she will focus on more advanced counseling skills such as case

conceptualization, treatment manual utilization, research activities, and explore different service

delivery modes. Finally, as the student moves into Phase III (Consultation), he or she seeks to

continue to develop the ability to self evaluate and acquire new skills. Schultz et al.

acknowledge that transitioning from Phase I to Phase II is crucial and should not necessarily

coincide with the student progressing from practicum to internship. Although the authors note

that prematurely transitioning a student to an advance stage may result in the counselor being

"overwhelmed," while delaying this transition for an able student will result in "apathy" or

"frustration," the practicum and internship timelines are reasonable (p.329).

In sum, both the SDM and Integrative Model offer an excellent resource for rehabilitation

counseling educators and students. Specifically, these models help organize the supervision

process and provide supervisors a means to understanding their role(s) in facilitating trainee

development. Despite these advantages, both models are limited in terms of trainee evaluation.

Specifically, these supervision models only describe a theoretical pattern of trainee development,

rather than specifying evaluation guidelines. Although the development of trainees cannot be

forced, supervisors are still faced with the reality of whether or not the trainee has earned a









passing grade or has demonstrated some minimum level of competency worthy of professional

endorsement.

Additionally, each supervision model provides little empirical evidence pertaining to the

actual sequence of counseling skill development. Without a thorough understanding of skill and

characteristic development, these models are limited by offering only a description of trainee

development. Although a development sequence of counseling skill and attitudes may appear

intuitive, empirical evidence outlining a clearer understanding of skill development will ensure

the appropriate skills are being required of novice and advanced students.

Moreover, trainees do not necessarily develop basic skills before advanced counseling

skills. Focusing novice counselors on "basic" counseling skills may be intuitively appealing,

however students will come into the supervision process with varying levels of ability in all

aspects of counseling, particularly counseling skills, personal suitability, and professionalism.

Therefore, supervisors would be better suited to identify several factors (i.e. interpersonal skills,

integrating theory, case conceptualization skills) and simultaneously evaluate the trainee. As a

result, supervisors will be able to better evaluate the trainee and meet the trainee's need for

supervision.

Clinical Supervision Instruments

Bernard and Goodyear (2004, p.28) state that, "there are as many evaluation instruments as

there are training programs in the helping professions." Because the professional literature

provides only general evaluation guidelines, counseling programs oftentimes rely on homegrown

instruments to evaluate trainee performance. As Eriksen & McAuliffe (2004) point out, many

evaluation instruments have not been subjected to rigorous validity or reliability checks, and may

require the evaluator to provide only general impressions of student performance. Moreover, the

authors identify several other shortcomings common to most evaluation instruments. First,

51









although expert opinion appears to be the most accurate indicator of trainee competence, some

instruments rely solely on client feedback to determine the student's ability level. Additionally,

the authors note that many evaluation instruments require the supervisor to simply record how

often the student demonstrates a skill, rather than how effective he or she is in utilizing the skill

during a counseling session. Finally, Eriksen and McAuliffe note that many supervision

instruments simply use dichotomous scales when evaluating a trainee, thus hindering the

precision of specific and meaningful feedback.

In addition to homegrown instruments, there are also a number of published evaluation

instruments. Unfortunately, they are highly variable regarding structure, types of items, and

psychometric information. Several authors (Eriksen & McAuliffe, 2003; Kerl et al., 2002;

Myrick & Kelly, 1971; Getting & Michaels, 1982; Robiner et al., 1994) have described various

published instruments that help counseling faculty members evaluate a student's counseling

performance. The following is a brief overview of several supervision instruments developed

within the last 35 years. One of the earliest published supervision instruments, the Counselor

Evaluation Rating Scale (CERS), was developed by Myrick and Kelly (1971) to address the need

for an instrument which could help supervisors evaluate a trainee's clinical performance. After

reviewing the literature, the authors selected 27 items to represent three areas of student

development: (a) the understanding of the counseling rationale, (b) counseling practice with

clients, and (c) exploration of self and the counseling relationship. These items were then

organized into a counseling performance domain (13 items), a supervision domain (13 items),

and one global item regarding the supervisor's overall recommendation. Each item is rated on a

7-point Likert type scale. Once the scores from the counseling and supervision domains are

totaled with the global recommendation, the composite score represents the trainee's overall









counseling effectiveness. Myrick and Kelly report a split-half correlation coefficient of .95.

Additionally, when correlating the counseling and supervision domain together, the authors

reported a coefficient of .86.

In 1983, Getting and Michaels developed a unique supervision tool, the Oetting/Michaels

Anchored Rating for Therapists (OMART), to help supervisors develop comprehensive

evaluations of trainee performance, and to serve as a communication device for supervisors and

students. The OMART consists of 34 counseling related items (i.e. relationship with client,

exploring issues, exploring feelings). The authors then described various types of trainee

behaviors that represent different ability levels for each item. In other words, each counseling

item had a corresponding hierarchy of behaviors that reflect various levels of trainee

development from needing remediation to professional therapist. Unfortunately, Getting and

Michaels provide no psychometric validity or reliability information. Additionally, because of

the OMART's size, Bernard and Goodyear (1992) suggest that some may view the instrument as

too lengthy.

Unlike the OMART (Oetting & Michaels, 1983), the Minnesota Supervisory Inventory

(MSI, Robiner et al., 1994) has undergone a more extensive validation and reliability testing than

most instruments. The MSI includes 112 items and a 3-point rating scale. The items represent

seven trainee performance areas: (a) assessment, (b) psychotherapy and intervention, (c)

consultation, (d) professional and ethical behavior, (e) supervision, (f) case

conference/professional presentations, and (g) site-specific functioning. When developing the

MSI, Robiner et al. reviewed the feedback from other University of Minnesota Psychology

Internship Consortium (UMPIC) supervisors, examined the job analysis of psychologists, and

appraised instruments from other internships. Despite its comprehensiveness, Robiner et al. note









that the MSI focuses on the trainee's skills and professionalism rather than personal

characteristics. The authors report adequate internal consistency within each of the seven

domains (rs=.61 to rs=.83). With regards to inter-rater reliability, Robiner et al. note that the

findings were lower than expected (rs=-.31 to rs=.28), however test-retest reliability after one

week ranged between .91 and 1.00. In addition to psychometric scrutiny, Robiner et al. surveyed

internship supervisors to compare their current evaluation instrument with the MSI.

Interestingly, the authors note that general reactions to the MSI were favorable, but interest in

using the instrument varied. Unfortunately, the authors provide no further explanation as to this

finding.

A fourth instrument, the Professional Counseling Performance Evaluation (PCPE, Kerl. et

al., 2002), was developed by faculty at Southwest Texas University to aid in due process

procedures. Originally referred to as the Professional Performance Fitness Evaluation (PPFE,

Lumadue & Duffey, 1999), the PCPE helps faculty members evaluate students and provides

them with feedback on several areas of professional and personal development. Although the

PCPE was not designed specifically for supervision, Kerl et al. note that the instrument is

appropriate for every course. Moreover, the instrument includes items related to counseling

performance. In all, the instrument contains 38 items arranged in five competency areas: (a)

counseling skills, (b) professional responsibility, (c) competence, (d) maturity, and (e) integrity.

The evaluator identifies whether the student consistently meets, minimally meets, or does not

meet the listed criteria. Although the PCPE contains areas of professional and personal

development, the instrument has not been psychometrically validated.

Finally, in an effort to develop a validated instrument for the counselor education field,

Eriksen and McAuliffe (2003) modified the Skilled Counseling Scale ([SCS] Urbani et al., 2002)









to create the Counseling Skills Scale (CSS). To improve the validity and reliability of the

instrument, the CSS went through an immense transformation that included: (a) addressing item

and rating scale issues, (b) establishing content and face validity by soliciting feedback from

relevant professors, and (c) pilot testing the CSS to obtain inter-rater reliability and construct

validity. The final instrument contains 22 items that represent six areas of counseling

performance. The rater evaluates the student on each item along a 5-point Likert type scale that

ranges from "well developed" to "major adjustment needed." To score the CSS, the evaluator

averages only those items that were used by the trainee during the session. Once each subscale is

averaged, the student's counseling competence is determined by totaling the six subscales

together. Using only 29 participants, the authors report a Cronbach's alpha of .90 and significant

pre- and postcourse change (construct validity) in the total score as well as on five of the six

subscales. Regarding item analysis, Eriksen and McAuliffe concluded that 11 items actually

correlated higher with another subscale than with its own, leading the authors to conclude that

the scales do not represent true factors.

In conclusion, the development of efficient and precise clinical supervision evaluation

tools remains a challenge for various psychology and counseling professions. Although several

clinical supervision instruments have been published, the majority of instruments contain a

limited amount of psychometric information or rely on summed scores to indicate the trainee's

clinical performance. The usefulness of summed scores may be minimal because every

counseling skills is considered to all contribute to the trainee development equally. However,

particular skills or items are undoubtedly more or less difficult for a trainee to develop.

Identifying a clear item hierarchy that outlines the trainee's skill development will provide a

more useful indicator of counseling performance. However, identifying what skills a trainee is









most likely able to perform in the early stages of clinical training compared to the latter stages is

complicated. Intuitively, certain basic counseling skills (i.e. simple reflection) are easier to

perform than other more advanced skills (i.e. positive confrontation), but for the vast majority of

counseling skills, educators may disagree over a clear item hierarchy of skill development. This

variability poses a practical challenge for educators when they develop their standards for

clinical evaluation.

In order to address these and other evaluation shortcomings, a clearer understanding of

trainee skill development must be understood empirically. Rather than continuing to rely on a

predominately theoretical understanding of trainee skill and attitude development, researchers

might be better served to study the typical progression of these concepts. Once a better

understanding of skill and attitude is empirically achieved, counseling educators might be better

able to establish and enforce a minimum standard of ability relative to specific timetables.

Undoubtedly, trainees will develop at various rates, however students who have been endorsed

by a counseling program should have acquired a minimum level of skill and personal suitability

in order to practice in the community. Therefore, it is imperative that the rehabilitation

counseling profession identify specific counseling skills and professional behaviors

representative of competent rehabilitation counseling trainees.

Additionally, the empirical understanding of counselor development not only ensures that

minimum criteria standards are accurate, but also allows supervisors and students to catalog their

development independent of practicum or internship. Therefore, educators can identify realistic

expectations that are just beyond the trainees' current ability, as well as pinpoint specific

interventions. In order to better understand trainee development and improve trainee evaluation,

researchers must rely on innovative research methods to construct instruments. On such method,









Rasch analysis, can address these and other measurement challenges associated traditional data

analysis techniques.

The Rasch Model

Rasch theory is based on a logistic model within Item Response Theory (IRT) that uses

probabilities and item difficulty calibrations to measure the person's ability level on some trait.

Unlike classical test theorists who assume that the ordinal data is interval, Rasch analysts believe

that in order to satisfy the fundamental rules of measurement, these raw scores must first be

converted into an actual interval scale (Rasch, 1960; Wright, 1997; Wright & Linacre, 1989).

This conversion is necessary because according to Wright and Linacre, a "score" on a test is

nothing more than the frequency count of events, but is oftentimes mistakenly treated as a

measure of ability. For example, a person who correctly answers more math items on a test than

another test taker would be considered to have more math ability. However, simply counting a

correct response as "one more," erroneously implies that every math item is an equal unit of

measurement. According to Wright and Linacre (1989), "the events counted are specific rather

than general, concrete rather than abstract, and thus varying rather than uniform in their import"

(p.858). In other words, certain items are indicative of more or less of an ability. Therefore,

"one more" can imply a different increment, and thus raw counts are insensitive to this detail

(Wright & Linacre).

In order to measure the ability of people, Rasch analysts suggest that measures should

mirror the instruments found in the physical sciences (e.g. ruler, scale, time). Specifically, raw,

ordinal data must first be converted into a linear, interval scale to allow the researcher to

examine the psychometric properties of specific items rather than the instrument as a whole. In

order to achieve equidistant scaling, Rasch experts rely on an alternative paradigm for analysis.

Unlike Classical Test Theory (CTT) and other IRT models, Rasch experts believe that data

57









obtained should not dictate the measurement model, but rather the measurement model should

first be established and then use data that conform to the model (Wright, 1997). As Wright

explains, a model should not imitate or accept any kind of data, but should be constructed to

define measurement.

The Rasch equation in its simplest form is as follows:

Log [Pni / 1 Pni] = Bn Di

where

Pni = the probability of person "n" passing an item

1 Pni = the probability of person "n" failing an item

Bn = ability of person "n"

Di = difficulty of item "i"

The left side of the equation represents the transformation of raw scores into an interval

measure whereby the probability of a person passing an item is divided by the probability of a

person failing an item. Conversely, the right side of the equation represents the various facets of

the measurement model. In its simplest form, the Rasch equation for dichotomous data utilizes

two facets: the ability of the person (Bn) minus the difficulty of the item (Di).

As Bond and Fox (2001) point out, "taking the natural log of the odds of successfully

passing an item results in the direct comparison between a person's ability and an item's

difficulty" (pp.202-203). Thus, by arranging people and items on the same linear continuum, the

Rasch model has generated parameter separation whereby the creation of a measure estimates a

person's ability independent of the specific test items and similarly estimates the difficulty of

items independent of the specific sample (Bond & Fox, 2001). In other words, the Rasch model

has created a measure that is both sample-free and test-free. Originally conceptualized by

Thurstone (1927, 1928), the concept of sample-free measurement refers to the idea that a









measurement scale should transcend any particular sample (Figure 2-1), just as a large ruler can

measure various people no matter what their height (Wright; 1997). On the other hand, test-free

refers to the ability to omit several questions from different levels of the measurement without

affecting the person's score (See Figure 2-1; Wright, 1997). This would be similar to measuring

the height of an adult who is approximately six foot without having to first determine whether he

or she exceeds the one, two, three, four, or five foot mark. In essence, requiring subjects to

answer an entire set of questions is unnecessary when determining their ability.



Measurement of a Measurement of a
variable should be variable should
f independent of the not require every
sample. test item.



_L 0L






Sample Free Test Free

Figure 2-1. Sample free and test free related to height

When using the Rasch formula, the most information about a person's ability level is

obtained when the person has a 50 percent probability of passing an item (Smith, 1994). If an

item is well below the person's ability, the probability of him or her correctly answering it

increases (i.e. 90 percent). Conversely, if the item is too difficult, then the person has a much

lower probability of passing the item (i.e. 10 percent). The most information about people is

obtained when one can match the types of items associated with their ability level.









By converting ordinal data into an interval scale, the Rasch model can examine the

psychometric properties of the individual items. One such property, an item / person measure,

arranges the items on a hierarchy from least to most difficult while simultaneously determining

the subject's ability level based on the types of questions he or she correctly answers.

Conceptualizing a person's ability level on a hierarchy, as opposed to relying on the summation

of correct answers, can provide clinical supervisors more useful information about the student

because the person's ability is attached to specific items or behaviors related to whatever

constructs one is measuring. A supervisor can then catalog what skills the student has the

capacity to demonstrate verses the skills that the student has mastered or has yet to demonstrate

at all. In other words, the quantitative nature of measurement would be understood more

qualitatively. The student score would finally be associated with specific counseling behaviors

and allow for a clearer organization of training and development. This information would be

useful for measuring change in the student's performance as well as improving communication

between the supervisor and the student. During clinical work, the supervisor would not only be

able to identify their students' counseling abilities by the types of items they can do, but also

target specific behavioral interventions that were not too easy nor too hard for the student.

Furthermore, because the Rasch hierarchy could determine a person's counseling ability

without requiring him or her to demonstrate every skill or behavior, the Rasch measurement

model is a useful tool for clinical supervision. Depending on the nature of the practicum or

internship site, students may not need to demonstrate every counseling skill or behavior.

However, a traditional instrument that relies on total scores may unfairly penalize a trainee for

not demonstrating a particular skill, even if the skill was unnecessary or did not affect his/her

overall counseling ability. Figure 2 illustrates that regardless of what types of skills









demonstrated, the trainee's ability is more accurately determined by considering the difficulty of

the skill. It is interesting to note that when using the total score method, persons B is considered

to have more counseling ability despite being unable to adequately perform more difficult skills.

Rasch Measurement Total Score

A B C
AA
Confrontation Confrontation NA 1 NA

Interpretation Interpretation 4 2 2
C Person A
Identify Themes Identify Themes 4 2 2

Paraphrase Paraphrase 4 4 3
C Person B
Clarify Clarify NA 4 1

Reflection Reflection 5 5 4
< Person C
Summarize Summarize 4 5 4

Open Questions Open-Ended Ques 5 4 4

Closed Questions Closed-Ended Ques 5 5 4

TOTAL SCORE 31 32 28

Figure 2-2. Hierarchy of counseling ability compared to total score

Finally, a minimum level of performance related to clinical skills, professional standards,

and personal suitability could be established to ensure that rehabilitation counseling programs are

endorsing only those students who have met these requirements. The development of several

hierarchies related to rehabilitation counseling performance will clearly organize what behaviors

and abilities are expected from counseling trainees. With this understanding, clinical supervisors









and trainees are better equipped to promote the trainee's professional and personal development

as a rehabilitation counselor.









CHAPTER 3
METHODOLOGY

Within the rehabilitation counseling profession, little is known about what competency

criteria are appropriate for a clinical supervision evaluation measure. Therefore, the purpose of

this study was to establish expert consensus regarding what specific competency indicators were

useful for clinical assessment. This researcher employed the Delphi research method to identify

these competency criteria. Specifically, participants partook in a three round Delphi procedure

designed to establish consensus among rehabilitation educators regarding the appropriateness of

specific performance indicators for clinical evaluation.

Research Question 1

What professional, personal, and clinical competency items are useful for the evaluation of

rehabilitation counseling trainees?

Study Design

To answer this research question, participants participated in a three round Delphi study to

identify the critical items necessary for clinical supervision evaluation. Originally developed by

the RAND Corporation for determining the likely targets of Soviet nuclear attacks, the Delphi

technique has become a widely used method for measuring, predicting, and decision making in

an array of disciplines (Rowe & Wright, 1999). The Delphi procedure allows researchers to

elicit and converge anonymous expert opinion by providing systematic feedback through a series

of questionnaires (Vazquez-Ramos, Leahy, & Hernandez, 2007). The first questionnaire asks an

open-ended questions) regarding the topic of interest. Once participants complete this

questionnaire, the researcher combines the panel's responses and creates the second round of

questions. In Round 2, participants are asked to rate the importance of all the Round 1 responses

on a Likert-type scale. After this information is returned and tabulated, the researcher develops









the Round 3 questionnaire. During this round, participants are provided their rating responses as

well as the statistical group data from Round 2 and allowed the opportunity to compare and

revise their previous responses. Ultimately, the goal of the Delphi procedure is to achieve

overall consensus or level of agreement among experts (Williams & Webb, 1994). This

technique builds on the expert's qualitative responses, while measuring the group's responses

quantitatively (McBride, Pates, Ramadan, & McGowan, 2003).

The Delphi methodology offers several advantages over other comparable techniques.

First, because the experts participating in this study are anonymous, any one member of the

panel can not overly influence the responses of other participants. According to Williams and

Webb (1994), this method encourages participants to offer their honest opinion because not only

are they uninhibited when responding, but junior participants are free to challenge more senior

participants anonymously. In general, this method eliminates any confrontation issues that may

be associated with similar techniques (Vazquez-Ramos et al., 2007). Additionally, because the

Delphi technique offers feedback in successive rounds, participant views can be "retracted,

altered, or added with the benefit of considered thought" (Williams & Webb, p. 181). Thus,

panel members have the opportunity to rethink and reflect on the feedback from other panel

members before submitting their final opinion. According to Rowe and Wright (1999), the

multiple iterations of the Delphi method may allow panel members to change their judgments

while saving face in the eyes of the other group members. A third advantage of this

methodology is the statistical nature of the process. Panel members receive statistical feedback

(i.e. mean, median, standard deviation, interquartile range) in between rounds, but final

consensus is determined using preset statistical criteria (Hakim & Weinblatt, 1993; Rowe &

Wright). The use of statistical feedback and preset standards for defining statistical consensus









ensures that researchers are not arbitrarily determining expert consensus based on personal

impressions. Finally, the Delphi approach allows researchers to collect data from a diverse panel

in terms of geographic location, experience, gender, and education. Particularly when developed

for online distribution, this type of survey is easily accessible to experts throughout the country.

Sampling

According to Williams and Webb (1994), determining the appropriate sample size for a

Delphi study is unclear; there are no established guidelines regarding appropriate sample size.

Delphi studies have typically included as few as three participants and as many as 80 participants

(Rowe & Wright, 1999). This trend is also evident within the rehabilitation counseling

profession where studies have ranged from 18 panel members (Thielsen & Leahy, 2001) to 111

panel members (Hakim & Weinblatt, 1993). Additional rehabilitation counseling Delphi studies

have included round 1 sample sizes of 23 (Rubin, McMahon, Chan, & Kamnetz, 1998), 44

(Currier, Chan, Berven, Habeck, & Taylor, 2001), and 31 (Shaw, Leahy, Chan, & Catalano,

2006). Although there is no generally accepted numbers of participants, Okoli & Pawlowski

(2004) suggest that a Delphi panel size is typically modest, ranging between 10-18 experts.

In addition to the ambiguity over the appropriate number of participants, Hasson, Keeney,

and McKenna (2000) report that the controversy over a Delphi sample size extends to the

difficulty in defining who qualifies as an "expert". In fact, few studies report any specific

criteria used to differentiate experts from non-experts (Williams & Webb, 1994). Within the

rehabilitation counseling literature, there are no specific definitions differentiating rehabilitation

counseling clinical supervision experts from non-experts. Therefore, this researcher defined a

rehabilitation counseling supervision expert as:

* Have supervised at least 15 rehabilitation counseling students during practicum or
internship,









* Have supervised at least three semesters of practicum or internship, and

* Have supervised at least three students during practicum or internship within the last three
years.

Further complicating the sample size issue is the fact that the Delphi procedure includes

several rounds, therefore researchers must account for the potential decline in response rate

(Hakim & Weinblatt, 1993). In fact, several of the rehabilitation counseling Delphi studies

reported a decline in response rate from one round to the next. Therefore in order to account for

a reduced response rate and ensure the number of panel experts in the final round remains at an

acceptable level, this researcher targeted a minimum of 23 participants for the Delphi study.

Potential participants were contacted through the National Council of Rehabilitation

Education (NCRE) listserv. The NCRE is the premier professional organization representing

rehabilitation educators who are dedicated to improving the lives of people with disabilities

through education and research (NCRE, 2007). After NCRE research committee approval, the

survey was made available to participants online. Collecting data through a Web-based survey

has several advantages including a reduced response time, lowered costs, ease of data entry, and

format flexibility (Granello & Wheaton, 2004).

The SurveyMonkey software program was used to distribute the questionnaire.

SurveyMonkey allows the user to collect and analyze data through a wide range of export and

statistical analysis functions (SurveyMonkey, 2007). In addition, this program provides a means

for immediately obtaining participant data through the internet as well as storing information on

a secure server.

Delphi Procedure

This investigator brainstormed with rehabilitation counseling educators to develop a

question strategy to identify what behaviors are appropriate/useful for the evaluation of trainees









during the clinical portion of their program. This open-ended question seeks to elicit trainee

behaviors that are indicative of rehabilitation counseling ability. Specifically, the question asks

participants to list behaviors that are reflective of three rehabilitation counseling trainees' areas:

(a) personal suitableness, (b) professional appropriateness, and (c) clinical ability (Appendix A).

Delphi round 1

As previously described, potential subjects were contacted via an email (Appendix B)

through the NCRE listserv. Prior to beginning the study, rehabilitation counseling educators

were notified of the purpose of the study, the general study procedure, their rights as participants,

potential risks and benefits of participating, and contact information of the primary investigator.

Subjects agreeing to participate in the study confirmed their understanding of the informed

consent information electronically (Appendix C). If a participant failed to consent their rights,

the survey closed and the participant did not see the questionnaire. At the end of the survey, the

respondents were instructed to submit their responses. A friendly reminder email was sent to the

panel experts two weeks following initial contact (Appendix D).

Once the data was received, this researcher engaged in the process of microanalysis. This

process involves a creative, unstructured analysis of the data whereby the researcher generates

initial concepts and deeper meaning by examining the data line-by-line (Strauss & Corbin,

1998). Throughout the analysis process, the researcher utilized analytic tools such as

questioning and comparative analysis in order to identify separate, substantially distinct items.

As new data were observed, this researcher constantly compared all of the data to look for

similarities and differences among the items (Corbin & Strauss, 1990). This process requires the

researcher to engage in several iterations of data analysis whereby incoming data are compared

to existing data.









Although the Delphi procedure is not typically considered a qualitative methodology,

certain elements of this process are conceptually similar to this type of data analysis process (e.g.

constantly comparing data). Therefore, in order to ensure a level of trustworthiness and

credibility in the data analysis process, this researcher has included an "audit trail" outlining the

round 1 data analysis (Appendix E). Specifically, the audit trail captures the researcher's

decision making process when grouping conceptually similar items. For example, conceptually

similar items such as "paraphrasing," "know how and when to paraphrase," and "ability to

paraphrase" were condensed into the item "paraphrase client statements." In this instance, the

researcher identified each item as being related to one's ability to paraphrase and therefore

combined the items.

On the other hand, items that appeared conceptually similar but determined to be

substantially different were separated into two distinct items. For example, the item "confidence

in approaching any interpersonal session with clients, site supervisor, or faculty supervisor" was

separated into "confident in approaching any interpersonal session with clients" and "confident

in approaching any interpersonal session with site or faculty supervisors." In this instance, the

researcher deemed the interpersonal interactions between the trainee/client and the

trainee/supervisor to be substantially distinct items. In other words, the level of confidence the

trainee exhibits during a counseling session was determined to be independent of the level of

confidence the trainee might exhibit with a supervisor. As a result, this one item was

conceptualized as two discrete items and separated accordingly.

Once the text was reviewed and some concepts were identified, this researcher then

categorized the data into theoretical competency domains. According to Strauss and Corbin

(1998), the analyst eventually realizes that previously conceptualized data can be grouped into









higher, more abstract concepts know as categories. The purpose of categorizing data are

twofold. First, it allows the researcher to work with a reduced number of data units which makes

the data analysis process more efficient. Second, by creating categories, the data now has the

potential to explain and predict (Strauss & Corbin).

Since the purpose of this research question is to generate items and competency domains to

be included on a rehabilitation counseling supervision measure, specific behaviors related to the

trainee's clinical ability, personal suitability, and professional adherence were identified and

used to develop the items for Round 2 of the Delphi Study.

Delphi round 2

Following the same methodology used in the original questionnaire, an email was sent to

all panel experts directing them to the Round 2 survey (Appendix F). Round 2 of the Delphi

study consisted of a series of closed-ended questions in which participants were asked to rate the

usefulness of each item generated from round 1. Within the instructions for round 2, the term

"useful" was further defined as any item related to rehabilitation counseling performance that

might help clinical supervisors evaluate a trainee (Appendix G). Participants rated each item on

a 5-point Likert-type scale ranging from 1 (strongly disagree) to 5 (strongly agree). In addition

to the item ratings, participants had the opportunity to respond with specific comments regarding

the items in Round 2. Like the previous round, a friendly reminder email was sent to the experts

one week following initial contact (Appendix H).

Once the Round 2 questionnaires were received, the data was analyzed in order to provide

each panel expert feedback about the group norms. Typically, researchers conducting Delphi

studies prepare a variety of information for each panel member including: (a) feedback about

their own ratings, (b) a statistical analysis of the group data, and (c) a summation of the

comments provided in Round 2 (Vazquez-Ramos et al., 2007). In order to provide appropriate

69









feedback, each item was analyzed using the Statistical Package for the Social Sciences software

(SPSS Inc., 2001) and a summary of participant comments was developed.

Although there are no specific rules regarding what specific statistics should be calculated,

Rowe and Wright (1999) state that researchers typically report at least one measure of central

tendency (i.e. mean, median) and one measure of dispersion (i.e. standard deviation, interquartile

range). For the purposes of this study, this researcher calculated and reported the mean and

standard deviation for each item. According to Cramer and Howitt (2004), the arithmetic mean

is the numeric average of scores and may or may not be the most common score. Although the

mean may be heavily influenced by outlying data, participants in this study are rating items on a

5 point Likert-type scale and therefore the undue influence of outliers on the calculation of the

mean will be minimal. In addition to the mean, this researcher calculated and reported the

standard deviation. The standard deviation is a common measure of variability appropriate for

interval data (Norman & Streiner, 2000). Specifically, the standard deviation is the square root

of the average of the squared deviations of each number from the mean of all the numbers

(Norman & Streiner, 2000). The closer the item ratings cluster around the mean (e.g. the smaller

the standard deviation gets), the smaller the variability and thus the larger the consensus.

Delphi round 3

Once the data from round two was calculated, the third round of the questionnaire was

developed and sent to the expert panel. This questionnaire consisted of the participant's original

item ratings along with group statistical norms (i.e. mean, standard deviation) and qualitative

feedback. Following the same methodology for rounds 1 and 2, an email was sent to all the

experts directing them to the round 3 survey (Appendix I). Round 3 instructions (Appendix J)

specifically asked participants to examine the group statistical data, consider the participant

comments, and re-evaluate their original item ratings. More specifically, the participants were

70









asked to re-rate only those items they decided to change. Similar to previous rounds, a friendly

reminder email was sent to the experts two weeks following initial contact (Appendix K).

Once all the data was collected and item statistics were calculated, this researcher

determined what items met the criteria for group consensus. Currently, there are no universal

standards for establishing consensus (Fink, Kosekcoff, Chassin, & Brook, 1984; Williams &

Webb, 1994). In fact, many researchers may not set a level of consensus prior to the study,

resulting in an arbitrary level of agreement (Webb & Williams). According to Webb and

Williams, "unless a [consensus] value is stipulated, the notion of a 'high' level of consensus

could almost be a movable feat which is unilaterally decided upon by the researcher" (p.183-

184). Hakim and Weinblatt (1993) suggest that in order to determine consensus

For the purposes of this study, the standards for "high" expert consensus were defined

according to two criteria:

* The mean difference between the second and third rounds cannot exceed plus or minus 1
(stability), and

* The standard deviation for each item after round 3 will be equal to or less than 1
(convergence)

In addition, for the purposes of this study, the standards for "moderate" expert consensus

were defined according to two criteria:

* The mean difference between the second and third rounds cannot exceed plus or minus 1
(stability), and

* The standard deviation for each item after round 3 will be equal to or less than 1.5
(convergence)

Pilot Testing

Once the initial Delphi question was developed and placed online, five rehabilitation

counseling educators associated with two universities were requested to complete the









questionnaire and provide any additional feedback regarding the survey. A summary of the

participant feedback, test findings, and actions taken are provided in Table 3-1.

Table 3-1. Pilot test findings and revisions
Comments Action Taken
Provide an comprehensive list of each Each school was listed with its corresponding
rehabilitation counseling school with its CORE region
associated CORE region
Delphi Round 1 question did not extract an Delphi was reworded to elicit more
exhaustive list of trainee clinical, personal, or comprehensive responses and the introduction
professional behaviors was reworded to encourage participants to
provide an exhaustive list of competency items

The first comment was in reference to the demographics page of the survey, in which

participants are asked to identify what CORE region their program is affiliated. The comment

suggested that many participants may erroneously select the wrong region or may not be aware

of their school's specific designation. Moreover, the questionnaire could be improved if such a

designation were clearer. To address this matter, this researcher reorganized the region options

designated in the survey. Rather than just listing the region, each school affiliated with CORE

was added to the survey under each specified region.

Next, in relation to the Delphi question, it was revealed that respondents did not provide

an exhaustive, detailed list of trainee clinical skills, professional attributes, or personal attributes

relative for a clinical supervision assessment tool. Responses only ranged from 3-8 items and

included broader items rather than specific skills or behaviors. To address this issue, this

researcher carefully rewrote the question to help invoke participant responses. Specifically, this

researcher added examples representative of the three evaluative areas, as well as added a prompt

to help supervisors better conceptualize the types of behaviors and skills useful for clinical

supervision evaluation. Additionally, this researcher reworded the introduction to emphasize the









nature of this study and encourage participants to provide a comprehensive list of trainee

competency items.

In sum, this study is designed to use qualitative methods to explore the inherent

challenges associated with student impairment and to identify specific performance indicators

useful for clinical assessment. Once this research study is completed, it is hoped that

rehabilitation counselors can more holistically conceptualize trainee competence during clinical

supervision. Furthermore, it is anticipated that the clinical supervisors could use the findings

from this research to help develop comprehensive measures of clinical competence (e.g. a Rasch

validated clinical supervision measure) that help establish clear performance expectations and

allow for more transparent evaluation.









CHAPTER 4
RESULTS

Delphi Study

Panel Demographics

Of the rehabilitation counseling educators solicited for this study, 21 participated in

Round 1. Participants averaged 14 semesters of practicum/internship supervision experience and

had supervised an average of 59 students throughout their career (Note: when participants

estimated the number of supervised students and/or semesters with a "+," this researcher used the

most conservative estimate for the calculation; one participant did not offer a numeric value

when asked the number of students supervised and was therefore not included in the calculation).

Additional information describing the participants is described in Table 4-1.

Participants held a variety of academic positions including professor emeritus (4.8%), full

professor (14.3%), associate professor (23.8%), assistant professor/tenured track (42.9%), and

assistant professor/non-tenured track (14.3%). Moreover, participants represented several CORE

academic regions. Specifically, participants were from regions I (14.3%), III (4.8%), IV

(47.6%), V (9.5%), VI (14.3%), VII (4.8%), and X (4.8). Regions not represented by the

participants included regions II, VIII, and IX. In addition to supervision experience and location,

participants held a variety of licenses and credentials. As outlined in Table 4-2, the majority of

participants were (9.5%) respectfully. Furthermore, the sample was comprised of a certified

disability management specialist, a certified psychiatric rehabilitation practitioner (pending), a

licensed clinical psychologist, licensed independent social worker, an occupational therapist

registered/licensed, and a registered mental health counselor intern.









Table 4-1. Participant demographics
Frequency
Factor Number %
Position
Professor Emeritus 1 4.8
Full Professor 3 14.3
Associate Professor 5 23.8
Assistant Professor/Tenured Track 9 42.9
Assistant Professor/Non-Tenured Track 3 14.3

Region
I 3 14.3
II 0 0
II 1 4.8
IV 10 47.6
V 2 9.5
VI 3 14.3
VII 1 4.8
VIII 0 0
IX 0 0
X 1 4.8

Delphi Round 1

Experts responded to Round 1 of the Delphi with 188 substantially different items to

consider when evaluating a rehabilitation counseling trainee during clinical supervision (Tables

4-3 through 4-7). These responses were based on an open-ended question which prompted

participants to think about their students who demonstrated either exceptional or poor counseling

ability and then list any specific counseling skills, characteristics, abilities, or attributes they

believed were useful or appropriate for evaluating trainees during clinical supervision.









Table 4-2. Participant credentials
Frequency
Credential Abbreviation Number (%)
American Board of Vocational Experts ABVE 1 4.8
Certified Disability Management Specialist CDMS 1 4.8
Certified Life Care Planner CLCP 2 9.5
Certified Psychiatric Rehabilitation Practitioner CPRP 1 4.8
Certified Rehabilitation Counselor CRC 20 95.0
Certified Vocational Evaluator CVE 2 9.5
Licensed Clinical Psychologist --- 1 4.8
Licensed Independent Social Worker LISW 1 4.8
Licensed Professional Counselor LPC 2 9.5
National Certified Counselor NCC 2 9.5
Occupational Therapist Registered/Licensed OTR/L 1 4.8
Registered Mental Health Counselor Intern RMHCI 1 4.8

Delphi Round 2

For the second round of the Delphi, 18 of the original 21 participants responded with their

level of agreement to whether each of the 188 items listed in Table 5 were "useful" for clinical

supervision evaluation. The term useful was defined as an item representative of some aspect of

rehabilitation counseling performance and could potentially help the supervisor evaluate the

trainee. The mean and standard deviation were calculated for each item and reported in Tables

4-3 through 4-7. These results correspond to the following five point Likert scale used to

measure the agreement for each item: strongly disagree, 2=disagree, 3=neutral, 4=agree, and

5=strongly agree. Following Round 2 data collect, 179 of the 188 items demonstrated a high

level of consensus. This was evident due to the fact that these items had a standard deviation

less than 1. In other words, 95 percent of the items rated by participants demonstrated high

consensus with respect to level of agreement. In addition, as indicated by a standard deviation

above 1, but lower than 1.5, 7 of the 188 items demonstrated moderate consensus.









Delphi Round 3

For the third round, 17 participants reevaluated their level of agreement to whether each of

the 188 items listed in Tables 4-3 through 4-7 were useful when evaluating trainee clinical

performance during clinical supervision. Like the previous round, means and standard

deviations ranges were reported as well as item stability. Of the 188 total items, 106 (56%)

resulted in a greater consensus than previously achieved during Round 2. Evidence of this

increase in consensus was seen in the greater convergence (decrease) of the standard deviation.

Sixty eight of the remaining 69 items demonstrated either equal consensus or minimum

divergence (.02 or less increase in standard deviation) from Rounds 2 to 3. In addition, all 188

items fell within the acceptable stability criteria. As described in Chapter 3, acceptable stability

criteria was defined as the mean difference between Rounds 2 and 3 being equal to or less than 1.

Round 3 resulted in 184 of the 188 items reaching "high" consensus. The remaining four

items that demonstrated "moderate" consensus include: "Healthy" (1.00), "Capable" (1.01),

"Diagnose (with some assistance)" (1.06), and "Pleasure to work with" (1.20). At the conclusion

of Round 3, 145 items (77%) resulted in a mean score between 4 (agree) and 5 (strongly agree)

(Tables 4-3 and 4-4). Item include, but are not limited to "Active listening," (5.00); "Builds

rapport," (4.94); and "Recognizes limits of competency," (4.53). In addition, 40 of the 188 items

(21%) resulted in a mean score between 3 (neutral) and 4 (agree) (Tables 4-5 and 4-6). These

items include, but are not limited to "Confident in approaching any interpersonal session with

site or faculty supervisor," (3.94); "Working within a theoretical approach," (3.76); "Balance the

needs of the faculty supervisor and site supervisor," (3.53); and "Diagnose (with some

assistance)," (3.59). Lastly, 3 of the 188 items (1.6%) resulted in mean score below 3 (neutral).

These items include "Ebullient/happy," (2.88); "Extroverted," (2.71) and "Introverted" (2.59)

(Table 4-7).









Table 4-3. Items with means between 4.50 and 5.00 arranged according to consensus
Round 1 Round 2
Mean Convergence


Item
Active listening (e.g. attending to client; listening and
responding)
Builds rapport
Dependable / reliable
Empathetic / Convey empathy to clients
Genuine
Non verbal skills (e.g. eye contact, body position, voice
tone, gestures, facial expressions, physical distance, and
appropriate touch)
Clearly explains limits of confidentiality to client
Demonstrates respect for client's values/beliefs
o0 Individualized treatment planning (i.e. plan interventions
and services that will assist the client in his or her goal)
Understands/practices in ethical manner (i.e. confidentiality,
representation of competence/qualifications, dual
relationships, informed consent)
Approachable / Non-threatening demeanor
Develops trust
Assess accuracy of interpretations with the clients
Identify client problems
Focuses on client rather than themselves in counseling
sessions
Sets realistic, attainable goals
Non-judgmental
Orients client to the counseling process
Interviewing skills


(X 1) (s.d.)


4.94
4.83
4.94
4.89
4.89



4.78
4.83
4.83

4.72



4.83
4.84
4.61
4.56
4.78

4.78
4.67
4.78
4.56
4.72


0.24
0.38
0.24
0.32
0.32



0.55
0.38
0.38

0.46



0.38
0.38
0.61
0.51
0.43

0.43
0.49
0.43
0.51
0.46


Round 3
Mean Convergence
(X 2) (s.d.)


5.00
4.94
4.94
4.88
4.88



4.82
4.82
4.82

4.82



4.82
4.82
4.76
4.76
4.76

4.76
4.76
4.76
4.71
4.71


0.00
0.24
0.24
0.33
0.33



0.39
0.39
0.39

0.39



0.39
0.39
0.44
0.44
0.44

0.44
0.44
0.44
0.47
0.47


Stability
(X 2)- (X 1)


0.06
0.11
0.00
-0.01
-0.01


0.04
-0.01
-0.01

0.10



-0.01
-0.02
0.15
0.20
-0.02

-0.02
0.09
-0.02
0.15
-0.01









Table 4-3. (continued)
Round 1 Round 2 Round 3
Mean Convergence Mean Convergence Stability


Item
Gather pertinent information from the client and relevant
others through assessments, observations, and facilitative
questions
Assess client strengths and weaknesses
Ability to reflect on one's practice
Willingness to continue to learn and develop professionally
Open minded
Flexible/adaptable
Respectful
Suicide risk assessment
Sets clear boundaries with client
, Observation skills
Reflect feeling of client statements
Use open ended questions
Assess client self-efficacy
Prioritize client problems
Identify goals that reflect the client's perspective
Regularly evaluates client progress
Non-defensive /receptive to clinical supervision
Participates in supervision
Evaluates relevance, value, and meaning of supervisory
feedback
Understands the challenges to communication with persons
with hearing, visual, or cognitive impairments
Aware of client's cultural differences/influences
Assess client motivation


(X I) (s.d.) (X 2) (s.d.) (X 2) (X 1)


4.67
4.61
4.61
4.61
4.72
4.67
4.72
4.67
4.62
4.61
4.56
4.61
4.56
4.50
4.61
4.61
4.67
4.67

4.56

4.56
4.56
4.50


0.49
0.50
0.50
0.61
0.46
0.49
0.46
0.59
0.50
0.50
0.51
0.50
0.51
0.71
0.50
0.50
0.59
0.49

0.62

0.51
0.62
0.51


4.71
4.71
4.71
4.71
4.71
4.71
4.71
4.71
4.65
4.65
4.65
4.65
4.65
4.65
4.65
4.65
4.65
4.65

4.65

4.65
4.65
4.59


0.47
0.47
0.47
0.47
0.47
0.47
0.47
0.47
0.49
0.49
0.49
0.49
0.49
0.49
0.49
0.49
0.49
0.49

0.49

0.49
0.49
0.51


0.04
0.10
0.10
0.10
-0.01
0.04
-0.01
0.04
0.03
0.04
0.09
0.04
0.09
0.15
0.04
0.04
-0.02
-0.02

0.09

0.09
0.09
0.09









Table 4-3. (continued)


Round 1


Item
Continues to move session toward the therapeutic goal
Identify and explore options with client
Provide accurate feedback
Demonstrates genuine motivation to become a competent
and ethical rehabilitation counselor
An ability to engage in critical self-reflection rather than
focusing solely on the mistakes of others when they
participate in clinical training, practicum and internship
experiences
Demonstrates emotional stability
Warm
00
o Crisis Management
Paraphrase client statements
Clarify client statements
Allows client to elaborate
Identifies and meets the client's needs
Summarize
Critical thinking ability
Identify short and long term goals
Prepares for supervision
Incorporates supervisor feedback into practice
Identifies own strengths and weakness related to counseling
performance/competency
Sets personal learning goals
Ability to handle client's strong emotions (e.g. crying)


Round 2
Mean Convergence
(X 1) (s.d.)
4.61 0.50
4.50 0.51
4.56 0.51


4.56




4.50
4.39
4.61
4.61
4.44
4.50
4.50
4.56
4.50
4.50
4.44
4.56
4.44

4.56
4.50
4.50


0.51




0.71
1.04
0.50
0.50
0.62
0.51
0.51
0.51
0.51
0.51
0.51
0.62
0.62

0.51
0.51
0.51


Round 3
Mean Convergence
(X2) (s.d.)
4.59 0.51
4.59 0.51
4.59 0.51


4.59




4.59
4.59
4.59
4.59
4.53
4.53
4.53
4.53
4.53
4.53
4.53
4.53
4.53

4.53
4.53
4.53


0.51




0.51
0.51
0.51
0.51
0.51
0.51
0.51
0.51
0.51
0.51
0.51
0.51
0.51

0.51
0.51
0.51


I


I


Stability
(X 2)- (X 1)
-0.02
0.09
0.03

0.03




0.09
0.20
-0.02
-0.02
0.09
0.03
0.03
-0.03
0.03
0.03
0.09
-0.03
0.09

-0.03
0.03
0.03









Table 4-3. (continued)
Round 1 Round 2 Round 3
Mean Convergence Mean Convergence Stability


Item
An awareness of when their own personal issues are
interfering with their ability to learn and practice in a
professionally and ethically responsible manner
Asks effective questions
Recognizes how their personal limitations, biases, and
beliefs impact the counseling relationship
Attentive
Uses appropriate language/terminology (e.g. person first
language; avoids heterosexist, ablest, racist, and sexist
language)
Manage own mental health disabilities
- Utilize simple techniques grounded in counseling theory
Providing client with choices
Preparation of written reports and progress notes (accuracy,
clarity, organization, professional presentation)
Fosters a therapeutic environment
Establishes a working alliance
Willingness to change when their own biases and beliefs
that interfere with their ability to learn how to function as
competent and ethical rehabilitation counselor
Recognizes limits of competency
Demonstrates unconditional positive regard
Understanding the impact of disability
Trustful


(X i) (s.d.) (X 2) (s.d.) (X 2) (X 1)


4.56
4.56

4.56
4.61



4.56
4.33
4.44
4.56

4.33
4.56
4.50



4.50
4.56
4.50
4.56
4.61


0.70
0.62

0.62
0.61



0.62
1.14
0.70
0.51

0.77
0.70
0.79



0.71
0.70
0.71
0.78
0.78


4.71
4.65

4.65
4.65



4.59
4.56
4.53
4.53

4.53
4.59
4.53



4.53
4.53
4.53
4.70
4.65


0.59
0.61

0.61
0.61


0.15
0.09

0.09
0.04


0.62
0.62
0.62
0.62

0.62
0.71
0.72


0.03
0.23
0.09
-0.03

0.20
0.03
0.03


0.72
0.72
0.72
0.77
0.79


0.03
-0.03
0.03
0.14
0.04









Table 4-4. Items with means between 4.00 and 4.49 arranged according to consensus
Round 1 Round 2
Mean Convergence


Item
Elicits motivating statements from client
Keeps client focused
Flexible with counseling skills (when necessary)
Applies structure to the counseling process (e.g. rapport
building, exploration, treatment, termination)
Reflects on the counseling session
Appropriately uses silence
Ability to be insightful
Clear understanding of theory(s)
Understanding of family and interpersonal dynamics
Enhance or help client to increase his or her self-esteem
Controls own feelings and personal thoughts about clients
to increase understanding and to decrease interference in
the counselor relationship
Appropriately uses confrontation
Understanding of and ability to develop accommodation
strategies
Redirects client (when appropriate)
Conceptualize the client's life (i.e. psychosocial,
vocational, etc.)
Willing to research and explore counseling resources (i.e.
journal articles, books, etc.)
Deals directly and appropriately with conflict rather than
avoiding it.
Recognition of the importance of self care
Confident (but not arrogant)


(X 1) (s.d.)
4.00 0.69
4.28 0.46
4.33 0.49


4.39
4.22
4.39
4.39
4.28
4.39
3.83



4.50
4.44

4.44
4.39

4.44

4.22

4.39
4.17
4.11


0.50
0.65
0.50
0.50
0.67
0.50
0.86



0.51
0.62

0.62
0.50

0.51

0.73

0.50
0.62
0.68


Round 3
Mean Convergence
(X 2) (s.d.)
4.06 0.43
4.30 0.47
4.35 0.49


4.35
4.35
4.35
4.35
4.35
4.35
4.00



4.47
4.47

4.47
4.41

4.41

4.41

4.41
4.18
4.18


0.49
0.49
0.49
0.49
0.49
0.49
0.50



0.51
0.51

0.51
0.51

0.51

0.51

0.51
0.53
0.53


0.19

0.02
0.01
0.07


Stability
(X 2)- (X 1)
0.06
0.02
0.02

-0.04
0.13
-0.04
-0.04
0.07
-0.04
0.17



-0.03
0.03

0.03
0.02

-0.03









Table 4-4. (continued)


Round 1

Item
Use close ended questions (when appropriate)
Disagreeing as needed, while maintaining a respectful and
supportive attitude
Ability to recognize when they are responding to supervisors,
clients, students and other professionals in a manner that puts
their own needs before those of their clients
Utilize advanced techniques grounded in counseling theory
Manages the time of the session appropriately
Reframes client statements
Uses self-disclosure appropriately
General ability to conceptualize
Manage/work on personal issues that arise during counseling
or supervision session
Use and understand solution skills (giving advice,
information, and directive)
An ability to use counseling skills intentionally/purposefully
Reflect meaning of client statements
Intervenes in a ways that produce client progress in achieving
his or her behavioral and/or emotional goals.
An awareness of when to seek supervision
Completes supervision tapes/reposts on time
Respects other professionals
Willingness to seek out professional counseling for
themselves when it is needed in order for them to function
ethically and professionally


Round 2
Mean Convergence
(X i) (s.d.)
4.17 0.70


4.28



4.28
4.22
4.28
4.33
4.22
4.39

4.28

4.00
4.50
4.39

4.44
4.50
4.44
4.44


0.75



0.67
0.65
0.75
0.59
0.73
0.61

0.75

0.69
0.62
0.98

0.62
0.62
0.62
0.62


Round 3
Mean Convergence
(X2) (s.d.)
4.29 0.59


4.29



4.29
4.12
4.35
4.35
4.35
4.35

4.35

4.00
4.47
4.47

4.47
4.47
4.47
4.47


0.59



0.59
0.60
0.61
0.61
0.61
0.61

0.61

0.61
0.62
0.62

0.62
0.62
0.62
0.62


Stability
(X 2)- (X 1)
0.12

0.01



0.01
-0.10
0.07
0.02
0.13
-0.04

0.07

0.00
-0.03
0.08


0.03
-0.03
0.03
0.03


4.44 0.70 4.47 0.62 0.03


4.44 0.70


4.47 0.62 0.03









Table 4-4. (continued)


Round 1

Item
Non-defensive
Manages resources
Patient
Uses confirmations/affirmations
Requests guidance as needed (during supervision)
Appropriate dress
Interprets client statements
Assess client interests, skills, and aptitudes
An understanding of and belief in the importance of
supervision
Comfortable with client
S Willing to experiment or take risks in session
Maintains awareness of personal feelings, limitations, and
experiences during the counseling session
Ability to be objective (lose their subjective thought)
Avoids countertransference
Group counseling skills
An ability to articulate why the counseling skills they learn
are crucial to all the roles and functions (e.g., vocational
rehabilitation counselor, case manager, job developer,
vocational evaluator) of rehabilitation counselors
Positive
Goal Oriented
Fiscal Management
Motivating
Kind/thoughtful/caring


Round 2 Round 3
Mean Convergence Mean Convergence
(X 1) (s.d.) (X 2) (s.d.)
4.50 0.62 4.47 0.62
4.39 0.70 4.41 0.62
4.39 0.61 4.41 0.62
4.39 0.61 4.41 0.62
4.44 0.62 4.41 0.62
4.22 0.65 4.24 0.66
4.28 0.67 4.29 0.69
4.17 0.86 4.29 0.69


4.39
4.28
4.00

4.45
3.94
4.33
4.06




4.17
4.17
4.06
4.06
4.17
4.22


0.70
0.67
0.77

0.78
0.87
0.77
0.80




0.92
0.79
0.87
0.87
0.79
0.81


4.29
4.29
4.12

4.41
4.00
4.24
4.06




4.29
4.12
4.12
4.00
4.18
4.24


0.69
0.69
0.70

0.71
0.71
0.75
0.75




0.77
0.78
0.78
0.79
0.81
0.83


I


Stability
(X ,)-(X,
-0.03
0.02
0.02
0.02
-0.03
0.02
0.01
0.12

-0.10
0.01
0.12

-0.04
0.06
-0.09
0.00




0.12
-0.05
0.06
-0.06
0.01
0.02









Table 4-4. (continued)
Round 1 Round 2 Round 3
Mean Convergence Mean Convergence Stability


Item
Conceptualize a case within a theoretical frame of
reference
Intuitive/perceptive
Mature
Displays common sense
Conscientious/prudent
Honest
Avoids advice
Demonstrates respect for the expertise of supervisors
Hard-working
Altruistic
( Capable


(X 1) (s.d.) (X 2) (s.d.) (X ) (X 1)


4.06
4.00
4.28
4.28
4.17
4.44
4.00
4.11
4.06
3.94
4.11


0.87
0.84
0.83
0.89
0.86
0.86
0.91
0.90
0.87
0.94
1.02


4.06
4.06
4.30
4.35
4.12
4.41
4.00
4.00
4.00
4.00
4.18


0.83
0.83
0.85
0.86
0.86
0.87
0.87
0.87
0.87
0.94
1.01


0.00
0.06
0.02
0.07
-0.05
-0.03
0.00
-0.11
-0.06
0.06
0.07










Table 4-5. Items with means between 3.50 and 3.99 arranged according to consensus
Round 1 Round 2
Mean Convergence


Item
Functions at any level of the helping relationship, from
limited to intense
Seamless delivery of the counseling skills
Creative
Able to convey an appropriate level of ease with
colleagues and supervisors
Sets career goals
Confidence in approaching any interpersonal session with
clients
Humorous
Educates client regarding referral information
00
mo Evaluates progress toward own goals
Ability to write and state a disclosure statement
Ability to change client's emotional arousal levels
Humble
Succint
Uses multiple techniques (i.e. eclectic style)
Job development and placement skills
Above average verbal and written expression
Avoids sympathy
Commitment to social justice
Assertive
Closes cases appropriately
Knowledge of career development
Energetic
Direct


(X 1) (s.d.)


3.89
3.94
4.00

3.72
3.61

3.78
3.61
3.94
3.89
4.00
3.56
3.67
3.56
3.89
3.83
4.00
3.67
3.89
3.83
3.94
4.00
3.72
3.67


0.68
0.73
0.69

0.57
0.78

0.65
0.78
0.64
0.83
0.77
0.92
0.77
0.78
0.76
0.99
0.77
0.77
0.76
0.86
0.80
0.77
0.89
0.91


Round 3
Mean Convergence
(X 2) (s.d.)


3.88
3.82
3.94

3.76
3.76

3.88
3.53
3.94
3.94
3.94
3.65
3.59
3.53
3.82
3.82
3.94
3.76
3.76
3.76
3.88
3.88
3.59
3.53


0.33
0.53
0.56

0.56
0.56

0.60
0.62
0.66
0.66
0.66
0.70
0.71
0.72
0.73
0.73
0.75
0.75
0.75
0.75
0.78
0.78
0.80
0.80


Stability
(X 2)- (X1)


-0.01
-0.12
-0.06

0.04
0.15

0.10
-0.08
0.00
0.05
-0.06
0.09
-0.08
-0.03
-0.07
-0.01
-0.06
0.09
-0.13
-0.07
-0.06
-0.12
-0.13
-0.14









Table 4-5. (continued)
Round 1 Round 2 Round 3
Mean Convergence Mean Convergence Stability
Item (X 1) (s.d.) (X 2) (s.d.) (X 2) (X 1)
Confident in approaching any interpersonal session with
site or faculty supervisor 3.83 0.92 3.94 0.83 0.11
Working within a theoretical approach 3.83 0.86 3.76 0.83 -0.07
Timely/opportune 3.83 0.86 3.76 0.83 -0.07
Calming/soothing 3.78 0.88 3.71 0.85 -0.07
Just 4.00 0.91 3.88 0.86 -0.12
Logical 3.83 0.92 3.76 0.90 -0.07
Intelligent/wise 3.67 0.91 3.71 0.92 0.04
Team player 3.72 0.96 3.71 0.92 -0.01
Balance the needs of the faculty supervisor and site
supervisor 3.56 1.10 3.53 0.94 -0.03
Healthy 3.61 1.10 3.65 1.00 0.04
Diagnose (with some assistance) 3.56 1.15 3.59 1.06 0.03
Pleasure to work with 3.56 1.20 3.53 1.18 -0.03

Table 4-6. Items with means between 3.00 and 3.49 arranged according to consensus
Round 1 Round 2 Round 3
Mean Convergence Mean Convergence Stability
Item (X i) (s.d.) (X 2) (s.d.) (X 2)-(X 1)
Has faith in intuitive responses 3.33 0.84 3.41 0.80 0.08
Develop ability to supervise others 3.50 0.86 3.41 0.80 -0.09
Experienced 3.44 0.78 3.41 0.80 -0.03
Moderating/judging 3.17 1.10 3.00 0.87 -0.17
Takes good notes during session 3.44 1.04 3.41 0.94 -0.03









Table 4-7. Items with means below 3.00 arranged according to consensus
Round 1 Round 2 Round 3
Mean Convergence Mean Convergence Stability
Item (X 1) (s.d.) (X 2) (s.d.) (X 2) (X 1)
Ebullient/happy 3.00 0.97 2.88 0.86 -0.12
Extroverted 2.78 0.94 2.71 0.92 -0.07
Introverted 2.67 0.97 2.59 0.94 -0.08









CHAPTER 5
DISCUSSION

Introduction

This study involved identifying and obtaining consensus on a variety of items useful for

clinical supervision evaluation. This chapter is divided into four sections, which discuss the

findings presented in the previous chapter. The first section includes an overview of the

significant findings from Delphi Study. The second section describes limitations of this study.

The third section discusses implications of the findings for education and public policy. The

final section includes recommendations for further research and development of the findings of

this study.

Overview of Significant Findings

Differences in Participant Response Sets

Of the educators who were solicited for this study, 21 completed the first round of the

Delphi. Educators were instructed to provide a comprehensive list of specific skills, behaviors,

and attributes that were useful/appropriate for clinical supervision evaluation. Round 1 data

collection included 410 items, with individual response sets ranging from 6 to 51 items, which

were later collapsed into 188 substantially distinct items. As expected, many rehabilitation

educators listed unique items that were occasionally identified by other participants (e.g. "willing

to research and explore counseling resources," "continues to move session toward the therapeutic

goal"). However, several items fundamental to the therapeutic process were only occasionally

identified by the participants. For example, of the 21 participants, only six suggested

"empathetic/convey empathy to clients" as useful/appropriate for clinical supervision evaluation.

Similarly, nine rehabilitation educators suggested "active listening (e.g. attending to client;









listening and responding)" as useful/appropriate for clinical supervision evaluation, while only

three participants listed "build rapport" and "paraphrase client statements".

In addition to item variability, participant response sets also varied. While participants

were not expected to provide identical items, entire response sets differed considerably from

expert to expert. This phenomenon may relate to what Gizara and Forrest (2004) characterize as

fundamental differences between supervisors regarding the purpose of clinical training. They

report that some supervisors may believe that the clinical supervision process/internship strictly

serves as a vehicle for the trainee's professional development, while other supervisors may

believe the clinical process functions as a gatekeeping mechanism (Gizara & Forrest). In

addition, individual programs and supervisors may have considerable flexibility when evaluating

trainee competence during clinical supervision (Robiner et al., 1993). This may be due to the

lack of specificity provided by professional organizations and standards highlighting what

supervisors should be evaluating during clinical supervision.

High variability among response sets in this study may suggest that supervisors

conceptualize trainee competency differently. Consequently, trainees are evaluated against

vastly different criteria which may lead to discrepancies in trainee preparedness and ultimately in

rehabilitation counseling performance. If one supervisor believes the trainee should be

personally fit to counsel, actively continue their professional development, and sufficiently

demonstrate certain clinical skills and treatment planning activities, while another supervisor

simply evaluates the trainee's personal suitability and potential to develop adequate clinical

skills, trainees may be unfairly subjected to different performance standards. Supervisor

opinions regarding trainee performance will invariably differ from supervisor to supervisor;

however, allowing supervisors to use arbitrary criteria to evaluate trainee performance will









unnecessarily complicate the evaluation process and all but guarantee trainees will be held to

different performance standards.

Item Identification

A comprehensive set of clinical skills, behaviors, and attributes representative of several

theoretical competency domains compose the final item list. The majority of items on this newly

constructed list are identical or conceptually similar to items commonly found in performance

instruments and in the counseling literature (Bernard, 1997; Chan et al., 2004; Corey; 2004;

Cromier & Cromier, 1991; Eriksen, & McAuliffe, 2003; Hackney & Cromier, 1994; Hill, 2004;

Lumadue & Duffey, 1999; Miller & Rollnick, 2004; Myrick & Kelly, 1971; Getting & Michaels,

1982; Robiner et al., 1994). These findings further emphasize the compatibility of the

rehabilitation counseling field with other counseling related professions.

While rehabilitation counseling is closely related to other counseling professions, there is

an emphasis on working with people with disabilities and reintegrating these clients into society.

Due to this emphasis it was expected that the experts would identify several items fundamental

to the rehabilitation counseling profession that are not commonly found on other instruments.

The experts in this sample listed items such as "understanding the impact of disability,"

"knowledge of career development," "job development and placement skills," and

"understanding of and ability to develop accommodation strategies". The identification of these

rehabilitation counseling items highlights the uniqueness of our field and the importance of

creating measures that can wholly account for the work of rehabilitation counselors.

Although an abundance of items were initially gathered through the Round 1

investigation (188 items), there were several items not identified by the experts that may be

useful for clinical supervision evaluation. The items that were not reported but expected to be

found are ones listed on other counseling performance measures, such as "student demonstrates









an awareness of power differences in the therapeutic relationship, and manages these differences

therapeutically" (Kerl et al., 2002), "evoking and punctuating client strengths" (Erikson &

McAuliffe, 2003), "improve client confidence/self-efficacy regarding change behavior" (Milner

& Rollnick, 2004), "counselor refrains from using social conversation" (Hackney & Cormier,

2004), "counselor helped client to develop action steps for goal attainment" (Hackney &

Cormier), and "counselor provided an opportunity for client to practice or rehearse action step"

(Hackney & Cormier). Due to the closely related nature of rehabilitation counseling with other

counseling fields, future investigations into supervision measures might consider items that are

already being used on instruments from other counseling professions.

Items Consensus

As previously noted in Round 3, 106 of the 188 items (56%) resulted in a greater

consensus than had been achieved during Round 2. Sixty eight of the remaining 69 items

demonstrated either equal consensus or minimal divergence (.02 or less increase in standard

deviations) from Rounds 2 to 3. Only one item, "Provides client with choices" demonstrated an

increase in standard deviation greater than .02 (.07). This may be partially explained by the fact

that this item had a Round 2 mean value almost directly in between 4 (Agree) and 5 (Strongly

agree) (4.56). Therefore, more participants most likely decided to simply change their ratings

from "Strongly agree" to "Agree."

Although the vast majority of items demonstrated "high" consensus, four items

demonstrated only "moderate" consensus. These items include: "Healthy" (1.00), "Capable"

(1.01), "Diagnose (with some assistance)" (1.06), and "Pleasure to work with" (1.20).

Three of the four items ("Capable," "Pleasure to work with," and "Healthy") were all

personal characteristics. Participants may believe that these terms are too general and

interpreting their meaning may be futile. One participant commented that:









I don't know what many of these words mean. I know what I mean by them, but I don't
know what others may mean by them. The shades of meaning of many of these words are
too variable for me to interpret-and therefore they are not useful (to me).

The vast majority of the items identified in this study represent specific skills or behaviors

for which participants may have a basic understanding. However personal characteristics are

more likely to manifest in various ways through out clinical supervision and therefore

participants are more likely to define these terms subjectively.

In addition, the term "Healthy" carries a certain connotation that may be less palatable to

rehabilitation counselors who almost exclusively work with people with disabilities. Health is

particularly variable and could imply several types including mental health, physical health,

and/or functional capacity. Additionally "Capable" may imply that the trainee should reach a

certain level of ability before successfully completing their clinical training. Supervisors may

view the clinical portion of the training as developmental (Gizara & Forrest, 1999) and therefore

may disagree that this item is useful for clinical evaluation. Finally, participants did not reach

"high" consensus on the term "Pleasure to work with." In particular, this item appears to be

highly subjective and therefore participants may believe this term is not useful for clinical

supervision evaluation. Clinical supervisors recognize that the trainee's clinical competency is

independent of whether they feel the student was a pleasure to work with. Quite simply,

supervisors and trainees may not work well together, but this relationship should not necessarily

determine the trainee's clinical ability.

The fourth item that demonstrated "moderate" consensus is "Diagnose (with some

assistance)." Interestingly, not only did this item exhibit moderate consensus, but the item also

had a Round 3 mean value of 3.59. This item may exhibit moderate consensus for a variety of

reasons. First, some clinical supervisors may believe that diagnosing is not a basic function of

rehabilitation counseling. Traditionally rehabilitation counselors focus in the vocational









assessment and placement arena. Although some clients may show signs of mental health

disorders, some participants may not believe it is the role of rehabilitation counselors to

"diagnose" a client. On the other hand, some participants may be affiliated with rehabilitation

counseling programs that serve as vehicle for licensed professional counselors or mental health

counselors. Therefore, this item may simply reflect the overall diversity of the rehabilitation

counseling profession.

While some items remained at "moderate" consensus, five items increased from

"moderate" consensus in Round 2 to "high" consensus in Round 3. Of these items, only

"Manages own mental health disabilities" and "Demonstrates emotional stability" also increased

their mean average from Round 2 to Round 3. The increased mean score in Round 3 suggests

that more participants agreed that these items are useful for clinical supervision evaluation.

Other items related to the trainees' self care, "Recognize the importance of self care," "An

awareness of when their own personal issues are interfering with their ability to learn and

practice in a professionally and ethically responsible manner," and "Willingness to seek out

professional counseling for themselves when it is needed in order for them to function ethically

and professionally," reached "high" consensus before Round 3. Interestingly, the three items

that demonstrated initial "high" consensus do not suggest that the trainee should manage any

mental health or emotional issues, but rather that the student simply be aware of and be willing

to seek help for these issues. In other words, some participants may have initially believed that it

is adequate for students to be seeking professional help for these issues, but that these issues did

not necessarily have to be resolved in order to receive professional endorsement.

Section G.3.a of the CRCC code of professional ethics (2002) clearly states that educators

should be aware of the student's personal limitations that might affect performance and help









secure remedial assistance when needed, and dismiss students whose personal limitations cause

them to be unable to provide competent service. Ironically, of all the items identified in this

study, it would appear that "Manages own mental health disabilities" and "Demonstrates

emotional stability" should have demonstrated initial "high" consensus because they are the most

closely related to any evaluation guidelines stated in the rehabilitation counseling professional.

Ultimately, both "Demonstrates emotional stability" and "Manages mental health disabilities"

achieved both "high" consensus (.51 and .62 respectively) and a high mean (4.59 and 4.56

respectively). These findings reinforce the fact that rehabilitation counseling educators are

bound by professional ethics to ensure that trainees manage their mental health issue and are

emotionally stable.

Unlike the previous two items, the remaining three items ("Balance the needs of the faculty

supervisor and site supervisor," "Moderating/judging," and "Takes good notes during session")

had mean scores that decreased from Round 2 to Round 3. Additionally, the mean value for each

of these three items fell between 3 (neutral) and 4 (agree). Thus, participants agreed that these

items were less useful for clinical supervision evaluation.

More specifically, clinical supervisors may believe it is inappropriate for trainees to be

responsible for addressing any conflicts between the department and training site and therefore

feel that the trainee should not balance the needs of their faculty supervisor and site supervisor.

Moreover, many supervisors may feel that this item is simply not representative of clinical

performance. Regarding the item "Moderating/judging," participants may interpret this term

differently. The term carries a strong connotation with which many educators may disagree.

Specifically, some participants may interpret the term as "judging the client," which conflicts

with the fundamental principles of rehabilitation counseling. On the other hand, other









participants may see the value of judging certain situations (e.g. a client situation). In these

instances, the trainee is not judging the person per se, but is making judgments about

information. Lastly, the item "Takes good notes during session" may have achieved "high"

consensus, but a lower mean rating, because more participants may have felt that note taking is

inappropriate during the counseling session. Clinical supervisors may view note taking as an

unnecessary distraction to the trainee, while others may simply feel that writing is simply not

related to clinical performance.

Limitations

The first limitation to this study is the rehabilitation counseling educator sample size. This

Delphi study solicited rehabilitation educators through the National Council of Rehabilitation

Education (NCRE) listserv. The NCRE is the leading professional association for rehabilitation

educators and represents over 90 higher education institutions and 600 individual memberships

(NCRE, 2008). While it is unrealistic to identify the exact number of NCRE members who

would qualify for this study, it was anticipated that a sizeable number of NCRE members met the

inclusion criteria and would participate. Although it is unrealistic to identify the specific number

of rehabilitation counseling educators who were eligible to participate in this study, a smaller

sample size was attained than expected.

Of the five rehabilitation counseling Delphi studies found within the literature review, only

two (Rubin et al., 1998; Thielsen & Leahy, 2001) had a Round 1 sample size near or below the

sample size attained in this investigation (23 and 18 respectively). Rubin et al. targeted 23

experts with a Round 1 response rate of 100%; whereas Thielsen and Leahy targeted 20 experts

with a Round 1 response rate of 90%. Unlike this present study, both Rubin et al. and Thielsen

and Leahy initially identified a select number of experts to participate in Round 1 of their

studies. Other rehabilitation counseling Delphi studies have included Round 1 sample sizes of









31 (Shaw et al., 2006), 44 (Currier et al., 2001), and 111 (Hakim & Weinblatt, 1993). While the

Round 1 sample size for this study may appear small, Okoli & Pawlowski (2004) suggest that the

size of a Delphi panel typically ranges between 10-18 experts. Therefore, the final number of

panel experts in this study is consistent with a typical Delphi study.

In addition to the Round 1 sample size, Delphi studies are inherently susceptible to

participant attrition. Because the Delphi procedure requires experts to respond to several

iterations of questions over time, it is uncommon for all participants to complete every round.

For this study, 18 of the 21 experts from Round 1 completed Round 2 (85.7% response rate), and

17 of the 18 participants from Round 2 completed Round 3 (94.4% response rate). Participant

attrition between evaluation rounds (here reported as rounds 2 and 3) may be of particular

concern because of its effect on item stability calculations (i.e. the item mean difference between

evaluation rounds). When high attrition rates occur between these rounds item stability becomes

less reliable. In comparing the response rate between evaluation rounds of this study to other

rehabilitation counseling Delphi studies, the results are encouraging. This study resulted in a

response rate between evaluation rounds of 94.4%, whereas other rehabilitation counseling

Delphi studies have yielded response rates of 68.8% (Currier et al., 2001), 71.4% (Shaw et al.,

2006), 80.2% (Hakim & Weinblatt, 1993), 89.0% (Thielsen and Leahy, 2001), and 90.0% (Rubin

et al., 1998). Even though participant attrition is always a concern during Delphi studies, the

level of attrition for this study is comparable to other Delphi studies in the rehabilitation

counseling literature. However, the attrition level from Round 2 to Round 3 may have slightly

impacted the standard deviation calculations. Because the standard deviation calculation is

effected by the size of the sample, 17 participants (rather than 18) may result in small, but

naturally occurring change in standard deviation. Therefore, any small incremental change in









standard deviation (e.g. plus or minus .02) may be the result of the decreased sample size rather

than item convergence or divergence.

In addition, only seven of the 10 CORE defined regions are represented in this study.

However, since regions are defined according to geographical location, different regions contain

an unequal number of rehabilitation education institutes. Of the three regions not represented,

region II contains 10 schools, region VIII contains three schools, and region IX contains seven

schools. Therefore the potential number of eligible participants from a particular region may be

considerably smaller than other regions. Lastly, 10 of the original 23 participants (47.6%) in this

study were from region IV (southeast). This region also has more rehabilitation counseling

education programs (22) than any other region. Although an abundance of participants from one

region may typically cause alarm, it is not apparent why participants from this region (or any

other region) would respond in a manner that is inconsistent with other rehabilitation counseling

supervisors. Therefore, it is believed that an over representation of participants from this region

would not necessarily bias the results.

Implications

Education

Clinical competency dimensions and benchmarks

Within the rehabilitation counseling profession, researchers have extensively written about

professional competence in a variety of arenas (e.g. multicultural). However, one area that has

received little attention from the educator's perspective is clinical supervision. With the

exception of two clinical supervision models (Maki & Delworth, 1996; Schultz, Copple, &

Ososkie, 1999), the rehabilitation counseling literature offers little related to clinical supervision

competency criteria.









Because the clinical portion of rehabilitation counseling training reflects how the trainee

will most likely perform in the field, it is perhaps the best opportunity for the supervisor to

evaluate the trainee's interpersonal and intrapersonal functioning. Thus, the rehabilitation

counseling profession will be well served to identify a comprehensive list of trainee skills,

personal characteristics, and other professional attributes useful for clinical supervision

evaluation. The findings from this study will have considerable implications for evaluating

trainee competence during clinical supervision, as well as building upon existing rehabilitation

counseling supervision models.

During the clinical portion of the program, supervisors must fulfill administrative and

evaluative duties as well as address the trainee's inter- and intrapersonal characteristics that

manifest throughout clinical training (Falvo & Parker, 2000). When students begin practicum or

internship, professional and personal expectations should be clear (Falvo &Parker). To help

clinical supervisors fulfill these and other supervisory roles, an extensive list of skills, behaviors,

and attributes will allow clinical supervisors to conceptualize and evaluate trainee performance

across multiple competency dimensions. According to Kaslow, Rubin, Bebeau, Leigh,

Lictenberg, Nelson et al., 2007, "benchmarks can capture the minimal level of each competency

for each stage of professional development and guide our conceptualization of competent and

problematic performance" (p.481).

Therefore, the general language used by CORE and CRCC to ensure rehabilitation

education programs endorse professionally competent and personally suitable trainees can be

further understood as specific competence dimensions, and more importantly specific

competency criteria. For example, the findings suggest that clinical supervisors may find it

useful to examine several dimensions including the trainees' general counseling skills (e.g.









ability to present open-ended questions; reframing; affirmation), critical thinking skills (e.g.

identifies client themes), intervention skills (e.g. establishes clear, measureable goals; evaluates

interventions), counseling session management (e.g. provides purpose of counseling session;

exhibits control of session), behavior during clinical supervision (e.g. appropriate case notes;

open to supervisory feedback), personal attributes exhibited during the counseling session (e.g.

genuine; non-judgmental), and professional behaviors (e.g. has appropriate boundaries;

empowers the client).

Much of the recent force behind identifying, training, and assessing professional

competence comes from the psychology literature (Elman, Illfelder-Kaye, & Robiner, 2005;

Kaslow, 2004; Kaslow, Borden Collins, Forrest, Illfelder-Kaye, Nelson et al., 2004; Kaslow,

Leigh, Smith, Bebeau, Lictenberg, Nelson, Portney, et al., 2007; Rubin, Bebeau, et al., 2007;

Kaslow, Rubin, Forrest, Elman, Van Home, Jacobs, et al., 2007). As a whole, this body of

literature offers a series of ideas, principles, and proposals to aid psychology programs in

assessing and improving current and future professional competence. Although the overall focus

of the literature relates to assessing a set of foundational, core, and specialty competencies

throughout the breadth of psychologists' professional development, many of the proposed

standards are applicable to a more narrowly defined area of training such as clinical supervision.

According to several researchers (Kaslow, 2004; Kaslow, Rubin, Bebeau, et al., 2007),

optimal assessment of professional competence includes a multi-trait approach. Multi-trait

analysis refers to concurrently assessing the trainee's knowledge, skills, attitudes, performance,

and ability to integrate these components across all competence domains. According to Kaslow,

Rubin, Bebeau, et al., the assessment process should include the evaluation of individual and

integrated competencies with multiple traits being evaluated simultaneously. When applying the









concept of multi-trait evaluation to the area of rehabilitation counseling clinical supervision, the

findings from this study identify a variety of competency dimensions as well as specific

performance criteria and personal attributes that will allow clinical supervisors to holistically

assess the trainee's counseling competency.

Although these clinical dimensions are apparent, albeit theoretical, they provide a

comprehensive framework for evaluating how students have integrated and applied their training

to the counseling session. Furthermore, the performance indicators identified from this study

provide specific behaviors and attributes which represent those dimensions. Thus, these research

findings target, arguably, the most fundamental aspect of rehabilitation counseling training: the

counseling session. With better defined dimensions of counseling competence, rehabilitation

counseling programs can ensure that trainees are held to the highest standards of the profession,

thus improving the gate-keeping process for their profession and ensuring novice rehabilitation

counselors are meeting the needs and expectations of their future clients.

In addition to general evaluation applications, the findings from this study can also be used

to supplement existing rehabilitation counseling supervision models. Currently, the

rehabilitation counseling literature offers two clinical supervision models that describe the

interaction between counseling trainee and supervisor. First, Maki and Delworth's Structured

Developmental Model (SDM) helps the supervisor catalog the trainees' professional

development by comparing his or her motivation, autonomy, and awareness relative to eight

competency domains. By conceptualizing these internal processes against the various

competency domains, the supervisor can classify the trainees' progress according to one of four

developmental stages (Level 1, Level 2, Level 3, and Level 3 Integrated). Depending on the









level of development, the model suggests various strategies to help supervisors facilitate the

counselor's development.

The findings from this study can be used to supplement the eight competence domains

found in the SDM. Specifically, the SDM categorizes the domains into one of two groups: (a)

primary domains, and (b) process domains. Primary domains, also known as meta-domains,

include: (a) sensitivity to individual differences, (b) theoretical orientation, and (c) professional

ethics and are continually examined during the counselor's development. The remaining

domains, known as process domains, include: (a) interpersonal assessment, (b) client assessment

within the environment, (c) case conceptualization, (d) treatment goals and plans, and (e)

intervention strategies (Maki & Delworth, 1995). While these domains are clearly organized to

reflect the various professional and personal processes found within a counseling session, they

fail to provide specific criteria useful for clinical competency evaluation. However, the findings

from this study can be used to help expand the eight competency domains to include specific,

behavioral criteria. Once each domain is defined by a cohesive set of specific competency

criteria, supervisors can more accurately and consistently evaluate the trainees' professional

development.

Like the SDM, the Integrated Model for supervision proposed by Schultz et al. (1999),

provides a thorough and practical model for the novice or expert supervisor. Because the

working alliance between supervisor and trainee is critical, the Integrated Model for supervision

proposes that the supervisor adopt a particular supervisory role (teacher, counselor, and

consultant) to suit the needs of the trainee as well as foster the professional development of the

student. Generally, the supervisor focuses on three areas of trainee competence (intervention

skills, conceptualization skills, and personalization skills) across three Phases of professional









development (technical, integrative, and consultative). The amount of attention each competence

domain receives depends on the specific developmental phase of the trainee. For example, Phase

I (Technical), focuses on intervention and personalization skills such as client centered

techniques as well as appropriate attitudes, self-trust, and internal evaluation processes, but

conceptualization skills are generally reserved for advanced phases of development. Unlike the

SDM (Maki & Delworth, 1995), the Integrative Model for Supervision (Schultz et al., 1999)

provides some specific competency criteria for trainee evaluation. However, most of the specific

competency criteria pertain to the trainee's personal attributes (e.g. willingness to experience,

develop trust in self, enhance internal evaluation, demonstration of unconditional positive regard,

acceptance, and accurate empathy), while clinical skills are described more as generic domains

(e.g. counseling micro skills, case conceptualization). Therefore, the findings from this study

can not only supplement the existing competency criteria (particularly micro counseling skills

and case conceptualization), but also allow the supervisor to assess the trainee's professional

development across more competency domains.

Supervision evaluation continuity

Although supervisors rely on a variety of sources to evaluate a trainee (e.g. site supervisor

evaluation), they typically depend on more direct observations such as reviewing counseling

session tapes and clinical supervision exchanges to evaluate the trainee's professional

development. The evaluation process is largely dependent on two separate, yet related,

components: (a) the criteria used to evaluate trainees, and (b) the supervisor's opinion regarding

trainee performance. While supervisory opinion is inherently variable, the criteria used to

evaluate trainee performance can be made relatively consistent throughout various training

programs. Rather than supervisors using arbitrary criteria or criteria developed by their program









to evaluate clinical competency, a set of empirically based competency criteria, standards, and

expectations might provide a foundation to ensure increased continuity across programs.

Establishing a comprehensive set of competency criteria not only provides all supervisors a

theoretical framework to evaluate the trainee's clinical performance holistically, but also helps to

ensure evaluation continuity between supervisors. Currently, rehabilitation education programs

are free to develop their own competency criteria to define the parameters of professional

competence and personal suitability. Although some differences in evaluation criteria are

necessary to accommodate the specific goals of the program, the rehabilitation counseling

profession would be better served to first identify specific clinical expectations of trainee

performance (Robiner et al., 1993) and then allow individual programs to build upon those

fundamental criteria with additional skills to suit their program needs and goals. Regardless of

which rehabilitation counseling program trainees attend or which clinical site they complete their

practicum or internship; trainees could be evaluated against the same minimal set of competency

criteria. Therefore, professional organizations, like CORE, can use the findings from this study

as a foundation for expanding the concepts of minimal professional performance and personal

expectations as well as offer additional benchmark items for continued professional

development.

Improving communication between supervisor and trainee

In addition to potentially improving evaluation continuity among supervisors, the findings

from this research study could positively impact the supervision process. Lehrman-Waterman

and Ladany (2001) found that the processes of effective goal setting and feedback positively

correlated with the supervisor/trainee working alliance, trainee self efficacy, and the trainees'

overall satisfaction with supervision; leading the authors to state that "new and current

approaches (to supervision) may need to reassess and highlight the importance of goal setting









and feedback" (p. 175). Supervisors who provide feedback based on explicit expectations or

criteria (i.e. clearly describing specific behaviors) have the potential to be better understood and

received by students (Freeman, 1985), while supervisors who provided vague, global, and/or

abstract feedback have been considered "lousy" (Magnuson, Wilcoxon, & Norem, 2000).

Clinical supervision typically requires the supervisor to engage in two types of evaluation

processes: (a) formative and (b) summative (Robiner et al., 1993). Formative evaluation is the

process whereby supervisors provide direct feedback to the trainees during supervision to help

foster their professional development. On the other hand, summative evaluation is the more

administrative process where the supervisor assesses the trainees' overall competence to ensure

the student qualifies for professional endorsement. The authors note that supervisors and

trainees oftentimes view summative evaluation with apprehension and consequently dislike the

summative evaluation process (Robiner et al., p.4). However, Bernard and Goodyear (2004)

caution that all aspects of clinical supervision, including the formative evaluation process, have

inherent evaluative undertones. For example, during formative evaluation, supervisors

continuously send spoken and unspoken evaluative feedback to students; "some of the

supervisor's evaluative comments are deliberately sent (encoded) by the supervisor to the

trainee; others are received (decoded) by the trainee and may or may not be an accurate

understanding of the supervisor's assessment" (Bernard & Goodyear, p.20).

To improve the formative and summative evaluation processes, supervisors should

communicate clear competency criteria to the trainee during supervision. Therefore, the findings

from this study can directly impact the supervisory process by improving the level and clarity of

supervisor feedback and establishing clear expectations for student performance. Supervisors

can use the findings from this research study to identify a comprehensive list of specific trainee









skills and attributes from a variety of competency domains. Once identified, supervisors can

communicate these criteria to students prior to clinical supervision. Therefore, trainees will have

a clearer understanding of what is expected of them during clinical supervision and supervisors

will be able to communicate clear feedback to the student.

Public Policy

Association of American Colleges and Universities / Council for Higher Education
Accreditation

In addition to specific educational implications, the findings from this study may also have

broader implications in shaping public policy. As part of their 2008 agenda, the Association of

American Colleges and Universities (AAC&U) and the Council for Higher Education

Accreditation (CHEA) state that it is imperative for the higher education community to improve

the quality of education in the United States so that graduates are "prepared to contribute to

society as knowledgeable, engaged, and active citizens" (p.1). In order to aid universities and

colleges to monitor the quality of student development, the organizations outlined a series of

principles and actions to help ensure that the collegial experience is challenging for students, and

assist higher education institutes in becoming more accountable and transparent.

AAC&U and CHEA posit that although the responsibility to establish better standards in

higher education is shared among a variety of organizations, associations, and governing bodies,

it is the colleges and universities themselves that must set a precedent in helping the American

higher education system maintain its designation as one of the world's foremost leaders.

Specifically, AACH and CHEA propose that the various organizational tiers within each

university (i.e. university, college, department, division) "should develop ambitious, specific,

and clearly stated goals for student learning appropriate to its mission, resources, tradition,

student body, and community setting" (p.2). As such, the findings from this research study will









help various counseling and psychology programs establish clearer expectations and higher

standards for student performance.

Specifically, an exhaustive list of specific counseling skills, personal attributes, and

professional behaviors will allow programs to conceptualize the various aspects of counselor

development more holistically. Thus, supervisors and trainees alike will have a clearer

understanding of what specific skills and attributes are being evaluated. Furthermore, by

identifying a comprehensive list of evaluation criteria, clinical supervisors will be able to

develop higher standards for student performance. Programs could begin to identify the specific

skills and behaviors related to various levels of student performance. Simply stated, counseling

programs can move toward the vision of AAC&U and CHEA by evaluating trainee performance

across specific counseling skills, personal attributes, and professional behaviors; thus ensuring

clear, ambitious goals for student performance as well as program accountability.

Council on Rehabilitation Education / Commission on Rehabilitation Counseling
Certification

While the principles and actions set forth by AAC&U and CHEA call for universities and

colleges to lead the way in maintaining America's global position in higher education, perhaps

professional organizations and specific accrediting bodies also have an obligation to assist in

identifying reasonable expectations and rigorous standards for professional development. As

previously discussed, the Council on Rehabilitation Education (CORE) and the Commission on

Rehabilitation Counseling Certification (CRCC) both acknowledge the need for rehabilitation

counseling trainees to demonstrate a level of professional competence and personal suitability.

Whereas the CORE accreditation standards basically outline a series of general curriculum

requirements, knowledge domains, and educational outcomes for student performance, the

CRCC code of professional ethics addresses the educator's responsibility to be aware of the









student's academic and personal limitations. Although this literature acknowledges the

educator's role in trainee development and provides a basic foundation for assessing

rehabilitation counselor ability, both are limited in terms of trainee evaluation. Specifically, the

performance outcomes provided by CORE are too general for clinical supervisors to truly

evaluate a rehabilitation counseling trainee. Furthermore, neither CORE nor CRCC define any

criteria related to personal suitability; thus this concept is left to broad interpretation.

Although the policies of CORE and CRCC are limited, the findings from this study could

have implications for how the rehabilitation counseling profession conceptualizes and defines

trainee competence. With a clearer understanding of the skills, attitudes, and behaviors

rehabilitation counseling educators feel are useful for clinical supervision evaluation,

professional organizations like CORE and CRCC can reexamine existing performance outcomes

to assure that key components of counseling competency are captured with their criteria and also

provide programs guidance about how to assess rehabilitation counseling performance and

personal suitability. Furthermore, these research findings may initiate dialogue among

rehabilitation counseling educators regarding the minimum skills and behaviors a trainee should

be able to demonstrate in order to qualify for professional endorsement.

Overall, the findings from this study can have implications for organizations at both the

national and professional levels. On the one side, rehabilitation counseling programs can uphold

the ideals of AAC&U and CHEA by ensuring trainees are subjected to clear, rigorous standards,

and on the other side, rehabilitation counseling organizations can improve trainee evaluation and

make sure people with disabilities are being served by competent, well-rounded individuals.

Future Research

As outlined in chapter 2, rehabilitation counseling educators face a variety of challenges

when evaluating a trainee during clinical supervision. Moreover, professional organizations such









as the Council on Rehabilitation Education and the Commission on Rehabilitation Counseling

Certification provide little guidance for evaluating clinical competency. Thus, rehabilitation

counseling programs have developed and relied upon in-house instruments to evaluate trainee

performance. Consequently, clinical competency standards may vary from program to program

resulting in inconsistent trainee evaluation.

The findings from this study resulted in a comprehensive item bank of skills, behaviors,

and personal attributes to consider when evaluating rehabilitation counseling trainees during

clinical supervision. This item bank will allow rehabilitation counseling educators to have a

thorough understanding of specific activities relating to the clinical performance. As such,

professionals should be able to refer to this item bank for use in future research on clinical

supervision evaluation. Although 10 theoretical competency domains were formed based on a

content analysis of the data, a factor analysis is necessary to determine whether the generated

items are actually representative of rehabilitation counseling clinical performance as well as

whether separate clinical competency constructs exist. Such an investigation will allow for the

creation of a measure for evaluating trainee clinical competency.

A validated clinical supervision instrument could help make the clinical supervision

evaluation process more consistent as well as assist educators and trainees to catalog the

student's professional development. Although clinical supervision instruments have been

previously developed, researchers have yet to validate a clinical supervision instrument using

Item Response Theory methodologies. To accomplish this task, supervisors should evaluate

trainee performance using the items identified in this study. Next, a factor analysis should be

performed to empirically determine if the items make up a unidimensional construct or multiple

constructs. Item response methodologies, such as Rasch analysis, would then be applied to each









construct to determine item fit and the hierarchal ordering of the items. Determining multiple

developmental hierarchies of clinical performance will provide a basis for understanding trainee

development. Once the instrument is fully developed, its usability and helpfulness can be tested

among supervisors.

In addition to developing a clinical supervision instrument, future research could be

conducted to examine how a validated and reliable instrument impacts the clinical supervision

experience. More specifically, a clinical supervision instrument can help supervisors establish

appropriate learning goals and provide specific feedback to trainees about their performance. As

previous research has suggest, these two practices have been positively correlated with

improving the supervisory working alliance, enhancing the trainee's self efficacy, and increasing

the trainee's overall satisfaction with supervision (Lehrman-Waterman & Ladany, 2001).

Building upon this research, the researcher can test how a Rasch validated clinical supervision

instrument impacts the supervisory relationship as well as trainee performance.

Conclusion

Although the clinical supervision process is a critical component of student preparation

and affords educators the opportunity to identify unsuitable trainees, rehabilitation counseling

literature and professional organizations provide little guidance related to specific performance

expectations and evaluation criteria. Consequently, training programs may rely solely on the

supervisor's own clinical judgment or use unstandardized, in-house evaluation instruments to

evaluate clinical performance. As a result, trainee evaluation during clinical supervision may

differ from program to program.

Despite this variability, rehabilitation counseling educators have identified and obtained

consensus on 183 items representative of clinical performance. These items were subsequently

organized into 10 theoretical competency domains including general counseling session items,









process skills, conceptualization/assessment items, identifying goals/treatment planning, personal

attributes, supervision items, self care items, professional behavior, general professional

development, and other items. With the identification of an exhaustive list of clinical

performance items, supervisors can better conceptualize trainee competency, offer specific,

behaviorally defined feedback to the trainee, and establish realistic goals to aid in the trainee's

professional development.

While identifying items for clinical assessment can improve the supervision process, future

research is needed to validate a clinical supervision instrument. Although a variety of clinical

supervision instruments exist, no instrument to date has been validated with Item Response

Theory (IRT) methods (e.g. Rasch analysis). The advantage of these methodologies compared to

classical validation methods is considerable. Rather than relying on a total score, IRT validated

instruments do not require the student to demonstrate each item and therefore is flexible enough

to accommodate all levels of students who train at various practicum/internship sites. Moreover,

an IRT validated instrument would arrange items within each competency domain on a hierarchy

from least to most difficult, thereby allowing supervisors to have an empirical understanding of

trainee development.












APPENDIX A
DELPHI ROUND 1 SURVEY


Dear Rehabilitation Counseling Educators,

The Rehabilitation Science Program at the University of Florida is conducting a research study titled, "Identifying
Benchmark Competency Crteria for a Rehabilitation Counseling Clinical Supervision Instrument: A Delphi Study."
Clinincal supervision is vital to professional gatekeeping and has emerged as an independent entity with its own
theories, standards, and research. Despite the importance of the supervision process, various accrediting bodies and
standards of practice provide little guidance regarding specific competency criteria for trainee evaluation, thus
rehabilitation counseling programs have considerable attitude when evaluating graduate level trainees. This
discrepancy complicates trainee evaluation/remediation and may hinder supervisors from identifying underdeveloped
students in rehabilitation counseling programs. Therefore, the purpose of this study is to aid rehabilitation counseling
educators and trainees during clinical supervision by identifying and reaching consensus on specific counseling skL Is,
personal attributes, and professional behaviors that are usefuliappropriate for clinical supervision evaluation.
Rehabilitation counseling educators agreeing to participate will complete an online, three round Delphi study.


Participants will be asked to complete a three round, online Delphi study. The Delphi procedure allows researchers to
elicit and converge anonymous expert opinion by providing systematic feedback through a series of rounds. The first
round of the Delphi study is included as part of this initial survey. This round asks participants to provide an
exhaustive list of specific counseling skills, personal attrbutes, and professional behaviors that would be
"useful/approprate" for a clinical supervision evaluation instrument. Once the Round 1 data has been collected and
analyzed, the participants will receive a second round of questions where they simply rate the
usefulness/appropriateness of all responses identified from the first round on a five point Likert scale. After the
Round 2 ratings have been collected and analyzed (means and standard deviations), the Round 3 questionnaire will
be developed. Round 3 of the DIephi will inc ude the overall group calculations, as well as the participant's original
ratings (participants will not receive other participant's personal ratings, only their own). Participants will then
compare their original ratings against the group statistics and have the opportunity to re-rate their original
responses (Round 3). Upon completion of the Round 3 ratings, final responses will be calculated and compared to
pre-determined consensus criteria.


Th s study involves collecting the expert opinion of rehabilitation educators who supervise graduate students during
their practicum and/or internship. If you agree to take part in this study, you will be asked to participate in an
anonymous, 3 round Delphi study. It is anticipated that the total amount of time to complete all 3 rounds of the
De phi will not exceed 3 hours (Round 1 1.5 hours; Round 2 1 hour; Round 3 .5 hours). However, when
considering the time associated with recruiting participants, analyzing the data, and allowing participants time to
complete each round, the entire Delphi process may take up to 3 months, There are no risks or discomforts by
participating on this study and participants may withdraw at any time without consequence. There will be no
compensation offered for your participation. Although there will be no direct benefits for participating in this study,
rehabilitation counseling educators will contribute to the greater good of the profession by helping to improve the
clinical supervision assessment process and aiding in the professional development of trainees.

Additionally, only authorized persons from the University of Florida involved in this research study have tne legal
rights to review the research records and will protect the confidentiality of those records to the extent by law or
court order. If the results of the research are published or presented, all expert identities will remain anonymous. For
questions regarding this study, please contact Michael Moorhouse, MHS, CRC at (352) 273-6491 or Linda Shaw,
Ph.D at (352) 273-6045. If you have any questions regarding your rights as a research subject, you may contact
tne Institutional Review Board (IRB) office at (352) 392-0433.

In order to be eligible for this study, participants must satisfy three inclusion criteria. First, educators should have
supervised a minimum of 15 rehabilitation counseling students during practicum or internship. Next, educators should
have supervised at least three semesters of practicum or internship. F.na Iv educators should have supervised at
least three students during practicum or internship within the last three years.


* By signing your name below (electronically), you acknowledge that you have read
and understood the informed consent and wish to proceed with the survey. If
however you decide that you no longer wish to participate now or at anytime during
the survey, you may exit the survey without consequence.











* Please enter your e-mail address (required)


Please identify your position

Q Assistant Piofessor/Tenured Track
O Assistant Professor/Non-Teinured Track
Q Associate Professor


O Professor Emeritus
O Aufiict Faculty
O Other


O l Professor

Please identify the region your school is associated with.


Q Region I (Assumption, Boston, Salve Regina, Springfield,
U. of Maine-Farmrington. U. Mass-Boston, U. of Souther Maine)
Q Region II (Cornell, Hilbert, Mofstra, Hunter, SUNY-Albany,
SUNY-Buffalo, St. thneis, Syracuse, U. of Medicine & Denistry of
hJ, U. Puerto Rico, Pontifcal Catholic U. of Puerto Rico)
0 Region III (Coppin St., East Stroudsberg, George
Washington, Pennsylvanis State, U. Maryland, U. Maryland
Eastern Shore, U. of Pittsburgh, U. Scranton, Virginia
Commonwealth, West Virginia)
0 Region IV (Auburn. East Carolina, Florida Atlantic. Florida
State, Fort Valley State, Georgia State, Jackson State,
Mississippi St., North Carolina Agriculturat & Technical St., South
Carolina St., Thomas, Troy State, U. Alabama, U. Florida, U.
Kentucky, U. Memphis, U. North Carolna-Chapal Hill. U. North
FTorida, U. of South Carolina, U. South Florida, U. Tennessee,
Winston-Salem St.)


O Region VI (Arkansas St., Arkansas Tech., East Central,
Langstan, Louisiana St., Southern, Stephen F. Austin St., Texas
Tech, U. Arkansas-Fayettevlle, U. Arkansas-Little Rock, U. North
Texas, U. Oaklahoma, U.Texas-Austin, U. Texas-Pan American,
U. Texas Southwestern Medical Center, Western New Mexico)
Q Region VII (Drake, Emporia St., Maryvile U. St. Louis, U
Iowa, U. Missouri)
Q Region VIII (Montana St.-Billings, U. Northern Colorado,
Utah St.)

O Region IX (Cal St.-Fresno. Cal St.-Los Angeles Cal St.-
Sacramento, San Diego St., San Francisco St., U. Arizona, U.
Hawaii-Manoa)
O Region X (Portland St., U. Idah,. Western Oregon, Western
Washington)


Q Region V (Bowling Green, Illinois Institue of Technology,
Kent St., Hichigan St., Minnesota St.-Mankato, Northeastern
Illinois, Northern Illinois, Ohio St., Saint Cloud St., Southern
Illnois-Carbondale, U. Illnois, U. Wisconsin-Madison, U.
Wisconin-Milwaukee, U Wisconsi-Stout, Wayne St., Western
Michigan. Wilberforce, Wright St.)

Please identify all your certifications and licenses.


D] CAP
fl CCH

SCDMS
SCLCP

OthCRC
Other (please specify)
I


0L4HC
O LPC
EJnce


Please identify the total number of master's level rehabilitation counseling students
have you supervised during their practicum or internship?









Please list the total number of semesters you have supervised master's level
rehabilitation counseling students?
I I
Have you supervised at least 3 master's level rehabilitation counseling students
during practicum or internship within the last 3 years?
OC)v oN

Think about some of your previous students over the years who demonstrated
either exceptional or poor counseling ability. With these students in mind, please
write an exhaustive list of specific counseling skills (basic and advanced), trainee
characteristics, abilities, or attributes that would be useful or appropriate for
evaluating a rehabilitation counseling trainee during clinical supervision.

In addition to your responses, please feel free to add any comments or elaborate on
any of your responses. Your comments will remain anonymous but may be included
in the Round 2 survey to help clarify items and/or aid other participants in their
decision making.









APPENDIX B
ROUND 1 EMAIL TO REHABILITATION COUNSELING EDUCATORS

Dear Rehabilitation Counseling Educators,

The Rehabilitation Science Program at the University of Florida is pleased to announce a new
research study titled, "Identifying Benchmark Competency Criteria for a Rehabilitation
Counseling Clinical Supervision Instrument: A Delphi Study." As part of this study, we are
soliciting the opinions of Rehabilitation Counseling educators who currently supervise or have
previously supervised master's level trainees during practicum and/or internship. More
specifically, we are looking to identify and draw consensus on an extensive list of counseling
skills, behaviors, and personality traits that clinical supervisors would find useful and/or
appropriate for a Rehabilitation Counseling clinical supervision instrument.

If you agree to take part in this study, you will be asked to participate in an anonymous, 3 round
Delphi study. There are no risks or discomforts by participating on this study and participants
may withdrawal at any time without consequence. There will be no compensation offered for
your participation. Although there will be no direct benefits for participating in this study,
rehabilitation counseling educators will contribute to the greater good of the profession by
helping to improve the clinical supervision assessment process and aiding in the professional
development of trainees.

The research team has been approved by both the National Council of Rehabilitation Education
and The University of Florida Institutional Review Board. This research project has obtained
permission to gain access to the NCRE membership for purposes of furthering the mission of the
association. Additionally, only authorized persons from the University of Florida involved in
this research study have the legal rights to review the research records and will protect the
confidentiality of those records to the extent by law or court order. If the results of the research
are published or presented, all expert identities will remain anonymous.

In order to be eligible for this study, participants must satisfy three inclusion criteria. First,
educators should have supervised at least 15 rehabilitation counseling students during practicum
or internship. Next, educators should have supervised at least three semesters of practicum or
internship. Finally, educators should have supervised at least three students during practicum or
internship within the last three years.

Your participation in this study is greatly appreciated. We understand your time limitations as a
Rehabilitation Counseling educator and seek to conduct this study in a thorough and efficient
manner. It is anticipated that the total amount of time to complete all 3 rounds of the Delphi
will not exceed 3 hours (Round 1 1.5 hours; Round 2 1 hour; Round 3 .5 hours). However,
when considering the time associated with recruiting participants, analyzing the data, and
allowing participants time to complete each round, the entire Delphi process may take up to 3
months.



If you are interested in participating, you may access the first round of the Delphi study at:









https://www.surveymonkey.com/s.aspx?sm=4263bzaxK7LtvuCwmk5 pg_3d_3d. For questions
regarding this study, please contact Michael Moorhouse, MHS, CRC at (352) 273-6491 or Linda
Shaw, PhD at (352) 273-6045. If you have any questions regarding your rights as a research
subject, you may contact the Institutional Review Board (IRB) office at (352) 392-0433.











APPENDIX C
INFORMED CONSENT


This study involves collecting the expert opinion of rehabilitate on educators who supervise graduate students during
their practicum and/or internship. If you agree to take part in this study, you will be asked to participate in an
anonymous, 3 round Delphi study. It is anticipated tnat the total amount of time to complete all 3 rounds of the
Delphi will not exceed 3 hours (Round 1 1.5 hours; Round 2 1 hour; Round 3 .5 hours). However, when
considering the time associated with recruiting partic pants, analyzing the data, and allowing participants time to
complete each round, the entire Delphi process may take up to 3 months, There are no risks or discomforts by
participating on this study and participants may withdraw at any time without consequence. There will be no
compensation offered for your participation. Although there will be no direct benefits for participating in this study,
rehabilitation counseling educators will contribute to the greater good of the profession by helping to improve the
clinical supervision assessment process and aiding in the professional development of trainees.

Additionally, only authorized persons From the University of Florida involved in this research study have the legal
rights to review the research records and will protect the confidentiality of those records to the extent by law or
court order. If the results of the research are published or presented, all expert identities will remain anonymous. For
questions regarding this study, please contact Michael Moorhouse, MHS, CRC at (352) 273-6491 or Linda Shaw,
Ph.D at (352) 273-6045. If you have any questions regarding your rights as a research subject, you may contact
the Institutional Review Board LRBI office at (352) 392-0433.

In order to be eligible for this study, participants must satisfy three inclusion criteria. First, educators should have
supervised a minimum of 15 rehabilitation counseling students during practicum or internship. Next, educators should
have supervised at least three semesters of practicum or internship. Fina Iv educators should have supervised at
least three students during practcum or internship within the last three years.

* By signing your name below (electronically), you acknowledge that you have read
and understood the informed consent and wish to proceed with the survey. If
however you decide that you no longer wish to participate now or at anytime during
the survey, you may exit the survey without consequence.









APPENDIX D
ROUND 1 FRIENDLY REMINDER EMAIL

Dear Rehabilitation Counseling Educators,

This is a remind email to follow up on a research study being conducted by the Rehabilitation
Science Program at the University of Florida titled, "Identifying Benchmark Competency
Criteria for a Rehabilitation Counseling Clinical Supervision Instrument: A Delphi Study." As
part of this study, we are soliciting the opinions of Rehabilitation Counseling educators who
currently supervise or have previously supervised master's level trainees during practicum and/or
internship. More specifically, we are looking to identify and draw consensus on an extensive list
of counseling skills, behaviors, and personality traits that clinical supervisors would find useful
and/or appropriate for a Rehabilitation Counseling clinical supervision instrument.

If you agree to take part in this study, you will be asked to participate in an anonymous, 3 round
Delphi study. There are no risks or discomforts by participating on this study and participants
may withdrawal at any time without consequence. There will be no compensation offered for
your participation. Although there will be no direct benefits for participating in this study,
rehabilitation counseling educators will contribute to the greater good of the profession by
helping to improve the clinical supervision assessment process and aiding in the professional
development of trainees.
The research team has been approved by both the National Council of Rehabilitation Education
and The University of Florida Institutional Review Board. This research project has obtained
permission to gain access to the NCRE membership for purposes of furthering the mission of the
association. Additionally, only authorized persons from the University of Florida involved in
this research study have the legal rights to review the research records and will protect the
confidentiality of those records to the extent by law or court order. If the results of the research
are published or presented, all expert identities will remain anonymous.

In order to be eligible for this study, participants must satisfy three inclusion criteria. First,
educators should have supervised at least 15 rehabilitation counseling students during practicum
or internship. Next, educators should have supervised at least three semesters of practicum or
internship. Finally, educators should have supervised at least three students during practicum or
internship within the last three years.

Your participation in this study is greatly appreciated. We understand your time limitations as a
Rehabilitation Counseling educator and seek to conduct this study in a thorough and efficient
manner. It is anticipated that the total amount of time to complete all 3 rounds of the Delphi
will not exceed 3 hours (Round 1 1.5 hours; Round 2 1 hour; Round 3 .5 hours). However,
when considering the time associated with recruiting participants, analyzing the data, and
allowing participants time to complete each round, the entire Delphi process may take up to 3
months.




If you are interested in participating, you may access the first round of the Delphi study at:









https://www.surveymonkey.com/s.aspx?sm=4263bzaxK7LtvuCwmk5 pg_3d_3d. For questions
regarding this study, please contact Michael Moorhouse, MHS, CRC at (352) 273-6491 or Linda
Shaw, PhD at (352) 273-6045. If you have any questions regarding your rights as a research
subject, you may contact the Institutional Review Board (IRB) office at (352) 392-0433.









APPENDIX E
AUDIT TRIAL

May 29, 2008

Combined the terms/phrases "empathy," "empathetic," "ability to convey empathy to clients,"
"conveying empathy," "feeling," and "empathetic," into "Empathetic / Convey empathy to
clients"

Combined the terms/phrases "rapport building," "building rapport," and "develop rapport" into
"Build rapport"

Split the phrase "engaging the client in the counseling process through attending and orienting"
into two sections: "engaging the client in the counseling process through attending" and
"orienting"

Split the phrase "being able to educate clients regarding what to expect out of the session and
any referral information" into "being able to educate clients regarding what to expect out of the
session" and "educates client regarding referral information"

Combine the terms/phrases "orienting," "explains the counseling process to client," and "being
able to educate clients regarding what to expect out of the session" into "orient client to the
counseling process"

Split "basic attending and observation skills" into two sections: "basic attending skills" and
"observation skills"

Combined the terms/phrases "basic attending skills," "listener," "good attending," "listening and
responding," "engaging the client in the counseling process through attending," "demonstrating
active listening," "attending skills," "active listening," and "attending skills training" into
"Active listening (e.g. attending to client; listening and responding)"

Combined the terms/phrases: "excellent observation skills," "attention to detail," "observation
skills" and "good observational skills" into "Observational skills/attention to detail"

Combined the terms/phrases "non-verbal behavior" and "trainees must know how to use good
non-verbal skill: eye contact, body position, attentive silence, voice tone, gestures, facial
expressions, physical distance, touch" but took out "attentive silence" because it was not
physical

Combined the terms/phrases "attentive silence," "appropriate use of silence," "good uses of
silence," and "appropriately uses silence" into "Appropriately uses silence"

Combined the terms/phrases "knowing how to set boundaries appropriately," "appropriate
boundaries," and "set clear boundaries with client" into "Set clear boundaries with client"









Combined the terms/phrases "paraphrasing," "know how and when to paraphrase" and "ability to
paraphrase" into "Paraphrase client statements"

Interpret "non directive approaches to goal setting" as working with the client to set goals

Split the phrase "set attainable short and long term goals in accordance with client wishes" into
three sections: "set attainable goals," "set short and long term goals" and "set goals in
accordance with client wishes"

Combine the terms/phrases: "set attainable goals" and "establishes goals that are measurable and
realistic" into "Sets realistic, attainable goals"

Combined the terms/phrases: "identifies goals with client," "goal setting that reflects the client's
perspective on wants and needs," "set goals in accordance with client wishes" and "non-directive
approaches to goal settings" into "Identify goals that reflect the client's perspective"

Combined the terms/phrases "goal setting," "trainees need to know how to implement
constructive goal setting," "developing goals," "obtaining goals" and "set short and long term
goals" into "Identify short and long term goals"

Split the phrase "controls and explores own feelings and personal thoughts about clients to
increase understanding and to decrease interference in the counselor relationship" into "explores
own feelings and personal thoughts about clients to increase understanding and to decrease
interference in the counselor relationship" and "controls own feelings and personal thoughts
about clients to increase understanding and to decrease interference in the counselor
relationship"

Combined the terms/phrases "explores own feelings and personal thoughts about clients to
increase understanding and to decrease interference in the counselor relationship," "identified
personal values and beliefs that may impact on ability to fulfill role of rehabilitation counselor,"
"recognizes personally sensitive areas in counseling content and/or process," "awareness of their
own limitations, biases, and beliefs and how these impact the counseling relationship" and "lack
of self awareness and impact on others" into "Recognizes how their personal limitations, biases,
and beliefs impact the counseling relationship"

Split the phrase "recognition of the importance of self care and ready willingness to seek out
professional counseling for themselves when it is needed in order for them to function ethically
and professionally" into "recognition of the importance of self care" and "willingness to seek out
professional counseling for themselves when it is needed in order for them to function ethically
and professionally"

Split the phrase "actively keep in touch with and working on their own issues" into "actively
keep in touch with own issues" and "actively work on own issues"

Split the phrase "personally knowing their own baggage and dealing with it" to "personally
knowing with their own baggage" and "deal with own baggage"











Combined terms/phrases "actively keep in touch with and working on their own issues," "self-
awareness," engages in self exploration," "an awareness of when their own personal issues are
interfering with their ability to learn and practice in a professionally and ethically responsible
manner," "demonstrates personal insight," "personally knowing with their own baggage," and
"willingness to explore discomfort honestly" into "An awareness of when their own personal
issues interfere with their ability to learn and practice in a professionally and ethically
responsible manner"

Combined the terms/phrases "do self-work on issues that arise in counseling and supervision,"
"emotional stability," "actively work on own issues," "self-care," "ability to manage personal
issues or make determinations regarding whether they should be in a master's program at this
time in their lives," "keep own personal problems out of counseling session," and "deal with own
baggage" to "Manage/work on personal issues that arise during counseling or supervision
sessions"

Interpreted the phrase "an ability to demonstrate that their motivation for participating in
required clinical learning experiences is not simply that it is required for their degree but that it is
necessary for them in order to provide competent, professional, and ethical services for people
with disabilities" as "Demonstrates genuine motivation to become a competent and ethical
rehabilitation counselor"

Combined the terms/phrases "ability to show deep reflection of feelings," "facilitates client
expression of thoughts and feelings," "demonstrating understanding of the client's ideas,
experiences, and beliefs through reflection of feelings and meanings of client statements,"
"reflective listen," "reflection," "reflecting feeling," "reflecting meaning," "using reflections,"
"reflection," and "accurately reflect client feelings" into two phrases "Reflect feeling of client
statements" and "Reflect meaning of client statements"

June 1, 2008

Combine the terms/phrases "confrontation," "knowing when to use confrontation," "confront"
and "confrontation" to "knowing when to use confrontation"

Split the phrase "Uses supervision, including preparation, participation, and follow through" into
three sections "prepares for supervision," "participates in supervision," and "follows through
with supervision"

Split the phrase "Evaluates relevance, value, and meaning of feedback and implements changes
as needed" into two sections "evaluates relevance, value, and meaning of supervisory feedback"
and "implements feedback"

Combine the terms/phrases: and "follows through with supervision," "ability to use supervision
to improve their skills," "how they make use of that feedback," "implements feedback," and "an









ability to demonstrate how they have incorporated feedback into their practice" into
"Incorporates supervisor feedback into practice"

Combine the terms/phrases: "self-disclosing in a way that facilitates formation of an effective
counseling relationship and serves to further client exploration," "appropriate use of self-
disclosure," "using self-disclosure appropriately, and self disclosure" into "Uses self-
disclosure appropriately"

June 2, 2008

Combine the terms/phrases: "receptivity to clinical supervision," "acceptance of feedback,"
"willingness to improve skills," "willing to incorporate supervisors' feedback into their
practice," "openness to feedback," and "ability to take constructive feedback some students
become defensive" into "Non-defensive /receptive to clinical supervision"

Split the phrase "above average verbal and written expression" into two phrases "above average
verbal expression" and "above average written expression"

Combine the terms/phrases: "report writing skills," "adequate case reporting skills," "above
average written expression," "ability to write at a master's level, some students cannot write case
notes," and "preparation of written reports, progress notes, and correspondence, emphasizing
clarity, organization, and professional presentation" into "Preparation of written reports and
progress notes (accurate, clarity, organization, professional presentation)"

Combine the terms/phrases: "genuineness" and "genuine" into the phrase "genuine"

Interpret the phrase "ability to approach all their clinical learning experiences openly and
nonjudgmentally" as two personal characteristics exhibited during a counseling session: open
and non-judgmental; and one characteristic during clinical supervision: open (open to supervisor
feedback)

Place the phrase "open (open to supervisor feedback)" under the "Non-defensive /receptive to
clinical supervision" item

Combine the terms/phrases: "the ability to suspend preconceived notions and judgments of
human behavior," non-judgmental," "non-judgmental attitude," and "judgmental" into the term
"non-judgmental"

Split the phrase/terms: "an ability to use counseling skills intentionally and to flex when needed"
into two phrases "An ability to use counseling skills intentionally" and "Flex counseling skills
when needed"

Interpret "flex counseling skills when needed" as "be flexible with counseling skills when
necessary"









Split the phrase "open and closed questions" into two sections "open questions" and "closed
questions"

Combine the terms/phrases: "open question" and "open ended questions" into "Uses open ended
questions"

Combine the terms/phrases: "flexible," "flexible (mental and emotional)," "adaptability is one of
the most important attributes," and "a trainee needs to be able to adapt and change to make the
therapeutic relationship beneficial to the client" to "Flexible/adaptable"

Combine the terms/phrases: "closed questions" and "use close ended questions appropriately"
into "Use close ended questions (when appropriate)"

Combine the terms/phrases "empowering," and "empower the client," "self-detemination," and
"ability to form empowering relationships with clients" into "Empowers the client"

Combine the terms/phrases: "ethical conduct," "confidential," "ethical," "understands of ethical
issues, including, for example, confidentiality, representation of competence and qualifications,
loyalties, and informed consent," "demonstration of ethical behavior Showing up on site some
students do not realize this is required and some site managers allow this to go on," and
"understands ethics of counseling relationship" into "Understands/practices in ethical manner
(i.e. confidentiality, representation of competence/qualifications, dual relationships, informed
consent, appropriate boundaries)"

Combine the terms/phrases "showing up on site" and "being on time for practicum/internship"
into "

Combine the terms/phrases: "providing client with choices" and "identifies and explores options
with client" to "Identify and explore options with client"

Combine the terms/phrases: "minimal encouragers," "using confirmations," and "uses
affirmations/positive reinforcement" into "Uses positive reinforcement/affirmations"

Split the phrase "requests guidance as needed while demonstrating openness to experimentation
and risk taking" into two sections "requests guidance as needed" and "willing to experiment or
take risks in session"

June 3, 2008

Combine the terms/phrases: "individualizing treatment," "planning for interventions and services
that will assist the client in his or her goal," and "being able to treatment plan" into "Plan
interventions and services that will assist the client in his or her goal (i.e. individualized
treatment planning)"

Split the phrase: "assessing client traits, problems, and needs in a comprehensive and specific
manner that emphasizes strengths and deficits in behavioral terms" into two sections "assess









client traits in a comprehensive and specific manner that emphasize strengths and deficits in
behavioral terms" and "assess client problems in a comprehensive and specific manner"

Split the phrase: "a trainee should be aware of each client's individual characteristics and
problems to be able to guide the therapeutic relationship to fit the client's needs" into two
sections "a trainee should be aware of each client's individual characteristics" and "a trainee
should be aware of each client's individual problems"
Combine the terms/phrases: "assess client traits in a comprehensive and specific manner that
emphasize strengths and deficits in behavioral terms," "a trainee should be aware of each client's
individual characteristics," "assess client's strengths and weaknesses" into the phrase "Assess
client strengths and weaknesses"

Combine the terms/phrases: "assess client problems in a comprehensive and specific manner," "a
trainee should be aware of each client's individual problems," and identifyy client
problems/issues" into "Identify client problems"

Combine the terms/phrases: "demonstrating cultural sensitivity," "acceptance of difference,"
"becoming cultural proficient," and "awareness of cultural differences/influences" into "Aware
of cultural differences/influences"

Combine the terms/phrases: "understanding the impact of disability" and "knowledge of
disability and its myriad effects" into "Understanding the impact of disability"

Combine the terms/phrases: "structuring session to yield success" and "ability to continue to
move session toward some therapeutic outcome" to "Continues to move session toward the
therapeutic goal"

Combines the terms/phrases: "avoids using heterosexist, ablest, racist, and sexist language in
interactions with other students, supervisors, clients, and other stakeholders in rehabilitation" and
"use person first language" into "Uses appropriate language/terminology (e.g. person first
language; avoids heterosexist, ablest, racist, and sexist language)"

Combines the terms/phrases: "demonstrates respect for clients and their point of view" and
"refrains from imposing personal values" to "Demonstrates respect for client's values/beliefs"

Combine the terms/phrases: "Summarizing process dynamics, themes, activities, and milestones
in the counseling relationship," "summarize," and "summarizing" into "Summarize"

Combine the terms/phrases: "provide feedback" and "provide accurate and honest feedback" into
"Provide accurate feedback"

Combine the terms/phrases: "ask effective questions" and "know how and when to use
questions" into "Ask effective questions"

Combine the term/phrases: "gives an accurate and balanced self-assessment of professional
competencies," "development of the ability to self-critique accurately," and "assesses own









strengths and weaknesses" into "Identifies own strengths and weakness related to counseling
performance/competency"

Combine the term/phrases: "conceptualize the client's life (i.e. psychosocial, vocational, etc.)"
and conceptualizess client from all aspects" into "Conceptualize the client's life (i.e.
psychosocial, vocational, etc.)"

Split the phrase "ability to apply theories and techniques appropriately to the client's situation"
into two sections: "ability to apply theory to a client's situation" and "ability to apply techniques
based in theory to the client's situation"

Combine the terms/phrases "ability to apply theory to a client's situation" and "working within a
theoretical approach" into "Working within a theoretical approach"

Combine the terms/phrases "ability to apply techniques based in theory to the client's situation,"
"specific counseling techniques based on a chosen theory," and "utilize simple techniques
grounded in counseling theory" into "Utilize simple techniques grounded in counseling theory"

Combine the terms/phrases "ability and willingness to research and explore resources (i.e. books,
journal articles, videos, training, etc.) that help the practitioner with ideas for techniques and
approaches" and "keep up to date on publications" into "Willing to research and explore
counseling resources (i.e. journal articles, books, etc.)"

Combine the terms/phrases "open" and "broad minded" into "Open minded"

Combine the terms/phrases "willing to help" and "altruistic" into "Altruistic"

Combine the terms/phrases "calm" and "soothing" into "Calming/soothing"

Combine the terms/phrases: "intuitive" and "perceptive" into "Intuitive/perceptive"

Combine the terms/phrases: "kind" and "thoughtful" into "Kind/thoughtful"

Combine the terms/phrases: "common sense" and "common sensical" into "Common sense"

Combine the terms/phrases" "prudent" and "conscientious" into "Conscientious/prudent"

Combine the terms/phrases: "intelligent" and "wise" into "Intelligent/wise"

Interpret the phrase "limit setting" as "establishes appropriate boundaries with clients"

Include "establishes appropriate boundaries with client" under the "Understands/practices in
ethical manner (i.e. confidentiality, representation of competence/qualifications, dual
relationships, informed consent)" item to form ""Understands/practices in ethical manner (i.e.
confidentiality, representation of competence/qualifications, dual relationships, informed
consent, establishes appropriate boundaries)"










Split the phrase "Able to convey an appropriate level of ease when they're working with clients,
colleagues, and supervisors" into two sections "Able to convey an appropriate level of ease with
clients" and "Able to convey an appropriate level of ease with colleagues and supervisors"

Combine the phrases "comfortable with client" and "able to convey an appropriate level of ease
when they're working with clients" into "comfortable with client"

Split the phrase "confidence in approaching any interpersonal session with clients, site
supervisor, or faculty supervisors" into "Confident in approaching any interpersonal session with
clients" and "Confident in approaching any interpersonal session with site or faculty supervisor"

June 4, 2008

Combine the items: "Builds rapport," "Active listening (e.g. attending to client; listening and
responding)," "Orient client to the counseling process," "Non verbal skills (e.g. eye contact,
body position, voice tone, gestures, facial expressions, physical distance, and appropriate
touch)," "Seamless delivery of the counseling skills," "Recognizes how their personal
limitations, biases, and beliefs impact the counseling relationship," "Set clear boundaries with
client," "Educates client regarding referral information," "Demonstrates respect for client's
values/beliefs," "Controls own feelings and personal thoughts about clients to increase
understanding and to decrease interference in the counselor relationship," "Maintains awareness
of personal feelings, limitations, and experiences during the counseling session," "Fosters a
therapeutic environment," "An ability to use counseling skills intentionally/purposefully,"
"Flexible with counseling skills (when necessary)," "Avoids countertransference," "Focuses on
client rather than themselves in counseling sessions," "Identifies and meets the client's needs,"
"Establishes a working alliance," "Continues to move session toward the therapeutic goal,"
"Clearly explains limits of confidentiality to client," "Manages the time of the session
appropriately," "Closes cases appropriately," "Develops trust," "Faith in intuitive responses,"
"Ability to be objective (lose their subjective thought)," "Keeps client focused," "Reflects on the
counseling session," "Functions at any level of the helping relationship, from limited to intense,"
"Confidence in approaching any interpersonal session with clients," "Use multiple techniques
(i.e. eclectic style)," "Takes good notes during session," and "Applies structure to the counseling
process (e.g. rapport building, exploration, treatment, termination)" into the
category/competency domain "General Counseling Process"

Combine the items: "Assess client interests, skills, and aptitudes," "Knowledge of career
development," "Ability to conceptualize," "Conceptualize a case within a theoretical frame of
reference," "Diagnose (with some assistance)," "Clear understanding of theory(s)," and
"Understanding the impact of disability" into the category/competency domain
"Assessment/Conceptualization"

Combine the items: "Observation skills," "Use and understand solution skills (giving advice,
information, and directive)," "Appropriately uses silence," "Paraphrase client statements,"
"Reflect feeling of client statements," "Reflect meaning of client statements," "Interpret client
statements," "Assess accuracy of interpretations with the clients," "Clarify client statements,"









"Appropriate use confrontation," "Disagreeing as needed, while maintaining a respectful and
supportive attitude," "Uses self-disclosure appropriately," "Utilize simple techniques grounded
in counseling theory," "Utilize advanced techniques grounded in counseling theory," "Working
within a theoretical approach," "Interviewing skills," "Group counseling skills," "Use open
ended questions," "Use close ended questions (when appropriate)," "Uses
confirmations/affirmations," "Ability to change client's emotional arousal levels," "Critical
thinking ability," "Reframes client statements," "Enhance or help client to increase his or her
self-esteem," "Avoids sympathy," "Avoids advice," "Ability to be insightful," "Summarize,"
"Provide accurate feedback," "Asks effective questions," "Intervenes in a ways that produce
client progress in achieving his or her behavioral and/or emotional goals," "Assesses client
motivation," "Elicits motivating statements from client," "Allows client to elaborate," "Assess
client's self-efficacy," "Ability to handle client's strong emotions (e.g. crying)," and "Redirects
client (when appropriate)" into the category/competency domain "Process skills"

Combine the items: "Identify goals that reflect the client's perspective," "Identify short and long
term goals," "Sets realistic, attainable goals," "Job development and placement skills,"
"Providing client with choices," "Identify and explore options with client," "Individualized
treatment planning (i.e. plan interventions and services that will assist the client in his or her
goal)," "Gather pertinent information from the client and relevant others through assessments,
observations, and facilitative questions," "Assess client strengths and weaknesses," "Identify
client problems," "Prioritize client problems," "Conceptualize the client's life (i.e. psychosocial,
vocational, etc.)," "Understanding of and ability to develop accommodation strategies,"
"Understanding of family and interpersonal dynamics," and "Regularly evaluates client
progress" into the category/competency domain "Identifying goals/Treatment planning"

Combine the items: "Evaluates relevance, value, and meaning of supervisory feedback," "Sets
personal learning goals," "Non-defensive /receptive to clinical supervision," "Prepares for
supervision," "Participates in supervision," "Incorporates supervisor feedback into practice,"
"Demonstrates respect for the expertise of supervisors," "An understanding of and belief in the
importance of supervision," "An awareness of when to seek supervision," "Ability to state a
disclosure statement," "Ability to write a disclosure statement," "Preparation of written reports
and progress notes (accuracy, clarity, organization, professional presentation)," "Ability to
reflect on one's practice," "Requests guidance as needed (during supervision)," "Balance the
needs of the faculty supervisor and site supervision," "Willing to experiment or take risks in
session," "Identifies own strengths and weakness related to counseling
performance/competency," "Willing to research and explore counseling resources (i.e. journal
articles, books, etc.)," and "Confident in approaching any interpersonal session with site or
faculty supervisor" into the category/competency domain "Supervision"

Combine the items: "Respects other professionals," "Demonstrates genuine motivation to
become a competent and ethical rehabilitation counselor," "Commitment to social justice,"
"Understands/practices in ethical manner (i.e. confidentiality, representation of
competence/qualifications, dual relationships, informed consent)," "Understands the challenges
to communication with persons with hearing, visual, or cognitive impairments," "Aware of
cultural differences/influences with clients," "Deals directly and appropriately with conflict
rather than avoiding it," "Ability to recognize when they are responding to supervisors, clients,









students and other professionals in a manner that puts their own needs before those of their
clients," "Uses appropriate language/terminology (e.g. person first language; avoids heterosexist,
ablest, racist, and sexist language)," "Recognizes limits of competency," "An ability to engage in
critical self-reflection rather than focusing solely on the mistakes of others when they participate
in clinical training, practicum and internship experiences," "Able to convey an appropriate level
of ease with colleagues and supervisors," "Appropriate dress," and "Willingness to change when
their own biases and beliefs that interfere with their ability to learn how to function as competent
and ethical rehabilitation counselor" into the category/competency domain "Professional
Behavior"

Combine the items: "Sets career goals," "Evaluates progress toward own goals," "Develop
ability to supervise others," "Above average verbal and written expression," "Willingness to
continue to learn and develop professionally," "An ability to articulate why the counseling skills
they learn are crucial to all the roles and functions (e.g., vocational rehabilitation counselor, case
manager, job developer, vocational evaluator) of rehabilitation counselors," and "Goal oriented"
into the category/competency domain "General Professional Development"

Combine the items: "Recognition of the importance of self care," "An awareness of when their
own personal issues are interfering with their ability to learn and practice in a professionally and
ethically responsible manner," "Manage/work on personal issues that arise during counseling or
supervision sessions," "Willingness to seek out professional counseling for themselves when it is
needed in order for them to function ethically and professionally," "Emotional stability," and
"Manage own mental health disabilities" into the category/competency domain "Self Care"

Combine the items: "Suicide risk assessment," "Manages resources," "Fiscal management,"
"Time management," and "Crisis management" into the category/competency domain "Other"

Combine the items: "Empathetic / Convey empathy to clients," "Dependable," "Demonstrates
unconditional positive regard," "Genuine," "Warm," "Open minded," "Non-judgmental,"
"Flexible/adaptable," "Non-threatening demeanor," "Patient," "Honest," "Trustful," "Humble,"
"Altruistic," "Motivating," "Energetic," "Assertive," "Calming/soothing," "Positive,"
"Attentive," "Humorous," "Respectful," "Caring," "Direct," "Gentle," "Creative,"
"Intelligent/wise," "Logical," "Experienced," "Confident (but not arrogant)," "Kind/thoughtful,"
"Mature," "Comfortable with client," "Non-defensive (with clients)," "Common sense,"
"Reliable," "Conscientious/prudent," "Intuitive/perceptive," "Extroverted," "Introverted,"
"Thoughtful," "Capable," "Approachable," "Ebullient/happy," "Hard-working,"
"Moderating/judging," "Timely/opportune," "Just," "Succinct," "Pleasure to work with," "Team
player," and "Healthy" into the category/competency domain "Personal Characteristics"

June 5, 2008

Combine the items: "Collaborates with client" and "Establishes a working alliance" into
"Establishes a working alliance"

Move the phrase "Recognizes how their personal limitations, biases, and beliefs impact the
counseling relationship" into the Professional Behavior Domain.









Move the items: "Assesses client motivation" and "Assess client's self-efficacy" from Process
Skills to Assessment/Conceptualization

Move item: "Intervenes in a ways that produce client progress in achieving his or her behavioral
and/or emotional goals" from Process Skills to Identifying goals/Treatment planning

Move the items: "Gather pertinent information from the client and relevant others through
assessments, observations, and facilitative questions" "Assess client strengths and weaknesses"
"Identify client problems" "Prioritize client problems" "Conceptualize the client's life (i.e.
psychosocial, vocational, etc.)" and "Understanding of family and interpersonal dynamics" from
Identifying goals/Treatment planning to Assessment/Conceptualization

Combine the terms/phrases: "Ability to state a disclosure statement" and "Ability to write a
disclosure statement" into one statement "Ability to state and write a disclosure statement"

Move the phrases: "Ability to state and write a disclosure statement" and "Willing to research
and explore counseling resources (i.e. journal articles, books, etc.)" from Supervision to General
Professional Development

Move the phrases: "Recognizes limits of competency" and "An ability to engage in critical self-
reflection rather than focusing solely on the mistakes of others when they participate in clinical
training, practicum and internship experiences" from Professional Behavior to General
Professional Development

Move the phrase: "Goal oriented" from General Professional Development to Personal
Characteristics

Combine the terms: "Dependable" and "Reliable" into "Dependable/reliable"

Combine the terms: "Non-threatening demeanor" and "Approachable" into "Approachable/Non-
threatening demeanor"

Combine the terms: "Kind," "Thoughtful," and "Kind/thoughtful" into the phrase
"Kind/thoughtful/caring"









APPENDIX F
ROUND 2 EMAIL TO REHABILITATION COUNSELING EDUCATORS

Dr. XXXXX,

Thank you for participating in round 1 of our clinical supervision Delphi study. The data has
been analyzed and we have identified 184 items representing 10 theoretical competency
domains. Per the Delphi format, we ask that you complete the second round of questioning
whereby you rate the "usefulness" of each item on a five point Likert scale. In order to remain
consistent with the purpose of this study, please read the instructions page before rating the
items.

In addition, we know that your time is valuable and we really appreciate your contribution to this
project. However, because we are on a limited time schedule we would really appreciate if you
could complete the second round by FRIDAY, JUNE 20th. Educators who pilot tested this round
completed it in approximately 30 minutes.

Survey Link: https://www.surveymonkey.com/s.aspx?sm=ShzVwljPeW1wOgt9VkLo_2bg_3d_3d

If you have any questions, please contact Michael Moorhouse at mmoorhou@phhp.ufl.edu or
352-273-6491.

Thank you,

Michael Moorhouse

Rehabilitation Science Doctoral Student

University of Florida

Department of Behavioral Science & Community Health










APPENDIX G
DELPHI ROUND 2 SURVEY

PLEASE READ THE FOLLOWING INSTRUCTIONS BEFORE COMPLETING ROUND 2
The purpose of this research study is to identify and draw consensus on a comprehensive set of skills, behaviors,
and personal attributes which are "USEFUL" for clinical supervision evaluation.
Specifica Iv for this study, we define a "USEFUL" item as any item tnat is representative oF some aspect of
rehabilitation counseling performance and could potentially help the supervisor evaluate the trainee. You will rate
each item on a 5 point Likert scale ranging from "strongly disagree" to "strongly agree."
We are emphasizing that you are rating tie item based on whether you believe the item COULD be used by clinical
supervisors to evalaute a trainee. For example, you may believe that students who are in their first practicum or
working at a particular site should not be required to "conceptualize a client within a theoretical framework."
However, you recognize that if the trainee did demonstrate this ability, the supervisor COULD use this information
to evaluate the tranee's clinical ability. Therefore, in this instance, you would agree that the item s "USEFUL" for
clinical supervision evaluation.

In addition to rating the items, there will be a section on the survey for any comments you may have regarding tne
items. These comments may be included in the final round of the Delphi to help participants reevaluate their ratings.


1. Please rate whether you agree each item is "useful" for clinical supervision
evaluation.
Strongly Strongly
Disagree Neutral Agree St
isaprree Agree
Bu.lds rapport 0 0 0 0 0
Develops trust O 0 0 0 0
Activ, listening (e.g. attending to cliet; histenig and responding) O O O O O
aon verbal skills (e.g. eye contact, body position. voice tone. O O O O O
gestures, facial expressris. physical distance, and appropriate
touch)
Oriit clnt to t he counseling process CO O O O
Set clear boundaries with client O O O O O
Clearly explains limits of confidletlailty to client 0 0 0 0 0
Demonstrates respect for client's valuesi/bliefs ( 0 0 0
Fosters a therapeutic environment O 0 0 0 0
Establishes a working alliance O 0 O 0
Seamless delivery of the counseling skills O 0( 0 0 0
An ability to use c.uns.ein g skills intentionally/purposefully ) O 0 0 0
Flexible with counseling skills (when necessary) 0 0 0 0 0
Use multiple techniques (i.e eclectic style) 0 0 0 0
Avoids cou ntertransference O C O O O
Identifies and meets the client's needs ) C) C O O)
Applies structure to thle counselig process (e.g. rapport building., O O O
exploration, treatment, termination)
Continues to move session toward the therapeutic goal 0 0 0 0 C
Manages tie time of tih session appropriately 0O ) O C CD C
Maintains awareness of personal feelings, limitations, and CO O O C
experiences during the counseling session
Controls own feelings and personal Itoughts about clieits to CO O O O
increase, understanding and to decrease interference during the
counseling session
Has faith in intuitive responses ) 0 0 0 0
Ability to be objective (lose their subjective thought) 0 0 O O









Keeps client focused 0 0 0 0 0
Focusres a client rather than themselves in counseling sessions O 0 C
Functions at any level of the helping relationship, from limited to ) O O O O
intense
Confident in approac ing any iintrpersonal session with dliets 0 0 0 O 0
Takes good notes during session 0 0 0 0 0
Educates client reardiing referral informati n 0 O O O
Clses cases appropriately 0 0 0 0
Reflects on the counseling session 0 0 O
Comments









* 2. Please rate whether you agree each item is "useful" for clinical supervision
evaluation.
Strongly Strongly
Disagree Nleutal Agree
Disagree Agree
Observataio skills 0 0 0 0 0
Use and understand solution skiMs (giving advice, Information, 0 0 0 0 0
and directive)
Paraphrase client statements 0 0 0 0 0
Reflect fec-ig of cli-et statements. 0 0 0 0
Reflect meaning of client statements O O O O O
Use open ended question 0 0 0 0 0
Us. close ended questloms (whieo appr oriate) 0 0 0 0
Reframes client statements 0 0 0 0 0
Interpret clwat statements 0 0 0 0 0
Assess accuracy of interpretations with the clients 0 0 0 0 0
Clarify cleat statements 0 0 0 0 0
Allows clent to elaborate 0 0 0 0
Uses confirmathiosi/alfrmations ) 0 0 0 0
Asks effective qusions. O O O O O
Redirects client (when appropriate) )0 0 0 0 0
S.ummariz 0 0 0 0 0
Provide accurate feedback ~ 0 0 0 0
Appropriately uses silence 0 0 0 0 0
Appropriate use confrontation 0 0 0 0 0
Ability to be Insightful 0 O O O 0
Elicits motivating statements from client C )0 0 0 0
Uses self-disclosure appropriately O ) ) C0 0 )
Ability to handle cldnt's strong emotions (e.g. crying) 0 0 0 0 0
Disagreeing as needed, while maintaining a respectful and 0 0 0 0 0
supportive attitude
Utilize simple tectlniques grounded in counseling theory 0 0 0 0 0
Utilize advanced techniques grounded in counseling theory 0 0 0 0
Working within a theoretical approach O O O 0)
Interview, g skills 0 0 0 0 0
Group counseling skills 0 0 0 0 0
Critical thinking ability O O O O 0
Abikty to chang client's emotional arousal levels 0 0 0 0 0
Enhance or help client to increase hs or her self-esteeam 0 0 0 0
Avoids sympathy 0 0 0 0 0
Avoids a.dvce 0 0 0 0 0
Comments
*-I









* 3. Please rate whether you agree each item is "useful" for clinical supervision
evaluation.


Gather pertinent information from the client and relevant others
through assessments, observations, and facilitative questions
Asssss clint self-efficacy
Assess client moetivalin
Assess clint strengths and weaknesses
General ability to concept lize
Conceptualize a case within a theoretical frame of reference
Conceptualize a client's life (i.e. psychosocial, vocational, etc)
Identity cient problems
Prriritize chakdt problems
Diagnose (with some assistance)
Clear understanding of theory(s)
Knowledge of career development
Understanding of famlly and interpersonal dynamics
Understanding the impact of disability
Comments


Strongly
Disagree
0
0
0
0
0
0
0
0
0
0
0
0
0
0
a
a


Disagree
0
0
0





0
0
0
0
0
0
O0

O
O
O0


Neutral
0
0
0
0
0
0
0
0
0
0
0
0
0
0
O


Agree
0

0
0
0
0
0
0
0
0
0
0
0
0
O
O


S4. Please rate whether you agree each item is "useful" for clinical supervision
evaluation.


Identify goals that reflect the clike's perspective
Identify short and long term goals
Sets realistic, attainable goals
Identify and explore options with chant
Providing clhnt with choices
Intervenes in a ways that produce client progress in achieving his
or her behavioral and/or emotional goals.
Individualized treatment planning (i.e. plan interventions and
services that will assist the chant in his or her goal)
Understanding of and ability to develop accommodation
strategies
Job development and placement skills
Regularly evaluates client progress
Comments


Strongly
Disagree
0
0
0
0
0
0
0
0
0
0
O


O
O


Disagree Neutral


Strongly
Ag ree


0
0


0
0
0
0
0
0
0
0


Strongly
gr Agree
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0


I










S5. Please rate whether you agree each item is "useful" for clinical supervision
evaluation.
Strongly Strongl~
Disagree Neutral Agree
Disagree Agree
An understanding of and blieff In the importance of supervision 0 O O O O
Deonstrates respect for the expertise of supervisor 0 0 0 0 0
Confident in approaching any interpersonal session with site or C O) 0C i
faculty supervisor
Non-dtefesive / receptive to clinical supervision C O 0C C0 03
repare-s for supervision 0 O 0 O 0
Participates in supervision 0 0 O O O
Ivaluatas relevance., alu., and meaning of supervisory feedback 0 0 0 0 0O
Incorporates supervisor feedback into practice 0 O O C O
An awareness of wie. to seek supervision O O 0 0
Requests guidance as needed (during supervision) C0 0 0 0 0
Preparation of written reports and progress notes (accuracy, CO 0O C0
clarity. organization, professional presentation)
Identifies own strengths and weakness related to counseling 0 0 0 0 0
performance/competency
Ability to reflect an one's practice 0 0 0 0 0
Man.agework on personal issues that arise during counseling or 0 O O O O
supervision sessions
Balance the needs of te faculty supervisor and sAit supervision C)0 0 0 0
Willing to research and eKplore counseling resources (ie. journal 0 0 0 0
articles, books, etc.)
Completes supervision tapes/reports on time 0 O 0 O 0
Sets personal learning goals 0 0 O O O
Comments
aI









* 6. Please indicate how much you agree that the item is "useful" for clinical


supervision evaluation.

Respects other professir ials
Demonstrates genuine motivation to become a competent and
ethical rehabilitation counselor
Commitment to social Iustic
Understands/practices in ethical manner (.e. confidentiality,
representation of competence/quallfications, dual relationships,
informed consent)
Understands the challenges to communication with persons with
hearing, viassl, or cognitive impairments
Aware of client's cultural drfferences/influences
Deals directly and appropriately with conflict rather thLa avoiding
it
Ability to recognize when they are responding to supervisors,
clients, students and other professionals in a manner that puts
their own needs before those of their clients
Uses appropriate languageterminology (e.g. person first
language; avoids hlterosexisit ablest, racist, and sexist
language)
Able to convey an appropriate level of ease with colleagues and
supervisors
Appropriate dress
Recognizes how their personal limitations, biases, and beliefs
impact the counseling relationship
Willingness to change when their own biases and beliefs that
interfere with their ability to lear how to function as competent
and etlhcal rehabilitatir n counselor
Comments


Strongly
Disagree
0
0
0
0

0
0
0
0
O


Disagree
0
0
0
0

0
0
0
0
O


O
O
O


0 0


Neutral
0
0
0
0

0
0
0
0
O

O

O
O


Agree
0
0
0
0

0
0
0
0
O


Strongly
Agree
0
0
0
0
0
0
0
0


0 0 0


* 7. Please indicate how much you agree that the item is "useful" for clinical


supervision evaluation.

Sets career goals
Evaluates progress toward own goals
Develop ability to supervise others
Above average verbal and written expression
Willingness to continue to learn and develop professionally
An ability to articulate why the counseling skills they learn are
crucial to all the roles and functions (e.g., vocational rehabilitation
counselor, case manage., job developer, vocational evaluator) of
rehabilitation counselors
Willimg to experiment or take risks in counseling session
Recognizes limits of competency
An ability to engage in critical self-reflection rather lhan focusing
solely on the mistakes of others when they participate in clinical
training, practcum and internship experiences
Ability to write and state a disclosure statement
Comments


Strongly
Disagree
0
0
0
0
0
0

0
0
0
O


Diagree
0
0
0
0
0
0

0
0
0
O


Neutral
0
0
0
0
0
0

0
0
0
O


Agree
0
0
0
0
0
0

0
0
0
O


Strongly
Agree
0
0
0
0
0
0

0
0
0
O
O

O

O


0 0 0 0 0










S8. Please indicate how much you agree that the item is "useful" for clinical
supervision evaluation.


Recognition of the impartarnce of self car
An awareness of when their own personal i sues are interfering
with their ability to learn and practice in a prafessionally and
ethically responsible manner
Willingness to seek out professional counseling for themselves
when it ks needed In order for them to function ethically and
professionally
Demonstrates emotional stability
Manages own metal health disabilities
Comments


Strongly
Disagree
0
0
O


Disagree
0
0
O


Neutral
0
0


Agree
0
0


Strongly
Agree
0
0



0 0 0 0 0








S9. Please indicate how much you agree that the item is "useful" for clinical
supervision evaluation.
Strongly Strongly
Strongly Agree Neutral Agree Strongly
Disagree Agree
Empathetic / Coivey empathy to dents O CO O O O
Dependable reliabl O O O O O
Deonsrtrates unconditional positve. regard O O O O O
Genui.e O O O O O
Warm 0 0 0 0 0
Ope minded O O O 0
Naom-udgmental O O 0 O O
lexible/daptabl O O O O O
Approachable / NaM-threatening O O O O O
patient 0 0 0 03
Honest 0 0 0 0 0

Altruist 0 0 0 0 0
Arst[, O O O O O


Energt- O O O O O
Notvat-ng 0 0 0 0 0
Energetic 0 0 0 0 0
Asseretiv 0 0 0 0 0
Po tive. 0 0 0 0 0
Hu.morou 0 0 0 0 0
,Respectfu O O O O
Direct 0 0 0 0 0
-Creatve 0 0 0 0 0
[Cntmlli/t.tn O O O O O

l..itl-ivsii 0 0 0 0 0



Mature 0 0 0 0 0
nO O O O O
tO O O O O





Comfortable with clent 0) 0 0 0 0
NLon-,. grfv O O O O O

Displays comorn sense C0 0 0 0 0C
Con.cientlous/prudl, t 0 0 0C 0 0
..tUiV.,pM....ptlVA. 0 0 0 0 0
Extroerted O O O O O
Introverted 0 0 0 0 0
ble...-. O O O O O
Comfortable 0e O O O

l.^.r.v. O O O O O
c....v* O O O O O









Ebu ll nit ppy 0 0 0 0 0
Hard-w rki- 0 0 0 0 0
Moderatng/Judg.g 0 0 0 0 0
Timntlyopportun.r 0 0 0 0 0
0 0 0 0 0
Succn 0 0 0 0 0
eans to work with Q 0 0 0 0
Team player. 0 0 0 0 0
Ha- hy 0 0 0 0 0
Go.l.oriented 0 0 0 0 0
Comments
1



S10. Please indicate how much you agree that the item is "useful" for clinical
supervision evaluation.
Stmro ly Strongly
Disagree Neutral Agree
Disagree Agree
Sncidea risi assessment O O 0 0 0
M.a..,e r.ourc 0 0 0 0 0
Fiscal management 0 0 0 0 0
Crisis management 0 0 0 0 0
Comments
-1









APPENDIX H
ROUND 2 FRIENDLY REMINDER EMAIL

Dr. XXXXX,

This is a reminder email regarding your participation in the University of Florida clinical
supervision Delphi study. As previously mentioned, the data from Round 1 has been analyzed
and we have identified 184 items representing 10 theoretical competency domains. Per the
Delphi format, we ask that you complete the second round of questioning whereby you rate the
"usefulness" of each item on a five point Likert scale. In order to remain consistent with the
purpose of this study, please read the instructions page before rating the items.

We know that your time is valuable and we really appreciate your contribution to this project.
However, because we are on a limited time schedule we would really appreciate if you could
complete the second round by FRIDAY, JUNE 20th. Educators who pilot tested this round
completed it in approximately 30 minutes.

Survey Link: https://www.surveymonkey.com/s.aspx?sm=SHzVwljPeW1wOgt9VkLo_2bg_3d_3d

If you have any questions, please contact Michael Moorhouse at mmoorhou@phhp.ufl.edu or
352-273-6491.

Thank you,
Michael Moorhouse









APPENDIX I
ROUND 3 EMAIL TO REHABILITATION COUNSELING EDUCATORS

Dr. XXXXX

Thank you for completing the first and second rounds of the clinical supervision Delphi study.
The third and final round is now ready for your participation. Similar to round 2, this round is
considerably quicker than round 1 and should take between 15-20 minutes to finish. Because we
are on a limited time schedule and this round takes considerably less time, we would really
appreciate if you could complete this final round by FRIDAY, JULY 4th.

The goal of the third round is to achieve consensus regarding whether or not each item is
"useful" for clinical supervision evaluation. Therefore, your participation in this round is highly
valuable.

You will now have the opportunity to review the group statistical data as well as your previous
ratings. Based on this information, please decide whether or not you would like to keep or
change your responses. Unlike the previous round you will not be expected to re-rate each item,
only those items you wish to change. Again we want to express our extreme gratitude to you for
participating in this study.

To begin the final round of this study, please click on the following link.

https://www.surveymonkey.com/s.aspx?sm=FHfex8ctiiWdtz9s2E1Kfw_3d_3d

Please contact Michael Moorhouse at (352) 273-6491 with any questions regarding this study.










APPENDIX J
DELPHI ROUND 3 SURVEY


1- Instructions

Thank you for completing the second round of the Delphi study.

For the final round, we ask that you compare your initial item ratings to the group
statistical data (i.e. item means and standard deviations: comments) and simply re-
rate only those item you want to change. For your convenience your initial item
ratings and group statistical data can be found in bold directly beneath ach item.
Keep in mind that unlike the previous round, you only rate th item you wish to
change For eKample, the itan avoidu colnttrtransfernce" appears below. If after
examining the item data you choose to change your rating fro "Agree(4)" to
Strongly Agree(S)," then you would select the "Strongly Agree (5)" cirde.
How.evr, if you n amine the data and choose to keep your it. rating asm "Agree
(4);" then simply sedct nothing and move on to the nKt item.

In addition to group statitical data, participant comments can be lound in two ares.
Comrrments which are related to a specific item are located directly beneath your
item/group ratings and general commnmnt can be found in the upper part of the
pag before the first item.

it s important to reiterate that you are rating items based on whether you believe
the item COULD be "usefulT" Ir clinical supervision evaluation. A "useful" it im ione
that is representative of some aspect of rehabilitation counseling performance and
potentially helps the supervisor evaluate the trainee.

>N OT E Plese enter your name at the completion of the survey even if you do not
diange any items.


MEIdu rfrd ( r r q.'
aft A r n4;1m I:1 SD -
(YON1 RA TM G:z 4; GROW, MHUM: 4-%Mt.M C 0EP










1. Plse re-rate whether you agree each item is useful for clinical suprvisoan
valuation.


m itrr nnmNihl gm- v
C Cly 0 r 0


"W.rTA M1t&G:; Ei )oClA:rl*. SD:.Ai1 C
Ictfi. bra elg e-sIEng h oI rin : Imruali rd riIpaicii) -
M.AR -Inb GzICMj GAMP HUM:Bf^ f0:.l.| C

W:ui uirt ikllk (&gA. mys nulrtj, 3u:c pFkhi., Wcs thm. gtiorm, klIl j
amprmial:n. plakil 411E:im. ale ipprcpr a ismck)
CeOwA Min& k; Rb WF htAle:4._L SL:.M|
g AIU~~4 9kiP Iw~, A&ICIJ 1

dClt c bt bto da wcoam 'l. pRe C
M At Gl4 =CLi UPiC:4Il.SL 'b

SaC dur ki:lrdr mk dairm C'r. C,

huari r empuri Ia hi a's cerlournfi' 4 e (o 0
pi. UM1xG64 GItOJP Im:4JL. SD:.Ml
DoMilnErim rmd r t CErIM h thaiJbital.i C ,O
YiO.A AM IG: GRWt Im:ekA:4Art SD:.



iaoubltn Ia w rg IO Im
(f'fA MLaG-iq:3; GR Imtakh: SO.5

i dh l eiry at heu c eaukt illag 'rl.e

An lbllMi:t G m lana4 kli nr Ir' II *p: *l
fT iA mlAnrGN4; G h IF All|:iM. SD:.Ak]

( 0BMfeING GIlur Lhii:Mi s D:.ni

(YrwA fAn&:4; N GI4Inr I. :k-" .E... ik
Mer.d c:IrurrIrlInrriu j i
LM'oI L'UmIG:4 GQOUFP IiL: jA. ~D:.S I D-
demll s11 ard TI l :cl l Meft i C'e
M At M1A& N GMIuP I:AlA.... :;rI:. I
Ap ,iwii = r& w cis aimakch promm |Ig r;F ~ m.11 cii, :pha. ,'"
trommit. biunaivisar
MATIImG4 GAbuP im4Ah:mS. SD:.MI
C4mitiiss t1 nEMe hrmc .ard anes Istrp*r&c Il .I
YO.DA MATlG:-4; GbMNF IhCmL:4a. SD:.|I


,0 O C

0 0 0

0 O 0


0 0 0

O 0 0


o 0 0



0 0 0



0 ")0

0 00
0 0 01
0 0


0 0 0

0 0 0


0 00
L3


Sikl a PP
(Yowf uInI: **; GOUP H u:mH:U sD:.a |











IWMu i ts he du mit. op at lppnrMitelS O O i
(Y tlITU l; Gilbu P IUCAk:4. SD:.)I

keui tis wumrm al t ;Irn :r l Felbru m. t a ca ind e -prir n ,t )
tL images r aisll, :
(D.rA MlAG:; GRWF ICbAh:".h. b::.HI

Cmnra maimr rmingl hid pdrAunl dagihL ibt chrm ta lit a e O
Lndlinllrd ng in idr daui wurfrarimn denil ci.a masll rg Auiiu
(Oil ,Mi4; Gl Crue lAJ:,, SD:.LI|

ili ikat r'A l.,th rhtemeI up C O
(YO UA MA 4 ; GIDuF l CA:LLU. SC:. lU

Ab b td iaWr 11m dr r ibe ancr csu O
M. A MTmGz4; Gh"P HMO: CS&.) 0 0 0 0


M Aep d aljs Gkw p Cns" O O O





(i tAulM:f GIAuF i.CAkl:L.t, mL:.iJ

Ferki at Giy l I.i o h rihhip Irmt Itri: to 1 "i? r
('o.A MLIANGz:; GIRNF ImCh:LHK. SP:.i4| '

CFIo het in hu l i urw hdlrpere.aal I nanrm att ) gt 1
(Ow ATH64; GMFuP Pll:L. fD:.)

Tdlmc gzcc n-Wo kliriosp wmiam
YO.A l-I h:J; GMRJF hCtAh:4. O:..:M

I c Lbs cblm nqll] ntvni bt rmtta Inj f~r latn~
-ihA -TI4:.2; GFoUFP hCAN:L4. SD:.C|

CL II ca I I nam E;;ratle 'D
(YO MlRAT:2; GIDJF iCAk.:LL4. C:.MI -


eOM MkT:; GIRCPUF hCILN:"L SD:.|S-











, Plase re-rat whether you agree mach iten i "us ful" Ior clinical su-vision
valuation,


ommvoii Nuftl Alm
Dizagr* Wr
at"(4aki p S 1c1a
(YO MllIG:*c GRWipP iCAh:.AL SD:.M c 0

U s ad ird4mraI a.iattio alh (IAMgi iadm r.miarnHaI, ird d r4e j-,, | (|
(YO.A AM"4; GPWP; I0. :LM. iD:.I -

Prapira* clM riiinnoiru m i C
(YNM MINE4; GRhlP iha:4rA. SD:.dk

P""'d o f'"i9 l :I"'-:t l 0Ah 0 0
(WA tCTnl4; GIWPF hWi:4-.S :.I

hI' oct rilmn rg .l dl It t.mlltIr l l o
(Y.A Md I.Q4; GRP ipl i:4-m. D:.M

UmyCK tzlm6.4 mcdo d L S
(YOA MAing4; GOeMPi lmAk:jmi.5c:.Mi


(mYON MQci; GhR MPrMi:_Adl, S. D:.Pil

br'*erini : IEc itlOts


tM CI1 4 I"kI I.L:'. 0 C 0I 0

w(YO.A Mlta:4 & IIUlA:Lk. SD:..|
(YA AMTn B4; GAMRP NWAI i: SD:.di.'
assg m*:d*am imhrpami Mb "/ c 0ts

(YOl- MLAT4G; cir Ai: L .:.MI
MDA fennck4; GbwuP fMLO.dft SD:.E*

a rlo :eil k hn CI 'w'
I(YCA MN1 : O4; F GIMr Ih:-l SD:.| I

4iw wem rreat cca sh tr
(YOA LA-IG:; GMulP @l : U.5:.i 0 0


(YO.A LM Q :R GIMF I k:4-.S &:.& ti

dre lld :ll Iwh i1 opp I a~
C(YO tIMB1 4; GRCuiP ICLh:4.. SD:.M|


Prwcaia s $sadbteCk






(YC.A PAIN4; GIhMF liAh:L4. SD:.MI


C 0 0 0


C0 0 0

C 0 0 0


0llo
HB












14bi be reignal Cci Oa Or O
C.,A AImnG4; GROj ILA h:4C:. Sh:.J -

-l "ra ridiai""q rm maii* 0 ,. _. O 0
C ra LAMM4; GROP I@h:LhI SD:i M

U s iar-d o. cm i.Fr. p 0i 0 0
(Y*A IA"I-Gs4; GRUP hC,:4-L SI:. s

Abt hadi ctuft stirin eig an:iala !is.. cnl~g
(CVO LNT U IM GRiOP NIA4:&LtS S_*.

DisgemIli a; Is cid. rIIf nillali] raepid.W ird ip:rat iahrd *
IMi.A MIX:G GkWP iNhCA:4t.50c:.7SI

Lhib. HIT pF akchiqae gmru"hdic le casral. ihry
C(Y. rAmcI :u4; GROU hiL:iA4i. SD~ik -il

ILth udiuelil thcir :ardd gr auIsIlI the :r
YckA MA1lG:4; GCMUP ICihL:4-kL SD:.I -

*:riI mdi'i a Ahgrc Cel -pp: ch C *
(YO~A R1ht, GROUP I :L%.h:At s.r.i

tr ryIhl abib k I
O-.A MA aIjGk GMU IsI isA:MLjlC:.4| '

Grm4.p ajiulg l ripg i al



(YO.A MAIAC: 5; GOUP I)CAh::.:I SD:.JL] -

tA o t claie Eerl Card M a ul a:I.l eulb O O 0
e(YOrA wFhU4; Srit MGuP M:L5, SD:1 M

Erhice or hII; : Isii i: Isemlm h s :r Ihw a ll-utu
M(a FAriCnG4; GROUP jU*:m1A. SD:._

kAfr iucl s tr p O O| *, ,
(wA fMhiA; GROCUP IatL:U LSD:.|.]


M.A MLAim:k; GRkUPF IsCAb:c.SC:.4M| S I










4. Please re-rate whether you agree each items is "useful" for clinical supervision
evaluation.

GENERAL COMMENT:Something in the back of my mind bugs me about using items
that reflect more of the career or vocational orientation in the supervision context.
While I agree that they are important and may be useful, I think the importance of
these is relative and depends highly on the focus of the practicum course, the
philosophy and mission of the porgram, and the students' career objectives. Of
course it would be terrific if students could experience the range of skills required of
rehab counseling at any given practicum site; but not all sites or students have that
option. I think it would be inappropraite for a student to be required to demonstrate
these skills in supervision with no opportunity to practice career
development/assessment activities at his/her practicum site.
Strongly St r gIylv
SDisagre Neutral Agree Sr
Disagree Agree
Gather pertinent Information frm the client and relevant iohers through O O O O O
assessments, observations, and faclitative questions
YOUR RATING B; GROUP NEAN:4.2. SO9..49)


Assss client interests, skills, and aptitudes
(YOUR RATINGr S; GROUP MEAN 4.17. SO-..8d)

Assess client self-afficacy
(YOUR RATINGi S; GROUP NEAN:4.S6. SO-.Sll


Assess client motivation
(YOUR RATINGi S; GROUP NLAN:4.,O. S-.5I)

Assss chlit trengts and wakinesses
(YOUR RATING S: GROUP NMAN:4.01. S&.S0)

General ability to conceptualize
(YOUR RATINGi S; GROUP NAN:4.39. SD.61)

Conceptualize a case within a theoretical frame of reference
(YOUR RATING 5: GROUP NEAN:4.06. SO-J7)

Conceptualize a client's 'ie l e psychosocial, vocational, etc.)
(YOUR RATING S9; GROUP NEAN:4.44. S..SL)

Identify chlnt problems
(YOUR RATINGi S; GROUP kMAN:4.78. S :-.43)

Prioritize clint problems
(YOUR RATING S; GROUP NMAN:4.L0, S-..71)

Diagnose (with some assistance)
(YOUR RATING 5; GROUP NEANt3.56. SD:1.1S)

Clear understanding of theory(s)
(YOUR RATING SI; GROUP NIAN:4.2B. S..69)

Knowledge of career devaopment
(YOUR RATINGi B; GROUP NEANL4.00, S&.77)


0 0 0 0 0


0 0 0 0 0


0 0 0 0 0


0 0 0 0 0


0 0 0 0 0


0 0 0 0 0


0 0 0 0 0


0 0 0 0 0


0 0 0 0 0


0 0 0 0 0


0 0 0 0 0















UlkirtaL I Df r aili arWd wmarpatlo I dpeurml:l
(I'O.A AiTMg :; CJPauP, k.dih:A-l. D-:.1Y
C.IatiMiIf uidintardmg :1 lm Ip lpreaI:s and Ilrly sIarp|p lk k big
Tprarm al at :a ;bk- arn bicui 1h MI.h ioan la rg

Lfdert MljN ItM ucpIAt is I .t3|
CrO.N RATIMA GROP, ktL%:AJL SD:.VSW


S, PMlse re-rate whether you agree ech item is "useful" for clinical supervision
evaluation.


GENERAL COHINENT: Many things depend on the circunmstnc of the client


lcwr.f (ilm ial mnriz cd dDw'r pnru'iplrrm
(M WFIMhc C GKup lilA:4AJL SD:.WI





SMIA GCIPF hILa:4.P. MD:A.]

LcHF'i ai rd 4c;lpk apikain A clh i
:.P A McAGi:4; G:JF ILih:Jt. MD:.U|I


M Ar LA G:4; GuF ILhbi:4.A& SI.Lk]


04 :bgly roon A


C 0 0 0 J
hrEu htLiTi .(r) 0

-0 C 0| 0




I. ,. I


C 0 I 0 0


toirom eI m i whi duht prmcfC chorr pIranI(c r icUing f i hc* bn ,1 I '
balarmiol iau :r c mu Fab. D U
.A lG,:,; GbF:,F hL C,,:rI,. S D:.UL

Lrdlfamtl Irebre (.u. plri C l.o. puis IimranrD m inr d w'ry cas ra t wi
inus ikt cit Ia irs ii or krga il
CK.(YoAt uIPMl MAn P ICL:CPR s:M Ad.


UlcerIll lie l Ild tiIf Ie eim Op KmnnKiin In11111ii
O.A AlG: 9; GbJFjF ImLA:a.4. :D:.iU|

ha Jiiis Epr-l .id ;Lb ranit ill l
0(`O1 LA1M :1 GMull sCA:AL. SD:.J|
feNtllrlLum n ir I:t ctmiiJfnD aE plEimcllt in plrt ~fE tHpu nklT
iprlnKA, Cre ht IPFpp:pnll to IlrAinIM du SiHu duEmui inu iiM
pSrvulMu. ir theyr u thIM the udicmi shbuM mal be pcal Ind rr rml
IPairg uo :pplisritf ton doma:lnt tHi .

RqNih'llf i Lin ci l i p;rorin
kWo.A MAIH R GbuPF AiM:&la. SD:5.


C il 0I 0
C; % C)









U 0 0 0 0
C.. I


.i i '

,r Cd


































































150












6. Pleae rr-ratu whether you agrmee u ch ite is "usul" f c dinical suplrvision
evaluation.


Aml JnI.'.ncii I ad IW t Ins tl inmpmraca :1 (4f p i m
1eOI IAe -4sA GIuP hci ah:4l-. fD:.MI

Dlramsls-rM nirpKec ea :ti upwrse of p? Irvbr
Ow.A MILm:; GJF i a-:4l ml. S:.M|I

Caililm a apFrmchii irny r'firoronl a mlai A afi hrwift

Mi -IMT Jt; GkuOP lCAfj: Lt SD:.|tl

Vha-dw"wsl r. cpat ID chikal nominl:r
|:.A- M1 &4; G-IF C :.Jl7U. "r:. I|

prmparen lar UJFefai m
(YCON. MIMLi4; GFOup ihbA:4SL SD:.U|


C 0 0 0 0




U 0 0 '-)


Fhr --o -a F iUd:I. i:rmi
if'uR IAlMC:4; G?:F PLhk:%lt. 0:.i|


I vim rmaami,, vikah.r, i nlun rg a' uprlinH'ap IlichKik
M(AtriMG; GIBi4 aru uiAlb:h h D:.UI|

lrLpraoWs sopers~tr lathrb fLai prNcHca
ier"IrA Mbl[A:4; GkjF lCAk:.4. SD:.UI


C C) 0 (,


n asrearin d hi:. mage aIrlnhm O I> )
tan u or Bt i S ( ) --- mc I1pm c
iCe W M ,,"I L4; Gi ,jP l lqah:,,n4 SD:.i l


MO.A PA-l-G:4; GuhF lCAb:4J4. SD:.|I


praflmual prnmati2E)
(Oti lTGMi, ; GFIuFp iach:4L. SD:.l|

mIdirts :Ar rragit ard mwran malmd ftB c:LrAil' ] C O
p*mu cafc pacmw,
(YOAA InA&:4; GFkJF Ch:4tU. Sc:.UI


COM MtINS4; GMlIS IIhAL:4JL SD:.U| C. ? C-) 0

Msgrbp umar cc pa Wuil iamu hikt ar m ILrm r m k ( (9 g (9)n (9

I"ICA MPaTG:4; GFuF hLAk:4t.5 D*.M|

kace th l* :atits riellr iprvilrw ird ail mpa. C O O O
IM.A N kr. QI M Ghou Gl PAed soW:1 e SD:L0 C
CIS11MIi I dnit hal thII ad rk hliu t hB pak lI th p:z ue c' karhg to
tblcii rri ths l od k aft oipovoEr ird iIh rhErhP amporir. Th tbao
prfbmKmnk sl it* hEJsti dold not bM eamkribg oiudso: in mhkh tM As
ia lias.


uai rg E-2 *mmurai am1e op re4 c" II, I rhiEHr:Mc (L.. amiur m.ircb,
bo cj,, evc.)
(YCIA M iG:4; GIkDIF CjULk:4AL SD:.n


C 0 0 0 0










I. '.( :;


':7 () -, ,


Canspkli Ui~ledun n IFIIm'mFoI mai ti1
A fta4; GlCut liAh:4. D:.dI

se ponr:l frli g bh
A PATG:4 GkouiF i.:4.i. SD:.U.I












7. Please re-rate whether you agree mach item is "useful" for clinical supervision
evaluation.


DIu.gro*

RIbupou e.ir p:rimm D h
(Y -AUG1:M4; .UIP iCAl:4AL4. SD:.d k C


DIrnz.raitm pir.au Ta.lr:I-al a kD t ii La rinHpmIIt ad ithuial
roohtai ti.ai

CaT rltnimr m dchIl IL|mHM
(YO.t M-mG:; SMuP MMAb: 1^ SD:.& |

LUdrllIr diprHs rk i rvm rru 1 a :me carr m)., r rli r nnracr D di
Drankeo rri4 sllIrainir, deal *Uihwihipsi, lil:rrmid :cmiiw jfi
I MUl I G4 G i:FP .M:L4A ;D:.M|


ur c L:n H Be umplruneiU
(YdO. MmUG:A *; .GRUP M.:^ fM t .Mrl]

o tcimars cok.aI dL4ThImchnfl.umias






ird arkir pzir*ifcah lis ia h aire Itlc pasn ithir c-i ruca betaru icosE cf




ttllr irN dM
Ml iA lM 4; GA EU:4. .: Sl P D:0

UMA MppMicfr.i I l]LasGjI umk[r 1-B r prom MnbelLMQ*; ramnds
tmalGe*cm IIm, slut, rmiui. sE b nuir IrgmIIgI
(OM.u NMI*ZA; cGuP MUkA:4- DIk:.kj

OB ism ct r jn ipprpnmils WYetM & mw* rit c:lt glsE a i&M nJFprm :n
IM-.A kM C.&4; GPOUP I- :X7L f:.0|


u(VCwI MIlNrf4; GARCF LMd :&UI, rSDT|h

c gibM ai ~ I ekilt smM niih: l tLtNr, ird hMGi Is-,,i ii

I''.H M~ia:4; GaMF IC= :m.sd.:.uI

ml rI I ii chil mitl ti r cam r ia d bIhal t h hammt I brih rgria Mir
bim.ki to lhri m 'l hKb. lteam a comptIIe and i hiical rrifablmis

(YO.A1 MIG4; GROF C:4UP : UP:.H|i
CdOMIri i NJa a ji11 gh A -as 11811l W lMe I gILdwii to wiinZi ID charge
wwo ctler cm h1111 aMl mill lebItaf ah theib Hr tldip:: 6Dr ena., him
k l Fpizblh I inhala m otedil in AIm lirai. Ic nmo nlt bi cameWe me
bMosiy =s 'raij, pm: mr rrabbeg i S I tk cbairse o r bca pserm m* mtg
tk chiarge il doD thi itt r g(I: mr.' tar m Mlt hM ahAdist k pa rg c iiir
Imd ms pFisgaig is Ihp rndinrc nlufhcirM INMHO, lit ; t th m%.c.l t
iFperve*r hi pr:mil gQlUIIce ird da L=w& recapiHamol w:rt mil tUm itbadit
wrt p-11111 tiu Lusk


(2f p) iii

C) 0 0 0


C 0 0




S, 0 0 0


0 0 00


~0 0 C


C. 0 0 0- 0


C0 0 0 0 0


0 0 0 0











*. PMaK re-rate whether you agre Ich itm is utul" for dlirncl suprvision


mealuation,

Hugrui

Lu car** k C
YO. M1G4; GiMP hi.Cl:ULL D:.: l


bitui pWoe hword ]1 |lt
YOA MInG:4; GCAUP Ith:Ah. SD.JIl

DIupi bi11y 1: *i.uran mBe alkir
Awgia ibrg I.rb i Un IIT tun am

(YOA ATG:4; GbDuF Is&:4MSiC:.I7

I I ~uIl m~ cralinlli ba him :Id mI Ip p*:hll aulp
(MOA iVG; GFiDuP IlI:EML :S:.ddI

An riin a runeil Ii r kiA IIIngu I b it:lh ilLn inir Iicrt al lt


(YO.A "nGc 1 G N FI :l.m. M.11,j


wi I q -: ampir, wm br tha hi cm udlIg muaeb
*lT WiM Gu P hIAm:.M*. 'D:.1]

lgi*lab Irll: al e p cp
M.A L'ImG:J-; GIuuF ICm :4j St:.H|


An Alslq tL ~m;ai m r Iu kl ulr.-reliai nilt. :lir iFccing i* f ai I1*
Inlhkin :1 :Alinr Im ti.p pem: piu in kiikl :rmung. prncluLn mdu
Ilrnurhp iplrtiuise
MA.A-Im G:A; GWP MMU Af:M SD:.]J
lMMII k- to ir ingr en cnl ulul-mrde r mwr ut
Mhi| :I It niluc d tL abu ttify pri'kikull t :liicil lfing.
prn.:jnm .h lhilrnLh p .Ipai-im Al irdli thl *iml iti Ilia.ijl I
tik thai mIy bI nFpirii, I: rmei rn:rii rnc u ai dh bit phaimn Tab ki
maitur lma ird cmn 1hi iMuaed in upiri u gias idm ind
n:iKCr ml o:rkC.

Ai I h Ia wr i'nd iti ,d id::urm itsinuit
(YOA WAIG: F h,*i:,M .iIP 1:.I71


0 0 0 0
Du..r, *mi5u


C 0 .
X-1 II


..h..


I_- I.


3 O I


(3I ,


*.0 *0


0 0











9. Pleae re-rat whether you agree Iach item is "u-1ful" fr dinial u pervision
ewaluation.


C 0
asI =sga


L. IV


c.ugih an a :vi I)Bartancii f i cai r
(YO.A 1UIf4; GinOJ hICMAL:4.. SD:.kL
UM I Nrw igiz c a' da pe nlir rci mf ar aur Ai immrinar dt
ther :H porssD Imur.i am klrtde w-h Rar ASimL to limrr and Fir.c
Is aprdirlu ih end dtcklpl nuapcnunki rTiimr *lilrdr4 It a:a;i
r.isnirma. Awlarmur :r rFcgirin In a trraL mr Inrt : ing ap araimnu d
wt k da ibMu*h Iard U~isquIrc f:l urI cirmu o Ir aci, nd I
bhta tio i ti ImpawI Ibe *lIIru a IhUI.

Ma mi as ul %at IthI ca prwws a rt-a-i u h l abiw *l
to tarm nid praha r I praelimauz:l ma ukrzlyf rmiprdHiba Tiamli
(YOf A MT 4i GFRF liiAlJl :a &C:..

WIE 11 a se b alt prtk iEi coarlatg lr r dfaonssu i at i n
raeded ib water fir I No: larcH ai tt acila ird p:fmuin cI ly
(YOtNwE MUI4; GOCpF MCAM: &A4 D:.H|

Drs reta es 0Mn& I a ib If
(YO.fA rAtIM:i G*Fd CuAlh:k.5L:M

lmrau s# r mansl hum diabiti
(Y'ow MTIQJ: GMu s:t :I|ll :
CP1MINU Tkl muM urn labuTlls am "MT imprilat. Is prctkr, th
uis rsia e m maI he t dkzbblI. I kbm Oitb id avmiteo d a uwi
ten e.


0


iAF


rlmrgg

(E
Aq


O 0 0 0 O



C 0 O 0 0



C 0 0 0 0


C 0 0 0 0
COOO


IC











10. Poass re-rate whether you agrae mch item is "useful" Ior clinical mupervison
evaluation.

GENERAL COIMENIT: I don't know what many of thews words man. I knaw what I
mean by them, but I don't know what adher may mean by ithm. The shades of
imaning af many af thau. wordn are ham variable for nm to interpirat-and thurwurm
they nar not useful (to rmet


fEpetwic e' Cwa ampcr it cAst
(YOM.Af : ~ 4 GFROUP ICtl:4.Jk. s:.J

M -A.A lLG:i G,:JuF Ih CAb:M. SD:.N

DamnsinrmE iiaiihiiial pculih *ejord


(YO.A I-I*G:! GRCUF fICAlh:*. SD:.h|

norT
(YOM LUVfUM4; GROUP IRhA:4JL SD:.MA|




FYA 3l,' Gi ECuI ll:L SblD
MYAf IAImG: R GROUP I@IAh:43t SDMASU

M k NA-ICG:; GIkF InC.:Ul?. iC:.4l*

Appr"uekmb I I K lh

'M4 A-ItG: N4 G UJP IIAih:4a, SD:.M



CY&A L M A GROUP kAW:4A SDM.WJ




(YO M'IG: GIuPJP IChC:M. rD:.Mn

(1~A UTA|GH:; GRFDjF ICLnA:_'I. SD:.IlM
(YO.A M eIG2; GIMI F RtEM:l. CS:.H|






rriugrtc
IYO.A AMi&G; GIauF ItAA:U1. C:.M|


A Ig N nbl AIrma



C 0 0 0 0



















C C) 0 0 0

o n 0 0 0

C O 0 0




C 0 0 0 0



C 0 0 0 0

C C0 0 0 0











NOW AtMG:; GUIP hCI.Ab:AlL S:.MI

tOA MPAING:-3; GIIuF I JrLA:31t. SC:.MI -
"9 0 C) 0t
MDA MaG:I4; GROUP MMA:LAT, Si :.


S"Mi:4 GfwM M'a.:.adl 0 -ID 0

IYA MINmG:4; RF CCA?:Ul :.|


(I'YA Cw Ci; G:'4 ,0G. 0: 0


SI.- CMl, 0 GR.P rAl:A7. fD:.I| 0 "D





-""
Owi~uIiI~ hI ~ kiI:,d_ 0 ~ )



I(Y.A L AInG:; iGh nui iU k:L.a SD:.MI
CK:Y MI :; GR:QUP14 G h:mP.5 :JL




(TlO.* MimG;* GROaUF MC::UI, SD4JtI D:.B ]




a:rlae nt CC r 0 0 0
It3l M l 4 GItpa _:4. 0D:. | _






C-3 I 11* ,It1 m"i f .. i"r










IA MtwGc a-k, jF MI, A. :L lt. E:ru:.d,_
(. MiinG:4; cf GIRDIP ^M:JL 5:.UI



(L'A Ml5'k t-:4 GAke NPCAk.:4t, SI D:.|I
P- PLA MTiU"t4; GIO ,'D:AJlC, 0 :.M| D

DIC tr'bhla p au th hftai


M.DA Ltm-lG:4; GuI- IPU4:a4at, SD*.Lk C
btanri ~ ~
COO" LaTmG:4; cliUP KhAW:u., f:.3*s ] 0


(YOw AIdG:F, CF IAk:Lnt.:.4M

MIY.A MAkG:2; GRePO- ILp.kle:U;. SP:.~i
M-iA RA-AG:J; GRWUF I:JkL7. f D:.X 'c Ol "D









Etbkiankihap

haw-waikrqn
I& Ml G:*; GFtkCJF li.:2:4. SD:.W I




(.YO U1UtfG:c GRtOiP lmll:t. SD:LI4
Nj.m





(.A MThNG:4i GFkCF C SI:.5:.MI




(..A MTGt ; GR CjP lC.:' 5 0:.Hl
IO.AT GuJPF ICAM:LJa SD:.MK


(Yt uMAG:; GIF J ICriM:lL 0SD:L.l



Gal arimba
(sA M GFO IM: Sa:.

M A G:PJF LC.i:.AL. "r:. iI


C 0 0 0 0

C 0 0 0 0

C 0C) 0 0 0

0 0 00








C 0 0 0 0

S0 0 0 0)



C 0 0 0











11. Plea re-rate whether you arge sch item ii "ueful" fIr dinical supervision


evaluation.




&hllll r1 .m:ul
M II1 MGiahjs G W IMP W!"A"'S


M.A Ahb5G:4; GukF m ikh:4.&b. SD:JIW

Fuul mnrglniurc
I(YO.A LAkh J Glr. nI.Abh:LM SD:.UI


M-:tr
DIuagrua


aJm

0-.


Hh#JTIJ

Cl~


IFI
04n


AFr


C 0 0 0 0


Cd-dl NuM-" r, C, Q 0 0
(OA MhG: GI WfF ICd : l S CD:




12. Thank you for compiling thil Delphi study. Your contribuion i vary much
uppreiated

In order for us to keep trck with your final submission, please include your narm In
the box below.
I









APPENDIX K
ROUND 3 FRIENDLY REMINDER EMAIL

Dr. XXXXX

This is a reminder email regarding your participation in the University of Florida clinical
supervision Delphi study. Thank you for completing the first and second rounds of the clinical
supervision Delphi study. The third and final round is now ready for your participation. Similar
to round 2, this round is considerably quicker than round 1 and should take between 15-20
minutes to finish. Because we are on a limited time schedule and this round takes considerably
less time, we would really appreciate if you could complete this final round by FRIDAY, JULY
4th.

The goal of the third round is to achieve consensus regarding whether or not each item is
"useful" for clinical supervision evaluation. Therefore, your participation in this round is highly
valuable.

You will now have the opportunity to review the group statistical data as well as your previous
ratings. Based on this information, please decide whether or not you would like to keep or
change your responses. Unlike the previous round you will not be expected to re-rate each item,
only those items you wish to change. Again we want to express our extreme gratitude to you for
participating in this study.

To begin the final round of this study, please click on the following link.

https://www.surveymonkey.com/s.aspx?sm=FHfex8ctiiWdtz9s2E1Kfw_3d_3d

Please contact Michael Moorhouse at (352) 273-6491 with any questions regarding this study.









LIST OF REFERENCES


Association of American Colleges and Universities & Council for Higher Education
Accreditation (January 30, 2008). New leadershipfor student learning and
accountability: A statement ofprinciples, commitments to action. Retrieved from:
http://www.chea.org/pdf/2008.01.30_NewLeadership_Statement.pdf

Chan, F., Berven, N.L., Thomas, K.R. (Eds.). (2004). Counseling either ie, and techniques for
rehabilitation health professionals. New York: Springer.

Baldo, T. D., Softas-Nall, B. C., & Shaw, S. F. (1997). Student review and retention in counselor
education: An alternative to Frame and Stevens-Smith. Counselor Education &
Supervision, 36, 245-253.

Bemak, F., Epp, L. R., & Keys, S. G. (1999). Impaired graduate students: A process model of
graduate program monitoring and intervention. International Journal for the
Advancement of Counselling, 21, 19-30.

Bernard, J. L. (1975). Due process in dropping the unsuitable clinical student. Professional
Psychology, 6, 275-278.

Bernard, J. M. (1997). The discrimination model. In C.E.Watkins (Ed.), Handbook of
jp/L 1h/theI.py supervision (pp. 310-327). New York: Wiley.

Bernard, J. M. & Goodyear, R. K. (1992). Fundamentals of clinical supervision. (2nd ed.).
Boston: Allyn & Bacon.

Bernard, J. M. & Goodyear, R. K. (2004). Fundamentals of clinical supervision. (3rd ed.).
Boston: Pearson, Allyn & Bacon.

Biaggio, M. K., Gasparikova-Krasnec, M., & Bauer, L. (1983). Evaluation of clinical psychology
graduate students: The problem of the unsuitable student. Profession Practice of
Psychology, 4, 9-20.

Blackwell, T. L., Strohmer, D. C., Belcas, E. M., & Burton, K. A. (2002). Ethics in rehabilitation
counseling supervision. Rehabilitation Counseling Bulletin, 45, 240-247.

Bond, T. G. & Fox, C. M. (2001). Applying the Rasch model: Fundamental measurement in the
human sciences. Mahwah, NJ: Lawrence Erlbaum Associates.

Boxley, R., Drew, C. R., & Rangel, D. M. (1986). Clinical trainee impairment in APA approved
internship programs. Clinical Psychologist, 39, 49-52.

Bradley, J. & Post, P. (1991). Impaired students: Do we eliminate them from counselor
education programs? Counselor Education & Supervision, 31, 100-108.









Chagnon, J., & Russell, R.K. (1995). Assessment of supervisee developmental level and
supervisory environment across supervisor experience. Journal of Counseling and
Development, 73, 553-558.

Commission of Rehabilitation Counselor Certification (2002). Code ofprofessional ,citiL \for
rehabilitation counselors. Rolling Meadows, IL.

Corbin, J., & Strauss, A. (1990). Grounded Theory research: Procedures, canons, and evaluative
criteria. Qualitative Sociology, 13, 3-21.

Corey, G. (2004). Theory and practice ofcounseling and, pyi /it, 1wtqy. (7th ed.). Pacific Grove,
CA: Brooks/Cole-Thompson Learning.

Cormier, W. H. & Cormier, L. S. (1991). Interviewing strategies for helpers -fundamental skills
and cognitive behavioral interventions. (3rd ed.). Pacific Grove: Brooks/Cole.

Council on Rehabilitation Education (2002). CORE accreditation standards and procedures
manual Rolling Meadows, IL.

Cramer, D., & Howitt, D. (2004). The SAGE dictionary of statistics: A practical resource for
students in the social sciences. Thousand Oaks, CA: Sage.

Currier, K.F., Chan, F., Berven, N.L., Habeck, R.V., & Taylor, D.W. (2001). Functions and
knowledge domains for disability management practice: A Delphi study. Rehabilitation
Counseling Bulletin, 44, 133-143.

Ellis, M.V. (1991). Research in clinical supervision: Revitalizing a scientific agenda. Counselor
Education and Supervision, 30, 238-251.

Elman, N.S., & Forrest, L. (2004). Psychotherapy in the remediation of psychology trainees:
Exploratory interviews with training directors. Professional psychology, research and
practice, 35(2), 123-130.

Elman, N.S., Illfelder-Kaye, J., & Robiner, W.N. (2005). Professional development: Training for
professionalism as a foundation for competent practice in psychology. Professional
Psychology: Research and Practice, 36, 367-375.

Eriksen, K. P. & McAuliffe, G. J. (2003). A measure of counselor competence. Counselor
Education & Supervision, 43, 120-133.

Falvo, D. R., & Parker, R. M. (2000). Ethics in rehabilitation education and research.
Rehabilitation Counseling Bulletin, 43, 197-202

Fink, A., Kosecoff, J., Chassin, M., & Brook, R.H. (1984). Consensus methods: Characteristics
and guidelines for use. American Journal of Public Health, 74, 979-983.









Forrest, L., Elman, N., Gizara, S., & Vacha-Haase, T. (1999). Trainee impairment A review of
identification, remediation, dismissal, and legal issues. Counseling Psychologist, 27, 627-
686.

Frame, M. W. & Stevens-Smith, P. (1995). Out of harm's way: Enhancing monitoring and
dismissal processes in counselor education programs. Counselor Education &
Supervision, 35, 118-129.

Freeman, E. M. (1985). The importance of feedback in clinical supervision: Implications for
direct practice. The Clinical Supervisor, 3, 5-26.

Gallessich, J. & Olmstead, K. M. (1987). Training in counseling psychology: Issues and trends in
1986. Counseling Psychologist, 15, 596-600.

Gizara, S. S. & Forrest, L. (2004). Supervisors' experiences of trainee impairment and
incompetence at APA-accredited internship sites. Professional Psychology-Research and
Practice, 35, 131-140.

Hackney, H. & Cormier, S. (1994). Counseling strategies and interventions. (4th ed.) Boston:
Allyn & Bacon.

Hahn, W. K. & Molnar, S. (1991). Intern evaluation in university counseling centers: Process,
problems, and recommendations. The Counseling Psychologist, 19, 414-429.

Hakim, S., & Weinblatt, J. (1993). The Delphi process as a tool for decision making: The case of
vocational training of people with handicaps. Evaluation andProgram Planning, 16, 25-
38.

Hasson, F., Keeney, S., & McKenna, H. (2000). Research guidelines for the Delphi survey
technique. Journal ofAdvanced Nursing, 32, 1008-1015.

Hensley-Choate, L. G., Smith, S. L., & Spruill, D. (2005). Professional development of
counselor education students: An exploratory study of professional performance
indicators for assessment. International Journal for the Advancement of Counselling, 27,
383-397.

Hensley, L. G., Smith, S. L., & Thompson, R. W. (2003). Assessing competencies of counselors-
in-training: Complexities in evaluating personal and professional development.
Counselor Education & Supervision, 42, 219-230.

Herbert, J. T. (2004). Analysis of clinical supervision practices as documented in rehabilitation
counseling syllabi and fieldwork manuals. Rehabilitation Education, 18, 13-33.

Herbert, J. T. & Richardson, B. K. (1995). Introduction to the special issue on rehabilitation
counselor supervision. Rehabilitation Education, 36, 278-281.









Herbert, J. T., Ward, T. J., & Hemlick, L. M. (1995). Confirmatory factor analysis of the
Supervisory Style Inventory and Revised Supervision Questionnaire. Rehabilitation
Counseling Bulletin, 38, 334-349.

Hill, C. E. (2004). Helping skills: Facilitating exploration, insight, and action. (2nd ed.)
Washington, D.C.: American Psychological Association.

Holloway, E.L. (1987). Developmental models of supervision: Is it developmental? Professional
Psychology: Research and Practice, 18, 209-216.

Holloway, E.L. (1988). Models of counselor development or training models for supervision:
Rejoinder to Stoltenberg and Delworth. Professional Psychology: Research and Practice,
19, 138-140.

Hoffman, M. A., Hill, C. E., Holmes, S. E., & Freitas, G. F. (2005). Supervisor perspective on
the process and outcome of giving easy, difficult, and no feedback to supervisees.
Journal of Counseling Psychology, 52, 3-13.

Huprich, S. K. & Rudd, M. D. (20074). A national survey of trainee impairment of clinical,
counseling, and school psychology doctoral programs and internships. Journal of Clinical
Psychology, 60, 43-52.

Janikowski, T. P. (1990). Rehabilitation counselor competencies: Recommendations for
identification and assessment. Rehabilitation Education, 4, 185-193.

Johnson, W. B. & Campbell, C. D. (2002). Character and fitness requirements for professional
psychologists: Are there any? Professional psychology, research and practice, 33, 46-53.

Kaslow, N.J. (2004). Competencies in professional psychology. The American Psychologist, 59,
774-781.

Kaslow, N.J., Borden, K.A., Collins, F.L., Forrest, L., Illfelder-Kaye, J., Nelson, P.D., et al.
(2004). Competencies conference: Future directions in education and credentialing in
professional psychology. Journal of Counseling Psychology, 60, 699-712.

Kaslow, N.J., Rubin, N.J., Bebeau, M.J., Leigh, I.W., Lichtenberg, J.W., Nelson, P.D., et al.
(2007). Guiding principles and recommendations for the assessment of competence.
Professional Psychology: Research and Practice, 38, 441-451.

Kaslow, N.J., Rubin, N.J., Forrest, L., Elman, N.S., Van Home, B.A., Jacobs, S.C., et al. (2007).
Recognizing, assessing, and intervening with problems of professional competence.
Professional Psychology: Research and Practice, 38, 479-492.

Kerl, S. B., Garcia, J. L., McCullough, C. S., & Maxwell, M. E. (2002). Systematic evaluation of
professional performance: Legally supported procedure and process. Counselor
Education & Supervision, 41, 321-332.









Knoff, H. M. & Prout, H. T. (1985). Terminating students from professional psychology
programs: Criteria, procedures, and legal issues. Professional Psychology-Research and
Practice, 16, 789-797.

Koch, L., Schultz, J., Conyers, L., & Hennessey, M. (2005). Rehabilitation research in the 21st
Century: Concerns and recommendations from members of the National Council on
Rehabilitation Education. Rehabilitation Education, 19, 5-14.

Kutz, S. L. (1986). Defining "impaired psychologist". American Psychologist, 41, 220.

Lamb, D. H., Presser, N. R., Pfost, K. S., Baum, M. C., Jackson, V. R., & Jarvis, P. A. (1987).
Confronting professional impairment during the internship: Identification, due process,
and remediation. Professional Psychology: Research and Practice, 18, 597-603.

Lamb, D. H., Cochran, D. J., & Jackson, V. R. (1991). Training and organizational issues
associated with identifying and responding to intern impairment. Professional
Psychology-Research andPractice, 22, 291-296.

Leahy, M. J., Shapson, P. R., & Wright, G. N. (1987). Rehabilitation practitioner competencies
by role and setting. Rehabilitation Counseling Bulletin, 87, 119-130.

Leigh, I.W., Smith, I.L., Bebeau, M.J., Lichtenberg, J.W., Nelson, P.D., Portney, S., et al.
(2007). Competency assessment models. Professional Psychology: Research and
Practice, 38, 463-473.

Li, C.-S. (2001). Non-academic behavioral indicators of student impairment in CACREP-
accredited master's level counseling programs. Dissertation Abstract International, 62(3-
A). (UMINo. 3010363) Retrieved August 10, 2006, from Dissertation and Theses
database.

Linderman, C. (1981). Priorities Within the Health Care System: A Delphi Study. Kansas City:
American Nurses Association.

Lumadue, C. A. & Duffey, T. H. (1999). The role of graduate programs as gatekeepers: A model
for evaluating student counselor competence. Counselor Education & Supervision, 39,
101-109.

Magnuson, S., Wilcoxon, S.A., & Norem, K. (2000). A profile of lousy supervision: Experienced
counselor's perspectives. Counselor Education & Supervision, 39, 189-202.

Maki, D. R. & Delworth, U. (1995). Clinical supervision: A definition and model for the
rehabilitation counseling profession. Rehabilitation Counseling Bulletin, 38, 282-293.

McAdams, C.R., & Foster, V.A. (2007). A guide to just and fair remediation of counseling
students with professional performance deficiencies. Counselor Education and
Supervision, 47(1), 2-13.









McBride, A.J., Pates, R., Ramadan, R., & McGowan, C. (2003). Delphi survey of experts'
opinions on strategies used by community pharmacists to reduce over-the-counter drug
misuse. Addiction, 98(4), 487-497.

McCarthy, H. & Leierer, S. J. (2001). Consumer concepts of ideal characteristics and minimum
qualifications for rehabilitation counselors. Rehabilitation Counseling Bulletin, 45, 12-23.

Mearns, J. & Allen, G. J. (1991). Graduate students experience in dealing with impaired peers,
compared with faculty prediction: An exploratory study. Ethics and Behavior, 1, 191-
202.

Michaelson, S. D., Estrada-Hernandez, N., & Wadsworth, J. S. (2003). A competency-based
evaluation model for supervising novice counselirs-in-training. Rehabilitation Education,
17, 215-223.

Miller, H. L. (1979). A procedure for nonacademic failure of graduate students in psychology.
Professional Psychology-Research and Practice, 10, 4-5.

Miller, H. L. & Rickard, H. C. (1983). Procedures and students' rights in the evaluation process.
Professional Psychology-Research and Practice, 14, 830-836.

Miller, W. R. & Rollnick, S. (2002). Motivational interviewing: Preparing people for change.
(2nd ed.) New York: Guilford Press.

Murray, G.C., Portman, T., & Maki, D.R. (2003). Clinical supervision: Developmental
differences during pre-service training. Rehabilitation Education, 17, 19-32.

Muthard, J. E. & Salomone, P. (1969). The roles and functions of the rehabilitation counselor.
Rehabilitation Counseling Bulletin, 13, 81-168.

Myrick, R. D. & Kelly, F. D. (1971). A scale for evaluating practicum students in counseling and
supervision. Counselor Education & Supervision, 10, 330-336.

National Counsel on Rehabilitation Education. (2007). National Counsel on Rehabilitation
Education. Retrieved November 7, 2007, from the World Wide Web:
http://www.rehabeducators.org/about.htm

Nelson-Jones, R. (2005). Introduction to counselling skills: Texts and activities. (5th ed.).
London: Sage.

Norman, G. R., Streiner, D. L. (2000). Biostatistics: The bare essentials (2nd ed.), Toronto: B. C.
Decker.

Getting, E. R. & Michaels, L. (1982). Oemnig _I\ ,/Ihael' anchored rating for therapists. Fort
Collins: Rocky Mountain Behavioral Science Institute.

Okoli, C., & Pawlowski, S.D. (2004). The Delphi method as a research tool: An example, design
considerations and applications. Information & Management, 42, 15-29.









Oliver, M. N. I., Bernstein, J. H., Anderson, K. G., Blashfield, R. K., & Roberts, M. C. (2004).
An exploratory examination of student attitudes toward "impaired" peers in clinical
psychology training programs. Professional Psychology: Research and Practice, 35, 141-
147.

Olkin, R. & Gaughen, S. (1991). Evaluation and dismissal of students in master's level clinical
programs: Legal parameters and survey results. Counselor Education & Supervision, 30,
276-288.

Pope, V. T. & Kline, W. B. (2007). The personal characteristics of effective counselors: What 10
experts think. Psychological Reports, 84, 1339-1344.

Procidano, M. E., Busch-Rossnagel, N. A., Reznikoff, M., & Geisinger, K. (1995). Responding
to graduate students' professional deficiencies: A national survey. Journal of Clinical
Psychology, 51, 426-433.

QSR International (2007). NVivo 7 [Computer software]. Melbourne: QSR International.

Ragg, D. M. (2001). Building effective helping skills: The foundation of generalist practice.
Boston: Allyn and Bacon.

Rasch, G. (1960). Probabilistic models for some intelligence and attainment tests. Copenhagen:
Danmarks Paedagogiske Institute.

Reising, G.N., & Daniels, M.H. (1983). A study of Hogan's model of counselor development
and supervision. Journal of Counseling Psychology, 30, 235-244.

Robiner, W., Fuhrman, M., & Ristvedt, S. (1993). Evaluating difficulties in supervising
psychology interns. The Clinical Psychologist, 46, 3-13.

Robiner, W., Fuhrman, M., Ristvedt, S., Bobbitt, B., & Schirvar, J. (1994). The Minnesota
supervisory inventory (MSI): Development, psychometric characteristics, and
supervisory evaluation issues. The ClinicalPsychologist, 47, 4-17.

Rowe, G., & Wright, G. (1999). The Delphi technique as a forecasting tool: Issues and analysis.
International Journal ofForecasting, 15, 353-375.

Rubin, S.E., Matkin, R.E., Ashley, J., Beardsly, M.M., May, V.R., Onstott, K., et al. (1984).
Roles and functions of certified rehabilitation counselors. Rehabilitation Counseling
Bulletin, 27, 199-224.

Rubin, S.E., McMahon, B.T., Chan, F., & Kamnetz, B. (1998). Research directions related to
rehabilitation practice: A Delphi study. Journal ofRehabilitation, 64, 19-26.

SPSS Inc. (2001). SPSS for Windows [Computer software]. Chicago: SPSS Inc.

Schultz, J. C., Copple, B. A., & Ososkie, J. N. (1999). An integrative model for supervision in
rehabilitation counseling. Rehabilitation Education, 13, 323-334.









Schultz, J.C., Ososkie, J.N., Fried, J.H., Nelson, R.E., & Bardos, A.N. (2002). Clinical
supervision in the public rehabilitation counseling setting. Rehabilitation Counseling
Bulletin, 45, 213-222.

Schwebel, M. & Coster, J. (1998). Well-functioning in professional psychologists: As program
heads see it. Professional Psychology-Research and Practice, 29, 284-292.

Scott, C.G., Nolin, J., & Wilburn, S.T. (2006). Barriers for effective clinical supervision for
counseling students and postgraduate counselors: Implications for rehabilitation
counselors. Rehabilitation Education, 20, 91-102.

Shaw, L.R., Leahy, M.J., Chan, F., & Catalano, D. (2006). Contemporary issues facing
rehabilitation counseling: A Delphi study of the perspectives of leaders of the discipline.
Rehabilitation Education, 20, 163-178.

Shebib, B. (2003). Choices: Counseling skills for social workers and other professionals.
Boston: Allyn and Bacon.

Smith, R. M. (1994). A comparison of the power of Rasch total and between item fit statistics to
detect measurement disturbances. Educational and Psychological Measurement, 54, 42-
55.

Stebnicki, M. A., Allen, H. A., & Janikowski, T. P. (1997). Development of an instrument to
assess perceived helpfulness of clinical supervisory behaviors. Rehabilitation Education,
11, 307-322.

Stebnicki, M. A. (1998). Clinical supervision in rehabilitation counseling. Rehabilitation
Education, 12, 137-159.

Stoltenberg, C.D., & Delworth, U. (1988). Developmental models of supervision: It is
developmental. A response to Holloway. Professional Psychology: Research and
Practice, 19, 134-137.

Stoltenberg, C.D., McNeil, B.W., & Crethar (1995). Persuasion and development in counsellor
supervision. The Counselling Psychologist, 23, 633-648.

Strauss, A., & Corbin, J. (1998). Basics of qualitative research: Techniques and procedures for
developing grounded theory (2nd ed.). London, England: Sage.

SurveyMonkey (2007). SurveyMonkey [Computer software]. Portland, OR: SurveyMonkey.

Tarvydas, V. M. (1995). Ethics and the practice of rehabilitation counselor supervision.
Rehabilitation Counseling Bulletin, 38, 294-306.

Tedesco, J. F. (1982). Premature termination of psychology interns. Professional Psychology-
Research and Practice, 13, 695-698.









Thielsen, V. A. & Leahy, M. J. (2001). Essential knowledge and skills for effective clinical
supervision in rehabilitation counseling. Rehabilitation Counseling Bulletin, 44, 196-208.

Thurstone, L. L. (1927). The unit of measurement in educational scales. Journal of Educational
Psychology, 18, 505-524.

Thurstone, L. L. (1928). Attitudes can be measured. American Journal of Sociology, 33, 529-
554.

Vacha-Haase, T.R. (1995). Impaired graduate students in APA-accredited clinical, counseling,
and school psychology programs. Dissertation Abstract International, 56(09-A). (UMI
No. 9539306) Retrieved July 10, 2006, from Dissertation and Theses database.

Vacha-Haase, T., Davenport, D. S., & Kerewsky, S. D. (2004). Problematic students:
Gatekeeping practices of academic professional psychology programs. Professional
Psychology-Research and Practice, 35, 115-122.

Vazquez-Ramos, R., Leahy, M., & Hernandez, N.E. (2007). The Delphi method in rehabilitation
counseling research. Rehabilitation Counseling Bulletin, 50, 111-118.

Walborn, F. S. (1996). Process variables: Four common elements of counseling and
tpL 1h' It /i .py. Pacific Grove: Brooks/Cole.

Wilkerson, K. (2006). Impaired students: Applying the therapeutic process model to graduate
training programs. Counselor Education & Supervision, 45, 201-217.

Wiley, M.O., & Ray, P.B. (1986). Counseling supervision by developmental level. Journal of
Counseling Psychology, 33, 439-445.

Williams, P.L., & Webb, C. (1994). The Delphi technique: A methodological discussion. Journal
ofAdvanced Nursing, 19, 180-186.

Woodyard, C. L. (1997). Indicators of impairment in incoming and ongoing master's-level
counseling students. Dissertation Abstracts International, 58(03-A). (UMI No. 9724633)
Retrieved August 10, 2006, from Dissertation and Theses database.

Worthington, E.L. (1987). Changes in supervision as counselors and supervisors gain
experience: A review. Professional Psychology: Research and Practice, 18, 189-208.

Wright, B. D. & Linacre, J. M. (1989). Observations are always ordinal; Measurements,
however, must be interval. Archives of Physical Measurement and Rehabilitation, 70,
857-860.

Wright, B. D. (1997). A history of social science measurement. Educational Measurement:
Issues andPractice, 16, 33-45, 52.









BIOGRAPHICAL SKETCH

Michael D. Moorhouse, MHS, CRC, is a doctoral candidate in the Rehabilitation Science

Doctoral program at the University of Florida College of Public Health and Health Professions

(Gainesville, FL). Mr. Moorhouse received his master's degrees in Rehabilitation Counseling

from the University of Florida in 2001. Accomplishments during Mr. Moorhouse's doctoral

student career include being a recipient of a four year alumni fellowship, which allowed him to

conduct his dissertation studies; recipient of the 2007 John Muthard Award for excellence in

research from the University of Florida College of Public Health and Health Professions,

Department of Behavioral Science and Community Health; and recipient of the inaugural Ronald

J. Spitznagel Outstanding Service Award. While completing the requirements for his doctoral

degree, Mr. Moorhouse worked as a research assistant for Drs. Jamie Pomeranz and Mary

Hennessey in the Department of Behavioral Science and Community Health at the University of

Florida.





PAGE 1

1 IDENTIFYING BENCHMARK COMPETENCY CRITERIA FOR A REHABILITATION COUNSELING CLINICAL SUPERVISION INSTRUMENT: A DELPHI STUDY By MICHAEL DEVIN MOORHOUSE A DISSERTATION PRESENTED TO THE GRADUATE SCHOOL OF THE UNIVERSITY OF FLOR IDA IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY UNIVERSITY OF FLORIDA 2008

PAGE 2

2 2008 Michael Devin Moorhouse

PAGE 3

3 To my parents Ken and Rose

PAGE 4

4 ACKNOWLEDGMENTS I would first like to thank m y dissertation comm ittee for all their mentorship and assistance through out this process. To Dr. Linda Shaw, my chair, I appreciate all your direction and supervision during my training; your expertise has contributed immens ely to this project. To Dr. Mary Ellen Young, thank you for your generosity during this study; you always made yourself available no matter how busy you were. To Dr. John Rosenbek, you are an exemplary educator and professional who continually demonstrates genuine care and intere st in your students professional development thank you. To Dr. Pe ter Sherrard, I appreciat e all your contributions to this project. You always helped challenged me to conceptualize my ideas holistically and with great thought. I would also like to thank to Drs. Mary He nnessey, Tracey Ba rnett, and Lynn Koch for their continued support and assistance during this study. In a ddition, I would like to extend a special thanks to my friend and colleague Dr. Jamie Pomeranz. Jamie, your mentorship has greatly impacted my professional development and I truly would not be the researcher or teacher I am today without your guidance. I would also like to acknowledge my familys continued support during these last five years. To my mother Rose, your unconditional love, support, and guidance through out the years have provided me with the strength to overcome many obstacles and made me a better person. To my father Ken, the immense work ethic, humility, and perseverance you have shown thro ugh out the years have been not only an example, but an inspiration to me. I love you bot h very much. Lastly, I would like to thank the love of my life, Donna. I cannot begin to ar ticulate the profound impact youve had on me. Everyday I think about how lucky I am to have you and cannot wait for us to begin our future together.

PAGE 5

5 TABLE OF CONTENTS page ACKNOWLEDGMENTS...............................................................................................................4 LIST OF TABLES................................................................................................................. ..........8 LIST OF FIGURES.........................................................................................................................9 ABSTRACT...................................................................................................................................10 CHAP TER 1 INTRODUCTION..................................................................................................................12 Background.............................................................................................................................12 The Clinical Supervision Role................................................................................................15 The Need for a Standardized Instrument to Evaluate Rehabilitation Counseling Trainee Com petency during Clin ical Supervision...........................................................................15 Methods for Identifying Competency Criteria.......................................................................23 Goals of Research.............................................................................................................. .....23 Research Question 1............................................................................................................ ...23 2 LITERATURE REVIEW.......................................................................................................24 Defining Impairment..............................................................................................................24 Rehabilitation Counseling Accreditation and Code of Ethics Literature ............................... 27 Prevalence of Trainee Impair m ent and Dismissal Rates........................................................ 29 Identifying Trainee Problems................................................................................................. 31 Due Process Procedures..........................................................................................................35 Clinical Supervision Evaluation Criteria................................................................................44 Supervision Models............................................................................................................. ...47 Clinical Supervision Instruments............................................................................................51 The Rasch Model....................................................................................................................57 3 METHODOLOGY................................................................................................................. 63 Research Question 1............................................................................................................ ...63 Study Design................................................................................................................... 63 Sampling..........................................................................................................................65 Delphi Procedure.............................................................................................................66 Delphi round 1..........................................................................................................67 Delphi round 2..........................................................................................................69 Delphi round 3..........................................................................................................70 Pilot Testing.....................................................................................................................71

PAGE 6

6 4 RESULTS...............................................................................................................................74 Delphi Study...........................................................................................................................74 Panel Demographics........................................................................................................ 74 Delphi Round 1................................................................................................................75 Delphi Round 2................................................................................................................76 Delphi Round 3................................................................................................................77 5 DISCUSSION.........................................................................................................................89 Introduction................................................................................................................... ..........89 Overview of Significant Findings........................................................................................... 89 Differences in Participant Response Sets........................................................................ 89 Item Identification.................................................................................................... 91 Items Consensus....................................................................................................... 92 Limitations.................................................................................................................... ..........96 Implications................................................................................................................... .........98 Education.........................................................................................................................98 Clinical competency dimensions and benchmarks................................................... 98 Supervision evaluation continuity.......................................................................... 103 Improving communication between supervisor and trainee...................................104 Public Policy..................................................................................................................106 Association of American Colleges an d Universities / Council for Higher Education A ccreditation..................................................................................... 106 Council on Rehabilitation Educatio n / Comm ission on Rehabilitation Counseling Certification..................................................................................... 107 Future Research....................................................................................................................108 Conclusion............................................................................................................................110 APPENDIX A DELPHI ROUND 1 SURVEY.............................................................................................112 B ROUND 1 EMAIL TO REHABILITA TION COUNSELING EDUCATORS ................... 115 C INFORMED CONSENT......................................................................................................117 D ROUND 1 FRIENDLY REMINDER EMAIL ..................................................................... 118 E AUDIT TRIAL.................................................................................................................... .120 F ROUND 2 EMAIL TO REHABILITA TION COUNSELING EDUCATORS ................... 131 G DELPHI ROUND 2 SURVEY.............................................................................................132 H ROUND 2 FRIENDLY REMINDER EMAIL ..................................................................... 141

PAGE 7

7 I ROUND 3 EMAIL TO REHABILITA TION COUNSELING EDUCATORS ................... 142 J DELPHI ROUND 3 SURVEY.............................................................................................143 K ROUND 3 FRIENDLY REMINDER EMAIL ..................................................................... 160 LIST OF REFERENCES.............................................................................................................161 BIOGRAPHICAL SKETCH.......................................................................................................170

PAGE 8

8 LIST OF TABLES Table page 2-1 Frequencies of impaired tr ainee characteris tics as reported by previous studies (select non-academic variables).................................................................................................... 34 3-1 Pilot test findings and revisions......................................................................................... 72 4-1 Participant demographics................................................................................................... 75 4-2 Participant credentials.................................................................................................... ....76 4-3 Items with means between 4.50 and 5.00 arranged according to consensus..................... 78 4-4 Items with means between 4.00 and 4.49 arranged according to consensus..................... 82 4-5 Items with means between 3.50 and 3.99 arranged according to consensus..................... 86 4-6 Items with means between 3.00 and 3.49 arranged according to consensus..................... 87 4-7 Items with means below 3.00 arranged according to consensus....................................... 88

PAGE 9

9 LIST OF FIGURES Figure page 2-1 Sample free and test free related to height......................................................................... 59 2-2 Hierarchy of counseling abil ity com pared to total score................................................... 61

PAGE 10

10 Abstract of Dissertation Pres ented to the Graduate School of the University of Florida in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy IDENTIFYING BENCHMARK COMPETENCY CRITERIA FOR A REHABILITATION COUNSELING CLINICAL SUPERVISION INSTRUMENT: A DELPHI STUDY By Michael Devin Moorhouse August, 2008 Chair: Linda Shaw Major: Rehabilitation Science The purpose of this study was to identify a nd draw consensus on a comprehensive set of competency criteria useful for clinical superv ision evaluation. An essential function of any counseling profession is to ensure that gradua te students are persona lly and professionally competent to practice within a community se tting. Although the rehabilitation counseling profession may provide general gui dance for competency standards, the responsibil ity to ensure that students have attained the necessary knowledge and skill routinely falls to the training faculty. Consequently, training programs ha ve a substantial amount of autonomy when developing these competency criteria. Even though some flexibility with th ese tasks is necessary for individual programmatic goals considerable variability prom otes inconsistently monitoring and evaluation of students. The clinical portion of the training programs is perhaps most vulnerable to these challenges due to the dual ro le of the supervisor, philosophical differences regarding the purpose of clinical training, supervisor preparedness, the supe rvisors fear of litigation, and difficulties in measuring clinical competence. Despite these challenges, this research was intended to identify a variety of clinical skills, attribut es, and behaviors to aid clinical supervisors measuring trainees compet ency during clinical supervision. Using the

PAGE 11

11 Delphi methodology, 21 rehabilitation counseling e ducators identified a variety of performance benchmarks useful for clinical supervision as sessment. The study resulted in 185 items useful for clinical competency evaluation. Items represented an assortment of competency domains including general indicators of the counseling process, specific clinical process skills, case conceptualization items, treatment planning items, professional behaviors, professional development items, self care ite ms, personal attributes, and item s specific to the supervision process. Implications for education and public policy are discussed as well as future development of a clinical supe rvision instrument measuring trainee clinical performance.

PAGE 12

12 CHAPTER 1 INTRODUCTION Background An essential function of any counseling profession is to ensure that graduate students are personally and professionally com petent to prac tice within a community setting. In accordance with this fundamental principle, counseling educators develop appropriate curriculum and learning objectives to make certa in that students receive the required training for professional practice. Although each counseling profession id entifies the general knowledge domains and expectations for adequate trainee preparation, th e responsibility to ensu re that students have attained the necessary knowledge and skill norm ally falls to the training faculty (Robiner, Fuhrman, Ristvedt, 1993). The same professi onal standards that re quire students to be professionally competent provide only general gu idelines regarding evaluation criteria and expectations of minimal student performance. Thus, faculty members typically rely on homegrown evaluation instruments that vary from program to program. Further complicating the evaluation problem is the fact that training programs may have difficulty identifying and addres sing their students nonacademic limitations (Hahn & Molnar, 1991). More specifically, programs frequently st ruggle to identify and intervene with students who perform well academically, but may have char acter or psychological issues that interfere with their counseling ability (Bemak, Epp, & Keys, 1999). The challenges associated with personal suitability coupled with non-specific evaluation criteria may result in inconsistent evaluation of the student. Consequently, progra ms may be faced with the difficult decision of endorsing questionable trainees for certification or licensure. By allowing impaired trainees to graduate without remediation, c ounselor educators not only endange r their professi onal integrity, but more importantly jeopardize their service to the public.

PAGE 13

13 Similar to other clinical counseling programs, the rehabilitation counseling field relies on professional organizations like the Commissi on on Rehabilitation Counselor Certification (CRCC) and the Council of Rehabilitation Education (CORE) to help guide trainee evaluation. However, these organizations provide only gene ral guidelines regarding trainee competence and therefore rehabilitation counseli ng programs must develop additional criteria to supplement the existing standards. For example, CORE (2002) li sts several requisite kno wledge domains for the rehabilitation counseling profession. One domain, Counseling and Consultation, (C.5) describes the general tasks for which a rehab ilitation counseling trainee is expected to demonstrate competency within the counseling process. More specifically, the domain contains tasks such as: (a) conduct individual counseli ng sessions with consumers (C.5.1), (b) develop and maintain a counseling relati onship with the individual (C.5.2), (c) recommend strategies to assist the consumer in solving identified problems that may im pede the rehabilitation process (C.5.6). While this information clearly facilitates the academic process and provides a general outline for assessing student competence, nowhere in the accreditation standards is competency operationally defined. Faculty members mu st therefore determine what constitutes competence and must also determine how to ev aluate the students rehabilitation counseling ability. Rehabilitation counseling programs ha ve a substantial amount of autonomy when developing these criteria and proced ures. Although some flexibility with these tasks is necessary to accommodate each programs goals, considerable variability between these programs may result in counseling programs inconsistently m onitoring and evaluating students (Robiner et al., 1993).

PAGE 14

14 Research on rehabilitation counselor ro les and functions (Leahy, Shapson, & Wright, 1987; Muthard & Salomone, 1969; Rubin, Matkin, et al., 1984) might be a resource in determining the competencies upon which students should be assessed. However, these studies focus only on the general tasks of th e rehabilitation counseling profe ssional. While this research provides invaluable informati on pertaining to the professiona l identity of rehabilitation counselors, the studies examining the roles and functions of rehabilita tion counselors provide limited information pertaining to the trainees co unseling ability. As Ja nikowski (1990) notes, the rehabilitation counseling literature has typi cally focused on the functions of the profession, rather than the competence of the professional. In this instance, Janikowsk i refers to function as the specific tasks or job duties related to occupation, whereas competency is the underlying characteristics of the professional th at lead to occupational success. For educational programs to consistently mon itor and evaluate trainees, supervisors should define competencies that trainees should acquire and the standards that the trainees should reach, as well as the criteria used to evaluate those co mpetencies and the thresholds at which trainees meet those standards (Robiner et al., 1993). Ev en though it is unlikely that uniform evaluation criteria will suit every program, the rehabilitation counseling pr ofession could, as a first step, operationally define the concept of counseling competence and develop a comprehensive set of behaviors (e.g. ability to empathi ze, ability to reflectively liste n, etc.) that will aid in trainee evaluation. By developing specific, behaviorally define d criteria, trainer feedback becomes more concise and faculty can more consistently eval uate whether the student meets program goals (Freeman, 1985). As Kerl, Garcia, McCullough, a nd Maxwell (2002) opine, objective evaluation helps faculty members communicate with their tr ainees as well as helps prevent capricious

PAGE 15

15 evaluator judgment. Thus, rehabilitation counse ling educators will strengthen the profession by ensuring programs produce appropriate counselors who demonstrate adequate counseling ability. The Clinical Supervision Role Even when trainee evaluation is continuous, m any student limitations are more likely to be identified during the clinical portion of the program (Lamb, Presser, Pfost, Baum, Jackson, & Jarvis, 1987). Perhaps one area of counselor tr aining most appropriate for identifying trainee competence is during clinical supe rvision. Unlike licensure exams or classroom tests, supervisors can assess how well the trainee applies classr oom knowledge to a counseling session and they therefore have the unique ability to determine the trainees sk ill and judgment (Robiner et al., 1993). Additionally, Pope and Kline (1999) recogni ze that within the counseling profession, the trainees personal characteristics play an integral role during evaluation. Therefore, clinical supervision may offer the best opportunity for training faculty to identify and address the students clinical skill and persona l characteristics. During clinical work, trainees are interacting with real clients rather than in simulated cl assroom settings, and ther efore faculty can better assess the trainees professional and personal shortcomings that might not have been apparent during coursework. The Need for a Standardized Instrument to E valuate Rehabilitation Counseling Trainee Competency during Clinical Supervision Clinical supervision is a basic component of c ounselor training that has developed into an independent body of theory and research (Ber nard & Goodyear, 2004). However, within the rehabilitation counseling professi on, clinical supervision research continues to be understudied (Herbert & Richardson, 1995; Herbert, War d, & Hemlick, 1995; Maki & Delworth, 1995; Stebnicki, 1998). More speci fically, the rehabilitation coun seling literature has vastly overlooked trainee evaluation from the supervisors perspective. Although Thielsen and Leahy

PAGE 16

16 (2001) have identified evaluation and assessm ent as a critical knowledge domain for fieldbased clinical supervis ion, there is a paucity of rehabilitation counsel ing research regarding competency criteria for evaluating rehabilitati on counseling trainees. In fact, when Koch, Schultz, Hennessey, and Conyers (2005) survey ed rehabilitation c ounseling educators, professionals, and students, they suggested that the identification and assessment of clinical supervision competencies was an area warrantin g further investigation in rehabilitation education (p.10). Despite this need, the reha bilitation counseling lite rature has focused on various other aspects of clinical supervision including identifying helpful supervisor practices (Stebnicki, Allen, & Janikowski, 1997), iden tifying current supervis ion methods use in rehabilitation counseling (Herbe rt, 2002), describing ethical considerations within clinical supervision (Blackwell, Strohm er, Belcas, & Burton, 2002; Tarvyda s, 1995), describing models to ensure proper trainee due process (Michaelson, EstradaHernandez, & Wadsworth, 2003), identifying the need for masters level supe rvision training (Scott, Nolin & Wilburn, 2006), determining what knowledge is es sential to provide effective supervision (Thielsen & Leahy, 2001), and developing supervision models for bot h the educational setting (Maki & Delworth, 1995; Schultz, Copple, & Ososkie, 1999) and publ ic setting (Schultz, Ososkie, Fried, Nelson, & Bardos, 2002). Although this body of research greatly expands our understanding and conceptualization of clinical supervision, it fails to fully a ddress what Bernard and Goodyear (2004) suggest may be the nucleus of clin ical supervision tr ainee evaluation. In 1995, Maki and Delworth recognized the need to operationally define clinical supervision specifically for th e rehabilitation counseling pr ofession. Using Bernard and Goodyears (1992) definition as a foundation, Maki and Delworth stated that rehabilitation counseling supervision is a distinct intervention, the use of which requires the trained supervisor

PAGE 17

17 to have specific knowledge and skills in multiple domains, including, but not limited to, education, consultation, and counseling (p.284). The authors hope d that this definition would help supervisors better characterize the relations hip between the supervis or and trainee (Maki & Delworth, 1995). Despite Maki and Delworths (1995) effort to define clinical s upervision relative to rehabilitation counseling, Stebnicki (1998) sugge sted that their defini tion was too general to encompass the unique characterist ics of the profession. Therefore, Stebnicki proposed a more elaborate clinical superv ision definition that soug ht to better reflect reha bilitation counseling. He defines clinical supervision as: a specialty area that facilitates a positive interp ersonal relationship w ith supervisees using a diversity of supervisory styl es and approaches in both an individual and group dynamic process and is facilitated using the roles of educator, consulta nt, and counselor to stimulate and increase the supervisees (a) process ski lls and psychosocial re habilitation counseling strategies to enhance positive and ethical client outcomes, (b) efficacy as a counselor being mindful of the supervisees ongoing developm ental and personal growth needs in the process, and (c) conceptualization of salient f eatures concerning the clients disability and other critical aspects related to his or her problem issues (p.139-140). Although each definition provides a general framework for rehabi litation counseling, both fail to mention the evaluative nature of clinical supe rvision. As Bernard and Goodyear (2004) state, evaluation is implicit in the supervisors mandate to safeguard clients, both those who will be seen by the supervisee and those who would be seen in the future (p.11). Specifically, supervisors evaluate students to ensure they reach a minimal level of professionally competence and exhibit suitable personal characteris tics for working with clients. In general, trainee evaluation reveals a set of complex issues that permeate any evaluative process. As previously stated, clinical supervisors within the academic setting aid in trainee development, as well as limit impaired train ees from entering the counseling profession. Robiner et al. (1993, p.4) assert that these dua l roles of nurturant teacher and vigilant

PAGE 18

18 gatekeeper may serve as the primary source of s upervisory conflict. A rehabilitation counselor, like any counselor or therapist, is trained to be non-judgmental and facilitate client growth. However, the qualities that make for a good counsel or may also interfere with the supervisors ability to evaluate a trainee. Hoffman, Hill, Ho lmes, and Freitas (2005) suggest that supervisors may feel uncomfortable when pr oviding the trainee subjective f eedback, particularly when it involves the trainees personality or professional issues. In fact, Gizara an d Forrest (2004) found that the personal impact for supervisors when dealing with an impaired trainee was an overwhelming concern. Specifically, the authors note that none of the responses brought the participant group together so dram atically as inquiries about the personal impact of dealing with intern impairment (p.136). As Bernard and Goodyear (2004) note, imagine a situation where the supervisor uses the trainees progress as the critical indicator for evaluation. The trainee has worked hard and improved duri ng the practicum, but continue s to perform below program expectations. Now the supervisor must weigh the students effort and progress against the ethical responsibility to the prof ession and potential clients (Ber nard & Goodyear). Thus, trainee evaluation can lead to a disturbing role sh ift for supervisors (Hahn & Molnar, 1991, p.417), and cause them some level of personal discord if remediation and/or di smissal procedures are employed, particularly if a relationship with the student exists (Gizara & Forrest). Similar to the personal struggle educators ma y face when evaluating a trainee, Gizara and Forrest (2004) found that supe rvisors may feel unprepared wh en dealing with problematic trainees. Specifically, the author s note that several participants felt ill-equipped to identify and address problematic trainees. Moreover, the au thors note that supervis ors may doubt their own ability to appropriately identify problematic train ees and in fact, may feel an increased sense of vulnerability when their own supervisory skills came to the forefront.

PAGE 19

19 Another challenge to effective tr ainee evaluation is the fear that trainees may initiate legal action upon the supervisor, program, and/or school. Several authors (Frame & Stevens-Smith, 1995; Tedesco, 1982; Vacha-Haase, Davenport, & Kerewsky, 2004) suggest that litigation may account for the faculty members reservations rega rding trainee evaluation. In fact, Vacha-Haase et al. reported that fear of litigation was highest when students were terminated for a lack of interpersonal skills and supervision difficulties. Specifically, the authors note that, it may be difficult to document more subjectively determ ined problematic behavi ors adequately, whereas problematic behaviors with objective criteria ma y be easier for faculty to identify confidently and act on with more certainty (Vacha-Haase et al., p.119). Despite th e authors assertion, several authors (Frame & Smith-Stevens; Kno ff & Prout, 1985; Olkin & Gaughen, 1991; Kerl et al., 2002) note that various court decisions have upheld the programs right to dismiss students who exhibit personal or profe ssional deficiencies. Although th ese precedents may support the counseling programs opinion, Knoff and Prout ca ution that these court rulings do require programs follow proper due process procedures. Consequently, several authors (Baldo, Soft as-Nall, & Shaw, 1997; Bemak, Epp, & Keys, 1999; Frame & Stevens-Smith, 1995; Lamb, Presse r, Pfost, Baum, Jackson, & Jarvis, 1987; Lumadue & Duffey, 1999) have writ ten specific remediation and dism issal procedures to ensure counseling and psychology programs provide adequate due process. Although the literature and professional organizations have re cognized the importance of due pr ocess, approximately half of the programs continue to operate without thes e policies and procedures in place (Olkin, & Gaughen, 1991; Vacha-Haase et al., 2004). This may be, at least in part because programs that have comprehensive policies and procedures in pl ace often find that the dismissal process can be

PAGE 20

20 complicated and result in additional programma tic modifications (McAdams, Foster, & Ward, 2007). A fourth challenge associated with trainee ev aluation is the level of support a supervisor receives from the department and fellow f aculty members may greatly impact the evaluation process (Gizara & Forrest, 2004). In fact, Vacha-Haase et al. (2004) found that training directors identified disagreement among faculty as the se cond biggest barrier to dismissing a student behind fear of litigation. In their qualitative stud y investigating the experiences of supervisors in dealing with impaired trainees, Gizara and Forre st found that the level of conflict within the supervisory group can impact the evaluation pro cess. The authors found that supervisory disagreement can manifest in a variety of ways For example, the way in which a supervisor conceptualizes the clinical proce ss can impact the level of supervis ory support. Specifically, one supervisor may view clinical supervision/internship to be a development process, while another supervisor may perceive that s upervision is primarily a gatek eeping function whereby trainees must meet some minimal level of competence. In addition to differences in perception about the purpose of clinical supervision, Gizara and Forrest reported that supervisor s also suggested that preexisting interpersonal conflict within the su pervisory group can in fluence the evaluation process to the point where some supervisors revealed that they may filter the amount of information they provide when discussing their supervisees. In addition to the challenges rehabilitati on counseling educators face when evaluating a problematic trainee, a programs failure to ad equately address the trainees deficits can negatively impact other student s in the training program. Oliver, Bernstein, Anderson, Blashfield, and Roberts (2007) surveyed gradua te students in clinical psychology to examine their attitudes toward impaired peers. Among other things, the authors found that students may

PAGE 21

21 not only resent problematic students for causing them additional work and lost opportunity, but may also resent faculty members who put off a ddressing a problematic tr ainee or simply allow him or her to proceed through the program without meeting some specified standard. Similarly, Mearns and Allen (1991) found that students viewed faculty member s as being significantly less active than faculty viewed themselves when a ddressing an impaired gr aduate student. In addition, the authors found that students demonstrate a variety of emotions when confronted with an impaired peer. Specifically, students fr equently reported feelings of angry, conflicted, frustrated, apprehensive, and concerned. Inte restingly, Mearns and Allen found that faculty members tended to overestimate the number of students who sought to protect their impaired peer. In fact, when confronted with an impaired peer, students reported that their feelings of ethical obligation and betrayal outweighed sentiments of loyalty to their problematic peers (p.198). Additionally, Mearns and Allen found that although the overall incidence of perceived impairment and ethical improprieties was low, the reported incidence of problematic functioning negatively correlated with the climate of the program. Thus, the chal lenges associated with properly evaluating and remediating a problemat ic trainee not only impacts the faculty and trainee, but may also alienate othe r trainees from the program. While the literature regarding clinical supe rvision may bring attention to problematic trainees, a consistent methodology for adequa tely measuring the trainees counseling competency remains elusive. Bernard and G oodyear (2004) recognize th at within clinical supervision, the difficulty of es tablishing criteria for evaluation and the equally difficult task of measuring them is a professional reality (p.23) According to Lamb, Cochran, & Jackson (1991), programs generally assess general student performance in three domains: (a) knowledge and application of professional standards, (b ) competency, and (c) personal functioning.

PAGE 22

22 Although the counseling and psychology literature is replete with articles and books describing various professional competencies and personal attributes appropr iate for these domains (Corey, 2004; Cormier & Cormier, 1991; Hensley-Choate Smith, & Spruill, 2005; Hill, 2004; Johnson & Campbell, 2002; McCarthy & Leierer, 2001; Miller & Rollnick, 2002; Nelson-Jones, 2005; Pope & Kline, 1999; Ragg, 2000; Shebib, 2002), the specifi c criteria used to evaluate the trainee among the programs is quite variable. As Robine r et al. (1993) state, until there is consensus regarding performance standards within spec ific competence domains, all evaluation is vulnerable to subjectivity. In an effort to aid in trainee evaluatio n, several authors (Bernard, 1997; Eriksen, & McAuliffe, 2003; Hackney & Cromier, 1994; Lu madue & Duffey, 1999; Myrick & Kelly, 1971; Oetting & Michaels, 1982; Robiner, Fuhrman, Ristvedt, Bobbitt, & Schirvar, 1994) have developed instruments to measure the trainee s professional competency and character. Although each instrument provides an excellent resource for clini cal supervisors, each one was developed for professions other than rehabi litation counseling. Additionally, no clinical assessment instrument has been validated usi ng Modern Test Theory (MTT). Compared to Classical Test Theory (CTT), MTT techniques li ke Rasch analysis allow the researcher to examine the psychometric properties of an instrument at the item level, rather than at the instrument level (Bond & Fox, 2001). As a result, the Rasch analysis pro duces a set of unique psychometric statistics, such as an item hierar chy, that allows supervisors to determine the trainees ability level based on specific performan ce behaviors arranged in hierarchal order from least to most difficult. In other words, the trai nees ability level is base d on the types of items near his or her ability level, ra ther than simply relying on a sum of scores. Practically speaking, the supervisor knows what types of counseling behaviors are belo w, at, and above the trainees

PAGE 23

23 ability level. Thus, supervisors can simulta neously measure trainee competence and target specific interventions while docum enting the trainees professiona l development over the course of his or her academic career. Methods for Identifying Competency Criteria One m ethod appropriate for identifying rehabilitation counseling competency criteria is the Delphi method. The Delphi method enables resear chers to elicit anonymous expert opinion, and then refines this information for experts to r eexamine their response(s) compared to the group response (Vazquez-Ramos, Leahy, & Estrada-Hern andez, 2007). The process consists of a minimum of three rounds with the ultimate goal of reaching consensus among the experts without the bias drawbacks of comparable techniques (Linderman, 1981). By identifying specific competency criteria relate d to professional skills and pe rsonal suitability, rehabilitation educators can ensure trainees are prepared to counsel people with disabilities. Goals of Research The prim ary goal of this research is to implement a Delphi study in order to identify specific competency criteria useful for the develo pment of a rehabilitation counseling clinical supervision measure. Research Question 1 What professional, personal, a nd clin ical competency items ar e useful for the evaluation of rehabilitation counseling trainees?

PAGE 24

24 CHAPTER 2 LITERATURE REVIEW Defining Impairment In order to u nderstand the inherent difficultie s associated with trainee evaluation, it is necessary to be familiar with the terminology that is often used to describe students who exhibit problematic professional and/or personal behaviors. Unfortunate ly, researchers usually do not operationalize these types of terms (Vacha-Hasse et al., 2004) and as a result, certain terms may be used generically or inappropriately to descri be different sets of student issues (Elman & Forrest, 2007). Gizara and Forrest (2004) caution that without accurately identifying a student issue, the nature of the problem can be obscure d. As a result, educat ors may not only use an ineffective strategy to address the issue, but also be unaware of the relevant literature that could be used to deal with the probl em (Gizara & Forrest). Among a ll the counseling professions, only the psychology literature (Elman & Forrest, 2004; Forrest, Elman, Gizara, & Vacha-Hasse, 1999; Gizara & Forrest; Vacha-Haase et al.) continues to debate the appropriateness of certain terms associated with professional and personal beha viors. Although the rehabilitation counseling profession clearly recognizes th e importance of endorsing only su itable trainees, dialogue about the appropriate use of certain wo rds remains largely unexplored. Although some counseling literature has referred to problematic students as unsuitable (Bernard, 1975; Biaggio, Gasp arikova-Krasnec, & Bauer, 19 83) or having professional deficiencies (Procidano et al., 1995), the most common term used to identify these types of students is impaired (Bemak et al., 1999; Boxley, Drew, & Rangel, 1986; Bradley & Post, 1991; Frame & Stevens-Smith, 1995; Lamb et al., 1987, Li, 2000; Mearns & Allen, 1991; Wilkerson, 2006; Woodyear, 1997). One of the earliest definitions of impairment is offered by Lamb et al. (p.598):

PAGE 25

25 An interference in professional f unctioning that is reflected in one or more of the following ways: (a) an inability and/or unwillingness to acquire and in tegrate professional standards into ones repertoire of professional behavior, (b) an inability to acquire professional skills in order to reach an acceptable level of competency, and (c) an inability to control personal stress, psychological dysfunction, and/or excessi ve emotional reactions that interfere with professional functioning. Although this definition provides a comprehensive summary of va rious trainee issues (i.e. professionalism, development of counseling skills, personal suitableness), the term impairment was originally reserved for professionals whose performance regressed due to such issues as mental illness, substance abuse, or emotional pr oblems (Forrest et al., 1999). The use of the same term in two very different contexts has prompted several authors (Elman & Forrest, 2007; Forrest et al.; Gizara & Forrest 2004; Kutz, 1986) to call for a cl earer delineation of these types of words. Specifically, these authors argue that the term impairment be reserved for situations where the trainee or practitioner first establishe s a baseline of adequate performance and then reverts to an unacceptable level of perf ormance. On the other hand, the term incompetent or not competent should refer to an individual who has not yet reached a minimum level of acceptable performance. In fact, the rehabilitation couns elor code of professi onal ethics (CRCC, 2002) appears to align with this model by reserving the word impairment to identify professionals whose physical, mental, or emotional problems are lik ely to harm the client or others (D.1.i). Similarity, the code uses the term competence when referring to the professionals knowledge (D.1.a, D.1.c., D.1.h, F.5.a, G.1.c). Ironically, Gizara and Forrest (2004) investigated, among ot her topics, how supervisors defined trainee impairment, rather than incompetence The authors report that the participants clearly articulated three themes indicative of trainee impairment, while a fourth was fairly inconsistent. The three identifiable themes were : (a) the interns behavior was professionally harmful or deficient, (b) the behavior was a clea r pattern, and (c) the beha vior was not resolving

PAGE 26

26 (Gizara & Forrest). The ambiguous fourth theme centered on whether or not trainees behavior required a noticeable shift in performance. During the interview, th e respondents categorized impairment in two different groups. The first ca tegory related to the diminished functioning of the trainee and the second category related to trai nees skill deficits (G izara & Forrest). This second category, skill deficits, included a wide range of issues such as clinical skills, interpersonal difficulties, lack of self awareness, and lack of professional responsibility. The authors acknowledge that the vast majority of examples of impairment offered spontaneously by these participants did not meet their original conception of a noticeable shift in performance but rather were more illustrative of skill deficiency (p.133). Hence, psychology intern supervisors appeared to associate the term impairment more closely with the trainees skill development, personal suitability, and adherence to professional st andards, rather than whether the trainee reverted back to an unacceptable level of performance. Although the argument for the appropriate use and understanding of certain terms to describe different trainee issues is clearly valid, impairment appears to be the most common term found in the literature that relates to issues associated with the students persona l suitability and professional development. However, as Elman and Forrest (2007) illustrate, using the term impairment overlaps with its use in the Amer icans with Disabilities Act (ADA; 1990) and the protections and guidance defined therein (p. 502). Specificall y, using the term impairment suggests the trainee has a physical or mental impairment that limits one or more major life activities, and therefore supervisor s may be vulnerable to legal ramifications if they use the word impairment. Consequently, for the purposes of this study, this author will use the terms competency or competent to refer to a trainee s ability to adequately demonstrate counseling

PAGE 27

27 skills, function in a personal appr opriate manner, and follow the pr ofessional standards set forth by the rehabilitation counseling profession. Rehabilitation Counseling Accreditation and Code of Ethics Literature During the clinical portion of a students training, rehabili tation counseling educators m ay be challenged to determine whether a trainee is qualified for professional endorsement. Unlike traditional coursework where students demons trate their competency through classroom participation and assignments, c linical supervisors evaluate the trainees ability to effectively counsel clients. Although this task is requi red by every counseling education program, the criteria used to evaluate a student during clinical supervis ion may vary considerably from program to program. This inconsistency is partic ularly noted during clini cal supervision because trainee evaluation is largely subjective and theref ore it may be more difficult to identify what characteristics and behaviors separate suit able trainees from impaired trainees. The importance of proper trainee evaluation is evidenced in both the 2002 Council of Rehabilitation Education (CORE) accreditation standards and th e 2002 Commission of Certified Rehabilitation Counseling (CRCC) code of professional ethics. CORE states that clinical faculty will review the progress of both practicum and internship students (D.1.7, D.2.4) as well as require a written procedure addressing problem atic trainees (D.1.8, D.3.4). More explicitly, Section G.2.b of the CRCC code of professional ethics further id entifies the evaluation role of faculty members: Rehabilitation counselor educators will clearly st ate, in advance of training, to students and internship supervisees, the levels of comp etency expected, appraisal methods, and timing of evaluation for both didactic and experien tial components. Rehabilitation counselor educators will provide students and internsh ip supervisees with periodic performance appraisal and evaluation feedback throughout the training program. Additionally, both CORE (2002) and the CRCC (2002) recognize that trainee development transcends the acquisition of counseling skills and extends into pers onal suitability. CORE

PAGE 28

28 defines a rehabilitation couns elor as someone who possesse s the knowledge, skill, and attitudes needed to collaborate with people with disabilities. Additionally, thes e accreditation standards are inundated with statements regarding the trainees ability to empower the client, be selfaware, and understand the importan ce of a collaborative relationshi p. Likewise, Section G.3.a of the CRCC recognizes the importance of personal suitability: Rehabilitation counselors, through ongoing evalua tion and appraisal, will be aware of the academic and personal limitations of students and supervisees that might impede performance. Rehabilitation counselors will a ssist students and supervisees in securing remedial assistance when needed, and will dismiss students or supervisees who are unable to provide competent service due to academic or personal limitations. Despite the emphasis on trainee evaluation, neither CORE (2002) nor the CRCC (2002) provide guidance relative to specific evaluation criteria for trai nees and students. In particular, the CORE accreditation standards list general competency domains composed of knowledge areas and trainee behaviors, but do not outline clear expectations for supervisors to evaluate trainee competency. For example, one compet ency domain, Counseling and Consultation, lists 15 knowledge areas (i.e. counseling and persona lity theory, mental health counseling, interviewing and counseling skill development) followed by 15 behaviors th at are indicative of that domain. Although these areas are behavioral, they are too broad to consider for evaluation criteria during clinical supervision. For inst ance, the behaviors include such items as: (a) conduct individual counseling sessions with cons umers (C.5.1), (b) develop and maintain a counseling relationship with the individual (C.5.2), and (c) assist consumers in modifying their lifestyles to accommodate individual functional limitations (C.5.14). Pr actically speaking, how does a faculty member determine whether a st udent is adequately conducting an individual counseling session (C.5.1) without defining what behaviors reflect this process? Moreover, when does the trainees ability level qualify him or her for professional endorsement? Without

PAGE 29

29 more specific indicators of pr ofessional performance, clinical supervisors may be unable to determine whether a trainee is counseling effectively. These general knowledge domains are particularly ineffective as evaluation criteria during clinical supervision beca use counseling is a coalescence of beha viors, skills, and attitudes that are continually evolving. During clinical work, the supervisor and trainee review and process counseling sessions and address other personal issues that may materialize. Although supervisors utilize various resour ces to evaluate the trainee and guide professional development, general evaluation criteria will ultimately lead to inconsis tent evaluation of trainees. Prevalence of Trainee Impairment and Dismissal Rates Several studies (Biaggio et al., 1983; B oxley et al., 1986; Bradley & Post, 1991; Gallessich & Olmstead, 1987; Huprich & Rudd, 2004; Mearns & Allen, 1991; Olkin & Gaughen, 1991; Procidano et al., 1995; Schwebel & Coster, 1998; Tedesco, 1982; Vacha-Haase, 1995) have examined trainee impairment and/or dism issal rates among their respective professions. Although Forrest et al. (1999) believ e that drawing conclusions from this literature is challenging because the authors of these studies requested data and reported findings differently, this literature does provide a general perspective on trainee impairme nt and dismissal trends among psychology and counseling programs. In general, studies that examined trainee impa irment rates found that the vast majority of programs regularly encounter an impaired student In fact, Olkin and Gaughen (1991) found that over two-thirds (76%) of maste rs programs in mental health (e.g. clinical and counseling psychology, counselor education, community psyc hology) identified one to three problematic students per year and 24% identif ied four or more problematic students per year. Similarly, Procidano et al. (1995) found that 89% of res ponding psychology departments reported one or more instances of non-academic professional deficiency among their students within the last 5

PAGE 30

30 years. Mearns and Allen (1991) found that 93% of faculty had perceived an impaired trainee within the last 5 years, while Sc hwebel and Coster (1998) reported that within the last 5 years, psychology programs averaged 2.55 impaired studen ts. On the other hand, Huprich and Rudd (2004) reported that within the last 10 year s, 98% of responding ps ychology doctoral programs and 68% of internship sites were aware of at least one impaired tr ainee. Current indications of trainee impairment suggested that 65% of docto ral programs identified one or more impaired students and 10% of internship sites identified one impaired st udent. Finally, Boxley et al. (1986) and Olkin and Gaughen found remarkably sim ilar annual trainee impairment rates despite surveying different populations. B oxley et al. reported an annu al trainee impairment rate of 4.6%, while Olkin and Gaughen found that the mean percentage of problematic students was 4.8%. In addition to trainee impairment studies, several studies focused on trainee termination rates. As expected, the prevalence of trai nee terminations among psychology and counselor education programs was lower than the reported student impairment rates. Tedesco (1982) surveyed 167 directors from APA approved internship sites and noted that within the last 5 years, 51 trainees did not complete their internship becaus e they either left on their own (n=24) or were terminated by the site director (n=27). Furthe rmore, the internship directors considered an additional 89 trainees for termination. Biaggi o et al. (1983) reported that 86% of doctoral programs and 75% of masters programs in c linical psychology had attempted at least 1 dismissal. Regarding annual te rmination rates, Gallessich and Olmstead (1987) found that over the past three years, counseling psychology programs have terminated an average of one student. Similarly, Vacha-Haase (1995) reported that 52% of responding clinical, counseling, and school psychology programs had at least on e student dismissal within the last 3 years. On the other

PAGE 31

31 hand, Bradley and Post (1991) indicated that w ithin the last 5 years, counselor education programs had dismissed anywhere from zero to 30 students. Of the programs reporting at least one student dismissal in the last five years (81%), the average number of terminations was 5.7 students. In sum, trainee impairment appears to be a common issue across psychology and counseling programs. Although the percentage of impaired trainees may appear relatively minor (4.6-4.8%) compared to the number of trainees in counseling programs, Biaggio et al. (1983) note that it is a steady trickle that requires attention (p.19). In particular, the amount of time counseling programs invest in impa ired trainees is considerably higher than with a non-impaired student (Vacha-Hasse, 1995). V acha-Hasse found that training dire ctors may devote as much as 20 hours a month to impaired students, while fa culty members may spend as much as 40 hours a month addressing student impairment issues. In reality, impaired trainees consume a disproportionate amount of faculty time and prog ram resources. Although one of the goals of counselor training is to develop co mpetent professionals, a supervisors first priority is to ensure that trainees do not harm clie nts and others (Frame & Stevens-Smith, 1995). Therefore, it is necessary for counseling programs to identify the types of non-academic trainee impairment issues that may interfere with th e students development. By fa miliarizing themselves with these impairment traits, counseling programs can not only ensure they identify impaired trainees, but also fulfill their obligation to protect the client population at large. Identifying Trainee Problems Although identifying the rate of problem atic students highlights the need for proper trainee evaluation, outlining what specific behaviors constitute trainee impairments can better prepare faculty members for evaluation and remediation procedures. By delineating various student issues, educators can simultaneously strengthen their professional gate keeping process while

PAGE 32

32 aiding in trainee development. Specifically, identifying the various trainee problems may improve faculty and student communication. Faculty may be more likely to effectively intervene with problematic students if they and trainees acknowledge that problems are likely to occur (Vacha-Haase et al., 2004). As previously mentioned, the ability to iden tify and remediate a problematic trainee is a fundamental component for any counseling prof ession. Researchers from various counseling professions (Boxley et al ., 1986; Huprich & Rudd, 2004; Li 2001; Olkin & Gaughen, 1991; Procidano et al., 1995; Vacha-Haase, 1995) have investigated how trai ning faculty and/or program directors classify problematic students. Collectively, these studies have identified several areas of trainee issues beyond the tradi tional academic coursework. Table 2-1 illustrates the most common non-academic trainee issues among counseling related programs. Despite this extensive research, drawing c onclusions from these studies may be difficult because of differences in sampling, questioning, and categorizat ion of trainee issues (Forrest et al., 1999). As first described by Forrest et al., studies solicited respondents from a variety of positions including chairpersons or training directors from psychology departments (Huprich & Rudd; Procidano et al.; Vacha-Haase), psychology inte rnship program directors (Boxley et al.), CACREP academic units (Li), and chairpersons from clinically or iented masters programs in mental health (Olkin & Gaughen). In addition, the manner in which the researchers phrased the questions varies considerably in these types of st udies (Forrest et al.). For example, Olkin and Gaughen provided a finite list of trainee problem s and asked participants to rank the top four, whereas Huprich and Rudd asked an open ende d question and reported only the five most frequent impairments. Finally, despite some overl ap between the studies, several student issues (adjustment disorder, anxiety problems, intraper sonal problems, lied, mari tal problems, maturity

PAGE 33

33 issues, misrepresented skills, and social proble ms) were reported in only one study. This may make comparison between studies particularly difficult because some specific problems from one study may be generic issues that are a part of a larger category in other studies (Forrest et al.). For example, Olkin and Gaughen used the category intrapersonal to comprise substance abuse issues, emotional problems, personality problems, rigidity, and immaturity, whereas Vacha-Hasse reported substance abuse, emotional problems, and personality issues as separate categories. The difference in methodologies makes direct comparison between the studies complicated. Despite these variations, a review of this lite rature suggests that so me trainee issues are more common than others. Based upon these st udies, it would appear that clinical skill deficits, interpersonal problems, and supervision difficulties are consistently identified as the most prevalent, non-academic, trainee problems. Clinical skill deficits were cited in three of the six studies (Olkin & Gaughen, 1991; Procidan o et al., 1995; Vacha-Hasse, 1995) and were identified as the most common trainee problem in all three studies (77%, 46%, and 65% respectively). Similarly, interper sonal problems were identified in three studies (Li, 2001; Olkin & Gaughen; Vacha-Hasse), and was ranked as the most prevalent trainee problem in one study (Li; tied with supervision difficulties), the second most prevalent issue in another study (Olkin & Gaughen,), and the third most prevalent impairme nt in a third study (Vacha-Hasse,). Finally, supervision difficulties ranked as the most identi fied trainee problem in one study (Li; tied with interpersonal problems) and the se cond most identified trainee issue in two other studies (Olkin & Gaughen; Vacha-Hasse). Supervision difficultie s were included in thre e of the six studies.

PAGE 34

34Table 2-1. Frequencies of impaired traine e characteristics as reported by previous studies (select non-academic variables) Boxley et al.Olkin & GaughenVacha-HasseProcidano et al.LiHuprich & Impairment(1986)a(1991) b (1995) c (1995) d (2001) e (200 4 Adjustment Disorder14Anxiety Problems25Clinical Skills Deficit 77 65 46Depression31 29 23Emotional Problems31 40 34*Ethical25* 8Fatigue or Burnout19 27Inappropriate Boundaries58Intrapersonal Problems54Interpersonal Problems70 42 73Lied42Marital Problems27Maturity23Misrepresented Skills36Personality Disorders35 15 34* 31 19Physical Illness27 10 40Refused Counseling36Social Problems19Substance Disorders10 15 4Supervision Difficulties58 52 73Unprofessional Behavio r 25* 27Note. Procidano et al. list emotional problems and personality disorders as one category. Note. Olkin and Gaughen listed ethical issues and rigidity, and unprofessional behavior as category and used intrapersonal problems to include substance abuse issues, emotional problems, personality disorders, immaturity. Note. Both Vacha-Hasse and Li are partial lists of trai n impairments. aRespondents were asked to provide information on the cases they felt were representative of impaired trainees. bRespondents ran ked their top four choices from a list of seven possibilities. cRespondents were asked to identify the frequency of current student difficulties. dRespondents were asked to affirm any nonaca demic deficiencies within the last five years. eRespondents were asked to indicate how frequent an impairment indicator was used as a basis for remediation. fIndicates the fiv e most commonly reported impairments for internship.*modified/expanded from Forrest, El man, Gizara, & Vacha-Hasse (1999)

PAGE 35

35 The absence of these issues from some of the studies is not surprising, as the way in which the authors define trainee defici encies shaped how the participants responded to the questioning. Two studies (Boxley et al., 1986; Huprich & Rudd, 2004) did not incl ude any of the three trainee problems; however, as previously mentioned, Huprich and Rudd focused only on the incidence of psychological impairments among trainees. Simila rly, Boxley et al defined trainee impairment as, any physical, emotional, or educ ational deficiency that interferes with the quality of the interns professional performance, education, or family life (p.51). Using this specific definition, clinical skill deficits, interper sonal problems, or supervision difficulties may not have qualified as an academic issue. A dditionally, Procidano et al. (1995) only examined general categories of trai nee issues. The authors simply diffe rentiated between clinical skills, emotional / personal problems, and ethical issues, ther eby condensing the list of trainee issues. In sum, when clinical skills deficits, inte rpersonal problems and supervision difficulties were included in a study, no other trainee issues were more prevalent. These three issues are directly associated with clinical supervision. Clin ical supervisors directly facilitate the trainees skill development as well as observe how the trainee interacts with their clients. This finding only reinforces the importance of proper trai nee evaluation during clin ical supervision and suggests the need for compre hensive evaluation tools. Due Process Procedures In 1975, Bernard suggested that counseling program s incorporate some general due process guidelines in order to protect themselves from potential lawsuits and ensure students are evaluated fairly. He recognized that due proces s procedures for addressi ng the trainees personal inadequacies were unclear and outlined four prin ciples for proper due process. Since Bernards article, several authors (Knoff & Prout, 1985; Lamb et al., 1987; Miller, 1979; Miller & Rickard, 1983) have written and expanded upon the more prominent aspects of due process. The

PAGE 36

36 following paragraph includes Bernards original outline as well as additional contributions provided by other due process literature. First, programs should provide students w ith written guidelines that outline what professional and personal behaviors are expected (Bernard, 1975; Lamb et al., 1987). These guidelines should include the specific proced ures for student evaluation and a statement stipulating that students may be terminated fo r personal shortcomings (Bernard; Knoff and Prout, 1985). Second, faculty members should continually evaluate all students a nd provide them with written evaluations (Bernard; Lamb et al.; Knoff & Proutt; Miller, 1979; Miller & Rickard, 1983). Although Bernard suggests th at faculty evaluate students at least once a year, other authors suggest more regular evaluations to en sure that programs identify and address student impairments within a reasonable timeframe. Thir d, if a problematic student is identified, the program should develop a written remediation pl an that includes: (a) defining the problem (Knoff & Proutt; Lamb et al.), (b) identifying expected behavior (Bernard; Knoff & Proutt; Miller), (c) specifying remediation options (Knoff & Proutt), (d) id entifying a reasonable timetable for completion (Bernard; Knoff & Proutt; Lamb et al), and (e) indicating clear consequences for failing to complete the remediat ion (Bernard). Finally, appeal procedures for students to challenge remediation or dismissal d ecisions should be in place (Bernard; Knoff & Proutt). As Knoff and Proutt note, although legal precedent does not require programs to have a formal appeal process, the hearing does represen t an important due process gesture, which again demonstrates a facultys sincere effort to make the termination process fair and open (p. 796). In addition to general due process parameters several authors (Bal do & Softas-Nall, 1997; Bemak, Epp, & Keys, 1999; Frame & Smith-Stevens, 1995; Lamb, Cochran, & Jackson, 1991; Lumadue & Duffey, 1999) have described student monitoring models to assist counseling

PAGE 37

37 programs in identifying and remediating problema tic trainees. As Vach a-Haase et al. (2005) state, an increased specificity in evaluation gu idelines would alleviate the burden an individual program may face when deciding what to do with a problematic trainee. Although these models vary in approach, each one seeks to improve professional gatekeeping procedures by implementing extensive due process policies and protecting individual facu lty members from the sole responsibility of student remediation. In 1991, Lamb et al. expanded their previous work on due process by describing a specific monitoring model. Although orig inally conceptualiz ed for counseling psychology programs, the authors contend that this model can be adapted to many academic programs and practicum sites. This model includes processes related to: (a) reco nnaissance and identifica tion of trainee issues, (b) discussion and consultation, (c) implementati on and review procedures, and (d) anticipating and responding to individual and organizational responses. Within each section, Lamb et al. describe various strategies and discuss several questions for faculty members to consider during the remediation process. Noteworthy topics included differentiati ng between problematic behaviors and impairments, the importance of ear ly identification and on-going review, and how a dismissed student can affect the program climate. Lamb et al. (1991) describe problematic behavi ors as, an interns behaviors, attitudes, or characteristics that may require remediation, but are perceived as not unexpected or excessive for professional training (p.292). In other words, problematic behavior s are situational deficiencies that trainees commonly exhibit throughout supe rvision (e.g. counseling pe rformance anxiety). Immediately after identifying a problematic be havior, the authors recommend the supervisor discuss the concerns with ot her training personnel, promptly develop and implement an intervention strategy, and allow the trainee an opportunity to addre ss the behavior. Therefore, if

PAGE 38

38 the problematic behavior does become an impa irment the documentation verifies the staffs proactive response and ensures the trainee was made aware of the problematic behaviors before more drastic measures were taken. On the other hand, when training personnel are unable to remediate the students problematic behavior, these defici encies may develop into an impa irment. According to Lamb et al. (1991), trainee impairments develop when most of the following occur (p.292): (a) the intern does not acknowledge, understand, or address the problematic behavior when it is identified, (b) the problematic behavior is not merely a reflection of a skill deficit that can be rectified by academic or didactic traini ng, (c) the quality of service delivered by the intern is consistently negatively affected, (d) the problematic behavior is not restricted to one area of professional functioni ng, (e) the problematic behavi or has potential for ethical or legal ramifications if not addressed, (f) a disproportionate amount of attention by training personal is required, (g) the interns behavior does not change as a function of feedback, remediation efforts, of time, and (h) the interns behavior negatively affects the public image of the agency. In situations where initial remediation attempts are unsuccessful, the authors recommend a systematic procedure whereby relevant personnel m eet to discuss the specifics of the students behavior and determine whether the behavior co nstitutes impairment. Lamb et al. recommend a more formal approach take place if training pe rsonnel deem the student impaired. Specifically, the student is notified, in writing, of the parameters surrounding the impairment as well as provided several ways in which to remediate the behaviors. The trainee then meets with all relevant training personnel to address the seri ousness of the circumstances. Once the probation period begins, supervisors continually review the trainees progress and in form the trainee about his/her performance. If the program decides to te rminate a student, a letter is sent to the student outlining the probation terms, the students actions, and the reasons for the dismissal. Finally, the trainee is given an opport unity to appeal the decision. Regardless of whether the program successfu lly remediates or dismisses the student, Lamb et al. (1991) recognize that this process affects people beyond the trai nee. Specifically, the

PAGE 39

39 academic institution, other interns, and those specifically involved with identifying the impairment and implementing the remediation plan, can all be negatively affected. Despite these concerns, the authors conclude that if training personnel handle the remediation / dismissal process humanely, this process can benefit both the intern and staff members. For example, an intern may reexamine his/her professional development and avoid future difficulties. Concurrently, training staff and other interns may also reexamine their professional behavior (Lamb et al.). Lamb et al. (1991) were one of the firs t to suggest a general model for student remediation/dismissal procedures. Although this article addresses many essential issues and raised several important questions, Frame and St evens-Smith (1995) were one of the first to evaluate how a specific model a ffects trainee evaluation. These authors described a three-step model that was developed and implemented at the University of Colorado at Denver. Before classes begin, students reviewed the student handbook and signed a statement indicating that they will abide by all policies. Within the ha ndbook, the faculty provide a policy statement that expressed, the facultys belief in the essential function of personal characteristics in the development of ethical and competent counselor s (p. 124), and the Pe rsonal Characteristics Evaluation Form (PCEF), an assessment that in cludes expected trainee characteristics. Each faculty member uses the PCEF to evalua te every student at the mid-term and end of the semester (Frame & Stevens-Smith, 1995). If a professor identifies a problematic student, he or she first discusses the student with the enti re faculty and then discusses the evaluation and remediation (if any) with the st udent. If a student receives more than one evaluation in a semester, or receives a form from the same profes sor in back-to-back semesters, the student must meet with his/her graduate advisor to discuss additional remediation or possible reconsideration

PAGE 40

40 in the program. Additionally, if a student recei ves three or more negative evaluations in one semester, the student will be required to meet with the faculty advisor and two other faculty members for possible program termination. One year after implementing these procedures, faculty members and students evaluated the effectiveness of this model (Frame & Steven s-Smith, 1995). Although only 50% of the faculty surveyed felt that the evaluation process provid ed them with a concrete approach to student evaluation (25% were neutral) counseling educators stated that they had become more intentional about evaluating a students personal characteri stics (Frame & Stevens-Smith, p.126). In relation to the students surveyed, 82% indicated that they were aware of the evaluation process. Unfortunately, Frame and Stevens-Smith present no data indicating the students thoughts about the evaluation process itself. The aut hors conclude that by utilizing due process procedures, programs can protect themse lves from unfair stude nt evaluation and more importantly protect future clients from harm (Frame & Smith-Stevens). In response to the Frame and Smith-Stevens (199 5) article, Baldo et al. (1997) believe that individual faculty members may be placing themse lves into a situation of unnecessary hardship if they directly evaluate and remediate problem atic students. Specifically, faculty members who identify and remediate problematic students may b ecome the target of that students aggression. Therefore, Baldo et al. assert that programs should include the entire facu lty during the evaluation process, thereby prot ecting individual facu lty members from any excessive duress. In essence, this type of mon itoring model allows faculty memb ers, at any time, to discuss student concerns during regularly scheduled faculty meetings. Once a year, program faculty members meet to specifically re view the students progress. If the program faculty members believe the students progress is unsatisfactory, th e students advisor, not the concerned faculty

PAGE 41

41 member, meets with the student to discuss possible remediation, voluntary resignation, or dismissal. If the faculty members do decide that remedia tion is necessary, the student, the students advisor, and a retention committee develop a reme diation plan. Once the plan is approved (or modified) by the program faculty, the student sign s the plan and begins the necessary steps to fulfill the obligations. Near the re-evaluation date, the student will present all necessary documentation to the advisor and re quest that his/her current professors evaluate the progress. This information is then reviewed by the entire faculty and they decide whether the student: (a) continues in the program without remediati on, (b) continues probation / remediation, (c) voluntary resigns, or (d) is dismisse d. If the faculty members decide to dismiss a student, he or she is notified in writing and has 30 days to pr esent a written request for re-evaluation. Upon receipt of the request, the faculty will allow the student to present his/he r case. Following the presentation, the faculty members will re-evaluate the student and notify the student of their final decision. The authors conclude that involving the enti re faculty ensures proper due process for both the faculty member(s) and student as well as helps prevent the possib ility of an individual faculty member experiencing unnecessary stress. Similar to the previous models, Bemak, et al (1999) offer a five-step model that aids counselor training programs with the remediation and/or dismissal of impaired counseling students. The authors note that the most ta xing concern facing counsel ing programs is how to properly evaluate and remediate st udents who present psychological or personality impairments, but otherwise perform basic counsel ing skills competently. Therefor e, Bemak et al. suggest that counseling programs incorporate the idea of ps ychological adjustment throughout the training process. Step one requires that the faculty cl arify and communicate program expectations before

PAGE 42

42 the admissions process. Like previous models, th e authors suggest that in addition to department standards that address traditional criteria like monitoring and dismissal procedures, an advisor should identify sound mental health and personal development as critical aspects of professional suitability (p.25). Once the pol icies are reviewed and agr eed upon, step two requires the students sign a contract indicating their understanding of these procedures. The third step, identifying trainee concer ns, requires that faculty members use departmental criteria to evaluate a students development. If a student exhibits any professional or personal deficits faculty members are to promptly contact the students advisor. Subsequently, several department members, incl uding faculty, supervisors, and the Department Chair, may convene to address the student issu e(s). Once these faculty members discuss the concerns, a remediation plan is developed with the student (step four). Finally, faculty must monitor the student and continually provide feedback regarding his/ her progress. If a student is unable to fulfill the requirements outlined in th e remediation plan he or she may appeal according to the institutions appeal process. Bema k et al. (1999) conclude that this model offers several advantages to studen t monitoring including: (a) an ongoing student assessment, (b) increased faculty communication regarding student impairment, (c) timely feedback to students, (d) proper student due process, and (e) a clear process that outlines a faculty members response to student impairment. Lumadue and Duffey (1999) discuss the difficult ies associated with student evaluation and the importance of properly implemented due process procedures. To improve upon the monitoring process, Lumadue and Duffey descri be an evaluation instru ment, the Professional Performance Fitness Evaluation (PPFE), which was developed by faculty at Southwest Texas State University (SWT). Unlike the PCEF (Fra me & Smith-Stevens, 1995) that uses abstract

PAGE 43

43 student characteristics, the PPFE utilizes spec ific student behaviors to evaluate trainee development. The authors contend that a sp ecific behavioral criterion ensures greater consistency among evaluators (Lumadue & Duffey). If a professor identifies an issu e, the student and the professor meet to discuss the matter. Then, if the issue remains unresolved, the profes sor contacts the department chairperson and the student meets with a retention committee. Th e committee reviews the issues and makes one of three recommendations: (a) the st udent is fit to continue, (b) the student should undergo remediation, or (c) the student should be dism issed. If the student chooses to appeal the committees decision, he or she must do so by writing a letter to the Department Chair. If the Chair upholds the committees decision, then the student can re-appeal to the Dean of the college. However, if the Dean also upholds th e decision, then the stud ent can initiate legal action. Although proper due process procedures protec t the counseling program and student from capricious evaluation, a considerable number of programs continue to operate without these policies and procedures in place. Boxley et al. (1986) found that 66% of psychology internship programs reported no adequate or judicious means of assuring the right of due process to interns who are dropped or deferred from continuing in the program (p.51). In contrast, Procidano et al. (1995) found that 74% of docto ral programs in professional psychology reported some policy for addressing a students professional deficien cy; however, only two-thir ds of these policies were in writing. In 1991, Olkin and Gaughen fo und that 45% of counseling oriented programs had no written policies for evaluating problematic students. Of the programs that did have a policy, over 40% did not generally gi ve these policies to their st udents. Similarly, Vacha-Hasse et al. (2004) found that 53% of APA accredited psychology programs reported that they did not

PAGE 44

44 have written policies for problem atic students. Furthermore, these authors note that of the programs that did have a policy on ly 46% routinely supplied this information to their students. Finally, although Huprich and Rudd (2004) li mited their questioning to psychological impairment, the authors found that only 58% of doctoral programs reported that they had a formal procedure to address thes e types of student issues. On the other hand, 84% of internship sites reported they had formal procedures in place to address a students psychological impairment. Despite some slight variability between these studies, a substantial number of programs are reported as not having written polic ies and procedures in place to address problematic students. As Vacha-Hasse et al. (2004) notes, the number of programs without these policies are, somewhat surprising, given that programs without such guidelines may be vulnerable on several fronts, including compliance with accreditation requirements and explanation of due process (p.118). Interestingly, accreditation bodies continue to allow programs to operate without proper due process procedures despite recognizing the need for programs to graduate personally and professionally competent trainees. Clinical Supervision Evaluation Criteria Without clearly identifying what skills are essential for clinical supervision, the process of deciding what skills are appropriate for ev aluation is an imperfec t process (Bernard & Goodyear, 2004). Fortunately, educators can exam ine the standards for their profession, the general counseling literature, and research to id entify what specific skills they want to include for clinical evaluation. Although accreditation and professional standards literature is limited, it provides an excellent starting point for understand ing the general types of items that should be included during evaluation. As previously mentioned, the rehabilitati on counseling code of professional ethics suggests that faculty endorse only those students w ho demonstrate adequate

PAGE 45

45 professional ability as well as are personally suitable to provide service to people with disabilities. In addition, the code outlines the professional standa rds which identify and define a rehabilitation counselor (e.g. en suring client autonomy, A.1.d ; respecting cultural diversity, A.2.a; promoting client empowermen t, C.1.c). Given that the pr ofessional standards refer to three types of criteria (professional skill, persona l suitability, and profe ssional standards), it is perhaps fitting that definitions of impairment as well as clinical supervision instruments also typically include these general criteria. Although each respec tive counseling profession may address general categories of trainee impairme nt, individual program s are responsible for developing their own evaluation cr iteria as well as reviewing th e professional literature to identify specific criteria that would aid in this process. Perhaps the most prolific source of clinical evaluation criteria comes from reviewing the general counseling literature. The counseling literature is in undated with various books and theories (e.g. Chan, Berven, & Thomas, 2004; Corey; 2004; Cromier & Cromier, 1991; Hill, 2004; Miller & Rollnick, 2004) that identify basic and advanced counseling skills that help facilitate the counseling session. Although there is an exhaustive amount of information to draw from, books and theories can vary with regards to content and organization. For example, Cormier and Cormier provide an excellent overv iew of various skills ranging from basic listening responses to the use of advanced clinical techniques. On the other hand, Walborn (1996) opines that the specific counseling skills are less important than the common elements of counseling. Specifically, he suggest s that four process variables ar e necessary for change: (a) the therapeutic relationship, (b) cognitiv e insight, (c) affective experience, and (d) appropriate client expectations. Therefore, alt hough there are a variety of resources that identify a mixture of counseling skills, consensus regarding what sk ills are necessary for counseling varies.

PAGE 46

46 In addition to counseling skills, the trainees personal attributes are oftentimes considered necessary for clinical evaluation. Although not a ll homegrown or published clinical instruments may explicitly address the trainees personal suitability, this issue has received increased attention within counseling training programs. More recently, supervis ion instruments have included sections to address the pe rsonal suitability of the student. For example, Bernard (1997) includes a personalization skills on an evaluation instrument This section assesses the students self-awareness, interp ersonal and intrapersonal depth and flexibility, as well as how well the trainee works with others. Likewise, th e evaluation instrument de scribed by Kerl et al. (2002) includes a six question sect ion dedicated to the maturity of the student. Despite the importance given to the trainees personal suitability, programs may struggle identifying students with character or psychological issues (Bemak et al., 1999). This may be due to the fact that personal impairment is usually difficult to iden tify within the counsel ing session because it may manifest as a skill deficiency or other professional impairment. For example, if the trainee is an abnormally anxious person, he or she may ask only superficial questions. In this instance, the supervisor may focus on developing the students use of open-ended questions or other probing techniques, rather than addressing the trainees anxiet y issue. Therefore, for the purpose of clinical evaluation, it is important to identify specific behaviors which are indicative of suitable personal functioning. A third area less commonly found in clinical supervision criteria are the students adherence to professional stan dards. As mentioned earlier each respective counseling profession adheres to specific st andards to help define and guide the profession. Within rehabilitation counseling, the professional standa rds address a variety of issues including the

PAGE 47

47 principles of ethical behavior, confidentiality issues, as well as underlying professional themes such as client empowerment (CRCC, 2002). In sum, establishing evaluation criteria may be challenging and perhaps the most laborintensive part of the evaluation process (B ernard & Goodyear, 2004). Typically, clinical evaluation criteria include counsel ing skills (basic and advanced sk ills), personal suitability, and adherence to professional standa rds. Although these three areas appear to encompass different aspects of counselor training, each component can potentially relate with one another. Supervision Models In 1995, Maki and Delworth sought to improve a students therapeutic competence and enhance client monitoring by describing a clin ical supervision model for the rehabilitation counseling profession. Specificall y, the authors developed the St ructured Developmental Model (SDM) by modifying Stoltenberg and Delworths Integrated Developmental Model (IDM, 1987). Basically, the IDM catalogs the counselors progression through four levels of professional development (Level 1, Level 2, Level 3, and Level 3 Integrated) while simultaneously providing supervisors with strategies to facilitate the counselor s development. In order to assess the students level of development, the supervisor examines the counselors motivation, autonomy, and awareness relative to eight competency dom ains. For example, Stoltenberg and Delworth (1987) describe a level 1 counsel or as highly motivated, highly dependent, and focused strictly on themselves rather than the client. These char acteristics are indicative of a new counselor who is relying on the supervisor to find the right way to counsel a client. As the counselor develops, he or she becomes more independent and aware of both se lf and the client. Additionally, after va cillating between high and low levels of motivation (level 2), the counselor begins to show more consistent motivation a nd develops his/her own professional identify.

PAGE 48

48 According to Stoltenberg and Delworth (1987), vertical development, or movement ahead to the next level, is explicitly stressed, but th e model in practice also attends to issues of horizontal development across domains (p.36). In other words, the counselor not only progresses to higher stages of development, but does so across the domains. Although the authors believe that counselor development is generally irreversible they recognize that counselors may periodically regres s to earlier stages of developm ent when faced with unfamiliar circumstances; however these regr essions do not indicate that the counselor has reverted back to a previous stage (Stoltenberg & Delworth). Using the IDM as a basis, Maki and Delwor th (1995) reorganized the eight competence domains to develop the SDM. Unlike the IDM which conceptualized counselor development through eight simultaneous domains, the SDM cate gorized the domains into one of two groups: (a) primary domains, and (b) pr ocess domains. Primary domains, also known as meta-domains, include: (a) sensitivity to individual differences, (b) theoretical orientation, and (c) professional ethics and are continually examined during th e counselors development. The remaining domains, known as process domains include: (a) interper sonal assessment, (b) client assessment within the environment, (c) case conceptualiz ation, (d) treatment goals and plans, and (e) intervention strategies. These five process domains mirror the convent ional service delivery sequence and are addressed within the context of the primary domains (Maki & Delworth). The authors contend that reorganizing the compet ence domains to better emulate the counseling process will aid in clinical supervision. Despite the obvious advantages of a clini cal supervision model, the debate over the usefulness of a developmental model, like th e IDM, to explain counselor development has endured (Ellis, 1991; Holloway, 1987; Holloway, 1988; Stoltenberg & Delworth, 1988;

PAGE 49

49 Stoltenberg, McNeil, & Crethar, 1995; Wort hington, 1987). Many researchers (e.g. Chagnon & Russell, 1995; Murray, Portman, & Maki, 2003; Reising & Daniels, 1983; Wiley & Ray, 1986) began investigating the developmental supervisor y process and supported th e notion that trainees do exhibit identifiable characteri stics across their development; how ever these differences were only evident when differentiating novice trainees from more advanced trainees. In conclusion, Bernard and Goodyear (2004) caution that alt hough developmental mode ls are intuitively appealing, they may mislead their supporters away fr om alternative explanatio ns of supervision. An alternative to the SDM (Maki & Delw orth, 1995) is the In tegrative Model of Supervision in Rehabilitation (S chultz et al., 1999). Recognizi ng that rehabilitation counselors work with a variety of populations in different environments, Sc hultz et al. suggest that any supervision model within th e rehabilitation profession s hould be flexible enough to accommodate the differences. Therefore, the Integrative Model was based on rehabilitation principles rather than a specifi c personality theory or psychologi cal process. The authors state that mastering the art of counseling before comp leting the clinical portion of the program is improbable; therefore, supervisees should be trai ned in multiple rehabilitation concepts, but not necessarily be expected to ma ster every professional skill be fore graduation. As a result, supervisors become responsible for not only teac hing counseling skills and fostering attitudes that allow the counselor to function independent ly, but also ensuring tr ainees understand that counselor development is a lifel ong process (Schultz et al.). In order for supervisors to fulfill their obligation, the Integrative Model of Supervision in Rehabilitation requires the educator be able to shift roles to or from teacher, counselor, and consultant (Schultz et al., 1999). Specifically, the authors reference Be rnards Discrimination Model (1997) where the supervisor adopts a different role depending on whether he or she

PAGE 50

50 wishes to focus on the trainees interventi on skills (e.g. empathy, confrontation, etc.), conceptualization skills (e.g. identify them es, discriminate information, etc.), or personalization skills (e.g. pers onality, cultural influence, etc.). In addition, the supervisor assesses the trainee progression th rough 3 phases of development. Phase I (Technical), focuses the trainee on developing basic interpersonal skills such as empathy as well as appropriate attitudes, self-trust, and intern al evaluation processes. Once th e trainee transitions to Phase II (Integrative), he or she will focus on more advanced counseling skills such as case conceptualization, treatm ent manual utilization, research activities, and explore different service delivery modes. Finally, as the student moves in to Phase III (Consultation) he or she seeks to continue to develop the ability to self eval uate and acquire new skills. Schultz et al. acknowledge that transitioning from Phase I to Phase II is crucial and should not necessarily coincide with the student progressing from pr acticum to internship. Although the authors note that prematurely transitioning a student to an advance stage may result in the counselor being overwhelmed, while delaying th is transition for an able student will result in apathy or frustration, the pract icum and internship timelines are reasonable (p.329). In sum, both the SDM and Integrative Model offer an excellent resource for rehabilitation counseling educators and students. Specifically, these models help organize the supervision process and provide supervisors a means to unders tanding their ro le(s) in facili tating trainee development. Despite these advantages, both models are limited in terms of trainee evaluation. Specifically, these supervision models only describe a theoretical pattern of trainee development, rather than specifying evaluation guidelines. Although the development of trainees cannot be forced, supervisors are still faced with the realit y of whether or not the trainee has earned a

PAGE 51

51 passing grade or has demonstrated some minimu m level of competency worthy of professional endorsement. Additionally, each supervision model provides li ttle empirical evidence pertaining to the actual sequence of counseling skill development. Without a thorough understanding of skill and characteristic development, these models are limited by offering only a description of trainee development. Although a development sequence of counseling skill and attitudes may appear intuitive, empirical evidence outlining a cleare r understanding of skill development will ensure the appropriate skills are be ing required of novice and advanced students. Moreover, trainees do not necessarily devel op basic skills before advanced counseling skills. Focusing novice counselors on basic co unseling skills may be intuitively appealing, however students will come into the supervision process with varying levels of ability in all aspects of counseling, particularly counseling sk ills, personal suitability, and professionalism. Therefore, supervisors would be be tter suited to identify several factors (i.e. interpersonal skills, integrating theory, case conceptualization skills) and simultaneously evaluate the trainee. As a result, supervisors will be able to better evalua te the trainee and meet the trainees need for supervision. Clinical Supervision Instruments Bernard and Goodyear (2004, p.28) state that, the re are as m any evaluation instruments as there are training programs in the helping professions. Beca use the professional literature provides only general evaluation guidelines, coun seling programs oftentimes rely on homegrown instruments to evaluate trainee performance. As Eriksen & McAuliffe (2004) point out, many evaluation instruments have not been subjected to rigorous validity or reliability checks, and may require the evaluator to provide only general impressions of stude nt performance. Moreover, the authors identify several other sh ortcomings common to most eval uation instruments. First,

PAGE 52

52 although expert opinion appears to be the most accurate indicator of trainee competence, some instruments rely solely on client feedback to de termine the students ability level. Additionally, the authors note that many evaluation instruments require the supervisor to simply record how often the student demonstrates a skill, rather than how effective he or she is in utilizing the skill during a counseling session. Finally, Eriksen and McAuliffe note that many supervision instruments simply use dichotomous scales wh en evaluating a train ee, thus hindering the precision of specific and meaningful feedback. In addition to homegrown instruments, ther e are also a number of published evaluation instruments. Unfortunately, they are highly vari able regarding structure, types of items, and psychometric information. Several authors (E riksen & McAuliffe, 2003; Kerl et al., 2002; Myrick & Kelly, 1971; Oetting & Michaels, 1982; Robiner et al., 1994) have described various published instruments that help counseling faculty members evaluate a students counseling performance. The following is a brief overview of several supervision instruments developed within the last 35 years. One of the earliest published supervision instruments, the Counselor Evaluation Rating Scale (CERS), wa s developed by Myrick and Kelly (1971) to address the need for an instrument which could help supervisors ev aluate a trainees clinical performance. After reviewing the literature, the authors selected 27 items to represent three areas of student development: (a) the understand ing of the counseling rationale (b) counseling practice with clients, and (c) exploration of self and the counseling relationship. These items were then organized into a counseling performance domain (13 items), a supervision domain (13 items), and one global item regarding the supervisors overall recommendation. Each item is rated on a 7-point Likert type scale. Once the scores from the counseling and supervision domains are totaled with the global recomme ndation, the composite score represents the trainees overall

PAGE 53

53 counseling effectiveness. Myrick and Kelly report a split-half correlation coefficient of .95. Additionally, when correlating th e counseling and supervision do main together, the authors reported a coefficient of .86. In 1983, Oetting and Michaels developed a unique supervision tool, th e Oetting/Michaels Anchored Rating for Therapists (OMART), to help supervisors develop comprehensive evaluations of trainee performan ce, and to serve as a communication device for supervisors and students. The OMART consists of 34 counseling related items (i.e. relationship with client, exploring issues, exploring feelings). The authors then described various types of trainee behaviors that represent different ability levels for each item. In other words, each counseling item had a corresponding hierarchy of behavior s that reflect various levels of trainee development from needing remediation to profes sional therapist. Unfortunately, Oetting and Michaels provide no psychometric validity or reli ability information. Additionally, because of the OMARTs size, Bernard and Goodyear (1992) sugge st that some may view the instrument as too lengthy. Unlike the OMART (Oetting & Michaels, 1983), the Minnesota Supervisory Inventory (MSI, Robiner et al., 1994) has undergone a more ex tensive validation and reliability testing than most instruments. The MSI includes 112 items a nd a 3-point rating scale. The items represent seven trainee performance areas: (a) assessm ent, (b) psychothera py and intervention, (c) consultation, (d) professional and ethical behavior, (e) supervision, (f) case conference/professional presentations, and (g) site-specific functioning. When developing the MSI, Robiner et al. reviewed the feedback fr om other University of Minnesota Psychology Internship Consortium (UMPIC) supervisors, exam ined the job analysis of psychologists, and appraised instruments from other internships. De spite its comprehensiveness, Robiner et al. note

PAGE 54

54 that the MSI focuses on the trainees skills and professionalism rather than personal characteristics. The authors report adequate internal consistency with in each of the seven domains (rs=.61 to rs=.83). With regards to inter-rater re liability, Robiner et al. note that the findings were lower than expected (rs=-.31 to rs=.28), however test-retest reliability after one week ranged between .91 and 1.00. In addition to psychometric scrutiny, Robiner et al. surveyed internship supervisors to compare their curre nt evaluation instrument with the MSI. Interestingly, the authors note that general reactio ns to the MSI were favorable, but interest in using the instrument varied. Un fortunately, the authors provide no further explanation as to this finding. A fourth instrument, the Professional Counse ling Performance Evalua tion (PCPE, Kerl. et al., 2002), was developed by faculty at Southwes t Texas University to aid in due process procedures. Originally referred to as the Pr ofessional Performance F itness Evaluation (PPFE, Lumadue & Duffey, 1999), the PCPE helps faculty members evaluate students and provides them with feedback on several areas of prof essional and personal development. Although the PCPE was not designed specifica lly for supervision, Kerl et al note that the instrument is appropriate for every course. Moreover, the in strument includes items related to counseling performance. In all, the instrument contains 38 items arranged in five competency areas: (a) counseling skills, (b) professional responsibility, (c ) competence, (d) maturity, and (e) integrity. The evaluator identifies whether the student cons istently meets, minimally meets, or does not meet the listed criteria. Although the PCPE c ontains areas of professional and personal development, the instrument has not been psychometrically validated. Finally, in an effort to develop a validate d instrument for the c ounselor education field, Eriksen and McAuliffe (2003) modified the Skilled Counseling Scale ([SCS] Urbani et al., 2002)

PAGE 55

55 to create the Counseling Skills Scale (CSS). To improve the va lidity and reliability of the instrument, the CSS went through an immense tran sformation that included: (a) addressing item and rating scale issues, (b) establishing conten t and face validity by soliciting feedback from relevant professors, and (c) pilot testing the C SS to obtain inter-rater reliability and construct validity. The final instrument contains 22 items that represent six areas of counseling performance. The rater evaluates the student on e ach item along a 5-point Li kert type scale that ranges from well developed to major adjustme nt needed. To score the CSS, the evaluator averages only those items that were used by the tr ainee during the session. Once each subscale is averaged, the students counse ling competence is determined by totaling the six subscales together. Using only 29 participants, the author s report a Cronbachs al pha of .90 and significant preand postcourse change (constr uct validity) in the total score as well as on five of the six subscales. Regarding item analysis, Eriksen an d McAuliffe concluded that 11 items actually correlated higher with another subs cale than with its own, leading the authors to conclude that the scales do not represent true factors. In conclusion, the development of efficient and precise clinical supervision evaluation tools remains a challenge for various psychology and counseling professions. Although several clinical supervision instrument s have been published, the majority of instruments contain a limited amount of psychometric in formation or rely on summed scores to indicate the trainees clinical performance. The usefulness of summed scores may be minimal because every counseling skills is considered to all contribute to the trainee development equally. However, particular skills or items are undoubtedly more or less difficult for a trainee to develop. Identifying a clear item hierarchy that outlines the trainees skill development will provide a more useful indicator of counsel ing performance. However, iden tifying what skills a trainee is

PAGE 56

56 most likely able to perform in the early stages of clinical training compared to the latter stages is complicated. Intuitively, certain basic counselin g skills (i.e. simple refl ection) are easier to perform than other more advanced skills (i.e. pos itive confrontation), but for the vast majority of counseling skills, educators may disagree over a clea r item hierarchy of skill development. This variability poses a practical challenge for edu cators when they develop their standards for clinical evaluation. In order to address these a nd other evaluation shortcomings a clearer understanding of trainee skill development must be understood empiri cally. Rather than c ontinuing to rely on a predominately theoretical understanding of traine e skill and attitude development, researchers might be better served to study the typical progression of thes e concepts. Once a better understanding of skill and attitude is empirically achieved, counseling educators might be better able to establish and enforce a minimum standard of ability relative to specific timetables. Undoubtedly, trainees will develop at various ra tes, however students who have been endorsed by a counseling program should have acquired a mi nimum level of skill and personal suitability in order to practice in the community. Theref ore, it is imperative that the rehabilitation counseling profession identify specific counseling skills and professional behaviors representative of competent re habilitation counseling trainees. Additionally, the empirical unders tanding of counselor development not only ensures that minimum criteria standards are accurate, but also allows supervisors and students to catalog their development independent of practic um or internship. Therefore, educators can identify realistic expectations that are just be yond the trainees current ability, as well as pinpoint specific interventions. In order to be tter understand tr ainee development and improve trainee evaluation, researchers must rely on innovative research methods to construct instruments. On such method,

PAGE 57

57 Rasch analysis, can address these and other measurement challenges associated traditional data analysis techniques. The Rasch Model Rasch theory is based on a logistic model w ithin Item Response Theory (IRT) that uses probabilities and item difficulty calibrations to measure the persons ability level on some trait. Unlike classical test theorists who assume that the ordinal data is in terval, Rasch analysts believe that in order to satisfy the f undamental rules of measurement, these raw scores must first be converted into an actual inte rval scale (Rasch, 1960; Wright, 1997; Wright & Linacre, 1989). This conversion is necessary because according to Wright and Linacre, a score on a test is nothing more than the frequency count of events, but is oftentimes mistakenly treated as a measure of ability. For example, a person who co rrectly answers more math items on a test than another test taker would be considered to have more math ability. However, simply counting a correct response as one more, erroneously implies that every math item is an equal unit of measurement. According to Wright and Linacre (1989), the events counted are specific rather than general, concrete rather than abstract, and thus varying rather than uniform in their import (p.858). In other words, certain items are indicati ve of more or less of an ability. Therefore, one more can imply a different increment, and thus raw counts are insensitive to this detail (Wright & Linacre). In order to measure the ability of people, Rasch analys ts suggest that measures should mirror the instruments found in the physical sciences (e.g. ruler, sc ale, time). Specifically, raw, ordinal data must first be converted into a lin ear, interval scale to al low the researcher to examine the psychometric properties of specific items rather than the instrument as a whole. In order to achieve equidistant sca ling, Rasch experts rely on an alte rnative paradigm for analysis. Unlike Classical Test Theory (CTT) and other IRT models, Rasch expert s believe that data

PAGE 58

58 obtained should not dictate the measurement mo del, but rather the measurement model should first be established and then use data that conform to the model (Wright, 1997). As Wright explains, a model should not imitate or accept an y kind of data, but should be constructed to define measurement. The Rasch equation in its simplest form is as follows: Log [Pni / 1 Pni] = Bn Di where Pni = the probability of person n passing an item 1 Pni = the probability of person n failing an item Bn = ability of person n Di = difficulty of item i The left side of the equation re presents the transfor mation of raw scores into an interval measure whereby the probability of a person passi ng an item is divided by the probability of a person failing an item. Conversely, the right side of the equation represents the various facets of the measurement model. In its simplest form, the Rasch equation for dichotomous data utilizes two facets: the abil ity of the person (Bn) minus the difficulty of the item (Di). As Bond and Fox (2001) point out, taking th e natural log of the odds of successfully passing an item results in the direct comparison between a persons ability and an items difficulty (pp.202-203). Thus, by arranging people and items on the same linear continuum, the Rasch model has generated parameter separation whereby the creation of a measure estimates a persons ability independent of the specific test items and similarly estimates the difficulty of items independent of the specific sample (Bond & Fox, 2001). In other words, the Rasch model has created a measure that is both sample-free and test-free. Originally conceptualized by Thurstone (1927, 1928), the concept of sample-free measurement refers to the idea that a

PAGE 59

59 measurement scale should transcend any particular sample (Figure 2-1), just as a large ruler can measure various people no matter what their height (Wright; 1997). On the other hand, test-free refers to the ability to omit se veral questions from different le vels of the measurement without affecting the persons score (See Fi gure 2-1; Wright, 1997). This would be similar to measuring the height of an adult who is approximately six foot without having to first determine whether he or she exceeds the one, two, three, four, or five foot mark. In essence, requiring subjects to answer an entire set of questions is unn ecessary when determining their ability. Sample Free Test Free Measurement of a variable should not require every test item. Measurement of a variable should be independent of the sample. Figure 2-1. Sample free and test free related to height When using the Rasch formula, the most in formation about a persons ability level is obtained when the person has a 50 percent probabili ty of passing an item (Smith, 1994). If an item is well below the persons ability, the proba bility of him or her correctly answering it increases (i.e. 90 percent). Conve rsely, if the item is too difficu lt, then the person has a much lower probability of passing the item (i.e. 10 pe rcent). The most information about people is obtained when one can match the types of items associated with their ability level.

PAGE 60

60 By converting ordinal data into an interv al scale, the Rasch model can examine the psychometric properties of the individual items. One such property, an item / person measure, arranges the items on a hierarchy from least to most difficult while simultaneously determining the subjects ability level based on the types of questions he or she correctly answers. Conceptualizing a persons ability level on a hi erarchy, as opposed to relying on the summation of correct answers, can provide clinical supervisors more useful information about the student because the persons ability is attached to spec ific items or behaviors related to whatever constructs one is measuring. A supervisor can then catalog what ski lls the student has the capacity to demonstrate verses the skills that the student has mastered or has yet to demonstrate at all. In other words, the quantitative na ture of measurement would be understood more qualitatively. The student score would finally be associated with specif ic counseling behaviors and allow for a clearer organization of training and development. This information would be useful for measuring change in the students performance as well as improving communication between the supervisor and the student. During clinical work, the superv isor would not only be able to identify their students counseling abiliti es by the types of items they can do, but also target specific behavioral interventions that we re not too easy nor too hard for the student. Furthermore, because the Rasch hierarchy co uld determine a persons counseling ability without requiring him or her to demonstrate every skill or behavior, the Rasch measurement model is a useful tool for clin ical supervision. Depending on th e nature of the practicum or internship site, students may not need to de monstrate every counseling skill or behavior. However, a traditional instrument that relies on to tal scores may unfairly penalize a trainee for not demonstrating a particular skill, even if the skill was unnecessary or did not affect his/her overall counseling ability. Figure 2 illustrates that regardless of what types of skills

PAGE 61

61 demonstrated, the trainees ability is more accura tely determined by considering the difficulty of the skill. It is intere sting to note that when using the tota l score method, persons B is considered to have more counseling ability de spite being unable to adequately perform more difficult skills. ABC NA1NA 422 Person A 422 443 Person B NA41 554 Person C 454 544 554 313228 Clarify Reflection Rasch Measurement TOTAL SCORE Summarize Open Questions Closed Questions Confrontation Total Score Open-Ended Ques Closed-Ended Ques Confrontation Interpretation Identify Themes Clarify Reflection Summarize Interpretation Identify Themes Paraphrase Paraphrase Figure 2-2. Hierarchy of counseling ability compared to total score Finally, a minimum level of performance related to clinical skills, professional standards, and personal suitability could be established to ensure that reha bilitation counseling programs are endorsing only those students who have met these requirements. The development of several hierarchies related to rehabilitation counseling performance will clearly organize what behaviors and abilities are expected from counseling trainees. With th is understanding, clin ical supervisors

PAGE 62

62 and trainees are better equipped to promote the trainees professional and personal development as a rehabilitation counselor.

PAGE 63

63 CHAPTER 3 METHODOLOGY W ithin the rehabilitation counseling professi on, little is known about what competency criteria are appropriate for a cl inical supervision evaluation meas ure. Therefore, the purpose of this study was to establish expert consensus rega rding what specific compet ency indicators were useful for clinical assessment. This researcher employed the Delphi research method to identify these competency criteria. Specifically, particip ants partook in a three round Delphi procedure designed to establish consensus among rehabilitation educators regarding the appr opriateness of specific performance indicators for clinical evaluation. Research Question 1 What professional, personal, a nd clin ical competency items ar e useful for the evaluation of rehabilitation counseling trainees? Study Design To answer this research questi on, participants participated in a three round Delphi study to identify the critical items necessary for clinical supervision evaluation. Originally developed by the RAND Corporation for determ ining the likely ta rgets of Soviet nuclear attacks, the Delphi technique has become a widely used method fo r measuring, predicting, and decision making in an array of disciplines (Rowe & Wright, 1999). The Delphi procedure allows researchers to elicit and converge anonymous expert opinion by providing systematic feedback through a series of questionnaires (Vzquez-Ramos, Leahy, & Hernndez, 2007). The first questionnaire asks an open-ended question(s) regarding the topic of interest. Once participants complete this questionnaire, the researcher combines the pa nels responses and crea tes the second round of questions. In Round 2, participants are asked to rate the importance of all the Round 1 responses on a Likert-type scale. After this information is returned and ta bulated, the researcher develops

PAGE 64

64 the Round 3 questionnaire. During this round, participants are provided their rating responses as well as the statistical group data from Round 2 and allowed the opportunity to compare and revise their previous responses. Ultimately, the goal of the De lphi procedure is to achieve overall consensus or level of agreement among experts (Williams & Webb, 1994). This technique builds on the experts qualitative responses while measuring the groups responses quantitatively (McBride, Pate s, Ramadan, & McGowan, 2003). The Delphi methodology offers several advantages over other comparable techniques. First, because the experts participating in th is study are anonymous, any one member of the panel can not overly influence the responses of other participants. According to Williams and Webb (1994), this method encourages participants to offer their honest opinion because not only are they uninhibited when responding, but junior pa rticipants are free to challenge more senior participants anonymously. In general, this method eliminates any confrontation issues that may be associated with similar techniques (VzquezRamos et al., 2007). Additionally, because the Delphi technique offers feedback in successive rounds, participant views can be retracted, altered, or added with the be nefit of considered thought (Williams & Webb, p. 181). Thus, panel members have the opportunity to rethink and reflect on the feedback from other panel members before submitting their final opinion. According to Rowe and Wright (1999), the multiple iterations of the Delphi method may allow panel members to change their judgments while saving face in the eyes of the other group members. A third advantage of this methodology is the statistical nature of the proce ss. Panel members receive statistical feedback (i.e. mean, median, standard deviation, inte rquartile range) in be tween rounds, but final consensus is determined using preset statisti cal criteria (Hakim & Weinblatt, 1993; Rowe & Wright). The use of statistical feedback and pr eset standards for defining statistical consensus

PAGE 65

65 ensures that researchers are not arbitrarily determining expert consensus based on personal impressions. Finally, the Delphi approach allows researchers to collect data from a diverse panel in terms of geographic lo cation, experience, gender, and educa tion. Particularly when developed for online distribution, this type of survey is easily accessible to expe rts throughout the country. Sampling According to Williams and Webb (1994), dete rmining the appropriate sample size for a Delphi study is unclear; there ar e no established guidelines regard ing appropriate sample size. Delphi studies have typically incl uded as few as three participants and as many as 80 participants (Rowe & Wright, 1999). This tr end is also evident within the rehabilitation counseling profession where studies have ranged from 18 panel members (Thielsen & Leahy, 2001) to 111 panel members (Hakim & Weinblatt, 1993). Add itional rehabilitation counseling Delphi studies have included round 1 sample sizes of 23 (Rubin, McMahon, Chan, & Kamnetz, 1998), 44 (Currier, Chan, Berven, Habeck, & Taylor, 2001), and 31 (Shaw, Leahy, Chan, & Catalano, 2006). Although there is no generally accepted nu mbers of participants Okoli & Pawlowski (2004) suggest that a Delphi pa nel size is typically modest, ra nging between 10-18 experts. In addition to the ambiguity over the appropria te number of partic ipants, Hasson, Keeney, and McKenna (2000) report that the controversy over a Delphi sample size extends to the difficulty in defining who qualifies as an expert In fact, few studies report any specific criteria used to differentiate experts from non-experts (Williams & Webb, 1994). Within the rehabilitation counseling literature, there are no specific definitions differentiating rehabilitation counseling clinical supervision experts from non-e xperts. Therefore, this researcher defined a rehabilitation counseling supervision expert as: Have supervised at least 15 rehabilitati on counseling students during practicum or internship,

PAGE 66

66 Have supervised at least three semest ers of practicum or internship, and Have supervised at least three students during practicum or internship within the last three years. Further complicating the sample size issue is the fact that the Delphi procedure includes several rounds, therefore researchers must accoun t for the potential decline in response rate (Hakim & Weinblatt, 1993). In fact, several of the rehabilitation c ounseling Delphi studies reported a decline in response rate from one round to the next. Therefore in order to account for a reduced response rate and ensure the number of panel experts in the final round remains at an acceptable level, this researcher targeted a mi nimum of 23 participants for the Delphi study. Potential participants were contacted through the Nationa l Council of Rehabilitation Education (NCRE) listserv. The NCRE is the premier professional organization representing rehabilitation educators who are dedicated to improving the lives of pe ople with disabilities through education and research (NCRE, 2007). Af ter NCRE research committee approval, the survey was made available to participants onlin e. Collecting data thr ough a Web-based survey has several advantages including a reduced response time, lowered costs, ease of data entry, and format flexibility (Granello & Wheaton, 2004). The SurveyMonkey software program was used to distribute the questionnaire. SurveyMonkey allows the user to collect and analyze data through a wi de range of export and statistical analysis f unctions (SurveyMonkey, 2007). In addition, this program provides a means for immediately obtaining particip ant data through the internet as well as storing information on a secure server. Delphi Procedure This investigator brainstorm ed with rehabi litation counseling educators to develop a question strategy to identify what behaviors are appropriate/useful for the evaluation of trainees

PAGE 67

67 during the clinical portion of th eir program. This open-ended que stion seeks to elicit trainee behaviors that are indica tive of rehabilitation counseling abi lity. Specifically, the question asks participants to list behaviors that are reflective of three rehabilitation c ounseling trainees areas: (a) personal suitableness, (b) profes sional appropriateness, and (c) c linical ability (Appendix A). Delphi round 1 As previously described, potential su bjects were contacte d via an email (Appendix B) through the NCRE listserv. Prior to beginning the study, rehabilitation counseling educators were notified of the purpose of the study, the genera l study procedure, their rights as participants, potential risks and benefits of pa rticipating, and contact informati on of the primary investigator. Subjects agreeing to participat e in the study confirmed their understanding of the informed consent information electronically (Appendix C). If a participant failed to consent their rights, the survey closed and the participant did not see the quest ionnaire. At the end of the survey, the respondents were instructed to submit their responses. A friendly reminder email was sent to the panel experts two weeks following in itial contact (Appendix D). Once the data was received, this researcher enga ged in the process of microanalysis. This process involves a creative, unstr uctured analysis of the data wh ereby the researcher generates initial concepts and deeper meaning by examin ing the data line-by-line (Strauss & Corbin, 1998). Throughout the analysis process, the rese archer utilized anal ytic tools such as questioning and comparative analysis in order to id entify separate, substantially distinct items. As new data were observed, this researcher constantly compared all of the data to look for similarities and differences among the items (Cor bin & Strauss, 1990). This process requires the researcher to engage in several iterations of data analysis whereby incoming data are compared to existing data.

PAGE 68

68 Although the Delphi procedure is not typically considered a qualitative methodology, certain elements of this process ar e conceptually similar to this type of data analysis process (e.g. constantly comparing data). Therefore, in or der to ensure a level of trustworthiness and credibility in the data analysis process, this re searcher has included an audit trail outlining the round 1 data analysis (Appendix E). Specifically the audit trail captures the researchers decision making process when grouping conceptually similar items. For example, conceptually similar items such as paraphr asing, know how and when to paraphrase, and ability to paraphrase were condensed into the item paraphrase client statements. In this instance, the researcher identified each item as being related to ones ability to pa raphrase and therefore combined the items. On the other hand, items that appeared con ceptually similar but determined to be substantially different were sepa rated into two distinct items. For example, the item confidence in approaching any interpersonal session with clie nts, site supervisor, or faculty supervisor was separated into confident in appr oaching any interpersonal sessi on with clients and confident in approaching any interpersonal session with site or faculty supervisors. In this instance, the researcher deemed the interpersonal intera ctions between the trainee/client and the trainee/supervisor to be substantially distinct ite ms. In other words, the level of confidence the trainee exhibits during a counsel ing session was determined to be independent of the level of confidence the trainee might exhibit with a s upervisor. As a result, this one item was conceptualized as two discrete items and separated accordingly. Once the text was reviewed and some concep ts were identified, this researcher then categorized the data into theore tical competency domains. According to Strauss and Corbin (1998), the analyst eventually realizes that previo usly conceptualized data can be grouped into

PAGE 69

69 higher, more abstract concepts know as categories. The purpo se of categorizing data are twofold. First, it allows the researcher to work with a reduced number of data units which makes the data analysis process more efficient. S econd, by creating categories, the data now has the potential to explain and pred ict (Strauss & Corbin). Since the purpose of this resear ch question is to generate items and competency domains to be included on a rehabilitation counseling supervision measure, sp ecific behaviors related to the trainees clinical ability, personal suitability, and professional adherenc e were identified and used to develop the items for Round 2 of the Delphi Study. Delphi round 2 Following the sam e methodology used in the orig inal questionnaire, an email was sent to all panel experts directing them to the Round 2 survey (Appendix F). Round 2 of the Delphi study consisted of a series of closed-ended questions in which participants were asked to rate the usefulness of each item generated from round 1. W ithin the instructions for round 2, the term useful was further defined as any item relate d to rehabilitation couns eling performance that might help clinical supervisors evaluate a trai nee (Appendix G). Partic ipants rated each item on a 5-point Likert-type scale ranging from 1 (strongly disagree) to 5 (strongly agree). In addition to the item ratings, participants had the opportunity to respond with specific comments regarding the items in Round 2. Like the previous round, a friendly reminder email was sent to the experts one week following initial contact (Appendix H). Once the Round 2 questionnaires were received, th e data was analyzed in order to provide each panel expert feedback about the group norms. Typically, resear chers conducting Delphi studies prepare a variety of information for each panel member including: (a) feedback about their own ratings, (b) a statis tical analysis of the group da ta, and (c) a summation of the comments provided in Round 2 (Vzquez-Ramos et al ., 2007). In order to provide appropriate

PAGE 70

70 feedback, each item was analyzed using the Statis tical Package for the Social Sciences software (SPSS Inc., 2001) and a summary of part icipant comments was developed. Although there are no specific rules regarding what specific sta tistics should be calculated, Rowe and Wright (1999) state that researchers typically report at least one measure of central tendency (i.e. mean, median) and one measure of di spersion (i.e. standard deviation, interquartile range). For the purposes of this study, this researcher calculated and reported the mean and standard deviation for each item. According to Cramer and Howitt (2004), the arithmetic mean is the numeric average of scores and may or ma y not be the most common score. Although the mean may be heavily influenced by outlying data, participants in this study are rating items on a 5 point Likert-type scale and therefore the undue influence of outliers on the calculation of the mean will be minimal. In addition to the mean, this researcher calculated and reported the standard deviation. The standard deviation is a common measure of variability appropriate for interval data (Norman & Streiner 2000). Specifically, the standard deviation is the square root of the average of the squared deviations of each number from the mean of all the numbers (Norman & Streiner, 2000). The cl oser the item ratings cluster around the mean (e.g. the smaller the standard deviation gets), th e smaller the variability and thus the larger the consensus. Delphi round 3 Once the data from round two was calculated, the third round of the questionnaire was developed and sent to the expert panel. This que stionnaire consisted of the participants original item ratings along with group statistical norms (i .e. mean, standard deviation) and qualitative feedback. Following the same methodology for r ounds 1 and 2, an email was sent to all the experts directing them to the round 3 survey (A ppendix I). Round 3 instructions (Appendix J) specifically asked participants to examine the group statistical data, consider the participant comments, and re-evaluate their original item ra tings. More specifically, the participants were

PAGE 71

71 asked to re-rate only those items they decided to change. Similar to previous rounds, a friendly reminder email was sent to the experts two weeks following initial contact (Appendix K). Once all the data was collected and item sta tistics were calculated, this researcher determined what items met the criteria for group consensus. Currently, there are no universal standards for establishing consensus (Fink, Kosekcoff, Chassin, & Brook, 1984; Williams & Webb, 1994). In fact, many researchers may not set a level of consensus prior to the study, resulting in an arbitrary level of agreemen t (Webb & Williams). According to Webb and Williams, unless a [consensus] value is stipulated, the notion of a high level of consensus could almost be a movable feat which is unilaterally decided upon by the researcher (p.183184). Hakim and Weinblatt ( 1993) suggest that in orde r to determine consensus For the purposes of this study, the standards for high expert consensus were defined according to two criteria: The mean difference between the second and third rounds cannot exceed plus or minus 1 (stability), and The standard deviation for each item after round 3 will be equal to or less than 1 (convergence) In addition, for the purposes of this study, th e standards for moderate expert consensus were defined according to two criteria: The mean difference between the second and third rounds cannot exceed plus or minus 1 (stability), and The standard deviation for each item after round 3 will be equal to or less than 1.5 (convergence) Pilot Testing Once the in itial Delphi question was devel oped and placed online, five rehabilitation counseling educators associated with two universities were requested to complete the

PAGE 72

72 questionnaire and provide any additional feedba ck regarding the survey. A summary of the participant feedback, test findings, and ac tions taken are provided in Table 3-1. Table 3-1. Pilot test findings and revisions Comments Action Taken Provide an comprehensive list of each rehabilitation counseling school with its associated CORE region Each school was listed with its corresponding CORE region Delphi Round 1 question did not extract an exhaustive list of trainee clinical, personal, or professional behaviors Delphi was reworded to elicit more comprehensive responses and the introduction was reworded to encourage participants to provide an exhaustive list of competency items The first comment was in reference to the de mographics page of the survey, in which participants are asked to identify what CORE region their program is affiliated. The comment suggested that many participants may erroneously select the wrong region or may not be aware of their schools specific desi gnation. Moreover, the questionnaire could be improved if such a designation were clearer. To address this matter, this researcher reorga nized the region options designated in the survey. Rather than just li sting the region, each school affiliated with CORE was added to the survey under each specified region. Next, in relation to the Delphi question, it was revealed th at respondents did not provide an exhaustive, detailed list of trai nee clinical skills, professional attributes, or personal attributes relative for a clinical supervision assessment t ool. Responses only ranged from 3-8 items and included broader items rather than specific skills or behaviors. To address this issue, this researcher carefully rewrote the question to help invoke participan t responses. Specifically, this researcher added examples representative of the th ree evaluative areas, as well as added a prompt to help supervisors better conceptualize the types of behaviors and skills useful for clinical supervision evaluation. Additionally, this research er reworded the introduction to emphasize the

PAGE 73

73 nature of this study and encour age participants to provide a comprehensive list of trainee competency items. In sum, this study is designed to use qua litative methods to explore the inherent challenges associated with student impairment and to identify specific performance indicators useful for clinical assessment. Once this research study is completed, it is hoped that rehabilitation counselors can more holistically conceptualize trainee competence during clinical supervision. Furthermore, it is anticipated that the clinical su pervisors could use the findings from this research to help develop comprehensiv e measures of clinical competence (e.g. a Rasch validated clinical supervision m easure) that help establish clea r performance expectations and allow for more transparent evaluation.

PAGE 74

74 CHAPTER 4 RESULTS Delphi Study Panel Demographics Of the rehabilitation counse ling educators solici ted for this study, 21 participated in Round 1. Participants averaged 14 semesters of practicum/internship supervision experience and had supervised an average of 59 students throughout their career (Note: when participants estimated the number of supervised students and/or semesters with a +, this researcher used the most conservative estimate for the calculation; one participant did not offer a numeric value when asked the number of students supervised and wa s therefore not included in the calculation). Additional information describing the partic ipants is described in Table 4-1. Participants held a variety of academic posi tions including professor emeritus (4.8%), full professor (14.3%), associate prof essor (23.8%), assistant profe ssor/tenured track (42.9%), and assistant professor/non-tenured tr ack (14.3%). Moreover, particip ants represented several CORE academic regions. Specifically, participants were from regions I (14.3%), III (4.8%), IV (47.6%), V (9.5%), VI (14.3%), VII (4.8%), a nd X (4.8). Regions not represented by the participants included regions II, VIII, and IX. In addition to s upervision experience and location, participants held a variet y of licenses and credentials. As outlined in Table 4-2, the majority of participants were (9.5%) respectf ully. Furthermore, the sample was comprised of a certified disability management specialist, a certified psychiatric rehab ilitation practitioner (pending), a licensed clinical psychologist, li censed independent social work er, an occupational therapist registered/licensed, and a registered mental health counselor intern.

PAGE 75

75 Table 4-1. Participant demographics Frequency Number% Position Professor Emeritus 14.8 Full Professor 314.3 Associate Professor 523.8 Assistant Professor/ Tenured Track 942.9 Assistant Professor/Non-Tenured Track 314.3 Region I 314.3 II 00 III 14.8 IV 1047.6 V2 9 5 VI 314.3 VII 14.8 VIII 00 IX 00 X1 4 8 Factor Delphi Round 1 Experts responded to Round 1 of the Delphi with 188 substantially different item s to consider when evaluating a rehabilitation counseling tr ainee during clinical supervision (Tables 4-3 through 4-7). These responses were base d on an open-ended question which prompted participants to think about thei r students who demonstrated eith er exceptional or poor counseling ability and then list any specific counseling skills, characteristics, abilitie s, or attributes they believed were useful or appropr iate for evaluating trainees during clinical supervision.

PAGE 76

76 Table 4-2. Participant credentials Frequency Credential AbbreviationNumber(%) American Board of Vocational Experts ABVE14.8 Certified Disability Management Specialist CDMS14.8 Certified Life Care Planner CLCP29.5 Certified Psychiatric Rehabilitation PractitionerCPRP14.8 Certified Rehabilitation Counselor CRC2095.0 Certified Vocational Evaluator CVE29.5 Licensed Clinical Psychologist ---14.8 Licensed Independent Social Worker LISW14.8 Licensed Professional Counselor LPC29.5 National Certified Counselor NCC29.5 Occupational Therapist Registered/LicensedOTR/L14.8 Registered Mental Health Counselor Intern RMHCI14.8 Delphi Round 2 For the second round of the Delphi, 18 of the or iginal 21 participants responded with their level of agreem ent to whether each of the 188 items listed in Table 5 were useful for clinical supervision evaluation. The term us eful was defined as an item repr esentative of some aspect of rehabilitation counseling performance and could potentially help the supervisor evaluate the trainee. The mean and standard deviation were calculated for each item and reported in Tables 4-3 through 4-7. These results correspond to the following five point Likert scale used to measure the agreement for each item: 1=strongly di sagree, 2=disagree, 3=neutral, 4=agree, and 5=strongly agree. Following Round 2 data co llect, 179 of the 188 items demonstrated a high level of consensus. This was evident due to th e fact that these items had a standard deviation less than 1. In other words, 95 percent of the items rated by participants demonstrated high consensus with respect to level of agreement. In addition, as indicated by a standard deviation above 1, but lower than 1.5, 7 of the 188 ite ms demonstrated moderate consensus.

PAGE 77

77 Delphi Round 3 For the third round, 17 participants reevaluated their level of ag reem ent to whether each of the 188 items listed in Tables 4-3 through 4-7 we re useful when evaluating trainee clinical performance during clinical s upervision. Like the previous round, means and standard deviations ranges were reported as well as it em stability. Of the 188 total items, 106 (56%) resulted in a greater consensus than previous ly achieved during Round 2. Evidence of this increase in consensus was seen in the greater co nvergence (decrease) of th e standard deviation. Sixty eight of the remaining 69 items demons trated either equal consensus or minimum divergence (.02 or less incr ease in standard deviation) from Rounds 2 to 3. In addition, all 188 items fell within the acceptable st ability criteria. As described in Chapter 3, acceptable stability criteria was defined as the mean difference between Rounds 2 and 3 being equal to or less than 1. Round 3 resulted in 184 of the 188 items reaching high consensus. The remaining four items that demonstrated moderate consensu s include: Healthy ( 1.00), Capable (1.01), Diagnose (with some assistance) (1.06), and Pleas ure to work with (1.20) At the conclusion of Round 3, 145 items (77%) resulted in a mean score between 4 (agree) and 5 (strongly agree) (Tables 4-3 and 4-4). Item include, but are not limited to Active lis tening, (5.00); Builds rapport, (4.94); and Recognizes lim its of competency, (4.53). In addition, 40 of the 188 items (21%) resulted in a mean score between 3 (neutr al) and 4 (agree) (Tables 4-5 and 4-6). These items include, but are not limited to Confident in approaching any inte rpersonal session with site or faculty supervisor, (3.94); Working within a theoretical approach, (3.76); Balance the needs of the faculty supervis or and site supervisor, ( 3.53); and Diagnose (with some assistance), (3.59). Lastly, 3 of the 188 items (1.6 %) resulted in mean score below 3 (neutral). These items include Ebullient/h appy, (2.88); Extroverted, (2.71) and Introverted (2.59) (Table 4-7).

PAGE 78

78Table 4-3. Items with means between 4.50 and 5.00 arranged according to consensus Round 1 MeanConvergenceMeanConvergenceStability Item (s.d.) (s.d.) Active listening (e.g. attending to client; listening and responding) 4.940.245.000.00 0.06 Builds rapport 4.830.384.940.24 0.11 Dependable / reliable 4.940.244.940.24 0.00 Empathetic / Convey empathy to clients 4.890.324.880.33 -0.01 Genuine 4.890.324.880.33 -0.01 Non verbal skills (e.g. eye contact, body position, voice tone, gestures, facial expre ssions, physical distance, and appropriate touch) 4.780.554.820.39 0.04 Clearly explains limits of confidentiality to client 4.830.384.820.39 -0.01 Demonstrates respect for clients values/beliefs 4.830.384.820.39 -0.01 Individualized treatment planni ng (i.e. plan interventions and services that will assist the client in his or her goal) 4.720.464.820.39 0.10 Understands/practices in ethical manner (i.e. confidentiality, representation of compet ence/qualifications, dual relationships, informed consent) 4.830.384.820.39 -0.01 Approachable / Non-threatening demeanor 4.840.384.820.39 -0.02 Develops trust 4.610.614.760.44 0.15 Assess accuracy of interpretations with the clients 4.560.514.760.44 0.20 Identify client problems 4.780.434.760.44 -0.02 Focuses on client rather than themselves in counseling sessions 4.780.434.760.44 -0.02 Sets realistic, attainable goals 4.670.494.760.44 0.09 Non-judgmental 4.780.434.760.44 -0.02 Orients client to th e counseling process 4.560.514.710.47 0.15 Interviewing skills 4.720.464.710.47 -0.01 Round 2 Round 3 (X 1)(X 2)(X 2) (X 1) (X 1)(X 2)

PAGE 79

79Table 4-3. (continued) Round 1 MeanConvergenceMeanConvergenceStability Item (s.d.) (s.d.) Gather pertinent information from the client and relevant others through assessments, observations, and facilitative questions 4.670.494.710.47 0.04 Assess client stre ngths and weaknesses 4.610.504.710.47 0.10 Ability to reflect on ones practice 4.610.504.710.47 0.10 Willingness to continue to learn and develop professionally 4.610.614.710.47 0.10 Open minded 4.720.464.710.47 -0.01 Flexible/adaptable 4.670.494.710.47 0.04 Respectful 4.720.464.710.47 -0.01 Suicide risk assessment 4.670.594.710.47 0.04 Sets clear boundaries with client 4.620.504.650.49 0.03 Observation skills 4.610.504.650.49 0.04 Reflect feeling of client statements 4.560.514.650.49 0.09 Use open ended questions 4.610.504.650.49 0.04 Assess client self-efficacy 4.560.514.650.49 0.09 Prioritize client problems 4.500.714.650.49 0.15 Identify goals that reflect the clients perspective 4.610.504.650.49 0.04 Regularly evaluate s client progress 4.610.504.650.49 0.04 Non-defensive /receptive to clinical supervision 4.670.594.650.49 -0.02 Participates in supervision 4.670.494.650.49 -0.02 Evaluates relevance, value, and meaning of supervisory feedback 4.560.624.650.49 0.09 Understands the challenges to communication with persons with hearing, visual, or cognitive impairments 4.560.514.650.49 0.09 Aware of clients cultural differences/influences 4.560.624.650.49 0.09 Assess client motivation 4.500.514.590.51 0.09 Round 2 Round 3 (X 1)(X 2)(X 2) (X 1) (X 1)(X 2)

PAGE 80

80Table 4-3. (continued) Round 1 MeanConvergenceMeanConvergenceStability Item (s.d.) (s.d.) Continues to move session toward the therapeutic goal 4.610.504.590.51 -0.02 Identify and explore options with client 4.500.514.590.51 0.09 Provide accurate feedback 4.560.514.590.51 0.03 Demonstrates genuine motivation to become a competent and ethical rehabilitation counselor 4.560.514.590.51 0.03 An ability to engage in critical self-reflection rather than focusing solely on the mistakes of others when they participate in clinical training, practicum and internship experiences 4.500.714.590.51 0.09 Demonstrates emotional stability 4.391.044.590.51 0.20 Warm 4.610.504.590.51 -0.02 Crisis Management 4.610.504.590.51 -0.02 Paraphrase client statements 4.440.624.530.51 0.09 Clarify client statements 4.500.514.530.51 0.03 Allows client to elaborate 4.500.514.530.51 0.03 Identifies and meets the clients needs 4.560.514.530.51 -0.03 Summarize 4.500.514.530.51 0.03 Critical thinking ability 4.500.514.530.51 0.03 Identify short and long term goals 4.440.514.530.51 0.09 Prepares for supervision 4.560.624.530.51 -0.03 Incorporates supervisor feedback into practice 4.440.624.530.51 0.09 Identifies own strengths and w eakness related to counseling performance/competency 4.560.514.530.51 -0.03 Sets personal learning goals 4.500.514.530.51 0.03 Ability to handle clients strong emotions (e.g. crying) 4.500.514.530.51 0.03 Round 2 Round 3 (X 1)(X 2)(X 2) (X 1) (X 1)(X 2)

PAGE 81

81Table 4-3. (continued) Round 1 MeanConvergenceMeanConvergenceStability Item (s.d.) (s.d.) An awareness of when their own personal issues are interfering with their ability to learn and practice in a professionally and ethica lly responsible manner 4.560.704.710.59 0.15 Asks effective questions 4.560.624.650.61 0.09 Recognizes how their pers onal limitations, biases, and beliefs impact the counseling relationship 4.560.624.650.61 0.09 Attentive 4.610.614.650.61 0.04 Uses appropriate language/te rminology (e.g. person first language; avoids heterosexist ablest, racist, and sexist language) 4.560.624.590.62 0.03 Manage own mental health disabilities 4.331.144.560.62 0.23 Utilize simple techniques grounded in counseling theory 4.440.704.530.62 0.09 Providing client with choices 4.560.514.530.62 -0.03 Preparation of written reports and progress notes (accuracy, clarity, organization, pr ofessional presentation) 4.330.774.530.62 0.20 Fosters a therapeutic environment 4.560.704.590.71 0.03 Establishes a working alliance 4.500.794.530.72 0.03 Willingness to change when their own biases and beliefs that interfere with their ability to learn how to function as competent and ethical rehabilitation counselor 4.500.714.530.72 0.03 Recognizes limits of competency 4.560.704.530.72 -0.03 Demonstrates uncond itional positive regard 4.500.714.530.72 0.03 Understanding the impact of disability 4.560.784.700.77 0.14 Trustful 4.610.784.650.79 0.04 Round 2 Round 3 (X 1)(X 2)(X 2) (X 1) (X 1)(X 2)

PAGE 82

82Table 4-4. Items with means between 4.00 and 4.49 arranged according to consensus Round 1 MeanConvergenceMeanConvergenceStability Item (s.d.) (s.d.) Elicits motivating statements from client 4.000.694.060.43 0.06 Keeps client focused 4.280.464.300.47 0.02 Flexible with counselin g skills (when necessary) 4.330.494.350.49 0.02 Applies structure to the counseling proce ss (e.g. rapport building, exploration, treatment, termination) 4.390.504.350.49 -0.04 Reflects on the counseling session 4.220.654.350.49 0.13 Appropriately uses silence 4.390.504.350.49 -0.04 Ability to be insightful 4.390.504.350.49 -0.04 Clear understanding of theory(s) 4.280.674.350.49 0.07 Understanding of family and interpersonal dynamics 4.390.504.350.49 -0.04 Enhance or help client to increase his or her self-esteem 3.830.864.000.50 0.17 Controls own feelings and pe rsonal thoughts about clients to increase understanding and to decrease interference in the counselor relationship 4.500.514.470.51 -0.03 Appropriately uses confrontation 4.440.624.470.51 0.03 Understanding of and ability to develop accommodation strategies 4.440.624.470.51 0.03 Redirects client (when appropriate) 4.390.504.410.51 0.02 Conceptualize the clients life (i.e. psychosocial, vocational, etc.) 4.440.514.410.51 -0.03 Willing to research and explor e counseling resources (i.e. journal articles, books, etc.) 4.220.734.410.51 0.19 Deals directly and appropriately with conflict rather than avoiding it. 4.390.504.410.51 0.02 Recognition of the importance of self care 4.170.624.180.53 0.01 Confident (but not arrogant) 4.110.684.180.53 0.07 Round 2 Round 3 (X 2) (X 1) (X 1)(X 2)

PAGE 83

83Table 4-4. (continued) Round 1 MeanConvergenceMeanConvergenceStability Item (s.d.) (s.d.) Use close ended questions (when appropriate) 4.170.704.290.59 0.12 Disagreeing as needed, while maintaining a respectful and supportive attitude 4.280.754.290.59 0.01 Ability to recognize when they are responding to supervisors, clients, students and other professionals in a manner that puts their own needs before those of their clients 4.280.674.290.59 0.01 Utilize advanced techniques grounded in counseling theory 4.220.654.120.60 -0.10 Manages the time of the session appropriately 4.280.754.350.61 0.07 Reframes client statements 4.330.594.350.61 0.02 Uses self-disclosure appropriately 4.220.734.350.61 0.13 General ability to conceptualize 4.390.614.350.61 -0.04 Manage/work on personal issues that arise during counseling or supervision session 4.280.754.350.61 0.07 Use and understand solution skills (giving advice, information, and directive) 4.000.694.000.61 0.00 An ability to use counseling skills intentionally/purposefully 4.500.624.470.62 -0.03 Reflect meaning of client statements 4.390.984.470.62 0.08 Intervenes in a ways that produce client progress in achieving his or her behavioral and/or emotional goals. 4.440.624.470.62 0.03 An awareness of when to seek supervision 4.500.624.470.62 -0.03 Completes supervision tapes/reposts on time 4.440.624.470.62 0.03 Respects other professionals 4.440.624.470.62 0.03 Willingness to seek out professional counseling for themselves when it is needed in order for them to function ethically and professionally 4.440.704.470.62 0.03 Round 2 Round 3 (X 2) (X 1) (X 1)(X 2)

PAGE 84

84Table 4-4. (continued) Round 1 MeanConvergenceMeanConvergenceStability Item(s.d.)(s.d.) Non-defensive 4.500.624.470.62-0.03 Manages resources4.390.704.410.620.02 Patient 4.390.614.410.620.02 Uses confirmations/affirmations 4.390.614.410.620.02 Requests guidance as needed (during supervision) 4.440.624.410.62-0.03 Appropriate dress 4.220.654.240.66 0.02 Interprets client statements 4.280.674.290.69 0.01 Assess client interests, skills, and aptitudes 4.170.864.290.69 0.12 An understanding of and be lief in the importance of supervision 4.390.704.290.69 -0.10 Comfortable with client 4.280.674.290.69 0.01 Willing to experiment or take risks in session 4.000.774.120.70 0.12 Maintains awareness of personal feelings, limitations, and experiences during the counseling session 4.450.784.410.71 -0.04 Ability to be objective (lose their subjective thought) 3.940.874.000.71 0.06 Avoids countertransference 4.330.774.240.75 -0.09 Group counseling skills 4.060.804.060.75 0.00 An ability to articulate why the counseling skills they learn are crucial to all the roles and functions (e.g., vocational rehabilitation counselor, case manager, job developer, vocational evaluator) of rehabilitation counselors 4.170.924.290.77 0.12 Positive 4.170.794.120.78 -0.05 Goal Oriented 4.060.874.120.78 0.06 Fiscal Management 4.060.874.000.79 -0.06 Motivating 4.170.794.180.81 0.01 Kind/thoughtful/caring 4.220.814.240.83 0.02 Round 2 Round 3 (X 2) (X 1) (X 1)(X 2)

PAGE 85

85Table 4-4. (continued) Round 1 MeanConvergenceMeanConvergenceStability Item(s.d.)(s.d.) Conceptualize a case within a theoretical frame of reference 4.060.874.060.830.00 Intuitive/perceptive 4.000.844.060.830.06 Mature 4.280.834.300.850.02 Displays common sense 4.280.894.350.860.07 Conscientious/prudent 4.170.864.120.86 -0.05 Honest 4.440.864.410.87 -0.03 Avoids advice 4.000.914.000.87 0.00 Demonstrates respect for the expertise of supervisors 4.110.904.000.87 -0.11 Hard-working 4.060.874.000.87 -0.06 Altruistic 3.940.944.000.94 0.06 Capable 4.111.024.181.01 0.07 Round 2 Round 3 (X 2) (X 1) (X 1)(X 2)

PAGE 86

86Table 4-5. Items with means between 3.50 and 3.99 arranged according to consensus Round 1 MeanConvergenceMeanConvergenceStability Item(s.d.)(s.d.) Functions at any level of the helping relationship, from limited to intense 3.890.683.880.33-0.01 Seamless delivery of the counseling skills 3.940.733.820.53-0.12 Creative 4.000.693.940.56-0.06 Able to convey an appropr iate level of ease with colleagues and supervisors 3.720.573.760.560.04 Sets career goals 3.610.783.760.560.15 Confidence in approaching any interpersonal session with clients 3.780.653.880.60 0.10 Humorous 3.610.783.530.62 -0.08 Educates client regarding referral information 3.940.643.940.66 0.00 Evaluates progress toward own goals 3.890.833.940.66 0.05 Ability to write and state a disclosure statement 4.000.773.940.66 -0.06 Ability to change clients emotional arousal levels 3.560.923.650.70 0.09 Humble 3.670.773.590.71 -0.08 Succint 3.560.783.530.72 -0.03 Uses multiple techniques (i.e. eclectic style) 3.890.763.820.73 -0.07 Job development and placement skills 3.830.993.820.73 -0.01 Above average verbal and written expression 4.000.773.940.75 -0.06 Avoids sympathy 3.670.773.760.75 0.09 Commitment to social justice 3.890.763.760.75 -0.13 Assertive 3.830.863.760.75 -0.07 Closes cases appropriately 3.940.803.880.78 -0.06 Knowledge of career development 4.000.773.880.78 -0.12 Energetic 3.720.893.590.80 -0.13 Direct 3.670.913.530.80 -0.14 Round 2 Round 3 (X 2) (X 1) (X 1)(X 2)

PAGE 87

87Table 4-5. (continued) Round 1 MeanConvergenceMeanConvergenceStability Item(s.d.)(s.d.) Confident in approaching any interpersonal session with site or faculty supervisor 3.830.923.940.830.11 Working within a theoretical approach 3.830.863.760.83-0.07 Timely/opportune 3.830.863.760.83 -0.07 Calming/soothing 3.780.883.710.85 -0.07 Just 4.000.913.880.86 -0.12 Logical 3.830.923.760.90 -0.07 Intelligent/wise 3.670.913.710.92 0.04 Team player 3.720.963.710.92 -0.01 Balance the needs of the faculty supervisor and site supervisor 3.561.103.530.94 -0.03 Healthy 3.611.103.651.00 0.04 Diagnose (with some assistance) 3.561.153.591.06 0.03 Pleasure to work with 3.561.203.531.18 -0.03 Round 2 Round 3 (X 2) (X 1) (X 1)(X 2) Table 4-6. Items with means between 3.00 and 3.49 arranged according to consensus Round 1 MeanConvergenceMeanConvergenceStability Item(s.d.)(s.d.) Has faith in intuitive responses 3.330.843.410.800.08 Develop ability to supervise others 3.500.863.410.80-0.09 Experienced 3.440.783.410.80-0.03 Moderating/judging 3.171.103.000.87-0.17 Takes good notes during session 3.441.043.410.94-0.03 Round 2Round 3 (X 2) (X 1) (X 1)(X 2)

PAGE 88

88Table 4-7. Items with means below 3.00 arranged according to consensus Round 1 MeanConvergenceMeanConvergenceStability Item(s.d.)(s.d.) Ebullient/happy 3.000.972.880.86-0.12 Extroverted 2.780.942.710.92-0.07 Introverted 2.670.972.590.94-0.08 Round 2Round 3 (X 2) (X 1) (X 1)(X 2)

PAGE 89

89 CHAPTER 5 DISCUSSION Introduction This study involved identifying and obtaining consensus on a va riety of items useful for clinical supervision evaluation. This chapter is divided into four sect ions, which discuss the findings presented in the previous chapter. The first section includes an overview of the significant findings from Delphi Study. The second section de scribes limitations of this study. The third section discusses imp lications of the findings for e ducation and public policy. The final section includes recommendations for furthe r research and development of the findings of this study. Overview of Significant Findings Differences in Participant Response Sets Of the educators who were solicited for th is study, 21 completed the first round of the Delphi. Educators were instructed to provide a comprehensive list of specific skills, behaviors, and attributes that were usefu l/appropriate for clinical superv ision evaluation. Round 1 data collection included 410 items, with individual response sets ranging from 6 to 51 items, which were later collapsed into 188 substantially distinct items. As expected, many rehabilitation educators listed unique items that were occasiona lly identified by other part icipants (e.g. willing to research and explore counseli ng resources, continues to move session toward the therapeutic goal). However, several items fundamental to the therapeutic process were only occasionally identified by the participants For example, of the 21 pa rticipants, only six suggested empathetic/convey empathy to clients as useful/a ppropriate for clinical s upervision evaluation. Similarly, nine rehabilitation educators suggest ed active listening (e.g. attending to client;

PAGE 90

90 listening and responding) as usef ul/appropriate for clinical supe rvision evaluation, while only three participants listed build rapport and paraphrase client statements. In addition to item variability, participant respon se sets also varied. While participants were not expected to provide identical items, entire response se ts differed considerably from expert to expert. This phenomenon may relate to what Gizara and Forres t (2004) characterize as fundamental differences between supervisors regard ing the purpose of clin ical training. They report that some supervisors may believe that the clinical supervision process/internship strictly serves as a vehicle for the trainees professi onal development, while other supervisors may believe the clinical process functions as a gatekeeping mechanism (Gizara & Forrest). In addition, individual programs and supervisors may have considerable flexibility when evaluating trainee competence during clinical supervision (Robiner et al., 1993). This may be due to the lack of specificity provided by professional organizations a nd standards highlighting what supervisors should be evaluating dur ing clinical s upervision. High variability among response sets in th is study may suggest that supervisors conceptualize trainee competency differently. Consequently, trainees are evaluated against vastly different criteria which may lead to discre pancies in trainee preparedness and ultimately in rehabilitation counseling performance. If one supervisor believes the trainee should be personally fit to counsel, activel y continue their professional development, and sufficiently demonstrate certain clinical skills and treatmen t planning activities, while another supervisor simply evaluates the trainee s personal suitability and potential to develop adequate clinical skills, trainees may be unfairly subjected to different performance standards. Supervisor opinions regarding trainee performance will inva riably differ from supervisor to supervisor; however, allowing supervisors to use arbitrary cr iteria to evaluate trainee performance will

PAGE 91

91 unnecessarily complicate the evaluation process and all but guarantee trainees will be held to different performance standards. Item Identification A comprehensive set of clinical skills, behavi ors, and attributes representative of several theoretical competency domains compose the final item list. The majority of items on this newly constructed list are identical or conceptually similar to items commonly found in performance instruments and in the counseling literature (Bernard, 1997; Chan et al., 2004; Corey; 2004; Cromier & Cromier, 1991; Eriksen, & McAuli ffe, 2003; Hackney & Cromier, 1994; Hill, 2004; Lumadue & Duffey, 1999; Miller & Rollnick, 2004; Myrick & Kelly, 1971; Oetting & Michaels, 1982; Robiner et al., 1994). These findings further emphasize the compatibility of the rehabilitation counseling field with other counseling related professions. While rehabilitation counseling is closely rela ted to other counseling professions, there is an emphasis on working with people with disabilitie s and reintegrating these clients into society. Due to this emphasis it was expected that the experts would identify se veral items fundamental to the rehabilitation counseli ng profession that are not comm only found on other instruments. The experts in this sample listed items such as understanding the im pact of disability, knowledge of career development, job development and placement skills, and understanding of and ability to develop accommodati on strategies. The identification of these rehabilitation counseling items highlights the uniqueness of our field and the importance of creating measures that can w holly account for the work of rehabilitation counselors. Although an abundance of items were in itially gathered through the Round 1 investigation (188 items), there were several items not identifie d by the experts that may be useful for clinical supervision evaluation. The items that were not reported but expected to be found are ones listed on other counse ling performance measures, such as student demonstrates

PAGE 92

92 an awareness of power differences in the therap eutic relationship, and manages these differences therapeutically (Kerl et al., 2002), evoking and punctuating client strengths (Erikson & McAuliffe, 2003), improve client confidence/self -efficacy regarding change behavior (Milner & Rollnick, 2004), counselor refrains from usi ng social conversation (Hackney & Cormier, 2004), counselor helped client to develop ac tion steps for goal attainment (Hackney & Cormier), and counselor provided an opportunity for client to pr actice or rehearse action step (Hackney & Cormier). Due to the closely related nature of rehabilitatio n counseling with other counseling fields, future investig ations into supervision measures might consider items that are already being used on instruments fr om other counseling professions. Items Consensus As previously noted in Round 3, 106 of th e 188 items (56%) resulted in a greater consensus than had been achieved during Round 2. Sixty eight of the remaining 69 items demonstrated either equal consensus or minimal divergence (.02 or less increase in standard deviations) from Rounds 2 to 3. Only one item, Provides client with choices demonstrated an increase in standard deviation greater than .02 (. 07). This may be partially explained by the fact that this item had a Round 2 mean value almost directly in between 4 (Agree) and 5 (Strongly agree) (4.56). Therefore, more participants most likely decided to simply change their ratings from Strongly agree to Agree. Although the vast majority of items demons trated high consensus, four items demonstrated only moderate consensus. Thes e items include: Healthy (1.00), Capable (1.01), Diagnose (with some assistance) (1.06) and Pleasure to work with (1.20). Three of the four items (Capable, Pleasur e to work with, and Healthy) were all personal characteristics. Participants may be lieve that these terms are too general and interpreting their meaning may be futile. One participant commented that:

PAGE 93

93 I don't know what many of these words mean. I know what I mean by them, but I don't know what others may mean by them. The shad es of meaning of many of these words are too variable for me to interpret-and th erefore they are not useful (to me). The vast majority of the items identified in this study represen t specific skills or behaviors for which participants may have a basic understanding. However pers onal characteristics are more likely to manifest in various ways th rough out clinical supe rvision and therefore participants are more likely to define these terms subjectively. In addition, the term Healthy carries a certain connotation th at may be less palatable to rehabilitation counselors who almo st exclusively work with people with disabilities. Health is particularly variable and could imply several types including mental health, physical health, and/or functional capacity. Additionally Capab le may imply that the trainee should reach a certain level of ability before successfully completing their clinical training. Supervisors may view the clinical portion of the training as de velopmental (Gizara & Fo rrest, 1999) and therefore may disagree that this item is useful for clinical evaluation. Finally, participants did not reach high consensus on the term Pleasure to work w ith. In particular, this item appears to be highly subjective and therefore participants may be lieve this term is not useful for clinical supervision evaluation. Clinical supervisors recognize that the tr ainees clinical competency is independent of whether they feel the student was a pleasure to work with. Quite simply, supervisors and trainees may not work well togeth er, but this relationshi p should not necessarily determine the trainees clinical ability. The fourth item that demonstrated mode rate consensus is Diagnose (with some assistance). Interestingly, not only did this item exhibit moderate consensus, but the item also had a Round 3 mean value of 3.59. This item ma y exhibit moderate consensus for a variety of reasons. First, some clinical supervisors may believe that diag nosing is not a basic function of rehabilitation counseling. Traditionally reha bilitation counselors focus in the vocational

PAGE 94

94 assessment and placement arena. Although some clients may show signs of mental health disorders, some participants may not believe it is the role of reha bilitation counselors to diagnose a client. On the other hand, some pa rticipants may be affiliated with rehabilitation counseling programs that serve as vehicle for licen sed professional counselo rs or mental health counselors. Therefore, this item may simply reflect the overall diversity of the rehabilitation counseling profession. While some items remained at moderate consensus, five items increased from moderate consensus in Round 2 to high consensus in Round 3. Of these items, only Manages own mental health disabilities and D emonstrates emotional stability also increased their mean average from Round 2 to Round 3. The increased mean score in Round 3 suggests that more participants agreed that these items are useful for clinical supervision evaluation. Other items related to the trai nees self care, Recognize the importance of self care, An awareness of when their own personal issues ar e interfering with thei r ability to learn and practice in a professionally and ethically responsible manner, and Willingness to seek out professional counseling for themselv es when it is needed in order for them to function ethically and professionally, reached high consensus be fore Round 3. Interest ingly, the three items that demonstrated initial high cons ensus do not suggest that the trainee should manage any mental health or emotional issues, bu t rather that the student simply be aware of and be willing to seek help for these issues. In other words, so me participants may have initially believed that it is adequate for students to be seeking professional help for these issues, but that these issues did not necessarily have to be resolved in orde r to receive professional endorsement. Section G.3.a of the CRCC code of professional et hics (2002) clearly st ates that educators should be aware of the students personal limitations that might affect performance and help

PAGE 95

95 secure remedial assistance when needed, and di smiss students whose personal limitations cause them to be unable to provide competent service. Ironically, of all the items identified in this study, it would appear that M anages own mental health di sabilities and Demonstrates emotional stability should have demonstrated initial high cons ensus because they are the most closely related to any evaluation guidelines stated in the rehabili tation counseling professional. Ultimately, both Demonstrates emotional stability and Manages mental health disabilities achieved both high consensus (.51 and .62 respectively) and a high mean (4.59 and 4.56 respectively). These findings reinforce the fact that rehabilitation counseling educators are bound by professional ethics to ensu re that trainees manage thei r mental health issue and are emotionally stable. Unlike the previous two items, the remaining th ree items (Balance the needs of the faculty supervisor and site supervisor, Moderati ng/judging, and Takes good notes during session) had mean scores that decreased from Round 2 to Round 3. Additionally, the mean value for each of these three items fell between 3 (neutral) and 4 (agree). Thus, participants agreed that these items were less useful for clinic al supervision evaluation. More specifically, clinical supervisors may believe it is inappropriate for trainees to be responsible for addressing any c onflicts between the department and training site and therefore feel that the trainee should not balance the needs of their faculty supervisor and site supervisor. Moreover, many supervisors may feel that this ite m is simply not representative of clinical performance. Regarding the item Moderating/ju dging, participants may interpret this term differently. The term carries a strong connotat ion with which many educators may disagree. Specifically, some participants may interpret th e term as judging the client, which conflicts with the fundamental principles of rehabil itation counseling. On the other hand, other

PAGE 96

96 participants may see the value of judging certain situations (e.g. a client situation). In these instances, the trainee is no t judging the person per se, but is making judgments about information. Lastly, the item Takes good notes during session may have achieved high consensus, but a lower mean rating, because more participants may have felt that note taking is inappropriate during the counseli ng session. Clinical supervisor s may view note taking as an unnecessary distraction to the trainee, while others may simply feel that writing is simply not related to clinical performance. Limitations The first limitation to this study is the rehabili tation counseling educator sample size. This Delphi study solicited rehabil itation educators through the Nati onal Council of Rehabilitation Education (NCRE) listserv. The NCRE is the l eading professional associ ation for rehabilitation educators and represents over 90 higher education institutions and 600 individual memberships (NCRE, 2008). While it is unrealistic to iden tify the exact number of NCRE members who would qualify for this study, it was anticipated th at a sizeable number of NCRE members met the inclusion criteria and would par ticipate. Although it is unrealistic to identify the specific number of rehabilitation counseling educat ors who were eligible to participate in this study, a smaller sample size was attained than expected. Of the five rehabilitation couns eling Delphi studies found with in the literature review, only two (Rubin et al., 1998; Thielsen & Leahy, 2001) had a Round 1 sa mple size near or below the sample size attained in this investigation ( 23 and 18 respectively). R ubin et al. targeted 23 experts with a Round 1 response ra te of 100%; whereas Thielsen and Leahy targeted 20 experts with a Round 1 response rate of 90%. Unlike this present study, both Rubin et al. and Thielsen and Leahy initially identified a select number of experts to participate in Round 1 of their studies. Other rehabilitation c ounseling Delphi studies have in cluded Round 1 sample sizes of

PAGE 97

97 31 (Shaw et al., 2006), 44 (Curri er et al., 2001), and 111 (Hakim & Weinblatt, 1993). While the Round 1 sample size for this study may appear sma ll, Okoli & Pawlowski ( 2004) suggest that the size of a Delphi panel typically ranges between 10-18 experts. Th erefore, the final number of panel experts in this study is cons istent with a typical Delphi study. In addition to the Round 1 sample size, Del phi studies are inherently susceptible to participant attrition. Because the Delphi procedure requires experts to respond to several iterations of questions over time, it is uncommon for all participan ts to complete every round. For this study, 18 of the 21 experts from Round 1 completed Round 2 (85.7% response rate), and 17 of the 18 participants from Round 2 completed Round 3 (94.4% re sponse rate). Participant attrition between evaluation rounds (here reported as rounds 2 a nd 3) may be of particular concern because of its effect on item stability cal culations (i.e. the item mean difference between evaluation rounds). When high at trition rates occur between these rounds item stability becomes less reliable. In comparing the response rate between evaluation rounds of this study to other rehabilitation counseling Delphi studies, the resu lts are encouraging. Th is study resulted in a response rate between evaluation rounds of 94. 4%, whereas other rehabilitation counseling Delphi studies have yielded response rates of 68.8% (Currier et al., 2001), 71.4% (Shaw et al., 2006), 80.2% (Hakim & Weinblatt, 1993), 89.0% (Thielsen and Leahy, 2001), and 90.0% (Rubin et al., 1998). Even though partic ipant attrition is always a con cern during Delphi studies, the level of attrition for th is study is comparable to other Del phi studies in the rehabilitation counseling literature. However, the attrition level from Round 2 to Round 3 may have slightly impacted the standard deviati on calculations. Because the st andard deviation calculation is effected by the size of the sample, 17 participants (rather than 18) may result in small, but naturally occurring change in standard deviation. Therefore, any small incremental change in

PAGE 98

98 standard deviation (e.g. plus or minus .02) may be the result of the decreased sample size rather than item convergence or divergence. In addition, only seven of the 10 CORE defi ned regions are represented in this study. However, since regions are defined according to geographical location, di fferent regions contain an unequal number of rehabilitation education institutes. Of the three regions not represented, region II contains 10 schools, region VIII contains three schools, and region IX contains seven schools. Therefore the potential num ber of eligible partic ipants from a particular region may be considerably smaller than other regions. Lastly, 10 of the original 23 partic ipants (47.6%) in this study were from region IV (southeast). This region also has more re habilitation counseling education programs (22) than any other region. Although an abundance of participants from one region may typically cause alarm, it is not appare nt why participants from this region (or any other region) would respond in a manner that is in consistent with other rehabilitation counseling supervisors. Therefore, it is be lieved that an over representation of participants from this region would not necessarily bias the results. Implications Education Clinical competency dime nsions and benchmarks Within the rehabilitation counseling profession, researchers have extensively written about professional competence in a variety of arenas (e .g. multicultural). However, one area that has received little attention from the educators perspective is clinical supervision. With the exception of two clinical supe rvision models (Maki & Delw orth, 1996; Schultz, Copple, & Ososkie, 1999), the rehabilitation co unseling literature offers little related to clinical supervision competency criteria.

PAGE 99

99 Because the clinical portion of rehabilitation counseling tr aining reflects how the trainee will most likely perform in the field, it is pe rhaps the best opportunity for the supervisor to evaluate the trainees interpersonal and intrap ersonal functioning. Thus, the rehabilitation counseling profession will be well served to id entify a comprehensive list of trainee skills, personal characteristics, and other professional attributes useful for clinical supervision evaluation. The findings from this study will ha ve considerable implications for evaluating trainee competence during clinical supervision, as well as build ing upon existing rehabilitation counseling supervision models. During the clinical portion of the program, supervisors must fulfill administrative and evaluative duties as well as address the trainee s interand intrapersonal characteristics that manifest throughout clinical trai ning (Falvo & Parker, 2000). When students begin practicum or internship, professional and pers onal expectations should be clear (Falvo &Parker). To help clinical supervisors fulfill these a nd other supervisory roles, an exte nsive list of skills, behaviors, and attributes will allow clinical supervisors to conceptualize and evaluate trainee performance across multiple competency dimensions. Acco rding to Kaslow, Rubin, Bebeau, Leigh, Lictenberg, Nelson et al., 2007, benchmarks can capture the minimal level of each competency for each stage of professional development and guide our conceptualization of competent and problematic performance (p.481). Therefore, the general language used by CORE and CRCC to ensure rehabilitation education programs endorse professionally compet ent and personally suitable trainees can be further understood as specific competence di mensions, and more importantly specific competency criteria. For example, the findings suggest that clinical supervisors may find it useful to examine several dimensions includi ng the trainees general counseling skills (e.g.

PAGE 100

100 ability to present open-ended questions; reframing; affirmation), critical thinking skills (e.g. identifies client themes), intervention skills (e .g. establishes clear, measureable goals; evaluates interventions), counseling session management (e.g. provides purpose of counseling session; exhibits control of session), be havior during clinical supervision (e.g. appropriate case notes; open to supervisory feedback), pe rsonal attributes exhibited du ring the counseling session (e.g. genuine; non-judgmental), and professional behaviors (e.g. has appropriate boundaries; empowers the client). Much of the recent force behind identifying, training, and assessing professional competence comes from the psyc hology literature (Elman, Illf elder-Kaye, & Robiner, 2005; Kaslow, 2004; Kaslow, Borden Collins, Forrest Illfelder-Kaye, Nels on et al., 2004; Kaslow, Leigh, Smith, Bebeau, Lictenberg, Nelson, Portne y, et al., 2007; Rubin, Bebeau, et al., 2007; Kaslow, Rubin, Forrest, Elman, Van Horne, Jacobs, et al., 2007). As a whole, this body of literature offers a series of ideas, principl es, and proposals to aid psychology programs in assessing and improving current and future prof essional competence. Although the overall focus of the literature relates to a ssessing a set of foundational, co re, and specialty competencies throughout the breadth of psychol ogists professional developm ent, many of the proposed standards are applicable to a more narrowly defined area of training such as clinical supervision. According to several researchers (Kaslow, 2004; Kaslow, Rubin, Be beau, et al., 2007), optimal assessment of professional competence in cludes a multi-trait approach. Multi-trait analysis refers to concurrently assessing the trainees knowledge, skills, attitudes, performance, and ability to integrate these components across all competence domains. According to Kaslow, Rubin, Bebeau, et al., the assessment process should include the evaluation of individual and integrated competencies with multiple traits being evaluated simultaneously. When applying the

PAGE 101

101 concept of multi-trait evaluation to the area of rehabilitation counsel ing clinical supervision, the findings from this study identify a variety of competency dimensions as well as specific performance criteria and personal attributes that will allow clinical supervisors to holistically assess the trainees counseling competency. Although these clinical dimensions are appa rent, albeit theoretic al, they provide a comprehensive framework for evaluating how studen ts have integrated an d applied their training to the counseling session. Furthermore, the pe rformance indicators iden tified from this study provide specific behaviors and attr ibutes which represent those dime nsions. Thus, these research findings target, arguably, the most fundamental aspect of rehabilitation counseling training: the counseling session. With better defined dimens ions of counseling competence, rehabilitation counseling programs can ensure that trainees are held to the highe st standards of the profession, thus improving the gate-keeping process for thei r profession and ensuring novice rehabilitation counselors are meeting the needs and expect ations of their future clients. In addition to general evaluation applications, the findings from this study can also be used to supplement existing rehabilitation counsel ing supervision mode ls. Currently, the rehabilitation counseling literature offers two c linical supervision mode ls that describe the interaction between counseling trainee and supervisor. First, Ma ki and Delworths Structured Developmental Model (SDM) helps the supe rvisor catalog the trainees professional development by comparing his or her motivation, autonomy, and awareness relative to eight competency domains. By conceptualizing th ese internal processe s against the various competency domains, the supervisor can classify the trainees progress acc ording to one of four developmental stages (Level 1, Level 2, Level 3, and Level 3 Integrated). Depending on the

PAGE 102

102 level of development, the model suggests various strategies to help supervisors facilitate the counselors development. The findings from this study can be used to supplement the eight competence domains found in the SDM. Specifically, the SDM categorizes the domain s into one of two groups: (a) primary domains, and (b) process domains. Prim ary domains, also known as meta-domains, include: (a) sensitivity to indivi dual differences, (b) theoretical orientation, and (c) professional ethics and are continually examined during th e counselors development. The remaining domains, known as process domains include: (a) interper sonal assessment, (b) client assessment within the environment, (c) case conceptualization, (d) treatment goals and plans, and (e) intervention strategies (Maki & Delworth, 1995). While these domains are clearly organized to reflect the various professional and personal processes found within a counseling session, they fail to provide specific criteria useful for clinical competency evaluation. However, the findings from this study can be used to help expand the eight competency domains to include specific, behavioral criteria. Once each domain is define d by a cohesive set of specific competency criteria, supervisors can more accurately and co nsistently evaluate the trainees professional development. Like the SDM, the Integrated Model for s upervision proposed by Schultz et al. (1999), provides a thorough and practical model for the novice or expert supervisor. Because the working alliance between supervisor and trainee is critical, the In tegrated Model for supervision proposes that the supervisor adopt a particul ar supervisory role (teacher, counselor, and consultant) to suit the needs of th e trainee as well as foster the professional development of the student. Generally, the supervisor focuses on three areas of trainee competence (intervention skills, conceptualization skills, and personaliza tion skills) across three Phases of professional

PAGE 103

103 development (technical, integrative, and consulta tive). The amount of attention each competence domain receives depends on the sp ecific developmental phase of th e trainee. For example, Phase I (Technical), focuses on intervention and personalization skills such as client centered techniques as well as appropriate attitudes, self-trust, and internal ev aluation processes, but conceptualization skills are generally reserved fo r advanced phases of de velopment. Unlike the SDM (Maki & Delworth, 1995), the Integrative M odel for Supervision (Schultz et al., 1999) provides some specific competency criteria for trai nee evaluation. However, most of the specific competency criteria pertain to the trainees pers onal attributes (e.g. willingness to experience, develop trust in self, enhance internal evaluation, demonstration of unconditional positive regard, acceptance, and accurate empathy), while clinical skills are described more as generic domains (e.g. counseling micro skills, case conceptualizati on). Therefore, the findings from this study can not only supplement the existing competency criteria (particularly micro counseling skills and case conceptualization), but also allow the supervisor to assess th e trainees professional development across more competency domains. Supervision evaluation continuity Although supervisors rely on a variety of sources to evaluate a trainee (e.g. site supervisor evaluation), they typically depend on more dire ct observations such as reviewing counseling session tapes and clinical supe rvision exchanges to evaluate the trainees professional development. The evaluation process is larg ely dependent on two se parate, yet related, components: (a) the criteria used to evaluate trainees, and (b) th e supervisors opinion regarding trainee performance. While supervisory opinion is inherently variable, the criteria used to evaluate trainee performance can be made re latively consistent thr oughout various training programs. Rather than supervisor s using arbitrary criteria or criteria de veloped by their program

PAGE 104

104 to evaluate clinical competency a set of empirically based comp etency criteria, standards, and expectations might provide a foundation to ensu re increased continu ity across programs. Establishing a comprehensive set of competency criteria not only provides all supervisors a theoretical framework to evaluate the trainees clinical performance holistically, but also helps to ensure evaluation continuity between supervisor s. Currently, rehabili tation education programs are free to develop their own co mpetency criteria to define th e parameters of professional competence and personal suitability. Although so me differences in evaluation criteria are necessary to accommodate the specific goals of the program, the rehabilitation counseling profession would be better served to first identify specific clin ical expectations of trainee performance (Robiner et al., 1993) and then al low individual programs to build upon those fundamental criteria with additional skills to su it their program needs and goals. Regardless of which rehabilitation counseling program trainees atte nd or which clinical site they complete their practicum or internship; trainees could be evaluated against the same minimal set of competency criteria. Therefore, professiona l organizations, like CORE, can use the findings from this study as a foundation for expanding the concepts of minimal professional performance and personal expectations as well as offer additional benchmark items for continued professional development. Improving communication betw een supervisor and trainee In addition to potentially improving evaluation continuity among supervisors, the findings from this research study could positively impact the supervision process. Lehrman-Waterman and Ladany (2001) found that the processes of effective goal setting and feedback positively correlated with the supervisor/trainee working al liance, trainee self efficacy, and the trainees overall satisfaction with supervision; leading the authors to state that new and current approaches (to supervision) may need to reasse ss and highlight the im portance of goal setting

PAGE 105

105 and feedback (p.175). Supervisors who provide feedback based on explicit expectations or criteria (i.e. clearly describing specific behaviors) have th e potential to be better understood and received by students (Freeman, 1985), while su pervisors who provided vague, global, and/or abstract feedback have been considered lousy (Magnuson, Wilcoxon, & Norem, 2000). Clinical supervision typically requires the supe rvisor to engage in two types of evaluation processes: (a) formative and (b) summative (Robine r et al., 1993). Forma tive evaluation is the process whereby supervisors provid e direct feedback to the trainees during supervision to help foster their professional development. On th e other hand, summative evaluation is the more administrative process where the supervisor asse sses the trainees overall competence to ensure the student qualifies for prof essional endorsement. The aut hors note that supervisors and trainees oftentimes view summative evaluation wi th apprehension and consequently dislike the summative evaluation process (Robiner et al., p.4). However, Bernard and Goodyear (2004) caution that all aspects of clinical supervision, including the formative evaluation process, have inherent evaluative undertones. For exampl e, during formative evaluation, supervisors continuously send spoken and unspoken evaluative feedback to students; some of the supervisors evaluative comments are deliberately sent (encoded) by the supervisor to the trainee; others are received (decoded) by th e trainee and may or may not be an accurate understanding of the superv isors assessment (Bernard & Goodyear, p.20). To improve the formative and summative ev aluation processes, supervisors should communicate clear competency criteria to the trai nee during supervision. Therefore, the findings from this study can directly impact the supervis ory process by improving th e level and clarity of supervisor feedback and establishing clear expe ctations for student performance. Supervisors can use the findings from this research study to identify a comprehensive list of specific trainee

PAGE 106

106 skills and attributes from a variety of competen cy domains. Once identified, supervisors can communicate these criteria to students prior to clini cal supervision. Therefore, trainees will have a clearer understanding of what is expected of them during clinical supervision and supervisors will be able to communicate clear feedback to the student. Public Policy Association of American Colleges and Universities / Council for Higher Education Accredita tion In addition to specific educational implications the findings from this study may also have broader implications in shaping public policy. As part of their 2008 agenda, the Association of American Colleges and Univ ersities (AAC&U) and the C ouncil for Higher Education Accreditation (CHEA) state that it is imperativ e for the higher education community to improve the quality of education in the United States so that graduates are prepared to contribute to society as knowledgeable, engaged, and active citi zens (p.1). In order to aid universities and colleges to monitor the quality of student development, the organizations outlined a series of principles and actions to help ensure that the collegial experience is ch allenging for students, and assist higher education institutes in becoming more accountable and transparent. AAC&U and CHEA posit that alt hough the responsibility to es tablish better standards in higher education is shared among a variety of organizations, associations, and governing bodies, it is the colleges and universitie s themselves that must set a pr ecedent in helping the American higher education system maintain its designation as one of the worlds foremost leaders. Specifically, AACH and CHEA propose that the various organizational tiers within each university (i.e. university, college, department, division) shoul d develop ambitious, specific, and clearly stated goals for student learning ap propriate to its mission, resources, tradition, student body, and community setting (p.2). As such, the findings from th is research study will

PAGE 107

107 help various counseling and ps ychology programs establish clearer expectations and higher standards for student performance. Specifically, an exhaustive list of specific counseling skills, personal attributes, and professional behaviors will allow programs to c onceptualize the various aspects of counselor development more holistically. Thus, supervis ors and trainees alike will have a clearer understanding of what specific skills and attributes are being evaluated. Furthermore, by identifying a comprehensive list of evaluation cr iteria, clinical supervisors will be able to develop higher standards for student performance. Programs could begin to identify the specific skills and behaviors related to va rious levels of student performa nce. Simply stated, counseling programs can move toward the vision of AAC& U and CHEA by evaluating trainee performance across specific counseling skills, personal attributes, and professi onal behaviors; thus ensuring clear, ambitious goals for student performance as well as program accountability. Council on Rehabilitation Education / Commission on Rehabilitation Counseling Certification While the principles and actions set forth by AAC&U and CHEA call for universities and colleges to lead the way in ma intaining Americas global positi on in higher education, perhaps professional organizations and specific accrediting bodies also have an obligation to assist in identifying reasonable expectati ons and rigorous standards for pr ofessional development. As previously discussed, the Council on Rehabili tation Education (CORE) and the Commission on Rehabilitation Counseling Certification (CRCC) both acknowledge the need for rehabilitation counseling trainees to demonstr ate a level of professional comp etence and personal suitability. Whereas the CORE accreditation standards basica lly outline a series of general curriculum requirements, knowledge domains, and educatio nal outcomes for student performance, the CRCC code of professional ethics addresses the educat ors responsibility to be aware of the

PAGE 108

108 students academic and personal limitations. Although this litera ture acknowledges the educators role in trainee development a nd provides a basic foundation for assessing rehabilitation counselor ability, bot h are limited in terms of train ee evaluation. Sp ecifically, the performance outcomes provided by CORE are too general for clinical supervisors to truly evaluate a rehabilitation counsel ing trainee. Furthermore, ne ither CORE nor CRCC define any criteria related to personal suit ability; thus this concept is left to broad interpretation. Although the policies of CORE and CRCC are limited, the fi ndings from this study could have implications for how the rehabilitation c ounseling profession conceptualizes and defines trainee competence. With a clearer understand ing of the skills, attitudes, and behaviors rehabilitation counseling educators feel are useful for clinical supervision evaluation, professional organizations like CORE and CRCC can reexamine existing performance outcomes to assure that key components of counseling comp etency are captured with their criteria and also provide programs guidance about how to assess rehabilitation counseling performance and personal suitability. Furthermore, these re search findings may initiate dialogue among rehabilitation counseling educators regarding th e minimum skills and behaviors a trainee should be able to demonstrate in order to qualify for prof essional endorsement. Overall, the findings from this study can have implications for organizations at both the national and professional levels. On the one si de, rehabilitation counseling programs can uphold the ideals of AAC&U and CHEA by ensuring trainees are subjected to clear, rigorous standards, and on the other side, rehabilita tion counseling organizations can improve trainee evaluation and make sure people with disabilities are being se rved by competent, well-rounded individuals. Future Research As outlined in chapter 2, rehabilitation c ounseling educators face a variety of challenges when evaluating a trainee during c linical supervision. Moreover, professional organizations such

PAGE 109

109 as the Council on Rehabilitati on Education and the Commissi on on Rehabilitation Counseling Certification provide little guidance for evaluatin g clinical competency. Thus, rehabilitation counseling programs have developed and relied upon in-house instruments to evaluate trainee performance. Consequently, clinical competency standards may vary from program to program resulting in inconsistent trainee evaluation. The findings from this study resulted in a co mprehensive item bank of skills, behaviors, and personal attributes to consider when eval uating rehabilitation couns eling trainees during clinical supervision. This item bank will allow rehabilitation counseling educators to have a thorough understanding of specific activities relating to the clinical performance. As such, professionals should be able to refer to this item bank for use in future research on clinical supervision evaluation. Alt hough 10 theoretical competency domains were formed based on a content analysis of the data, a factor analysis is necessary to determine whether the generated items are actually representative of rehabilita tion counseling clinical performance as well as whether separate clinical competency constructs exist. Such an investigation will allow for the creation of a measure for evaluating trainee clinical competency. A validated clinical supervision instrument could help make the clinical supervision evaluation process more consistent as well as assist educators and trainees to catalog the students professional development. Although clinical supervision instruments have been previously developed, researcher s have yet to validate a clinic al supervision instrument using Item Response Theory methodologies. To accomplish this task, supervisors should evaluate trainee performance using the items identified in th is study. Next, a factor analysis should be performed to empirically determine if the items make up a unidimensional construct or multiple constructs. Item response methodologies, such as Rasch analysis, would then be applied to each

PAGE 110

110 construct to determine item fit and the hierarchal ordering of the items. Determining multiple developmental hierarchies of clinical performa nce will provide a basis for understanding trainee development. Once the instrument is fully devel oped, its usability and helpfulness can be tested among supervisors. In addition to developing a clinical supervision instrument, future research could be conducted to examine how a validat ed and reliable instrument imp acts the clinical supervision experience. More specifically, a clinical supervision instrument can help supervisors establish appropriate learning goals and provide specific feedback to trainees about their performance. As previous research has suggest, these two pr actices have been positively correlated with improving the supervisory working alliance, enhanc ing the trainees self efficacy, and increasing the trainees overall satisfaction with supervision (Lehrman -Waterman & Ladany, 2001). Building upon this research, the researcher can te st how a Rasch validated clinical supervision instrument impacts the supervisory relations hip as well as trainee performance. Conclusion Although the clinical supe rvision process is a critical co mponent of student preparation and affords educators the opportunity to identify unsuitable trainees, rehabilitation counseling literature and professional organizations provide little guidance related to specific performance expectations and evaluation crit eria. Consequently, training pr ograms may rely solely on the supervisors own clinical judgment or use unstandardized, in-house evaluation instruments to evaluate clinical performance. As a result, trainee evaluation during c linical supervision may differ from program to program. Despite this variability, reha bilitation counseling educators have identified and obtained consensus on 183 items representative of clinical performance. These items were subsequently organized into 10 theoretical competency domai ns including general counseling session items,

PAGE 111

111 process skills, conceptualization/assessment items identifying goals/treatment planning, personal attributes, supervision items, self care items professional behavior, general professional development, and other items. With the iden tification of an exhaustive list of clinical performance items, supervisors can better conceptualize trainee competency, offer specific, behaviorally defined feedback to the trainee, and establish realistic goals to aid in the trainees professional development. While identifying items for clinical assessment can improve the supervision process, future research is needed to validate a clinical superv ision instrument. Although a variety of clinical supervision instruments exist, no instrument to date has been validated with Item Response Theory (IRT) methods (e.g. Rasch analysis). Th e advantage of these methodologies compared to classical validation methods is c onsiderable. Rather than relyi ng on a total score, IRT validated instruments do not require the student to demons trate each item and therefore is flexible enough to accommodate all levels of students who train at various practicum/internship sites. Moreover, an IRT validated instrument woul d arrange items within each competency domain on a hierarchy from least to most difficult, th ereby allowing supervisors to ha ve an empirical understanding of trainee development.

PAGE 112

112 APPENDIX A DELPHI ROUND 1 SURVEY

PAGE 113

113

PAGE 114

114

PAGE 115

115 APPENDIX B ROUND 1 EMAIL TO RE HABILITATION COUN SELING EDUCATORS Dear Rehabilitation Counseling Educators, The Rehabilitation Science Program at the University of Florida is pleased to announce a new research study titled, Identifying Benchmar k Competency Criteria for a Rehabilitation Counseling Clinical Supervision Instrument: A De lphi Study. As part of this study, we are soliciting the opinions of Rehabilitation Counseli ng educators who currently supervise or have previously supervised masters level trainees during practicum and/or internship. More specifically, we are looking to identify and draw consensus on an extensive list of counseling skills, behaviors, and personality traits that clinical supervisors w ould find useful and/or appropriate for a Rehabilitation Counseling clinical supervision instrument. If you agree to take part in th is study, you will be asked to pa rticipate in an anonymous, 3 round Delphi study. There are no risk s or discomforts by participati ng on this study and participants may withdrawal at any time w ithout consequence. There will be no compensation offered for your participation. Although there will be no direct benefits for participating in this study, rehabilitation counseling educators will contri bute to the greater good of the profession by helping to improve the clinical supervision assessment process and aiding in the professional development of trainees. The research team has been approved by both th e National Council of Re habilitation Education and The University of Florida Institutional Review Board. This research project has obtained permission to gain access to the NCRE membersh ip for purposes of furthering the mission of the association. Additionally, only authorized persons from the University of Florida involved in this research study have the legal rights to review the research reco rds and will protect the confidentiality of those records to the extent by law or court order. If the results of the research are published or presented, all expert identities will remain anonymous. In order to be eligible for this study, participan ts must satisfy three incl usion criteria. First, educators should have supervised at least 15 rehabilitation couns eling students during practicum or internship. Next, educators s hould have supervised at least three semesters of practicum or internship. Finally, educators s hould have supervised at least th ree students during practicum or internship within the last three years. Your participation in this study is greatly appreciated. We unde rstand your time limitations as a Rehabilitation Counseling educator and seek to conduct this study in a thorough and efficient manner. It is anticipated that the total amount of time to complete all 3 rounds of the Delphi will not exceed 3 hours (Round 1 1.5 hours; Ro und 2 1 hour; Round 3 .5 hours). However, when considering the time associated with re cruiting participants, analyzing the data, and allowing participants time to co mplete each round, the entire Delphi process may take up to 3 months. If you are interested in partic ipating, you may access the first round of the Delphi study at:

PAGE 116

116 https://www.surveymonkey.com/s.aspx?sm=4263bzaxK7LtvuCwmk51pg_3d_3d. For questions regarding this study, please c ontact Michael Moorhouse, MHS, CRC at (352) 273-6491 or Linda Shaw, PhD at (352) 273-6045. If you have any questions regarding your rights as a research subject, you m ay contact the Institutional Review Board (IRB) offi ce at (352) 392-0433.

PAGE 117

117 APPENDIX C INFORMED CONSENT

PAGE 118

118 APPENDIX D ROUND 1 FRIENDLY REMINDER EMAIL Dear Rehabilitation Counseling Educators, This is a rem ind email to follow up on a res earch study being conducte d by the Rehabilitation Science Program at the University of Florid a titled, Identifying Benchmark Competency Criteria for a Rehabilitation Couns eling Clinical Supervision Instrument: A Delphi Study. As part of this study, we are so liciting the opinions of Rehabilit ation Counseling educators who currently supervise or have previ ously supervised masters level trainees during practicum and/or internship. More specifically, we are looking to identify and draw consensus on an extensive list of counseling skills, behaviors, a nd personality traits that clini cal supervisors would find useful and/or appropriate for a Rehabilitation Counsel ing clinical supervisi on instrument. If you agree to take part in this study, you w ill be asked to participate in an anonymous, 3 round Delphi study. There are no risks or discomfort s by participating on this study and participants may withdrawal at any time wit hout consequence. There will be no compensation offered for your participation. Although there will be no direct benefits for participating in this study, rehabilitation counseling educators will contri bute to the greater good of the profession by helping to improve the clinical supervision assessment process and aiding in the professional development of trainees. The research team has been approved by both the National Council of Re habilitation Education and The University of Florida Institutional Revi ew Board. This research project has obtained permission to gain access to the NCRE membership for purposes of furthering the mission of the association. Additionally, only authorized pers ons from the University of Florida involved in this research study have the le gal rights to review the research records and will protect the confidentiality of those records to the extent by law or court order. If the results of the research are published or presented, all expert identities will remain anonymous. In order to be eligible for this study, participan ts must satisfy three inclusion criteria. First, educators should have supervised at least 15 rehabil itation counseling student s during practicum or internship. Next, educators should have supe rvised at least three semesters of practicum or internship. Finally, educators s hould have supervised at least th ree students during practicum or internship within the last three years. Your participation in this study is greatly apprec iated. We understand your time limitations as a Rehabilitation Counseling educator and seek to conduct this study in a thorough and efficient manner. It is anticipated that the total amount of time to complete al l 3 rounds of the Delphi will not exceed 3 hours (Round 1 1.5 hours; Round 2 1 hour; Round 3 .5 hours). However, when considering the time associated with r ecruiting participants, analyzing the data, and allowing participants time to complete each round, the entire Delphi process may take up to 3 months. If you are interested in partic ipating, you may access the first round of the Delphi study at:

PAGE 119

119 https://www.surveymonkey.com/s.aspx?sm=4263bzaxK7LtvuCwmk51pg_3d_3d. For questions regarding this study, please contact Michael Moorhouse, MHS, CRC at (352) 273-6491 or Linda Shaw, PhD at (352) 273-6045. If you have any que stions regarding your rights as a research subject, you may contact the Institutional Review Board (IRB) offi ce at (352) 392-0433.

PAGE 120

120 APPENDIX E AUDIT TRIAL May 29, 2008 Com bined the terms/phrases empathy, empathe tic, ability to convey empathy to clients, conveying empathy, feeling, and empathetic into Empathetic / Convey empathy to clients Combined the terms/phrases rapport building, building rapport, and develop rapport into Build rapport Split the phrase engaging the client in the coun seling process through attending and orienting into two sections: engaging the client in the counseling process through attending and orienting Split the phrase being able to educate clients rega rding what to expect out of the session and any referral information into being able to ed ucate clients regarding what to expect out of the session and educates client re garding referral information Combine the terms/phrases orienting, explains the counseling process to client, and being able to educate clients regarding what to expect out of the session into orient client to the counseling process Split basic attending and observation skills in to two sections: basic attending skills and observation skills Combined the terms/phrases bas ic attending skills, listener, good attending, listening and responding, engaging the client in the counsel ing process through attending, demonstrating active listening, attending skill s, active listening, and a ttending skills training into Active listening (e.g. attending to client; listening and responding) Combined the terms/phrases: excellent observati on skills, attention to detail, observation skills and good observational skills into Observational skills/attention to detail Combined the terms/phrases non-verbal beha vior and trainees must know how to use good non-verbal skill: eye contact, body position, atte ntive silence, voice tone, gestures, facial expressions, physical distance, touch but t ook out attentive silence because it was not physical Combined the terms/phrases attentive silence, appropriate use of s ilence, good uses of silence, and appropriately uses silence into Appropriately uses silence Combined the terms/phrases knowing how to set boundaries appropriately, appropriate boundaries, and set clear boundaries with clie nt into Set clear boundaries with client

PAGE 121

121 Combined the terms/phrases paraphrasing, kno w how and when to paraphrase and ability to paraphrase into Paraphrase client statements Interpret non directive approaches to goal settin g as working with the client to set goals Split the phrase set attainable s hort and long term goals in accordance with client wishes into three sections: set attainable goals, set short and long te rm goals and set goals in accordance with client wishes Combine the terms/phrases: set attainable goals and establishes goals that are measurable and realistic into Sets realistic, attainable goals Combined the terms/phrases: identifies goals with client, goal setting that reflects the clients perspective on wants and needs, set goals in a ccordance with client wishes and non-directive approaches to goal settings into Identify goa ls that reflect the clients perspective Combined the terms/phrases goal setting, trainees need to know how to implement constructive goal setting, developing goals, obtaining goals and set short and long term goals into Identify short and long term goals Split the phrase controls and explores own feel ings and personal thoughts about clients to increase understanding and to decrease interferen ce in the counselor relationship into explores own feelings and personal though ts about clients to increase understanding and to decrease interference in the counselor re lationship and controls own feelings and personal thoughts about clients to increase understanding and to decrease interference in the counselor relationship Combined the terms/phrases explores own feel ings and personal thoughts about clients to increase understanding and to decrease interfer ence in the counselor relationship, identified personal values and beliefs that may impact on abili ty to fulfill role of rehabilitation counselor, recognizes personally sensitive ar eas in counseling content and/or process, awareness of their own limitations, biases, and belief s and how these impact the couns eling relationship and lack of self awareness and impact on others into Recognizes how th eir personal limita tions, biases, and beliefs impact the counseling relationship Split the phrase recognition of th e importance of self care and ready willingness to seek out professional counseling for themselv es when it is needed in order for them to function ethically and professionally into recognition of the import ance of self care and willingness to seek out professional counseling for themselv es when it is needed in order for them to function ethically and professionally Split the phrase actively keep in touch with an d working on their own issues into actively keep in touch with own issues and actively work on own issues Split the phrase personally knowi ng their own baggage and dealing with it to personally knowing with their own baggage and deal with own baggage

PAGE 122

122 Combined terms/phrases actively keep in touch with and working on their own issues, selfawareness, engages in self exploration, an aw areness of when their own personal issues are interfering with their ab ility to learn and practice in a prof essionally and ethically responsible manner, demonstrates personal insight, per sonally knowing with their own baggage, and willingness to explore discomfort honestly in to An awareness of wh en their own personal issues interfere with their ab ility to learn and practice in a professionally and ethically responsible manner Combined the terms/phrases do self-work on issu es that arise in counseling and supervision, emotional stability, actively work on own issues self-care, ability to manage personal issues or make determinations re garding whether they should be in a master's program at this time in their lives, keep own personal problems out of counseling session, and deal with own baggage to Manage/work on personal issues that arise during couns eling or supervision sessions Interpreted the phrase an ability to demonstrat e that their motivation for participating in required clinical learning experiences is not simply that it is required for their degree but that it is necessary for them in order to provide competen t, professional, and ethical services for people with disabilities as Demonstrates genuine motivation to become a competent and ethical rehabilitation counselor Combined the terms/phrases ability to show deep reflection of feelings, facilitates client expression of thoughts and feelings, demonstrating understanding of the clients ideas, experiences, and beliefs through reflection of fee lings and meanings of client statements, reflective listen, ref lection, reflecting fee ling, reflecting meaning, using reflections, reflection, and accurately reflect client feelings into two phras es Reflect feeling of client statements and Reflect mean ing of client statements June 1, 2008 Combine the terms/phrases confrontation, kno wing when to use confrontation, confront and confrontation to knowing when to use confrontation Split the phrase Uses supervision, including prepar ation, participation, and follow through into three sections prepares for s upervision, participates in supervision, and follows through with supervision Split the phrase Evaluates relevance, value, a nd meaning of feedback and implements changes as needed into two sections evaluates relevanc e, value, and meaning of supervisory feedback and implements feedback Combine the terms/phrases: and follows through with supervision, ability to use supervision to improve their skills, how they make use of that feedback, implements feedback, and an

PAGE 123

123 ability to demonstrate how they have incorp orated feedback into their practice into Incorporates supervisor feedback into practice Combine the terms/phrases: self-disclosing in a wa y that facilitates formation of an effective counseling relationship and serves to further client exploratio n, appropriate use of selfdisclosure, using self-disclosure appropriately, and self disclosure into Uses selfdisclosure appropriately June 2, 2008 Combine the terms/phrases: receptivity to clin ical supervision, acceptance of feedback, willingness to improve skills, willing to in corporate supervisors feedback into their practice, openness to feedback, and ability to take constructive feed back some students become defensive into Non-defensive /receptive to clinical supervision Split the phrase above average verb al and written expression into two phrases above average verbal expression and above average written expression Combine the terms/phrases: report writing ski lls, adequate case reporting skills, above average written expression, ability to write at a masters level, some students cannot write case notes, and preparation of written reports, pr ogress notes, and correspondence, emphasizing clarity, organization, and profe ssional presentation into Pre paration of written reports and progress notes (accurate, clarity, orga nization, professional presentation) Combine the terms/phrases: genuineness and genuine into the phrase genuine Interpret the phrase ability to approach all their clinical learning experiences openly and nonjudgmentally as two personal characteristics exhibited du ring a counseling session: open and non-judgmental; and one characteristic during clin ical supervision: open (open to supervisor feedback) Place the phrase open (open to supervisor f eedback) under the Non-de fensive /receptive to clinical supervision item Combine the terms/phrases: the ability to su spend preconceived notions and judgments of human behavior, non-judgmental, non-judgmental attitude, and judgmental into the term non-judgmental Split the phrase/terms: an ability to use counseling skills intentiona lly and to flex when needed into two phrases An ability to use counseling skills intentiona lly and Flex counseling skills when needed Interpret flex counselin g skills when needed as be flexib le with counseling skills when necessary

PAGE 124

124 Split the phrase open and closed questions into two sections open questions and closed questions Combine the terms/phrases: open question and open ended questions into Uses open ended questions Combine the terms/phrases: flexible, flexible (m ental and emotional), adaptability is one of the most important attributes, a nd a trainee needs to be able to adapt and change to make the therapeutic relationship beneficial to the client to Flexible/adaptable Combine the terms/phrases: closed questions a nd use close ended questions appropriately into Use close ended questions (when appropriate) Combine the terms/phrases empowering, and em power the client, self-detemination, and ability to form empowering relationships w ith clients into Empowers the client Combine the terms/phrases: ethical conduct, c onfidential, ethical, understands of ethical issues, including, for example, confidentiality representation of comp etence and qualifications, loyalties, and informed consent, demonstration of et hical behavior Showing up on site some students do not realize this is required and some site managers allow this to go on, and understands ethics of counseli ng relationship into Understands /practices in ethical manner (i.e. confidentiality, representation of competence/qualifications, dual relationships, informed consent, appropriate boundaries) Combine the terms/phrases showing up on sit e and being on time for practicum/internship into Combine the terms/phrases: providing client with choices and identifie s and explores options with client to Identify and explore options with client Combine the terms/phrases: minimal encouragers, using confirmations, and uses affirmations/positive reinforcement into Uses positive reinforcement/affirmations Split the phrase requests guidance as needed wh ile demonstrating openness to experimentation and risk taking into two secti ons requests guidance as needed and willing to experiment or take risks in session June 3, 2008 Combine the terms/phrases: individualizing treatment, planning for interventions and services that will assist the client in his or her goal, and being able to treatment plan into Plan interventions and services that will assist the client in his or her goal (i.e. individualized treatment planning) Split the phrase: assessing client traits, problem s, and needs in a comprehensive and specific manner that emphasizes strengths and deficits in behavioral terms into two sections assess

PAGE 125

125 client traits in a comprehensive and specific manner that emphasize strengths and deficits in behavioral terms and assess client problems in a comprehensive and specific manner Split the phrase: a trainee shoul d be aware of each client's individual characteristics and problems to be able to guide th e therapeutic relationship to fit the client's needs into two sections a trainee should be aware of each clie nts individual characteristics and a trainee should be aware of each clients individual problems Combine the terms/phrases: assess client traits in a comprehensive and specific manner that emphasize strengths and deficits in behavioral terms, a trainee s hould be aware of each clients individual characteristics, a ssess clients strengths and wea knesses into the phrase Assess client strengths and weaknesses Combine the terms/phrases: assess client proble ms in a comprehensive and specific manner, a trainee should be aware of each clients individual problems, and indentify client problems/issues into Identify client problems Combine the terms/phrases: demonstrating cultur al sensitivity, acceptance of difference, becoming cultural proficient, and awareness of cultural differences/influences into Aware of cultural differences/influences Combine the terms/phrases: understanding the impact of disability and knowledge of disability and its myriad effects into Understanding the impact of disability Combine the terms/phrases: structuring session to yield success and ability to continue to move session toward some therapeutic outcome to Continues to move session toward the therapeutic goal Combines the terms/phrases: avoids using hetero sexist, ablest, racist, and sexist language in interactions with other st udents, supervisors, clients, and other stakeholders in rehabilitation and use person first language into Uses appr opriate language/terminology (e.g. person first language; avoids heterosexist, able st, racist, and se xist language) Combines the terms/phrases: demonstrates resp ect for clients and thei r point of view and refrains from imposing personal values to Demons trates respect for clients values/beliefs Combine the terms/phrases: Summarizing proce ss dynamics, themes, activities, and milestones in the counseling relationship, summari ze, and summarizing into Summarize Combine the terms/phrases: provide feedback a nd provide accurate and honest feedback into Provide accurate feedback Combine the terms/phrases: ask effective questions and know how and when to use questions into Ask effective questions Combine the term/phrases: gives an accurate and balanced self-assessment of professional competencies, development of the ability to self-critique accurately, and assesses own

PAGE 126

126 strengths and weaknesses into Identifies ow n strengths and weakness related to counseling performance/competency Combine the term/phrases: conceptualize the client s life (i.e. psychosocia l, vocational, etc.) and conceptualizes client from all aspects into Conceptualize th e clients life (i.e. psychosocial, vocational, etc.) Split the phrase ability to apply theories and tech niques appropriately to the clients situation into two sections: ability to apply theory to a cl ients situation and ability to apply techniques based in theory to the clients situation Combine the terms/phrases ability to apply theory to a clients situation and working within a theoretical approach in to Working within a theoretical approach Combine the terms/phrases ability to apply technique s based in theory to the clients situation, specific counseling techniques based on a chosen theory, a nd utilize simple techniques grounded in counseling theory into Utilize simp le techniques grounded in counseling theory Combine the terms/phrases ability and willingne ss to research and explore resources (i.e. books, journal articles, videos, training, etc.) that help the practitione r with ideas for techniques and approaches and keep up to date on publications into Willing to research and explore counseling resources (i.e. j ournal articles, books, etc.) Combine the terms/phrases open and broad minded into Open minded Combine the terms/phrases willing to he lp and altruistic into Altruistic Combine the terms/phrases calm and soothing into Calming/soothing Combine the terms/phrases: intuitive and perceptive into Intuitive/perceptive Combine the terms/phrases: kind a nd thoughtful into Kind/thoughtful Combine the terms/phrases: common sense an d common sensical into Common sense Combine the terms/phrases prudent and conscientious into Conscientious/prudent Combine the terms/phrases: intelligent and wise into Intelligent/wise Interpret the phrase lim it setting as establishes appropr iate boundaries with clients Include establishes appropriate boundaries with client under the Understands/practices in ethical manner (i.e. confidentiality, repres entation of competence/qualifications, dual relationships, informed consent) item to form Understands/practices in ethical manner (i.e. confidentiality, representation of competence/qualifications, dual relationships, informed consent, establishes a ppropriate boundaries)

PAGE 127

127 Split the phrase Able to convey an appropriate level of ease when theyre working with clients, colleagues, and supervisors into two sections Able to convey an appropriate level of ease with clients and Able to convey an appropriate level of ease with colle agues and supervisors Combine the phrases comfortable with client and able to conve y an appropriate level of ease when theyre working with clients into comfortable with client Split the phrase confidence in approaching any interpersonal session with clients, site supervisor, or faculty supervisors into Confid ent in approaching any interpersonal session with clients and Confident in appro aching any interpersonal session w ith site or faculty supervisor June 4, 2008 Combine the items: Builds rappor t, Active listening (e.g. atte nding to client; listening and responding), Orient client to the counseling process, Non verb al skills (e.g. eye contact, body position, voice tone, gestures, facial expr essions, physical distance, and appropriate touch), Seamless delivery of the counseli ng skills, Recognizes how their personal limitations, biases, and beliefs impact the couns eling relationship, Set clear boundaries with client, Educates client regarding referral in formation, Demonstrates respect for clients values/beliefs, Controls ow n feelings and personal though ts about clients to increase understanding and to decrease inte rference in the counselor relati onship, Maintains awareness of personal feelings, limitations, and experien ces during the counseli ng session, Fosters a therapeutic environment, An ab ility to use counseling skills intentionally/purposefully, Flexible with counseling skill s (when necessary), Avoids c ountertransference, Focuses on client rather than themselves in counseling sessi ons, Identifies and meets the clients needs, Establishes a working alliance, Continues to move session toward the therapeutic goal, Clearly explains limits of confidentiality to client, Manages th e time of the session appropriately, Closes cases appr opriately, Develops trust, F aith in intuitive responses, Ability to be objective (lose th eir subjective thought), Keeps client focused, Reflects on the counseling session, Functions at any level of th e helping relationship, from limited to intense, Confidence in approaching any interpersonal se ssion with clients, Use multiple techniques (i.e. eclectic style), Takes good notes during session, and App lies structure to the counseling process (e.g. rapport building, exploration, treatment, term ination) into the category/competency domain G eneral Counseling Process Combine the items: Assess client interests, skills, and aptitudes, Knowledge of career development, Ability to conceptualize, Conc eptualize a case within a theoretical frame of reference, Diagnose (with some assistance) Clear understanding of theory(s), and Understanding the impact of disability into the category/competency domain Assessment/Conceptualization Combine the items: Observation skills, Use and understand solution skills (giving advice, information, and directive), Appropriately us es silence, Paraphrase client statements, Reflect feeling of client statements, Reflect meaning of c lient statements, Interpret client statements, Assess accuracy of in terpretations with the clients, Clarify client statements,

PAGE 128

128 Appropriate use confrontation, Disagreeing as needed, while maintaining a respectful and supportive attitude, Uses self-disclosure a ppropriately, Utilize simple techniques grounded in counseling theory, Utilize advanced techniques grounded in counseling theory, Working within a theoretical approach, Interviewi ng skills, Group counsel ing skills, Use open ended questions, Use close ended questions (when appropriate), Uses confirmations/affirmations, Ability to change clients emotional arousal levels, Critical thinking ability, Reframes client statements, Enhance or help client to increase his or her self-esteem, Avoids sympathy, Avoids advice, Ability to be insightful, Summarize, Provide accurate feedback, Asks effective qu estions, Intervenes in a ways that produce client progress in achieving his or her behavioral and/or emot ional goals, Assesses client motivation, Elicits motivating statements from client, Allows client to elaborate, Assess clients self-efficacy, Ability to handle clients strong emotions (e.g. crying), and Redirects client (when appropriate) into the category/competency domain Process skills Combine the items: Identify goals that reflect th e clients perspective, Identify short and long term goals, Sets realistic, attainable goals, Job development and placement skills, Providing client with choices, Identify and explore options with client, Individualized treatment planning (i.e. plan interventions and serv ices that will assist the client in his or her goal), Gather pertinent inform ation from the client and relevant others through assessments, observations, and facilitative ques tions, Assess client streng ths and weaknesses, Identify client problems, Prioritize client problems, C onceptualize the clients life (i.e. psychosocial, vocational, etc.), Understanding of and ab ility to develop accommodation strategies, Understanding of family and interpersonal dynamics, and R egularly evaluates client progress into the category/competency dom ain Identifying goals/Treatment planning Combine the items: Evaluates relevance, value, and meaning of supervisory feedback, Sets personal learning goals, Non-defensive /receptive to clinical superv ision, Prepares for supervision, Participates in supervision, Inc orporates supervisor feedback into practice, Demonstrates respect for the expertise of superv isors, An understanding of and belief in the importance of supervision, An awareness of when to seek supervision, Ability to state a disclosure statement, Ability to write a disclo sure statement, Preparation of written reports and progress notes (accuracy, clar ity, organization, professional presentation), Ability to reflect on ones practice, Reque sts guidance as needed (duri ng supervision), Balance the needs of the faculty supervisor and site supervision, Willing to experiment or take risks in session, Identifies own strengths and weakness relate d to counseling performance/competency, Willing to research and explore counseling re sources (i.e. journal articles, books, etc.), and Confident in approa ching any interpersonal session with site or faculty supervisor into the categor y/competency domain Supervision Combine the items: Respects other professiona ls, Demonstrates genuine motivation to become a competent and ethical rehabilitation counselor, Commitment to social justice, Understands/practices in ethical manner (i.e. confidentiality, representation of competence/qualifications, dual relationships, informed consent), Understands the challenges to communication with persons with hearing, vi sual, or cognitive impairments, Aware of cultural differences/influences with clients, Deals directly and appropriately with conflict rather than avoiding it, Abili ty to recognize when they are responding to supervisors, clients,

PAGE 129

129 students and other professionals in a manner that puts their own needs before those of their clients, Uses appropriate language/terminology (e .g. person first language; avoids heterosexist, ablest, racist, and sexist language ), Recognizes limits of competency, An ability to engage in critical self-reflection rather than focusing solely on the mistakes of others when they participate in clinical training, practicum and internship ex periences, Able to conv ey an appropriate level of ease with colleagues and supe rvisors, Appropriate dress, and Willingness to change when their own biases and beliefs that interfere with th eir ability to learn how to function as competent and ethical rehabilitation counselor into the category/competency domain Professional Behavior Combine the items: Sets career goals, Evalu ates progress toward own goals, Develop ability to supervise others, Above average verbal and written expression, Willingness to continue to learn and develop pr ofessionally, An ability to arti culate why the counseling skills they learn are crucial to all the roles and functio ns (e.g., vocational rehabilitation counselor, case manager, job developer, vocational evaluator) of rehabilitation counselors, and Goal oriented into the category/competency domain General Professional Development Combine the items: Recognition of the importance of self care, An awareness of when their own personal issues are interfering with their abil ity to learn and practice in a professionally and ethically responsible manner, M anage/work on personal issues that arise during counseling or supervision sessions, Willingness to seek out prof essional counseling for themselves when it is needed in order for them to function ethically and professionally, Emotional stability, and Manage own mental health disabilities into the category/competency domain Self Care Combine the items: Suicide risk assessment, Manages resources, Fiscal management, Time management, and Crisis management into the category/competency domain Other Combine the items: Empathetic / Convey empat hy to clients, Dependable, Demonstrates unconditional positive regard, Genuine, Warm, Open minded, Non-judgmental, Flexible/adaptable, Non-threatening demeanor, Patient, Honest, Trustful, Humble, Altruistic, Motivating, Energetic, Assertive, Calming/soothing, Positive, Attentive, Humorous, Respectful, C aring, Direct, G entle, Creative, Intelligent/wise, Logical, E xperienced, Confident (but not arrogant), Kind/thoughtful, Mature, Comfortable with client, Non-defensive (with clients), Common sense, Reliable, Conscientious/prudent, Intuitive /perceptive, Extroverted, Introverted, Thoughtful, Capable, Approachable, Ebullient/happy, Hard-working, Moderating/judging, Timely/opport une, Just, Succinct, Pleasure to work with, Team player, and Healthy into the category/c ompetency domain Personal Characteristics June 5, 2008 Combine the items: Collaborate s with client and Establish es a working alliance into Establishes a working alliance Move the phrase Recognizes how their persona l limitations, biases, and beliefs impact the counseling relationship into th e Professional Behavior Domain.

PAGE 130

130 Move the items: Assesses client motivation and Assess clients self-efficacy from Process Skills to Assessment/Conceptualization Move item: Intervenes in a ways that produce c lient progress in achieving his or her behavioral and/or emotional goals from Process Skills to Identifying goals/Treatment planning Move the items: Gather pertinent information fr om the client and relevant others through assessments, observations, and facilitative questi ons Assess client strengths and weaknesses Identify client problems Pri oritize client problems Concep tualize the clients life (i.e. psychosocial, vocational, etc.) and Understandi ng of family and interpersonal dynamics from Identifying goals/Treatment planning to Assessment/Conceptualization Combine the terms/phrases: Ability to state a di sclosure statement and Ability to write a disclosure statement into one statement Abil ity to state and write a disclosure statement Move the phrases: Ability to state and write a disclosure statement and Willing to research and explore counseling resources (i .e. journal articles, books, etc.) from Supervision to General Professional Development Move the phrases: Recognizes limits of competency and An ability to engage in critical selfreflection rather than focusing sole ly on the mistakes of others when they participate in clinical training, practicum and internship experiences from Professional Behavior to General Professional Development Move the phrase: Goal oriented from Gene ral Professional Development to Personal Characteristics Combine the terms: Dependable and R eliable into Dependable/reliable Combine the terms: Non-threatening demeanor and Approachable into Approachable/Nonthreatening demeanor Combine the terms: Kind, Thoughtful, and Kind/thoughtful into the phrase Kind/thoughtful/caring

PAGE 131

131 APPENDIX F ROUND 2 EMAIL TO RE HABILITATION COUN SELING EDUCATORS Dr. XXXXX, Thank you for participating in round 1 of our clin ical supervision Delphi study. The data has been analyzed and we have identified 184 item s representing 10 th eoretical competency domains. Per the Delphi format, we ask th at you complete the second round of questioning whereby you rate the usefulness of each item on a fi ve point Likert scale. In order to remain consistent with the purpose of th is study, please read the instru ctions page before rating the items. In addition, we know that your time is valuable a nd we really appreciate y our contribution to this project. However, because we are on a limited time schedule we would r eally appreciate if you could complete the second round by FRIDAY, JUNE 20th. Educators who pilo t tested this round completed it in approximately 30 minutes. Survey Link: https://www.surveymonke y.com/s.aspx?sm=ShzVwI jPeW1wOgt9VkLo_2bg_3d_3d If you have any questions, pleas e contact Michael Moorhouse at mmoorhou@phhp.ufl.edu or 352-273-6491. Thank you, Michael Moorhouse Rehabilitation Science Doctoral S tudent University of Florida Department of Behavioral Science & Community Health

PAGE 132

132 APPENDIX G DELPHI ROUND 2 SURVEY

PAGE 133

133

PAGE 134

134

PAGE 135

135

PAGE 136

136

PAGE 137

137

PAGE 138

138

PAGE 139

139

PAGE 140

140

PAGE 141

141 APPENDIX H ROUND 2 FRIENDLY REMINDER EMAIL Dr. XXXXX, This is a re minder email regardi ng your participation in the Un iversity of Florida clinical supervision Delphi study. As previously menti oned, the data from Round 1 has been analyzed and we have identified 184 items representing 10 theoretical competency domains. Per the Delphi format, we ask that you complete the second round of questioning whereby you rate the usefulness of each item on a five point Likert scale. In order to remain consistent with the purpose of this study, please read the inst ructions page before rating the items. We know that your time is valuable and we really apprecia te your contribution to this project. However, because we are on a limited time schedule we would really appreciate if you could complete the second round by FRIDAY, JUNE 20th. Educators who pilot tested this round completed it in approximately 30 minutes. Survey Link: https://www.surveymonke y.com/s.aspx?sm=SHzVwI jPeW1wOgt9VkLo_2bg_3d_3d If you have any questions, pleas e contact Michael Moorhouse at mmoorhou@phhp.ufl.edu or 352-273-6491. Thank you, Michael Moorhouse

PAGE 142

142 APPENDIX I ROUND 3 EMAIL TO RE HABILITATION COUN SELING EDUCATORS Dr. XXXXX Thank you for com pleting the first and second rounds of the clinic al supervision Delphi study. The third and final round is now ready for your participation. Si milar to round 2, this round is considerably quicker than round 1 and should take between 15-20 minutes to finish. Because we are on a limited time schedule and this round takes considerably less time, we would really appreciate if you could complete th is final round by FRIDAY, JULY 4th. The goal of the third round is to achieve c onsensus regarding whether or not each item is useful for clinical supervision evaluation. Therefore, your partic ipation in this round is highly valuable. You will now have the opportunity to review the group statistical data as well as your previous ratings. Based on this information, please deci de whether or not you would like to keep or change your responses. Unlike the previous roun d you will not be expected to re-rate each item, only those items you wish to change Again we want to express our extreme gratitude to you for participating in this study. To begin the final round of this study please click on the following link. https://www.surveymonkey.com/s.aspx?sm=FHfex8ctiiWdtz9s2E1Kfw_3d_3d Please contact Michael Moorhouse at (352) 273-6491 with any questions regarding this study.

PAGE 143

143 APPENDIX J DELPHI ROUND 3 SURVEY

PAGE 144

144

PAGE 145

145

PAGE 146

146

PAGE 147

147

PAGE 148

148

PAGE 149

149

PAGE 150

150

PAGE 151

151

PAGE 152

152

PAGE 153

153

PAGE 154

154

PAGE 155

155

PAGE 156

156

PAGE 157

157

PAGE 158

158

PAGE 159

159

PAGE 160

160 APPENDIX K ROUND 3 FRIENDLY REMINDER EMAIL Dr. XXXXX This is a re minder email regardi ng your participation in the Un iversity of Florida clinical supervision Delphi study. Thank you for comple ting the first and second rounds of the clinical supervision Delphi study. The third and final ro und is now ready for your participation. Similar to round 2, this round is considerably quicker than round 1 and should take between 15-20 minutes to finish. Because we are on a limited time schedule and this round takes considerably less time, we would really appreciate if you c ould complete this final round by FRIDAY, JULY 4th. The goal of the third round is to achieve c onsensus regarding whether or not each item is useful for clinical supervision evaluation. Therefore, your partic ipation in this round is highly valuable. You will now have the opportunity to review the group statistical data as well as your previous ratings. Based on this information, please deci de whether or not you would like to keep or change your responses. Unlike the previous roun d you will not be expected to re-rate each item, only those items you wish to change Again we want to express our extreme gratitude to you for participating in this study. To begin the final round of this study please click on the following link. https://www.surveymonkey.com/s.aspx?sm=FHfex8ctiiWdtz9s2E1Kfw_3d_3d Please contact Michael Moorhouse at (352) 273-6491 with any questions regarding this study.

PAGE 161

161 LIST OF REFERENCES Association of Am erican Colleges and Universities & C ouncil for Higher Education Accreditation (January 30, 2008). New leadership for student learning and accountability: A statement of pr inciples, commitments to action. Retrieved from: http://www.chea.org/pdf/2008.01.30_New_Leadership_Statement.pdf Chan, F., Be rven, N.L., Thomas, K.R. (Eds.). (2004). Counseling theories and techniques for rehabilitation health professionals. New York: Springer. Baldo, T. D., Softas-Nall, B. C., & Shaw, S. F. (1997). Student review an d retention in counselor education: An alternative to Frame and Stevens-Smith. Counselor Education & Supervision, 36, 245-253. Bemak, F., Epp, L. R., & Keys, S. G. (1999). Im paired graduate students: A process model of graduate program monitoring and intervention. International Journal for the Advancement of Counselling, 21, 19-30. Bernard, J. L. (1975). Due process in dropping the unsuitable clinical student. Professional Psychology, 6, 275-278. Bernard, J. M. (1997). The discrimina tion model. In C.E.Watkins (Ed.), Handbook of psychotherapy supervision (pp. 310-327). New York: Wiley. Bernard, J. M. & Goodyear, R. K. (1992). Fundamentals of clinical supervision. (2nd ed.). Boston: Allyn & Bacon. Bernard, J. M. & Goodyear, R. K. (2004). Fundamentals of clinical supervision. (3rd ed.). Boston: Pearson, Allyn & Bacon. Biaggio, M. K., Gasparikova-Krasnec, M., & Bauer, L. (1983). Evaluation of clinical psychology graduate students: The problem of the unsuitable student. Profession Practice of Psychology, 4, 9-20. Blackwell, T. L., Strohmer, D. C ., Belcas, E. M., & Burton, K. A. (2002). Ethics in rehabilitation counseling supervision. Rehabilitation Counseling Bulletin, 45, 240-247. Bond, T. G. & Fox, C. M. (2001). Applying the Rasch model: Fundamental measurement in the human sciences. Mahwah, NJ: Lawrence Erlbaum Associates. Boxley, R., Drew, C. R., & Rangel, D. M. (1986). Clinical trainee impairment in APA approved internship programs. Clinical Psychologist, 39, 49-52. Bradley, J. & Post, P. (1991). Impaired stude nts: Do we eliminate them from counselor education programs? Counselor Education & Supervision, 31, 100-108.

PAGE 162

162 Chagnon, J., & Russell, R.K. (1995). Assessmen t of supervisee developmental level and supervisory environment across supervisor experience. Journal of Counseling and Development, 73, 553-558. Commission of Rehabilitation C ounselor Certification (2002). Code of professional ethics for rehabilitation counselors. Rolling Meadows, IL. Corbin, J., & Strauss, A. (1990). Grounded Theory research: Procedures, canons, and evaluative criteria. Qualitative Sociology, 13, 3-21. Corey, G. (2004). Theory and practice of counseling and psychotherapy. (7th ed.). Pacific Grove, CA: Brooks/Cole-Thompson Learning. Cormier, W. H. & Cormier, L. S. (1991). Interviewing strategies for helpers fundamental skills and cognitive behavioral interventions. (3rd ed.). Pacific Grove: Brooks/Cole. Council on Rehabilitation Education (2002). CORE accreditation standards and procedures manual Rolling Meadows, IL. Cramer, D., & Howitt, D. (2004). The SAGE dictionary of statis tics: A practical resource for students in the social sciences. Thousand Oaks, CA: Sage. Currier, K.F., Chan, F., Berven, N.L., Habeck, R.V., & Taylor, D.W. (2001). Functions and knowledge domains for disability mana gement practice: A Delphi study. Rehabilitation Counseling Bulletin, 44, 133-143. Ellis, M.V. (1991). Research in clinical supervision: Revita lizing a scientific agenda. Counselor Education and Supervision, 30, 238-251. Elman, N.S., & Forrest, L. ( 2004). Psychotherapy in the reme diation of psychology trainees: Exploratory interviews w ith training directors. Professional psychology, research and practice, 35(2), 123-130. Elman, N.S., Illfelder-Kaye, J., & Robiner, W. N. (2005). Professional development: Training for professionalism as a foundation for competent practice in psychology. Professional Psychology: Research and Practice, 36, 367-375. Eriksen, K. P. & McAuliffe, G. J. (2003). A measure of counselor competence. Counselor Education & Supervision, 43, 120-133. Falvo, D. R., & Parker, R. M. (2000). Ethics in rehabilitation education and research. Rehabilitation Counseling Bulletin, 43, 197-202 Fink, A., Kosecoff, J., Chassin, M., & Brook, R.H. (1984). Consensus methods: Characteristics and guidelines for use. American Journal of Public Health, 74, 979-983.

PAGE 163

163 Forrest, L., Elman, N., Gizara, S ., & Vacha-Haase, T. (1999). Trainee impairment A review of identification, remediation, dismissal, and legal issues. Counseling Psychologist, 27, 627686. Frame, M. W. & Stevens-Smith, P. (1995). Out of harm's way: Enhancing monitoring and dismissal processes in couns elor education programs. Counselor Education & Supervision, 35, 118-129. Freeman, E. M. (1985). The importance of feedback in clinical supervisio n: Implications for direct practice. The Clinical Supervisor, 3, 5-26. Gallessich, J. & Olmstead, K. M. (1987). Training in counseling psychology: Issues and trends in 1986. Counseling Psychologist, 15, 596-600. Gizara, S. S. & Forrest, L. (2004). Supervisor s' experiences of trainee impairment and incompetence at APA-accredited internship sites. Professional Psychology-Research and Practice, 35, 131-140. Hackney, H. & Cormier, S. (1994). Counseling strategies and interventions. (4th ed.) Boston: Allyn & Bacon. Hahn, W. K. & Molnar, S. (1991). Intern evalua tion in university counse ling centers: Process, problems, and recommendations. The Counseling Psychologist, 19, 414-429. Hakim, S., & Weinblatt, J. (1993) The Delphi process as a tool for decision making: The case of vocational training of people with handicaps. Evaluation and Program Planning, 16, 2538. Hasson, F., Keeney, S., & McKenna, H. (2000). Re search guidelines for the Delphi survey technique. Journal of Advanced Nursing, 32, 1008-1015. Hensley-Choate, L. G., Smith, S. L., & Spru ill, D. (2005). Professional development of counselor education students: An explor atory study of professional performance indicators for assessment. International Journal for the Advancement of Counselling, 27, 383-397. Hensley, L. G., Smith, S. L., & Thompson, R. W. (2003). Assessing competencies of counselorsin-training: Complexities in evaluating personal and professional development. Counselor Education & Supervision, 42, 219-230. Herbert, J. T. (2004). Analysis of clinical supe rvision practices as documented in rehabilitation counseling syllabi and fieldwork manuals. Rehabilitation Education, 18, 13-33. Herbert, J. T. & Richardson, B. K. (1995). In troduction to the special issue on rehabilitation counselor supervision. Rehabilitation Education, 36, 278-281.

PAGE 164

164 Herbert, J. T., Ward, T. J., & Hemlick, L. M. (1995). Confirmatory factor analysis of the Supervisory Style Inventory and Re vised Supervision Questionnaire. Rehabilitation Counseling Bulletin, 38, 334-349. Hill, C. E. (2004). Helping skills: Facilitating ex ploration, insight, and action. (2nd ed.) Washington, D.C.: American Psychological Association. Holloway, E.L. (1987). Developmental models of supervision: Is it developmental? Professional Psychology: Research and Practice, 18, 209-216. Holloway, E.L. (1988). Models of counselor deve lopment or training models for supervision: Rejoinder to Stoltenberg and Delworth. Professional Psychology: Research and Practice, 19, 138-140. Hoffman, M. A., Hill, C. E., Holmes, S. E., & Fr eitas, G. F. (2005). Supervisor perspective on the process and outcome of giving easy, di fficult, and no feedback to supervisees. Journal of Counseling Psychology, 52, 3-13. Huprich, S. K. & Rudd, M. D. (20074). A national survey of trainee impairment of clinical, counseling, and school psychology doct oral programs and internships. Journal of Clinical Psychology, 60, 43-52. Janikowski, T. P. (1990). Rehabilitation c ounselor competencies: Recommendations for identification and assessment. Rehabilitation Education, 4, 185-193. Johnson, W. B. & Campbell, C. D. (2002). Charact er and fitness requirements for professional psychologists: Are there any? Professional psychology, research and practice, 33, 46-53. Kaslow, N.J. (2004). Competencies in professional psychology. The American Psychologist, 59, 774-781. Kaslow, N.J., Borden, K.A., Collins, F.L., Forrest L., Illfelder-Kaye, J., Nelson, P.D., et al. (2004). Competencies conference: Future directions in education and credentialing in professional psychology. Journal of Counseling Psychology, 60, 699-712. Kaslow, N.J., Rubin, N.J., Bebeau, M.J., Leigh, I.W., Lichtenberg, J.W., Nelson, P.D., et al. (2007). Guiding principles and recommendati ons for the assessment of competence. Professional Psychology: Re search and Practice, 38, 441-451. Kaslow, N.J., Rubin, N.J., Forrest, L., Elman, N. S., Van Horne, B.A., Jacobs, S.C., et al. (2007). Recognizing, assessing, and intervening with problems of professional competence. Professional Psychology: Re search and Practice, 38, 479-492. Kerl, S. B., Garcia, J. L., McCullough, C. S., & Maxwell, M. E. (2002). Systematic evaluation of professional performance: Legally supported procedure and process. Counselor Education & Supervision, 41, 321-332.

PAGE 165

165 Knoff, H. M. & Prout, H. T. (1985). Termin ating students from professional psychology programs: Criteria, procedures, and legal issues. Professional Psychology-Research and Practice, 16, 789-797. Koch, L., Schultz, J., Conyers, L., & Hennessey, M. (2005). Rehabilitation research in the 21st Century: Concerns and recommendations from members of the National Council on Rehabilitation Education. Rehabilitation Education, 19, 5-14. Kutz, S. L. (1986). Defining "impaired psychologist". American Psychologist, 41, 220. Lamb, D. H., Presser, N. R., Pfost, K. S., Baum M. C., Jackson, V. R., & Jarvis, P. A. (1987). Confronting professional impairment during th e internship: Identification, due process, and remediation. Professional Psychology: Research and Practice, 18, 597-603. Lamb, D. H., Cochran, D. J., & Jackson, V. R. (1991). Training and organizational issues associated with identifying and responding to intern impairment. Professional Psychology-Research and Practice, 22, 291-296. Leahy, M. J., Shapson, P. R., & Wright, G. N. (1987). Rehabilitation practitioner competencies by role and setting. Rehabilitation Counseling Bulletin, 87, 119-130. Leigh, I.W., Smith, I.L., Bebeau, M.J., Lichtenbe rg, J.W., Nelson, P.D., Portney, S., et al. (2007). Competency assessment models. Professional Psychology: Research and Practice, 38, 463-473. Li, C.-S. (2001). Non-academic behavioral indicators of student impairment in CACREPaccredited masters level counseling programs. Dissertation Abstract International, 62(3A). (UMI No. 3010363) Retrieved August 10, 2006, fr om Dissertation and Theses database. Linderman, C. (1981). Priorities Within the Heal th Care System: A Delphi Study. Kansas City: American Nurses Association. Lumadue, C. A. & Duffey, T. H. (1999). The role of graduate programs as gatekeepers: A model for evaluating student counselor competence. Counselor Education & Supervision, 39, 101-109. Magnuson, S., Wilcoxon, S.A., & Norem, K. (2000). A profile of lousy supervision: Experienced counselors perspectives. Counselor Education & Supervision, 39, 189-202. Maki, D. R. & Delworth, U. (1995). Clinical supervision: A definition and model for the rehabilitation counseling profession. Rehabilitation Counseling Bulletin, 38, 282-293. McAdams, C.R., & Foster, V.A. (2007). A guide to just and fair rem ediation of counseling students with professional performance deficiencies. Counselor Education and Supervision, 47(1), 2-13.

PAGE 166

166 McBride, A.J., Pates, R., Ramadan, R., & McGo wan, C. (2003). Delphi survey of experts opinions on strategies used by community pha rmacists to reduce over-the-counter drug misuse. Addiction, 98(4), 487. McCarthy, H. & Leierer, S. J. (2001). Consumer concepts of ideal char acteristics and minimum qualifications for rehabilitation counselors. Rehabilitation Counseling Bulletin, 45, 12-23. Mearns, J. & Allen, G. J. (1991) Graduate students experience in dealing with impaired peers, compared with faculty prediction: An exploratory study. Ethics and Behavior, 1, 191202. Michaelson, S. D., Estrada-Hern andez, N., & Wadsworth, J. S. (2003). A competency-based evaluation model for supervising novice counselirsin-training. Rehabilitation Education, 17, 215-223. Miller, H. L. (1979). A procedure for nonacademic failure of graduate students in psychology. Professional Psychology-Res earch and Practice, 10, 4-5. Miller, H. L. & Rickard, H. C. (1983). Procedures and students' rights in the evaluation process. Professional Psychology-Res earch and Practice, 14, 830-836. Miller, W. R. & Rollnick, S. (2002). Motivational interviewing: Preparing people for change. (2nd ed.) New York: Guilford Press. Murray, G.C., Portman, T., & Maki, D.R. ( 2003). Clinical supervision: Developmental differences during pre-service training. Rehabilitation Education, 17, 19-32. Muthard, J. E. & Salomone, P. (1969). The role s and functions of the rehabilitation counselor. Rehabilitation Counseling Bulletin, 13, 81-168. Myrick, R. D. & Kelly, F. D. (1971). A scale fo r evaluating practicum st udents in counseling and supervision. Counselor Education & Supervision, 10, 330-336. National Counsel on Rehabilita tion Education. (2007). National Counsel on Rehabilitation Education. Retrieved November 7, 2007, from the World Wide Web: http://www.rehabeducators.org/about.htm Nelson-Jones, R. (2005). Introduction to counselling skills: Texts and activities. (5th ed.). London: Sage. Norman, G. R., Streiner, D. L. ( 2000). Biostatistics: The bare essentials (2nd ed.), Toronto: B. C. Decker. Oetting, E. R. & Michaels, L. (1982). Oetting/Michaels anchored rating for therapists. Fort Collins: Rocky Mountain Behavioral Science Institute. Okoli, C., & Pawlowski, S.D. (2004). The Delphi method as a research tool: An example, design considerations and applications. Information & Management, 42, 15-29.

PAGE 167

167 Oliver, M. N. I., Bernstein, J. H., Anderson, K. G., Blashfield, R. K., & Roberts, M. C. (2004). An exploratory examination of student attitu des toward "impaired" peers in clinical psychology training programs. Professional Psychology: Research and Practice, 35, 141147. Olkin, R. & Gaughen, S. (1991). Ev aluation and dismissal of student s in master's level clinical programs: Legal parameters and survey results. Counselor Education & Supervision, 30, 276-288. Pope, V. T. & Kline, W. B. (2007). The personal characteristics of effective counselors: What 10 experts think. Psychological Reports, 84, 1339-1344. Procidano, M. E., Busch-Rossnagel, N. A., Re znikoff, M., & Geisinger, K. (1995). Responding to graduate students' professiona l deficiencies: A national survey. Journal of Clinical Psychology, 51, 426-433. QSR International (2007). NVivo 7 [Computer software]. Melbourne: QSR International. Ragg, D. M. (2001). Building effective helping skills: Th e foundation of ge neralist practice. Boston: Allyn and Bacon. Rasch, G. (1960). Probabilistic models for some in telligence and attainment tests. Copenhagen: Danmarks Paedagogiske Institute. Reising, G.N., & Daniels, M.H. (1983). A study of Hogans model of counselor development and supervision. Journal of Counseling Psychology, 30, 235-244. Robiner, W., Fuhrman, M., & Ristvedt, S. ( 1993). Evaluating difficulties in supervising psychology interns. The Clinical Psychologist, 46, 3-13. Robiner, W., Fuhrman, M., Ristvedt, S., Bobbitt, B., & Schirvar, J. (1994). The Minnesota supervisory inventory (MSI): Developmen t, psychometric characteristics, and supervisory evaluation issues. The Clinical Psychologist, 47, 4-17. Rowe, G., & Wright, G. (1999). The Delphi techni que as a forecasting tool: Issues and analysis. International Journal of Forecasting, 15, 353-375. Rubin, S.E., Matkin, R.E., Ashley, J., Beards ly, M.M., May, V.R., Onstott, K., et al. (1984). Roles and functions of certif ied rehabilitation counselors. Rehabilitation Counseling Bulletin, 27, 199-224. Rubin, S.E., McMahon, B.T., Chan, F., & Kamnetz, B. (1998). Research directions related to rehabilitation practic e: A Delphi study. Journal of Rehabilitation, 64, 19-26. SPSS Inc. (2001). SPSS for Windows [Computer software]. Chicago: SPSS Inc. Schultz, J. C., Copple, B. A., & Ososkie, J. N. (1999). An integrative model for supervision in rehabilitation counseling. Rehabilitation Education, 13, 323-334.

PAGE 168

168 Schultz, J.C., Ososkie, J.N., Fried, J.H., Ne lson, R.E., & Bardos, A.N. (2002). Clinical supervision in the public rehabi litation counseling setting. Rehabilitation Counseling Bulletin, 45, 213-222. Schwebel, M. & Coster, J. (1998). Well-functioni ng in professional psyc hologists: As program heads see it. Professional Psychology-Res earch and Practice, 29, 284-292. Scott, C.G., Nolin, J., & Wilburn, S.T. (2006). Barriers for effective c linical supervision for counseling students and postgraduate counsel ors: Implications for rehabilitation counselors. Rehabilitation Education, 20, 91-102. Shaw, L.R., Leahy, M.J., Chan, F., & Catala no, D. (2006). Contemporary issues facing rehabilitation counseling: A Delphi study of th e perspectives of leaders of the discipline. Rehabilitation Education, 20, 163-178. Shebib, B. (2003). Choices: Counseling skills for social workers and other professionals. Boston: Allyn and Bacon. Smith, R. M. (1994). A comparison of the power of Rasch total and between item fit statistics to detect measurement disturbances. Educational and Psychological Measurement, 54, 4255. Stebnicki, M. A., Allen, H. A., & Janikowski, T. P. (1997). Development of an instrument to assess perceived helpfulness of c linical supervisory behaviors. Rehabilitation Education, 11, 307-322. Stebnicki, M. A. (1998). Clinical supe rvision in rehabilitation counseling. Rehabilitation Education, 12, 137-159. Stoltenberg, C.D., & Delworth, U. (1988). Deve lopmental models of supervision: It is developmental. A response to Holloway. Professional Psychology: Research and Practice, 19, 134-137. Stoltenberg, C.D., McNeil, B.W., & Crethar ( 1995). Persuasion and development in counsellor supervision. The Counselling Psychologist, 23, 633-648. Strauss, A., & Corbin, J. (1998). Basics of qualitative research: Techniques and procedures for developing grounded theory (2nd ed.). London, England: Sage. SurveyMonkey (2007). SurveyMonkey [Compute r software]. Portland, OR: SurveyMonkey. Tarvydas, V. M. (1995). Ethics and the practice of rehabilita tion counselor supervision. Rehabilitation Counseling Bulletin, 38, 294-306. Tedesco, J. F. (1982). Premature termination of psychology interns. Professional PsychologyResearch and Practice, 13, 695-698.

PAGE 169

169 Thielsen, V. A. & Leahy, M. J. (2001). Essent ial knowledge and skills for effective clinical supervision in rehabilitation counseling. Rehabilitation Counseling Bulletin, 44, 196-208. Thurstone, L. L. (1927). The unit of measurement in educational scales. Journal of Educational Psychology, 18, 505-524. Thurstone, L. L. (1928). Attitudes can be measured. American Journal of Sociology, 33, 529554. Vacha-Haase, T.R. (1995). Impaired graduate st udents in APA-accredited clinical, counseling, and school psychology programs. Dissertation Abstract Inte rnational, 56(09-A). (UMI No. 9539306) Retrieved July 10, 2006, from Di ssertation and Theses database. Vacha-Haase, T., Davenport, D. S., & Kerewsky, S. D. (2004). Problematic students: Gatekeeping practices of academic professional psychology programs. Professional Psychology-Research and Practice, 35, 115-122. Vzquez-Ramos, R., Leahy, M., & Hernndez, N. E. (2007). The Delphi method in rehabilitation counseling research. Rehabilitation Counseling Bulletin, 50, 111-118. Walborn, F. S. (1996). Process variables: Four comm on elements of counseling and psychotherapy. Pacific Grove: Brooks/Cole. Wilkerson, K. (2006). Impaired students: Applyi ng the therapeutic process model to graduate training programs. Counselor Education & Supervision, 45, 201-217. Wiley, M.O., & Ray, P.B. (1986). Counse ling supervision by developmental level. Journal of Counseling Psychology, 33, 439-445. Williams, P.L., & Webb, C. (1994). The Delphi technique: A methodological discussion. Journal of Advanced Nursing, 19, 180-186. Woodyard, C. L. (1997). Indicators of impair ment in incoming and ongoing masters-level counseling students. Dissertation Abstracts Interna tional, 58(03-A). (UMI No. 9724633) Retrieved August 10, 2006, from Di ssertation and Theses database. Worthington, E.L. (1987). Changes in superv ision as counselors a nd supervisors gain experience: A review. Professional Psychology: Re search and Practice, 18, 189-208. Wright, B. D. & Linacre, J. M. (1989). Obse rvations are always ordinal; Measurements, however, must be interval. Archives of Physical Meas urement and Rehabilitation, 70, 857-860. Wright, B. D. (1997). A history of social science measurement. Educational Measurement: Issues and Practice, 16, 33-45, 52.

PAGE 170

170 BIOGRAPHICAL SKETCH Michael D. Moorhouse, MHS, CRC, is a docto ral candidate in the Rehabilitation Science Doctoral program at the University of Florida College of Public Health and Health Professions (Gainesville, FL). Mr. Moorhouse received his masters degrees in Rehabilitation Counseling from the University of Florid a in 2001. Accomplishments during Mr. Moorhouses doctoral student career include being a reci pient of a four year alumni fellowship, which allowed him to conduct his dissertation studies; recipient of the 2007 John Muth ard Award for excellence in research from the University of Florida College of Public Health and Health Professions, Department of Behavioral Science and Community Health; and recipient of the inaugural Ronald J. Spitznagel Outstanding Service Award. While completing the requirements for his doctoral degree, Mr. Moorhouse worked as a research assistant for Drs. Jamie Pomeranz and Mary Hennessey in the Department of Be havioral Science and Community Health at the University of Florida.