1 THE IMPACT OF A PROFESSIONAL DEVELOPMENT UNIT ON THE PROGRAM EVALUATION SKILLS OF IN SERVICE SCHOOL COUNSELORS By NICOLE MERLAN CARR A DISSERTATION PRESENTED TO THE GRADUATE SCHOOL OF THE UNIVERSITY OF FLORIDA IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY UNIVERSITY OF FLORIDA 2010
2 2010 Nicole Merlan Carr
3 Logan and Bryce, T hank you for your inspiration I love you
4 ACKNOWLEDGMENTS First and foremost, I thank G d for inspiring me to live my greatest life. Special appreciation goes to my advisor, Dr. Mary Ann Clark. She has been a mentor and a friend throughout this process. I appreciate her belief in me. Her wisdom and support w ere essential in the completion of this process. I thank my committee members, Dr. Sondra Smith, Dr. David Miller, and Dr. Harry Daniels. Their guidance and direction with my work helped the forge the path for not only this project, but my future goals. I thank the school counselors for the hard work they do every day. Participants in this study were both gracious, and eager. I value the time we shared developing and implementing a process so dear to my heart. I am grateful to have had Dr. Behrokh Ah madi as a mentor and dear friend. I am thankful to have worked with such impeccable members of the Research and Accountability Department of Pinellas County. I am fortunate to work in Pinellas County Schools where compassionate colleagues have always su rrounded me. I am thankful for my friends and family. They helped sustain me. Always it was a friend who could swoop in and wrap a blanket of support around me when I began to doubt myself. I am thankful to have been the child of my mother. Her ins piration has been a powerful force in all I do. Without her unconditional love I could not be the woman I hope to become. I thank my sons Logan and Bryce, who constantly provide me with opportunities to put life into perspective. The life that shines in them constantly inspires me to learn more. I am grateful to have been blessed to share the world with them.
5 TABLE OF CONTENTS page ACKNOWLEDGMENTS ................................ ................................ ................................ ............... 4 LIST OF TABLES ................................ ................................ ................................ ........................... 8 LIST OF FIGURES ................................ ................................ ................................ ......................... 9 ABSTRACT ................................ ................................ ................................ ................................ ... 10 CHAPTER 1 INTRODUCTION ................................ ................................ ................................ ................... 11 Accountability ................................ ................................ ................................ ......................... 11 Early Educational Accountability Systems ................................ ................................ ..... 11 Accountability Today ................................ ................................ ................................ ...... 12 School Counselor Accountability ................................ ................................ .................... 13 Scope of the Problem ................................ ................................ ................................ .............. 16 Statement of the Problem ................................ ................................ ................................ ........ 18 Need for the Study ................................ ................................ ................................ .................. 19 Theoretical Ra tionale ................................ ................................ ................................ .............. 20 Program Evaluation Theory ................................ ................................ ............................ 20 Social Cognitive Theory ................................ ................................ ................................ .. 21 Purpose of the Study ................................ ................................ ................................ ............... 22 Research Questions ................................ ................................ ................................ ................. 22 Definition of Terms ................................ ................................ ................................ ................ 22 Overview of the Remainder of the Dissertation ................................ ................................ ..... 23 2 REVIEW OF THE LITERATURE ................................ ................................ ......................... 24 Theoretical Framework ................................ ................................ ................................ ........... 24 Social Cog nitive Theory ................................ ................................ ................................ .. 24 Self Efficacy ................................ ................................ ................................ .................... 27 Determining Self Efficacy ................................ ................................ ............................... 27 Research on Self Efficacy ................................ ................................ ............................... 28 Professional Development ................................ ................................ ................................ ...... 29 Best Pra ctices in Professional Development ................................ ................................ ... 30 Methods of Evaluating Professional Development ................................ ......................... 33 Professional Development in the Area of Program Evaluation ................................ ....... 36 Program Evaluation ................................ ................................ ................................ ................ 38 Historical Ove rview ................................ ................................ ................................ ......... 38 Program Evaluation Models ................................ ................................ ............................ 39 Systematic Approach to Program Evaluation ................................ ................................ .. 40 Current State of Program Evaluation Among School Counselors ................................ .......... 43 Research Methodology ................................ ................................ ................................ .... 44
6 Instrumentation ................................ ................................ ................................ ................ 46 Program Evaluation In School Counseling ................................ ................................ ..... 47 Summary of the Literature ................................ ................................ ................................ ...... 51 3 METHODOLOGY ................................ ................................ ................................ .................. 52 Overview of the Study ................................ ................................ ................................ ............ 52 Relevant Variables ................................ ................................ ................................ .................. 52 Population ................................ ................................ ................................ ............................... 52 Sampling Procedures ................................ ................................ ................................ .............. 53 Research Design ................................ ................................ ................................ ..................... 55 Research Questions ................................ ................................ ................................ ......... 56 Hypotheses ................................ ................................ ................................ ...................... 57 Measurement Procedures/ Instrumentation ................................ ................................ ............ 57 Personal Data Sheet ................................ ................................ ................................ ......... 57 Essential Competencies for Program Evaluators Self Assessment ................................ 58 School Counselor Self Efficacy Scale ................................ ................................ ............. 60 Implementation Survey ................................ ................................ ................................ ... 61 Professional Development Unit ................................ ................................ .............................. 62 Data Analyses ................................ ................................ ................................ ......................... 64 Summary ................................ ................................ ................................ ................................ 67 4 RESULTS ................................ ................................ ................................ ................................ 69 Demographic Characteristics ................................ ................................ ................................ .. 69 Descriptive Statistics ................................ ................................ ................................ .............. 71 Inferential Statistics ................................ ................................ ................................ ................ 72 Research Question One ................................ ................................ ................................ ... 73 Rese arch Question Two ................................ ................................ ................................ ... 74 Research Question Three ................................ ................................ ................................ 76 Summary ................................ ................................ ................................ ................................ 78 5 DISCUSSION ................................ ................................ ................................ .......................... 88 Overview of the Study ................................ ................................ ................................ ............ 88 Research Q uestion One ................................ ................................ ................................ ... 88 Research Question Two ................................ ................................ ................................ ... 89 Research Question Three ................................ ................................ ................................ 90 Implications ................................ ................................ ................................ ............................ 91 Practice ................................ ................................ ................................ ............................ 91 Theory ................................ ................................ ................................ .............................. 94 Limita tions ................................ ................................ ................................ .............................. 95 Recommendations ................................ ................................ ................................ ................... 97 Summary ................................ ................................ ................................ ................................ 99 APPENDIX A EMAIL FROM GUIDANCE SUPERVISOR ................................ ................................ ...... 101
7 B EMAIL FROM PRINCIPAL ................................ ................................ ................................ 102 C S CRIPT PRESENTED BY R ESEARCHER AT GUIDANC E MEETING ......................... 103 D PERSONAL DATA SHEET ................................ ................................ ................................ 104 E SCHOOL COUNSELOR SEL F EFFICACY SCALE ................................ ......................... 105 F ESSENTIAL COMPETENCI ES FOR PROGRAM EVALU ATORS SELF ASSESSMENT ................................ ................................ ................................ ..................... 106 G IMPLEMENTATION SURVE Y ................................ ................................ .......................... 108 H INFORMED CONSENT ................................ ................................ ................................ ...... 110 I OUTLINE OF PROFESSIO NAL DEVLOPMENT UNIT ................................ .................... 111 LIST OF REFERENCES ................................ ................................ ................................ ............. 115 BIOGRAPHICAL SKETCH ................................ ................................ ................................ ....... 126
8 LIST OF TABLES Table page 3 1 Research design ................................ ................................ ................................ ...................... 68 4 ................................ ................................ ............... 79 4 2 Item statistics ECPE ................................ ................................ ................................ ................ 79 4 2 Item statistics ECPE continued ................................ ................................ ................................ 80 4 3 Summary item statistics ECPE ................................ ................................ ................................ 81 4 4 Item total statistics ECPE ................................ ................................ ................................ ........ 81 4 4 Item total statistics ECPE continued ................................ ................................ ....................... 82 4 5 SCES item statistics ................................ ................................ ................................ ................. 83 4 6 Summary item statistics SCES ................................ ................................ ................................ 84 4 7 Item total statistics SCES ................................ ................................ ................................ ........ 84 4 8 Paired samples statistics pretest posttest ECPE ................................ ................................ ....... 84 4 9 Paired samples test (T Test) ECPE ................................ ................................ .......................... 85 4 10 Paired samples statistics pretest posttest SCSE ................................ ................................ ..... 85 4 11 Paired samp les test (T Test) SCSE ................................ ................................ ........................ 85 4 12 Test between subject effects (ANCOVA) posttest difference ECPE ................................ .... 85 4 12 Test between subject effects (ANCOVA) pretest posttest difference ECPE continued ........ 86 4 13 Test between subject effects (ANCOVA) pretest posttest difference SCSE ......................... 86 4 14 Implementation survey results items 8 13 ................................ ................................ ............. 86 4 15 Implementation survey comments ................................ ................................ ......................... 86
9 LIST OF FIGURES Figure page 3 1 Professional development unit ................................ ................................ ................................ 68
1 0 Abstract of Dissertation Presented to the Graduate School of the University of Florida in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy THE IMPACT OF A PROFESSIONAL DEVELOPMENT UNIT ON THE PROGRAM EVALUATION SKILLS OF IN SERVICE SCHOOL COUNSELORS By Nicole Merlan Carr December 2010 Chair: Mary Ann Clark Major: School Counseling and Guidance The purpose of this study was to examine the impact of a professional development unit on program evaluation on in service elementary school evaluation, (b) self efficacy level towards performing program evaluation, and (c) ability to develop and conduct program evaluation in the school setting. This study provided a four session professional development u nit on program evaluation to a group of elementary school counselors in Pinellas County, Florida. There were 29 participants who completed the study. Results indicated after participating in a the professional development unit, based on Social Cognitive Theory, elementary school counselors increased their knowledge of program evaluation, increased their perceived self efficacy toward program evaluation skills, and applied the learning from the professional development in their school settings. Although f urther research is necessary, the findings of this study suggest that implementation of a professional development unit, like the one tested in this study might be a useful step toward increasing the program evaluation skills of school counselors. Addition ally, these results further validate the applications of Social Cognitive Theory to professional development.
11 CHAPTER 1 INTRODUCTION In 2001, the Elementary and Secondary Education Act (ESEA) was reauthorized to include No Child Left Behind (NCLB). The reauthorization stemmed from the standards based reform movement, which proposed (a) high academic standards for all students, (b) assessments to measure those expectations, and (c) accountability for those who work with students to meet these standards (E akin, 1996). In 1988, Congress established a National Assessment Governing Board who worked to create a standardized national assessment of what American students know. The National Assessment of Educational Programs (NAEP) assessment instrument was admi nistered nationally to students in grades 4, 8, and 12. Before NCLB, the results of the NAEP indicated achievement gaps between white students and black students and between white students and Hispanic students (National Center for Educational Statistics, 2009). The increased documentation of the achievement gap provided a heightened awareness, which strongly influenced the stringent regulations outlined in NCLB. Accountability Early Educational Accountability Systems Prior to the NCLB, the state accou ntability systems used to monitor student achievement, were not standardized across states nor did they accurately reflect the progress of all children. For example, some states reported student achievement data without disaggregating (Florida Department of Education, 2008). Often times, there were no minimum participation rates set, allowing some students or groups of students not to be assessed and thereby not represented in the reported data (Florida Department of Education, 2008). NCLB requires each state to have an approved accountability plan in place, which is based primarily on annual academic assessments and has established aggregate groups and minimum participation rates (U. S. Department of
12 Education, 2008). Annual measurable objectives are es tablished by each state with ensuing consequences on districts and schools for failing to meet standards. NCLB provides states, districts and schools with very specific requirements, such as detailed expectations of how funds are to be allocated, how sp ecific programs such as Reading First and Even Start Family Literacy program (U.S. Department of Education, 2010a) are to be implemented, and how targeted populations, like disadvantaged youth, homeless, migrant, and neglected and delinquent youth are to b e supported (U.S. Department of Education, 2008). Failure to meet federal guidelines can result in loss of federal funding. This level of legislated accountability is unprecedented and has altered all aspects of the educational system in America. Accoun tability Today The role of accountability continues to play a significant role in American education. In March of 2008, the United States Department of Education (USDOE) offered states the opportunity to participate in a Differentiated Accountability Pilo t Program (United States Department of Education, 2010b). USDOE requested that each state propose an accountability system that differentiated schools and apply appropriated interventions and consequences to support student achievement. Seventeen state proposals were approved. In 2010, Differentiated Accountability was written into Florida state statute (Florida Senate, 2010). It established criteria to differentiated schools with the greatest need of improv ement from schools with less intense needs for interventions based on student achievement data over time. Along with the categorizing criteria, came specific strategies and supports to be implemented by schools and school districts. In Florida, one requ ired strategy to be implemented is a mandatory, comprehensive instructional review of schools. Each district is required to develop an instructional monitoring process that includes classroom, school leadership team and school wide monitoring. Data is to
13 be reviewed to determine effectiveness of all programs and class offerings (Florida Bureau of School Improvement, 2010). Schools categorized as having the greatest need for intervention, based on student achievement data, are required to have an instruct ional review completed by the state department of education. These state instructional reviews include the review of all programs within a school (Florida Bureau of School Improvement, 2010). Another requirement of the Florida state accountability model is the establishment of a defined school based leadership team, which is to include the principal and assistant principal(s), school counselor, social worker, psychologist, and other school based personnel (Florida Bureau of School Improvement, 2010). Thi s team reviews data and uses problem solving to determine what is and what is not working to accomplishing highest student achievement. The ongoing increased levels of state and federal government monitoring have enhanced the need for school personnel t o expand their knowledge of program evaluation. Schools need to know if the programs they implement are doing what is required to accomplish the highest student achievement. School Counselor Accountability Specifically, the evaluation of school counselor programs has an important role in contributing to the accountability system in education (Fitch & Marshal, 2004; Loesch & Ritchie, 2008). State accountability systems are based heavily on student achievement in the areas of reading, math, writing, scienc e and sometimes social studies (U.S. Department of Education, 2010b). School counselors do not have a prescribed set of curriculum standards that are measured in a state assessment. However, school counselors have a role in student success (Stone & Dahir 2006). The mission of the school counselor is to contribute to the mission of the School Counseling Association, 2010). The importance of prevention and interventi on programs
14 is evident in the prescribed interventions found in the state accountability models. Many of the interventions address the social and emotional needs of students (US Department of Education, 2010b). As a result, the school counselors must be i ncreasingly responsible for the evaluation of their school counseling programs. (Educational Trust, 2007; House and Hayes, 2002). In 2002, The Educational Trust, an independent non profit organization, implemented the National School Counselor Training Initiative (NSCTI). NSCTI collaborated with private organizations, state departments, school counseling associations, higher education institutions, and school distric ts to promote school counselors as agents of change at their schools and in their districts. NSCTI has established the inclusion of the school counselor in the accountability system by defining the school counselor as a change agent who fosters student ac ademic achievement. NSCTI works to promote the school counselor as a change agent by providing in service school counselors with information to increase data driven decision making skills, and promoting research on effective school counseling practices to be used in standards based systems. Comprehensive school counseling programs reflect an understanding that school counselors are working in a climate of standards based education and accountability. The American School Counseling Association (ASCA) def ines comprehensive school counseling 2009). The value placed on set standard s and assessments is clearly a foundation of comprehensive school counseling programs.
15 The ASCA National Model (2005), a framework for school counseling programs, specifies standards of an effective school counseling program. It focuses on four areas: f oundation, delivery, management, and accountability. Accountability includes being engaged in continuous program evaluation activities (ASCA, 2010). It is expected that school counseling programs be evaluated and are linked to student achievement. The AS CA National Model (2005) provides a Program Audit to assist in this evaluation process. It is a rubric that assesses how closely a school counseling program aligns to the ASCA National Model (2005). The ASCA performance appraisal for school counselors is based on annually. These competencies reflect the four components of The ASCA National Model s the knowledge, abilities, skills and attitudes necessary to monitor and evaluate the processes and results of a school ASCA recognizes the value of program evaluati on knowledge to the school counselor. The Council for Accreditation of Counseling and Related Educational Programs (CACREP) identifies the need for studies which provide an understanding of research methods, statistical analysis, needs assessment, and prog ram evaluation as a program element in the revised CACREP accredited programs standards (CACREP, 2009). The CACREP organization sets the standards for school counselor preparation programs and has recognized the importance of research and program evaluati on. Ed Trust, ASCA, and CACREP see the need for school counselors to engage in program evaluation and illustrate the impact standards based performance has had on the school counseling profession.
16 Beyond external accountability, there is an ethical obligation for school counselors to be proficient in the area of program evaluation. If school counselors are to determine their effectiveness, then they must be able to evaluate what they are doing (ASC A, 2005). Ethical 2005). The school counselor conducting a small group guidance unit aimed at decreasing truancy needs to know if the group has achieved the s et objective, and school counselors need to have the ability to use that information to make future decisions about the program. School counselors are frequently in schools with very little clinical supervision (Borders, 2005; Borders & Brown, 2005). Sch ool principals, who regularly function as the school counselor supervisor, often lack the counseling theory to judge counselor effectiveness (Lambie & Williamson, 2004). Without clinical supervision the school counselor is left without a method to monitor and evaluate the quality of services (Bernard and Goodyear, 1998, 2004; Borders, 2005, Borders & Brown, 2005). This lack of supervision leaves the school counselor ethically bound to the task of evaluating their program (Borders, 2005). The school counse lor must know if the guidance curriculum they are implementing is effectively meeting the needs of their population (Loesch & Ritchie, 2008; Myrick, 1990, 1997, 2003; Vacc & Loesch, 2000). Thus, legislative requirements, professional counseling organizat ions, and ethical code expect school counselors to engage in the practice of program evaluation. Yet, despite the identification of program evaluation as an essential means to provide accountability, it is not fully implemented in school counselor practic e. (Studer, Oberman, & Womack, 2006; Trevisan, 2002a; Walsh, Barrett & DePaul, 2007 ) Scope of the Problem As of June 2010, all fifty states, the District of Columbia, and Puerto Rico have
17 esponsible to close existing achievement gaps. The decisions made at the state, district and school level are to be determined by data based evaluations. In 2007, Florida identified 937 Schools Identified in Need of Improvement (SINI) for failing to meet the set annual measurable objectives for student achievement (Florida Department of Education, 2009). These schools in SINI have sanctions imposed on them, including increased monitoring and evaluating of school and district level programs (U.S. Departm ent of Education, 2009). Continued failure to demonstrate student lev els of sanction result in a decrease in decision making powers at the school and district level (U.S. Department of Education, 2009). Increased oversight for these schools includes ectly to student achievement and targeting individual professional development plans to school reform efforts. In order to be in compliance with these mandates, school counselors must have the capacity to evaluate their programs and determine the impact o n student achievement. These sanctions have a direct connection to the role of the school counselor and the level of program evaluation knowledge (Florida Department of Education, 2009). The broad scope of these mandates illustrates the great need for sc hool counselors to have an understanding of program evaluation. The ASCA is an ever expanding professional organization for pre service and in service counselors as well as counselor educators and other professionals and currently has over 23,000 member s (ASCA, 2008). CACREP has accredited 184 school counseling university training programs (CACREP, 2009). In 2006, there were 121,608 elementary and secondary school counselors with a projected increase of 10.3 % by 2016 (Bureau of Labor Statistics, 2008). With
18 increasing numbers of in service school counselors as well as pre service counselors in training, working in highly accountable school systems, it is important to further explore methods to aluation. Statement of the Problem While there is increasing emphasis on program evaluation, school counselors are not systematically implementing program evaluation into their way of work (Fairchild & Steeley, 1995; Isaccs, 2003; McGannon, Carey & Dimmi tt, 2005). School counselors are not monitoring the implementation processes nor are anticipated program outcomes being evaluated. Program evaluation includes process evaluation, to address the monitoring of implementation and outcomes evaluation, and to assist in determining effectiveness (Patten, 2004). Both the process and outcomes are important elements if the program evaluation is going to be utilized (Patton, 2004). However, the lack of practice of program evaluation will no longer be tolerated in t he current educational climate (Educational Trust, 2007; Loesch & Richie, 2008; McGannon, Carey & Dimmitt, 2005; U.S. Department of Education, 2001). As a social science profession, it is essential that a program evaluation be more than just a checklist of duties and that value be assigned to the tasks (Scriven, 2004). Because school counselors work with people, largely children, it is imperative to determine the effectiveness of the school counseling curriculum (Lipsey, 2008). The school counselor is e xpected to go beyond determining effectiveness and establish that what is being done meets the needs of the population (Loesch & Ritchie, 2008; Myrick, 1990, 2003; Vacc & Loesch, 2000). Engaging in ongoing program evaluation is the first step in making th ose determinations. Program evaluation allows school counselors to utilize the evaluation process to develop more effective practices. In order to determine how counselors are effectively serving students in the academic, career and social emotional domai ns as outlined in the standards promoted by ASCA, the program
19 evaluation process must be valued and understood by the school counselor and their supervisors (Clark & Amatea, 2004; Eisner, 2004). It is often difficult to define the effectiveness of the scho ol counselor. Being engaged in program evaluation activities allows school counselors to constantly reflect on goals of the program. Without program evaluation, school counselors lack clear measurable objectives for their program. The school counselors who fail to demonstrate the impact of their program on student achievement begins to lose control of their program and are relegated to perform tasks and duties determined by others in the building (Adelman & Taylor, 2002). Need for the Study Evaluation o f the guidance curriculum has always been an element in the comprehensive guidance models (Gysbers & Henderson, 2000; Myrick, 1993, 2003; Vacc & Loesch, 2000). The ASCA National Model (2005) further established the expectation that school counselors will provide program evaluation (McGannon, Carey & Dimmitt, 2005). The ASCA has made tremendous strides to create and promote effective tools school counselors can use to determine if their program is aligned to The ASCA National Model (2005). If school couns elors use the ASCA program audit, performance appraisal, and the standards prescribed in the framework they will be able to assess if their school counseling programs are aligned to The ASCA National Model (2005). However, it is possible that a lack of ho listic understanding of program evaluation makes even the best instruments ineffective to the user (Guba & Lincoln, 2004). Without the ability to link student achievement to the school counseling program, school evaluation skills are required to make those connections. What causes school counselors not to practice program evaluation? Trevisan (2002a) identifies lack of training, mistrust of the evaluation proc ess, and perceived difficulty in
20 measuring outcomes as contributing factors to the lack of program evaluation implemented among school counselors. CACREP programs do require coursework intended to provide the school counselor with the appropriate training However, not all counselor education programs are CACREP accredited. Even with some of the best pre service training, there is no guarantee that the in service school counselor will be effective in carrying out program evaluation (Bandura, 2007; Wittme r & Loesch, 1986 ). Another difficulty may be in a lack of motivation by the school counselor to conduct evaluation. Motivational factors are categorized as those that push and those that pull (Trevisan, 2002b). For the school counselor the external push factors are evident in legislation and professional organizations. However, the push may not be as strong at the school level where pull factors exist in cou nselors as the professional ethics, but the deficiency of skills may hinder the utilization of evaluation. In order to address the lack of motivation, training, trust, and perceived difficulty surrounding school counselor program evaluation, a training ne eds to be conducted and evaluated, which addresses these potential causes (Trevisan, 2002b). Theoretical Rationale Program Evaluation Theory Program evaluation is largely viewed as the assessing of program outcomes (Patton, 1997; Rossi, Lispey and Free man, 2004). However, evaluation has a broader scope than outcome assessment. Rossi et al (2004) identify five areas of assessment in Program Evaluation: (a) Need for the Program, (b) Program Theory, (c) Program Process, (d) Impact of the Program, and ( e) Cost Analysis. Each of these dimensions builds upon the next. Identifying the need is the first step. Without a clearly defined need for the program the evaluator cannot proceed in evaluating the
21 appropriateness of the program design and theory. The process can only be effective when it is based on a theoretically sound practice. The impact cannot be determined if the implementation of the process has not taken place. Finally, without clear outcomes it is not possible to determine the cost effecti veness. Each level of assessment is interwoven with the one before it when determining the value of a program. Understanding of the relationships between these elements is the precursor to determining if a program is effective. According to Rossi, Lis pey and Freeman (2004), program evaluation should review these procedures for gathering, analyzing, and interpreting evidence about the performance of a al, 2004 p.16). The evaluation should be utilized to inform future decisions related to the program (Patton, 1997; Rossi et al, 2004; Schwitzer, 1997). It is the assignment of value to the goals, processes and impacts of a program or practice that gives the evaluation a a Systematic Approach to program evaluation will provide a theoretical framework for this study. Social Cognitive Theory Social cognitive the ory proposes that learning is strongly influenced by observation (Bandura, 1986, 1994, 2007). Through modeling and comparing oneself to someone else, an individual can cultivate his or her thoughts and feelings about a skill (Bandura 1986, 1994; Zimmerman feelings about a skill influence the learning. In social cognitive theory, the internal belief that one could change is termed self efficacy (Bandura, 1977, 1994, 2007). Self efficacy is the perception an individual has of their ability to complete an action (Bandura 1986, 2007). Social
22 cognitive theory, specifically the concept of self efficacy, will also provide a theoretical framework for this study. Purpose of the Study The purpose of this experimental study will be to examine the impact of a Program Evaluation training unit on in (b) self efficacy level towards performing program evaluation, and (c) ability to develop and conduct program evaluation in the school setting. The professional development unit will be delivered to a voluntary sample of elementary school counselors from a large school district in Florida. This research will provide initial data and implications regarding strategies for increasing the application of program evaluation knowledge and skills of in service counselors Research Questions This study will address the following research questions: 1. Will school counselors have a greater understanding of program evaluation principles after taking part in a professional development unit? Will demographic factors influence performance on the Essential Competencies for Program Evaluators Self Assessme nt? efficacy towards efficacy towards program evaluation implementation as measured by the School Counselor Self Efficacy Scale ? 3. Will school counselors be able to develop a program evaluation in their own school setting as a result of having received professional development training? Definition of Terms The following terms are defined as they are used in this study:
23 Accountability is in the broadest sense being responsible for actions. In the field of education, individuals are responsible to the public as well as the individuals they are serving. School Counselors are accounta ble to all stakeholders ( Astramovich & Coker, 2007; Herr. 2001; House & Hayes, 2002; & Loesch, 2001; Loesch & Ritchie, 2008) In service school counselor a counselor who has completed the training to be a state certified school counselor and is working a s a school counselor in the school setting. Instrument the tool used to measure a defined concept, skill, or variable (Sink and Spencer, 2007; Struder, Oberman & Womak, 2006; Whiston & Aricak, 2005, 2007) Professional Development Unit a unit in this stu dy is a multiple session professional development. It is not a one time meeting where information is shared, but instead information is progressive and builds upon prior sessions. Program Evaluation the systematic approach to determining the value of a p rogram, policy or procedure (Rossi, Lipsey, & Freeman, 2004) Self Efficacy is a component of Social Cognitive Theory. It is the perception an individual has of their ability to complete an action (Bandura 1986, 2007). Social Cognitive Theory is based on the concept that human actions are the result of the (Bandura 1986, 2007). Overview of the Remainder of the Dissertation This first chapter provided an introduct ion to the study. Chapter 2 review s the literature in several important areas : school counselors as evaluators, program evaluation theory, professional development theory and self efficacy theory The variables, population sampling procedures, research design, measu rement procedures, professional development unit, and data analysis are presented in Chapter 3. The results of the study are presented in Chapter 4. Chapter 5 discuss es the significant findings, limitations, implications, and recommendations for further research.
24 CHAPTER 2 REVIEW OF THE LITERA TURE The purpose of this study will be to examine the impact a professional development unit on program evaluation has on in (b) self efficacy level towards performing program evaluation, and (c) ability to develop and conduct program evaluation in the school setting. This chapter reviews literature relevant to th is study including the theoretical framework of social cognitive theory; self efficacy theory and instrumentation; effective characteristics of professional development as revealed in the evaluation of professional development; program evaluation theory; a nd the current state of program evaluation amongst school counselors. Theoretical Framework Social Cognitive Theory he realized that behavior was influenced by more than external rewards and punishments (Bandura, 1977, 2007). Social Cognitive Theory is based on the concept that human actions are factors (Bandura 1986, 2007). The attribution of behavior to the external factors is evident in together to get what they want (Bandura, 2007). In Social Cognitive Theory, this concept is kn own as reciprocal determinism; the idea that behaviors impact the environment and the environment impacts the behaviors (Bandura, 1986, 2007). The internal factors are defined as thoughts, feelings, and biological make up (Bandura, 2007). Social Cognitive Theory views the person both the strategic thinker about how to manage the environment and later the evaluator of
25 the adequacy of his or her knowledge, thinking skills, capabilities, and action strategies (Bandura, 2007). According to Social Cognitive Theory, learning is strongly influenced by observation (Bandura, 1986, 1994, 2007). By modeling, a process of comparing oneself to someone else, an individual can develop his or her thoughts and feelings about a skill (Bandura 1986, 1994; Zimmerman & Schu nk, 2004). To learn from modeling, the individual must have the capacity to pay attention; retain what is being modeled; reproduce what is being modeled; and be motivated. Strong and effective modeling allows an individual to acquire skills and have the capacity to apply those skills in an appropriate way in different settings. Mindful application of the skills implemented in a routine is required to develop efficient acquisition of those skills (Bandura, 1999). The reinforcement of the skill provides the opportunity to self regulate the learning (Wang & Lin, 2006) Making the application a routine can be beneficial when the skills can be the skills cannot be adapted (Bandura, 2007). Since people do not always do everything they learn Social Cognitive Theory makes a distinction between learning and performing (Bandura, 2007). Social Cognitive Theory postulates that there are three main motivational incen tives that influence performance: direct motivation, vicarious learning, and self produced skill (Bandura 1986, 1994; Zimmerman & performance. Vicarious learning is the influence the observation of others has on the performance, and the self produced skill is the identification the person has on the performance
26 performance. People are more likely to perform if they anticipate positive outcomes than if they anticipate negative outcomes (Bandura, 2007). and application of computer skills. Early work of Compeau & Higgins (1995) and later Wang, & Lin (2006) examine the impact of personal, behavioral and environmental influences. Compeau & Higgins (1995) compared a behavior modeling to a lecture based approach and found the modeling the more effec tive training model. Additionally, watching others perform the skills attributed to higher levels of self efficacy and influenced performance (Compeau & Higgins, 1995). Wang & Lin (2006) also applied the idea of self regulated learning in the development of the application used to teach the skills. Adult learners were able to monitor their learning and adjust to acquire the skills. Findings indicated the participants with hig her levels of motivation were able to respond more appropriately and effectively in the web based learning environment (Wang & Lin, 2006). Social Cognitive Theory based interventions with teachers are also found in the literature. The impact of a Social Cognitive Theory based intervention with Physical Education teachers indicated an increased efficacy in teaching Exemplary Physical Education Curriculum (EPEC) and overcoming barriers (Martin, McCaughtry, Kulinna, & Cothran, 2009). The study applied mode ling, and vicarious learning by pairing experienced teachers with new teachers during the intervention, which included professional development on teaching EPEC. used in a Business English course with 24 freshman university students (VanSteendam, Rijlaarsdam, Sercu, & VandenBergh, 2010). The approach used observation, modeling, and self
27 regulated learning. Findings indicated if emulation happened then observation and pra ctice were equally effective in impacting the acquisition of the strategy (VanSteendam et, al., 2010). Social Cognitive Theory is also prevalent in teacher preparation texts. Practical application and use of Social Cognitive Theory, including the use of modeling, self regulated learning, and vicarious learning are seen in these texts (Kumpulainen, 2001; Pajares, 2002). Social Cognitive Theory, including, modeling, self regulated learning and vicarious learning is the framework used in the development of the professional development unit for this study. Self Efficacy Social Cognitive Theory discusses how modeling, motivation, and the experience of doing can each lead to an internal belief that one could change a behavior (Bandura, 1977, 1994, 2007). T he internal belief that one could change a behavior -Self efficacy -is a central component of Social Cognitive Theory (Bandura, 1977, 1994, 2007). Self actions. Individuals who believe they will be effective will expect a p ositive outcome and those who believe they will be ineffective will expect a negative outcome (Bandura, 1977, 1994, 2007). Personal efficacy contributes to the acquisition of the knowledge and development of skills and can be influenced by the choice of a (Bandura, 2007). If the school counselor receives the training to implement a new program, but has a low self efficacy towards their ability to put that training into practice, they are unlikely to impleme nt the program. Without initial implementation, feedback cannot occur and ongoing change in practice will not be sustained. Determining Self Efficacy Self efficacy measures have ranged from broad overviews to domain specific assessments (Pajares, 1991). Unless the self efficacy instrument is specific to the expected outcome it is unlikely to predict the future behaviors (Bandura, 1986). The skills being tested and the skills
28 one hopes to assess should be similar skills (Pajares, 1991). There are a numbe r of instruments developed to determine the self efficacy of teachers (Brouwers & Tomic, 2004; Pajares, 1991; self efficacy of school counselors (Bodenhorn, 2005, 201 instrument development related to counseling self efficacy is found in chapter 3. Determining the self efficacy of the school counselor following a professional development will not only but also begin to ascertain the participant learning, and assess if there is the potential for the learning to be applied (Guskey, 2002). Research on Self Efficacy The investigation into self efficacy is a highly visible area of research in counseling and teaching (Paulsen & Betz, 2004). A substantial body of research can be found in the area of self efficacy and career counseling (Bodenhorn, Wolf, & Airen, 2008; Cantrell & Hughes, 2008; Guskey, 2002; Kosin, Steger, & Duncan, 2008; Larson & Daniels, 1 998; Paulsen & Betz, 2004; Yuen & Ma, 2008). The relationship between self efficacy and career decision making has been examined in the areas of self appraisal, collecting occupational information, setting goals, career planning, and problem solving (Paul sen & Betz, 2004). Numerous studies examining the self efficacy career decision making have identified the important relationship that exists between the expectation for success in a career and the decision to pursue that career (Betz & Voyten, 1997; Feld t & Woelfel, 2009; Kosine, Steger, & Duncan, 2008; Paulsen & Betz, 2004). The relationship between self efficacy and acquisition and application of skills are examined in the literature. Work done by Yuen and Ma (2008) targeted a group of in service teache rs studying in a graduate program at a university in Hong Kong. The study sought to learning technology. The findings
29 from 152 questionnaires indicated computer self efficacy was one of the tw o significant determinants for perceived ease of use (Yuen & Ma, 2008). Cantrell and Hughes (2008) examined self efficacy of teachers in their evaluation of an extended professional development on content literacy. The study examined the effects of a year long professional development with sixth and ninth grade teachers. Results indicated an efficacy toward the literacy teaching and concluded that teachers who had a higher self efficacy prior to the professional developmen t were more likely to implement what they had learned (Cantrell & Hughes, 2008). In Larson and Daniels (1998) review of the counseling self efficacy literature, it was concluded that higher counseling self efficacy was related to perseverance and the inc reased capacity to integrate evaluative feedback. They found mindfulness was a significant predictor of counseling self efficacy. In a national study of 860 American School Counseling Association members, school counselor self efficacy was shown to have a relationship to a greater awareness of achievement gap data (Bodenhorn, Wolfe, & Airen, 2010). Greason and Cashwell (2009) examined the predictive relationship between mindfulness and counseling self efficacy in 179 counseling students. Literature with a focus on self efficacy supports the concept that self efficacy effects the professional pathway and practice. It is evident that the level of self efficacy influences career decisions and the acquisition and implementation of professional skills. Profe ssional Development Professional development is a broad term used to describe learning to develop an formal course leading to a specific skill, an embedded on g oing reflection for continuous improvement, or mandated training on specific skills or procedures (Guskey, 2002). Many professional development models are grounded in social cognitive theory (Guskey, 2002; Fullen,
30 2001; Shaw et al., 1991). The concept th at the leaders of professional development can change the skills and knowledge learned in the professional development is a first step (Guskey, 2002). Thre e key areas of professional development were considered in designing the professional development unit used in this study: overall best practices in professional development, methods of evaluating professional development, and professional development in t he specific area of program evaluation. Each of these three areas will be considered before developing and evaluating the impact of a professional development unit on in service school Best Practices in Professiona l Development Although this study involves a professional development unit with school counselors, the theoretical basis of the study must draw on research on professional development done with teachers. This is because most of the literature surrounding professional development in education comes from the work done with teachers. Historically, the theoretical framework of Teacher change theory is often based on psychotherapeutic change models (Bechtel & about improved student academic achievement, is considered effective (Guskey, 2002). Guskey (2002) argues that a broader definition of student learning outcomes should include not only academic achievement, but also other measures such as behavior and attitude. These other measures of student learning outcomes could be measured with standardized assessments, attendance reco rds, behavior evaluations, and motivation evaluations. Guskey (2002) cites the study by Harootunian &Yargar (1980), which found that teachers
31 development. Scho ol counselors are similar to teachers in that they also define success in terms of student accomplishment, and have found value in linking their impact on students to their sense of mattering. (Rayle, 2006) zed, carefully structured and the needs of the targeted audience (Lipsey, 2008). In designing a successful professional development unit, it is helpful to rev iew relevant research on theories about why professional professional development models failed because they lacked teacher ownership. Without teacher buy in, change would not occur. Fullan (2001) emphasizes need for both support and pressure ome perturbed enough to embrace a change and must be active in the development of a new vision for change. This model suggests that the change occurs over time and is a repeated process for the teacher. Both of these theories explain change as the result beliefs. Once the attitude and beliefs of the teacher change, the behavioral change can occur. The more recent work of Guskey (2002) asserts that most professional development work fails because it does not pr operly address what motivates change and the process of change. Guskey (2002) asserts that motivation to change arises from both the belief that the professional development will enhance teaching and that it will be pragmatic (Guskey, 2002). Like other m odels, this model asserts that sustained change results from a change in beliefs and attitudes (Fullan, 2001; Shaw et al, 1991). However, this model of teacher change reverses the usual order of change. Whereas change in belief and attitudes is followed by implementation in other
32 models, in this model professional development is followed by change in teacher classroom practice, then change in student outcomes, and then finally change in teacher beliefs and attitudes (Guskey, 2002). This concept stems fro m the idea that changes in attitudes and beliefs are the result of the experience with the implementation of the skills and knowledge obtained in the professional development. For example, when teachers have implemented a change and can see results, they respond with a change in their attitudes and beliefs. In order to implement a successful professional development unit using this model, continual feedback needs to be provided to the participants (Guskey, 2002). One way to provide feedback so the teache r can monitor the change that is a result of the professional development unit is to implement a monitoring system and support for this system as a part of the professional development unit. A recent noteworthy contribution to professional development r esearch was the Institute Development Affects Student Achievement (2007). This report identified 1,359 studies that investigated the effect of professional development on student achievement. Of all the studies investigated, only nine studies met the What Works Clearinghouse (2002) evidence standards. From those nine studies, several important conclusions were drawn about what make professional development programs ef fective (Guskey & Yoon, 2009). The effective professional development models identified in the IES study have common development showed a positive and signifi cant effect on student achievement from professional greatest impact on student achievement, the average contact time was slightly over 53 hours
33 (range: 30 hours t o 83 hours), over a period of time from four months to 12 months (Yoon, et al., 2007). With the exception of one 4 week intensive summer training, all of the professional development models in the review were workshops that provided follow up (Yoon, et al., 2007). In all of the professional development models, the training went directly to the teacher; that is, none of the studies utilized the common practice of training one person to go back and then train other groups. This is often referred to as a Guskey, 2000). The professional development sessions were all conducted by the researcher or a trainer external to the school site. A review of the content description of the nine studies indicated that successful ac tivities focused on increasing content knowledge and pedagogic practice (Guskey & Yoon, 2009). Successful activities also contained well planned, purposeful learning (Guskey report, Reviewing The Evidence on How Teacher Professional Development Affects Student Achievement (2007), highlighting the importance of duration of training, follow up, delivery method and content, provide useful guidelines for what should be included i n an effective professional development model. Methods of Evaluating Professional Development In addition to providing useful guidelines for professional development research based on published research, the IES study demonstrates the importance of soun d evaluation of professional development (Yoon, et al., 2007). Many of the 1,350 quasi experimental and randomized controlled design studies in the study were excluded because they lacked baseline data. Because studies without baseline data were devalued by the IES study, the researcher was careful to include baseline data obtained through a pretest in this study.
34 Although scholars agree that professional development training programs benefit from comprehensive evaluation, there is evidence that many pro fessional development programs lack appropriate evaluation (Guskey, 2000; Trevisan, 2002b ; Trevisan, 2004). Many professional development programs are not evaluated (Trevisan, 2002b ; Trevisan, 2004). Some programs utilize tally sheets of attendance or n arrative descriptions of the programs as the only evaluation (Dahir & Stone, 2003; Guskey, 2000). Other evaluations of professional development are simple questionnaires asking the participants if they enjoyed the workshop or felt they had learned somethi ng from it. These self assessments lack the baseline data to assert any real evidence (Yoon et al., 2007). Although many self assessment methods used to evaluate professional development have lacked the depth needed to draw conclusions, self assessment has been studied to determine its & Nation, 2008; construct validity of self assessment of professional development workshops based on effect along with earlier works, which looked at correlation between retrospective self assessments and objective ratings provide potential resources in evaluating the effectiveness of professional development (Skeff et al., 1992; Pratt et, al, 2000). s (2000) text on evaluating professional development provides a broad theoretical framework of what he defines as the levels of evaluating professional development. He does not provide an evaluation model. Instead he offers a theoretical framework ground ed in the Joint Committee Evaluation Standards, a set of standards for evaluation. The Joint Committee is a group professional association with the common purpose to develop standards
35 for program evaluation. Since being formed in 1975 the committee has p ublished program evaluation standards (Yarbrough, D. B., Shulha, L. M., Hopson, R. K., and Caruthers, F. A., 2011). Guskey (2000) offers a perspective to consider when evaluating professional development. ) Participant Reaction; (2) Participant Learning; (3) Organizational Support and Change; (4) Participant Use of Knowledge and Skills; (5) Student Learning Outcomes. Participant Reaction, the first level in this theoretical framework, is the one most oft en investigated. While this first level is the most basic and the most easily performed, it remains a fundamental step in the evaluation of any professional development program. Participant reaction determines what perceptual information needs to be gath ered and helps decide the best method of data collection. Participant Learning, the second level in this theoretical framework, is frequently referred to in educational pedagogy as the assessment of the measurable learning objective (MLO). The MLO of the professional development would relate to the expected learning outcomes. professional development must be systematic and purposeful. The participants need to k now what is expected and how it will be measured. Organizational Support and Change, the third level in this framework, involves assessing the organizational support. This third level is not always addressed in the evaluation but is identified as a fac tor in teacher change (Fullen, 2001; Shaw et al, 1991; Guskey 2003b). The idea that support and pressure influence teacher change, makes this level of investigation a key aspect of any professional development program. An understanding of the system the training
36 will be implemented within provides the evaluator with a broad perspective and allows the evaluator to better assess the professional development program. Knowing how the professional development participant is going to be received when he or she tries something different at school can impact the type of ongoing support and follow up required from the professional development program. level in an assessm ent of a professional development program. An evaluation of the important component of gauging the success of the program. The participants use their knowledg e and skills learned, suggesting that they have mastered the content, and value it enough to use it in their teaching. Student Learning Outcomes, the fifth and final level in this framework, requires sufficient baseline data and frequent measurement of the The concept of one level being interrelated and required before the next level can be investigated is congruent program evaluation. This theoretical framework also functions within a context, considering the inputs and process along with products. The concept of considering inputs, proc ess and products context, input, process, and product. Professional Development in the Area of Program Evaluation Ghere, King, Stevahn, and Minnema (2006) develop ed taxonomy of program evaluator competencies. These competencies were crosswalks between The Program Evaluation Standards
37 (1994); the Essential Skills Series in Evaluation (1999); and the Guiding Principles for Evaluators (1995). They include six catego ries: technical aspects of evaluation; (c) situational analysis: understanding and attending to the contextual and political issues of an evaluation; (d) project managem ent: the nuts and bolts of expertise as well as needs for professional growth; and (f) interpersonal competence: the skills people need to work with diverse groups of (Ghere et al., 2006) Their study used these competencies as a starting point for the participating teachers to reflect upon. They strove to create awareness and develop knowledge of these competencies. Parti cipants first reflected on these competencies in the context of their environment and then self assessed their level and subsequently developed a professional development plan based on their reflections. The two hour session has been delivered to multiple audiences at a variety of times. Anecdotal data was collected to determine satisfaction and perception of the professional development but implementation and impact were not assessed. While there is a lack of evaluation, this study does provide the init ial development of some expected competencies, which were collected using a system of crosswalks between multiple established essential skills. at providing guid ance to faculty assigned to teach program evaluation coursework, and offers some insight on what should be included in a professional development unit on program evaluation. In general, practical experience for students has been recommended consistently i n the literature (Trevisan, 2004). The examples of hands on teaching strategies addressed in the
38 review include: simulation, role play, single course projects, and practicum experiences. In order to include any of these approaches support and supervision is required (Trevisan, 2004). Based on what is discussed above regarding best practices in professional development, the current program evaluation frameworks for professional development, and the previous knowledge garnered from studies on training indi viduals in program evaluation, there are some clear expectations and directions set for the development of affective professional development unit on program evaluation. Program Evaluation Before developing affective professional development unit focusi ng on program evaluation, there needs to be an understanding of the history of Program Evaluation in education. An understanding of the history can guide any decisions made about the best program evaluation model to be used by in service school counselors Historical Overview The early days of program evaluation in American Education began in 1845, when students in Boston were first tested (Madaus & Stufflebeam, 2000). The concept of using testing scores as a method for assessing student anticipated out comes has a rich history, starting in 1887 with Joseph Mayer Rice, who first began to formally evaluate students by comparing test scores between groups of students (Shadish, Cook, & Leviton, 1995). There have been periods of growth in program evaluation related to the historical context in which they occurred. One peak came with Ralph W. Tyler who began to move program evaluation beyond the teacher and local level and developed larger scale evaluations in the 1940s. The development of standardized test ing in the 1950s along with the large amount of federal funding that came with The National Defense Education Act (1958) brought about additional growth in program evaluation. The Elementary and Secondary Education Act (1965)
39 was quick to follow and with it came Title I evaluation requirements. These increases in mandated educational evaluation began to define program evaluation as a business and a profession (Shadish et al, 1995). The more recent reauthorization of the Elementary and Secondary Education Act, NCLB (2002) with explicit evidence based evaluation requirements solidified the significant role program evaluation has in American Education (Dollarhide & Lemberger, 2006). An awareness of the consistent presence of evaluation is important because of the mistaken impression held by some that that evaluation was born from NCLB. While the recent impact of the prescriptive nature of NCLB has influenced the prevalence of Educational evaluation, program evaluation in education is not new. No matter the political climate, programs will continue to be scrutinized through evaluation (Rossi, Lipsey, & Freeman, 2004). The same National Defense Education Act (1958) that brought growth to the standardized testing and evaluation industry gave birth to what in the 1980s became known as Developmental School Guidance programs (Gysbers & Henderson, 2000; Myrick, 2003). The school counseling profession has been and will continue to be influenced by some of the same educational policies that have impacted program ev aluation. Knowing the historical evolution and political context provides a greater understanding of the environment the school counselor will have in which to practice program evaluation. Program Evaluation Models One of the most widely referenced Progra m Evaluation Models is the CIPP Model. Developed by Daniel Stufflebeam, in 1966, the CIPP Model is now in its fifth version. The original focus of this model was aimed at including both the process and the product in the evaluation model (Stufflebeam, 20 04). The idea that the end product was not the only area worthy of evaluation and that the process leading up to the outcomes was also worthy of
40 evaluation represented a new direction in program evaluation. Later versions incorporated the idea that the Mo del is within a context and both the context and the Inputs, which already exist, should be considered in the whole approach to the evaluation. The most recent version of the CIPP Model has evolved to include more detail related to the product. There i s now the inclusion of the effectiveness of the product; not only is the impact on the target group evaluated, but an investigation into the quality and significance is also included (Stufflebeam, 1999). The product is also more closely examined to determ ine sustainability and transportability. These constant revisions in this model exemplify the tenets of program evaluation. As the model itself is evaluated within the context of the work it does, and those inputs taking place, the process and product ar e evaluated. Focused Evaluation, developed in 1978, is another model that addressed the importance of using the program evaluation (Patton 1997). Patton defines program out a potential broad range of topics for a Focused Evaluation moves from determining the value of a program in an evaluation to determining why the value is important and what wil l be done with the knowledge of this value (Patton, 1997). Systematic Approach to Program Evaluation Rossi, Lipsey, and Freeman (2004) provide a model that, like those of Patton (1997) and Stufflebeam (2004) is systematic. The Systematic Approach includ es five levels that are assessed when doing a program evaluation: (a) Need for the Program, (b) Program Theory, (c) Program Process, (d) Impact of the Program, and (e) Cost Analysis. Each level builds upon the next. When evaluating programs, there must b e some success at each of the levels in order to make a determination that the program was successful. The lessons learned from assessments at each level provide value to the users of the program evaluation.
41 The first level of the Systematic Approach to program evaluation is Needs Assessment. In order to evaluate the program, there must be an understanding of the problem and the population it was aimed at addressing. What was the need and does this program address that need? Program designers sometimes fail to align the program with the need. For example, a reading intervention, which is designed to teach initial sound fluency, may be utilized with a population of students that, upon investigation, did not need a reading program focused on fluency. The Needs Assessment might have found that this group of students had a high level of fluency and a low level of reading endurance. Their need was a program in reading endurance, not sound fluency. If the needs do not align with the intended outcomes of a p rogram, it is difficult to evaluate the program. In this example, since the students all began the program proficient in initial sound fluency, they all would have done well on the post test. If there had not been a pre test assessment, one might wrongly conclude that the program resulted in the improved fluency. The program would have incorrectly been deemed successful. Following the Needs Assessment, the second level of the Systematic Approach to program evaluation is Program Theory. Once there ha s been a determination of need and an intervention is deemed necessary there is an assessment of the Program Theory: Was the program based on sound theoretical knowledge? An example of a problem at this second level of program theory would be the implemen tation of a math program in all the schools even though research on the program indicated that it did not improve student math performance. delinquents with incarcera ted criminals with the hopes that this relationship will decrease the delinquent behavior amongst the juveniles. Research shows this approach does not work and in
42 fails the second level of program evaluation, Program Theory, it remains an ongoing program. The third level of the Systematic Approach to program evaluation in Rossi et al. (2004) is the Program Process. Was the program implemented in the way it was p rescribed? How was the implementation monitored? Programs are put into practice differently. When evaluating a program, it must be clear what aspects of the program were critical to its success. When a program is executed differently for one group then that different method of execution must be investigated. At this level, the question becomes, is it the program or the individual individual, one must then a sk, what is it about the individual that is making a difference in program effectiveness? Outcomes Assessment, the fourth level of the Systematic Approach to program evaluation, involves looking at the outcomes of a program to determine if the program successfully achieved its goals. Outcomes assessment sometimes is mistakenly the first step th at people performing program evaluation start take, before assessing the needs, theory, or process (Lipsey, 2008). Unfortunately, often times this level cannot be properly assessed because, in order to attribute any outcomes to a program, the program has t o have successfully considered the previous three levels. Working in isolation at this level makes it impossible to draw conclusions about effects of a program (Lipsey, 2008). The final level of the Systematic Approach to program evaluation is a Cost A nalysis, a measure of the efficacy (Rossi et. al., 2004). The examination of the cost and benefits of the program has the greatest impact on the future direction of a program (Rossi, et al., 2004). There are many technical and advanced applications to us e when conducting a cost analysis. However,
43 cost and benefit are not always monetary. They may come in the form of time or student performance. Broadly defining and understanding the value of cost and benefit assists in the measuring of the program effic acy. Each of these levels of the Systematic Approach to program evaluation is related to one another. Knowing how they relate and how they are used to evaluate a program can influence the planning of an effective program. Ideally program evaluation mod els are considered at the planning stage of a program (Patton, 1997; Rossi, et al. L, 2004; Stufflebeam, 2004). However, in practice, they are often used after a program has been implemented when someone is responsible for determining the program outcomes or cost effectiveness. Current State of Program Evaluation Among School Counselors American education is currently in an environment of standards based reform. With the reauthorization of the No Child Left Behind Act (NCLB) expected in the near future, it is unlikely there will be a reduction in the levels of accountability expected from those involved in the United States K 12 educational system (Manasevit, 2008). The accountability levels f work (Dollarhide & Lemberger, 2006). The call for increased levels of accountability is not new to the school counseling profession (Brown & Trusty, 2005; Gysbers, 2004; Gysbers & Henderson, 2001; Myrick, 2003; Wheeler & Loesch, 1981) However, the sens e of urgency for school counselors to practice program evaluation has been heightened by the external pressures of educational policy and national budget reductions (Dahir & Stone, 2009; Herr, 2001; House & Hayes, 2002; McGannon, Carey & Dimmitt, 2005). T he increased attention to school counselor accountability has generated an increase in the accountability efforts. The Fairchild & Zins (1986) study indicated that 45% of school counselors take part in some accountability. Six years later, Fairchild (199 3) repeated the study
44 and found that the percentage of school counselors taking part in accountability efforts increased from 45% to 65%. The Council of Accreditation of Counseling and Related Educational nts to be provided with an understanding of program evaluation while enrolled in a counselor preparation program (CACREP, 2009). Despite the value placed on program evaluation by CACREP, as evident in the 2009 survey of school counselor certification requirements found only 19 states and Washington, D.C. require some form of program evaluation skills as demonstrated by completed coursework. Within those 20 locations, only Washington, D.C. and Colorado mandate a level equal to that found in CACREP standards (Trevisan, 2000). ASCA has responded to the current standards based climate in education, by establishing clearly defined standards in the ASCA (2005) National Model which provide a benchmark for school co unselors to evaluate their programs (Dahir & Stone, 2009). Additional efforts to improve school counselor accountability presented in the literature a variety of methods to improve school counseling programs, provide evidence of effective practice, and improve the 2006; Whiston, 1996, 2002). The recent literature supporting school counselor accountability includes articles calling for and presenting methodology for increased research; articles presenting instruments school counselors could use to measure counselor effectiveness; and research that specifically promotes the use of program evaluation (Astramovich & Coker, 2007; Carey, Dimmitt, Hatch, Lapan, & Whist on, 2008; Dahir & Stone, 2009; Struder, Oberman & Womak, 2006). Research Methodology Dahir & Stone (2009) respond to the call for increased accountability by summarizing the method of
45 counting duties is being replaced with an outcomes based approach. School counselors are no longer maintaining timesheets of activities, but are now looking at the measurable outcomes that can be shared with others. Dahir & Stone use their own six step action research model, M.E.A.S.U.R.E. (mission, elements, analyze, stakeholders unit, results, and educate) (Dahir & evidence based research, Dahir & Stone (200 9) contend that action research begins to link school In the Dahir & Stone (2009) study counselors and other stakeholders selected one goal of the school improvement plan for the counseling program to focus upon. The study found that all but two of the research plans had a positive impact on the selected goal. This work reiterates and further supports some of the earlier works which promote action research as the means for increased empirical evidence in the fiel d (Rowell, 2006; Rowell, 2005; Whiston, 1996, 2003). The establishment of a National Panel for Evidence Based School Counseling has begun to address the need for empirical evidence based studies in school counseling (Carey et al., 2008). The joint effort on the part of ASCA and the Association for Counselor Education and (C arey et al., 2008, p. 196). A major contribution of this group has been the development of a research coding protocol. Strongly influenced by the What Works Clearinghouse Study Design and Implementation Assessment Device (Valentine & Cooper, 2003) and (2003) Procedures and Coding Manual, the protocol allows the panel to consistently review the literature with a uniform standard to determine the effectiveness of a given intervention. The
46 protocol has seven domains: Measurement, C omparison Groups, Statistical Analysis of Outcome Variables, Implementation Fidelity, Replication, Ecological Validity, and Persistence of Effect (National Panel for Evidence Based School Counseling, 2005). Each domain is assigned a value of strong, promi sing, or weak. Through the development of a research coding protocol, The National Panel for Evidence Based School Counseling has begun identifying school counseling practice that benefits student improvement in areas like student achievement (Carey & D immitt, 2006). The results of their meta analysis approach review of current practice allows for a consistent standard (Lipsey & Wilson, 2000). This standard not only contributes to the body of best practices, by working to meet the standards of NCLB (20 02) evidence based practice, it also provides researchers with a guide to develop their research methodology. Instrumentation In addition to guidelines for the practice of research, the literature offers school counselors instrumentation to better measur e their practice and more accurately align the outcomes with the intended effect. Struder, Oberman, and Womack (2006) emphasize the need for counselors to assess their programs and to describe an instrument design process. Their checklist for developing an assessment instrument offers a simple straightforward approach that a school counselor may use to demonstrate some measure of effect. The simple collection of results may not result in empirical data, but it can serve as a means for educating others ab out the role of the school counselor and their potential impact on student success (Struder, Oberman & Womak, 2006). Whiston and Aricak (2008), and Sink and Spencer (2007) investigated the psychometrics of instruments used to measure school counselor pr ograms. The School Counseling Program Evaluation Survey (SCoPES) is a 64 item measure with items aligned to each of the three
47 domains of in the National Standards for School Counseling Programs (Campbell & Dahir, 1997). The initial results of the psychom etric analysis of SCoPES showed promising results, but additional studies with comparison groups may offer greater evidence that SCoPES is an effective instrument. The My Class Inventory Short Form (MCI SF) is a self report measure used to assess elementa modified this instrument to collect teacher perceptions of the classroom. The survey has five dimensions: satisfaction, peer relationships, competitiveness, difficulty, and s chool counselor impact. The examination of reliability and factorial validity indicate that this quick and easy instrument could provide school counselors with a useful tool to assess a classroom guidance unit (Sink & Spencer, 2007). The ASCA National Model (2005) provides a Program Audit for school counselors to use when developing a new program or monitoring an established program. The ASCA Program Audit serves as a check sheet where counselors self report or obtain input from other stakeholders, su ch as an advisory group. The action items listed align with the standards defined in the ASCA National Model (2005). This needs assessment/ monitoring tool is a valuable resource (American School Counseling Association, 2010). The development of these checklists and other instruments are a piece of the process needed for school counselors to be more accountable for their practice. These instruments provide valuable program feedback. They can improve a dialogue between the school counselor and parents teachers, and principals about the impact of a school counselor. Program Evaluation In School Counseling An understanding of instrument development, and research methods are necessary for school counselors to effectively practice program evaluation ( Loesch & Ritchie, 2008).
48 Researchers in the field of school counseling, by expanding knowledge in the area of program evaluation, support efforts of school counselors to be accountable to themselves and others (Astramovich & Coker, 2007; Herr. 2001; House & Hayes, 2002; Loesch, 2000). In order to be Seeley, 1995 p. 10; Loesch & Ritchie, 2005). Counselors should utilize needs assessments; advisory committees and stakeho lder evaluations of counseling; and external assessments to assist in the determination of effective practice (Fairchild & Seeley, 1995; Lapan, 2001; Vacc, Rhyne Winkler, & Poidevant, 1993). While often used interchangeably, there has been a differentiatio n between the terms accountability and program evaluation (Astramovich & Coker, 2007; Herr. 2001; House & Hayes, 2002; Loesch, 2001). Accountability is the external pressure for the school counselor to be able to show the results of their programs (Gysber s, 2004; Herr, 2001; Isaacs, 2003). Program evaluation is the comprehensive approach used not only to demonstrate effectiveness, but also the process by which the school counselor can improve and develop the comprehensive guidance program (Astramovich & C oker, 2007; Loesch & Ritchie, 2008; Wheeler & Loesch, 1981). Through effective practice of program evaluation, school counselors can be accountable for their programs. The research aimed at the specific practice of program evaluation includes the devel opment of program evaluation models for counselors (Astramovich & Coker, 2007; Lapan, 2001; Schwitzer, 1997). The Bridge Model, A Program Evaluation Framework for Counselors provides counselors with a continuous cycle of program evaluation (Astramovich & Coker,
49 Input, Process and Product evaluation (Stufflebeam, 2004). The Bridge M odel emulates this direction by making connections between the needs assessment and the feedback from stakeholders. The feedback then influences planning, implementation, monitoring, and outcomes d and the feedback from stakeholders drives the strategic planning and the needs assessment in a continuous loop (Astramovich & Coker, 2007). As the profession of counseling continues to respond to the need for increased counselor accountability, there has been some investigation into the possible reasons why school counselors fail to practice accountability methods, specifically program evaluation. There are ce program evaluation (Trevisan, 2002a). One external factor is the organizational environment. For school counselors to effectively practice program evaluation, they must work in an ed or institutionalized based administrators, support and expect the practice of program evaluation, the school counselor has an external push to practice program eva luation. The lack of internal pull to practice program evaluation can be the result of fear of the unknown. Counselors often lack training to adequately prepare them to practice program evaluation (Astramovich & Coker, 2007; Lusky & Hayes, 2001). Whil e school counseling preparation programs sometimes require a research course, program evaluation theory and practice are not always emphasized (Astramovich & Coker, 2007). In addition to fear of attempting program evaluation because they lack the skills t o do so, some counselors may fear the results of a program evaluation (Isaac, 2003; Loesch & Richie, 2008). They may fear what
50 those results will indicate and how those results will be used (Trevisan, 2002). A poor understanding of how to perform program e valuation combined with an uncertainty as to what the results of such a program will be and how those results will be used can make the implementation of program evaluation a daunting task that many school counselors are reluctant to pursue (Trevisan, 2002 ). In order to address the lack of knowledge of program evaluation, Astramovich and Coker (2005) conducted a study where school counselors were given training to provide a foundation for the school counselors to evaluate their program. Based on ASCA Nati onal Model (2003) concepts, the group received a three hour workshop designed to help participants: understand the emphasis on accountability and program evaluation in the A SCA National Model; (c) define program evaluation;(d) understand the evaluation process including the role of needs assessment, program planning, program implementation, and assessing outcomes; and (e) plan to implement their own school counseling progr am evaluation. (Astramovich & Coker, 2005 p. 52) After training the first group of five counselors, the district director asked those five counselors to provide the training to an additional 23 school counselors. When this group was surveyed, many of their responses reiterated what had been previously s aid by much of the literature on program evaluation in the school counseling profession. Many (53.6%) of the counselors had no graduate coursework in program evaluation; 78.5% of them had not received professional development in program evaluation; 85.7% of them thought school counselors ought to make time for program evaluation; and 82.1% of the group agreed that outcome data on school counseling programs contributed to school accountability. After completion of the
51 training program, participants were wi lling to conduct evaluations and felt they understood why program evaluation is important, but many participants were unsure how to conduct a program evaluation. Follow up with the participants identified time constraints and lack of understanding as the barriers to conducting program evaluation. The Astramovich & Coker (2005) study took an important first step in meeting the professional development needs of school counselors. This study, along with the description of accountability methods, instrumen ts and checklist found in the literature, emphasizes the need to develop and effectively evaluate professional development for school counselors in the area of program evaluation. Summary of the Literature This review of the literature has provided a clo ser look at the overarching theoretical framework for this study, social cognitive theory including the self efficacy component and how self efficacy is determined. Best practices in professional development and the current methods used to evaluate profes sional development were additional areas of interest in the development of this study. Along with the historical perspective of program evaluation, several major evaluation models with specific emphasis on the systematic approach to program evaluation wer e reviewed above (Rossi et al., 2004). Lastly, the current state of program evaluation, in school counseling profession was reviewed. Chapter three will describes the variables, population sampling procedures, research design, measu rement procedures, p rofessional development unit, and data analysis of this study.
52 CHAPTER 3 METHODOLOGY Overview of the Study This study examined the impact of a professional development unit on program evaluation on in edge of program evaluation, (b) self efficacy level towards performing program evaluation, and (c) ability to develop and conduct program evaluation in the school setting. This chapter describes the variables, population sampling procedures, research des ign, measurement procedures, professional development unit, and data analysis of this study. Relevant Variables The independent variable for this study was participation in a four part professional development unit. The dependent variables were school cou efficacy, program evaluation competencies, and implementation of program evaluation. Data was collected for demographic information including: (1) gender, (2) age, (3) race, (4) ethnicity, (5) years of experience as a school counselor, and (6 ) graduation from a Council for Accreditation of Counseling and Related Educational Programs (CACREP) accredited program. Baseline data rating of their competencies in the area of program evaluation and self effic areas of program evaluation, self efficacy, and their application of program evaluation development and use following the four part professional development unit. Population The popula tion for this study included in service, elementary school counselors employed by the Pinellas County School District located on the west coast of Florida. In 2007, Florida, the fourth largest state, had an estimated population of 18,680,367 (Office of Ec onomic and
53 Demographic Research, 2009) with 67 public school districts. In the 2007 2008 school year, there were 2,653,377 students enrolled in state funded schools (Florida Department of Education, 2009). The racial breakdown of students in Florida as reported by Florida Department of Education is 45.9% white, 23.1% black, 24.7% Hispanic, 2.4% Asian, 3.6% multiracial, and 0.3% American Indian. Nearly half, (45.9%) of Florida students receive free or reduced lunch. English speakers of other languages m ake up 11.9%, 14.4% are classified as students with disabilities, and 4.9% are defined as gifted (Florida Department of Education, 2009). The Florida Department of Education provides state certification in the area of school counseling. Certification req counseling or counselor education which must include at least three semester hours or supervised counseling, and the specific course work is described in Florida Administrative Rule 6A 4.0181. Certification does include a requirement of three semester hours in student appraisal including administration and interpretation of standardized tests, but d oes not require coursework in program evaluation. Sampling Procedures In Pinellas County Schools, there are 77 elementary school counselors working in 80 elementary schools. Some counselors are assigned to multiple school sites, hence the difference in nu mber of participants and number of elementary schools. In 2008, Pinellas County was home to 923,066 residents. The school district has over 13,000 employees and is the largest single employer in Pinellas County. The Pinellas County School district is th e 25th largest school district in the nation and the seventh largest of the 67 districts in Florida. There are 104,717 students enrolled in kindergarten through twelfth grade; 44,955 are in the 80 elementary schools. The racial breakdown of students enro lled is 62.8% white, 19.2% black, 9.3% Hispanic,
54 3.7% Asian, 4.7% multiracial, and 0.3% American Indian. Pinellas County Schools have 40.8% percent of students receiving free or reduced lunch. English speakers of other languages make up 4.8%; 15.5% are c lassified as students with disabilities; and 5.4% are defined as gifted (Florida Department of Education, 2009). In the 2008 2009 school year, The Pinellas County School Board employed 230 school counselors with a counselor to student ratio of 1:459 in kin dergarten to grade twelve traditional schools. In the 2009 2010 school year, at the elementary level, the counselor to student ratio was 1:634. All Pinellas County School Counselors hold a Florida state certification in the area of school couns eling. Pri or to the first meeting the 2009 2010 school year, all 77 of the elementary school counselors were told that the professional development unit on program evaluation would be a part of their monthly guidance meetings, and they were invited to participate in the study via a district email from the guidance supervisor (Appendix A). In addition, they received information about the professional development unit on program evaluation from their principal (Appendix B). The researcher attended the first district wide guidance meeting in August and described the study to the school counselors and invited them to participate. The script of the invitation is shown in Appendix C. Over a period of six months, the school counselors met three times for a one to two hou r long professional development unit. School counselors were given the opportunity to volunteer to participate in the study as a supplemental activity to their district level meeting. Since participation in professional development was strongly encouraged all of the elementary school counselors received the training while attending the district meeting. Forty seven of the elementary counselors volunteered to participate in the study by completing the instruments. Of
55 the 47 who volunteered, 29 completed a ll three sessions of the professional development unit. The study group was made up of 29 counselors who received all three sessions of the professional development unit; volunteered to participate in the study; and completed the Persona l Data Sheet, the S CSES (Bodenhorn & Skaggs, 2005) the ECPE Self Assessment (Stevahn, et al., 2005) and the Implementation Survey. The researcher did not provide any monetary or time compensation for participation in the study. Research Design gain scores were assessed to determine the effectiveness of the intervention (Ravid, 2000). There was not a control group for this study. The district, which volu nteered to allow the researcher to conduct the study, stipulated all elementary school counselors were to have the opportunity to receive the professional development. The research design is shown in Table 3 1. This One Group Pretest Posttest Design invo lved administering the Personal Data Sheet (Appendix D), ECPE Self Assessment (Appendix E), and the School Counselor Self Efficacy Scale (SCSES) (Appendix F) to participants at the start of the professional development. Participants then took part in the professional development unit conducted over the period of six months. The ECPE Self Assessment (Ghere, King, Stevahn, & Minnema, 2006) and the SCSES (Bodenhorn & Skaggs, 2005) were re administered at the conclusion of the professional development unit. Two months after the completion of the professional development unit, participants were sent an Implementation Survey (Appendix G) via the interdepartmental mail system. In this Implementation Survey, participants were asked to self report if they had co nducted a program evaluation in their school. Counselors were specifically asked if they had conducted a needs assessment, selected a program to evaluate, assessed the purpose of the program, defined
56 anticipated outcomes, assessed the program implementati on, assessed the program outcomes, or shared their findings with others. Each question was linked to a step in the program evaluation process identified in The Systematic Approach to Program Evaluation (Rossi et al, 2007). Additionally, participants wer e asked three open ended questions: What did they like about the unit; what would they change; and how did this program influence their way of work? To insure confidentiality, participants were assigned a number. The instruments used in the study were la beled with the number assigned to the participant. The key connecting the names to the numbers was kept secure by the researcher. An electronic version of the key was kept on a password protected private computer. A paper version of the key was stored in a locked cabinet. When the study was completed and the data were analyzed, the key was destroyed. Names were not published in any report. Research Questions Questions addressed by the study are listed below. 1. Will school counselors have a greater understanding of program evaluation principles after taking part in a professional development unit? Will demographic factors influence performance on the Essential Competencies for Program Evaluators Self Assessme nt? efficacy towards efficacy towards program evaluation implementation as measured by the School Counselor Self Efficacy Scale ? 3. Will school counselors be able to develop a program evaluation in their own school setting as a result of having received professional development training?
57 Hypotheses As related to research questions one and two, several null hypotheses were investigated in this study: mean scores on the ECPE Self Assessment (Stevahn, et al., 2005). Ho1b: There will not be a significan t relationship between years of experience and the performance on the ECPE Self Assessment (Stevahn, et al., 2005). Ho1c: There will not be a significant relationship between graduation from a CACREP accredited program and the performance on the ECPE Se lf Assessment (Stevahn, et al., 2005). scores on the SCES (Bodenhorn & Skaggs, 2005). Ho2b: There will not be a significant relationship between years of experience and the performance on the SCSES (Bodenhorn & Skaggs, 2005). Ho2c: There will not be a significant relationship between graduation from a CACREP accredited program and the performance on the SCSES (Bodenhorn & Skaggs, 2005). Measurement Procedures/ I nstrumentation The personal data sheet, the ECPE Self Assessment (Stevahn, et al., 2005), SCES (Bodenhorn & Skaggs, 2005), and the Implementation Survey were the instruments used in this study. Personal Data Sheet Questions in the personal data sheet ask ed the participants their gender, race, ethnicity, and years of experience as a school counselor. The race and ethnicity codes were based on Florida
58 Department of Education reporting codes effective for the 2010 2011 academic year (Florida Department of E ducation, 2008). The participants had the option to choose yes or no for Hispanic or Latino ethnicity and yes or no for each of the Florida Department of Education race categories. Participants could choose multiple race categories. Participants who sele cted multiple race categories were recorded and analyzed as multiracial. Participants were also asked if they had graduated from a Council for Accreditation of Counseling and Related Educational Programs (CACREP) accredited program. Information collected from this data sheet was used to determine if there were relationships between the demographic factors and the dependent variables. Essential Competencies for Program Evaluators Self Assessment Stevahn, King, Ghere, and Minnema initially developed Essent ial Competencies for Program Evaluators (ECPE) in 2001. In 2005, they revised the original competencies and cross walked the competencies to the standards of the Joint Committee Program Evaluation Standards (1994), American Evaluation Association Guiding Principles (1995), and Canadian Evaluation Society Essential Skills Series (1999) (Stevahn, King, Ghere, & Minnema, 2005). The ECPE items were grouped into six major categories within the instrument: (1) professional practice, (2) systematic inquiry, (3) s ituational analysis, (4) project management, (5) reflective practice, and (6) interpersonal competence. The ECPE were later utilized as a self reporting instrument for school counselors to use, when reflecting on their professional development needs (Gher e, et al., 2006). The ECPE Self Assessment Instrument (Ghere, et al., 2006) directs respondents to select their level of competency with specific knowledge and skills. Respondents defined themselves as a tool for the participants to self reflect on their professional development needs.
59 Recently, the ECPE were used to measure health care workers program evaluation competencies (Fournier, Banza, Tourigny, & Dieudonne, 2009). Essential evaluation competencies were assessed by means of a questionnaire (pretest posttest, one year after) adapted from the work of Stevahn et al., 2005. (Fournier et al, 2009). The participants in the study were asked to rate thei r abilities to accomplish on a four compared to assess if the participants had learned and used the skills t aught in the program evaluation course (Fournier et al, 2009). The prior work done to develop the ECPE Self Assessment consisted of identifying the essential competencies (Stevahn, et al., 2005). The essential competencies were identified and used in a se lf assessment tool (Ghere, et al., 2006). The results of the self assessment tool were used to help identify needed areas of professional development. Like the work done by Fournier et al. (2009), the ECPE Self Assessment tool was adapted from the work do ne by Stevahn et al., 2005 and Ghere et al., 2006 for this study. The self assessment tool was formatted into an instrument asking participants to gauge their confidence in their ability for each of the competency items. A five point scale was used: 1 = n ot confident; 2 = slightly confident; 3 = moderately confident; 4 = generally confident and 5 = highly confident. Participants were asked to circle the number that best represented their response for each item. These directions mimic those found in the S CSE (Bodenhorn & Skaggs, 2005) also used by the participants. The ECPE Self Assessment The instrument was reviewed by a panel of experts and piloted with a small group for readability and ease of use (Fraenkel & Wallen, 2008).
60 School Counselor Self Effi cacy Scale Permission to use items from the School Counselor Self Efficacy Scale (SCSE) was obtained from the author, Nancy Bodenhorn by means of personal correspondence on March 29, 2009 ,w ho with Garry Skaggs developed the 43 item scale (Bodenhorn & Skag gs, 2005). The SCSE is based on the National Standards for School Counseling and the CACREP standards (2001). Bodenhorn and Skaggs (2005) conducted three studies when developing the SCSE. The first study consisted of a panel of experts who evaluated can didate items and determined which items should be included (Bodenhorn & Skaggs, 2005). The second study was conducted to analyze reliability, group differences, and item analysis. The sample group consisted of in service school counselors (Bodenhorn & Sk aggs, 2005). In this stage of the scale development, items were deleted based on insufficient response, lack of discrimination, or the excessive variability. The coefficient alpha for the scale score was 0.95 (Bodenhorn & Skaggs, 2005). The overall mean score for the items was a 4.21 on a five point scale with a standard deviation of 0.67. The Correlation matrix indicated 93% of the items responses correlated between 0.2 and 0.6 (Bodenhorn & Skaggs, 2005). Analysis of variance (ANOVA) with a 0.05 alpha level indicated no significant difference for groups working at different levels or in different settings. According to Bodenhorn & Skaggs (2005), there were differences based on gender, teaching experience, and number of years practicing. In the study, s elf efficacy was stronger among the female than the male participants F (1, 223) = 6.813, p < 0.05, R= 0.03. Those with teaching experience reported significantly stronger self efficacy than those without teaching experience F (1, 223) = 8.236, p < 0.01, R= 0.04. In addition, those who had practiced school counseling for 3 or more years reported significantly stronger self efficacy than those who had practiced less than 3 years F (1, 220)= 7.037 p < 0.01, R= 0.03 (Bodenhorn & Skaggs, 2005)
61 The third study (Bodenhorn & Skaggs, 2005) was c onducte It was composed of correlation studies between the SCSE and (1) Counseling self efficacy scales (COSE) (Larson, Suzuki, Gillespie, Potenza, Bechtel, & Toulouse, 1992), (2) Social Desirability Scale (SDS) (Crowne & Marlowe, 1960), (3) Six Trait Anxiety Inventory (STAI) (Spielberger, 1983), and (4) Tennessee Self Concept Scale, second edition (TSCS: 2) (Fitts & Warren, 1996) Validity was determined based on the COSE scores being posi tively correlated with SCSES (correlation= 0.41) and correlated with SDS (correlation=0.30) (Crowne & Marlowe, 1960). There was no correlation with the TSCS: 2 and the SCES. However, there was a significant difference in SCSE scores when comparing this gr oup of students with the group of practitioners sampled in Study 2, F (l, 340) = 29.89, p < 0.0001, R=0 .08. The authors concluded that the completion of the program and at least 1 year of experience resulted in an increase in school counseling self effic acy (Bodenhorn & Skaggs, 2005). For this study 18 items from SCSES were used to measure school counselor self efficacy. The items selected were specific school counselor activities related to program evaluation. The researcher selected the 18 items. To provide content validity, experts in the areas of both school counseling and program evaluation reviewed the items (Rand, 2000). Implementation Survey The researcher created the 16 item Implementation Survey using the steps in survey design and review ed the instrument using principles prescribed by the Tailored Design Method (Dillman, 2007). The first eight survey items were dichotomous items asking the participants if they had or had not developed and implemented a program evaluation since the profes sional development unit. The following five survey items (items 9 13) were five point Likert type the time, and ability to continue the practice of program ev aluation. The last three survey items
62 were open ended questions asking the participants what they liked about the professional development; what they would change; and how the professional development influenced their way of work. Distribution of this surv includes five points of contact (Dillman, 2007). Pre notice was given at the final session of the professional development unit. Delivery of the survey was initially done through the school district interoffice mail. Three reminder notices, with the survey as an attachment, were sent by means of email. Professional Development Unit Approach to Program Eval uation (Rossi et al., 2004). The first session of the three part participant training focused on the topic of Needs Assessment and included an introduction reinforcing the need for school counselors to conduct a program evaluation. The second and third top ics of the unit focused on the topics of Program Alignment and Assessing Program Design and Process. The Process included assessing program implementation and monitoring. The fourth topic and final of the three sessions, focused on discussions on the top ic of Sharing Outcomes. A figure illustrating the four topics within the unit can be seen in Figure 3 1. A complete outline for each of the topics of the professional development unit can be found in Appendix I. The delivery of the professional developm ent was based on Social Cognitive Theory. Social cognitive theory asserts an individual thoughts and feelings about a skill influence the learning and that learning is strongly influenced by observation (Bandura, 1986, 1994, 2007). An and feelings about a skill can be developed through modeling and comparing oneself to someone else (Bandura 1986, 1994; Zimmerman & Schunk, 2004).
63 Based on these concepts of Social Cognitive Theory, the four part unit was structured to include lecture, discussion, collaborative learning activities, and individual reflection. The initial session included time to review the informed consent (Appendix VIII); and time to administer the personal data sheet, ECPE Self Assessment (Stevahn, et al., 2005), and SC SES (Bodenhorn & Skaggs, 2005). The final session allocated time to complete the retest of the ECPE Self Assessment (Stevahn, et al., 2005), and SCES (Bodenhorn & Skaggs, 2005). The professional development unit provided participants with several experi ences: 1. Defining and conducting a needs assessment, 2. Developing questions to determine the purpose of a program, 3. Assessing if the purpose of a program aligns to a need, 4. Defining anticipated outcomes of a program, 5. Discussing methods for mo nitoring implementation of a program, 6. Providing examples of expected outcomes of a program, and 7. Demonstrating methods of sharing results of the implementation of a program. The researcher, a former school counselor and the current coordinator of ac countability in the district, provided the training model. The elements on needs assessment, program alignment, program implementation and program monitoring were previously presented to groups of instructional staff and administrators but were not previo usly presented to groups of school counselors. Feedback from these prior presentations to groups of instructional staff and administrators was considered in the development of the unit for school counselors. The professional development unit took place as a supplemental activity to the scheduled meetings between elementary school counselors and their district level guidance supervisor. These meetings were held monthly. At these monthly meetings, school counselors usually
64 received professional development along with relevant information related to their duties and functions. The study was introduced at the August meeting and the professional development took place in October, December and January. Those school counselors participating in this study were asked to volunteer to complete an Implementation Survey in March, two months after the final session of the professional development unit to determine if they had met the expectation of conducting a program evaluation. The reminder of the Implementation Su rvey was given to participants at the final session in January. The survey was delivered using the school district interoffice mail with directions for completion and was returned to the researcher using the same school district interoffice mail. Regardl ess of voluntary participation in the study group t he guidance supervisors expected all school counselors to conduct an evaluation of a program at their school as a result of the professional development unit. Data Analyses Data analysis was run using Sta tistical Package for Social Science (SPSS) Version 17. Descriptive statistics including frequencies for demographic information on gender, age, race, ethnicity, years of experience as a school counselor, and graduation from a CACREP accredited program wer e performed. The mean age and years of experience were calculated and are presented in the description of the results. Comparisons between the resultant sample and the population of school counselors were done. Analyses of the ECPE Self Assessment ( Stevahn, et al., 2005) and the SCES (Bodenhorn & Skaggs, 2005) instruments were done to examine the strength of the instruments. Results from both the ECPE Self Assessment (Stevahn, et al., 2005) and the SCES (Bodenhorn & Skaggs, 2005) pretest were used to determine the reliability and construct validity of both instruments (Rand, 2006). In order to examine the internal consistency of the items on the ECPE Self Assessment (Stevahn, et al., 2005) and the SCES (Bodenhorn & Skaggs, 2005), a
65 Cronbach reliability to use because of the scale type responses (Shavelson & Towne, 2002). Item total correlation analysis was completed with both pretests to exa mine the performance of each individual item (Ravid, 2000). An inter item correlation matrix was also completed to determine the relationship between items and to ensure an absence of consistently negative correlations between items that may indicate that an item is not measuring the same construct as the other items (Shavelson & Towne, 2002). An examination of both the ECPE Self Assessment (Stevahn, et al., 2005) and the SCES (Bodenhorn & Skaggs, 2005) pretest items for Skewness and Kurtosis was also done. Skewness and Kurtosis characterize the data to examine symmetry and normal distribution (Shavelson & Towne, 2002). greater understanding of program evaluation pri nciples after taking part in a professional test to determine if there were any significant differences between the pretest and posttest scores on the ECPE Self Assessment (Stevahn, et al., 20 05). The dependent means t test assumes a normal distribution of the data and equal variances (Shavelson, 1996). The test retest design insured that the samples are dependent. The pretest and posttest mean scores; maximum, minimum, and standard deviatio ns of the ECPE Self Assessment (Stevahn, et al., 2005) were calculated. participation in a CACREP accredited program be related to performance on the ECPE Self nswering this question required the use of an analysis of covariance (ANCOVA). The ANCOVA was used to determine if years of experience or participation in a CACREP accredited program was related to performance on the ECPE Self Assessment. The
66 posttest sc ores were used as the dependent variable. Participation in a CACREP program was the fixed factor and years of experience and pretest score were the covariates. The ANCOVA assumes the groups are independent of each other (Rand, 2006). In this study partici pants completed their own individual instruments and there was no threat to independence. ANCOVA also assumes the dependent variable is normally distributed within the population; also know n as normality (Rand, 2000). The normality test on the pretest an d posttest of the ECPE Self Assessment (Stevahn, et al., 2005) was the Kolmogorov Smirnov (K S). The Shapiro Wilk was also applied to assess normality because it is an appropriate application for a small sample (Glass & Hopkins, 1996). The second researc efficacy towards program evaluation test to determine if there were any significa nt differences between the pretest and posttest scores on the SCES (Bodenhorn & Skaggs, 2005). The dependent means t test assumes a normal distribution of the data and equal variances (Shavelson, 1996). The test retest design insures that the samples are dependent. The pretest and posttest mean scores, maximum, minimum, and standard deviations of the SCES (Bodenhorn & Skaggs, 2005) were calculated. participation in a CACREP efficacy analysis of covariance (ANCOVA). The ANCOVA was used to determine if years of experience or parti cipation in a CACREP accredited program was related to performance on the SCES (Bodenhorn & Skaggs, 2005). The posttest scores were used as the dependent variable.
67 Participation in a CACREP program was the fixed factor and years of experience and pretest s core were the covariates. The ANCOVA assumes the groups are independent of each other (Rand, 2006). There are several assumptions of the ANCOVA. They are: independence, normality and homogeneity of variances (Shavelson & Towne, 2002). Independence was achieved by having the participants complete their instruments independent of each other. Normality was assured by applying the K S test of normality (Glass & Hopkins, 1996) and, due to the small sample size, the Shapiro W ilk to the data. program evaluation in their own school setting as a result of the Professional Development e Implementation Survey. Descriptive statistics were calculated for the Implementation Survey. Furthermore, the frequency of responses was calculated. In addition, the maximum, minimum, and standard deviations for five Likert scale set of questions were calculated. The final three open ended questions were sorted into like groups and summarized. Summary In this third chapter, methodology for this study was explained. Relevant variables, population, sampling procedures, research design, measurement proc edures and instrumentation, the professional development unit, data analysis, and methodological limitations were presented. In Chapter 4, results will be described, including demographic characteristics, descriptive statistics, and inferential statistics
68 Table 3 1 Researc h d esign SCSES Y 1 X Y 2 ECPE Y 1 X Y 2 Implementation Survey X Y Note: Y indicates the measure. X indicates the Professional Development Unit Figure 3 1 Professional development u nit
69 CHAPTER 4 RESULTS The purpose of this study was to examine the impact a professional development unit in program evaluation on in evaluation, (b) self efficacy level towards performing program evaluation, and ( c) self perceived ability to develop and conduct program evaluation in the school setting. This chapter presents the demographic data of the sample population including: gender, age, race, and years of experience and graduation from a Council for Accredi tation of Counseling and Related Educational Programs (CACREP) accredited program. This chapter also presents t he pretest posttest results of the Essential Competencies for Program Evaluators (ECPE) Self Assessment ( Stevahn, King, Ghere, & Minnema, 2005) (Appendix E ). The ECPE (Stevahn, et al., 2005) consisted of 60 items and used a five point Likert type scale to measure perceived self confidence level with evaluation competencies Pretest posttest results from the 18 icacy Scale (SCSES) (Bodenhorn & Skaggs, 2005) (Appendix F) are presented. The SCSES (Bodenhorn & Skaggs, 2005) measured perceived self confidence toward school counselor skills using a five point Likert type scale. The analysis of the difference between the pretest posttest results and an analysis of the relationship between years of experience and graduation from a CACREP accredited program and the pretest posttest results are also presented. Lastly, the themes and feedback gathered from the Implementa tion Survey are presented. Demographic Characteristics Seventy seven elementary school counselors participated in the first of the three professional development trainings and 47 agreed to participate in the study Of the 47 volunteers study participants 29 comple ted all three professional development trainings and both
70 the pre test and posttest. Forty four of the 47 volunteers completed two of the three trainings. However, the third training date was rescheduled with less than two weeks notice. As a r esult of the rescheduled meeting time, 29 counselors participated in all three trainings. The pretest data from the initial 47 participants was used in the descriptive analysis, of the SCSES (Bodenhorn & Skaggs, 2005) and ECPE Self Assessment (Stevahn, et al., 2005) The study group comprised t he 29 counselors who received all three sessions of professional development unit; volunteered to participate in the study; and completed the Persona l Data Sheet, the SCSES (Bodenhorn & Skaggs, 2005) the ECPE Self A ssessment (Stevahn, et al., 2005) and the Implementation Survey. Demographic and inferential statistics were based on the study group of 29 counselors. Demographic characteristics of the elementary school counselor study group participants collected dur ing the first meeting with the m via the P ersonal Data Sheet (Appendix D), are displayed in Table 4 1. Of the 29 study subjects, 28 (96.6%) were female and one was male (3.4%). The participants had the option to choose yes or no for Hispanic or Latino ethni city and yes or no for each of the Florida Department of Education race categories. Participants could choose multiple race categories. In the area of ethnicity there was one participant who identified himself or herself as Hispanic (3.4%). In the category of race, 28 participants identified themselves as white (96.6%); one identified themselves as black (3.4%) Twelve of the 29 participants (41.4%) were graduates of a CACREP accredited program 14 (48.3%) were not. Three (10.3%) were unsure The mean age of the participants was 44.5 years with a standard deviation of 10.5 years and a range of 27 years to 61 years T he mean number of years of experience was 11 with a standard deviation of 9.9 with a range between one year and 35 years
71 De scriptive Statistics The Statistical Package for Social Sciences (SPSS) version 17 was used for the data summaries and analysis. An alpha level of p = .05 was chosen for all of the analys e s. A total of 40 participants complete d the ECPE Self Assessment (S tevahn, et al., 2005) and 43 completed the SCSES (Bodenhorn & Skaggs, 2005) p retests. Individual item means were calculated for the ECPE Self Assessment (Stevahn, et al., 2005) pretest. The item mean on the five point Likert type scale of the ECPE Self A ssessment (Stevahn, et al., 2005) p retest was 2.89 (N=40). Complete detail of item statistics are shown in Table 4 2. On the five point scale, the minimum item mean was 1.8 and the maximum item mean was a 4.4 with a variance between means of 0.35. The i tem variance was calculated to investigate the quality of the individual items. Variance had a mean of 1.12 with a minimum of 0.7 and a maximum of 2.04 with a 0.05 variance. An item summary, including the variance is shown in Table 4 3. In order to exa in a 0.98. Examination of individual items indicated none of the individual items strongly Alpha test for internal reliability are presented in Table 4 4. Further investigation into the stability of the items was done through the examination of items and their correlation with the total. The item total correlation revealed a range from 0. 449 to 0.815. This wide range indicated an inconsistent variation between the individual items and their correlation with the total. An inter item correlation matrix was also conducted to determine if any negative correlations between items existed. No n egative correlations were found.
72 Individual item means were calculated for the SCSES (Bodenhorn & Skaggs, 2005). The SCSES (Bodenhorn & Skaggs, 2005) had a mean of 3.3 (N = 43) on a 5 point scale measuring self confidence with counseling skills. Complet e detail of item means are shown in Table 4 5. The minimum mean was a 2.7 and maximum w as 4.3 with a range of 1.6. The variance between the mean was 0 .256. In order to examine the item strength item variances were examined. Item v ariances had a mean of 0 .906 with a minimum of 0 .613 and a maximum of 1.36 with a range of 0 .749 and a variance of 0 .027. Complete i tem summary statistics can be seen in Table 4 6. In order to examine the internal reliability a (Boden horn & Skaggs, 2005). The SCSE indicated an Alpha of 0 .943. An examination of items and their correlation with the total was done and item total correlation ranged from a 0 .479 to 0.836. No item had a strong impact on the Alpha when deleted. Details of the individual item correlations are shown in Table 4 7. An i n ter item correlation matrix was completed and all items were positively correlated. The test retest research design implies normality (Rand, 2000). Skewness and Kurtosis was examined. Skewn ess and Kurtosis characterize the data to examine symmetry and normal distribution (Shavelson & Towne, 2002). ECPE had a S kewness of 0 .137 and Kurtosis of 0 .334; SCSE Skewness was 0 .653 and Kurtosis of 1.65. Inferential Statistics In order to address research questions and the null hypotheses investigated in this study, analysis of the data obtained from the instruments completed by the 29 participants was done. The first instrument was the ECPE (Stevahn, et al., 2005), which consi sted of 60 items and used a five point Likert type scale. The second instrument was SCSES (Bodenhorn & Skaggs, 2005), which consisted of 18 items and also used a five point Likert type scale. The ECPE (Stevahn, et al., 2005), and the SCSES (Bodenhorn & Skaggs, 2005) pretests were administered to participants during the first professional development session. The posttests for both ECPE
73 (Stevahn, et al., 2005) and SCSES (Bodenhorn & Skaggs, 2005) were administered at the fourth and final session of the p rofessional development unit. The calculation of differences between pretest and posttest was based on those results collected from the 29 participants. Results of the pretest were linked to the posttest results and tied to the initial personal data coll ection sheet, which collected the demographic information of participants. The Implementation Survey was sent to participants via district interoffice mail two months after the fourth and final session of the professional development unit. It included se ven type scale questions to determine level of agreement; and three open ended questions. Research Question One unselors have a greater understanding of program evaluation principles after taking part in a professional development unit? Will years of experience or participation in a CACREP accredited program be related to performance on the ECPE Self hree null hypotheses were determined to respond to this question. The first null hypothesis proposed was: mean scores on the ECPE Self Assessment (Stevahn, et al. 2005). A dependent sample t test was conducted to address this hypothesis. The ECPE (Stevahn, et al., 2005) had a five point scale. The pretest mean score for the ECPE (Stevahn, et al., 2005) was 165.93 and the posttest mean score was 206.59. Details including standard deviation, maximum, and minimum are shown in Table 4 8. The t test was applied using the pretest and posttest means. The t = 4.6 (df =28). The calculated confidence interval had a lower bound of 58.79 and an upperbound of 22.532. The large size of the range could be attributed to the small
74 sample size (Shavelson, 1996). Results of the t test are shown in Table 4 9. The first Null hypothesis was rejected because there was a significant difference between the pretest posttest means (p = .009). The second and third null hypotheses were: Ho1b: There will not be a significant relationship between years of experience and the performance on the ECPE Self Assessment (Stevahn, et al., 2005). Ho1c: There will not be a significant relationship between graduation from a CACREP accredited program and the performance on the ECPE Self Assessment (Stevahn, et al., 2005). One analysis was done to address both of the second and third hypotheses. An analysis of covariance was done using the posttest scores of the ECPE (Stevahn, et al., 2005) as the dependent variable with graduation from a CACREP program as the fixed factor and years of experience and pretest scores of the ECPE (Stevahn, et al., 2005) as the covariates to determine if there is a significant relationship. The effect of years of experiences was not significant, F(1, 24) = 2.057, p = .164 nor was graduation from a CACREP program, F(2, 24) = .005, p = .947. Results from the ANCOVA are show n in Table 4 12. There was no statistical significance indicating either years of experience or graduation from a CACREP program affected the posttest. Therefore the null hypotheses were not rejected. Research Question Two The second research question a efficacy towards program evaluation implementation? Will years of experience or participation in a CACREP accredited program be related to school elf The fourth null hypothesis proposed in this study was:
75 scores on the SCSES (Bodenhorn & Skaggs, 2005). An ind ependent samples t test was conducted to addres s this hypothesis. The pretest mean score for the SCSES (Bodenhorn & Skaggs, 2005) was 57.83 and the posttest mean was 69.10 on a five point scale. The t test was applied to the pretest posttest mean differe nce (t = 5.52; df = 28). The calculated confidence interval ranged from a lower level of 15.46 to an upper level of 7.09. Results from the t test are shown in Table 4 10. There was a statistical difference between pretest posttest scores on the S CES (p = .002). Therefore, the null hypothesis was rejected. The fifth and sixth null hypotheses proposed in this study were: Ho2b: There will not be a significant relationship between years of experience and the performance on the SCSES (Bodenhorn & Sk aggs, 2005). Ho2c: There will not be a significant relationship between graduation from a CACREP accredited program and the performance on the SCSES (Bodenhorn & Skaggs, 2005). An analysis of covariance (ANCOVA) was done to address both hypotheses. An analysis of covariance was done using the posttest scores of the SCSES (Bodenhorn & Skaggs, 2005) as the dependent variable with graduation from a CACREP program as the fixed factor and years of experience and pretest scores of the SCSES (Bodenhorn & Skagg s, 2005) as the covariates to determine if there is a significant relationship. Th e effect of years of experience was not significant, F(1, 24) = .467, p = .501 nor was graduation from a CACREP program, F(2, 24) = .382, p = .687. Results from the ANCOVA ar e shown in Table 4 13. There was no statistical significance in the analyses; therefore the null hypotheses were not rejected.
76 Research Question Three The third research question asked in this study was: Will school counselors develop a program evaluati on in their own school setting as a result of the Professional Development Unit? The Implementation Survey was administered to determine if school counselors were able to develop an evaluation. Twenty of the 29 study participants responded to the 16 item Implementation Survey. When asked to respond to I have conducted a needs assessment 20 (70%) reported yes. Nineteen of 20 (95%) indicated they had selected a guidance program to evaluate Seventeen (85%) said they had assessed the theory and purpose of the program. Sixteen (80%) assessed how the program purpose aligns with the needs and defined the anticipated outcomes. Nineteen (95%) had assessed the implementation of the program they chose to evaluate, and fifteen ( 75%) had assessed the outcomes of the program When asked with Council, Pri ncipal, Parents, School Leadership Team, Teachers, and Other. Twelve said they had shared results with principals; eleven with other counselors; ten with teachers; four with parents; four with school leadership teams; none had shared with school advisory c ouncils and none selected other. Participants were also asked five Likert type questions. They were directed to indicate their level of agreement: Strongly Agree, Agree, Neutral, Disagree or Strongly Disagree. During data input, the researcher assigned evaluation of a program in their guidance program, the mean of the responses was a 4.37. The mean s core was 4.32 when participants were asked if they felt supported by other school counselors. The mean score was a 4.21 when participants were asked if the other staff in the
77 building supported them. The lowest mean score was 3.74 when participants were a sked if they had time to complete this process. When asked if participants will continue to practice program evaluation the mean response was a 4.42. Complete results, including maximum, minimum, and standard deviations for responses to these five questi ons are shown in Table 4 14. In addition to the seven dichotomous yes / no questions and the five Likert type questions, the survey asked participants what they liked about the professional development training, what they would change, and how the profess ional development influenced their way of work. The comments related to what participants liked, and the comments related to how the professional development experience influenced their way of work had similar themes. The first theme was related to an i mproved structure by which counselors could reflect on It provides me with a more effective way of fine It allowed me to reflect on th Allowed us to think about how we evaluate our The second theme centered on comments related to improved understanding of data. Ther e Comments related to how the professional development could be improved were la rgely related to time. There were two comments about the need to begin the professional development on program evaluation at the very start of the school year as well as the need for more time to include things like d irect guided and hands on practice in the training. A complete list of the comments (N=8) can be found in Table 4 15.
78 Summary In this chapter, results described included demographic characteristics, the descriptive statistics, and inferential statistics, and presented the results of the Implem entation Survey. In the fifth and final chapter an overview of the study conclusions, limitations, implications, and recommendations will be presented.
79 Table 4 haracteristics Variable Percent (N) Gender Male 3.4 (1) Female 96.6 (28) Race Black 3.4 (1) White 96.6 (28) Ethnicity Hispanic 3.4 (1) Non Hispanic 96.6 (28) Graduate from a CACREP accredited program Yes 41.4 (12) No 48.3 (14) Unsure 10.3 (3) Age Range: 27 to 61 years Mean: 44. 5 years s.d.: 10.5 Years of Experience Range: 1 to 35 years Mean: 11.2 years s.d.: 9.9 Table 4 2 Item s tatistics ECPE Item Mean Std. Deviation N 1 4.40 .955 40 2 3.30 1.137 40 3 2.95 .986 40 4 3.45 1.011 40 5 2.43 1.059 40 6 3.30 1.043 40 7 2.65 1.051 40 8 2.63 1.055 40 9 2.85 .949 40
80 Table 4 2 Item s tatistics ECPE continued Item Mean Std. Deviation N 10 2.65 1.001 40 11 2.45 .959 40 12 3.75 .981 40 13 3.08 .917 40 14 3.28 1.062 40 15 3.30 .939 40 16 2.60 1.429 40 17 2.68 1.047 40 18 2.55 1.061 40 19 2.65 1.027 40 20 3.18 1.059 40 21 3.38 1.030 40 22 2.23 1.025 40 23 1.85 .834 40 24 2.85 1.075 40 25 2.25 .954 40 26 2.83 1.107 40 27 2.28 .960 40 28 2.45 .986 40 29 2.50 1.013 40 30 2.88 1.042 40 31 2.68 1.071 40 32 1.83 .958 40 33 3.03 1.000 40 34 2.55 1.154 40 35 2.43 1.035 40 36 2.55 1.011 40 37 2.55 1.085 40 38 2.58 .903 40 39 2.85 1.231 40 40 3.33 1.228 40 41 4.03 1.025 40 42 3.13 .992 40 43 4.28 .933 40 44 3.83 1.196 40 45 3.28 1.012 40 46 2.45 1.154 40 47 2.28 1.109 40
81 Table 4 2 Item s tatistics ECPE continued Item Mean Std. Deviation N 48 1.93 1.095 40 49 2.18 1.196 40 50 2.43 1.217 40 51 3.38 1.192 40 52 3.08 1.163 40 53 3.68 1.118 40 54 3.25 1.171 40 55 1.93 .997 40 56 1.95 .932 40 57 2.00 .987 40 58 2.53 1.062 40 59 4.40 .955 40 60 3.30 1.137 40 Table 4 3 Summary item s tatistics ECPE Mean Minimum Maximum Range Maximum/ Minimum Variance N Item Means 2.819 1.825 4.400 2.575 2.411 .350 58 Item Variances 1.116 .695 2.041 1.346 2.937 .050 58 Table 4 4 Item total s tatistics ECPE Item Number Scale Mean if Item Deleted Scale Variance if Item Deleted Corrected Item Total Correlation Squared Multiple Correlation Cronbach's Alpha if Item Deleted 1 159.08 1753.404 .539 .981 2 160.18 1730.199 .697 .981 3 160.53 1739.640 .691 .981 4 160.03 1734.076 .740 .980 5 161.05 1742.613 .607 .981 6 160.18 1733.635 .722 .981 7 160.83 1739.840 .644 .981 8 160.85 1741.208 .626 .981 9 160.63 1743.574 .668 .981 10 160.83 1736.712 .716 .981 11 161.03 1748.230 .602 .981 12 159.73 1741.897 .667 .981
82 Table 4 4 Item total s tatistics ECPE continued Item Number Scale Mean if Item Deleted Scale Variance if Item Deleted Corrected Item Total Correlation Squared Multiple Correlation Cronbach's Alpha if Item Deleted 13 160.40 1756.913 .516 .981 14 160.20 1736.933 .671 .981 15 160.18 1738.456 .742 .980 16 160.88 1736.471 .495 .981 17 160.80 1730.421 .757 .980 18 160.93 1725.199 .807 .980 19 160.83 1728.558 .795 .980 20 160.30 1737.549 .665 .981 21 160.10 1736.144 .702 .981 22 161.25 1732.603 .747 .980 23 161.63 1748.958 .685 .981 24 160.63 1738.651 .643 .981 25 161.23 1727.256 .873 .980 26 160.65 1727.977 .742 .980 27 161.20 1742.831 .669 .981 28 161.03 1739.820 .689 .981 29 160.98 1727.820 .815 .980 30 160.60 1733.118 .729 .980 31 160.80 1738.164 .651 .981 32 161.65 1741.772 .685 .981 33 160.45 1742.869 .642 .981 34 160.93 1725.199 .740 .980 35 161.05 1730.203 .768 .980 36 160.93 1729.507 .795 .980 37 160.93 1725.712 .783 .980 38 160.90 1732.451 .854 .980 39 160.63 1731.215 .632 .981 40 160.15 1729.105 .655 .981 41 159.45 1745.690 .592 .981 42 160.35 1740.336 .678 .981 43 159.20 1755.549 .524 .981 44 159.65 1751.054 .449 .981 45 160.20 1749.446 .555 .981 46 161.03 1724.025 .752 .980 47 161.20 1735.087 .661 .981 48 161.55 1740.151 .614 .981
83 Table 4 4 Item total s tatistics ECPE continued Item Number Scale Mean if Item Deleted Scale Variance if Item Deleted Corrected Item Total Correlation Squared Multiple Correlation Cronbach's Alpha if Item Deleted 49 161.30 1721.087 .755 .980 50 161.05 1724.562 .706 .981 51 160.10 1730.810 .658 .981 52 160.40 1735.579 .624 .981 53 159.80 1740.574 .596 .981 54 160.23 1724.487 .736 .980 55 161.55 1731.126 .787 .980 56 161.53 1739.640 .732 .981 57 161.48 1740.102 .684 .981 58 160.95 1734.869 .695 .981 59 159.08 1753.404 .539 .981 60 160.18 1730.199 .697 .981 Table 4 5 SCES item s tatistics Item Number Mean s.d. N 1 3.79 .871 42 2 4.14 .783 42 3 3.24 .932 42 4 3.69 .924 42 5 2.81 .917 42 6 4.29 .944 42 7 3.19 .890 42 8 3.95 .936 42 9 2.67 .928 42 10 2.69 .897 42 11 3.26 .964 42 12 2.83 1.167 42 13 3.07 .947 42 14 3.07 1.091 42 15 2.79 .951 42 16 3.07 .921 42 17 2.83 .986 42 18 3.21 1.025 42
84 Table 4 6 Summary item s tatistics SCES Mean Minimum Maximum Range Maximum/ Minimum Variance N of Items Item Means 3.255 2.667 4.286 1.619 1.607 .256 18 Item Variances .906 .613 1.362 .749 2.221 .027 18 Table 4 7 Item total s tatistics SCES Item Number Scale Mean if Item Deleted Scale Variance if Item Deleted Corrected Item Total Correlation Squared Multiple Correlation Cronbach's Alpha if Item Deleted 1 54.81 137.768 .507 .652 .943 2 54.45 137.181 .604 .750 .941 3 55.36 135.943 .555 .760 .942 4 54.90 133.503 .681 .743 .939 5 55.79 134.319 .646 .623 .940 6 54.31 134.658 .608 .739 .941 7 55.40 134.344 .666 .762 .940 8 54.64 137.503 .479 .719 .943 9 55.93 134.068 .649 .690 .940 10 55.90 131.357 .813 .842 .937 11 55.33 129.593 .836 .789 .936 12 55.76 128.820 .706 .735 .939 13 55.52 131.573 .755 .824 .938 14 55.52 131.914 .630 .764 .941 15 55.81 132.548 .705 .790 .939 16 55.52 131.768 .769 .879 .938 17 55.76 130.966 .751 .833 .938 18 55.38 130.583 .736 .758 .938 Table 4 8 Paired samples statistics pretest postt est ECPE Instrument Mean N Std. Deviation Std. Error Mean ECPE Pre test 165.93 29 37.195 6.907 ECPE Post test 206.59 29 52.198 9.693
85 Table 4 9 Paired samples t est (T Test) ECPE Paired Differences t d.f. Sig (2 tail) 95% Confidence Interval of the Difference Mean Std. Deviation Std. Error Mean Lower Upper Pre test Post test 40.655 47.644 8.847 58.778 22.532 4.595 28 .000 Table 4 10 Paired samples statistics p re test postt est SCSE Instrument Mean N Std. Deviation Std. Error Mean SCSE Pre test 57.83 29 9.067 1.684 SCSE Post test 69.10 29 12.982 2.411 Table 4 11 Paired samples t est (T Test) SCSE Paired Differences t d.f. Sig (2 tail) 95% Confidence Interval of the Difference Mean Std. Deviation Std. Error Mean Lower Upper Pre test Post test 11.276 11.003 2.043 15.461 7.091 5.519 28 .000 Table 4 12 Test between subject e ffects (ANCOVA) posttest d ifference ECPE Source Type III Sum of Squares df Mean Square F Sig. Corrected Model 22322.093a 4 5580.523 2.482 .071 Intercept 13604.763 1 13604.763 6.050 .021
86 Table 4 12 Test between subject e ffects (ANCOVA) pretest posttest d ifference ECPE continued Source Type III Sum of Squares df Mean Square F Sig. experience 4625.065 1 4625.065 2.057 .164 E CPE 19504.056 1 19504.056 8.673 .007 CACREP 246.314 2 123.157 .055 .947 Error 53968.941 24 2248.706 Total 1313949.000 29 Corrected Total 76291.034 28 a. R Squared = .293 (Adjusted R Squared = .175) Table 4 13 Test between subject e ffects (ANCOVA) pretest posttest d ifference SCSE Source Type III Sum of Squares df Mean Square F Sig. Corrected Model 1542.981a 4 385.745 2.915 .043 Intercept 399.356 1 399.356 3.018 .095 experience 61.818 1 61.818 .467 .501 SCSE 1149.317 1 1149.317 8.686 .007 CACREP 101.033 2 50.516 .382 .687 Error 3175.708 24 132.321 Total 143202.000 29 Corrected Total 4718.690 28 a. R Squared = .327 (Adjusted R Squared = .215) Table 4 14 Implementation survey results i tems 8 13 Item Number Mean sd Maximum Minimum 9 4.37 1.012 5 1 10 4.21 .787 5 2 11 4.32 .749 5 3 12 3.74 .933 5 2 13 4.42 .507 5 4 Table 4 15 Implementation s urvey c omments What did you like about the professional development? It allowed me to reflect on the +and of the program. I was able to collaborate with others in my school to get feedback The support and help with data collection
87 A new way to see my data It gave me direction and a starting point It provided a lot of specific information and allowed us to think about how we evaluate our programs It pr ovides me with a more effective way of fine tuning my guidance program components Made things clearer Professional development in this area is always helpful I had already received state DOE training on MEASURE and adapted it to my IPDP so had already planned for the year what to do What would you change about the professional development? I was not always clear how to respond to the questions being asked Not hing We need more training earlier in the year and possibly tie it to the inquires that schools do Allow more time for hands on and Q&A Perhaps Nicole could follow up and work with small groups so we could implement our plan with her direct guidance. Some practice. It should be offered in the beginning of the year Explanation of the new template we are using this year More time to work on the concept How will this professional development influence your way of work? I will try to conduct a needs assessment next year Be more reflective, use data more to drive guidance program Always keep in mind the purpose and how my program purpose aligns with the needs, data, data, data and more data I will continue to practice program evaluation I will try this approach with my program; unfortunately there is little time and support to implement it effectively It will guide my progr am creation, implementation, and assessment It will make delivery of services more meaningful More conscientious and deliberate in making data work for me
88 CHAPTER 5 DISCUSSION The purpose of this study was to examine the impact of a professional development unit on program evaluation on in evaluation, (b) self efficacy level towards performing program evaluation, and (c) ability to develop and conduct program evaluation i n the school setting. This chapter is divided into four sections. The first section provides an overview of the study. The second section delineates the implications of the study for the practice of counseling. The third section discusses limitations of t he study. The fourth and final section recommends potential avenues of future research. Overview of the Study This study provided a four session professional development unit on program evaluation to a group of elementary school counselors in Pinellas County, Florida. The 29 participants took part in all four professional development sessions and they completed: a personal data sheet; a pretest and posttest of the Essential Competencies for Program Evaluators Self Assessment (ECPE); a pretest and postt est of the School Counselors Self Efficacy Scale (SCSES); and an implementation survey. The participants were primarily white (96.6%) and female (96.6%) ranging in age from 27 to 61. All had at least one years experience working as a school counselor. Re search Question One The first research question investigated the impact the professional development unit had relationship between years of experience or participation i n a CACREP accredited program and
89 Assessment (Stevahn, et al., 2005), which measured self confidence with their program evaluation skills. This finding demonstrates that participants in the professional development unit increased their self confidence towards their knowledge of program evaluation. On the other hand, ther pretest posttest mean scores on the EPCE Self Assessment (Stevahn, et al., 2005) based on years p retest posttest mean scores on the EPCE Self Assessment (Stevahn, et al., 2005) based on graduation from a CACREP accredited program. Therefore, neither years or experience, nor graduation from a CACREP program were differentiating factors in the increase of mean scores from pretest to posttest. Research Question Two The second research question investigated the impact the professional development unit efficacy towards program evaluation skills. Additionally, it examined t he relationship between years of experience or participation in a CACREP accredited efficacy towards program evaluation skills. For this study 18 items from SCSES were used to measure school counselor self efficacy. The items selected were specific school counselor activities related to program evaluation. The provided preliminary indication the 18 item instrument was reliable A statistically significant scores on the SCSES (Bodenhorn & Skaggs, 2005). Therefore, participants in the professional development unit increased their per ceived self efficacy toward utilizing program evaluation skills.
90 posttest mean score gains on the SCSES (Bodenhorn & Scaggs) based on their years of counseling experience. There was also no significant difference between participants pretest posttest mean score gains on the SCSES (Bodenhorn & Scaggs) based on their graduation from a CACREP accredited program. Therefor e, neither years of experience nor graduation from a CACREP program were differentiating factors. Research Question Three The third question investigated the impact the professional development unit had on aluation in their own school setting. The Professional Development Unit was based on The Systematic Approach of Rossi, Lipsey, and Freeman (2004). This approach prescribes five levels to be assessed when conducting a program evaluation: (a) Need for the Pr ogram, (b) Program Theory, (c) Program Process, (d) Impact of the Program, and (e) Cost Analysis. In this study, the professional development unit taught needs assessment, evaluating program alignment with need, program design and theory, program implemen tation and monitoring, and the topic of sharing outcomes. Two months after the final session of the professional development unit, an Implementation Survey was given to the participants to complete. The Implementation Survey collected information on the use of the program evaluation skills taught during the professional development unit. Results from the Implementation Survey indicated that 70% of participating school counselors conducted a needs assessment after having received the profe ssional development unit. Interestingly, 80% of participating school counselors assessed how the purpose of the program they selected aligned with the needs assessment. Results indicated more respondents identified they had looked at program alignment wit h need than had conducted a needs assessment. The difference could have been attributed to the fact that some school counselors
91 aligned their program to the school wide improvement plan, which had pre identified the needs. Therefore, they would not have conducted a separate needs assessment. Finally, 95% of the respondents evaluated the implementation of a program in their school setting, and 75% of them assessed the outcomes of a program within their school counseling curriculum. These results indicate that the majority of respondents applied the learning from the professional development in their school setting. The results of the Implementation Survey indicated that the participants felt supported by the principal and other staff in their building to implement a program evaluation. On a 5 point scale, the mean of the responses was a 4.37 when participants were asked if they felt supported by their principal in the evaluation of a program in their guidance program. When asked if supported by other co unselors the mean response was 4.32 and mean response was 4.21 when asked if they felt supported by other staff in the building. This finding indicated the perceived presence of organizational support and is supported by previous literature that organizat ional support is a required element in the implementation of skills learning in professional development (Gurskey, 2002). Additionally, the Implementation Survey results identified both the need for more training time and for hands on support as two are as that could be improved in the professional development unit. This finding supports previous literature on program evaluation, which identifies duration and experiential learning as factors in effective professional development models (Guskey &Yoon, 200 9; Trevisan, 2004; Yoon, Duncan, Lee, Scarloss, & Shapley, 2007). Implications Practice Program evaluation is a comprehensive approach used to demonstrate effectiveness for the purpose of accountability and as a means for the school counselors to improv e and develop a
92 comprehensive guidance program (Astramovich & Coker, 2007; Loesch & Ritchie, 2008; Wheeler & Loesch, 1981). The understanding of the impact that this professional development unit on program evaluation had on in service elementary school co unselors has implications for the practice of school counseling program evaluation practices. There has been a long standing need for school counselors to be more accountable (Brown & Trusty, 2005; Gysbers, 2004; Gysbers & Henderson, 2001; Myrick, 2003; W heeler & Loesch, 1981). The current climate of standards based reform has increased the sense of urgency for school counselors to be accountable for their contribution to student achievement (Dahir & Stone, 2009; Herr, 2001; House & Hayes, 2002; McGanno n, Carey & Dimmitt, 2005). Emphasis is evident in National School Counselor Training Initiative (NSCTI), established by The Educational Trust, to promote the inclusion of the school counselor in the accountability system. NSCTI works to promote the schoo l counselor as a change agent who fosters student academic achievement (Educational Trust, 2007). Additionally, the ASCA National Model (2005), a framework for school counseling programs, specifies standards of an effective school counseling program and s pecifically includes being engaged in continuous program evaluation activities as a means to address accountability (ASCA, 2005). The sense of urgency to link the school counselor to student achievement has become even more pressing with new initiatives l Education, 2010c). Race to the Top is a $4.35 billion federally funded competitive grant awarded to states who demonstrate the development of conditions which promote significant improvement to student outcomes in the areas of student achievement, graduation rates, and the closing of achievement gaps (U. S. Department of Education, 2010c). State applications for Grants like Race to the Top provide prescriptive interventions, which focus on compr ehensive
93 educational reform (U. S. Department of Education, 2010d). Educational reform plans impact the systems and process in place at the school level. School counselors are in integral part of the school reform efforts (ASCA, 2005; Dahir & Stone, 2009 ; Educational Trust, 2007; Herr, 2001; House & Hayes, 2002; McGannon, Carey & Dimmitt, 2005). Individual state accountability models and school reform measures now require the development of a measure which links student achievement to school and in some cases district personnel (U. S. Department of Education, 2010b). These requirements often include the school counselor (Florida Department of Education, 2010). Counselors must have the confidence in their program evaluation skills and employ the practic e of program evaluation as a first step in understanding the impact their programs have on student achievement (Rossi et al, 2004) Yet, despite the recognized growing importance of school counselor accountability, school counselors are not systematica lly implementing program evaluation into their way of work (Fairchild & Steeley, 1995; Isaccs, 2003; McGannon, Carey & Dimmitt, 2005). Internal (Trevisan, 20 02a). Externally, the organization of the school, including support from principal and other staff to conduct program evaluation are factors in contributing to the implementation of program evaluation. Internally, counselors often lack training to adequa tely prepare them to practice program evaluation (Astramovich & Coker, 2007; Lusky & Hayes, 2001). This study addresses both the lack of training and examines the perceived support for program evaluation The results of this study were encouraging; the professional development confidence towards their knowledge of program evaluation skills efficacy with initiating program evaluation practices. Results from t he follow up Implementation Survey showed that
94 more than 95% of the participants in the study did select a program in their school counseling curriculum to evaluate. Results indicated there were high levels of implementation in each of the program evaluat ion components addressed in the professional development. The components included, needs assessment, program alignment, program design and purpose, and sharing understandi ng of program evaluation skills and improved self efficacy implies an increased likelihood that the participants will utilize the skills acquired from the professional development unit (Bandura, 1977, 1996, 2007). These findings have strong implications for practicing school counselors. There is an identified need for the implementation of program evaluation amongst school counselors ( Dahir & Stone, 2009; Educational Trust, 2007; Fairchild & Steeley, 1995; Herr, 2001; House & Hayes, 2002; Isaccs, 2003; Mc Gannon, Carey & Dimmitt, 2005.) This study found a professional implementation of program evaluation. Theory Moreover, the evaluation of the professional develop ment unit supported the evidence that use of a professional development unit was an effective strategy for increasing self confidence towards program evaluation knowledge and the application of these skills among the participating school counselors. In t he theoretical framework proposed by Gurskey (2002), the evaluation of professional development has five levels. The five levels include: (1) Participant Reaction; (2) Participant Learning; (3) Organizational Support and Change; (4) Participant Use of Know ledge and Skills; and (5) Student Learning Outcomes. In this study, these five levels of program evaluation were assessed as follows: (1) Participant Reaction the Implementation Survey indicated that the participants found the professional development uni t valuable; (2)
95 Participant Learning -the difference between the pretest and posttest ECSE demonstrated that the participants increased self confidence in their knowledge of the skills being taught; (3) Organizational Support and Change -the Implementatio n Survey results provided evidence of organizational support; (4) Participant Use of Knowledge and Skills -the Implementation Survey indicated that the majority of participants implemented what they had learned through the professional development traini ng unit at their schools. The delivery of the professional development was based on Social Cognitive Theory, and included lecture, discussion, collaborative learning activities, and individual reflection. This study provided evidence that use of the prof essional development unit delivered in this study was confidence towards their knowledge and the application of program evaluation skills. These results further validate the applications of Soci al Cognitive Theory to professional development. This study is timely and the results were significant. The participants reported they had implemented program evaluation. The study has the potential to inform the future development of professional devel opment on program evaluation skills for school counselors. Limitations This study used a pretest posttest pre experimental design: it did not utilize a control group. As a pre experimental design, this study lacked internal validity (Ary, Jacobs, & Razavi ech, 1996). The observed changes cannot definitively be attributed to this professional development unit and not to other possible causes. There are other variables that could have been responsible for the observed changes between the pretest results and the posttest results found in this study. Two such possible extraneous variables were history and maturation (Ary et al., 1996). History as a source of change would include any external event or factor outside of s posttest score, such as outside training, reading, or other
96 learning (Rand, 2006). Maturation as a source of change would include any physical or mental Another limitation in the pre experimental design was reactivity; the concept that study participants may act differently in response to being observed (Ary et al., 1996). In this study, participants were all receiving the professional development as part of the monthly school counsel or meetings. The supervisor of school counseling for the school district was a present observer at the meetings and strongly supported participation in the professional development. Reactivity may have been a factor in the observed pretest posttest chang e. In addition, familiarity, where test scores improve over time because test subjects do better on a test once they become more familiar with it, may affect the results of this study. Familiarity with the items, rather than the result of the professiona l development, may have contributed to the observed pretest posttest change (Ary et al, 1996). There are also limits to the generalizability of the study findings. First, the sample size was small. The professional development was delivered to all 77 of the elementary school counselors employed by Pinellas County Schools. Forty six elementary school counselors volunteered to participate in the study. Of those who volunteered, 29 were able to attend all three sessions of the professional development un it and completed both the pretest and posttest instruments. Although the study started with a large group, the end sample was small (N=29), further limiting generalizability of the results. Secondly, the sample of school counselors participating in the study was not randomly selected. Participants chose to take part in the study. The common characteristic of being willing to take part in the study may have somehow made them a special subgroup of the population and therefore may not be representative of the population of all school counselors. Finally, the
97 person delivering the training program was the person carrying out this research project. As the trainer and researcher, the person leading the training program was highly motivated to see the projec t succeed. This factor could introduce bias and make the study results less likely to generalize to other situations in which the person leading the training program would not be similarly motivated. Recommendations Weaknesses of this study could be eas ily addressed in future studies. This study lacked a control group, was conducted with a small group of elementary school counselors in one district in Florida, and was performed with the researcher also delivering the training program. The primary recom mendation for future research is to replicate this study with a control group to include a larger group of elementary, middle and high school counselors in multiple locations, and to utilize training program leaders who are not the researchers. Despite req uiring a substantial time investment for the researcher and participants, the results of this study indicated a high level of participant satisfaction. Therefore, additional study in this area could be well received by school counselors. Including a cont rol group in future studies would benefit the study by increasing the strength of the study (Rand, 2000) by increasing the confidence that the positive changes were attributed to the professional development unit. Expanding the scope of future studies to include a larger group of school counselors at all school levels would allow for better generalizability of the findings. Including different levels would also help determine if there is a difference in acquisition of program evaluation skills, self effic acy or implementation of program evaluation between school counselors practicing at elementary, middle, and high school levels. Expanding this research beyond one district would provide the opportunity to examine the effect of the professional development training on program evaluation on counselors working in different
98 school districts. This could include a variety of settings such as small and large districts and different regions of the United States. Such expansion would allow an examination into the difference between participants in different areas, in acquisition of program evaluation skills, self efficacy or implementation of program evaluation. Finally, separating the role of the researcher from the role of the training program leader would remov e a potentially important source of bias from future studies. Additional studies in this area, with the above changes would provide a more comprehensive understanding of the impact of this professional development on program evaluation on in service school counselors. Future studies could be improved by heeding recommendations gleaned from the Implementation Survey. Participants indicated the need for beginning this training at the start of a school year. A recommendation could be to start this professional developmen t at the very beginning of the school year. The results of the Implementation Survey also indicated a need for additional direct support with the development of an evaluation and hands on activities during the professional development. A recommendation c ould be to increase the duration of the professional development to last the entire nine months of the school year. Doing so would allow additional time to provide targeted support and incorporate teaching techniques that foster guided instruction, indepe ndent activities, and feedback. This study was designed to determine the impact a professional development unit had on however, investigate the impact that implem entation of program evaluation had on the overall goals of the work done by the counselor. A recommendation would be to add a research question, which could address did the evaluation done by the school counselor at their school result in improvements to the program they evaluated? A reasonable next step in the research
99 would be to further investigate the impact of an evaluation done by the school counselor had on the program they evaluated. Additionally, the theoretical framework of Gurskey (2002) propo ses evaluating the impact a professional development had on student learning outcomes There are ever increasing mandates to link educational personnel in the schools to student learning outcomes ( U. S. Department of Education, 2010c ). The examination of the link between how the professional evaluation would have strong implications for the development and implementation of professional development for school counsel ors. An important next step, which could further elicit additional recommendations for the direction of ongoing research, is the sharing of the results with the participants. It is important to share the outcomes of the research with the participants to v alidate their participation (Dillman, 2007). Discussions between the researcher and the participants may also bring out previously unaddressed concerns with the professional development content or delivery. Sharing results with participants is feasible a nd could be accomplished by the researcher utilizing the same monthly guidance meeting forum used to deliver the professional development unit. Summary This study showed that after participating in a professional development unit on program evaluation, elementary school counselors increased their knowledge of program evaluation, increased their perceived self efficacy toward program evaluation skills, and applied the learning from the professional development in their school settings. The ASCA National M odel (2005) standards of an effective school counseling program reflect the current climate of reform, making these finding encouraging. The majority of school counselors who participated in the study did implement the principles they learned, felt increa sed self efficacy in doing so, and
100 indicated a sense of school level support for using program evaluation. Although further research is necessary, the findings of this study suggest that implementation of a professional development unit like the one test ed in this study might be a useful step toward increasing the program evaluation skills of school counselors.
101 APPENDIX A EMAIL FROM GUIDANCE SUPERVISOR To: School Counselors From: Guidance Supervisor RE: Back to School Guidance Meeting Date: D uring our district guidance meetings we will be working to develop our program evaluation skills with Nicole Carr, a doctoral candidate from the University of Florida. The materials and skills gathered from this professional development are aimed at assist ing you in completing the annual measure you submit to me at the end of the year. It is expected that you will conduct an evaluation of a component of the guidance program at your school as a result of the professional development unit. Nicole will be de livering the professional development unit as part of a research study. As a participant you will benefit from learning program evaluation skills, which are important to you as a professional school counselor. Your participation in the research study is v oluntary. As a voluntary participant, at the start of the professional development unit, you will be asked to anonymously complete (1) a personal data sheet, (2) a questionnaire regarding program evaluation, and (3) a school counselor self efficacy scale. At the end of the professional development you will again be asked to anonymously complete a questionnaire regarding program evaluation, and a school counselor self efficacy scale. Two months after completing the professional development unit, you will be asked to complete a survey to determine if you have applied what you have learned and actually evaluated your guidance program. You can receive the professional development and opt not to participate in the study by not completing the forms for the stud y.
102 APPENDIX B EMAIL FROM PRINCIPAL To: School Counselor From: Principal RE: Guidance Meeting Date: During your annual guidance meetings this year, you will have to opportunity to participate in a professional development unit on Program Evaluation Skills. I hope you take advantage of this opportunity and attend all three of the sessions offered. They will be available during your scheduled countywide guidance meetings. As a participant you will benefit from learning program evaluation skills. It is expected that you will conduct an evaluation of a component of the guidance program at our school as a r esult of the professional development unit. Nicole Carr, a doctoral candidate from the University of Florida, will be delivering the professional development unit as part of a research study. Your participation in the research study is voluntary. Her study will request that anonymously complete (1) a personal data sheet, (2) a questionnaire regarding program evaluation, and (3) a school counselor self efficacy scale. At the end of the professional development you will again be asked to anonymousl y complete a questionnaire regarding program evaluation, and a school counselor self efficacy scale. Two months after completing the professional development unit, you will be asked to complete a survey to determine if you have applied what you have lear ned and actually evaluated a component of the school guidance program.
103 APPENDIX C SCRIPT PRESENTED BY RESEARCHER AT GUIDAN CE MEETING Hello, information with you t oday. Currently, I am a doctoral candidate in the Department of Counselor Education, at the University of Florida. As part of my dissertation I am examining the impact a professional development unit on program evaluation has on in service school counsel ors. I am asking you to volunteer to participate in this study. The study includes participation in a professional development unit. You will be asked to meet three times for one to two hours, over the course of the semester. At the start of the prof essional development unit, you will be asked to anonymously complete (1) a personal data sheet, (2) a questionnaire regarding program evaluation, and (3) a school counselor self efficacy scale. At the end of the professional development you will again be asked to anonymously complete a questionnaire regarding program evaluation, and a school counselor self efficacy scale. Two months after completing the professional development unit, you will be asked to complete a survey to determine if you have applie d what you have learned and actually evaluated your program. The follow up survey will take between 5 10 minutes to complete. There are not anticipated risks. As a participant you will benefit from learning program evaluation skills You are free to w ithdraw your consent to participate and may discontinue your participation in the study at any time without consequence. The researcher wi ll not provide monetary or time compensation for participation in this study. Your identity will be kept confidentia l to the extent provided by law. Your information will be assigned a code number. The list connecting your name to this number will be kept on a When the stu dy is completed and the data have been analyzed, the list will be destroyed. Your name will not be used in any report. If you have any questions about this research protocol, please contact me, my name and the name of my faculty advisor along with contact information are provided on the informed consent. Questions or concerns about your rights as a research participant rights may be directed to the IRB02 office, which can also be found on the informed consent.
104 APPENDIX D PERSONAL DATA SHEET The purpose of these questions is to gather demographic information about participants in this research study. You will not be asked to provide your name. Your responses will only be reported in general terms as they are related to the variables being an alyzed. Please provide your information like this : not like this : 1. Gender: Male Female 2. Ethnicity: Yes No Hispanic or Latino 3. Select all that apply Race: Yes No American Indian or Alaskan Asian Black or African American Native Hawaiian or Other Pacific Islander White 4. 5. Number of years of experience as a school counselor: _______ 6. Age : _______ Thank you for participating! Program Yes No Did you complete a Masters Program in Counselor Education? Did you graduate from a CACREP accredited program?
105 APPENDIX E SCHOOL COUNSELOR SEL F EFFICACY SCALE Below is a list of activities representing many school counselor responsibilities. Indicate your confidence in your current ability to perform each activity by circling the appropriate answer next to each item according to the scale defined below. Please answer each item based on one current school, and based on how you feel now, not on your anticipa ted (or previous) ability or school(s). Remember, this is not a test and there are no right answers. Use the following scale: 1 = not confident, 2 = slightly confident, 3 = moderately confident, 4 = generally confident, 5 = highly confident. Please circle the number that best represents your response for each item. 1. Advocate for integration of student academic, career, and personal development into the mission of my school. 1 2 3 4 5 2. Recognize si tuations that impact (both negatively and positively) student learning and achievement. 1 2 3 4 5 3. Analyze data to identify patterns of achievement and behavior that contribute to school success 1 2 3 4 5 4. Advocate for myself as a professional school counselor and articulate the purposes and goals of school counseling. 1 2 3 4 5 5. Develop measurable outcomes for a school counseling program which would demonstrate accountability. 1 2 3 4 5 6. Consult and collaborate with teachers, staff, administrators and parents to promote student success. 1 2 3 4 5 7 Select and implement applicable strategies to assess school wide issues. 1 2 3 4 5 8 Promote the use of counseling and guidance activities by the total school community to enhance a positive school climate. 1 2 3 4 5 9 Develop school improvement plans based on interpreting school wide assessment r esults. 1 2 3 4 5 10 Identify aptitude, achievement, interest, values, and personality appraisal resources appropriate for specified situations and populations. 1 2 3 4 5 11 Analyze data to identify needs of students in my school. 1 2 3 4 5 12 Differentiate needs from means. 1 2 3 4 5 13 Identify expected program outcomes of a program in my counseling curriculum. 1 2 3 4 5 14 Review research on existing programs. 1 2 3 4 5 15. Develop a process to monitor implementation of a program. 1 2 3 4 5 16. Assess the effectiveness of a program in my counseling curriculum. 1 2 3 4 5 17. Analyze the impact of a program. 1 2 3 4 5 18. Articulate the outcomes of a program. 1 2 3 4 5
106 APPENDIX F ESSENTIAL COMPETENCI ES FOR PROGRAM EVALU ATORS SELF ASSESSMENT Below is a list of Program Evaluator Competencies. Indicate your confidence in your current ability to perform each activity by circling the appropriate answer next to each item according to the scale defined below. Please answer each it em based on how you feel now, not on your an ticipated (or previous) ability. Remember, this is not a test and there are no right answers. Use the following scale: 1 = not confident, 2 = sl ightly confident, 3 = moderately confident, 4 = generally confident, 5 = highly confident. Please circle the number that best represents your ability for each item. 1 Act ethically and strives for integrity and honesty in conducting ev aluations 1 2 3 4 5 2 Address conflict in an evaluation 1 2 3 4 5 3 Analyze data 1 2 3 4 5 4 Analyze situations 1 2 3 4 5 5 Analyze the political considerations of an evaluation 1 2 3 4 5 6 Apply professional evaluation standards 1 2 3 4 5 7 Assess reliability of data 1 2 3 4 5 8 Assess validity of data 1 2 3 4 5 9 Attend to the issues of organizational change 1 2 3 4 5 10 Attend to the issues surrounding the use of an evaluation 1 2 3 4 5 11 Budget time and resources for an evaluation 1 2 3 4 5 12 Build professional relationships that will enhance program evaluation practice 1 2 3 4 5 13 Collect data 1 2 3 4 5 14 Communicate with stakeholders throughout and evaluation process 1 2 3 4 5 15 Conduct an evaluation in a non disruptive manner 1 2 3 4 5 16 Conduct literature reviews 1 2 3 4 5 17 Conduct meta evaluations 1 2 3 4 5 18 Consider the general and public welfare in evaluation practice 1 2 3 4 5 19 Contribute to the knowledge base of evaluation 1 2 3 4 5 20 Convey personal evaluation approaches and skills to potential stakeholders 1 2 3 4 5 21 Demonstrate cross cultural competence in an evaluation 1 2 3 4 5 22 Describe a program 1 2 3 4 5 23 1 2 3 4 5 24 Develop evaluation designs 1 2 3 4 5
107 Use the following scale: 1 = not confident, 2 = slightly confident, 3 = moderately confident, 4 = generally confident, 5 = highly confident. Please circle the number that best represents your response for each item. 25 Develop recommendations 1 2 3 4 5 26 Examine the organizational context of the evaluation 1 2 3 4 5 27 Facilitate constructive interpersonal interaction to assist in evaluation 1 2 3 4 5 28 Frame evaluation questions 1 2 3 4 5 29 Identify data sources 1 2 3 4 5 30 Identify needed resources for an evaluation 1 2 3 4 5 31 Identify the interests of stakeholders 1 2 3 4 5 32 Interpret data 1 2 3 4 5 33 Justify the cost of an evaluation 1 2 3 4 5 34 Make judgments 1 2 3 4 5 35 Modify a study when needed 1 2 3 4 5 36 Negotiate with stakeholders before and evaluation begins 1 2 3 4 5 37 Note strengths and limitations of an evaluation 1 2 3 4 5 38 Present an evaluation in a timely manner 1 2 3 4 5 39 Provide rationales for decisions throughout the evaluation process 1 2 3 4 5 40 Pursue professional development in program evaluation 1 2 3 4 5 41 Reflect on my competencies and areas for growth 1 2 3 4 5 42 Remain open to input from others 1 2 3 4 5 43 Report evaluation procedures and results 1 2 3 4 5 44 Respect clients, respondents, program participants and other stakeholders 1 2 3 4 5 45 Respect the uniqueness of the evaluation site and client 1 2 3 4 5 46 Respond to requests for evaluations 1 2 3 4 5 47 Specify program theory 1 2 3 4 5 48 Supervise others conducting an evaluation 1 2 3 4 5 49 Train others in evaluation 1 2 3 4 5 50 Understand the knowledge base of evaluation (terms, concepts, theories, assumptions) 1 2 3 4 5 51 Use appropriate technology for an evaluation 1 2 3 4 5 52 Use Interpersonal skills in program evaluation 1 2 3 4 5 53 Use negotiation skills in program evaluation 1 2 3 4 5 54 Use verbal / listening skills in program evaluation 1 2 3 4 5 55 Use written communication skills in program evaluation 1 2 3 4 5 57 Knowledgeable about mixed methods 1 2 3 4 5 58 Knowledgeable about qualitative methods 1 2 3 4 5 59 Knowledgeable about quantitative methods 1 2 3 4 5 60 Aware of self as an evaluator 1 2 3 4 5
108 APPENDIX G IMPLEMENTATION SURVE Y The purpose of these questions is to determine if the Professional Development Unit had an not an evaluation of the school counselor. You are asked not to prov ide your name or name of your school. Please provide your information like this : not like this : Since completing the Professional Development Unit: Yes No 1. I have conducted a needs assessment. 2. I have selected a guidance program to evaluate. 3. I have assessed the theory and purpose of the program. 4. I have assessed how the program purpose aligns with the needs. 5. I have defined the anticipated outcomes. 6. I have assessed the implementation of the program. 7. I have assessed the outcomes of the program. 8. I have shared the results with: (Please check all that apply) Other Counselors School Advisory Council Parents School Leadership Team Teachers Other Please indicate your level of agreement with the following statements: Strongly Agree Agree Neutral Disagree Strongly Disagree I felt supported by my principal in the evaluation of a program in my guidance program I felt supported by the other staff in the building. I felt supported by other school counselors. I had time to complete this process. I will continue to practice program evaluation as a way of work. What did you like about the professional development on Program Evaluation?
109 What would you change about the professional development? How will this professional development influence your way of work?
110 APPENDIX H INFORMED CONSENT Dear School Counselor: I am a doctoral candidate in the Department of Counselor Education, at the University of Florida As part of my dissertation I am examining the impact a profession al development unit on program evaluation has on in service school counselors. I am asking you to volunteer to participate in this study. In this study you will be asked to volunteer to participate in a four part professional de velopment unit. You will be asked to meet three times for one to two hours over the c ourse of the semester. At the start of the professional development unit, you will also be a sked to anonymously complete a personal data sheet, a questionnaire regardin g program evaluation and a school counselor self efficacy scale. At the end of the professional development yo u will be asked to anonymously complete a questionnaire regarding program evaluation, and a s chool counselor self efficacy scale. Two months after completing the professional developm ent unit, you will be asked to complete a survey to determine if you have applied what you have learned in the professional development unit and conducted an evaluation of your program. The follow up survey will t ake between 5 10 minutes to complete. There are no anticipated risks. As a participant you will benefit from learning program evaluation skills. You are free to withdraw your consent to participate and may discontinue your participation in the study at any time without consequence T he researcher will not provide monetary or time compensation for participation in this study. Your identity will be kept confidential to the extent provided by law. Your information will be assigned a code number. The list c onnecting your name to this num ber will be kept on a password protected computer and paper copies will be locked in a file in the the study is completed and the data have been analyzed, the list wil l be destroyed. Your name will not be used in any report. If you have any questions about this research protocol, please contact me, Nicole Carr, at (727) 643 4658 firstname.lastname@example.org or my faculty advisor at Mary Ann Clark, PhD. Department of Counselor Education, University of Florida and (352) 273 4331 Questions or concerns about your rights as a research participant rights may be directed to the IRB02 office, University of Florida, Box 112250, Gainesville, FL 32611 Please sign and retur n this copy of the letter in the enclosed envelope. A second copy is provided for your records. By signing this letter, you give me permission to report your responses anonymously in the final dissertation. Participant: _________________________________ ________________Date:__________ Principal Investigator: _________________________________________Date: _________
111 APPENDIX I OUTLINE OF PROFESSIO NAL DEVLOPMENT UNIT I. Unit I Needs Assessment A. Basic Program Evaluation 1. Why do we evaluate? 2. How do we evaluate? 3. When do we evaluate? B. How to of Program Evaluation 1. Cost Analysis 2. Effectiveness 3. Implementation 4. Theory/ Design 5. Needs Assessment Needs Assessment D. Determine what it is we need before we can determine what program aligns with our needs. E. ACTIVITY (whole group): School Counselors 1. Who are we? 2. What do we do? 3. What do we want to be? F. The gaps between what is and what is the g oal. G. ACTIVITY (individual) Personal example H. The Data I. What data do we have available? 1. AYP 2. School Grades 3. Individual Student 4. Climate Surveys 5. Discipline 6. Attendance 7. Targeted Behavior 8. Other J. ACTIVITY (small group) 1. Where is your program now? 2. What does it look like? 3. What are the gaps/ problems/ needs? K. Next Steps 1. Set goals that align with the needs 2. Sound Theory 3. Design Plan 4. Implement
112 II. Unit II Program Alignment A. How do we know what will meet our needs? 1. What are our needs? 2. How did we determine our needs? Where are we and where do we want to be? What are the gaps between what is and what should be? The gaps are the need s. B. Avoid starting with the means 1. At first we all think of the MEANS (resources solutions, money, more teachers, computers, more programs, more testing) 2. Be sure you have a clear understanding of what it is you really need. 3. If you are unclear what the needs are you will not be able to align your programming to fit the needs C. Understanding what we have 1. What means are already in place to meet these needs? a. Are these means aimed at meeting other needs? b. Are they w orking? c. How do we know? D. ACTIVITY (small group) 1. Your school needs 2. Your school has what in place to meet these needs E. What Works 1. Obtaining a clear definition of the objective of a program 2. What is it intended to do? (Refer to What Works Clearinghouse ) F. Be Specific 1. What is the goal of your program? a. Does this goal match my need? b. Is the goal clear? 2. Who will this program impact? 3. How will I monitor the pro gress? 4. How will I know this program is working? a. What can I measure to see this is working? b. What will indicate this program is not working? G. ACTIVITY (individual/ whole group share) 1. Identify a need in your school 2. Your s chool has what in place to meet these needs 3. Next steps H. Program Design 1. What does the theory tell you? 2. What are the objectives of the program? 3. Is this something that can be done? I. Remember: Effectiveness Implementa tion Theory/ Design Needs Assessment
113 III. Unit III Program Design and Process A. Review 1. Assessment of Program Outcomes 2. Assessment of Program Process 3. Assessment of Program Design 4. Assessment of Need for Program B. Sound Theory C. Design 1. Is the design grounded in theory? 2. Is the design feasible? 3. Are the objectives clear and measurable? D. Is the program a means to filling the need? Does this program objective align with what is needed? E. Implemen tation 1. Describe what took place. How was it implemented? 2. Was the program implemented the way it was intended? objective? F. Monitoring G. ACTIVITY (small group) 1 What was the goal of your program? a. Did this goal match my need? b. Was the goal clear? 2. Who did this program impact? 3. How am I monitoring the progress? 4. How will I know this program is working? a. What can I measure to see this is working? b. What will indicate this program is not working? H. Assessing the process I. Remember: Effectiveness Implementation Theory/ Design Needs Assessment
114 IV. Unit IV Sharing Outcomes A. What happened? 1. What were the goals? 2. What need/ problem was being addressed? 3. What took place? (describe it) 4. What were the results? 5. How were these results determined? B. Evidence verses Proof C. Who needs to know? D. ACTIVITY (whole group) 1. Who needs this information? 2. How will I determine who needs the information? 3. How will I know what to share with them? E. How do I tell them? 1. Not one size fits all no template 2. You know your audience F. ACTIVITY (small group) 1. What was the goal of your program? a. Did this goal match my need? b. Was the goal clear? 2. Who did this program impact? 3. How did I monitor the progress? 4. How did I determine if this program worked? a. What did I measure to s ee if it worked? b. What were my results? 5. How will I share my results?
115 LIST OF REFERENCES Alderman, H. S., & Taylor, L. (2002). S chool counselors and school re form: New directions. Professional School Counseling 5 235 248 American School C ounseling Association. (2005). American School Counseling Association National Model: A framework for school counseling programs (2 nd ed.). Alexandria, VA: Author. American School Counseling Association (2007). School counseling standards: s chool counselor competencies. Retrieved October 19, 2007 from http://www.schoolcounselor.org/files/competencies.pdf American School Counseling Association (2008). School Counselor Per formance Standards. Retrieved December 2008 from: http://www.ascanationalmodel.org/content.asp?pl=33&sl=35&contentid=35 American School Counseling Association (2010). ASCA National Model. Retrieved September 2010 from: http://www.ascanationalmodel.o rg/ Ary, D., Jacobs, L. C., & Razaviech, A. (1996). Introduction to Research in Education (5 th ed.) Florida: Harcourt Brace & Company. Astramovich, R.L. & Coker, J.K. (2007). Program evaluation: The accountability bridge Model for counselors. Journal of Counseling and Development 85 162 172. Astramovi ch, R.L., & Coker, J.K. (2005). Training school counselors in program evaluation. Professional School Counseling 9 49 54. satisfaction and commitment: Correlates and predictors. Professional School Counseling 9 197 205 Bandura, A. (1977). Self Efficacy: Toward a unifying theory of behavior change. Psychological Review 84 191 215. Bandura, A. (1986). Social foundations of thought and action: A social cognitive theory Englewood Cliffs, NJ: Prentice Hall. Bandura, A. (1994). Self Efficacy. In V.S. Ramachaudran (Ed.), Encyclopedia of human behavior (Vol 4.) New York: Academic Press (Reprinted from Encyclo pedia of mental health H Friedman (Ed.) 1998 San Diego: Academic Press). Bandura, A. (1999). Social cognitive theory: An agenetic perspective. Asian Journal of Social Psychology 2(1), 2141. Bandura, A. (2007). Self Efficacy: The exercise of control (9 th E d.) New York: MacMillan.
116 Bax, S. (2002). The social and cultural dimensions of training the trainer. Journal of Education for Teaching 28, 167 178. what we now know. Journal of Teaching in Physical Education 25, 363 378. Bernard, J.M., & Goodyear, R.G. (1998). Fundamentals of Clinical Supervision (2 nd E d.). Needham Heights, MA: Allyn & Bacon. Berna rd, J.M., & Goodyear, R.G. (2004 ). Fundamentals of Clinical Supervision (3 rd E d.). California: Merrill Betz, N., & Voyten, K. K. (1997). Efficacy and outcome expectations influence career exploration and decidedness. The Career Development Quarterly 46, 179 189. Bodenhorn, N. & Skaggs, G. (2005). Develop ment of the school counselor self efficacy scale. Measurement and Evaluation in Counseling and Development 38 14 21. Bodenhorn, N., Wolfe, E., Airen, O. (2010). Program choice and self efficacy: Relationship to the achievement gap and equity. Profess ional School Counselor 13, 165 174. Borders, L. D. (2005). Snapshot of clinical supervision in counseling and counselor education: A five year review. The Clinical Supervisor 24 (1 2), 69 113. Borders, L. D., & Brown, L. L. (2005). The New handbook o f counseling supervision Mahwah, NJ: Lahaska/Lawrence Erlbaum. Brouwers, A., & Tomic, W. (2001). The factorial validity of scores on the teacher interpersonal self efficacy scale. Educational and Psychological Measurement, 6(3), 433 445. Br own D., & Trusty, J. (2005). School counselors, comprehensive school counseling programs, and academic achievement: Are school counselors promising more than they can deliver? Professional School Counseling 7 91 99. CACREP (2009) Directory of programs. Retrieved December 12, 2009 from http:// www.cacrep.org Cantrell, S. C., and H. K. Hughes. 2008. Teacher efficacy and content literacy implementation: An exploration of the effects of extended professional development with coaching. Journal of Literacy Research 40:95 127. Carey, J., & Dimmitt, C. (2006). Resources for school counselors and counselor educators: The Center for School Counseling Outcome Research. Professional School Counseling, 9 416 420.
117 Carey, J. C., Dimmitt, C., Hatch, T. A ., Lapan, R.T., & Whiston, S. C. (2008). Report of the National Panel for Evidence Based School Counseling: Outcome Research Coding Protocol and evaluation of Student Success Skills and Second Step Professional School Counseling, 11, 197 206. Carey, J ., Harrity, J., & Dimmitt, C. (2005). The development of a self assessment instrument to measure a school district's readiness to implement the ASCA national model. Professional School Counseling, 8 305 312. Campbell, C. A., & Dahir, C.A. (1997). The n ational standards for school counseling programs Alexandria, VA: American School Counseling Association. Clark, M. A. & Amatea, E. (2004). Teacher perceptions and expectations of school counselor contributions: Implications for program planning and training. Professional School Counseling, 8 12 140. Compeau, D. R., & Higgins, C. A. (1995). Application of social cognitive theory to training for computer skills. Information Systems Research 6. 118 143. Crowne, D. P., & Marlowe, D. (1960). A new scale of social desirability independent of psychopathology. Journal of Consulting Psychology 24 349 354. assessment to detect workshop success: Do they work? American Journal of Ev aluation 29 92 98. Dahir, C., & Stone, C. (2003). Accountability: A M.E.A.S.U.R. E. of the impact school counselors have on student achievement. Professional School Counseling, 6 214 221. Dahir, C., & Stone, C. (2009). School Counselor accountab ility: The path to social justice and system change. Journal of Counseling and Development 87 12 20. Dillman, D. (2007) Internet, Mail, and Mixed Mode Surveys: The Tailored Design Method. (2 nd E d.) New York: Wiley. Dollarhide, C. & counselors. Professional School Counseling, 9 295 304. Eakin, S. (1996) National education summit. Technos 5 16 25. Educational Trust. (2007). Transforming school counse ling. Ret rieved January 13, 2009 from http://www2.edtrust.org/EdTrust/Transforming+School+Counseling/main Elementary and Secondary Education Act, Pub. L. No.89 10, 79 Stat 27, 20 U.S.C. ch 70 (1965) Eisner, E. (2001). What does it mean to say a school is doing well? Phi Delta Kappan 82 367 372.
118 Fairchild, T. N., & Seeley, T.J. (1995). Accountability strategies for school counselors: A School Counselor 42 377 392. Feldt, R. C., & Woelfel, C. (2009). Five factor personality domains, Sel f efficacy, career outcome expectations and career indecision College Student Journal 43, 2, 429 437. Fitch, T. J. & Marshal, J. L. (2004) What counselors do in high achieving schools: A study on th e role of the school counselor. Professional Sch ool Counselor 7 172 177. Fitts, W. H., & Warren, W. L. (1996). Tennessee Self Concept Scale: TSCS:2 Los Angeles: Westem Psychological Services. Florida Department of Education (2009). Florida School Indicator Reports. Retrieved March 30, 2009, fro m http://www.fldoe.org/eias/eiaspubs/xls/fsir/2007 08/Mem_Category_Dist_0708.xls Florida Department of Education (2008) Race/ Ethnicity, Survey 6 and Other Chang es. Retrieved March 29, 2009, from: http://www.fldoe.org/eias/databaseworkshop/word/racesvy6.doc Florida Senate (2010) The 2010 Florida Statutes. Retrieved September 11, 2010 from: http://www.leg.state.fl.us/statutes/ Florida Bureau of School Improvement (2010) The Florida Department of Education: Bureau of School Improvement. Retrieved September 11, 2010 from: http://flbsi.org/ Fournier, P., Banza, B., Tourign y, C., & Dieudonne, O. ( 2009 ) Programme evaluation training for health professionals in francophone Africa: Process, competence acquisition and use Human Resources for Health, 7 ( 3 ) Fraenkel, J., & Wallen, N. (2008). How to design and evaluate research in education (7 th Ed.) New York: McGraw Hill. Fullen, M. (2001). Leading in a culture of change. San Francisco: Jossey Bass. Ghere, G., Ki ng, J. A., Stevahn, L., & Mimmema, J. (2006). A professional development unit for reflecting on program evaluation competencies American Journal of Evaluation, 27, 108 123. Greason, P. B. & Cashwell, C. S. (2009) Mindfulness and counseling self efficac y: The mediating role of attention and empathy. Counselor Education and Supervision 49, 2 18. Glass, G.V., & Hopkins, K. D. (1996). Statistical Methods In Education and Psychology. (3 rd Ed.) Massachusetts: Allyn and Bacon.
119 Guba, E. & Lincoln, Y. ( 2004). Fourth Generation Evaluation (6 th Ed.) Thousand Oaks, California: Sage. Guskey, T.R. (2000). Evaluating Professional Development Thousand Oaks, CA: Corwin Press. Guskey, T R. (2002). Professional development and teacher change. Teachers and Teaching: Theory and Practice 8 381 391. Guskey, T. R. (2003, April). What makes profes sional development effective. P resented at annual meeting of American Educational Research Association, Chicago. Guskey, T. R. & Yoon, K.S. (2009) What works in professional development ? Phi Delta Kappan 90 495 500. Gysbers, N. C. (2004). C omprehensive guidance and counseling programs: The evolution of accountability. Professional School Counseling, 8 1 4. Gysbers N. C., & Henderson, P. (2000) Developing and Managing your school guidance program (3 rd E d.). Alexandria, VA: American Counseling Association. Gysbers N. C., & Henderson, P. (2001) Comprehensive guidance and counseling programs: A rich history and bright future. Professional School Counseling 4, 246 256. Gysbers N. C., Hughey, K., Starr, M., & Lapan, R (1992) Improving school guidance programs: a framework for program, personnel, and results evaluation. Journal of Counseling and Development 70 565 57 0. Herr, E.L. (2001) The impact of national policies, economics, and school reform on comprehensive guidance programs. Professional School Counseling 4 236 245. s. Washington, DC :ERIC Clearinghouse on Teacher Education House, R. M., & Hayes, R.L. (2002) School counseling: Becoming key players in school reform. Professional School Counseling 5 249 256. Isaacs, M. L. (2003). Data driven decision making: T he engine of accountability. Professional School Counseling, 6 288 295 The Joint Committee on Standards for Educational Evaluation (2008). The program evaluation standards: How to assess evaluation of educational programs ( 2 nd Ed.). Michigan: Author. Kosine, N, Steger, M, & Duncan, S. (2008). A strengths based approach to finding meaning and purpose in careers. Professional School Counseling 12, 133 136.
120 Kumpulainen, K. Classroom Interaction and Social Learning: From Theory to Practice London: Routledge Falmer, 2001. Krumboltz, J.D. (1998). Counselor actions needed for the new career perspective. British Journal of Guidance & Counseling 26, 559 564. Lambie, G., & Williamson, L. (2004). The challenge to change from guidance counseling to professional school counselor. Professional School Counseling 12 124 131. Lapan, R. T. (2001). Results based comprehensive guidance and counseling program: A framework for planning and evaluation. Professional School Counseling, 4 289 299. Lapan, R. T., Gysbers N. C., & Sun, Y. (1997). T he impact of more fully implemented guidance programs on the school experiences of high school students: A statewide evaluation study. Journal of Counseling and Development 75 292 302. Larson, L. M., Suzuki, L. A., Gillespie, K. N., Potenza, M. T., Bechtel, M. A., & Toulouse, A. L. (1992). Development and validation of the Counseling Self Estimate Inventory. Journal of Counseling Psychology 39 105 120. Larson, L. M. & Daniels, J. A. (1998) Review of the counseling self efficacy literature. The Counseling Psychologist 26, 179 218. Lee, C., & Workman, D. (1992). School counselors and research: Current status and future direction. School Counselor 40 15 20. Lipse y, M., & Wilson, D.B., (2000). Practical meta analysis ( 3 rd E d. ) Thousand Oaks, CA : Sage Loesch, L. C., & Ritchie, M. (2008). The accountable school counselor (2 nd Ed.). Austin, TX: ProEd Inc. Loesch, L. C. (2001). Counseling program evaluation: Insi de and outside the box. In D. C. Locke, J. E. Myers, & E. L. Herr (Eds.), The handbook of counseling (pp. 513 525). Thousand Oaks, CA: Sage. Lusky, M. B., & Hayes, R. L. (2001). Collaborative consultation and program evaluation. Journal of Counseling & Development 79 26 38. Madaus, G. E., & Stufflebeam, D. L. (2000). Program evaluation: a historical overview. In Stufflebeam, D.L., Madaus, G.F. & Kellaghan, T. ( Eds .). Evaluation Models: Viewpoints on Educational and Human Services Evaluation (pp.3 1 8) ( 2 nd ed. ) New York: Springer
121 Martin, J., McCaughtry, N., Kulinna, P. H., & Cothran, D. (2009). The Impact of a Social Cognitive Theory Based Intervention on Physical Education Teacher Self Efficacy. Professional Development in Education 35, 511 529. McGannon, W., Carey, J., & Dimmitt, C. 2005, May). The current status of school counseling outcome research (Research Monograph No. 2). Amherst, MA: Center for School Counseling Outcome Research, School of Education, University of Massachusett s. Myrick, R. D. (1990). Retrospective measurement: An accountability tool. Elementary School Guidance and Counseling 25 21 29 Myrick, R. D. (1997). Developmental guidance and counseling: A practical approach (3 rd ed.). Minneapolis, MN: Educational Media. Myrick, R. D. (2003). Accountability: Counselors count. Professional School Counseling 6 175 179. 29, 2009, from: http://nces.ed.gov/nationsreportcard/ Patton, M. Q. (2004). Roots of Utilization Focused Evaluation. In Alkin, M. C. (Ed.), Evaluation influences. Thousand Oaks, California: Sage. Patton, M.Q. (1 997). Utilization Focused Evaluation: The new century text (3 rd Ed.) Thousand Oaks, CA : Sage. Patton, M.Q. (2008). Utilization Focused Evaluation: The new century text (4 th Ed.) Thousand Oaks, CA : Sage. Pajares, F. (1991). Current directions in self efficacy research. In M. Maehr & P. R. Pintrich (Eds.). Advances in motivation and achievement Volume 10, (pp. 1 49). Greenwich, CT: JAI Press. Pajares (2002). Overview of social cognitive theory and of self efficacy. Retrieved: September 10, 2010 from http://www.emory.edu/EDUCATION/mfp/eff.html Paulsen, A. & Betz, N. (2004). Basic confid ence predictors of career decision making self efficacy. The Career Development Quarterly 52, 354 362. Pratt, C., McGuigan, W., & Katzev, A. 2000. Measuring program outcomes using retrospective pre test methodology. American Journal of Evaluation 21, 341 350. Ravid, R. (2000). Practical Statistics for Educators (2 nd ed.). New York: University Press of America.
122 Rayle, A. D. (2006). Do school counselors matter? Mattering as a moderator between job stress and job satisfaction. Professional School Counselor, 9, 206 215. Riggs, I., & Enochs, L. (1990). Toward the development of an elementary teacher's science teaching efficacy belief instrument. Science Education, 74, 625 638. Rossi, P., Lipsey, M., & Freeman, H. (2004). Evaluation: A Systemati c Approach .(7 th Ed.). Thousand Oaks, CA: Sage. Rowell, L.L. (2005). Collaborative action research and school counselors. Professional School Counseling 9 28 36 Rowell, L. L. (2006). Action research and school counseling: Closing the gap between research and practice. Professional School Counseling 9 376 384. Scarborough, J. L. (2005). The school counselor activity rating scale: An instrument for gathering process data. Professional School Counseling, 8 274 283. Schwitzer, A.M. (1997). Utilization focused evaluation: Proposing a useful method of program evaluation for college counselors and student development programs. Measurement and Evaluation in Counseling and Development 30 50 61. Scriven, M. (2004). Reflections. In Alkin, M. C. (Ed.), and influences. Thousand Oaks, California: Sage. Shadish, W., Cook, T., & Leviton, L. (1995). Foundations of program evaluation (3 rd Ed.). Thousand Oaks, California: Sage. Shavelson, R & Town, L. (2002). Scientific Research in Education Washington: National Academies Press. Shaw, K., Davis, N., & McCarty, B. (1991). A cognitive framework for teacher change. In Proceedings of the thirteenth annual meeting of the North American chapte r of the international group for the psychology of mathematics education, edited by R.G. Underhill, p. 161 167. Blacksburg, VA. Sink, C. A., & Spencer, L. R. (2005). My Class Inventory Short Form as an accountability tool for elementary school counselo rs to measure classroom climate Professional School Counseling, 9, 37 48. Sink, C. A., & Spencer, L. R. (2007). Teacher version of the my class inventory short form: an accountability tool for elementary school counselors. Professional School Counsel ing, 11, 129 139. Spielberger, C. (1983). State Trait Anxiety Inventory (Form Y). Redwood City, CA: Mind Garden.
123 Stevahn, L., King, J., Ghere, G., & Minnema, J. (2005). Establishing essential competencies for program evaluators. American Journal of Evaluation 26 43 59. Stone, C. & Dahir, C. (2006). School Counselor Accountability: A MEASURE of Student Success. (2 nd ed.) New York: Prentice Hall. Studer, J. R., Oberman, A. H., & Womack, R. H. (2006). Producing evidence to show counseling effectiv eness in the schools. Professional School Counseling 9 385 391. Stufflebeam, D. (2004). The 21 st century CIPP model; Origins, development, and use In Alkin, M. C. (Eds.), Thousand Oaks, California: Sage. Trevisan, M. S. (2000). The status of program evaluation expectations in state school counselor certification requirements. American Journal of Evaluation 21 81 94. Trevisan, M. S. (2001). Implementing comprehensive guidance program evaluation support: Lessons learned. Professional School Counseling 4 225 228. Trevisan, M. S. (2002a). Enhancing practical evaluation training through long term evaluation projects. American Journal of Evaluation 23 81 92. Trevisan, M. S. (2002b). Evaluation capacity in K 12 school counseling programs. American Journal of Evaluation, 23 291 305. Trevisan, M. S. (2004). Practical training in evaluation: A review of the literature. American Journal of Evaluation, 25 255 272. U.S. Department o f Education (2001). No Child Left Behind Act (Pub. L. No. 107 110). Retrieved March 10, 2009, from http://www.ed.gov/nclb/overview/intro/execsumm.html U.S. Department of Education (2008). Research and Statistics. Retrieved March 29, 2009, from http://www.ed.gov U.S. Department of Education (2010a) Catalog of Federal Domestic Assistance. Retrieved September 18, 2010, from http://www2.ed.gov/programs/reading first/index System Under No Child Left Behind. Retrieved September 18, 2010 from, http://www2.ed.gov/admins/lead/account/nclbfinalassess/index.html U.S. Department of Education (2010c) Race to the Top Program Executive Summary Retrieved: October1, 2010: http://www2.ed.gov/programs/racetothetop/executive summary.pdf
124 U.S. Department of Education (2010d) Race to the Top Fund. Retrieved October1, 2010 http://www2.ed.gov/programs/racetothetop/index.html Vacc, N. A. & Loesch L. C. (2000). Professional orientation to counseling (3 rd E d. ). Philadelphia : Brunner Routledge Vacc, N. A., Rhyne Winkler, M. C., & Poidevant (1993). Evaluation and accountabili ty of counseling services: Possible implications for a midsize school district. The School Counselor 40 260 266. Valentine, J & Cooper, H. (2003). Effect size substantive interpretation guidelines: Issues in the interpretation of effect sizes. Washi ngton, DC: What Works Clearinghouse. VanSteendam, E., Rijlaarsdam, G., Sercu, L., & VandenBergh, H. (2010). The effect of instruction type and dyadic or individual emulation on the quality of higher order peer feedback in EFL. Learning and Instructio n 20, 316 327. Vogt, M. E., and B. A. Shearer. 2007. Reading specialists and literacy coaches in the real world. (2 nd Ed.) Boston: Pearson. Walsh, M. E., Barrett, J. G., & DePaul, J. (2007). Day to day activities of school counselors: Alignment with new directions in the field and the ASCA national model. Professional School Counseling 10 370 378. Wang, S.L., & Lin, S.J. (2007). The application of social cognitive theory to web based learning through NetPorts British Journal of Educational Tech nology 38, 600 612. Wheeler, P. T., & Loesch, L. (1981). Program evaluation and counseling: Yesterday, today and tomorrow. Personnel and Guidance Journal 51 573 578. Whiston, S. C., & Aricak, O. T., (2008). Development and Initial Investigation of th e School Counseling Program Evaluation Scale Professional School Counseling 11 (4), 253 261. Whiston, S.C. (1996). Accountability through action research: Research methods for practitioners. Journal of Counseling and Development 74, 616 623. Whiston, S. C. (2002). Response to the past, present, and future of school counseling: Raising some issues. Professional School Counseling 5 148 155. Wittmer, J., & Loesch, L.C. (1990). Roses, ducks, and doctoral degrees in counselor education. Counsel or Education and Supervision, 30 (2), 156 162. Yarbrough, D. B., Shulha, L. M., Hopson, R. K., and Caruthers, F. A. (2011). The program evaluation standards: A guide for evaluators and evaluation users (3rd ed.). Thousand Oaks, CA: Sage
12 5 Yoon, K. S., Duncan, T., Wen Yu S., Scarloss, B. and Shapley, K.L. Reviewing the Evidence on How Teacher Professional Development Affects Student Achievement. Issues and Answers Report, REL 2007 -No. 033. Washington, D.C.: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory Southwest, 2007. Wheeler, P. T., & Loesch, L. (1981). Program evaluation and counseling: Yesterday, today and tomorrow. The Personne l and Guidance Journal 51 573 578. Yuen, A. & Ma, W. (2008). Exploring teacher acceptance of e learning technology. Asia Pacific Journal of Teacher Education 36, 3, 229 243. Zimmerman, B.J., & Schunk, D. (2004). Self regulating intellectual processes and outcomes: A social cognitive perspective. In Dai, D.Y., & Sternberg, R.J. (Eds.), Motivation, emotion, and cognition: Integrative perspectives on intellectual functioning and development, (pp. 323 349). Mahwah, N.J. Erlbaum. Zunker, V.G. (2002) Career counseling: Applied concepts of life planning. Pacific Grove, CA: Brooks/Cole.
126 BIOGRAPHICAL SKETCH Born in Rhode Island, Nicole Merlan Carr wanted to become a teacher. After receiving a Bachelor of Arts degree in English and Secondary Education from Rhode Island College, she began a teaching career as a high school English teacher in Zuni New Mexico. She then moved to Florida and continued to teach English. In 1998 she graduated with concurrent degrees fro m the University of Florida, a Master of Education and a Specialist in Education majoring in school counseling and guidance and mental health counseling. She worked as a high school and middle school counselor in Washington and Florida for several years. During that time, she became a National Board Certified Counselor. In 2006 she returned to the University of Florida to pursue a doctorate in counselor education at the University of Florida While enrolled she accepted a position as the full time Title I Research Specialist for Pinellas County Schools, and was later promoted to the Senior Coordinator of Accountability.