|UFDC Home||myUFDC Home | Help|
This item has the following downloads:
1 THE RELATIONSHIP BETWEEN UNDERGRADUATE RE SEARCH AND CRITICAL THINKING SKILLS By JOHN PATRICK DENNY A DISSERTATION PRESENTED TO THE GRADUATE SCHOOL OF THE UNIVERSITY OF FLORIDA IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY UNIVERSITY OF FLORIDA 2012
2 2012 John Patrick Denny
3 To my mother, Beverly Ann Denny
4 ACKNOWLEDGMENTS First and foremost, I must express my sincere appreciation to my wife, Christine. Her love, support and encouragement were constant throughout this challenging process. She never doubted my abilities and was always there to make me smile and persevere. This study would not have been possible without the many weekends and evenings she spent wrangling ch ildren on her own I also thank my children, Ella and Tyson who served as a source of inspiration while keeping me grounded and never letting me forget what is truly important in life. I am grateful to my dissertation committee members at the Universit y of Florida: co chairs, Dr. David S. Honeyman and Dr. Dale F. Campbell as well as Dr. David Miller and Dr. Albert Matheny. Their efforts in assisting me through this process were much appreciated. I also wish to thank Dr. Cyndi Garvan whose guida nce, teaching and encouragement were particularly important in completing this study. I have been fortunate to work with many fine colleagues over a long career at the University of Florida many of whom have offered guidance and words of support to complete th is process. In particular, I wish to thank my supervisor at the University Honors Program Dr. Kevin P. Knudson. His encouragement and understanding have been invaluable. I must also recognize my colleague Dr. Jeanna Mastrodicasa. She was always there to help me clarify my thoughts and offer her support. Navigating the complexities of this experience would have been much more difficult without her. Finally, I would like to extend my sincere appreciation to my friend Dr. James H. Crozier. His wise cou nsel, wit and words of reassurance sustained me throughout this process.
5 TABLE OF CONTENTS page ACKNOWLEDGMENTS ................................ ................................ ................................ .. 4 ABSTRACT ................................ ................................ ................................ ..................... 8 CHAPTER 1 INTRODUCTION ................................ ................................ ................................ ....... 10 Theoretical Framework ................................ ................................ ........................... 14 Purpose of this Study ................................ ................................ .............................. 15 Research Questions ................................ ................................ ......................... 15 Hypotheses ................................ ................................ ................................ ...... 16 Definition of Terms ................................ ................................ ................................ .. 16 Significance of the Study ................................ ................................ ........................ 19 Limitations ................................ ................................ ................................ ............... 20 Summary ................................ ................................ ................................ ................ 21 Organization of the Study ................................ ................................ ....................... 21 2 LITERATURE REVIEW ................................ ................................ ............................. 23 Student Involvement Theory ................................ ................................ ................... 23 Evolution of Undergraduate Research Programs ................................ ................... 25 Overview of the Literature on Undergraduate Research ................................ ......... 27 Ways of Measuring Critical Thinking ................................ ................................ ....... 36 Potential Benefits for Colleges and Universities ................................ ..................... 41 Summary ................................ ................................ ................................ ................ 42 3 METHODOLOGY ................................ ................................ ................................ ...... 44 Research Questions ................................ ................................ ............................... 45 Design of the Study ................................ ................................ ................................ 45 The Participants ................................ ................................ ................................ ...... 46 Operational Definition of Variables ................................ ................................ ......... 46 Instrumentation ................................ ................................ ................................ ....... 47 Survey of Experiences in the Young Scholars Program ................................ ... 47 California Critical Thinking Skills Test (CCTST) ................................ ............... 48 Data Collection ................................ ................................ ................................ ....... 51 Procedure ................................ ................................ ................................ ............... 51 Data Analysis ................................ ................................ ................................ .......... 52 Statistical Analysis ................................ ................................ ................................ .. 52 4 DATA ANAYLSIS AND RESULTS ................................ ................................ ............ 55 Estimating Internal Consistency Reliability ................................ ............................. 55
6 Analysis of Descriptive Statistics ................................ ................................ ............ 56 Demographics ................................ ................................ ................................ .. 56 Descrip tive Statistics for Academic Performance and CCTST scores .............. 57 Descriptive Statistics for Educational Outcomes ................................ .............. 57 Descriptive Stat istics for Semesters of Experience ................................ .......... 58 Descriptive Statistics for Research Experiences ................................ .............. 59 Analysis of Research Question 1 ................................ ................................ ............ 59 Analysis of Research Question 2 ................................ ................................ ............ 61 Analysis of Research Question 3 ................................ ................................ ............ 62 Analysis of Research Question 4 ................................ ................................ ............ 64 Analysis of Research Question 5 ................................ ................................ ............ 66 Analysis of Research Question 6 ................................ ................................ ............ 67 Summary ................................ ................................ ................................ ................ 69 5 DISCUSSION ................................ ................................ ................................ ............ 70 Overview ................................ ................................ ................................ ................. 70 Conclusions ................................ ................................ ................................ ............ 71 Critical Thinking Skills ................................ ................................ ....................... 71 Undergraduate Research Experiences and Educational Outcomes ................. 75 Limitations ................................ ................................ ................................ ............... 77 Future Research Recommendations ................................ ................................ ...... 78 Implications for Hig her Education ................................ ................................ ........... 79 APPENDIX A INVITATION TO PARTICIPATE AND INFORMED CONSENT ................................ 83 B SURVEY OF EXPERIENCES IN THE YOUNG SCHOLARS PROGRAM ................ 86 LIST OF REFERENCES ................................ ................................ ............................... 9 4 BIOGRAPHICAL SKETCH ................................ ................................ .......................... 102
7 LIST OF TABLES Table page 4 1 ................................ ....... 56 4 2 Major Categories and Actual Majors ................................ ................................ ....... 56 4 3 Descriptive Statistics for Academic Performance and California Critical Thinking Skills Test Scores ................................ ................................ ............................... 57 4 4 Frequency of Educational Outcomes ................................ ................................ ...... 58 4 5 Semesters of Research Experience ................................ ................................ ........ 59 4 6 Descriptive Statistics for Research Experiences ................................ ..................... 59 4 7 CCTST Scores Compared by Gender ................................ ................................ ..... 60 4 8 CCTST Scores Compared by Major Category ................................ ........................ 60 4 9 CCTST Scores Compared by Semesters of Experience ................................ ......... 61 4 10 Relationships Between Research Experiences and CCTST Scores ..................... 63 4 11 Relationships Between R esearch Experiences and Submitting Findings to a Peer Reviewed Journal ................................ ................................ ...................... 66 4 12 CCTST Correlations with Academic Performance ................................ ................ 67
8 Abstract of Dissertation Presen ted to the Graduate School of the University of Florida in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy THE RELATIONSHIP BETWEEN UNDERGRADUATE RESEARCH AND CRITICAL THINKING SKILLS By John Patrick Denny May 2012 Chair: Dale F. Campbell Co chair: David S. Honeyman Major: Higher Education Administration Undergraduate research programs have the potential to enhance the critical thinking skills of student researchers While many college administrators gene rally view these programs as a positive experience, there is still relatively little research that provides an objective assessment of outcomes associated with undergraduate research. The ability to document educational outcomes of college students, in pa rticular critical thinking skills is especially important given the increased emphasis on accountability by college adminis trators, legislators and the public The purpose of this study was to examine the relationships between participation in an underg raduate research program, critical thinking skills, common educational outcomes and academic performance. The study examined whether factors such as field of study, specific experiences in the research program and length of time engaging in undergraduate research had any effect on the critical thinking ability and/or educational outcomes of students In addition, the study looked at correlations between academic performance as measured by standardized test scores and grade point average and critical thi nking ability.
9 The primary significant relationship found, using a Wilcoxon Rank Sum was that students who had four or more semesters of experience with undergraduate research tended to have higher scores on the California Critical Thinking Skills Test (C CTST) than those with three or less semesters of experience. A Spearman Correlation Coefficient indicated that a ll of the measures of academic performance (SAT scores, ACT scores and GPA) were correlated with CCTST scores. The two self reported research experiences that correlated with CCTST scores were Collaborati ve Learning and Active Learning. Three correlations between educational outcomes and resea rch experiences were observed; Collaborative Learning, Active Learning and Time on Task were all correl ated with submitting an article for publication. The findings of this study provide several implications for higher education administrators. Undergraduate research programs that enlist student s early in their college career s and that encourage collabora tive learning, may result in enhanced critical thinking ability and positive educational outcomes.
10 CHAPTER 1 INTRODUCTION The declining quality of undergraduate education has been a concern for educators, administrators and policy makers for well over t hirty years (Pascarella & Terezini, 2005). As the United States competes in an ever increasing global economy, greater attention has been focused on the skills gleaned by college students during their undergraduate experience. In 1983, a report commissio ned by the Secretary of Education entitled A Nation at Risk lacked higher order thinking skills that allowed them to draw inferences from written United States National Commission on Excellence in Education 1983). A response to this report was the highly influential opus, Reinventing Undergraduate Education often referred to as the Boyer Commission report. While the report made by the Boyer Commission did spur reform in many colleges and univer sities, it did not silence the alarm being sounded by researchers and policy makers. The Spellings Commission pointed out that the United States no longer held the lead in access to higher education for its citizens A lmost a dozen countries had surpasse d the United States in educating greater numbers of their citizens at more advanced levels. The report not only bemoaned the declining numbers of Americans who were attaining post secondary education, it stated that those who could access college were not getting the quality of education they deserved. Specifically, the report charged that employers were reporting Recommendat ions on how to improve the state of the higher education system included a commitment to implement new teaching methods which embraced the use of
11 technology while placing emphasis on objective, measureable results of educational outcomes (U.S. Department o f Education, 2006). As pointed out in the Spellings Commission report, critical thinking is widely accepted as a desirable outcome of undergraduate education (Astin, 1993, Bok, 2006, U.S. Department of Education, 2006). The emphasis on thinking in modern educational practice can be traced to educational reformer John Dewey (Dewey, 1910 as cited in Giancarlo & Facione, 2001). Dewey expressed his concerns about the mechanization of education through lecture and memorization. He contended that students were not intellectually educated unless they gained the ability to distinguish evidence based beliefs from conjecture and opinion. He championed the idea of learning based on p 27 28, 34). Over 80 years later, Robert Reich would describe the qualities of a highly valued worker in the 21 st century. In his widely read book, The Work of Na tions Reich defined the symbolic analyst. This modern professional would have many qualities commonly associated with critical thinkers. The symbolic analyst would have a command of abstractions meaning that he or she would be adept at identifying patte rns and analyst ; in addition to offering a solution to a problem, they would be able to asse ss why the problem arises and examine how the problem may be connected to other problems. A symbolic analyst will be able to test ideas from a variety of angles and perspectives.
12 The use of judgment and interpretation will allow for the visualization of new possibilities and choices. Finally the symbolic analyst will possess a willingness to collaborate and the ability to communicate in an articulate a nd precise fashion. He or she will be able to identify problems and find answers, seek and accept criti cism and appreciate a point of view from other perspectives (Reich, 1991). analyst could also be applied to an undergraduate researcher. Today, students and educators face the challenge of learning and teachin g in a technologically advanced and continuously evolving world. Vast amounts of new and archived knowledge can be easily disseminated through the Internet (Leiner, 1999). The widespread use of mobile devices combined with Internet penetration of 76 per cent in the United States has resulted in an unprecedented access to information (Jones, 2011). Thus, students and faculty have the opportunity to keep pace with the latest advances in any given discipline. While access to this wealth of data has many pot ential benefits in education and commerce, it requires users to have the ability to evaluate information sources and context to determine its reliability. Perhaps the best skill that institutions of higher education can offer its students is the ability t o process information and then use good judgment about how to best use that information (Astin, 1993 p. 47, Bok, 2006 p.109). This complex cognitive process is commonly referred to as critical thinking (Giancarlo & Facione, 2001). Expanding upon this definition, Rand states that people who are criti cal thinkers: Raise vital questions and problem s formulating them clearly and precisely gather and assess relevant information, using abstract ideas to interpret it effectively come to well reasoned conc lusion and solutions, testing them against relevant criteria and standards
13 think open mindedly with alternative systems of thought, recognizing and assessing, as need be, their assumptions, implications and practical consequences communicate effectively with others in figuring out solutions to complex problems (Rand, 2008) Designing and implementing programs that allow for the development of critical thinking skills is a fundamental question in higher education. One potential answer may be found through undergraduate research programs. Undergraduate research programs are increasing nationwide. In a joint statement, the Council on Undergraduate Research and National Conferences on 21 st curriculum ( National Conferences on Undergraduate Research 2011 ) Undergraduate Research Programs replace the traditional hierarchy of professor and stude nt with that of a collaborative learning effort that benefits both students and faculty (National Conference on Undergraduate Research, 2005). These programs have increased across all institutions and disciplines from the mid les, 2007). Undergraduate research gained prominence in1998 after the Boyer Commission called for inquiry based study and undergraduate research to become an integral part of teaching at research universities. This call for reform was to counter the This shortchanging was the result of pedagogical techniques that failed to engage students and did not allow for the development of critical thinking skills (Kinkead, 2003). The commission suggested that students who participated in research experiences
14 would become active learners and have an easier transition to graduate programs. Those students that opted for the workforce rather than graduate or professional education wou ld benefit from developing abilities to define, examine and offer solutions to complex problems. These students would become more attractive to potential employers and become more engaged and active citizens (Boyer, 1998). Theoretical Framework This study Student Involvement Theory as its theoretical framework. 1984 pp. 307 ). In volvement can take many forms including academ ic work, participation in extra curricular activities and interactions with faculty. Undergraduate research programs combine all of these qualities. By the nature of research activities, these programs typica lly attract students who are highly motivated and will invest the necessary time and diligence to succeed. Students who participate in undergraduate research are afforded many ways in which they can invest their time including but not limited to, selectin g a research topic, collecting and analyzing data, compiling a literature review, writing up results and preparing results for presentation. As such, the theory fit well in providing an underpinning for this study and provide d guidance in the development of the research questions The theory suggests that students who are most involved in their college education are the ones who will achieve the greatest level of student learning and personal growth. For the purpose of this study, student learning and pe rsonal growth will be measured by scores on the California Critical Thinking Skills Test and by the achievement of common educational outcomes associated with undergraduate research.
15 Purpose of this Study The purpose of this study is to examine correlati ons between participation in an undergraduate research program and critical thinking skills. Further, the study will Seven Principles of Good Practice in Undergraduate Education in an u ndergraduate research program are correlated with critical thinking ability (1991). A secondary purpose of the study is to determine if factors such as field of study, gender, or length of time engaging in undergraduate research have any effect on the cr itical thinking ability of students and/or common educational outcomes associated with undergraduate research programs. The educational outcomes of interest for this study are writing a senior thesis, attendance at a professional academic conference and s ubmitting a manuscript for publication in a peer reviewed journal. These outcomes have been described as It will also examine the relationship between academic measures such as standardized test scores and grade point average with critical thinking skills. Research Questions undergraduate research and critical thinking skills as measured by the California Critical Thinking Skills Test? undergraduate research and outcomes of undergraduate research? Are their particular experiences with undergraduate research that is correlated with critical thinking skills? Whi ch independent variables are correlated with outcomes of undergraduate research? Does academic performance as measured by standardized test scores and grade point average correlate with critical thinking skills?
16 Does academic performance as measured by standardized test scores and grade point average correlate with outcomes of undergraduate research? Hypotheses Critical thinking ability will not be influenced gen der, but will be influenced by prior experience conductin g research. undergraduate research. Prior experience with undergraduate research will have a positive influence on outcomes of undergraduate research. Students whose experiences include t Principles of Good Practice in Higher Education will be more likely to write a senior thesis, present at a professional conference or publish research in a peer reviewed journal upon completion of an undergraduate researc h program. Definition of Terms For the purposes of this study the following terms are defined : Active Learning This term is defined investment in the research experience. Participants were asked about how much their personal opinion s were valued as the project progressed, the confidence they had in explaining their projects to others, the frequency with which they were expected to report to their mentor and/or research team and if they were encouraged to sugges t new ideas for research projects. Collaborative Learning This term i s defined as working as part of a research team. The research team may be composed of undergraduate students, graduate students or a mixture of both. Those participants who were part of research teams were asked about the quality of the interactions with members of the team and the role the team played in the overall research experience.
1 7 Critical Thinking This term regulatory judgment that results in interpretation, analysis and evaluation and inference, as well as explanation of the evidential, conceptual, methodological, criteriological or contextual Facione, 1990 p.2 ). In this study, critical th inking refers to the combination of cognitive skills measured by the Californi a Critical Thinking Skills Test (CCTST). Diverse Learning Th is term is defined as if they felt like their mentor encouraged creative thinking, welcomed other points of view and felt like there were multiple ways to solve a particular problem. Participants were also asked about their comfort level in asking questions. Educational Outc omes. This refers to specific, measurable outcomes as a result of participating in the undergraduate research program. For the purposes of this study, the products include: senior thesis, publication in a peer review ed journal and presentation at a profes sional academic conference. Mentor Expectations This term is defined as how the research mentor evaluated d additional reading or writing tasks related to the research project. They were also asked about outcomes such as writing a senior thesis, presentation of findings at a professional conference and submitting finding s to a peer reviewed journal. Prompt F eedback This term is defined as the quality and timeliness of performance feedback from the research mentor and/or the research team. Participants
18 were asked about the frequency with which they received detailed evaluations of their effort and if they we re required to maintain a log or record of their progress. Participants were also asked about the accessibility of their mentors with regard to answering questions as well as the timeliness with which questions were answered. Student Faculty Contact Thi s term is defined as the amount of time the participant spent interacting with his or her faculty mentor. The quality and depth of these interactions was assessed by asking if the mentor offered advice beyond the scope of the research project and if the m entor shared personal experiences and values with the participant. The participant was also asked if the mentor encouraged questions as well as the likelihood of continuing research with the mentor upon completion of the undergraduate research program. T ime on Task This term is defined as the amount of time spent conducting research. Students were asked about the average hours per week spent on the project Undergraduate Res earch Program This is defined as a year long program in which undergraduates are paired with a faculty mentor to conduct research. Participants voluntarily apply for this extra curricular activity. The selection process to participate in the program is competitive. Standardized Test Scores This term refer s to the Scholastic Aptitude Test (SAT) and the Academic Achievement Test (ACT) Grade Point Average This term average at the end of the semester in which the undergraduate research program was completed.
19 Significance of the Study participation in an undergraduate research program, they often fall short of providing evidenc e based outcomes (Seymour, 2004). The Spellings Commission encouraged higher education institutions to measure and report meaningful student outcomes (U.S. Department of Education, 2006). This study has the potential to provide one of these measures. Th e study will specifically target critical thinking using an established psychometric instrument. The results will provide some insight about the potential enhancement of a transferable skill while giving faculty and administrators information about how un dergraduate research programs might give the university a measureable educational outcome. This research also has the potential to inform faculty about the potential benefits students receive when they are invited to join them as colleagues in the discove ry of know ledge. The findings examine potential differences in critical thinking outcomes among the academic disciplines, gender of the participants and length of time in the research program. The findings also comment on differences in experiences among the academic disciplines. Finally, they give faculty some insight about the level of expectation they should place upon their students by examining whether there is a relationship between deliverables from the research experience and its impact on critic al thinking. Effective undergraduate research programs invo lve a sig nificant investment from university administration ; i nfrastructure, faculty time as well as a financial commitment are all important considerations when implementing a research program. Access to a state of the art research facility is often listed by faculty as an essential feature of a quality undergraduate r esearch program (Lopatto, 2003). Developing advanced
20 research skills in undergraduates, particularly by engaging them in an inte nsive full time summer program, is far more costly and time consuming than using students as research technicians (Kremer, 1990). The cost of materials as well as the time required to properly mentor an undergraduate researcher can be a deterrent to facul ty participation (Stevens and Reingold 2000; Zydney et al. 2002). In a survey of research mentors, Zydney found that 50 percent of the faculty spent between 3 and 5 hours per week mentoring undergraduates (2002). Furthermore, this investment in mentorin g is not often accounted for by universities as they evaluate the productivity of their faculty members (National Science Foundation, 1989). Evidence based research on the outcomes of undergraduate research programs can help administrators make decisions about the human and fiscal resources the university and external funding agencies need to commit to launch and maintain these often costly programs (Kardash, 2000). Limitations The following limitations may affect the data collected by this study: Data wi ll be collected from only one undergraduate research program at one university Findings may be specific to the institution and to the specific program which was studied. G eneralization of the results will be limited The undergraduate research program th at will be studied is of a competitive nature. Paid stipends are disbursed to students who frequently have more than one semester of research experience. Results may differ from non competitive programs or programs that are designed for students with no prior research experience. The small sample size resulted in participants grouped into four broad major categories. A larger sample size may have allowed for more precise groupings of resulting in a closer examination of each indiv experiences. students who have completed an undergraduate research program. It is likely that these students, who have all voluntarily chosen to participate in thi s endeavor
21 already had well developed critical thinking skills prior to beginning the undergraduate research program. Summary Educational researchers, administrators and politicians have all questioned the effectiveness of undergraduate education in recent decades. Questions have been rapidly evolving and technically advanced workplace. The chapter began with a brief overview of some of the major calls for educational re form and continued with a discussion of critical thinking as an important educational outcome for college students It was suggested that participation in undergraduate research programs may be an effective way to enhance critical thinking skills. These programs are generally seen as valuable experiences that enhance the undergraduate experience. Effective programs exact a cost in university resources as well as faculty time. Therefore, documentation of the outcomes of participation in these programs is important. The findi ngs of this study are one source of objective data related to the educational outcomes of undergraduate research The study also provides suggestions for best practices when designing undergraduate research programs. Organization of t he Study Chapter 1 in this study includes an introduction, the purpose of the study, the research questions, hypothesis, a definition of terms, significance of the study and its limitations. Chapter 2 features a review of the literature including a brief history of the evolution of undergraduate research programs, an examination of descriptive and evidence based benefits of undergraduate research, the importance of critical thinking as an outcome of undergraduate education as well as a review of instrument s used to
22 measure critical thinking. Chapter 3 provides a detailed description of the research methodology employed and a detail ed description of the survey and instruments. Analysis of data and results are covered in Chapter 4. Conclusions and recommen dations are presented in Chapter 5.
23 CHAPTER 2 LITERATURE REVIEW The litera ture review is divided into five parts. It begins with a brief discussion of the theory used to guide the study and continues with a discussion of the development of institution wi de undergraduate research programs and professional organizations related to these activities It then examines the descriptive and evidence based benefits of undergraduate research. I nstruments used to measure critic al thinking will be discussed. The l iterature review will conclude with a discussion of how colleges and universities may benefit from supporting institutionalized undergraduate research programs. Student Involvement Theory Student Involvement Theory provides a simple but powerful framework for planning effective educational programs. The central hypothesis of the theory is that educational practices that intentionally increase student involvement are the most effective in facilitating learning and personal development (Astin, 1999) Involv ement theory is not unique to a specific discipline. The academic subject is not important; effectiveness is gauged by the motivation and investment of the student. Involvement incorporates a variety of activities, many of which are pertinent to undergra duate research. Examples include immersion in academic work, interaction with faculty, collaboration with peers and graduate students and preparation for oral and written presentations of findings. The research questions for this study are guided in part by uggestions for further research: D o particular forms of involvement facil itate student development ? A re student characteristics related to different types of
24 involvement? D oes a specific form of involvement produce different outcomes for dif ferent types of students (Astin, 1999)? Student Involvement Theory and the decades of research that led to its Seven Principles for Good Practice in Undergraduate Education nciples which Chickering invited Astin and other researchers with expertise in the college experience to compose a statement of principles that could be easily disseminated among the academic community. The resulting principles were based on tenants of Student Faculty Inventory and Institutional Inventory as an easy way for faculty and institutions to gauge their adherence to the seven principles. The Survey of Experiences in the Young Scholars Program one of the instruments used in this study is based upon items included in Ch Faculty Inventory The participatory quality of u ndergraduate research programs allows for students to have meaningful interactions with faculty mentors, learn collaboratively, play an active role in their education and receive a critique of t heir performance. Participants have the opportunity to meet high expectations placed by their mentors by investing significant amounts of time and energy to their research. Because of these qualities, some researchers have referred to undergraduate resea rch as the epitome of engaged learning (Lopatto, 2006).
25 Evolution of Undergraduate Research Programs Several prominent criticisms of the quality of undergraduate education helped shape research programs on college campuses. College: The Undergraduate Experience in America concluded that u ndergraduates were more likely to express dissatisfaction with the quality of teaching at large research universities compared to their peers at other types of institutions These students reported that professors wer e less likely to take a personal interest in their academic progress, less likely to encourage students to express their opinions on important issues and were less likely to believe that professors took a personal interest in their academic progress (Boyer 1987) This report was followed by Scholarship Reconsidered (1990) in which Boyer promoted the s cholarship of discovery, daring teaching versus research debate and define in more creative ways what it means t o be based learning the standard 15 16) in Universities (1998). (Dewey, 1910), the Boyer Commission made the case for faculty, graduate students an d undergraduate students to join together in the research process. They assert that this collaborative approach benefits undergraduates by stimulating interest in the discipline by linking concepts with applied problem solving. Graduate students benefit by honing their teaching skills and learning to be effective mentors. Faculty members profit from exposure to new ideas through interaction with the grads and undergrads. isions
26 these suggestions through the implementation of formalized undergraduate research programs. Institution wide undergraduate research programs debuted on college campuses over 40 y ears ago. Founded in 1969, the Undergraduate Research Opportunity Program (UROP) at the Massachusetts Institute of Technology was one of the earliest undergraduate research programs in the United States The UROP plays a large role in facilitating the cu lture of research on campus with approximately 80 percent of students and more than 45 percent of faculty participating in the program. Ten years later, the Summer Undergraduate Research Fellowships (SURF) program was founded at the California Institute skills by encouraging student faculty collaboration on research proposals. The program also has high expectations with regard to communication skills. Participants are expected to write technical papers and give oral presentations at a SURF seminar day. As a result of this expectation of excellence, approximately 20 percent of students coauthor articles in peer reviewed journals or present at a professional conference in their discipline (Merkel, 2003). Other universities with early institution wide programs included Stanford, Rutgers and the University of Delaware (Bauer & Bennett, 2003). Development of professional associations has enhanced the quality of undergraduate research program s by facilitating communication with agencies and foundations to promote research opportunities for faculty and students. The Council on Undergraduate Research was established in 1978 to provide like minded faculty members with a sense of community and a forum to share best practices This not for
27 profit educational organization serves as an advocate for faculty and students who are engaged in research and provides information to state and federal government as well as private foundatio ns on the importan ce of these experiences (Halstead, 1997). While the CUR provides higher level support and advocacy for its members, the National Conferences on Undergraduate Research provide a forum that celebrates undergraduate researchers by hosting an annual conferenc e attended by nearly two thousand faculty and s tudents (Kinkead, 2003). While professional associations were promoting the merits of undergraduate research, o pportunities for funding were significantly expanded in the mid 1980 with the establishment of th e Research Experiences for Undergraduates program by the National Science Foundation (Merkel, 2003). Undergraduate research programs gained a solid foothold on college campuses once t he concept of integrating teaching and research gained momentum after t he report of the Boyer Commission on Educating Undergraduates in the Research University was published in 1998 (Hu et al ., 2007). Since that time, many descriptive accounts, promotional and discussion articles and histories of undergraduate research exper iences have been published, yet few provide evidence based findings to support the benefits of undergraduate research (Seymour et al. 2003). Overview of the Literature on Undergraduate Research The interest in learning more about undergraduate researc h outcomes is gaining momentum, particularly within the last 15 years. The National Science Foundation and the Howard Hughes Medical Institute have both funded three large research efforts to help define the benefits of these programs (Guterman, 2007). W hile rigorous examination of undergraduate research is beginning, there is little consensus on the
28 criteria used to assess outcomes of undergraduate research programs (Davis & Glazer, 1997). Previous research has attempted to define outcomes and measure t he extent to which the participants benefited, yet relatively few studies have produced evidence based benefits of these outcomes. Much of the literature on undergraduate research is anecdotal; descriptions of educational programs using a research based a pproach and the development of critical thinking abilities are often vague and not well supported (Seymour, 2004). If undergraduate research is to earn a permanent place in the curriculum, it must be subjected to further research to determine the specific benefits for students as well to inform faculty of best practices in developing and improving programs. Involving students in research programs holds great potential as an effective way to enhance the undergraduate experience Undergraduate research has been hailed as (Crowe, 2006). Research on the outcomes of undergraduate research is rapidly expanding In a qualitative research study involving 76 students with undergr aduate research experiences Seymour (2004) found that a great majority, 91 percent, report a positive experience that involved personal growth with many transferable benefits including gains in basic research skills, critical thinking and problem solving skills, oral communication of findings, and enhanced comprehension of the nature and deve lopment of scientific knowledge. Similar results were described by Mabrouk and Peters (2000) in which 320 students majoring in physical or biological sciences were s urveyed. Nearly all students, 98 percent, related a positive experience that they would recommend to a friend.
29 In an even larger study involving 4500 students who participated in research programs sponsored by the National Science Foundation, Russell (2 007) found that 83 percent of students described increased confidence in their ability to conduct research and 68 percent reported greater interest in the fields of science, technology, engineering and mathematics. Similar findings were reported by Kremer and Bringle. In a well des ig ned study the researchers compared 22 students who participated in a 10 week, full time intensive summer program, with a control group of students who were matched in the following categories: field of study (all were psycholo gy majors), GPA, SAT score and score on the advanced psychology test on the Graduate Record Exam. When compared to the control group, the students who participated in the research program reported a higher level of research abilities. Research abilities included conducting a literature review, use of computer software to analyze statistics, writing results and writing discussion. In addition, they were more likely to choose a research oriented career and they were more likely to be accepted to research intensive graduate programs (Kremer and Bringle, 1990). Undergraduate research experiences are a unique blend of self directed inquiry and student faculty collaboration (Nagda, 1998). Pascarella and Terenzini (2005) report that undergraduate research expe riences provide the following: active involvement in their own learning, increased and more meaningful inte raction with faculty members, opportunities to apply course related theory and skills in solving real problems, and a challenging intellectual activity (p. 406) Hakim noted that all undergraduate research experiences share four main characteristics: mento rship, originality, acceptability and dissemination. According to
30 Hakim, undergraduate research involves frequent interaction between the student researcher and faculty mentor, the opportunity for the student to make a meaningful contribution to the field of study, the use of research techniques that are the standard of the discipline and culmination of the experience by sharing their findings with the scientific community. Participation in undergraduate research allows s tudents to experience an initiati on into the discipline, make a contribution to the discipline and then have an integra tion of skills and competencies (Hakim, 1998). This integration is apply knowle dge and skills to a specific research problem (Seymour, 200 4 ). Students may experience a greater understanding of the nature of science. Using qualitative research methods, Ryder, Leach and Driver interviewed 11 students in the physical or biological sci ences who had recently completed a 5 8 month research project under the supervision of a faculty mentor. The students were interviewed at the beginning of their project and again when the project was near completion. The researchers found that the majori ty of the students believed that knowledge claims must be established on empirical grounds. They also developed their understanding of how theoretical developments within a discipline influences scientific inquiry (Ryder, et al., 1998). All of the element Seven Principles of Good Practice in Undergraduate Education (1991) may be applied in undergraduate research settings. In particular, faculty student contact, high faculty expectations and prompt feedback seem to be especially well suited for this endeavor. Not surprisingly, evidence in the literature suggests gains in academic persistence and degree completion, primarily when the students develop a strong mentoring relationship with a faculty member
31 (Seymour, 2004). Mentors hip established through collaborative research exemplifies active ctual maturity (Ei gren & Hensel, 2006). Mentoring relationships with faculty members have a significant impact engagement with the college or university and have been cited as a key predictor of student retention and academic performance (Kuh, 2001). These relationships also allow faculty to offer advice on achieving academic goals, facilitate professional networ with publications and presentations (NCR, 2003) The literature describes many additional benefits of participating in undergraduate research. Enhanced retention has been reported, parti cularly with underrepresented students in the sciences (Nagda, Gregermann, Jonides, von Hippel & Lerner, 1998). In addition, minority students who advanced to post baccalaureate study and participated in undergraduate research were more likely to enter me dical school or law school than similar students who did not. Survey responses from a randomized sample of alumni who participated in an undergraduate research program were compared to a similar group of students who did not. The sample groups were match ed on the basis of field of study, race or ethnicity, date of graduation and cumulative grade point average. The study found that former students who participated in either a formalized undergraduate research program or in some other form of research prog ram were more likely to attend graduate school, were more likely to attend professional school, were more likely to continue in some sort of research activity post graduation, were more likely to have received a recommendation from faculty and were more l ikely to keep in contact with faculty at the undergraduate institution tha n those who had no undergraduate research
32 experience. Even stronger effects were noted from underrepresented students of color (Hathaway, Nagda and Gregerman, 2002). In a similarly designed study of alumni who participated in undergraduate research experiences, Bauer and Bennett report that 67% of those surveyed furthered their education beyond a baccalaureate degree compared to 57% of the alumni who did not participate in an under graduate research experience. This difference is even more pronounced when considering alumni who went on to obtain a doctoral degree; 43% of alumni with undergraduate experience earned their doctorate compared to 23% of alumni who earned doctorates with no undergraduate research experience (Bauer et al 2003). These results are consistent with those found by Russell (2007) in which the research experience helped students who had not considered attending graduate school to alter their education goals and apply for a Ph.D program. I ncreased interest graduate school enrollment has also been demonstrated with underrepresented minorities Using archival research methods, Foertsch, observed that 52 percent of minority students who participated in a summer un dergraduate research program went on to enroll in graduate studies and 23 percent enrolled in professional schools. Although a matched sample was not utilized in this study, this group far exceed ed the national average of 8.8 percent enrollment for minori ties in graduate school and 8.2 percent enrollment in professional schools during a similar time frame (Foertsc h, et al. 1997). Using the Survey of Undergraduate Research Experiences (SURE), Lopatto found that minority students who participated in a summ er research experience were more likely to choose a career in the sciences. He also observed greater learning gains than comparable non minority students ( 2004). Undergraduate research participation
33 has also been reported to have a positive effect on suc cess in graduate school for first generation, low income and underrepresented minorities (Nnadozie, Ishiyama & Chon, 2001). Undergraduates who participate in research gain a unique opportunity to develop critical thinking skills, increase persistence, incr ease the chances of graduate school study and help to define future career choice s (Pascarella & Terenzini, 2005). Other benefits of undergraduate research include increased intellectual curiosity, the abilities to acquire and analyze information indepen dently, comprehension of scientific findings, enhanced oral presentation skills and leadership abilities (Bauer & Bennett, 2003). interpretation, interpret results within the larger context of the discipline and verbally communicate the findings of the research project (Kardash, 2000). Evidence for these findings was provided in a qualitative study by Carol Anne Kardash. In her study, a pre test/post test design was used with 57 undergraduates who were enrolled in a summer undergraduate research program. During the first week of the academic term, participants were asked to rate their ability to perform 14 skills related to research activities. The participants rated the ir abilities on the same 14 items once the term was complete. Kardash also surveyed asking them to rate their students using the same scale. The analysis looked for the convergence or divergence of the responses. The only skill that the students reported was enhanced included data collection, comprehension of modern concepts in their disciplines and d. The ability to write for scholarly publications
34 was reported as the skill that had been enhanced the least. Unexpectedly, Kardash found that more sophisticated skills including identifying a research question, developing a hypothesis, testing the hypo thesis and refining the hypothesis based on line with those of their faculty mentors (Kardash, 2000). Common objectiv es of undergraduate research include helping students comprehend the fundamentals of the research process, providing them with the skill set collaboratively with faculty and peers (Hancox & Shaw, 2006). Not surprisingly, one of the most common benefits of undergraduate research noted in the literature is the ability research report a greater ability to engage in research activities and complete research proposals than those who do not. This enhanced ability to engage in the research process is reported by males and females and from across disciplines (Bauer et al 2003). Faculty mentors re port that the ability to acquire information independently is an essential skill for students who participate in undergraduate research (Lopatto, 2003). The ability to conduct effective literature searches has been listed as a learning objective for under graduate research students (Hancox et al 2006). Research indicates that students who participate in undergraduate research benefit from an increased ability to obtain information autonomously and to analyze literature critically (Bauer et al 2003).
35 U ndergraduate research has the potential to give students a better understanding by Ares. Ares comments that all too often researchers do not have an adequate understanding of highly advanced machinery (which is often housed in beige boxes) used in the laboratory. By allowing his students to have a better comprehension of the what the instruments do, they will ultimately have a better understanding of what they are trying t o accomplish in the laboratory (Ares, 2004). Evidence also suggests that the length of time committed to undergraduate research has a positive relationship with the overall perceived benefit. A survey of alumni who participated in undergraduate research r e vealed that the experience was beneficial, regardless of the length of time involved with it. However students who devoted the longest time to undergraduate research reported the greatest benefit (Bauer & Bennett, 2003). In an extensive survey of 3,200 undergraduate researchers in the social, behavioral and economic sciences (SBES), students who had participated in research activities for 12 months or more reported increased confidence, understanding and awareness than those with less than 12 months expe rience. The same group reported that the research process was more influential in making career decisions (SRI International, 2005). Student Involvement Theory in that the students that invested the most energy rea ped the greatest benefit from the experience. Another learning objective for undergraduate research students is the ability to present research findings in both oral and written presentations (Hancox, 2006). In research done on participants of a REU pro gram, Shellito (2001) identified the ability to
36 produce a professional quality research project as a result of the technical writing expected of the students. Enhanced ability to speak effectively when presenting research projects has been reported as an outcome of participation in undergraduate research (Bauer & Bennett, 2003; Karadash 2000). Elements of critical thinking abilities have been described as outcomes of the undergraduate research experience (Seymour et al 2003). Bauer and Bennett found th at undergraduate research alumni experienced the largest gains in becoming intellectually curious, independent, logical thinkers and problem solvers (2003). All of the aforementioned qualities are described as characteristics of critical thinkers (Facione et al 1994). Hakim comments that students who integrate the skills and competencies necessary for undergraduate ther enhanced qualities related to critical thinking include growth in originality, creativity, initiative, curiosity and enthusiasm (Davis & Glazer, 1997). Students who participated in a summer undergraduate research experience exhibited higher level tho ught processes as quantified by the Measure of Epistemological Reflection (Magolda, 2001). While enhanced critical thinking skills have been described as a beneficial outcome, other studies make a less compelling claim. Karadash reports that while underg raduate research experiences are clearly successful in enhancing basic scientific skills, there is less evidence to demonstrate that these experiences promote higher order thought processes that make up the foundation of critical thinking (2000). Ways of M easuring Critical Thinking A large body of research on the development of critical thinking in college students
37 increases while in college, there is little agre ement on the factors that influence its growth Further, critical thinking is generally agreed upon as an important element of undergraduate education; few studies have tested the effectiveness of producing critical thinking as such an outcome (Tsui, 1998 ). In response to the emphasis on accountability in education over the last several decades, several instruments have been developed to assess critical thinking skills. In Glaser Critical Thinking Appraisal (WGCTA) meas ures critical thinking skills by utilizing an 80 item, multiple choice test with five subscales: inference, recognition of assumptions, deduction, interpretatio n and evaluation of arguments. While it is considered to be a well constructed instrument, revi thinking as being a combination of attitudes, knowle dge and skills (Gray, 1987) A lternative s to the WGCTA were developed by Robert Ennis. He coauthored both the Ennis Weir Critical Thinking Essay Test and the Cornell Tests of Critical Thinking. Ennis on Ennis, 1985, p. 8). The Cornell Test of Critical Thinking, Level Z is intended for use with college students and assesses critical thinking skills using a 52 item, multiple choice examination. The seven thinking abili ties include: detecting equivocal judgment, discovering various types of assumptions, evaluating the reliability of observations and judging the authenticity of sources (Gray, 1987). While the Cornell Test is another option to assess this skill set, resea rchers must note the technical construction of the WGCTA was rated above the Cornell Test by the American
38 Psychological Association (Woehlke, 1987). As its names suggests, the Ennis Weir valuate an argument using a written essay format. As stated in the Tenth Mental Measurements Yearbook, and assumptions, stating nking skills that (Tompkins, 1989). The report of the Boyer Commission along with other educational reform initiatives in the improving the quality of undergraduate education while increasing accountability These calls for change helped to ignite the critical thinking movement. The American Philosophical Association, Committee on Pre College Philosophy identified the need to investigate definition s of critical thinking and how it was assessed. A Delphi research study was conducted, initiated b y Peter A. Facione. Experts from the fields of critical thinking assessment, instruction and theory were gathered to form the 46 member Delphi study panel. most worthwhile contribution could be the articulation of a clear and correct conceptualization of cr p.6). The Delphi study produced a consensus statement to serve as a guide to critical thinkin g assessment and to provide a description of the ideal critical thinker: We understand critical thinking to be purposeful, self regulatory judgment which results in interpretation, analysis, evaluation and inference, as well as explanation of the evidentia l, conceptual, methodological, criteriological, or contextual consideration upon which that judgment is based. CT (critical thinking) is essential as a tool of inquiry. As such, CT is a liberating force in education and a powerful resource i n nal and civic life. While not synonymous with good thinking, CT is a persuasive and self rectifying human phenomenon. The ideal critical thinker is habitually minded in
39 evaluation, honest in facing personal biases, prudent in making judgments, willing to reconsider, clear about issues, orderly in complex matters, diligent in seeking relevant information, reasonable in the selection of criteria, focused on inquiry, and persistent in seeking results which are as precise as the subject and the circumstances of inquiry permit. Thus, educating good critical thinkers means working toward this ideal. It combines developing CT skills with nurturing those dispositions which consistently yield useful insights and which are the basis of a rational and dem ocratic so ciety. (Facione, 1990 p.2) In addition to this definition, the Delphi panel came to a consensus on a list of cognitive skills required of a critical thinker. These skills include: interpretation, analysis, evaluation, inference, explanation and self reg ulation (Facione, 1990 p. 6). Logically, Dr. Facione used this list of cognitive skills to develop subscales for the California Critical Thinking Skills Test (CCTST). First published in 1990, the CCTST is an instrument intended to measure critical thi nking in college aged individuals. It is a 34 item multiple choice exam in which test takers are required to complete a range of tasks including: interpreting or analyzing information presented in a narrative format or in charts or images, drawing correct and justified inferences, explaining the merits of a given evaluation or inference and determining why inferences represent strong or weak reasoning. Results of the CCTST are reported as a total score and five individual scale scores which include: ana lysis, inference, evaluation, inductive reasoning and deductive reasoning (Facione, 2010). To use critical thinking skills, one must invoke a core set of cognitive skills: analysis, interpretation, inference, explanation, evaluation and self regulation ( Giancarlo, 2001). However, these skills are but one factor when considering the concept of critical thinking. A more complete analysis of critical thinking must include a ges, situations to evaluate or decisions to render; this tendency to utilize critical thinking skills
40 is referred to as critical thinking disposition (Giancarlo, 2001). The California Critical Thinking Disposition Inventory (CCTDI) was developed two years after the CCTST. It also draws upon the definition of critical thinking established by the APA Delphi Report. The CCTST measures skills associated with critical thinking, the CCTDI measures disposition toward critical thinking (Facione, et al., 1994a). Giancarlo measured critical thinking dispositions in college students using a longitudinal study with a pre test/post test design. She administered the CCTDI to a group of freshmen enrolled at a private, Catholic university in 1992 and a group of senior s at the same institution in 1996. The self implied that critical thinking disposition either experience (Giancarlo, 2001). More recently, extensive studies that surveyed large populations of undergraduates suggest that college students are making little, if any gains in critical thinking skills. In the Wabash Study of Liberal Arts Education, Char les Blaich administered the Collegiate Assessment of Academic Proficiency (CAAP) Critical Thinking Test to over three t housand students at nineteen institutions. Developed by the ACT, this nationally recognized standardized test is intended for use by postsecondary institutions to assess learning outcomes, including critical thinking. Using a pre test/post test design, o ver the course of a six month period, the data indicated a 0.57 point change in critical thinking skills as measured by the CAAP.
41 more discouraging finding, sociologists, Arum and Roska used the College Learning Assessment (CLA) to survey over two thousand students to assess critical thinking skills. The CLA is also intended for use by posts econdary institutions to assess higher order competencies by: Presenting realistic problems that require students to analyze complex written responses to the tasks are evaluated to assess their abilities to think critically, reason analytically, solve problems and communicate clearly and cogently (Collegiate Learning Assessment, 2012). Their findings indicated that at least 45 percent of the participants exhibited no statisticall y significant change in critical thinking skills during their first two years in college (Arum, 2011). Potential Benefits for Colleges and Universities While participation in undergraduate research has the potential to enhance the perience, it also offers potential benefits to the institution by providing evidence of an additional educational outcome. One measure of the ual college rankings of U.S. News and World Report (Eigren, 2006). Undergraduate Research is also one of the items on the National Survey of Student Engagement ( National Survey of Student Engagement, 2007 ). Rising costs of higher education, increased glo bal competition for highly skilled workers and public demand for greater transparency of educational processes are all reasons for institutions to do a better job of clearly defining educational outcomes (U.S. Department of Education, 2006). According to the Spellings Commission, employers are reporting that many recent college graduates lack basic skills that are required in
42 the workplace including critical thinking (U.S. Department of Education, 2006). As a result of these shortcomings, the commission is calling for higher education to implement innovative teaching practices that encourage the use of technology to improve student learning outcomes (U.S. Department of Education, 2006). In addition to a richer experience and improved learning outcomes fo r institutions, u ndergraduate research has the potential to bridge the roles of the professor as a scientist and an instructor (Lane, 1996) Many consider these two roles to be discordant; a professor devoted to research could not possibly invest adequate time to teaching undergraduates. Since tenure relies heavily on publications and bringing in grant tenure portfolio (Sample, 1972) Undergraduate research has the pot ential to mitigate this gap between teaching and research by blending the two into a combined pedagogy; instruction and scholarship becoming parts of a shared process (National Conference on Undergraduate Research, 2005). Summary As one response to the dec lining quality of the college experience, u ndergraduate research programs have been evolving on college campuses for over forty years. While this educational experience is not new, assessment of the benefits of participation in these programs is still in its early stages. In the study of undergraduate research programs, multiple studies have concluded that the majority of students perceive these experiences in a positive light and feel they are an enhancement of their college experience. E ducational rese archers have documented a variety of benefits including, increases in retention rates, entrance to graduate or professional school (especially in underrepresented minorities) the ability to interpret data and the ability to apply
43 research findings within the larger context of the discipline. Enhancement of written and oral presentation skills has been observed Improvement in areas commonly associated with critical thinking has been reported including thinking creatively and logically, ability to evaluat e the credibility of a data source, problem solving, initiative and enthusiasm for the subject. Much of the data in the literature rely upon self report ing from the participants. Relatively few studies provide an objective measure of the effects of par ticipation in an undergraduate research program. Studies that have employed standardized instruments to evaluate critical thinking skills in undergraduates have shown modest gains or in some cases, no significant changes. Undergraduate research programs a re becoming a common feature on many college campuses as evidenced by its inclusion as a category on nationally recognized benchmarks such as the National Survey of Student Engagement T hese programs give colleges and universities another way to document an educational outcome and have the potential to significantly enhance the unde rgraduate experience.
44 CHAPTER 3 METHODOLOGY Undergraduate Research Programs are becoming an increasingly prevalent aspect of the undergraduate educational experience. These pr ograms often require a significant investment in university resources and time on the part of faculty mentors (Kardash, 2000). While many educators agree that participation in an undergraduate research program is beneficial, there remains very little obje ctive data in the literature to support this claim (Seymour, 2004). The purpose of this study is to determine if there is any correlation between critical thinking skills and educational outcomes. The experiences this study is Seven Principles of Good Practice in Undergraduate Education and include a) the student mentor relationship b) collaborative learning c) active learning d) prom pt feedback e) time on task f) high expectations and g) respect for diverse talents and different ways of learning (Chickering, 1991). The study will also explore relationships between critical thinking skills and educational outcomes gender, academic major, academic performance (as measured by standardized test scores and grade point average) and prior experience with undergraduate research. The educational outcomes of interest in the study are: presenting results at a professional co nference, submitting results to a peer reviewed journal and using the results as the basis of a senior thesis. The findings will add an objective measure to the literature that describes the outcomes of undergraduate research experiences and may offer som e insight into best practices for undergraduate research programs.
45 Research Questions The following questions were the focus of this study: and prior experience with undergraduate research and critical thin king skills as measured by the California Critical Thinking Skills Test? undergraduate research and educational outcomes of undergraduate research? Do particular experiences with un dergraduate research correlate with critical thinking skills? Which ind ependent variables are correlated with outcomes of undergraduate research? Does academic performance as measured by standardized test scores and grade point average correlate with criti cal thinking skills? Does academic performance as measured by standardized test scores and grade point average correlate with outcomes of undergraduate research? Design of the Study The goal of the study was to investigate the relationship between the depe ndent variable of the California Critical Thinking Skills Test (CCTST) total score and the independent variables of gender, major, academic performance and specific experiences in an undergraduate research program. The ability to predict educational outco mes (writing a senior thesis, publishing results in a peer reviewed journal and presenting at a professional conference) using the independent variables of specific experiences within the undergraduate research program was also evaluated. The study took place at a large Research Intensive Carnegie classification university in the Southeastern United States. The college is accredited by the Southern Association of Schools and Colleges.
46 The Participants This study surveyed participants in the Young Schol ars Program (this is a pseudonym), a year long program in which undergraduates are paired with a faculty mentor to conduct an independent research project. Students from all colleges and all disciplines are eligible to participate. The Young Scholars P ro gram is well established on campus, having been in existence for twelve years at the time of this study. It is facilitated by a Center for Undergraduate Research The CUR is charged with promoting undergraduate research as an integral part of the college experience and linking undergraduates with faculty mentors. The program receives over $400,000 in funding to sponsor 205 students each year. The selection process to participate in the program is competitive. Students submit research proposals to their colleges. Each college appoints a faculty committee to review all proposals and select the best ones to participate in the program. Students who were selected received a $2000 research stipend. The experience culminates in a presentation at an undergra duate research symposium and/or the submission of their findings for publication. Some students will become co authors with their faculty mentors in a peer reviewed journal, while others will submit a manuscript to an online journal maintained by the univ ersity. As part of the informed consent process, the participants gave permission to the researcher to access academic information including standardized test scores, cumulative grade point average and average number of earned credit hours per semester Operational Definition of Variables The dependent variables in this study are the critical thinking skills score as measured by the California Critical Thinking Skills Test (CCTST), standardized test scores (SAT and ACT), grade point average and the edu cational outcomes of
47 participation in the program. Specifically, the educational outcomes consist of: writing a senior thesis, presentation of the research at a professional conference and publishing the research in a peer reviewed journal as a primary o r secondary author. The with undergraduate research. Prior experience with undergraduate research is defined as the amount of time (number of semesters) the student has s pent conducting research under the supervision of a faculty mentor. Other independent variables collaborative research (working as part of a research team), active learning, prompt diverse talents and different ways of learning. For a detailed explanation of each research experience and a copy of the research experiences survey, please see Appendix A. Instrumentation Survey of Experiences in the Young Scholars Program Participants began the study by completing a 45 item questionnaire that was a mix of fixed choice and open ended questions. This survey was based upon Chickering and s Seven Principles for Good Practice in Undergraduate Education (1991). The survey was adapted from the faculty inventory developed by Chickering which was designed to help faculty members examine individual behaviors, institutional policies and practice s to determine if their teaching practices were consistent with the Seven Principles. The survey was divided into seven sections one section for each of The sections consisted of: contact with a mentor, collaborative learn ing, active learning, prompt feedback from the mentor, time on task,
48 high expectations of the mentor respect for diverse talents or ways of learning. assigned a numer ical value and a subtotal was calculated for each section. A final section asked participants about additional research related activities and career plans. The survey was piloted with a small group of undergraduate students as well as the director of th e Center for Undergraduate Research to determine clarity of the questions and ease of navigation with the online interface. Please see Appendix 1 for a copy of the survey questions. The survey was administered online via Survey Monkey ( www.surveymonkey.com ). California Critical Thinking Skills Test (CCTST) The CCTST is a 34 item instrument designed for college students and adults. Used most widely with college and university undergraduate students and community college students, the CCTST assesses the critical thinking skills identified and defined by the American Philosophical Association Delphi study (Facione, 2005 ). It reports the total critical thinking score and five subscale scores of a) analysis b) inte rpretation, c) evaluation, d) inductive reasoning, and e) deductive reasoning (Facione, 1990, Facione & Facione, 1994). Published in 1990, this was the first instrument developed based on the Delphi method and the critical thinking objectives of the Calif ornia State University system executive order 338 (Facione, 1990). The Delphi group selected 34 multiple choice items for their ability to discriminate critical thinking skills and their high item total correlations (Facione & Facione, 1994). This study u sed the CCTST Form 2000, the most recent version of the CCTST. The updated test form provides item contexts that are more broadly representative of the reasoning required to be a skillful critical thinker and updates these contexts with questions that ar e more
49 appropriate to the analysis, inference and evaluation expectations of the 21 st century (Facione, 2005). Norm referencing of the CCTST is based on the analysis of 781 tests administered to a representative sample of college students at a comprehensiv e state university. The test scores were distributed in a normal bell curve with a homogeneous variance. None of the students within the norm group had enrolled in a college level critical thinking course (Facione, 1991). Reliability and validity for the CCTST were established on the pilot instrument from a pretest/posttest, case/control study design that included a series of four simultaneous quasi experimental studies. The data for the pilot instrument was obtained from 1,196 college students at Califo rnia State University Fullerton. Internal consistency reliability estimates for pretest and posttest groups were calculated separately with the Kuder Richardson 20 formula. The KR 20 reliability estimate of the pilot experimental group ranged from .68 to .70 (Facione, 1990, Facione et al 1994). The Facione and Facione report ed that a KR 20 above .70 is considered sufficient in support of the alpha level for an instrument with multidimensional scales. They also state, Although a reliability coefficient of .80 is suggested for internal consistency on instruments intended to target a single, homogenous ability, lower coefficients are often seen in the case of dichotomously scored test s particularly those tha t test a multifaceted concept like critical thi nking. Reliability ranges of .65 to .75 have been suggested to be considered sufficient for placing confidence in instruments of this type (Nunnally as quoted by Facione and Facione, 2010, p. 34). The aggregated KR 20 estimate for the CCTST Form 2000 is between .78 and .82 (Facione et al 2010).
50 Because the subscales are interconnected and interdependent, the instrument authors do not recommend using the subscale scores on the CCTST to evaluate individual achievement. In addition, the reliability for decision making about an factor analysis aimed at parsing out the differences betw een the s k ills of inference, (Faci one, P.A., & Facione, 1994, p.15 ) Students who took a required college level critical thinking course were tested to provide construct validity for the CCTST. A .74 scor e increase, which was significant at a .008 level was noted across administrations using a pretest / posttest design. A matched pairs analysis revealed that the average student in the paired sample jumped from the 55 th to the 70 th percentile as compared to pretest scores with an average 1.45 point score gain (Lambert, 2007). The expert consensus of critical thinking as defined by the APA Delphi Report provides additional construct validity of the CCTST (Facione et al 1994). The instrument authors state t hat face validity is established by the identify correct analysis, draw inferences, evaluate reasoning and justify their inferences 1994, p .15). Concurrent validity of the CCTST was supported by the pilot validation study. Pretest scores correlated with college GPA (.20), SAT Verbal (.55), SAT Math (.44) and Nelson Denny Reading scores (.40) (Facione et al 1994). In addition, a signific ant correlation at .72 was demonstrated for the Graduate Record Examination Score (Lambert, 2007).
51 internal consistency, reliability and validity data (Lambert in Mental Measurements Yearbook, 2010). Data Collection A random sample of two hundred participants in the University Scholars Program was approached with consent. Participants were given an i nformed consent form that was retained by the researcher. Students who signed the consent form gave permission to the researcher to access academic information including standardized test scores (SAT and ACT) and grade point average. Academic information was obtained via a data request to the Office of the University Registrar. The study ultimately collected complete responses from 45 participants just over 22 percent of the sample population. While this response rate is somewhat low, a 20 percent resp onse experienced by many undergraduates (Lipka, 2011) Procedure The survey, entitled Research Experiences in the University Scholars Program was administered online usin g Survey Monkey. Upon completion of the survey, participants were directed to the Insight Assessment website at http://www.insightassessment.com/home.html to complete the CCTST. All participants were assigned a confidential user name and password. Student identification numbers, assigned by the university were used to match test results with the survey responses. Students were asked to supply demographic information including gender, academic m ajor, field of study of their research project, prior experience with undergraduate
52 research and self reported expectations from their faculty mentors. The online testing platform used drop down menus with a fixed selection of responses for each of the de mographic questions on the CCTST and questions about research experiences on the survey. No other identifying information was collected. The test takers had access to their individual scores and could view the results of their critical thinking skills t est upon completion. Participants were able to request results of the study. The survey and the CCTST were completed after the semester in which they completed the University Scholars Program research experience. Data Analysis The CCTST was scored by the online service at Insight Assessment. Data was analyzed using SAS version 9.3 nternal consistency reliability of the CCTST and was compared to the data reported in the CCTST test manual. The independent variables examined in this study include a) gender, b) academic major, c) field of study of the research project d) prior experience with undergraduate research and e) experiences in the University Scholars Program. The dependent variables are the CCTST total scores and sub scores as well as the educational outcomes of participation in the program. Descriptive statistics were used to calculate the means and standard deviations of the dependent variables as well as each of the independent variables. Sta tistical Analysis A variety of inferential statistics were used to examine the data. The CCTST scores were not normally distributed; therefore nonparametric statistical tests were used. For the first research question, a bivariate analysis of the dependen t variable of
53 the CCTST total score with each of the independent variables was conduct ed. A Wilcoxon Rank Sum was used to test for differences between gender and CCTST total score. The Wilcoxon Rank Sum was also used to test for differe nces between gender and undergraduate research experience and major category and undergraduate research experiences. The second research question, a bivariate analysis of the independent variables gender, major and experience with undergraduate research with the dependent variables of educational outcomes were conducted using a chi square test of independence. Research question three examined the correlation between specific experiences within the undergraduate research program and CCTST scores. To det ermine which independent variables correlated with educational outcomes, a Spearman Correlation Coefficient was calculated with the educational outcomes as dependent variables and gender, major and experience with undergraduate research as independent vari ables. The fourth research question is concerned with which of the independent variables of experiences with the USP correlated with the dependent variables of educational outcomes. To examine this relationship, the Research Experiences in the University Scholars Program survey was scored with a simple points system. Responses of given one point. The points for each section of the survey were totaled and a subtotal for each of the seven sections was determined. A Spearman Correlation coefficient was calculated with the educational outcomes as dependent variables and the subtota ls of each of the survey sections as independent variables.
54 Research question five looked at the correlation between critical thinking skills with grade point average and standardized test scores, specifically the Scholastic Aptitude Test (SAT) and the A merican College Testing exam (ACT). The correlation between grade point average (GPA) and CCTST scores was also examined. For this question, a Spearman Correlation Coefficient was calculated to compare the dependent variable of CCTST total score with the independent variables of GPA, SAT and ACT scores. Research question six examined the correlation between educational outcomes and grade point average and standardized test scores. Using the same technique as question five, a Spearman Correlation Coeffi cient was used to compare the dependent variables of educational outcomes with the independent variables of grade point average and standardized test scores.
55 CHAPTER 4 DATA ANAYLSIS AND RE SULTS The purpose of this study was to determine if participatio n in an undergraduate California Critical Thinking Skill s Test (CCTST). In addition, the study examined if there were correlations between specific experiences with undergraduate research and critical th inking s kills. T he study also looked at correlations between undergraduate research experiences and educational outcomes as well as correlations between critical thinking skills and traditional measures of academic pe rformance such as standardized test scores and grade point average. The setting for the study was a Research Intensive Carnegie classification university with an established tradition of promoting undergraduate research. The sample was drawn from the You ng Scholars Program, a year long program in which the participants conducted an independent research project under the supervision of a faculty mentor. The program was open to students from all academic disciplines. The selection process for the program was competitive. Estimating Internal Consistency Reliability The scores from the study participants who took the CCTST (n = 45) were used to estimate internal consistency reliability. The sample was compared to the population used in the standar dization s amples provided in the CCTST Form 2000 test manual. The authors report a Kuder Richardson 20 ranging from .78 to .82. They state: For an instrument with multidimensional scales, a KR 20 above.70 indicates a high level of internal consistency (Facione a nd Facione, 2010). The analysis of internal consistency alpha of 0.97.
56 Analysis of Descriptive Statistics Demographics All of the study participants were traditionally aged undergraduate students in their third or fo urth year of college. Twenty three students were female (51.11%). Twenty two students were male (48.89%). For the purposes of the study, the students were grouped into four major categories: Engineering and Mathematics (9 participants) Humanities (7 pa rticipants) Physical or Biological Sciences (15 participants) and Social Sciences (14 participants) Table 4 1 shows the frequency of major categories. Table 4 2 shows the actual majors that made up each category group. Table 4 1 Frequency of Major Cat Major Category Male Female Total Engineering and Mathematics 3 6 9 Humanities 3 4 7 Physical and Biological Sciences 8 7 15 Social Sciences 8 6 14 Table 4 2 Major Categories and Actual Majors Major Category Actual M ajors Engineering and Mathematics Agricultural Engineering, Electrical Engineering, Environmental Engineering, Chemical Engineering, Materials Science Engineering, Mathematics, Mechanical Engineering Humanities English, History, Linguistics, Music, Theat re Physical and Biological Sciences Applied Physiology and Kinesiology, Biology, Chemistry, Microbiology, Neuroscience, Nutrition, Physics, Wildlife Ecology Social Sciences Anthropology, Business Administration, Economics, Journalism, Management, Politi cal Science, Psychology, Public Relations
57 Descriptive Statistics for Academic Performance and CCTST scores Academic performance was measured by scores on the Scholastic Aptitude Test rade point average (GPA). Another objective academic measure was average number of credit hours (ACH). The ACH was calculated by looking at the number of credit hours successfully completed during the course of fall and spring semesters (summer semesters were excluded). The mean SAT score was 1983.18 (n = 44, SD = 237.90). The mean ACT score was 29.73 (n = 26, SD = 3.44). The mean GPA was 3.75 ( SD = 0.24). The mean ACH for participants was 14.45 ( SD = 1.43). Table 4 3 displays the distribution of acad emic performance scores and CCTST scores. Table 4 4 shows the CCTST scores compared by gender. Table 4 3 Descriptive Statistics for Academic Performance and California Critical Thinking Skills Test Scores Academic Measure N Mean Median Std Dev Minimum M aximum SAT 44 1983.18 2025 237.9 1300 2310 ACT 26 29.73 31 3.44 18 34 GPA 45 3.75 3.82 0.24 3.16 4.00 Average Credits 37 14.45 14.38 1.43 10.75 18.00 CCTST 45 81.83 84.6 10.33 55.20 97.20 Descriptive Statistics for Educational Outcomes The education al outcomes of interest in this study were writing a senior thesis, attendance at a professional academic conference and submitting an article for publication in a peer reviewed journal. Twenty five of the participants reported they were very likely to wr ite a senior thesis while twenty of the participants stated they were either not likely to or were neither likely n or unlikely to w rite a senior thesis. Intention
58 to submit findings to a peer reviewed journal had an identical frequency with 25 particip ants stating that they intended to submit their findings and 20 participants saying that they were unlikely or neither likely n or unlikely to submit their findings to a peer review ed journal. Intention to attend a professional conference was less equall y distributed with 16 participants stating that they were likely to attend a professional conference and 29 participants who were either unlikely to attend a professional conference or neither likely n or unlikely to attend a professional conference. Tabl e 4 5 displays the frequency of the three educational outcomes measured in this study Table 4 4 Frequency of Educational Outcomes Educational Outcome Likely to have outcome Unlikely or neither likely nor unlikely to have outcome Senior Thesis 25 20 Att end a professional conference 16 29 Submit findings to a peer reviewed journal 25 20 Descriptive Statistics for Semesters of Experience Students were asked how many semesters of experience they had while engaged in unde rgraduate research activities unde r the supervision of a faculty mentor. Ten students reported six or more semesters of experience. Three students reported five semesters of experience. Nine students reported four semesters of experience Eleven students reported three semesters of expe rience and twelve students reported two semesters of experience. Since these subgroups were small, participants were re organized into two groups: students with three or less semesters of experience (n = 22) and four or more semesters of experience (n = 2 3). Table 4 6 shows the frequency of semesters of experience with undergraduate research.
59 Table 4 5 Semesters of Research Experience Semesters of Experience Number of Participants Six or more 10 Five 3 Four 9 Three 11 Two 12 Descriptive Statistics for Research Experiences Participants began the study by completing a 45 item questionnaire that was a mix of fixed choice and open ended questions. Please see Appendix 1 for a listing of survey questions. The survey had seven categories with five items in each category. for each category. Table 4 5 shows the distribution of scores from the Survey of Research Experiences. Table 4 6 Descriptive Statistics for Research Experiences Research Experience N Mean Median Std Dev Minimum Maximum Student Faculty Contact 45 20.96 21 3.03 11 25 Collaborative Learning 41 15.88 16 3.72 4 20 Active Learning 45 21.67 22 2.84 10 25 Prompt Feedback 45 21.51 22 3 12 25 Time on Task 4 5 16.96 17 3.82 8 24 High Expectations 45 22.09 23 2.67 14 25 Diverse Leaning 45 21 22 2.99 13 25 Analysis of Research Question 1 A category and semesters of experience with undergraduate research and cr itical thinking skills as measured by the
60 California Critical Thinking Skills Test? Since the CCTST scores were not normally distributed, non parametric statistical analysis was used. The CCTST scores were rank ordered and a Wilcoxon Rank Sum with a cont inuity correction of 0.5 wa s used to evaluate the data. Aver age scores were used for ties. Males and females did not differ significantly with their CCTST scores, T S =1.8667, p = 0.0686. Table 4 6 compares CCTST scores for males and females. Table 4 7 CCTST Scores Compared by Gender Gender n Mean CCTST score Median CCTST score Std Dev Minimum Maximum Female 23 79.30 83.20 10.20 56.60 95.80 Male 22 84.47 85.30 10.02 55.20 97.20 A Kruskal Wallis test was used to examine differences in CCTST scores a nd the four major categories The outcome of the test indicated no significant differences among the major categories, H = 0.2256, (3, N =45) p = 0.9734. Table 4 7 compares CCTST scores by major categories. Table 4 8 CCTST Scores Compared by Major Cate gory Major Category n Mean CCTST score Median CCTST score Std Dev Minimum Maximum Engineering and Mathematics 9 81.64 83.20 10.46 56.60 91.60 Humanities 7 82.00 86.00 11.92 55.20 88.80 Physical and Biological Sciences 15 83.01 83.20 10.02 60.80 97.20 S ocial Sciences 14 80.60 84.60 10.81 60.80 94.40 Students reported the number of semesters they had been engaged in undergraduate research activities. For the purposes of this analysis, the students were
61 classified into two groups: students with four or more semesters of experience and students with three or less semesters of experience A Wilcoxon Rank Sum with a continuity correction of 0.5 was used to approximate a Z score. Students with high experience scored higher on the CCTST than those with low experience, T S = 2.6635, p = 0.0108. Table 4 8 compares the means CCTST scores of students with high experience and low experience. Table 4 9 CCTST Scores Compared by Semesters of Experience Experience n Mean CCTST score Median CCTST score Std Dev Minimum Maximum High 22 86.64 86.70 5.84 77.60 97.20 Low 23 77.23 81.80 11.65 55.20 93.00 Note: High denotes students with four or more semesters of experience in underg raduate research. Low denotes students with three or less semesters of experience Analysis of Research Question 2 The second research question asks, a major category and prior experience with undergraduate resear ch and educational outcomes of undergraduate research? The only significant correlation was between high experience students and students who had submitted their findings for publication. For each of the educational outcomes (attending a professional a cademic conference, writing a senior thesis and submitting results to a peer reviewed journal) the students were grouped into two categories: those who were likely or very likely to produce the outcome in question and those who were unlikely or neither li kely or in gender and educational outcomes. There was no significant difference with gender and attending a professional conference, F = 17, p = 0.2214. There was no s ignificant
62 difference with gender and writing a senior thesis, F = 14, p = 0.5544. There was no significant difference with gender and submitting results to a peer reviewed journal, F = 13, p = 0.1362. There was no significant difference between major ca tegory and attending a professional conference, X 2 (3, n = 45) = 3.2143, p = 0.3598. For this part of the analysis, 50% of the cells had expected counts less than 5; therefore the Chi Square may not have been a valid test. There was no significant differe nce between major category and writing a senior thesis, X 2 ( 3, n = 45) = 3.4393, p = 0.3287. For this part of the analysis, 38% of the cells had expected counts less than 5; therefore the Chi Square may not have been a valid test. There was no significant difference between major category and submitting results to a peer reviewed journal, X 2 (3, n = 45) = 5.3486, p = 0.1480. For this part of the analysis, 38% of the cells had expected counts less than 5; therefore the Chi Square may not have been a valid Test was used to determine if there was a difference in semesters of experience and educational outcomes. There was no significa nt difference in semesters of experience and attendance at a professional conference, F = 12, p = 0.2214. There was no significant difference in semesters of experience and writing a senior thesis, F = 14, p = 0 .3726. However, the study did find a correlation between students with hig h experience and those who submit ted their findings to a peer reviewed jou rnal F = 6, p = 0.0361. Analysis of Research Question 3 T he third research question asks : w hich experiences reported by participants in the Survey of Experiences in the Young Scholars Program correlate with critical thinking skills. To answer this questio n the survey responses were rank ordered and compared
63 with the total score on the CCTST. A Spearman correlation was calculated for each of the reported experience categories. Analysis revealed that there was no significant correlation between student f aculty contact and CCTST score, r = 0.17610, n = 45, p = 0.2472. There was however a significant c orrelation between collaborative learning and CCTST score, r = 0.46087, n = 41, p = 0.0024 as well as active learning and CCTST score, r = 0.28599, n = 45, p = 0.0569. Analysis revealed that there was no significant correlation between prompt feedback and CCTST score, r = 0.04854, n = 45, p = 0.7515. Analysis revealed that there was no significant correlation between time on task and CCTST score, r = 0.040 61, n = 45, p = 0.7911. Analysis revealed that there was no significant correlation between high expectations and CCTST score, r = 0.03341, n = 45, p = 0.8275. Analysis revealed that there was no significant correlation between respect for diverse ways o f learning and CCTST score, r = 0.15774, n = 45, p = 0.3007. Table 4 10 Relationships Between Research Experiences and CCTST Scores Research Experience R N P value Student Faculty Contact 0.17610 45 0.2472 Collaborative Learning 0.46087 41 0.0024 Active Learning 0.28599 45 0.0569 Prompt Feedback 0.04854 45 0.7515 Time on Task 0.04061 45 0.7911 High Expectations 0.03341 45 0.8275 Diverse Learning 0.15774 45 0.3007
64 Analysis of Research Question 4 The fourth research question asks w hich experiences r eported by participants in the Survey of Experiences in the Young Scholars Program are correlated with outcomes of undergraduate research. To answer this question the survey responses were rank ordered and a Wilcoxon Rank Sum with a continuity correction of 0.5 was used to approximate a Z score while comparing research experiences and outcomes of undergraduate research. CCTST scores were also compared with each of the educational outcomes (writing a senior thesis, attendance at a professional conference a nd submitting results to a peer reviewed journal) Average scores were used for ties. No significant relationships were found between any of the reported research experiences and writing a thesis. The details are as follows: There was no significant r elationship between writing a thesis and CCTST score, | T S | = 0.0573, p = 0.9546. There was no significant relationship between writing a thesis and student faculty contact, | T S | = 0.9548, p = 0.3449. There was no significant relationship between writin g a thesis and collaborative learning, | T S| = 0.4588, p = 0.6488. There was no significant relationship between writing a thesis and active learning, | T S | = 0.0116, p = 0.9908. There was no significant relationship between writing a thesis and prompt f eedback, | T S| = 0.7405, p = 0.4629. There was no significant relationship between writing a thesis and time on task | T S | = 1.0785 p = 0. 2867 There was no significant relationship between writing a thesis and high expectations, | T S| = 1.1657, p = 0.25 There was no significant relationship between writing a thesis and respect for diverse ways of learning, | T S | = 0.5310, p = 0.5981. No significant relationships were found between any of the reported research experiences and attendance at a professio nal conference. The details are as follows:
65 There was no significant relationship between attendance at a professional conference and CCTST score, | T S | = 0.0951, p = 0.9247. There was no significant relationship between attendance at a professional co nference and student faculty contact, | T S | = 0.7882, p = 0.4348. There was no significant relationship between attendance at a professional conference and collaborative learning, | T S | = 0.6764, p = 0.5027. There was no significant relationship between attendance at a professional conference and active learning, | T S | = 1.8435, p = 0.0720. There was no significant relationship between attendance at a professional conference and prompt feedback, | T S | = 0.012, p = 0.4952. There was no significant relati onship between attendance at a professional conference and time on task, | T S | = 0.3811, p = 0.705. There was no significant relationship between attendance at a professional conference and high expectations, | T S | = 1.3419, p = 0.1865. There was no sign ificant relationship between attendance at a professional conference and respect for diverse ways of learning, | T S | = 0.1078, p = 0.9146. Significant relationships were found between submitting findings to a peer reviewed journal and three of the report ed research experiences. Students who reported higher levels of collaborative learning, active learning and time on task were more likely to submit their findings to a journal. Details are as follows: There was no significant relationship between submit ting findings to a peer reviewed journal and CCTST score, | T S | = 1.1336, p = 0.2631. There was no significant relationship between submitting findings to a peer reviewed journal and student faculty contact, | T S | = 1.0354, p = 0.3062. There was a signifi cant relationship between submitting findings to a peer reviewed journal and collaborative learning, | T S | = 2.0036, p = 0.0519. There
66 was a significant relationship between submitting findings to a peer reviewed journal and active learning, | T S | = 1.984 8, p = 0.0534. There was no significant relationship between submitting findings to a peer reviewed journal and prompt feedback, | T S | = 0.3471, p = 0.7302. There was a significant relationship between submitting findings to a peer reviewed journal and t ime on task, | T S | = 1.9389 p = 0. 0589 There was no significant relationship between submitting findings to a peer reviewed journal and high expectations, | T S | = 0.9926 p = 0. 3263 There was no significant relationship between submitting findings to a peer reviewed journal and respect for diverse ways of learning, | T S | = 0. 2886 p = 0. 7743 Table 4 11 Relationships Between Research Experiences and Submitting Findings to a Peer Reviewed Journal Research Experience | T S | P value Student Faculty Contact 1.0354 0.3062 Collaborative Learning 2.0036 0.0519 Active Learning 1.9848 0.0534 Prompt Feedback 0.3471 0.7302 Time on Task 1.9389 0.0589 High Expectations 0.9926 0.3263 Diverse Learning 0.2886 0.7743 Analysis of Research Q uestion 5 The fifth research question asks d oes academic performance as measured by standardized test scores and grade point average correlat e with critical thinking skills. To answer this question the various measures of academic performance (SAT score ACT score and GPA) were rank ordered and compared with the total score on the
67 CCTST. A Spearman correlation was calculated for each of the measures of academic performance A ll of the academic performance measures were significantly correlated with CCTS T score s Details are as follows: Analysis revealed that there was a significant correlation between SAT score and CCTST score, r = 0.57792, n = 44, p < 0.001. There was also a significant correlation between ACT score and CCTST score, r = 0. 55076 n = 26 p = 0.0035 as well as GPA and CCTST score, r = 0.32109 n = 45, p = 0.0315 These correlations are similar to those d escribed in the CCTST test manual Correlations were found with the four year college freshman norm group between CCTST score and ACT score, r = .402, N = 446, p = < .001 and between CCTST score and GPA, r = .20, N = 473, p = < .001 (Facione and Facione, 2010). Anal ysis revealed that there was a significant correlation between average number of credits and CCTST score, r = 0. 32291 n = 37 p = 0. 0513 Table 4 12 CCTST Correlations with Academic P erformance Variable r N p value SAT 0.57792 44 <.001 ACT 0.55076 26 0.0035 GPA 0.32109 45 0.0315 ANC 0.32291 37 0.0513 Analysis of Research Question 6 The sixth research question as ks : d oes academic performance as measured by standardized test scores and grade point average correlate with outcomes of undergraduate research? To answer this question the various measures of academic performance were rank ordered and a Wilcoxon Rank Su m with a continuity correction
68 of 0.5 was used to approximate a Z score while comparing academic performance and outcomes of undergraduate research. CCTST scores were also compared with each of the educational outcomes (writing a senior thesis, attendance at a professional conference and submitting results to a peer reviewed journal). Average scores were used for ties. There was no significant relationship between SAT scores and writing a thesis, | T S | = 0.2253, p = 0.8228. There was no significant rel ationship between GPA and writing a thesis, | T S | = 0.3657, p = 0.7164. There was no significant re lationship between ACT scores and writing a thesis, | T S | = 0.6815, p = 0.5018. A significant relationship was found between average number of credits and writing a thesis, | T S | = 1.9478, p = 0.0593. There was no significant relationship between SAT scores and attendance at a professional conference, | T S | = 0.9644, p = 0.3402. There was no significant relationship between GPA and attendance at a professio nal conference, | T S | = 1.4828, p = 0.1452. There was no signif icant relationship between ACT scores and attendance at a professional conference, | T S | = 1.5989 p = 0.1224. There was no significant relationship found between average number of credits an d attendance at a professional conference, | T S | = 0.2546 p = 0. 8004 There was no significant relationship between SAT scores and submitting results to a peer reviewed journal, | T S | = 0, p = 1. There was no significant relationship between GPA and subm itting results to a peer reviewed journal, | T S | = 0.3885, p = 0.6995. There was no significant relationship between ACT scores and submitting results to a peer reviewed journal, | T S | = 0.6754, p = 0.5056. There was no significant
69 relationship found bet ween average number of credits and submitting results to a peer reviewed journal, | T S | = 0.5946, p = 0.5558. Summary Analysis of particip n umber of semesters involved with undergraduate research a nd CCTST scores were correlated. Participants with four or more semesters of research experiences tended to have higher CCTST scores than those with three or less semesters of experience. A ll of the measures of academic performance (SAT scores, ACT scores and GPA) were strongly correlated with CCTST scores Collaborative Learning and Active Learning were the two self reported research experiences that were correlated with higher CCTST scores. category and research experiences except for one: students in the Humanities and Social Sciences were the least likely to report a collaborative learning experience. Three correlations between educational outcomes and research experiences were observed. Collaborative Learning, Active Learning and Time on Task were all correlated with submitting an article for publication.
70 CHAPTER 5 DISCUSSION This chapter provides a summary and discussion of the findings, suggestions for further research and implicatio ns for higher education. The purpose of this study wa s to examine the relationship between participation in an un dergraduate research program critical thinking skills and academic performance Specifically, t he study was interested in critical thinking skills as measured by the California Critical Thinking Skills Test (CCTST) and the outcomes of participation in an un dergraduate research program The study examined if factors such as field of study, gender and length of time engaging in undergraduate re educational outcomes. The study was also interested in determinin g if specific experiences were correlated with common educational outcomes associated with undergraduate research program s. Finally, the study looked at correlations between academic performance as measured by standardized test scores, grade point average and CCTST scores. This chapter will provide an overview of the study, summarize the findings and offer suggestions for further research. Overview C ritical thinking has been recognized as an important educational outcome for all undergraduates ( Bok, 2006, Paul 1995 ) Th e abundance of easily accessible information via the Internet converging with the widespread use of mobi le devices connected to the Internet makes critical thinking skills, particularly skills related to the ability to evaluate and sort information, especially valuable. As such, development of these skills need s to be an intentional part of the educational experience (Halpern, 1999).
71 College administrators are under increased pressure from a variety of stakeholders to provide measures of accountability with an emphasis on student outcomes (Liu, 2011) While many suggest that participation in undergraduate r esearch programs is a positive experience there are still relatively few studies that attem pt to quantify the experience ( Seymour, 2004, Tusi, 1998). Examining critical thinking ability and linking this to specific experiences within undergraduate resear ch has the potential to allow colleges to offer an objective measure of educational effectiveness as well as identify best practices in undergraduate research programs. Conclusions The California Critical Thinking Skills Test (CCTST), an objective measure of critical thinking skills The study Seven Principles of Good Practice in Undergraduate Education ( 1991) to see if any of these experiences were related to critical thinking ability or the likelihood of common educational outcomes for undergraduate researchers. Critical Thinking Skills Perhaps the most encouraging finding of t his study was that students who had more experience with undergraduate research tended to have higher CCTST scores. Students with four or more semesters of research experience scored higher on the CCTST than those with three or less semesters of experience, T S = 2.6635, p = 0.0108. While this finding cannot imply causation, it does suggest that it is possible that more experience with undergraduate research leads to enhanced critical thinking skills. This finding supports the Student Involvement Theory in that students with higher levels of involvement, in this case semesters of research experience under the supervision of a
72 faculty mentor experienced greater personal growth as measured by CCTST scores. The observation that students with a longer duration of research experience e xperienced a more positive outcome is consistent with findings by SRI International in a survey of students who participated in research p rograms funded by the NSF. The survey did not measure critical thinking skills, but it did indicate that positive out comes specifically expectations of obtaining a PhD, were strongly associated with the overall duration of research experiences ( Russell, 2007 ). While it is enticing to think that the undergraduate research experience played a role in this change, other s tudies in which students self reported growth in knowledge and intellectual skills over the course of the undergraduate experience reported similar changes in critical thinking skills. In a study examining national norms for undergraduate students, 38.8% reported that their critical as compared to their freshman year (Astin, 1993 ). The Student Experience in the Research University ( SERU ) data obtained at the same institution in which this study was cond ucted revealed a similar trend; 52% of graduating seniors reported gains in analytical and critical thinking skills ( Office of Institutional Planning and Research 2011). The study did not demonstrate any statistically significant difference in gender or major category and CCTST scores. The median score for females was 83.2. gender and their CCTST score a p value of 0.0686 suggests that a larger sam ple size may have yielded a statistically significant difference in scores between males and females.
73 Of the seven categories of research experiences (student faculty contact, collaboration active learning, prompt feedback, time on task, high expectatio ns and respect for diverse talents/ways of learning), two correlated with CCTST scores. A Spearman Correlation Coefficient revealed that Collaborative Learning and CCTST scores were significantly related, r = 0.46087, n = 41, p = 0.0024. Active Learning and CCTST scores were also significantly related, r = 0.28599, n = 45, p =0.0569 Also, students who reported more Active Learning (i.e. students who felt their opinions were valued, who had confidence in explaining their research to others and who repor ted they had a hig h level of autonomy in selecting their research topic) also tended to have higher critical thinking scores. These correlations are supported by Student Involvement Theory as well as Seven Principles for Good Practice in Unde rgraduate Education in that the use of educational practices that facilitate student activity and involvement, especially those practices that encou rage interactions among student peer groups, are likely to yield better outcomes when affective learning or high level cognitive gains are the goal (Sorcinelli, 1991 ) This suggests to higher education administrators and faculty mentors that encouraging the formation of research teams may lead to more positive outcomes from the und ergraduate research experience This finding is potentially promising as a best practice; the concept of collaborative learning deserves more exploration. This study did inquire about the experiences and perceptions of students who were part of research teams; however a more robust attempt at linking collaborative learning to the enhancement of critical thinking skills is warranted (Arum, 2011). The data also revealed that students in the social sciences and the humanities were less
74 likely to report being part of a research team tha n students in engineering/mathematics or the physical and biological sciences. A Kruskal Wallis test was used to examine differences in the scores from the Survey of Research Experiences and the four major categories. The median collaborative learning sc ore for student s in the social sciences ( n =12) and the Humanities ( n =6) categories was 12. The median collaborative learning score for student s in the engineering/mathematics category ( n =8) was 19.5. The median collaborative learning score for physical a nd biological sciences category ( n =15) was 18. The outcome of the test indicated a statistically significant difference among the major categories, H = 16.0825, (3, n = 45) p = 0.0011. This finding is consistent with observations made by Bost (1992) in which he argues that research in the Humanities is a solitary endeavor. Although team research may not be as common in the humanities and social sciences, this finding suggests that it may be beneficial for faculty mentors in these disciplines to consider innovative ways to introduce team based research for their undergraduate students. This can be facilitated by creatively breaking down the research tasks and sharing these tasks with able undergraduates (Lancy, 2003). The study found several significant correlations between critical thinking skills and traditional indicators of academic performance (SAT scores, ACT scores and grade point average). Analysis revealed that there was a significant correlation between SAT score s and CCTST score s r = 0.57792, n = 44, p < 0.001. There was also a significant correlation between ACT score s and CCTST score s r = 0.55076, n = 26, p = 0.0035 as well as GPA and CCTST score s r = 0.32109, n = 45, p = 0.0315. While this finding does not imply that students with lower standardized test scores would not benefit from
75 participation in an undergraduate research program, it does give higher education administrators another potential predictor for success in these programs and a potential target audience for recruitment effo rts The undergraduate research program in this study require d a minimum GPA f or participation ; therefore the correlatio n between GPA and CCTST scores wa s to be expected Perhaps high scores on the ACT and SAT exams may be an additional factor to conside r when students apply for compe titive research opportunities; however this should certainly not be the only factor. Hathaway, Nagda and Gregerman demonstrated that students with a range of incoming abilities can benefit from participating in undergraduate research (2002). A significant correlation was found between average number of credit hours and CCTST scores, r = 0.32 109, n = 37, p = 0.0513. This may be attributed to the fact that all of these students had the option of conducting research for credit While it cannot be said that students who take more credits per semester tend to have higher critical thinking skills, this finding support s Student Involvement Theory in that students who invested more time in their academic work experienc ed a greater level of personal growth. Additionally, it is a serendipitous correlation for administrators who are looking for ways to encourage students to increase their average credit registration rates in an effort to decrease time to degree completion. Undergradua te Research Experiences and Educational Outcomes Common educational outcomes for participants in an undergraduate research program included writing a senior thesis, attending a professional conference and submitting findings for publication. These outcom es have been described as part of the The three research experiences that were correlated with these activities were collaborative learning, active learning and time on
76 task. Using a Spearman Correlation Coefficient the data showed a correlation between collaborative learning and submitting research findings for publication, T S = 2.0036, p = 0.0451, and active learning and submitting research findings for publication, T S = 1.9848, p = 0.0472. One may conclude the p ossibility that being part of a research team increases the chances for a publication, especially as a secondary author. Students were not asked if they were submitting their findings as primary or secondary authors. The correlation between active learn ing and submitting findings for publication suggests the possibility that students who are more invested in their research projects are more likely to produce the caliber of research that warrants publication (questions from the Active Learning section of autonomy when selecting a research topic and their perception of how much their opinion was valued by the research mentor). The other experience correlated with submitting research findings for publication w as Time on Task, T S = 1.9389, p = 0.0525. Given the time consuming nature of preparing a manuscript, the correlation between this experience and educational outcome is not surprising. It is also supported by Student Involvement Theory While no signifi cant correlations between research experiences and the educational outcomes of writing a senior thesis and attendance at a professional conference were demonstrated, one correlation coefficient did come close. Students who had high survey responses in Ac tive Learning trended towards attending a professional conference, T S = 1.8435, p = 0.0653. A similar trend was found in students who had high survey responses in Faculty Expectations and attendance at a
77 professional conference, T S = 1.13419, p = 0.1796 A larger sample size may have yielded statistically significant results. Other factors, such as availability of funding for travel and timing conflicts with the academic calendar may also have influenced ferences. No significant correlations were found between student faculty contact and educational outcomes. This finding is consistent with that of the Evaluation of NSF Support for Undergraduate Research Opportunities conducted by SRI International. Thei interactions and perception of mentors had no effect on their research outcomes. The synthesis report of this study does not suggest that the role of the faculty mentor is insignificant, rather the complexity of the role is difficult to measure on a survey (Russell, 2007). The results did not demonstrate any significant difference with regard to gender and educational outcomes. This is in line with the findings of Bauer (2003) who reported that enhanced ability t o engage in the research process was demonstrated by both males and females. Limitations Several factors were identified as having the potential to l imit the findings of this study. The study was limited to participants in a selective undergraduate rese arch program at a single university. This limits the ability to generalize the results. The mean standardized test scores and grade point averages of the sample indicate that these were high achieving students. One might assume that their critical think ing skills were already well developed before they were selected for the program. Another limitation was the online format of the CCTST and the Survey of Undergraduate Research Experiences. Since students were allowed to complete the CCTST and
78 survey at their own convenience, there was no consistency in the testing environment. The variation in testing environment could have affected outcomes of the study. A couple of limitations could be traced to the small sample size. Students were approached with consent to participate toward the end of the research experience. Sample size may have increased had the students been notified at the beginning of the program, with one or more reminders/encouragements to participate throughout the course of the program. Finally, due to the small sample size, students had to be classified into four major categories. While the researcher made every effort to define major categories with similar characteristics ( i.e. engineering and mathematics), a larger sample size may have allowed for further exploration of more specific disciplines i.e. History, Chemistry, Anthropology, etc. Future Research Recommendations More research on the connection between critical thinking skills and undergraduate research programs is warran ted based on the findings of this study. The following research is recommended to further explore the educational effects of participation in undergraduate research and identify best practices: ticular group of students, all of whom participated in the same research program. A comparative study using a matched sample of students who participated in an undergraduate research program and similar students who had no experience with undergraduate re search is recommended. Participants would be matched by gender, major category and college classification (first year, second year students, etc.). It is further suggested that this study implement a pre test, post test design to see if CCTST scores chan ged after having completed the program. This study sampled students who participated in a single program. Based on the finding that students with four or more semesters of experience with undergraduate research tended to have higher CCTST scores, a study that looked at the total undergraduate research experience rather than a single program is recommended. A possible design could involve administering the CCTST as a pre test during the first semester of beginning research under the supervision of a
79 facu lty mentor and as a post test in the final sem ester of undergraduate research. Because all of the participants in this study were engaged with the same selective undergraduate research program, some similarities in experiences are to be expected. If the participant pool was widened to include all students who were conducting research under the supervision of a mentor, greater variation in experiences may be observed. The S urvey of Experiences in the Young Scholars Program was piloted by administering th e survey to several undergraduates as well as the director of the survey could lead to an assessment tool that would be useful across institutions to determine the effectiveness o f their undergraduate research programs and help to inform administrators and faculty mentors of best practices. Another variation would be pairing the CCTST with the Survey of Undergraduate Experiences (SURE) funded by the Howard Hughes Medical Institute The research experiences that were the most objective in nature were also the ones that were most likely to result in correlations (time on task, collaborative learning and active learning). In future studies these qualities could have greater emphasis perhaps through adding qualitative survey questions, while experiences such as respect for diverse ways of learning could receive less emphasis or simply be omitted. This study was interested in exploring positive relationships between undergraduate r esearch experiences, critical thinking skills and educational outcomes. While it is natural to look for the positive effects of any educational experience, it is also important to look at students who did not have positive experiences. Taking a closer lo ok at students who did not achieve the desired outcomes or perhaps non completers of research programs may shed light on characteristics of students who are not well suited for undergraduate research experiences and/or yield information on how these progra ms may be improved. Implications for Higher Education The results of this study have implications for university administrators and faculty who supervise undergraduate researchers. The skills students gain in their undergraduate experience, especially tho se skills related to preparing students to compete in a rapidly evolving and technologically advanced workforce, are of great concern to colleges and universities, legislators, students and parents. Undergraduate research experiences are one way to hone t hese highly valued and transferrable skills. It is essential that colleges and universities find ways to assess the outcomes of
80 participation in undergraduate research programs. Benefits of assessing these programs include informing faculty of the ways th ey can facilitate the best possible experience for their students as well as providing administrators with an added measure of student outcomes in an age that demands high levels of accountability. The findings demonstrated a correlation between students who had the most experience condu cting undergraduate research under the supervision of a faculty mentor and higher scores on the CCTST. In light of this finding, higher education administrators may wish to consider implementing programs that orient first and second year students to research opportunities and encourage faculty to recruit underclassmen as research assistants. This strategy may be implemented through seminar classes designed for first year students, research symposiums encouraging these stu dents to apply for summer research programs and online searchable databases in which faculty can post their projects and recrui t new students. Lecture style courses promoting undergraduate research opportunities have the adva ntage of reaching a large numb er of potential participants H owever care should be exercised to keep these courses engaging and interactive (Behar Horenstein, 2010). Care should also be taken to be sure only students who are truly motivated to engage in research are recruited, or else little benefit will be gained (Stevens and Reingold, 2002). The findings demonstrated a correlation between students who worked as part of a research team and higher CCTST scores. In light of this finding, faculty should consider forming research teams wh enever possible. The formation of a research team may not only benefit the students experience ; it may also help faculty members by delegating some routine supervisory roles to graduate students and/or more
81 experienced undergraduates. This in turn would benefit the more advanced students in giving them a higher level of responsibility, especially those who are considering a career in academia or research. Some effort may also be devoted to thinking of innovative ways to establish team based research in the social sciences and humanities. Results from this study found that students from these two major categories were the least likely to work as a part of a research team. Training faculty and graduate students in these disciplines in facilitating team b ased research may provide better outcomes. Additionally, programs that feature cross disciplinary research teams such as the Community and Environmental Transitions in Metropolitan Trenton at the College of New Jersey may be an effective way to facilitate team research for students in these disciplines. This innovative program, implemented by the Department of Sociology and Anthropology had students in the social sciences researching such topics as affordable housing and gang violence working in concert with students in the physical and biological sciences who researched contamination of the Delaware River and issues related to pollutants as former industrial sites were redeveloped ( Trenton College of New Jersey Update, 2007 ) The correlation between acti ve learning and CCTST scores reinforces what good educators have known for quite some time; students who have a high level of autonomy, whose opinions are valued and who are required to regularly report on their progress are more invested in their studies and are more likely to have positive outcomes. Mentors who consistently demonstrate these qualities should be commended and recognized on their campuses. Administrators shoul d consider developing workshops and symposiums to allow the best faculty mentors to share their
82 experiences and mentoring styles with their colleagues and especi ally with graduate students. When possible, funding should be allocated to allow student scholars and mentors to attend regional and national conferences such as the Nationa l Conference on Undergraduate Research or discipline specific professional meetings (Stevens and Reingold, 2000).
83 APPENDIX A INVITATION TO PARTIC IPATE AND INFORMED C ONSENT Invitation to Participate Dear Undergraduate Researcher, You have been randomly selected to participate in a study that is investigating relationships between critical thinking skills, educational outcomes and experiences in undergraduate research programs. Participants will complete an online survey regarding their USP experience a nd the California Critical Thinking Skills Test (CCTST). The time required to complete the survey and the CCTST will be approximately one hour. Your participation may help to identify best practices and lead to improvements for future USP participants. Participation in this study is completely voluntary. There is no penalty for not participating. If you wish to participate, please visit http://www.honors.ufl.edu/apps/protected/survey/ at your earliest convenience. You will be asked to login with your Gatorlink username and password. This website contains an informed consent form and further instructions on participation in this study. The second part of the survey will be administere d by Insight Assessment, a company that specializes in measuring critical thinking skills. Once the first part of the survey is complete, you will be re directed to the Insight Assessment website. Login instructions are below. Please save this email in the event you have questions about the login process. This entire process should take approximately 45 55 minutes. Please be sure that you have allowed yourself plenty of time and, if using a laptop, have plenty of battery life to complete both sections o f the test. 1. Begin by opening Internet Explorer and going to www.insightassessment.com 2. Please read the remainder of these instructions before moving on. Once t he test taker interface has opened it can be minimized so that you can refer back to these instructions with questions. 3. On the Right side of the page locate but do not select the Blue Hyperlink labeled that will assist you if you have any trouble logging in. 4. Next, locate and select check to make sure that your computer has an updated version of Java. Please be patient and follow any instruct
84 Instructions or go to http://www.java.co m/getjava and choose Password: User ID Password University Scholars Program USPGator 6. Please fill in the perso nal profile page being sure to fully complete all fields. *You fields have been filled out* 7. Select the CCTST 8. Accept the User Agreement Terms 9. Read the test instructions and continue to take the exam 10. Be advised that the exam has a time limit. You can see how much time remains in the bottom right hand corner of your screen while testing. If you do not finish all questions in the allotted time you will be unable to complete the test but your partial data will still be submitted. button Sincerely, John Denny
86 A PPENDIX B S URVEY OF EXPERIENCES IN THE YOUNG SCHOLAR S PROGRAM Introduction Dear Undergraduate Researcher, This is the first part of a two part survey. It is very important that BOTH parts of the survey are complete d. The purpose of this study is to examine the relationships between experiences in undergraduate research programs, critical thinking skills and academic performance. Your participation is essential. Please answer the survey questions as completely a nd accurately as possible. The survey questionnaire is a mix of fixed choice and open ended questions. Data will be reported in aggregate form only and comments will be pooled by theme. No individual scores or survey results will be published. The surv ey is asking for your student ID so that your responses can be matched with your scores on the critical thinking skills test. This study has been approved by the University of Florida Institutional Review Board 02, Protocol # 2001 U 0390. If you have any questions about your participation, please contact John Denny ( firstname.lastname@example.org ) What is your UF ID? (please enter without a dash: 11112222 Student Faculty Contact 1. How often were you in contact with you r research mentor? a. Daily b. More than once per week c. Once per week d. Every other week e. Once per month or less 2. My Mentor offered me advice beyond the scope of my research project (career advice, graduate school, recommended classes, etc.) a. Strongly agree b. Agree c. Ne ither agree or disagree d. Disagree e. Strongly disagree 3. My mentor shared personal experiences, attitudes and values with me.
87 a. Strongly agree b. Agree c. Neither agree or disagree d. Disagree e. Strongly disagree 4. I will continue research with my mentor after my USP projec t is complete. a. Strongly agree b. Agree c. Neither agree or disagree d. Disagree e. Strongly disagree 5. My mentor encouraged me to ask questions. a. Strongly agree b. Agree c. Neither agree or disagree d. Disagree e. Strongly disagree Collaborative Learning 1. Did you work as part of a r esearch team? a. Yes, I was part of a team that actively collaborated on my research project b. Yes, I worked on a team, but only had contact with other team members on an occasional and infrequent basis c. No, I was not part of a formal research team but I did hav e a group of students that I could go to for support d. No, I primarily worked alone on my project 11 2. I got to know about the interests and backgrounds of other students on my research team. a. St rongly agree b. Agree c. Neither agree or disagree d. Disagree e. Strongly disagree 3. I received meaningful feedback from members of my research team. a. Strongly agree
88 b. Agree c. Neither agree or disagree d. Disagree e. Strongly disagree 4. I felt comfortable in discussing concepts w ith members of my team who had backgrounds or viewpoints that were different from my own. a. Strongly agree b. Agree c. Neither agree or disagree d. Disagree e. Strongly disagree 5. I will continue conducting research with my research team. a. Strongly agree b. Agree c. Neither agr ee or disagree d. Disagree e. Strongly disagree Active Learning 1. My opinions and ideas were valued and had influence on how the research project progressed. a. Strongly agree b. Agree c. d. Neither agree or disagree e. Disagree f. Strongly disagree 2. I feel confident when explaini ng my research project to others. a. Strongly agree b. Agree c. Neither agree or disagree d. Disagree e. Strongly disagree 3. As part of my project, I was asked to regularly report on my progress to my mentor or my research team. a. Strongly agree b. Agree
89 c. Neither agree or disag ree d. Disagree e. Strongly disagree 4. I was encouraged to suggest new ideas for research projects. a. Strongly agree b. Agree c. Neither agree or disagree d. Disagree e. Strongly disagree 5. I had a high level of autonomy while selecting my research topic a. Strongly agree b. Agree c. Ne ither agree or disagree d. Disagree e. Strongly disagree Prompt Feedback 1. My mentor and /or research team gave me detailed evaluations of my work on a regular basis. a. Strongly agree b. Agree c. Neither agree or disagree d. Disagree e. Strongly disagree 2. I was asked to keep a log or record of my progress. a. Strongly agree b. Agree c. Neither agree or disagree d. Disagree e. Strongly disagree 3. My mentor answered questions promptly. a. Strongly agree b. Agree c. Neither agree or disagree d. Disagree e. Strongly disagree
90 4. My mentor was accessible any time I h ad a question. a. Strongly agree b. Agree c. Neither agree or disagree d. Disagree e. Strongly disagree Time on Task 1. My mentor clearly communicated the amount of time I should be spending on research. a. Strongly agree b. Agree c. Neither agree or disagree d. Disagree e. Strongly disag ree 2. During the summer semester, my mentor expected my USP project to be the primary focus of my time. a. Strongly agree b. Agree c. Neither agree or disagree d. Disagree e. Strongly disagree 3. On average, how many hours per week did you devote to your research project? a. 2 0 or more hours b. 15 20 hours c. 10 15 hours d. 5 10 hours e. 5 or less hours 4. I spent the most time working on my research project: a. During the summer semester while I was only enrolled in research credits b. During the summer semester while I was enrolled in research c redits and other courses c. About equal time during the summer and fall semesters d. During the fall semester
91 High Expectations 1. My mentor helped me set challenging goals a. Strongly agree b. Agree c. Neither agree or disagree d. Disagree e. Strongly disagree 2. My mentor sugge sted or required additional reading or writing tasks related to my research project. a. Strongly agree b. Agree c. Neither agree or disagree d. Disagree e. Strongly disagree 3. I plan to use my research project as the basis of my senior thesis. a. Strongly agree b. Agree c. Neit her agree or disagree d. Disagree e. Strongly disagree 4. My mentor expects me to present my research project at a professional conference (other than the Undergraduate Research Symposium). a. Strongly agree b. Agree c. Neither agree or disagree d. Disagree e. Strongly disagree 5. My mentor expects me to submit my research project to a peer reviewed journal for publication (other than the UF Journal of Undergraduate Research) a. Strongly agree b. Agree c. Neither agree or disagree d. Disagree e. Strongly disagree
92 f. Respect for diverse talents/way s of learning 1. My mentor encouraged creative thinking to solve problems. a. Strongly agree b. Agree c. Neither agree or disagree d. Disagree e. Strongly disagree 2. My mentor welcomed other points of view, even if they were different than his/her own. a. Strongly agree b. Agree c. N either agree or disagree d. Disagree e. Strongly disagree 3. a. Strongly agree b. Agree c. Neither agree or disagree d. Disagree e. Strongly disagree 4. Most often my mentor felt his/her ideas were the only correct method to solve a problem a. Strongly agree b. Agree c. Neither agree or disagree d. Disagree e. Strongly disagree Other questions 1. How many semesters have you been involved in undergraduate research? a. Two to three b. Three to four c. Four to five d. Five to six e. Seven or more
93 2. When did you begin conducting research under supervision of a mentor? a. Freshman (first year) b. Sophomore (second year) c. Junior (third year) d. Senior (fourth year) e. I began supervised research as a high school student 3. My participation in my research project has influ enced my career decision a. Strongly agree b. Agree c. Neither agree or disagree d. Disagree e. Strongly disagree 4. My participation in my research project has compelled me to attend graduate school a. Strongly agree b. Agree c. Neither agree or disagree d. Disagree e. Strongly disagree 5. My future career will involve research activities. a. Strongly agree b. Agree c. Neither agree or disagree d. Disagree e. Strongly disagree 6. Discuss your research activities. Please select all of the following activities that were part of your research experience. a. Col lected/analyzed data b. Made research decisions on my own c. Choose a research project on my own d. Completed the project I set out to do e. Compiled a literature review/accessed primary resources f. Wrote up research results g. Prepared results for an oral presentation h. Pre pared results for a poster presentation
94 LIST OF REFERENCES Adderly Kelly, B. (2003). Promoting the scholarship of research for faculty and students. ABNF Journal, 14 (2), 41 44 Arnold, K. D., & King, I. C. (1997). College student development and academic life : Psychological, intellectual, social, and moral issues / edited with an in troduction by Karen Arnold and I lda C arreiro king New York: Garland Pub. Arum, R., & Roksa, J. (2011). Academically adrift : Limited learning on college campuses / Richard a rum and Josipa R oksa Chicago: University of Chicago Press. Astin, A. W. (1993). What matters in college? : Four critical years revisited (1st ed.). San Francisco: Jossey Bass. Astin, A. W. (1999). Student involvement: A developmental theory for higher e ducation. Journal of College Student Development, 40 (5), 518 529. Astin, A. W., & American Council on Education. (1991). Assessment for excellence: The philosophy and practice of assessment and evaluation in higher education New York: American Council on Education; Toronto; New York: Macmillan; Collier Macmillan Canada; Maxwell Macmillan International. Baron, J. B., & Sternberg, R. J. (1987). Teaching thinking skills : Theory and practice New York: Freeman. Bauer, K. W., & Bennett, J. S. (2003). Alumni perception used to assess undergraduate research experience. Journal of Higher Education, 74 (2), 210 230. Baxter Magolda, M. B. (2001). Making their own way : Narratives for transforming higher education to promote self development (1st ed.). Sterling, V a.: Stylus. Behar Horenstein, L., & Johnson, M. L. (2010). Enticing students to enter into undergraduate research: The instrumentality of an undergraduate course. Bok, D. C. (2006). Our underachieving colleges : A candid look at how much students learn a nd why they should be learning more Princeton: Princeton University Press. Bost, D. (1992). Seven Obstacles to Undergraduate Research in the Humanities (and Seven Solutions!). Council on Undergraduate Research Newsletter, 13 (1), 35. Boyer Commission on E ducating Undergraduates in the Research University. (1998). Reinventing undergraduate education : A blueprint for research universities Princeton, N.J.?: Boyer Commission funded by the Carnegie Foundation for the Advancement of Teaching.
95 Boyer, E. L., & Carnegie Foundation for the Advancement of Teaching. (1987). College : The undergraduate experience in America ; the Carnegie foundation for the advancement of teaching / Ernest L. Bo yer (1st ed. ed.). New York: Harper & Row. Chickering, A. W., & Gamson, Z. F. (1991). Applying the seven principles for good practice in undergraduate education San Francisco, Calif.: Jossey Bass Inc. Collegiate Learning Assessment. (2012). CLA: Returning to Learning Retrieved from http://www.collegiatelearningassessment.org/index.html Crowe, M. (2006). Creative scholarship through undergraduate research. Peer Review, 8 (1), 16 18. Davis, D.S. & Glazier, D. (1997). How do we evaluate undergraduate research? Council on Undergraduate Research Quarterly, 18 73 76. Dewey, J. (1910). How we think Boston, D.C: Heath & co. Eigren, T., & Hensel, N. (2006). Undergraduate research experiences: Synergies between scholarship and teaching. Peer Review, 8 (1) 4 7. Elgin, S. C. R., Mardis, E. R., Buhler, J., Trosset, C. S., & Lopatto, D. E. (2007). The importance of research in the undergraduate curriculum: Explorations in genomics. FASEB Journal, 21 (5), A42 A42. Ennis, R. H. (1985). A logical basis for meas uring critical thinking skills. Educational Leadership, 43 (2), 44. Ennis, R. H. (1990). The extent to which critical thinking is subject specific: Further clarification. Educational Researcher, 19 (4), pp. 13 16. Ennis, R. H., Millman, J., & Tomko, T. N. (1985). Cornell critical thinking tests level X & level Z microform : Manual (3rd ed.). Pacific Grove, CA: Midwest Publications. Ertmer, P. A., Strobel, J., Cheng, X., Chen, X., Kim, H., Olesova, L., Tomory, A. (2010). Expressions of critical thinki ng in role playing simulations: Comparisons across roles. Journal of Computing in Higher Education, 22 (2), 73 94 Facione, P. A. (1984). Toward a theory of critical thinking. Liberal Education, 70 (3), 253 261. Facione, P. A. (1986). Testing college level critical thinking. Liberal Education, 72 (3), 221 231 Facione, P. A. (1990). Critical thinking: A statement of expert consensus for purposes of educational assessment and instruction. Millbrae, CA: California Acade mic Press.
96 National League for Nursing accreditation requirement in critical thinking. Retrieved February 3, 2008 from ERIC Document Reproduction Service No. ED38 0509 Facione, P. A., & Facione, N. C. (2007). Talking critical: Thinking. Change, 39 (2), 38 45. Facione, P. A., Facione, N. C., Winterhalter, K. (2010). California critical thinking skills test CCTST Test Manual. Millbrae, CA: California Academic Press. Fischer, K. W. (1980). A theory of cognitive development: The control and construction of hierarchies of skills. Psychological Review, 87 (6), 477 531. doi:10.1037/0033 295X.87.6.477 Fischer, K. W., & Farrar, M. J. (1987). Generalizations about generalizat ion: How a theory of skill development.... International Journal of Psychology, 22 (5), 643. Fischer, K. W., & Wozniak, R. H. (1993). Development in context : Acting and thinking in specific environments Hillsdale, N.J.: L. Erlbaum. Foertsch, J. (2000). Summer research opportunity programs (SROPs) for minority undergraduates: A longitudinal study of program outcomes, 1986 1996. Council on Undergraduate Research Quarterly, 21 (3), 114. Gale Group, & Guthrie, J. W. (2003). Encyclopedia of education [electro nic resource] (2nd ed.). New York: Macmillan Reference USA. Gamson, Z. F. (1991). A brief history of the seven principles for good practice in undergraduate education New Directions for Teaching and Learning, 1991 (47), 5 12. Giancarlo, C. A., & Facione, P. A. (2001). A look across four years at the disposition toward critical thinking among undergraduate students. Journal of General Education, 50 (1), 29 55. Gonzalez Espada, W. J. (2006). Evaluation of the impact of the NWC REU program compared with othe r undergraduate research experiences. Journal of Geoscience Education, 54 (5), 541. Gray, M. J., Association for the Study of Higher Education, Astin, A. W., Ayala, F., ERIC Clearinghouse on, H. E., & United States Office of Educational Research and Improv ement. (1987). College student outcomes assessment : A talent development perspective College Station, Tex: Association for the Study of Higher Education. Greenwald, D. A. (2010). Faculty involvement in undergraduate research: Considerations for nurse ed ucators. Nursing Education Perspectives, 31 (6), 368 371. doi:10.1043/1536 5026 31.6.368
97 Guterman, L. (2007). What good is undergraduate research anyway? Many students benefit, but studies show weaknesses in current practices. The Chronicle of Higher Educa tion p. A12 Halpern, D. F. (1999). Teaching for critical thinking: Helping college students develop the skills and dispositions of a critical thinker. New Directions for Teaching and Learning, 1999 (80), 69 74. doi:10.1002/tl.8005 Halstead, J. A. (1997). Council on undergraduate research: A resource (and a community) for science educators. Journal of Chemical Education, 74 (2), 148 149. Hathaway, R. (2002). The relationship of undergraduate research participation to graduate and professional education purs uit: An empirical study. Journal of College Student Development, 43 (5), 614 621 Hathaway, R. S., Nagda, B. (. A., & Gregerman, S. R. (2002). The relationship of undergraduate research participation to graduate and professional education pursuit: An empir ical study. Journal of College Student Development, 43 (5), 614 631. Hu, S., Kuh, G., & Gayles, J. (2007). Engaging undergraduate students in research activities: Are research universities doing a better job? Innovative Higher Education, 32 (3), 167 177. do i:10.1007/s10755 007 9043 y Hu, S., Scheuch, K., & Gayles, J. (2009). The influences of faculty on undergraduate student participation in research and creative activities. Innovative Higher Education, 34 (3), 173 183. doi:10.1007/s10755 009 9105 4 Hunter, A., Laursen, S. L., & Seymour, E. (2007). Becoming a scientist: The role of undergraduate research in students' cognitive, personal, and professional development. Science Education, 91 (1), 36 74. doi:10.1002/sce.20173 Jacobs, G., & Murray, M. (2010). Dev eloping critical understanding by teaching action research to undergraduate psychology students. Educational Action Research, 18 (3), 319 335. Jin, G., Bierma, T. J., & Broadbear, J. T. (2004). Critical thinking among environmental health undergraduates an d implications for the profession. Journal of Environmental Health, 67 (3), 15 20. Jones, T., Books24x7, I., & Dewing, C. (2011). Future agenda [electronic resource] : The world in 2020 / Tim J ones and Caroline D ewing Oxford, U.K: Infinite Ideas Limited. Kardash, C. M. (2000). Evaluation of undergraduate research experience: Perceptions of undergraduate interns and their faculty mentors. Journal of Educational Psychology, 92 (1), 191 201. doi:10.1037/0022 06220.127.116.11
98 Katkin, W. (2003). The B oyer commiss ion report and its impact on undergraduate research. New Directions for Teaching & Learning, (93), 19. Kerr, C. (2001). The uses of the university (5th ed.). Cambridge, Mass.: Harvard University Press. King, P. M., & VanHecke, J. R. (2006). Making connec tions: Using skill theory to recognize how students build and rebuild understanding. About Campus, 11 (1), 10 16. Kinkead, J. (2003). Learning through inquiry: An overview of undergraduate research. New Directions for Teaching & Learning, (93), 5. Kitchen er, K. S., Lynch, C. L., Fischer, K. W., & Wood, P. K. (1993). Developmental range of reflective judgment: The effect of contextual support and practice on developmental stage. Developmental Psychology, 29 (5), 893 906. doi:10.1037/0012 1618.104.22.1683 Kreme r, J. F., & Bringle, R. G. (1990). The effects of an intensive research experience on the careers of talented undergraduates. Journal of Research and Development in Education, 24 (1), 1 5. Kuh, G. D. (2009). What student affairs professionals need to know about student engagement. Journal of College Student Development, 50 (6), 683 706. Kurfiss, J. G., Association for the Study of Higher Education, ERIC Clearinghouse on, H. E., & United States Office of Educational Research and Improvement. (1988). Critical thinking : Theory, research, practice, and possibilities College Station, TX: Association for the Study of Higher Education. Lampert, N. (2007). Critical thinking dispositions as an outcome of undergraduate education. JGE: The Journal of General Educati on, 56 (1), 17 33. Lancy, D. F. (2003). What one faculty member does to promote undergraduate research. New Directions for Teaching & Learning, (93), 87. Lipka, S. (2011). Want data? Ask students. Again and again. The Chronicle of Higher Education Retrie ved from http://chronicle.com/article/Want Data Ask Students Again/128537/ Liu, O. L. (2011). Outcomes assessment in higher education: Challenges and future research in the context of voluntary system of accountability. Educational Measurement: Issues and Practice, 30 (3), 2 9. doi:10.1111/j.1745 3992.2011.00206.x Lopatto, D. (2004). Survey of undergraduate research experiences (SURE): First findings. Cell Biology Education, 3 (4), 270 277.
99 Lopatto, D. (2005). The benefits of undergraduate research. Academi c Leader, 21 (2), 3 3. Lopatto, D. (2006). Undergraduate research as a catalyst for liberal learning. Peer Review, 8 (1), 22 25. Lopatto, D. (2007). Undergraduate research experiences support science career decisions and active learning. CBE Life Sciences Education, 6 (4), 297 306. Lopatto, D. (2010). Undergraduate research as a high impact student experience. Peer Review, 12 (2), 27 30. Malachowski, M. R. (2003). A research across the curriculum movement. New Directions for Teaching & Learning, (93), 55. Merkel, C. A. (2003). Undergraduate research at the research universities. New Directions for Teaching & Learning, (93), 39. National Conferences on Undergraduate Research (2011). Joint Statement of Principles in Sup port of Undergraduate Research, Schol arship and Creative Activities Retrieved from: http://www.ncur.org/ugresearch.htm National Research Council (NRC) and the Committee on Undergraduate Biology Education to Prepare Research Scientists fo r the 21st Century. ( 2003 ) Bio 2010: Transforming undergraduate education for future research biologists. Washington, DC: The National Academies Press. National Science Foundation (1989). Report on the National Science Foundation disciplinary workshops on undergraduate education Washington, DC: Author. National Survey of Student Engagement. (2007). Annual report. Retrieved from http://nsse.iub.edu/NSSE_2007_Annual_Report/ docs/withhold/NSSE 2007_Annual_Report.pdf Office of Institutional Planning and Res earch (2011). UF Student Experience in the Research University (SERU) Reports Unpublished survey Gainesville, FL: University of Florida. Pascarella, E. T., Cruce, T., Umbach, P. D., Wolniak, G. C., Kuh, G. D., Carini, R. M., Zhao, C. (2006). Institutio nal selectivity and good practices in undergraduate education: How strong is the link? Paul, R., Binker, A. J. A., & Sonoma State University. Center for Critical Thinking and Moral Critique. (1990). Critical thinking : What every person needs to survive i n a rapidly changing world Rohnert Park, CA: Center for Critical Thinking and Moral Critique, Sonoma State University. Paul, R., Binker, A. J. A., Foundation for Critical Thinking, & Willsen, J. (1995). Critical thinking : How to prepare students for a r apidly changing world (4th ed.). Santa Rosa, CA: Foundation for Critical Thinking.
100 Phillips, D. C., & Soltis, J. F. (1991). Perspectives on learning (2nd ed.). New York: Teachers College Press. Reich, R. B. (1992). The work of nations : Preparing ourselv es for 21st century capitalism (1 Vintage Books ed.). New York: Vintage Books. Russell, S. H. (2007). Benefits of undergraduate research experiences. Pos : Proceedings of Science, 316 (5824), 548. Russell, S. H. (2007). Survey shows research is good for u ndergrads. Drug Discovery & Development, 10 (11), 18 21. Ryder, J., Leach, J. and Driver, R. (1999), Undergraduate science students' images of science. Journal of Research in Science Teaching 36: 201 219. Seymour, E., Hunter, A., Laursen, S. L., & DeAnto ni, T. (2004). Establishing the benefits of research experiences for undergraduates in the sciences: First findings from a three year study. Science Education, 88 (4), 493 534. doi:10.1002/sce.10131 Shellito, C. (2001). Successful mentoring of undergraduat e researchers: Tips for creating positive student research experiences. Journal of College Science Teaching, 30 (7), 460 ?. Sorcinelli, M. D. (1991). Research findings on the seven principles. New Directions for Teaching and Learning, 1991 (47), 13 25. Ste vens, J., and D. Reingold. 2000. The role of research in undergraduate chemical education. Retrieved from http://www. chem.vt.edu/confchem/2000/a/stevens/ researchrole.html Thorp, H. H., & Goldstein, B. (2010). Engines of inn ovation : The entrepreneurial university in the twenty first century Chapel Hill: University of North Carolina Press. Tompkins, G.E. (1989). Review if the Ennis Weir Critical Thinking Essay Test. In J.C. Conoley & J.J. Kramer (Eds.), The tenth mental me asurements yearbook (pp.290 291). Lincoln, NE: The Buros Institute of Mental Measurements, University of Nebraska Press. Tsui, L. (1998). A review of research on critical thinking. ASHE annual meeting paper., 34. Tsui, L. (2001). Faculty attitudes and the development of students' critical thinking. Journal of General Education, 50 (1), 1 28. Tsui, L. (2002). Fostering critical thinking through effective pedagogy: Evidence from four institutional case studies. Journal of Higher Education, 73 (6), 740 763.
101 Un ited States National Commission on Excellence in Education, & United States Dept. of Education. (1983). A nation at risk : The imperative for educational reform : A report to the nation and the secretary of education, united states department of education Washington, D.C.; The Commission; Supt. of Docs., U.S. G.P.O: distributor. Mind in society : The development of higher psychological processes Cambridge: Harvard University Press. Waite, S., & Davis, B. (2006). Coll aboration as a catalyst for critical thinking in undergraduate research. Journal of further & Higher Education, 30 (4), 405 419. Wenzel, T. J. (2004). Meeting report: Outcomes from the undergraduate research summit. Cell Biology Education, 3 (3), 150 151. Wingspread Group on Higher Education., Brock, W. E., & Johnson Foundation (Racine, W. ). (1993). An American imperative : Higher expectations for higher education [Racine, Wis.]: Johnson Foundation. Glaser Critical Thinkin edited by Daniel J. Keyser and Richard C. Sweetland. Kansas City, Mo.: Test Corporation of America. Zahorski, K. J., & Zahorski, K. J. (2002). Scholarship in the postmodern era : New venues, new values, new visions San Fra ncisco: Jossey Bass. Zydney, A.L., J. Bennett, A. Shahid, and K. Bauer. 2002. Faculty perspectives regarding the undergraduate research experience in science and engineering. Journal of Engineering Education 91 (3): 291 297.
102 BIOGRAPHICAL SKETCH John De nny was born and raised in Lakeland, Florida. After attending high school in his hometown, he sychology from the University of Florida in 1992. He worked in commu nity mental health for two years before beginning his gradua te studies at the University of South Florida where he compl eted his m aster o f social w ork. John interned at the Moffitt Cancer Center after receiving a scholarship in Oncology Social Work from the American Cancer Society. His research at Moffitt examined to cancer and was published in Health and Social Work He has six years of experience in health care providing medical social work services and has worked with several patient populations in cluding: oncology, infectious disease and acute care rehabilitation John has maintained his status as a licensed clinical social worker since 1998. A change in career path brought him to the University of Florida in 2001 as Assistant Dean of Students and Director of Disability Resources. He was instrumental in opening the Disability Resource Center at Reid Hall, the first ever designated center for students with disabilities a t the University of Florida. He joined the Honors Program as an Assistant Director in 2006 served as Interim Director from 2007 to 2009 and currently serves as Associate Director. In addition to academic advising, John teaches Introduction to Professional Development and enjoys offering classes through the (Un)Common Reading Program; focusing on books that involve sustainability themes land use and wilderness experiences. He has a keen interest in international education providing oversight for the United World Scholars program and is the Fulbright Program advisor f or the University of Florida
103 John lives in Gainesville Florida He is an avid music lover who enjoys spending time in the outdoors with his wife Christine, daug hter, Ella and son, Tyson. He likes to pl ay guitar, hike and kayak. He also dabbles in applied biochemistry as an award winning amateur zymurgist