|UFDC Home||| Help|
|PRIVATE ITEM Digitization of this item is currently in progress.|
This item is only available as the following downloads:
1 ONLINE EMBEDDED LIBRA RIAN CASE STUDY: A DESCRIPTIVE EVALUATION By MARY EDWARDS A DISSERTATION PRESENTED TO THE GRADUATE SCHOOL OF THE UNIVERSITY OF FLORIDA IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DE GREE OF DOCTOR OF EDUCATION UNIVERSITY OF FLORIDA 2011
2 2011 M ary E dwards
3 To my loving husband Eddie and beautiful daughters Allison and Samantha
4 ACKNOWLEDGMENTS First and foremost I wish to express my utmost gratitude to my husband Eddie for his support during this extended journey I could have never finished without his help, encouragement, and willingness to put up with a slightly messy house I thank my daughters Allison and Samantha for their patience and understanding I know t hey are For Allison the novelty of doing her homework with mommy has long since worn off and now she just wants her mommy back Samantha has never known her mommy not in school and will be pleas antly surprised with how much more fun mommy is now that the word is no longer in regular use in our home To the rest of my family mom Don, dad, Mona, Ariel, Janna nana, Jolene, Ryan and Jimanie your support and encouragement has been invaluable Dr. Kathryn ith family because you have been supportive since day one of the Foundations of Educational Technology class fall 2006 when we both started on this path You told m e then that I could and would finish with a doctorate, and with your help, you were right I can never repay you for the chocolate espresso beans and willingness to read my chapters no matter how busy you were with your own teaching and research! I think Kathryn. My advisors Dr. Erik Black and Dr. Swapna Kumar have been amazingly supportive during this process Their availability, encouragement, and feedback molded this project, the document, and me as a researcher/practitioner comment to me during our first meeting spring of 2007 when he gave me a gentle shove in the direction of purs u ing a doctorate; little did he know that he would become so
5 involved with my prog r es s Swapna, you ha ve been both advisor and friend and I look forward to many future collaborations I also thank my entire committee, Dr Black, Dr. Kumar, Dr. Lombard, and Dr. Tennant. Your commitment to improving my work is greatly appreciated. While not on my committee, the entire Educational Technology department has been supportive of me and my work: Dr. Cavanaugh, Dr. Dawson, Dr. Ritz haupt and Dr. Sessum s you have my sincerest gratitude My department chair and committee member, Dr. Michele Tennant has been unfailingly supportive of my desire to pursue this degree, even before she volunteered to serve on my committee the day before my proposal defense I thank the rest of the Health Science Center Library faculty and staff especially the Bi omedical and Health Information Services department H annah Norton your help with serving as a ra ter for my citation analysis and volunteering to proofread the whole document was invaluable. Kaitlin Blackburn, you have my utmost appreciation for proofread ing my dissertation I know you will go far in whatever you do. I also thank Dr. Randy Graff for auditing my coding and lending an ear To my colleague and friend Marilyn Ochoa thanks for working with me on the pilot p roject and supportively listening to my progress. Last, but certainly not least I thank the others who have been helpful and understanding these last few years including the parents and teachers of Expressions Learning Arts Academy (especially Dr. Dave Price for his reminders that sleep is necessary) and my educational technology support group: Nicola Wayer, Ben Campbell Vasa Burap h adeja, Feng Liu, Nate Poling, Irvika Francois, and Melissa Johnson. Finally, a special thanks to all of my EdD cohort colleagues and especially my support group, Polly Haldman, and Lisa Holmes.
6 TABLE OF CONTENTS page ACKNOWLEDGMENTS ................................ ................................ ................................ .. 4 LIST OF TABLES ................................ ................................ ................................ ............ 9 LIST OF FIGURE S ................................ ................................ ................................ ........ 10 LIST OF ABBREVIATIONS ................................ ................................ ........................... 11 ABSTRACT ................................ ................................ ................................ ................... 12 CHAPTER 1 INTRODUCTION ................................ ................................ ................................ .... 14 Background ................................ ................................ ................................ ............. 14 Evolving Roles of Librarianship ................................ ................................ ........ 16 Library Support for Distance Education ................................ ............................ 17 Types of Online Library Instruction ................................ ................................ ... 18 Embedded Librarians ................................ ................................ ....................... 19 Previous Related Research ................................ ................................ .............. 19 The Project ................................ ................................ ................................ ............. 21 Purpose ................................ ................................ ................................ ............ 21 Research Questions ................................ ................................ ......................... 21 Methodology ................................ ................................ ................................ ..... 22 Design ................................ ................................ ................................ .............. 23 The Researcher ................................ ................................ ................................ ...... 23 Significance of the Project ................................ ................................ ...................... 24 2 LITERATURE REVIEW ................................ ................................ .......................... 26 Introductory Remarks ................................ ................................ .............................. 26 Supporting Distance Education ................................ ................................ ............... 27 Growth & Trends in Distance Education & Online Learning ............................. 27 Need for library support in Distance Education ................................ ................ 28 Library Instruction and Information Literacy ................................ ............................ 31 Library Instruction for Distance Education ................................ .............................. 33 Stand Alone Library Instruction for Distance Education ................................ ... 33 Course Integrated Library Instruction for Distance Education .......................... 36 Embedded Librarians ................................ ................................ .............................. 39 3 METHODOLOGY ................................ ................................ ................................ .... 43 Introductory Remarks ................................ ................................ .............................. 43 Methodol ogy Review ................................ ................................ ............................... 43
7 Assessing Self Efficacy ................................ ................................ .................... 43 Citation Analysis ................................ ................................ ............................... 46 Case Study and Qualitative Inquiry ................................ ................................ .. 49 The OnMed Embedded Librarian Case ................................ ................................ .. 50 Overview ................................ ................................ ................................ .......... 50 Setting ................................ ................................ ................................ .............. 50 Participants ................................ ................................ ................................ ....... 52 The Embedded Librarian Project ................................ ................................ ...... 53 Theories Influencing the Embedded Librarian Content Design ........................ 56 Research Design and Data Analysis ................................ ................................ ....... 59 Overview ................................ ................................ ................................ .......... 59 Assessing Self Efficacy and Library Skill Performance ................................ .... 60 Citation Analysis ................................ ................................ ............................... 61 Qualitativ e Data Analysis ................................ ................................ .................. 63 Additional perspectives ................................ ................................ .............. 65 Validity ................................ ................................ ................................ ....... 65 4 RESULTS ................................ ................................ ................................ ............... 72 Introductory Remar ks ................................ ................................ .............................. 72 Quantitative Data ................................ ................................ ................................ .... 73 Student Access of Instructional Materials ................................ ......................... 73 Pre Post Assessment Data ................................ ................................ .............. 74 Information literacy self efficacy ................................ ................................ 74 Paired t test ................................ ................................ ................................ 74 Library skill performance ................................ ................................ ............ 75 Citation Analysis ................................ ................................ ............................... 76 Qualitativ e Data ................................ ................................ ................................ ...... 7 7 Participant Reflections ................................ ................................ ...................... 77 Themes ................................ ................................ ................................ ............ 77 Annotated bibliographies and critical analysis ................................ ............ 77 Library instructional materials ................................ ................................ .... 78 Search terms and strategies ................................ ................................ ...... 78 Finding useful and relevant articles ................................ ............................ 79 Minor themes ................................ ................................ ............................. 79 Prior library experience ................................ ................................ .............. 80 Prio r library instruction ................................ ................................ ............... 80 Instructor Reflections ................................ ................................ ........................ 81 Quality of participant annotated bibliography assignments ........................ 81 Library instructional materials ................................ ................................ .... 82 Limited interaction between students and librarian ................................ .... 82 Literature search capabilities of the participants ................................ ........ 82 Summary of Findings ................................ ................................ .............................. 83 5 DISCUSSION, IMPLICATIONS, AND CONCLUSIONS ................................ .......... 92 Introductory Remarks ................................ ................................ .............................. 92
8 Discussion of Major Findings ................................ ................................ .................. 97 Research Question: how does the presence of an online embedded librarian influence graduate students experience in an online educational technology research class? ................................ ................................ ........... 97 Librarian presence ................................ ................................ ..................... 98 Additional findings ................................ ................................ .................... 100 Discussi on Summary ................................ ................................ ...................... 101 Findings and the Literature ................................ ................................ ................... 102 Embedded Librarianship ................................ ................................ ................. 102 Summary of the Findings in the Context of the Literature ............................... 105 Threats to Validity ................................ ................................ ................................ 105 Implications ................................ ................................ ................................ ........... 106 Overview ................................ ................................ ................................ ........ 106 Implications for Professional Librarianship ................................ ..................... 106 Policy and Administrative Implications ................................ ............................ 112 Curricular and Educational Implications ................................ ......................... 116 Directions for Future Research ................................ ................................ ............. 119 Conclusions ................................ ................................ ................................ .......... 122 APPENDIX A PRE/POST TEST SELF EFFICACY AND LIBRARY SKILLS INSTRUMENT ...... 127 B OBJECTIVE CITATION RUBRIC SCORING SCALE ................................ ........... 129 C SUBJECTIVE RUBRIC SCORING SCALE ................................ ........................... 130 D PARTICIPANT REFLECTION QUESTIONS ................................ ........................ 131 E INSTRUCTOR INTERVIEW QUESTIONS ................................ ........................... 132 LIST OF REFERENCES ................................ ................................ ............................. 133 BIOGRAPHICAL SKETCH ................................ ................................ .......................... 159
9 LIST OF TABLES Table page 3 1 Embedded library content ................................ ................................ ................... 68 4 1 Embedded library instructional material access ................................ .................. 85 4 2 Paired t test results for the self efficacy assessment ................................ .......... 86 4 3 Descriptiv e statistics for each participant ................................ ............................ 87 4 4 Citation analysis summary ................................ ................................ .................. 88 4 5 Descriptive statistics of citation analysis ................................ ............................. 89 4 6 Reflection themes with examples ................................ ................................ ....... 90 5 1 Pre and post test scores for each participant ................................ .................... 124
10 LIST OF FIGURES Figure page 3 1 Objective Rubric ................................ ................................ ................................ 70 3 2 Subjective Rubric ................................ ................................ ................................ 71 5 1 ................................ ................................ .......... 125 5 2 Health/Medical Accrediting Agencies ................................ ............................... 126
1 1 LIST OF ABBREVIATIONS ACRL Association of College and Research Libraries AHC Academic Health Center ARL Association of Research Libraries CMS Course Management System DCT Dual Coding Theory EBM Evidence Based Medicine EL Embedded Librarian HSC Health Science Center IL Information Literacy OL Online Learning SACS Synchronous Audiographic Conferencing Software SoTL Scholarship of Teaching and Learning
12 Abstract of Dissertation Presented to the G raduate School of the University of Florida in Partial Fulfillment of the Requirements for the Degree of Doctor of Education ONLINE EMBEDDED LIBRARIAN CASE STUDY: A DESCRIPTIVE EVALUATION By Mary Edwards December 2011 Chair: Erik Black Major: Curriculum and Instruction The increase in online programs has been accompanied by the need for library instruction and support for online students. Students enrolled in off campus and/or online programs have to be able to successfully access and use digital library resources to complete course requirements. Using and evaluating resources is an element of information literacy, a key component in critical thinking and lifelong learning, which is a ddressed by accrediting agencies An embedded librarian project in an online graduate educational technology course demonstrated the feasibility of an embedded librarian for online graduate courses at the University of Florida result ing in benefits for st udents and a pilot for this OnMed Embedded Librarian case study. This case study describes the development and evaluation of an embedded librarian project for the OnMed program OnMed is a Masters level graduate educational technology program aimed at equi pping academic health care professionals with skills to teach and conduct educational research in the 21 st century. The librarian was embedded in an eight week course, EDG 6931 Issues and Current Research in Educational Technology. The results of paired pre and post assessment questionnaires, narrative reflections and an instructor interview indicate that embedded
13 librarians can greatly help students in their online assignments ; however, the specific features and design of the successful embedded librar ian program should vary according to the context and learner characteristics. Course level embedded librarian programs such as the one described and evaluated in this case can have implications for the practice of librarianship, policy and administrative i mplication s and curricular implications. Further research and recommendations for the implementation and assessment of embedded librarian projects are discussed including the need for further studies to investigate course level embedded librarians in vari ous contexts and the use of other methodologies to provider stronger empirical evidence to support the efficacy of embedding librarian s in online courses.
14 CHAPTER 1 INTRODUCTION Chapter 1 provides an introduction and framework for the embedded librar ian program The introduction highlights the need for library support and instruction for distance and online education in the context of prior research, introduces the researcher (including prior research, qualifications and niche), describes the project and context (including the purpose, research questions, methodology and design) and finally emphasizes the significance of designing and evaluating an online embedded librarian experience. Background As distance and online education enrollments continue to increase and the trend for growth in online learning continues educators are challenged to ensure quality distance education experiences (Allen & Seaman, 2010; Accrediting Commission of the Distance Education and Training Council, 2010 ) For the past eight years, the Sloan Consortium has sponsored national surveys of online learning. According to these reports, there is a trend of significantly increasing enrollments in onli ne courses (Allen & Seaman 2002; 2003 ; 2004 ; 2005 ; 2006 ; 20 07 ; 2008 ; 2009). During the fall 2008 semester, over 4.8 million students took an online course, which was a seventeen percent increase from 2007 (Allen & Seam a n, 2010). The newest report published in n early thirty percent of all college an d university students now take at p. 2 ). While enrollments are increasing, faculty perception of the quality of online instruction is also increasing ( Allen & Seaman, 2011 ). Sixty six percent of faculty (spec ifically chief academic officers)
15 face to face instruction (Allen & Seaman, 2010). As distance education evolved technologically and pedagogically, it transitioned from the boundaries of educational legitimacy to its current status as a valid alternative and/or supplement to traditional higher education (Bruder, 1989; Casey 2008 ; Garrison, & Shale, 1987; Go och, 1998; Matthews, 1999; Means, Toyama, Murphy, Bakia, & Jones, 2009; Moore, & Kearsley, 2004; Schlosser, 1994) Support for distance education and online learning includes student, faculty, and library support (Bird & Morgan, 2003; Cain & Lockee, 2002; Carnwell & Harrington, 2001; Fisher & Baird, 2005; Kelly, 2001; Ludwig Hardman & Dunlap, 2003; Rama & Hope, 2009; Simpson, 2002; Tait, 2000; Tait, & Mills, 2003; Thorpe, 2002; Wheeler, 2002) Various factors contribute to quality assurance in distance educ ation at the course, program, and administrative levels including faculty support (the use of instructional designers and technologists, etc.) and various levels of student support ( Distance Education & Training Council, 2010 ) In terms of access to librar y materials, a cademic librarians can provide necessary support for both faculty and students thereby contributing to a quality distance education experie nce and providing distance students with library services similar to those available to face to face s tudents ( Baird & Wilson, 2002; Black, 2003; Burge, 2002; Burge & Judith 2000; Burich, 2004a; Burich, 2004b; Cassner & Adams, 2005; Coffman, 2001; Gandhi, 2003; Haynes, 2002; Lorenzo, 2003; Pace, 2001; Perrone, 2000 ) The Association of College and Researc h Libraries (ACRL) emphasize s
16 (ACRL, 2007) In addition to enumerating service (i.e. document deliver/interlibrary loan, reserves, etc.), personnel and policy requirements, the standards unequivocally state that students enrolled in distance education programs are entitled to the same library services as their face to face counterparts. Evolving Roles of Librarianship Steven Bell and John S (p. 372) which emphasizes the evolving roles of librarianship and the need for librarians who are skilled in curriculum/instruction and educational technology (Bell & Shank, 2004; Shank, 2006). Librarian involvem ent with distance education expands upon the features of traditional academic librarianship collaborative partners in teaching and learning (Bell, 2005; Bell & Shank, 2004; Shank, 2006; Sinclair, 2 009) Blende d librarians support the needs of distance education faculty, staff and students through their knowledge of instructional design, pedagogy/andragogy and technology ( Bell & Shank, 2004; Shank, 2006) While there are many professional development opportuniti es to assist librarians in gaining the knowledge, skills and attitudes described by Bell and Shank, there is a need background in theory and research in order to lead professional development in ble nded librarianship acquired through advanced graduate study ( Shank, 2006 ) As a librarian and doctoral candidate studying educational technology, the author, heretofore referred to as the librarian researcher exemplifies and lead s the profession by contributing to and strengthening the quality of scholarship on library roles in instruction In addition to holding an American Library Association (ALA) accredited Degree in Library and Information Science the librarian researcher has six years of
17 professional library experience and five years of advanced graduate coursework in curriculum and instruction (including educational technology, instructional design and learning theories). Online library instruction br idges the academic disciplines of library and information science and educational technology (Johnson, 2008). Therefore, i n addition to the more traditional areas of library instruction and information literacy librarians providing support to online progr ams should be knowledgeable about significant issues in distance education and online teaching and learning (Cassner & Adams, 2004, 2006; Fritts & Casey, 2010; Johnson, 2008; Jones, 2007, 2003; Washburn, 2007; Yang, 2005) Library Support for Distance Educ ation Library support for distance and online learning and the benefits of such support for student s and faculty are well documented in the library and information science literature ( Baird & Wilson, 2002; Black, 2003; Burge, 2002; Burge & Judith 2000; Bu rich, 2004a; Burich, 2004b; Cassner & Adams, 2005; Coffman, 2001; Gandhi, 2003; Haynes, 2002; Lorenzo, 2003; Pace, 2001; Perrone, 2000; Ramadevi, 1999) Providing additional evidence of learning outcomes (through assessment efforts described in the literature and in this embedded librarian project) can help ensure future library involvement with distance and online education and enhance quality assurance b y providing empirical evidence for evidence based librarianship ( Birdsall, 2008; Bogel, 2008; Brettle, 2008; Glynn, 2008; Helliwell, 2007; Loertscher, 2009; Lyons, 2009; Mirijamdotter, 2009; Nobisso, 2008; Todd, 2008; Wakimoto, 2010). Evidence based practi ce is a trend towards data based decision making in many professions including librarianship, education medicine and other health professions (Johnson, 2008).
18 Types of Online Library Instruction There are two broad categories of library instruction: stan d alone instruction that occurs free of any ties to a particular course or curricula and course integrated instruction that is contextualized and designed to supplement coursework (Allegri, 1 985; Badke, 2009; Beile, 2003; Bordonaro & Richardson, 2004; Cmor & Marshall, 2006; Mellon, 1984; Stein & Lamb, 1998). Library instruction for distance education includes both stand alone and course integrated instruction ( Cipkin, 2002; Jobe & Deborah 2000; Markgraf, 2004) An examination of the literature reveals that librarians use multiple modes to provide library instruction for distance education (Arnold, & Jingping, 2002; Dunlap, 2002; Fourie, 2001; Garnsey, 2002; Henner, 2002; Holmes, 2002; Hricko, 2001; Jayne, 2000; Kinder, 2002). Stand alone library instruction for distance and online instruction focuses on online tutorials and other instructional content not developed to support a specific course (Churkovick & Oughtred, 2002; Ferguson & Ferguson 2005; Lindsay Cum mings, Johnson, & Scales 2006 ; McLean & Dew 2006 ; Orme, 2004 ). Course integrated library instruction can include integration in the course management system, synchronous instruction (using video conferencing or synchronous audiographic conferencing tools ) and more recently online embedded librarians ( Cipkin, 2002; Jobe & Deborah 2000; Markgraf 2004 ; Moore, 2004; Ramsay & Kinne, 2006; Kearley & Phillips, 2004) The library literature supports the fact that library instruction is most effective when students are directly applying the content to an assignment (Badke, 2009; Stein, 1998) because the instruction is contextualized to the needs of a particular course or assignment.
19 Embedded Librarians Various levels of course and program integration exist within online library model ( Kesselman & Watstein, 2009; Rudin, 2008 ) T borrowed from the practice of embedding journalists in combat zones during recent military conflicts ( Dugan 2008; Kesselman & Watstein 2009). Embedded librarianship will be thoroughly discussed in the literature review, but defined generally an embedded librarian is thoroughly integrated into the college, department, and courses he/she serves and provides contextualized support and instruction As noted by Edwards, Kumar, and Ochoa (2010) the theoretical basis for the course level o nline embedded librarian is founded in the theories of contextualized instruction (Brown, Collins & Dugid, 1990), authentic instruction ( Collins, Brown & Newman, 1989 education theories of transactional distance and interaction in dis tance education (Moore, 1989 & 1993 ). Additional conceptual support for the embedded librarian project is founded in the concept of applying an instructional design model (specifically the Morrison, Ross, and Kemp model 2006) to the development of the library instruction that serves as the instructional intervention used in the study The embedded librarian project described in this document was based on prior education research and best practices in library instruction for di stance and online education. Previous R elated R esearch The project is based on the findings of earlier research including pilot studies completed in the areas of assessing online embedded librarians and using synchronous audiographic tools to provide synch ronous library instruction to professional students enrolled in a graduate educational technology program The written documentation of
20 those projects was submitted in partial fulfillment of the librarian researcher qualifying examination. Both projects were piloted with online graduate level educational technology students in collaboration with the education librarian Synchronous instruction is used in online learning but is not thoroughly documented or evaluated in the library literature ( Bell & Shank 2006; Blakeslee & Johnson, 2002; Henning, 2001 ; Kontos & Henkel, 2008; Reeves, 2005) A pilot project allowed the librarian and researcher to investigate the use of Elluminate to provide synchronous online library instruction The project was evaluated w ith a post instruction survey, but due to several issues with the assessment instrument and methodology, the data wa s of little use However, the anecdotal findings indicated that given careful preparation and topic selection, synchronous instruction via E lluminate is a viable platform for providing online library instruction and the overall experience (from both the librarian and student perspective) informs future work in this area. Based on an identified gap in the literature it can be concluded that th ere is a need to further implement and assess online synchronous library instruction for satisfaction and learning outcomes. Exploring another type of integrated online library instruction was the goal of the embedded librarian pilot The concept of the embedded online librarian was first explored and evaluated in a pilot project that involved collaboration between the education librarian, the course instructor and the librarian researcher Results from that project indicate that students and faculty foun d both the presence of a librarian and integrated library content useful While the embedded librarian pilot was a successful Lessons learned
21 from both pilot projects informed the design and evaluation of this embedd ed librarian project. The Project Purpose The purpose of this project wa s to not only develop an embedded librarian program (focusing on instruction) but to also develop a comprehensive evaluation plan to assess program success and determine its effect on student learning As was established in the above discussion of the evolving nature of librarianship and distance education, contextual library support of distance education wa s essential While librarians are acknowledg ing this need by creating online embedded librarian programs, there is still a need to evaluate and provide empirical evidence as to the efficacy of such endeavors This project has not only fill ed an important instructional need, it has also demonstrate d a comprehensive evaluation plan and guide for future work in the development and assessment of online embedded librarian programs. Research Questions The primary research question guiding this project wa s: H ow does the presence of an online embedded librar ian influence graduate students experience in an online educational technology research class as defined by these specific aspects : Self efficacy with information literacy and library skills as measured by changes in performan ce on a pre and post assessment Library skill performance as measured by changes in performance on a pre and post assessment ns in an annotated bibliography
22 Exploring these questions will provide the evidence for future online embedded librarians and further integration collaboration between librarians and faculty. Methodology All of the participants in this case study are members of the inaugural cohort of the OnMed program The OnMed program was desi gned to provide faculty in the health science colleges with a Masters of Education focusing on Educational Technology The goal of the program is to equip participating practitioners with the knowledge and skills necessary to utilize technology in their cl inical teaching areas ( Cavanaugh, Dawson & Black, 2010). The needs of the OnMed students include d both medical education and educational technology literature and resources. This new program provided an excellent fit for the librarian researcher because it required a blend of her developing expertise in the field of educational technology with professional experience s as a health science librarian in an academic medical center Prior research provide d valuable insight into the methodologies employed in the evaluation of this embedded librarian project Two relevant areas include d assessing self efficacy with information literacy/library skills and citation analysis as an outcome measure for student library skills Both qualitative and quantitative methods provide d a complete picture of the impact of the embedded librarian program Quantitative assessments include d a paired pre/post assessment of self efficacy with information literacy and library skill performance and citation analysis of student writing ar tifacts to evaluate the quality of citations students use (an indirect measure of their research skills) perceptions of the program (from narrative reflections ) provide d qualitative data and
23 were triangulated with the quantitative data to provid e a complete program evaluation of the OnMed case Design C onsistent with th e Morrison, Ross, and Kemp (2006 ) instructional design model, the exact configuration of the instructional components that comprise d the embedded librarian experience were designed and developed after a full analysis of the learners, context and tasks required of the students The learner analysis was informed by the OnMed survey of technology use and collaborati on /consultation with the course instructor. For the OnMed stud ents the instructional goals focus ed on increasing medical/ health science education and educational technology The library instruction include d foundational, to th e program as a whole and other topics specific to the needs of the course Library content for the course focused on optional asynchronous interaction via the discussion board, optional synchronous help facilitated through the integration of a Meebo chat widget and instructional content Instructional materials included several instructional websites (called LibGuides), asynchronous presentations and instructional videos The Researcher The embedded librarian project provide d evidence of the librarian res earcher teaching as she employ ed an instructional design approach to create an instructional plan, develop ed instructional materials and integrate d herself in the OnMed course By conducting a rigorous evaluation of the program, the librarian researcher demonstrate d expertise in the areas of research and scholarship of teaching and learning The project coincide d with her professional responsibilities as an academic medical librarian
24 specializing in distance education and liaising to departments in the co lleges of Medicine and Public Health & Health Professions at the University of Florida Within the librarian researcher professional role her expertise in educational technology h as facilitated service opportunities within the university libraries, in t he Health Science Center, on campus and nationally Two relevant and significant committees include the Academic Health Center Faculty Development Workgroup and the Academic Health Center Educational Technology Advisory Council Her participation in those groups is beneficial both to the library, to the committees on which the librarian researcher serve s and to the Health Science Center at large. Benefits to the libraries include participat ion in discussions significant to the future of the health science center and the establish ment of the libraries contribution to academic health education. The HSC Library is in a unique situation as a neutral entity providing service to all HSC college without having ties to a specific college This neutrality is beneficial to the HSC and the committees on which the librarian researcher serve s because she is able to provide valuable input while remaining objective Significance of the Project The need for library support of distance and online l earning is well established in the literature ( Baird & Wilson, 2002; Black, 2003; Burge, 2002; Burge & Judith 2000; Burich, 2004a; Burich, 2004b; Cassner & Adams, 2005; Coffman, 2001; Gandhi, 2003; Haynes, 2002; Lorenzo, 2003; Pace, 2001; Perrone, 2000; R amadevi, 1 999) Libraries use a variety of methods to provide support and instruction, including established embedded librarian programs and services (Moore, 2004; Ramsay & Kinne, 2006; Kearley & Phillips, 2004)
25 In order to demonstrate the continued need and benefits of these programs existing and new services must be evaluated To date there is no documentation of a thorough and rigorous evaluation of an online embedded librarian program demonstrating a clear impact on student performance Completing such an evaluation provide d evidence for the continuation and expansion of embedded librarian programs in the Health Science Center and in a broader context at the University of Florida Additionally, t he results of this project ha ve implications for library s upport of online learning. As such, the written documentation of the project contribute s to the body of library science literature by describing and providing evidence to support the practice of providing contextual instruction for online and distance educ ation and providing practicing librarians with a model for thorough and valid research and assessment practices.
26 CHAPTER 2 LITERATURE REVIEW Introd uctory Remarks As previously stated the context, niche and methodology for this embedded librarian project are grounded in the educational technology and library science literature The two main ideas present in this project relate to library instruction/information literacy and distance and online education. Both concepts provide foundational support for onlin e library instruction in general and specific support for embedded librarian ship doctoral study (p. 3) To this end, this chapter will focus on an in depth exploration of the relevant literature. The following literature review includes four main sections Supporting Distance Education, Library Instruction and Information Literacy Library Instruction for D istance Education, and Embedded Librarianship with subsections as needed The first section (Supporting Distance Education) introduces the topic of support for distance and online education including discussion of the growth in distance and online educat ion student and faculty support, and conclud es with a discussion of general library support. Section two (Library Instruction and Information Literacy ) reviews the literature on information literacy and library instruction. It is important to discuss this concept prior to discussing library instruction for distance education for several reasons. First, library instruction and information literacy are central themes for the project. Second, traditional information literacy instruction preceded and informed library instruction for distance and online education. Section three ( Library Instruction for
27 Distance Education ) focus es on library instruction for distance education and the articles are categorized by whether they describe stand alone or course integra ted instruction. The final section (Embedded Librarianship) provides an in depth discussion of relevant embedded librarian literature, which directly informs the design of the embedded librarian pr oject described in this document Supporting Distance Education Growth & Trends in Distance & Online Education Distance Education has evolved from the early days of correspondence courses via mail, radio, television, and videoconference (Casey 2008; Moore & Kearsley, 2005). Advances in technology have facili tated growth in parallels between the development of distance learning and the expanding role of technology in mass communication suggest that technology is the most compelling p. 45 ) As distance education expands, it continues to gain acceptance as a legitimate educational option. T he Internet has become a nearly ubiquitous technology, and the scope of distance education opportunities has shifted in favor or web based courses and onl ine learning Lee and Nguyen (2007) provide a thoughtful and thorough review of the literature highlighting key articles on the growth and evolution of online learning. Since 2002, the Sloan Consortium has funded an annual report on the growth of online education. The most recent report (Allen & Seaman, 2009) describes the growth of online learning and factors influencing its success. Research questions guiding the 2009 study include d issues investigating the growth of and support for online learning : How Many Students are Learning Online, Is Online Learning Strategic, Has Faculty Acceptance of Online Learning Increased, and Do Faculty Receive Training for
28 Teaching Online? Answers to these research questions provide d useful insight into the online learning environment with recommendations and directions for institutions of higher education. Quality assurance and acceptance (by the public and those involved in higher education) of online education is an issue addressed thoroughly in the literature Addition ally, the literature is replete with studies (utilizing primarily experimental and controlled quasi experimental designs) comparing the effectiveness of online (or hybrid/blended) instruction and traditional classroom instruction. A 2009 meta analysis from the U.S. Department of Education examined the literature regarding the quality of learning in online and hybrid (or blended) courses (Means et al. 2009). Th is analysis found slightly increased learning outcomes from purely online learning (compared to fa ce to face instruction) and that blended instruction resulted in significantly increased learning outcomes (compared to both traditional and online only instruction) Meta analyses are included with systematic reviews as the highest (and most rigorous) lev el of evidence (Bangert Drowns, 1986; Bangert Drowns & Rudner, 1991; Glass McGaw & Smith, 1981; Hoffert, 1997; Hunter & Schmidt 1990; Kulik & Kulik, 1989) and as such, the conclusions from the Means et al. study are a significant contribution to the field There are many factors influencing the quality of online learning including the level of institutional support for both students enrolled in online courses and faculty members providing online instruction To this end, the next section of the liter ature review discusses the literature concerning support for distance education and online learning Need for library support in Distance Education Supporting distance education is a complex administrative issue and while administrators should consider is sues in all a reas, instruction is still the focus and
29 ensuring academic quality is paramount The extraordinary growth in online learning noted in Allen and Seaman (2010) indicates that students are flocking to pursue online educational opportunities Now that students are viewing online learning as a viable alternative to face to face courses, universities must shift focus from marketing online learning to fully supporting the needs of online students Student affairs support of traditional college studen ts has been covered extensively in higher education literature (Cook, 2009) S upport for distance education students is an emerging topic of research (Husmann & Miller, 2001; LaPadula, 2003; Levy, 2003; Levy & Beaulieu, 2003; McLendon & Cronk, 2003; Peters 1998; Visser & Visser, 2000) Key topics in this burgeoning research area focus on the link between student perception s of connectedness (both with other students and the institution) ; retention, completion, and satisfaction ; and d eveloping and fostering a sense of community (Cain & Lockee, 2002; Kelly, 2001; Tait, & Mills, 2003) Dare, Zapata, & Thomas (2005) surveyed and compared distant and campus based students regarding a variety of issues including their relative experiences, perceptions of support needs, and use of and satisfaction with existing programs and services The researchers found a gap between the experiences and satisfaction of campus based and distant students, indicating a need for student support services to target distant students. In addition to technology and pedagogical support, institutional support for distance education should include library support The library can (and should) play a key role in supporting the needs of both faculty and students involved in distance education ( ACRL 2007; ARL 2005; Baird & Wilson, 2002; Black, 2003; Burge, 2002; Burge & Judith 2000; Burich, 2004a; Burich, 2004b; Cassner & Adams, 2005; Coffman,
30 2001; Gandhi, 2003; Haynes, 2002; Lorenzo, 2003; Pace, 2001; Perrone, 2000; Ramadevi, 1999) Libraries are using a variety of methods to provide support and instruction, including establish ing embedded librarian programs and services ( Kearley, & Phillips, 2004; Moore, 2004; Ramsay & Kinne, 2006) The next sections of the l iterature review describe the ways in which libraries are supporting distance education and how this project fills a need for increased contextualized online library instruction. Distance education support is a growing area of research and institutional administrators are, for the most par t, aware of the need for supporting faculty and students (Bird & Morgan, 2003; Cain & Lockee, 2002; Carnwell & Harrington, 2001; Fisher & Baird, 2005; Kelly, 2001; Ludwig Hardman & Dunlap, 2003; Rama & Hope, 2009; Simpson, 2002; Tait, 2000; Tait & Mills, 2 003; Thorpe, 2002; Wheeler, 2002) L ibrary support for distance education is a topic thoroughly discussed in the library and information science literature. To acknowledge the need for library support of distance education, the Association of College and R esearch Libraries (ACRL) created a set of standards : the Standards for Distance Learning Library Services (ACRL, 2007). The current iteration of the se standards has evolved from a set of guidelines documented in 1963 to the most recent 2008 version A defi ning facet of the ACRL Standards is the university community are entitled to equal access to library resources and services regardless of location and proximity to campus The Standards document is founded on the access entitlement principle and describes specific elements necessary to fulfill the principle Examples include accessible interlibrary loan and document delivery services, electronic materials (books,
31 journals, a nd databases) and access to instruction and reference services In addition to t he ACRL Standards, there are several professional organizations and conferences dedicated to the provision of library services for distance learners the most prominent of which include the Distance Learning Section (DLS) of the ACRL and the Off Campus Library Services Conference (OCLSC). The key professional library journal dealing with Journal of Library Services for Distance Learning L ibrarians serving significant populations of distance learners should keep up to date with the issues and current research in the field through professional participation and scholarship. Library Instruction and Information Literacy Prior to discussing lib rary instruction for distance and online education, it is necessary to provide background and context of library instruction and information literacy in general. This overview includes a brief introduction to information literacy, a discussion of library i nformation literacy instruction, and an examination of the connection between self efficacy and information literacy. The Association of College and Research Libraries defines information literacy as ACRL further expanded this definition of informatio n literacy and developed a set of competency standards to delineate what an individual needs to be able to do to be considered information literate The five standards include performance indicators and outcomes In addition to the operational definition o f information literacy presented by the ACRL, Livingstone, Van Couvering, and Thumin present a practical definition that
32 extends beyond the confines of higher education into the realm of society and participate in the world of & Thumin 2008 in Corio, Knobel, Lankshear & Leu, 2008, p. 110 ). The role of libraries and librarians in information literacy has become a defining characteri stic of academic librarianship; Marcum (2002) characterizes information years in the making ( p. 1). Librarians have accepted the banner of information literacy and transformed library instruction to include information literacy concepts in addition to traditional bibliographic instruction (Grassian, 2004; Thompson, 2002). Neely (2000) describes gaps in information literacy research surrounding psychological and socia l aspects Self efficacy could be an important factor that influences information literacy and library skills (Kurbanoglu, 2003) The following section discusses the theoretical underpinnings of self efficacy and its relationship to information literacy an d library instruction. Self e tasks (Bandura, 1977; Kear, 2000) Self efficacy was initially described by Bandura (1977) as a component of his social cognitive learning theory and according to Pajares (2006) Bandura remains the most prominent researcher studying the concept. Two significant characteristics of self efficacy relate to the embedded librarian project First ly self efficacy is context specific and can vary in spe cific domains. Secondly, self efficacy is positively linked to performance (Bandura, 1986).
33 These two aspects of the concept of self efficacy relate to self efficacy with library skills and information literacy There is a noticeable lack of published res earch on self efficacy and information literacy, with few studies considering all aspects of information literacy ; an in depth literature search revealed only five articles on the topic (Beile & Boote, 2005; Fletcher, 2005; Kurbanoglu, 2003; Kurbanoglu, Ak koyunlu, & Umay, 2004; Therefore, it is necessary for an evaluation of the embedded librarian project to include measures of self efficacy to attempt to both investigate the psychological and social aspects of inf ormation literacy and fill a gap in the literature. Library Instruction for Distance Education L ibrary services are an essential component of support for distance education (Bird & Morgan, 2003; Cain & Lockee, 2002; Carnwell & Harrington, 2001; Fisher & Ba ird, 2005; Kelly, 2001; Ludwig Hardman & Dunlap, 2003; Rama & Hope, 2009; Simpson, 2002; Tait, 2000; Tait & Mills, 2003; Thorpe, 2002; Wheeler, 2002) The focus of this project online embedded librarianship for the OnMed program necessitates an in depth discussion and evaluation of the literature related to library instruction for distant learners. To that end, t he following section examine s various modalities for providing library instruction to distance students in the context of stand alone or course i ntegrated instruction. Stand Alone Library Instruction for Distance Education Stand alone instruction occurs free of any ties to a particular course or curricula Several of the articles reviewed described a multi pronged approach to providing stand alone library instruction to distance learners ( Ferguson & Ferguson 2005 ; McLean & Dew 2006 ; Lindsay Cummings, Johnson & Scales 2006)
34 In 2005 Ferguson and Ferguson discuss ed the information literacy efforts of Charles Stu a rt University, an Australian inst itution comprised of a large proportion of distance learning students The university library explored email reference and instruction and online tutorials They int egrated each within the curricular structure but without customizing them for curricular ou tcomes or for particular courses While each of the instructional tools is discussed in detail, the authors include no evaluation or assessments and make no conclusions as to their success McLean and Dew (2006) reported on strategies employed by libraries to provide library instruction to distant students primarily focusing o n the use of asynchronous library tutorials distributed via CD ROM. In 2006 Lindsay, Cummings, Johnson and Scales of W ashington State University used a variety of criteria to assess s everal of their library online instructional materials The librarians developed assessments for various tutorials including text based, virtual tour and animated tutorials. The results of the assessments indicate that the tutorials increase student confidence with the materials they cover However, the quizzes at the end of the assessment show ed that although confidence levels increased, student perfo rmance and ability to demonstrate skills was poor even after completing the tutorials Much as online learning has evolved, online library instruction has evolved from linking to handouts, online pathfinders and web quests, to the development of a variety of online tutorials Unlike much of the other online library instruction literature, many of the articles dealing with online tutorials consistently include assessment efforts with detailed description s of research methodologies.
35 One of the higher qualit y examples of assessing online library tutorials is evident in the work of Churkovick and Oughtred (2002) The librarians at Deakin University in Australia compared learning outcomes between one of their online tutorials (Smart Searcher) and face to face l ibrary instruction The assessment instrument included questions relating to confidence with information literacy skills and a multiple choice quiz to test information literacy proficiency. Quizzes were administered pre and post instruction. The assessmen t results indicated that post test scores improved considerably regardless of instruction method, but score s were higher for face to face instruction than for the online tutorial. Analysis showed a significant difference between the methods of instructiona l delivery Using the data collected, the authors assert that to face library instruction is preferred to online instruction and teaching efforts will focus on face to face instruction accordingly. The Texas Information Literacy Tutorial (TILT) was one of the first online, interactive, comprehensive information literacy tutorials in the United States After the tutorial was launched, university libraries nationwide either linked directly to the tutorial or modified it for their ow n libraries As this tutorial wa s widely used, it wa s imperative to evaluate its effectiveness Orme (2004) studied TILT by dividing participants into cohorts : o ne cohort was assessed with no information literacy instruction, one cohort participated in onl y face to face instruction, one cohort completed TILT, and the last cohort completed both instructional formats. The author of this study took a novel approach to assess learning after completion of the instruction, the librarians tested the students the semester after instruction occurred. The purpose of waiting was to determine if persistent learning
36 occurred or if the learning were superficial and limited to the semester in which the instruction took place. Results indicate d that TILT instruction was as effective as traditional face to face instruction Continuing to develop and assess stand alone library instruction for distance and online learning will help librarians provide a comprehensive i nstruction program regardless of the format of educational delivery. However, while both of these studies report findings from well designed studies a wealth of education research indicates that when comparing instructional delivery types, such as compari ng face to face and online instruction, there is usually a finding of no significant difference (Russell, 1999) In light of the no significant difference debate, (Clark 1983; Kozma 1991) the results from these and similar studies should be thoroughly evaluated. Therefore assessing the impact of stand alone online library instruction should focus on evaluating efficacy rather than comparing delivery methods ; a more valid research qu estion would investigate the difference in instructional design methods rather than delivery mediums. Course Integrated Library Instruction for Distance Education Course integrated instruction is contextualized and designed to supplement coursework. Librar y research supports the need for course integrated, contextualized instruction (Badke, 2009; Stein, 1998) The theories of situated cognition (Brown, Collins & Dugid, 1989) and cognitive apprenticeship (Collins, Brown & Newman, 1990) provide the theoretica l basis for the above findings regarding contextualized library instruction. Situated cognition describes the need for instruction to be tied to authentic activities and environments to facilitate knowledge acquisition, application and synthesis (Collins, Brown & Dugid, 1989) Cognitive apprenticeship is conceptually similar to situated cognition and refers to the need for instruction to be integrated into authentic
37 activities (and their appropriate social context) to further equip learners with the functio nal knowledge appropriate to disciplinary domains (Collins, Brown & Newman, 1990) By applying these theoretical constructs to library instruction it can be conclude d that librarians must integrate the principles of information literacy and library / resear ch skills to the course and/or curriculum and connect them to specific products and assignments for library instruction (regardless of delivery format) to have maximum efficacy. In the context of distance and online learning course integrated instruction includes integration into the course management system, synchronous instruction, and more recently, online embedded librarians Librarians providing course integrated instruction to distant or online students are using one of the previously mentioned moda lities or combining modalities. For example, Librarians at Nova Southeastern University attempted three types of instruction: asynchronous (via streaming media), synchronous (via compressed video and NetMeeting), and blended (via WebCT). The authors descri be their experiences with providing instruction to remote sites using compressed video and comment that this method was unsuccessful due to logistical and technical concerns, but offer no evaluative dat a to support their conclusions Synchronous instructio n with NetMeeting was more successful and the authors collected dat a from their trials to support its continued use. Many of the 4.6 million online students (Allen & Seaman, 2011) are taking courses supported by the expanding course management industry K ey players in this field include Sakai, Moodle (both open source products), Desire2learn, Angel, and Blackboard Learn TM (all proprietary products) One trend identified regardless of the
38 platform is the increasing presence of librarians and libraries withi n the course management system s (CMS) ( Cipkin, 2002; Jobe & Deborah 2000; Kearley & Phillips, 2004; Markgraf, 2004; Moore, 2004) Library integration into CMSs ranges from including library links, to monitoring a library related discussion board, and in some cases offering library specific credit bearing courses Tunon (2002) describes her experiences with a research literacy course for Education Doctoral Studen ts in a distance education program and compares the online course with a face to face version Both courses focused on teaching students how and when to find information for their literature reviews and instructional topics included using subscription ele ctronic databases and indices, free Internet resources and print resources. The online course was conducted through WebCT, the Feedback indicates that while library content was beneficial to doctoral student s, the amount of work required to participate in a course was more than the single credit hour earned. Libraries have been providing course integrated synchronous instruction to remote and off campus learners for years, primarily face to face (by travelin g to remote sites) or through various types of video conferencing ( Chakraborty & Victor 2004; Dunlap 2002; Henner 2002; Hricko 2001; Sochrin 2004) New technologies provide new opportunities for synchronous instruction to distance students While the education literature is replete with articles describing web based synchronous communication for distance education (Anderson & Garrison, 1995; Finkelstein, 2006; Hampel, 2003 & 2006; Macdonald, 1998; Schullo et al. 2005), the library literature is bereft of literature on the topic The few articles dealing with online synchronous communication in
39 libraries were describing the use of tools for purposes other than instruction (primarily to facilitate virtual reference) One of the only clear examples of usi ng synchronous audiographic tools for library instruction is illustrated in Kontos and Henkey (2008) T he instruction, identif y strengths and weakness of the platform, and pro vide standard tips for librarians wishing to use this tool for library instruction. More research is needed to evaluate the usefulness and efficacy of providing synchronous library instruction via synchronous audiographic tools. One incarnation of online e mbedded librarianship is a form of course integrated instruction that utilizes many of the stand alone and integrated instructional components discussed above including online tutorials, integrating into the course management system and synchronous instr uction. Embedded L ibrarians As previously discussed, online embedded librarianship is an emerging service and is being reported with increasing frequency in the literature. While the focus of this librarian has a variety of meanings including librarian involvement and integration at the macro (college, department, program, research team) and mic r o (course) levels Regardless of of a librarian into the environment at various levels (Bozeman & Owens 2008; Edwards, Kumar & Ochoa 2010; Dewey 2004; Dugan 2008; Freiburger & Kramer 2009; Hall 2008; Lillard Norwood, Wise, Brooks & Kitts 2009; Kesselman & Watstein 2009; Matthew & Schroder 2006 ; Rudin 2008; Shumaker & Talley, 2009; York & Vance 2009) While various models of embedded librarianship ar e described in the
40 literature (including librarians integrated in to research teams and the clinical environment), this project focuses on librarians embedded granularly at the course level Edwards, Kumar & Ochoa ( 2010 ) provide a detailed literature revie w in their article discussing the assessment of an online embedded librarian project for graduate students enrolled in educational technology In their mixed method assessment, the authors found that while the presence of an online embedded librarian was b eneficial for both students and faculty, more detailed and rigorous assessment efforts are required. Th e identified need for additional research is directly addressed by this project, specifically regarding evaluating and comparing the quality of student a rtifacts. The articles in this section of the literature review focus on librarians embedded into online courses, as that is the focus of the embedded librarian project described in this document. An in depth literature review of articles discussing librarians embedded into online courses is covered in York and Vance (2009). The authors performed a literature review and surveyed librarians t o develop their best practices The survey consisted of a 2 1 item questionnaire distributed to librarians via professional lists. Through the resulting analyses, York and Vance developed seven recommendations for librarians providing embedded librarian services in online courses. Three recommendations highlight th e significance of a working relationship with the CMS (Course Management System) administrator, having a permanent link to the library in the CMS, and going beyond that library link. Other best practices include avoiding overextending the librarian, select ively
41 choosing courses that will reap the maximum benefit from the experience, actively participating in the class, and marketing the service to other faculty members. As indicated in the literature it is clear that embedded librarianship is an area libra ries are exploring with generally positive feedback Lillard (2009) took a novel approach to train ing future librarians for their roles as potential embedded librarians by embedding library science graduate students into online graduate courses. The librar y science students received course credit for their participation in the project by enrolling in a special topics course and acting as embedded librarians for distance nursing students. The student librarians collaborated with the course faculty members pr ior to the start of the course to discuss their level of participation. (Lillard et al. 2009, p. 14) This experience working with faculty provided the nascent librarians with important collaborative experience and reinforced the significance of working wi th faculty members. Student embedded librarians were allowed varying levels of involvement in the courses in which they were embedded ranging from complete immersion and relative freedom to contact students to a more conservative approach. There were tw o aspects to the feedback for Lillard et al. the nursing students regarding their embedded librarian experience and feedback from the library students regarding their perceptions. The nursing students had generally positive feedbac k and reported that the usefulness of the embedded librarian was related to the course topic and how far along in their program the course occurred. Several students indicated that the instruction and embedded librarian experience would have been more bene ficial if it had been offered earlier in their program. The embedded student librarians perceived the experience as helpful for their graduate
42 library education and had mixed reflections. Not surprising, library students who were allowed more freedom and c ontact with their nursing student patrons reported higher levels of satisfaction with the experience. Lillard et al. have utilized feedback to implement changes and are planning the second semester of the project ( 2009 ). Design and implementation informati on from the embedded librarian projects described above directly influenced the embedded librarian project being evaluated in this document It is clear from the literature that librarians are investigating innovat ive methods to provide library instruct ion to support online and distance education. However, g eneral trends in the library instruction literature include a noticeable gap in the research; there is a lack of valid research practices and assessment. Generally the articles focus on implementation and are lacking in rigorous assessment of the discussed instruction (Behr 2004; Ferguson 2005; Kontos 2008, etc) Assessment is crucial because librarians must be able to evaluate the effectiveness of their instruction to demonstrate alignment with accept ed standards such as the ACRL Information Literacy Standards (ACRL, 2000). The lack of assessment noted in the literature searches for this project emphasize the need for a valid, multi face ted assessment approach to demonstrate the impact of the em bedded librarian.
43 CHAPTER 3 METHODOLOGY Introd uctory Remarks Drawing upon the literature and prior research presented in chapter two, the embedded librarian project is described in this section Elements of the methodology section include a review of the r elevant methodology literature, a description of the case including the setting, participants, design of the embedded library content (focusing on the instructional design), and description of the theories influencing the instructional design of the materi als. T he chapter concludes with a detailed description of the research design data analysis and validity. As with all human subject research at the university, the research protocol was submitted and approved by the Institutional Review Board ( IRB 02 for non medical research U 405 2011). Methodology Review The instructional intervention used in this study (the embedded librarian experience) was evaluated and described using a mixed methods case study approach including pre and post tests (of self effic acy with library skills and library skills performance), log data analysis, citation analysis to evaluate the quality of the sources used in the participants annotated bibliography assignments, qualitative analysis of narrative reflections, and a post cour se interview with the instructor The following literature review will describe studies employing similar methods and when possible make connections to the evaluation of the embedded librarian project. Assessing S elf E fficacy As mentioned previously, there is a lack of research in the area of assessing library skills/information literacy self efficacy. Two studies describe the development of
44 validated instruments to measure self efficacy of information literacy (Kurbanoglu, Akk and two discuss evaluation of self efficacy of information literacy as a component of a larger project (Beile & Boote, 2005; Fletcher, 2005; Ren, 2000) Because the use of a reliable, validated instrument decreases the chances for error in a study and increases the overall validity ( Seliger & Shohamy 1989) the validated instruments will be discussed in detail In 2004 Kurbanoglu et al. first created and validated a forty item questionnaire (alpha reliab ility coefficient = 0.78) and then developed a shorter seventeen item instrument with an alpha reliability coefficient of 0.82 After initial testing the authors created a twenty eight item instrument with acceptable validity and reliability ( C al pha scores of .92 for the Turkish version and .91 for the version translated into English) This was the first attempt documented in the literature to develop and validate an instrument to measure self efficacy with information literacy skills. Another sel f efficacy information literacy instrument was developed and tested by and Diaz in 2005 and the questions are correlated with outcomes enumerated by the ACRL Standards (ACRL, 2000) This direct correlation with established standards provid es the instrument with practical significance as the library instruction efforts of many institutions are also aligned with the se standards. A description of the instrument development and validation is included under the study design and data analysis hea ding in this section. The authors comment that their instrument is useful to measure changes in self efficacy that may occur as a result of exposure to library instruction The instrument is particularly useful to the embedded librarian project to evaluate pre and post implementation self efficacy scores and
45 investigate possible correlations between self efficacy prior to participation in the embedded librarian project. As previously mentioned, two studies that efficacy with information literacy did not utilize validated instruments While the authors did not use va lidated instruments, the studies are of sufficient quality to be useful for researchers seeking to investigate similar topics. The first of those studies was conduc ted by Ren (2000) in which the author conducted a quasi experimental study to determine if students self efficacy with library searches improved after being exposed to library instruction Student participants in this study were asked to complete thirteen tasks and question their self efficacy while performing the tasks Search performance was self assessed and externally assessed by librarians Conclusions from this study include d an increa se in self efficacy a s a result of training and students who exhibited relatively higher self efficacy in the pre test also exhibited relatively higher self efficacy in the p ost test This study paved the way for future research including the Beile and Bo ote (2004) study. Beile & Boote (2004) conducted a well designed and well analyzed quasi experimental study to compare self efficacy scores and library skills performance for face to face and online instruction The authors clearly describe their methodolo gy and data analysis detailing the various statistical tests performed, which included a multivariate analysis of variance (MANOVA), multivariate analysis of co variance (MANCOVA) (with pre instruction self efficacy levels, pre instruction library skills s cores, and exposure to prior library instruction as covariates) and paired t tests The analyses utilized were appropriate to the research questions and study design and the
46 discussion of results confirmed the data. In addition to solid design and analyse s, the authors clearly indicate study limitations, which include the use of a single perhaps limited performance measure (a library skills test) and a lack of validation with the population sample Another unstated limitation is the use of un val idated instruments to measure self efficacy and library skills performance. The Monoi, (2005) instrument was adapted and utilized to evaluate information literacy self efficacy and library skill performance in the OnMed case and this will be further discussed in the research design section. Citation A nalysis Citation analysis is a form of bibliometrics that has been employed by librarians for years in a variety of contexts. Historically it has been employed in a collection management capac ity to evaluate library collections (Hirose & Nakazawa, 2007) More recently, citation analysis has been used to evaluate the differences between expert and novice searchers ( Whipple, McGowan, Dixon & Zafar, 2009 ), t he quality of written artifacts includin g journals (Bauerly & Johnson, 2005; Brazzeal & Fowler, 2005; Bruer, 1985; De Groote, 2008; Drori, 2003; Eckel, 2009; Edzan, 2008; Gao, Yu, & Luo, 2009; Herther, 2009; Jiao, Onwuegbuzie, & Waytowich, 2008; Olatokun, 2009; Sam & Tackie, 2007; Sherriff, 2010 ; Thomas, 2000; Tunon & Brydges, 2006 & 2009), and the effectiveness of library instruction (Brunvand & Pashkova Balkenhol, 2008; Hurst & Leonard, 2007) Citation analysis is a general term used to describe an analytic study of the number and type of cita tions in a given document Neuhaus and Daniel (2008), Herther (2009) and Sun (2007) discuss the various uses of citation analysis in library and information science research
47 of Science, Journal Cita provide times cited information about specific journals and articles. Librarians and researchers use these databases as one source of data in citation analysis Other data sources in citation analys is focus on written artifacts submitted by students, especially at the graduate level. For the purposes of this study, the literature review will focus on relevant studies describing the use of citation analysis to evaluate the quality of student writing o r the impact of library instruction Most of the articles reviewed analyzed written artifacts from traditional, on campus institutions and courses; however, Tunon & Brydges (2009) analyzed citations from the reference lists of doctoral dissertations and made comparisons between traditional and non traditional (primarily distance education) institutions. The authors used a multi pronged approach to analyze the dissertations including a traditional statistical analysis (descriptive statistics of citation p atterns), an objective rubric and a subjective rubric. Both rubrics were validated a process which was discussed at length in Tunon & Brydges (2005) The objective rubric assigned points for the quality of a citation based on the type and age of referenc es while the subjective rubric assessed various aspects of the breadth and depth of references cited in doctoral dissertations. Tunon and Brydges conclude d that the quality of dissertations in traditional and non traditional institutions have more commonal ities and few differences with the exception in a single aspect of citation quality The breadth of the resources used was significantly higher for the traditional group demonstrating that the students enrolled in traditional doctoral programs used a wid er variety of resources. The use of the two rubrics is an interesting
48 and informative approach to gauge the quality of dissertations and can be used to evaluate the effectiveness of instruction Two key studies use citation analysis as an indicator of the quality of student written artifacts and evaluate the effectiveness of library instruction based on the quality In 2007, Hurst and Leonard compared the term papers of three groups of students, two who received library instruction and one who received no library instruction The reference lists of the papers were analyzed for two main factors : the total number of unique citations and the variety of source types (peer reviewed journal articles, non peer reviewed journal articles, books, proceedings, web sites, etc.). Basic descriptive statistics including tests of differences, such as means and proportions were utilized to determine if the differences in number and type s of citations was statistically significant. The authors found that there were statistically significant differences in the quantity of citations and the variety of resources cited between the students who were exposed to the library instruction and those who were not This type of citation analysis was used to evaluate t he written artifacts of students who participate d in the voluntary embedded librar ian project and provide d empirical evidence for requiring intensive library/information literacy instruction. Rather than using citation analysis to evaluate the effectivenes s of library instruction, Pashkova Balkenhol (2008) used citation analysis to determine the need for incorporating new topics (government documents research) into existing library instruction sessions The analysis showed definitively that students primari ly cited books and journals and made little use of government information. Data from the analysis was used to support the incorporation of new topics into the traditional information literacy
49 sessions From the literature discussion above, it is clear that libraries are using citation analysis to evaluate research quality and instructional impact. Case Stud y and Qualitative I nquiry Case studies are commonly used in the social sciences, particularly educational settings including libraries and health care (Y in, 2009) In fact, the OnMed program itself is being evaluate d using a case study approach (Clark, 2010 ), therefore a case study approach to evaluate the embedded librarian within an OnMed course fits well with the overall program evaluation The use of c ase studies in library research is described by Powell (1997) in the third edition of his book Basic Research Methods for Librarians which is frequently used in library and information science education to prepare librarians to conduct research. Powell de scribes case study research as a qualitative methodology and its uses in answering research questions relevant to practicing librarians (Powell, 1997) While others may argue with the description of case study study research in librarianship is illustrated by the number of case studies published in the field ( Clayton, 1995; Foote, 2001; Watson B oone, 2000). Regardless of whether or not case study research can be characterized as qualitative research, qualitative research methodologies are a natural fit within the research of librarianship (Davis 1990, p. 327). Ethnographic approaches are used in academic libraries to understand the culture of the users and help librarians improve services and/or instruction A prime (and heavily cited) example is University of underg raduate students to rethink their services and library spaces (Foster & Gibbons, 2007).
50 The OnMed Embedded Librarian Case Overview The following sections will describe the details of the case including the setting and participants outline the embedded l ibrarian content and finally discuss the theories influencing the instructional design of the embedded librarian materials. Setting Distance and online learning at the University of Florida has exhibited tremendous growth and continues to expand as the in itiatives were provided official administrative support from the university p resident : To facilitate this change, we are seeking a distance education infrastructure company to help us increase the 52 programs we now ( Machen, 2010) The univer sity offers b oth graduate and undergraduate degrees through distance learning O nline learning in both the College of Education and the Health Science Center colleges is important to the context of the project because of the overlapping nature of the OnMed Program. Distance learning programs within the College of Education includ e topic areas such as Educational Leadership, Educational Administration & Policy, Higher Education, and Educational Technology Distance education courses vary in format and delivery mechanisms and include online courses, remote site courses, and blended courses. Online courses are asynchronous and deliver content via network technology utlizing Moodle as a course management system The College of Education is the only c ollege on campus that uses Moodle; the rest of campus used WebCT until recently when Sak a i was adopted Using a non campus supported course management system necessitate d that the product be supported within the college. The College of Education has a dedicated Distance Learning office to support Moodle and online learning. The
51 use of different course management systems is significant to this project because the OnMed participants tak e courses in Moodle ; however, they use Sakai to facilitate their online teaching The Health Science Center (HSC) is comprised of six colleges: Dentistry, Medicine, Nursing, Pharmacy, Public Health & Health Professions, and Veterinary Medicine Faculty in those colleges can have multiple roles as teachers, researchers, and practitione rs /clinicians. Faculty development typically occurs at the college level; however, recently an interest group was convened to discuss collaborating on faculty development efforts across the HSC Improving teaching practices is one of the areas included in the faculty development curricula The College of Medicine has created a certificate program to focus on clinical teaching and pedagogy of the attending physi cians referred to as the Master Educator Fellowship (MEF) program All of the OnMed cohort members completed the fellowship previously and have similar prior knowledge about pedagogy and clinical education. There is a growing amount of distance and online learning occurring i n the HSC. Distance education efforts in the HSC include fully distance degree s (through remote sites and online) and online courses for campus based students In keeping with the rest of the university (except for the College of Education), the HSC college s use Sakai as the course management system Faculty teaching in all of the s ix HSC colleges need familiarity with concepts and practices in educational technology, specifically as it relates to teaching and learning online. The OnMed program was designed to fill the niche of providing HSC faculty with the background and skills nee ded to employ educational technology to enhance clinical teaching. Faculty, staff, and students in the
52 HSC have access to a comprehensive specialized library known as the Health Science Center Library. The Health Science Center Library (HSC Library) is housed within the Health Science Center in direct proximity to the college s it serves. While it is physically housed in the HSC administratively the HSC L ibrary is a part of the larger campus library sys tem known as the Smathers Libraries. The library is well known for its established liaison program ( Tennant, Butson, Rezeau, Tucker, Boyle & Clayton, 2001; Tennant, Cataldo, Sherwill Navarro & Jesano, 2006; Feree, Schaefer, Butson & Tennant, 2009 ) wherein a librarian liaises to a particular college or department facilitating the establish ment of expertise with the subject area and the develop ment of strong working relationships with the users The HSC L ibrary currently provides face to face course integrate d instruction as well as stand alone instruction on various topics including database searching and citation management software. Liaison librarians are integrated into their distance and online programs to a varying extent, but to date there have been no opportunities for a truly embedded librarian program. Participants Participants in this study are practicing health care professionals enrolled in the inaugural cohort of the OnMed program. While OnMed participants are part time graduate students in the Co llege of Education, they are full time faculty in the Health Science Center, primar ily in the Colleges of Dentistry, Medicine and Pharmacy The first cohort is comprised of six students representing the Colleges of Dentistry, Medicine, and Pharmacy Four o f the students are College of Medicine faculty and attending physicians at the UF & Shands HealthCare system representing various departments including Anesthesia and Pediatrics.
53 The Embedded Librarian Project Course integrated and contextualized instruc tion occurred in the course EDG 6931 Issues in Educational Technology Research The course is specific to the OnMed program and as such is comprised solely of practicing health professionals. Rather than focusing on specific methodologies, the course is designed to provide an introduction to and overview of research in educational technology S tudents were required to develop a research q uestion, search the literature, critically evaluate research articles and synthesize the research. The final course deliverable is a brief ( ten item) annotated bibliography focusing on a research question of interest Of the courses offered during the spri ng 2010 semester it is the best fit for the librarian researcher knowledge and experiences. Building upon the literature and prior experience with online embedded librarianship for graduate educational technology, the instruction al intervention was desi gned to include primarily asynchronous components, with occasional synchronous options as necessary to provide optimal support for the students. The EME 505 4 Foundations of Educational Technology embedded librarian project described in Edwards, Kumar and O choa (2010) serves as the pilot for the OnMed project and informed not only the implementation of the embedded librarian project but also the case study evaluation and description of the implementation of that project as well Findings from the pilot suppo rt the creation of customized library instructional material and the integrated availability of a librarian, both synchronously and asynchronously Specific experiences from the pilot informed the OnMed design For example, the pilot project included week ly optional synchronous office hours, which none of the participants used and the office hours were discontinued in the OnMed implementation
54 option in the form of a M eebo instant message chat utility students could use to contact the librarian. One of the recommendations for future research from the pilot concentrated on the quantitative assessment of the embedded librarian and suggested that future pre and post asses sments be paired for improved data analysis, a recommendation that was implemented in the OnMed embedded librarian case Using conclusions and lessons learned from the pilot, the librarian researcher designed and implemented an embedded librarian instance based upon the Morrison, Ross, and Kemp (MRK) instructional design model (Morrison, Ross, & Kemp, 2006). C onsistent with th e Morrison, Ross, and Kemp (2007 ) instructional design model, the exact configuration of the instructional components that comprise th e embedded librarian experience was influenced by the needs analysis, including analysis of the learners, context of the study and tasks required of the students The needs analysis was based in part upon a survey administered by the OnMed program coord inators. It was determined from the survey that participants had a range of technology skills, but the majority of respondents were open to learning and using technology (OnMed Evaluation Materials Participant Survey 2010). Additional information about needs of the learners was gleaned through close collaboration with the course instructor, as faculty collaboration is a key component of embedded librarianship ( Bozeman & Owens, 2008; Dewey, 2004; Dugan, 2008; Hall, 2008; Kesselman & Watstein, 2009; Lillard et al. 2009; Love & Norwood, 2007; Matthew & Schroeder, 2006; Moore, 2004; Ramsay & Kinne, 2006; Kearley & Phillips, 2004) From consultations with the instructor it was determined that library support
55 should not only include procedural task orie nted support (a demonstration of the MeSH browser tool for example), but it should also include higher level cognitive tasks to reinforce instructor developed content (for example critical evaluation of research). For the OnMed students the instructional g oals focus ed information literacy skills in the areas of health science education and educational technology The library instruction included foundational to educational technology and other con cepts specific to the needs of the course Based upon the OnMed survey data and discussion with the course instructor, the embedded librarian content was developed to provide general course support and to support the course modules for six of the eight we eks General resources included an OnMed LibGuide, a VPN installation demonstration video, and an optional asynchronous library help discussion forum See T able 3 1 for details on the course modules and corresponding library content. While the course instr uctor and librarian decided that most of the library content should be asynchronous to allow for maximum flexibility for the participants (especially due to their demanding clinical responsibilities), initially a synchronous session was scheduled to provid e training on the use of RefWorks, a library support ed bibliographic management system. The College of Education uses and supports Elluminate (now referred to as BlackBoard Connect) for synchronous audiographic communication T he librarian researcher was an experienced Elluminate user; therefore Elluminate was the chosen platform for the synchronous session An additional benefit of Elluminate is that sessions can be recorded for later viewing thus increasing the flexibility for users who either could no t attend the synchronous session or wished to review the content Due to
56 scheduling issues, the synchronous RefWorks session was rescheduled and then it was convert ed to an asynchronous mode using the recording feature in Elluminate. Case study data was co llected by the librarian researcher who acted as a participant observer in the course by interacting with the students and instruct or in order to establish familiarity with the course content, participants, and participant research interests As mentioned previously, the embedded librarian concept exemplified in this project extends beyond the creation of predetermined instructional materials to interactive participation in relevant discussions and the creation of adaptive content to suit not only the struc tured course content but also an unstructured organically developed content as well. Theories Influencing the Embedded Librarian Content Design who incorporate educationa l technology/instructional design skills and knowledge with the traditionally held librarian roles (Bell & Shank, 2004 ; 2006) The literature reflects the efficacy of using an instructional design approach in general and more research is needed to explore the effect of various instructional design models on library instruction. The instructional intervention described in this project was developed using the Morrison, Ross, and Kemp (MRK) (2006) iterative design model. Message design is a component of the M RK instructional design model and is increasingly significant as multimedia is used often to support online instruction & 2005) in the area of multimedia and cogni tion influence the design of learning objects developed to support the online library instruction in this project. Through his research with cognition, Paivio identifies three types of processing (representational, referential,
57 and associative) and suggest s that recall is improved by presenting information in verbal and visual formats According to dual coding theory (DCT), cognition involves verbal and non verbal subsystems and instruction that requires both subsystems will maximize cognition and recall ( Paivio, 1991) types of processing directly inform research on the cognition of multimedia learning Representational processing refers to the direct activation of either the verbal or non verbal subsystems ; referentia l processing describes the activation of one of the subsystems by the other (the nonverbal system activates the verbal or vice versa) ; and associative processing refers to the same system (verbal or nonverbal) being activated a variety of research studies and his work is used to inform other theoretic al work including the cognitive theory of multimedia learning. Cognitive theories of multimedia learning influence the design of online learning objects. Mayer is a leading researcher in this area. Mayer and Moreno (1998) use Clark to inform their research and make conclusions regarding multimedia learning In a 1998 article Mayer and Moreno discuss the results of their research regarding a split attention effect in multimedia learning and use it to support dual processing theory T conclusion that connections between visual and auditory channels can only occur when both images and verbal inf ormation are stored in working memory simultan eously (Mayer & Moreno, 1998).
58 design, a 2005 paper (Mayer & Moreno) directly outlines five principles for multimedia design to guide instructional design of multimedia learning objects and direct further Multiple Representation I t is better to explain a concept using more than one mode. Contiguity Principle W hen presenting information verbally and visually the words and images should be presented simultaneously rather than separately. Split Attention Principle Words should be presented through narration rather than visually (through text) to reduce the cog nitive load on the visual processing channel and split the load between the verbal and visual channels. Individual Differences Principle The multimedia effects on cognition vary according to differences in the individual learners. Coherence Principle Shorter summaries of information (with fewer words and important words and images highlighted) result in more learning than longer, more detailed descriptions. T hese principles are research based and grounded in learning theories and as such they provid e concrete guidelines for designing effective learning objects Because this project involves online library instruction it is important to ground the distance and in teractions i n distance education Regarding t ransactional distance, this project focuses on minimizing the transactional distance between distance learning students and library content by providing appropriately designed learning situations. The aim of lib rary instruction is to increase interactions, especially learner content interactions by facilitating access to and critical evaluation of relevant literature and other resources.
59 Research Design and Data Analysis Overview As previously stated, the purpos e of this project was to develop and evaluate an embedded librarian program The primary research question is H ow does the presence of an online embedded librarian influence graduate students experience in an online educational technology research class as defined by these specific aspects: Self efficacy with information literacy and library skills as measured by changes in performan ce on a pre and post assessment Library skill performance as measured by changes in performance on a pre and post assessment ns in an annotated bibliography Reflections on the embedded librarian experience and processes of searching and critical ly evaluating the literature The online embedded instruction was assessed using a mixed methods, multi pronged evaluation strategy in a case study research design As noted by Yin, case study methodology allows the researcher to explore social phenomena and answer questions concerning While traditionally case studies focus on qualitative methods (Patton, 2001; Yin, 2009), the research questions and nature of this project necessitate quantitative data as well as qualitativ e data. The research questions seek to describe if and how an embedded librarian influences the OnMed Issu es in Education Research course; therefore the purpose of the research fall s into the category of an exploratory case study The selection of the uni t of analysis (or case) for this project was based upon the information sources, and the health care practitioners who comprise the study
60 population. Additional factors influenci ng case selection focus ed on the unique context and online graduate students. Quantitative assessment strategies include d a pre/post survey designed to gauge eived self efficacy with using library resources and performance on select library related tasks Additional quantitative assessments focus ed on an analysis of the learnin g. Performance on the pre/post tests and citation analysis results were compared by pairing the data for each respondent and performing paired t tests. T tests w ere performance on both the overall assessment and individual item s Statistical analysis was conducted using the Statistical Package for the Social Sciences (SPSS) version 1 9. Assessing Self Efficacy and Library Skill Performance efficacy with library resource and information literacy concepts a twelve was adapted with permission. The instrument was aligned to learning outcomes described by the ACRL Information Literacy Standards and has been thoroughly validated (Appendix A). A full description of the design and development of the instrument is included under the methodology review heading. In addition to the self efficacy scale, the pre/ post test performance on library related tasks. The information literacy/library skills assessments were administered through an online survey ( Z oomerang ) during the first and last weeks of the course (weeks 1 and 8) Responses were paired by the course instructor and de identified prior to sending to
61 the embedded librarian. Descriptive and inferential statistical analysis methods allowed for a thorough understanding of the data and investigation of a change in performa nce after the embedded librarian program is complete. Unpaired data was disregarded and paired t tests were conducted on the remaining data to measure individual change before and after the intervention Citation Analysis As previously stated, citation a nalysis (a form of bibliometrics) is traditionally a form of an annotated bibliography) were evaluated by the librarian researcher with rubrics (used with permission from a similar study by Tunon & Brydges 2006) to make comparisons between participants and non participants. The two rubrics used in the citation analysis procedure provide both objective and subjective assessment of the references without evaluating how t he citations were used within the body of the assignment. The methodology review section includes more detail regarding the design and evaluation of the Tunon and Brydges (2006) rubrics. Although reliability data for the rubrics is available from the Tunon and Brydges studies, because the context and researchers differ from the original study, it was important to establish reliability specific to the context of the OnMed course. An Inter rater reliability value was measured by having two librarians use the rubrics to evaluate The 13 which indicates moderate correlation and sufficient inter rater reliability to proceed with usi ng the rubrics to evaluate the annotated bibliography assignments.
62 In the embedded librarian evaluation project citation analysis (used as a measure of quality of student learning) included general descriptive analysis of the citations in the annotated bi bliography assignment that focus ed on citation patterns such as the mean age and frequency of references The objective rubric assessed a point value to each reference (according to criteria such as age, type of reference, etc.) and a total score for eac h reference list was calculated (Appendix B) The subjective rubric measured the quality of reference lists through expert review of the references on criteria including breadth, depth, and appropriateness for the topic (Appendix C ) See F igures 3 1 and 3 2 for a graphical overview of how the two rubrics were used. Citations were processed and analyzed using a procedure similar to the one employed by Tunon and Brydges (2006), with some modifications allowing for the shift from dissertation reference list (u sed by Tunon & Brydges) to annotated bibliography assignment and the overall purpose of the analysis. Specifically, the purpose of the Tunon & Byrdges (2006) analysis was comparative while the purpose of this analysis is strictly descriptive. The citatio ns were gathered from anonymous participant generated annotated bibliographies. Each citation was assigned a unique alpha numeric identifier that included a single letter to identify the annotated bibliography document (A F for six individual bibliographie s) and a number referencing the order the citation appeared in the bibliography . Once the citations wer e removed from the word document bibliographies they were entered into an excel spreadsheet listing all citations Categories matching the categories in the
63 objective rubric for document type were created and each citation was categorized In cases where a document meets several categories, they were assigned to the category for the function rather than form of the document For example, an online government report was counted as a report rather than a web page. See the results section for a breakdown of the citation categories and number of citations in each category Once the citations were categorized, each citation was scored according to both rubrics In addition to scoring each citation separately, a compiled score was generated for each annotated b ibliography by computing the mean scores of the citations included in the bibliography Only citations with an annotation were analyzed to focus on the intent of the annotated bibliography assignment, which was to locate, select and critique research arti generally not an issue as most annotated bibliography assignments included only those citations being annotated However, in one of the bibliographies a participant included additional ref erences for each annotation ; i n this case only the citation being annotated was included in the analysis. SPSS version 19 was used to provide descriptive statistics on the citations and bibliographies and the results are reported in chapter four. Qualitative Data A nalysis Qualitative data sources focus on narrative reflections in the form of participant narrative responses to questions designed to elicit reflective feedback The qualitative data was analyzed with a grounded theory approach (Glaser & Strauss, 1967) to embedded librarian, which includes instructional content and the prolonged presence of a librarian within the course Analysis was facilitated by the u se of the sof tware, NVivo 9 NVivo was useful for creating first level codes and querying the coded data to
64 facilitate the development of patterns and themes. Participants were asked to respond in writing to the following questions: What role did the libr arian (who offered instructional support and assistance) play in your process of completing the annotated bibliography assignment (describing research questions, refining your question, searching for literature, and finally evaluating and synthesizing the literature)? In what ways/How did your prior experiences in clinical research and literature searching influence your completion of the annotated bibliography assignment and use of the embedded librarian support? Reflections were submitted to the course in structor in writing, de identified, and forwarded to the researcher for analysis. One of the benefits of having the participants respond in writing is that the data did not require transcription and subsequent member checking to determine accurate represen tation allowed the researcher to not only query the participants to gain insight into the embedded librarian experience (thus helping to answer the research questions), but also tailor the questions according to the state of the embedded librarian project For of interaction with the librarian researcher Those observations caused the lib rarian researcher to hypothesize that the cause was possibly rooted in their years of experience as health care practitioners who are theoretically familiar with searching and evaluating the medical literature As a result of those observations one of the reflective questions was developed to explore the lack of interaction perceived high quality responses and to question the working hypothesis
65 Additional perspectives While most of the data collection strategies focus ed exper iences with the embedded librarian, it was important to consider the course project. A post implementation, semi structured interview was conducted to provide qualitative feedback. The interview was conducted face to face, recorded electronically and transcribed. To follow quality qualitative research protocol, the interview transcript was sent to the course instructor for member checking. The transcript was coded using an open approach and analyzed thematically See Appendix D for the interview instrument. Informal field notes were used in order to document the experiences of the librarian researcher Notes were organized by week and recorded in a web based document tool, Google Docs All data quantitative and qualitative, w ere analyzed and triangulated to evaluate the success of the embedded librarian program and describe the experience. A nalysis (including development of first and second level codes and thematic analysis ) of the participant reflections was discussed and verified with an external qualitative researcher to increase validity. Data from the course instructor and field notes from the librarian researcher serve d as triangulation points and were used to compare to the primary data from the participants in order to provide additional perspectives. Validity In an effort to provide an accurate evaluation and description of the embedded librarian program and holistic description of the experience, it is im portant to establish validity As Yin (2009) notes, the four factors describing the validity of social science research design apply to case study research as a subset of social science research.
66 According to Seliger and Shohamy (1989), internal validity is controlled by research design and data collection: "Findings can be said to be internally invalid because they may have been affected by factors other than those thought to have caused them, or because the interpretation of the data by the researcher is not clearly supportable" ( p. 95) In the evaluation of the online embedded librarian case study internal validity is affected by the design and instruments used (including the pre/post self efficacy measure and citation analysis rubrics) All data collection instruments were borrowed (with permission) from researchers who created and validated them in an attempt to ensure valid results. The self efficacy instrument was modified by the removal of one item and addition of two items (a multi part item to measure library skill performance [ item #12 parts a c ] and an item to measure confidence in completing the performance item [ item #13 ] ) The citation analysis rubrics were used without modification. Content validity on both instruments was fu rther established by expert evaluation by experie nced health science librarians Internal validity in case study research can be addressed in the research design and data analysis phases by using pattern matching, explanation building, addressing rival exp lanations, and using logic models (Yin 2009) External validity refers to the extent to which the study findings can be generalized to another context or a larger group: "Findings can be said to be externally invalid because [they] cannot be extended or applied to contexts outside those in which the research took place" (Seliger & Shohamy, 1989 p. 95). One of the criticisms of case study research is its perceived low level of external validity (Yin, 2009). Yin differentiates between statistical generaliz ations (common in survey research) and
67 analytic generalizations use to generalize case study findings to theory (Yin 2009). Despite these threats to external validity, it is possible to influence the validity and accuracy of the case study by member check ing qualitative data and compiling an audit trail for data and documents collected during the study (Yin, 2009) To that end all of the data collection instruments, transcripts, and reflective data are included in the audit trail.
68 Table 3 1. Embedded library content Week Course Module Embedded Library Content 1 Introduction 1) Introduction Forum 2) 2) Pre Assessment 2 Reading R esearch 1) RefWorks Elluminate Session 2) RefWorks Handout 3) What is an annotated bibliography 4) Critical Analysis LibGuide 3 Research Q uestions N/A 4 Mid Point 1) 2) 3) Journal Citation Reports Video 5 Building Your A nnotated B ibliography 1) ERIC demonstration video 2) MeSH demonstration video 3) Why Use MeSH video
69 Table 3 1 Continued Week Course Module Embedded Library Content 6 Working on your A nnotated B ibliography Search Support (including presentation on finding medical education literature and general database search techniques) 7 Annotated B ibliography Continued search support 8 Issues and C urrent R esearch N/A
70 Figure 3 1. Objective Rubric
71 Figure 3 2. Subjective Rubric
72 CHAPTER 4 RESULTS Introduc tory Remarks As academic librarians have continue d to integrate themselves into curricular activities the concept of the embedded librarian has evolved and implementation of embedded librarian projects are being described in the professional literature ( Cipkin, 2002; Jobe & Deborah, 2000; Markgraf, 2004; Moore, 2004; Ramsay & Kinne, 2006; Kearley & Phill ips, 2004; Shumaker & Talley, 2009 ) Currently assessment of these embedded librarians is limited to reacti ve feedback from students, with some data describing the experience from the perspective of the course instructor (Bozeman & Owens 2008; Edwards, Ku mar & Ochoa 2010; Dewey 2004; Dugan 2008; Freiburger & Kramer 2009; Hall 2008; Lillard et al. 2009; Kesselman & Watstein 2009; Matthew & Schroder 2006; Rudin 2008; York & Vance 2009) To that end, the librarian researcher developed a study of a context specific embedded librarian implementation with the purpose of describing the experience and measuring impact. The results chapter describes the data collected from a mixed methods case study that investigates how the presence of an embedded librarian infl uences an online graduate course Participants are full time health care practitioners and academic health care faculty in the areas of medicine, pharmacy, and dentistry enrolled in the OnMed program OnMed is a clinical educators with a background in educational technology. Data was collected for eight weeks from the sixth of March through the twenty second of April in 2011 Complete data (including pre and post test scores, citation metrics, and reflections) w ere available from four of the six OnMed students enrolled in
73 the course. Citation metrics and re flective data were available from all six students. A variety of data was collected on each participant, providing sufficient data with which to describe the e xperience and develop a working theory for how the librarian influenced participants to answer t he primary research question H ow does the presence of an online embedded librarian influence graduate students experience s in an online educationa l technolog y research class as defined by their self efficacy with information literacy, research quality, and descriptive narrative reflections ? This chapter reports the results beginning with the quantitative data ( resource access, pre and post tests and citation analysis) then move s on to the more descriptive qualitative data (narrative reflections and instructor interview) Informal observations from the librarian researcher were used to help triangulate the data to provide validity and reliability. Quantitative Data Student Access of Instructional Materials Six students were enrolled in the eight week course, some of whom availed themselves of the embedded library resources and provided feedback While direct interaction with the students was limited (one quest ion was received and it came during the last week of the course), analysis of the M oodle access log data shows that the instructional elements were viewed, in many cases multiple times by the same individuals Table 4 1 displays the access information for the library content The most heavily viewed resources include d the ERIC demonstration (11 views), the peer review and the OnMed Libguide (7 views).
74 Pre Post Assessment Data Information l iteracy s elf efficacy Participants completed voluntary pre and post implementation assessments designed to measure self efficacy with information literacy and library skills (Likert items) as well as performance on three library/research oriented tasks (o pen ended items). The pre and post implementation assessment data was collected via an online survey application (Zoomerang) and analyzed first to check if the data meet the assumptions for conducting paired t tests (independent observations, dependent var iable measured on an interval scale, normal distribution) by construction of a histogram, nearest quartile (NQ plot), and box plot After confirming that a paired t test was appropriate for the data, each of the L ikert scale assessment questions was evalua ted The descriptive statistics for the pre test scores shown in T able 4 2 indicate that the most frequently occurring confidence value is 2.0, demonstrating that for all 14 Likert s cale questions respondents fell between not confident and neutral, with a high level of variance in scores (.850) Post test scores demonstrated a much lower amount of variance (.216) and a mode falling between confident and very confident. Paired t test The null hypothesis for the paired t test was that the pre test and post test means were equivalent and the alternate hypothesis showed the pre and post test means were not equal. Results indicate that t here was a significant d ifference in the scores for the pre test s (M=3.06, SD=.92) and post test scores ( (M=4.23, SD=.46); ( t(13)= 7.17, p = .000 )) Due to the p value the null hypothesis is rejected, meaning the pre test means
75 did not equal the post test means. Statistically significant results indicate an increase in information literacy self efficacy as measured by the assessment. In addition to performing statistical analysis for the mean scores of all participants for each Likert item, a mean score ( for all the Likert items) was calculated for each participant. The descriptive statistics for pre and post tests for each participant ar e illustrated in T able 4 3, while T able 4 2 illustrates the paired t test data Data indicates that in all cases, there was a positive increase between the pre and post test and the exact change in mean scores ranges from .714 to 2.65. Table 4 3 illustrates all delta mean values. Library s kill p erformance In addition to assessing self efficacy, the assessment also contain ed items requiring participants to utilize library resources to complete various tasks Participants were asked to locate electronic access to a specific journal title and specify the dates available, use a database of their choosing to find an article on blended learning in undergraduate medical education, and use the library catalog to find a book in electronic format The per formance results for the first question regarding electronic access to a journal indicate that there was a 33.4% increase in correc t responses between the pre and post assessments. Pre test results from this question demonstrated reliance on general sources including Google Scholar with no mention of the use of subject headings In the post test all respondents indicated that they used a library database, and two mentioned the use of subject headings ( MeSH ) demonstrating improved search performance The final performance item required participants to use the library catalog to find an ebook on a
76 specific topic (blended learning) and all participants were able to complete this task satisfactorily in both the pre and post implementation assessments. Citation A nalysis Citations used in participant ge nerated annotated bibliographies were analyzed as an indirect measure of research quality The citation analysis used two validated rubrics to assign a numerical value to citations based on specified criteria Objective scores were calculated based upon th e type of resource (peer reviewed journals, books, dissertations, etc.) and the age of the citation (less than five years old, more than five years old, but less than or equal to ten years old). Each citation within a document was scored and the scores we re averaged to achieve a total score for the document. Subjective scores were calculated (by evaluating the document as a whole rather than evaluating each citation individually), based upon several factors including the type of source, age of source, dept h of the sources listed, and relevancy to the topic. Analysis of the citations from the annotated bibliography indicated that students used high quality sources Both objective and subjective scores for all bibliographies were relatively high Objective s cores ranged from 2.75 to 2.9 (3.0 was the maximum score) and subjective scores ranged from 17 to 20 (out of a total of 20 points available) See T able s 4 4 and 4 5 for a summary of citation scores and a breakdown of the descriptive statistics Almost a ll citations used in the annotated bibliography assignment were from peer reviewed publications, and the primary variance in the citation categories was in the currency of the articles When the bibliographies were analyzed with the subjective rubric crite ria the primary areas of variance included the currency of articles and the relevancy to the research question.
77 Qualitative Data Participant Reflections All six participants completed reflections, and the reflective data was analyzed for themes to develop a theory of how the presence of a librarian influenced participants. To increase validity, an external researcher verified that the data analysis procedure was appropriate and voiced no dissent with either the first and second level codes or themes that e merged from the data. However, recommendations were made to the librarian researcher to extend the text of some of the comments reflected in T able 4.6, when feasible, to provide additional context. Themes A variety of themes emerged from the data includi ng both major themes and minor themes There were four major themes identified: 1 T he nature of the annotated bibliography assignment and critical analy sis required in the annotations 2 U se of the library inst ructional materials 3 T he literature search process in general and specifically the search terms and strategies used within the process 4 T he process of finding useful and relevant articles The last two themes are both related to literature searching in general and are separate components of the process. Each of these major themes are discussed in more detail below See T able 4 6 for an illustration of the core themes with corresponding examples from the data. Annotated bibliographies and critical a nalysis A number of the responses discussed the nature of the annotated bibliography assignment and strategies used to approach the assignment While the concept of an
78 annotated bibliography was foreign to many of the participants, the intent of analysis and synthesis of the research was familiar and participants co mpared the process to familiar tasks such as writing a literature review for an article manuscript. In addition to strategies regarding the annotated bibliography assignment, participants commented on the critical analysis aspect of the assignment stating how the analysis of articles influenced their decisions and research. Library instructional materials One of the prevalent themes in the reflections was the customized library instructional content Librarian created instructional videos were mentioned in all A minor theme associated with the instructional content was the reduction of search anxiety as a result of viewing the materials One comment in particular highlighted a reduction in search anx iety after Overall, I definitely experienced much less anxiety about the literature search with a good research question in hand and with a few of the tips I picked up from the instructional tutorial on annotated ( OnMed reflections, line s 122 126) Similar comments regarding the instructional videos are displayed in T able 4 6 Search terms and strategies A significant theme within the reflections concerns the use of search terms and construction o f search strategies While some participants voiced experience and confidence with their literature searching ability (literature searching in general, not necessarily the use of search terms to construct search strategies), others expressed a lack of conf idence and anxiety about the search process To further complicate matters, some participants expressed both sentiments in the same reflection One particularly
79 enlightening comment relate d that while the participant wa s comfortable with searching for clin ical literature, searching for educational literature wa s different and introduce d a The experience as a clinician and literature searching tricks were helpful, however, when you branch into a new field of vocabulary and research I felt lost especially when it feels abstract in addition to new territory of design and ( OnMed reflections, lines 161 164) Additional examples of comments supporting the search term and strategy theme are illustrated in T able 4 6 Finding useful and relevant articles The final major theme uncovered is related to the literature search process in general and is also a component of critical analysis One of the final steps in the literature search is the ability to not only find results, but t o find results that are both useful and related to the topic in question Several participants commented on this topic and the relevance of citations to the research topic was one of the subjective cr iteria in the citation analysis See T able 4 6 for exam ples of comments supporting this theme. One particularly interesting comment describes the difficulty of finding relevant articles as a function of the massive increase in scholarly publications in recent years on it [my research topic] has li terally exploded so it is hard to sort through what is ( OnMed reflections, lines 49 50). Minor t hemes In addition to the major themes uncovered in the analysis of the participant reflections, a few significant minor themes emerg ed focusing on participants previous interactions with library resources and library instruction. These minor themes are discussed below.
80 Prior library experience A minor theme that emerged from the reflective data regarded prior library experience Particip ants commented on prior experience including prior literature searching experience, past experiences using a library, a lack of prior library experience, and also discussed prior library instruction and/or contact with a librarian Comments about prior ex perie nce with literature searching fe ll into two opposing broad categories a lack of confidence in literature searching abilities and confidence with literature searching Statements regarding a lack of confidence in searching are reported within the sea rch strategies and terms theme An example of a comment regarding search anxiety is: when you branch into a new field of vocabulary and OnMed reflections line 161) This participant expressed feelings of search anxiety and lack of confidence particularly regarding searching in an unfamiliar area or discipline. Illustrative examples of comments regarding positive experiences (or confidence in capabilities) with literature searching include d PubMed and was pretty facile at finding the literature OnMed reflections lines 32 22) and I am familiar with searches using PubMed, Google scholar, GALE etc and organizing references with Endnote and reference support for this" ( OnMed reflections, lines 64 67) The connections between this self expressed confidence with searching and results of the self efficacy and library performance assessments are discussed in the tri angu lation section in chapter five. Prior library instruction Prior library instruction and interaction with a librarian was a sub theme represented with in the theme of prior library experience One example of
81 prior interactions with a librari an caused confusion about the types of services provided and the ways in which to best utilize a librarian: A HSC librarian presents to the pediatric residents each year in order to prepare them to lead J ournal Club A statement was made that has stuck with me where the librarian involve a librarian in the early ( OnMed reflections, lines 74 92) This comment has particularly relevant implications for library services and the library liaison program Instructor Reflections While the focus of the case study was on how the participants view the embedded library experience, a defining feature of a course level embedded librarian project was close collaboration with course instructor perspective, a post course interview was conducted Them es from the instructor interview include the quality of participant annotated bibliography submissions, an emphasis on library instructional materials, limited interaction between the students and librarian, and finally the literature search capabilities o f the participants. Quality of participant annotated bibliography assignments I was pleasantly surprised with the majority of the products that were produced ( instructor int erview, lines 22 23) and I felt that if I was to compare the product the annotated bibliography products I think that they were more on par with what I would see from a first year doc student rather than what I see from masters students ( instructor int erview lines 23 25)
82 Library instructional materials Comments regarding the instructional materials describe the quality of the learning I think the materials that you developed were not just instructionally sound from a librarianship perspective, but also from an instructional design persp ( instructor interview, lines 14 16) When the i nterviewer/ librarian researcher noted that the log data indicated usage of the instructional materials, the instructor supported that instructor interview, line 353). Limited interaction between students and librarian Another theme to emerge from the post course interview concerned interactions between the embedded librarian researcher and t he course participants Specific comments that I was assuming that many of the students also had issues of interface [with librarians] and that students were not interfacing and that at a minimum they would begin to establish r elationships [with the librarian] on their instructor interview, lines 19 interaction of the course, because of how much they interacted with the embedded co instructor interview, lines 343 345) The theme of limited interaction is further analyzed in the discussion section Literature search capabilities of the participants In a d iscussion of faculty literature search patterns, the course instructor commented: I should be interfacing with the academic librarians myself and probably as guilty as the next p erson of underutilizing and being over confident in my abil ities to appropriately search ( instructor interview, lines 7 10).
83 Many of the themes uncovered in analysis of the interview data are related to the influence of the embedded librarian on the cours e participants, but an important consideration in an embedded librarian project is the collaboration between the course instructor and librarian The librarian researcher took the opportunity when interviewing the instructor to elicit feedback regarding no t only the collaboration during this embedded librarian project, but also future collaborations between teaching faculty and academic librarians. Further discussion of this theme will be used to triangulate the case study data in the next section. Summary of Findings While there was only one interaction between the embedded librarian and a course participant, log data indicates that the instructional materials were utilized Paired t test analysis of pre and post implementation assessment s indicate d a sign ificant difference between the pre test and post test means. When participant annotated bibliographies were analyzed, both subject and objective rubric scores were high. An analysis of participant reflections revealed several major themes, including: the nature of the annotated bibliography assignment and critical analysis required in the annotations, use of the library instructional materials, the literature search process in general and specifically the search terms and strategies used within the process and the process of finding useful and relevant articles Themes from the instructor interview include an emphasis on the quality of participant annotated bibliographies, the use of library instructional materials, limited interaction between the students and librarian, and finally the literature search capabilities of the participants L og data indicates that the instructional materials we re utilized Paired t test analysis of
84 pre and post assessment indicates a significant difference between the pre test and post test means.
85 Table 4 1 Emb edded library instructional material access Resource Name Times Accessed VPN Installation Video 2 Library Help 2 OnMed LibGuide 7 Refworks Handout 6 Refworks Elluminate 0 7 9 JCR Video 4 ERIC Demonstration 11 MeSH Demonstration 7 Search Support 6
86 Table 4 2 Paired t test results for the self efficacy assessment Test: N Mean SEM Median Mode Std. deviation Variance Range Pre test 14 3.058 .2464 2.875 2.0 .9221 .850 3.17 Post test 14 4.232 .1242 4.375 4.5 .4647 .216 1.75 Key: 1= Not confident, 2= Somewhat Confident; 3=Neutral, 4= Confident; 5=Very C onfident, N=number of questions
87 Table 4 3 Descriptive statistics for each participant Participant Test : N Mean Mean SEM Median Mode Std. dev Variance Range 1 Pr e 14 2.214 .714 .3176 2.000 1.0 1.1883 1.412 3.00 Post 14 2.928 .2864 2.500 2.0 1.0716 1.148 3.00 2 Pre 14 3.5714 1.143 .2716 3.500 3.0 1.0165 1.033 3.00 Post 14 4.7143 .1252 5.000 5.0 .46881 .220 1.00 3 Pre 14 2.2143 2.643 .3805 2.000 2.0 1.4238 2.027 4.00 Post 14 4.8571 .0970 5.00 5.0 .36314 .132 1.00 4 Pre 14 3.357 .859 .3722 4.00 4.0 1.39286 1.940 4.00 Post 14 4.4286 .2020 5.00 5.0 .75593 .571 2.00 Key: 1= Not C onfident, 2= Somewhat Confident; 3=Neutral, 5= Confident; 5=Very C onfident, N=number of questions
88 Table 4 4 Citation analysis summary Statistics Objective Rubric Subjective Rubric N 6 6 Mean 2.835 18.00 Std. Error of the Mean .0224 .5164 Median 2.850 17.50 Mode 2.85 17.00 Std. Deviation .0550 1.265 Variance .003 1.600 Range 15 3.0 Legend Out of 3.0 points Out of 20 points
89 Table 4 5 Descriptive statistics of citation analysis Bibliography ID Objective Rubric Score Subjective Rubric Score A 2.75 17 B 2.79 18 C 2.9 20 D 2.85 17 E 2.87 19 F 2.85 17 Mean 2.835 18
90 Table 4 6. Reflection themes with examples Theme Examples Annotated Bibliographies and Critical Analysis I approached the annotations as though I was going to use them to write a manuscript which helped. I was really not getting it until I realized that it is essentially a summary of what I grasp from a research paper when I am reading it to quote in a manuscript. ional papers that support or refute it. I realized these were the steps that I had been missing that forced me to slow down and review the articles in a way to know what direction to take my project. It also helped to see what design and statistical analysis methods were used how I should further evaluate the pre and post survey information that I have collected Library Instructional Materials The videos from the librarian were helpful to understand the annotation concept and how it is used. tips I picked up from the instructional tutorial on annotated bibliographies what I used and learned from the tutorials I certainly utilized some of I did use the brief educational segments by the librarian Search terms and strategies I still have an elementary understanding of how to decide on terms and combine them in such a way as to produce a useful list of articles I still struggled a bit during the search my confidence in my ab ility to conduct sound searches Finding out about the MeSH search terms did help. finding the correct search terms
91 Table 4 6 Continued Theme Examples Finding useful and relevant articles challenges I had with this assignment were finding publications that were relevant but not repetitive. finding the correct search terms to get the arti cles that answered my question. literature on it has literally exploded so it is hard to sort through what is relevant and what is not.
92 CHAPTER 5 DISCUSSION, IMPLICATIONS AND CONCLUSIONS Introduc tory Remarks As the occurrences of face to face and online course level embedded librarians increase ( Bozeman & Owens 2008; Edwards, Kumar & Ochoa 2010; Dewey 2004; Dugan 2008; Freiburger & Kramer 2009; Hall 2008; Lillard et al. 2009; Kesselman & Watstein 2009; Matthew & Schroder 2006; Rudin 2008; York & Vance 2009 ), the importance of describing and evaluating instances in which embedded librarians are active in online courses (often taking a leadership and/or instructor role) is paramount to their successful co ntinuation (Bowler & Street, 2008; Edwards, Kumar, & Ochoa 2009; Lillard et al. 2009 ) The purpose of this project was to describe a particular case of an online embedded librarian and evaluate the manner in wh ich the librarian influenced course participan ts. Specific contextual features of this case include d the status of the students as full time health care professionals in addition to their status as graduate educational technology students. The librarian researcher used best practices from the literatu re and instructional design principles to develop an embedded librarian program customized to the needs of the OnMed students A mixed methods case study was designed and implemented to describe and Data collection occurred during the spring 2 term (March 6 th April 22 nd ). Data collected includes Moodle access data regarding the number of times resources were viewed, pre/ post test s to measure information literacy self efficacy and library skill performance, a citation analysis of from all participants, and an interview with the course instructor Quantitative data
93 analysis indicated that the instructional participants viewed the instructional content and results of the paired t test showed a significant difference between pre and post test scores Emergent themes from the qualitative data focus ed on use of the library a concept and process. This chapter discuss es triangulation of the variety of data, discuss es the major findings, transition s to a discussi on of the findings in relation to the literature, describe s the threats to validity, and finally present s the implications, suggestions for future research, and conclusions. Triangulation In a mixed methods case study th e diversity of data (both sources an d types) require that the data be triangulated to construct and support working theories concerning the libr Of the four types of triangulation discussed by Patton (2002), triangulation of data sources was conducted in this st udy because all the data is aimed at corroborating the impact of the embedded librarian Primary data sources include Moodl e access log data, pre and post test scores, citation analysis data, and reflective data Additionally, the instructor interview and librarian researcher data triangulation and served as check points to inform the findings. Moodle l og data supports the reflective data concerning library instructional materials and commented specifically on their use of the library instructional videos : I did use the brief educational segments by the librarian about ERIC, etc (OnMed reflections, lines 35 36) which is demonstrated by the number of times the materials were accessed. As an
94 additional data point, the course in structor perspective regarding the instructional content is consistent with both the log data and the narrative reflections Additionally, the instructor reflected upon not only the use of the instructional materials, but their quality and the idea that they were instructionally sound: I think the materials that you developed were not just instructionally sound from a librarian ship perspective, but also from an ins 17). It is evident from analyzing both data sources ( M oodle log data and participant reflections) that the instructional content was utilized and this obs ervation was supported by the course instructor. By analyzing the pre and post test results it was determined that prior to the embedded librarian experience information literacy self efficacy scores were comparatively low contrasted to the post test sc ores (the pre test scores ranged from 2.21 to 3.57 where post test score ranges from 2.9 to 4.7), which reflects moderate confidence in information literacy skills. On average, the difference between pre test and post te st scores was 1.17. See T able 5 1 fo r a summary of the scores. This quantitative data outcome is supported by the participant reflections as several commented specifically on low searching confidence and/or anxiety with literature searching: ed reflections, lines 110 111). Post test results that demonstrate increased confidence is supported by the findings from the citation analysis, which indicate that participants used high quality peer reviewed sources. An additional theme from the data ana lysis concerns the literature search process, specifically, difficulties in locating relevant articles Several participants commented that :
95 medical records relevant to (OnMed reflections, lines 49 50) and this was observed in the subjective citation analysis results as some citation lists received 3 out of 4 points for article relevancy. Reflections comment on a lack of interaction with the embedded libr arian which is a theme supported by observations from the librarian researcher Learners were not required to interact with the librarian in the discussion forum, but they were strongly encouraged to view the instructional materials and comp lete the pre and post implementation assessments and reflections. A defining feature of a course level et al. 2009; York & Vance 2009) and a comment from one of the participants suggest ed that the librarian researcher was not felt within this course but I honestly did not (OnMed reflections, lines 74 75). However, the librarian researcher s describe several attempts to engage students and participate in the course: Thursday March 10th: mention of information makes me happy I think it fits in well with the information literacy and health care accreditations part of the embedded librarian presentation I just gave the needs of this course to highlight the links between accreditation standards and information literacy (field notes, lines 15 20) No one is commenting on my announcements in the discussion forum I I posted the presentation to the discussion forum. ( f ield notes lines 35 36) Specific examples of attempts to interact with and engage students include creating and e.g. a five slide video clip illustrating the connections between information literacy and the
96 accreditation documents from the health science colleges), post ing announcements and reminders for relevant presentations, and posting links to interesting sites to support the objectives of the course. Additionally there was a prominent Meebo chat widget in the course to facilitate instant messaging as a medium for c ontacting the librarian. The instructor commented that the lack of active discussion issue was a factor affecting the entire course, not just the library content: This course was very frustrating because it only consisted of seven students So from an onl ine instructional perspective it was a bear. I would have rather lumped these students in with the masters students or actually probably with the EdD students and taught a larger cohort Then the students would have fed off of each other to a larger degree there would have more opportunities for organic discussion. When there are seven students its literally impossible to get any sort of discussion dialogue going in an asynchronous context ( instructor interview, lines 85 91) From the interview it appears as if the course instructor noticed the librarian researcher interaction was due (at least in part) to the small class size: ourse, because low enrollment I think this experiment was heavily impacted by low enrollment numbers. A nd even if we were even to focus on these seven individuals if you added another eleven into the mix you think you would have a very different course dynamic. ( instructor interview, lines 343 347) In addition to commenting on the lack of felt presence, the participant also made suggestions for how the librarian can increase student interactions certainly highlight the resources (the tutorials) that will be used throughout the course more I think she can also elaborate on how sh e could M ed reflections, lines 81 83).
97 Discussion of Major Findings Research Question: H ow does the presence of an online embedded librarian influence graduate students experience in an online educational technology research class? Anal ysis of the quantitative measures of information literacy self efficacy and qualitative reflections of the experience lead to the conclusion that the embedded librarian had a p ositive impact upon OnMed students enrolled in the EDG 6931 course. The pre and post test scores for information literacy self efficacy illustrated a significant increase in self efficacy. From the reflective comments regarding usage of the instructional content one can conclude that viewing the librarian created instructional conte nt at least temporarily improved information literacy self efficacy Scores for the citation analysis (both objective and subjective measures) were consistently high, demonstrating that the participants used high quality sources in their annotated bibliog raphy; however it is not possible to correlate the high quality bibliographies with the embedded librarian or any other single factor The course instructor also noted the high quality of student submissions and suggested that ced academic health care professionals is one possible explanation (instructor interview, lines 23 25; 353 358). One of the citation analysis criteria is related to the type of resource cited, with peer reviewed articles receiving the highest point value The library instructional content directly addressed this as one of the videos demonstrated how to use a tool to determine the peer review status of a publication Student reflections commented on the usefulness of that specific video, which provided furt her empirical evidence to support the usefulness of the embedded librarian
98 Librarian p resence Findings suggest ed within the course beyond the existence of the customized library instruction al content necessarily feel lines 74 75). However, the learner goes on to comment that although the pr esence of the librarian was not felt (perhaps in terms of learner librarian interactions), the Reflecting back on what I used and learned from the tutorials and thinkin g about the latter part of the previous sentence I recognize it is not a fair statement The embedded librarian provided ( OnMed reflections, lines 75 78 ) This statement corroborates not only the fact that the instructional materials were used, but also that the learner content interactions were a reflection of librarian presence whether or not participants realized them as such. Narrative comments from the course instructor and librarian researcher support the findings rega rding lack of interaction with the librarian. Moore describes three types of interaction: learner learner interactions, learner content interactions, and learner instructor interactions (Moore, 1986) Learner learner interactions can be described as ; learner content interactions occur between the learner and the content or subject being studied ; and finally learner instructor interactions describe interactions between the learner and the instructor, content creator, or subject expert. Wh ile learners did not interact with the librarian synchronously or asynchronously through the discussion boards, chat, or email, the learners did interact with the librarian
99 created instructional content. Analysis of the assessment, citation analy sis, and reflective data demonstrates the efficacy of the learner content interactions. Prolonged sustained contact with a librarian who is actively integrated into a course is one of the features of an embedded librarian One of the benefits of this level of contact is to allow the learners to become familiar with the librarian. Increasing learner familiarity with the librarian is an attempt to decrease anxiety about approaching a librarian ( Collins & Veal, 2004; Jiao, Onwuegbuzie, & Lichtenstein, 1996; Me llon, 1986, 1988; Kuhlthau, 1988, 1991; Onwuegbuzie, 1997) By decreasing the well documented uncertainty about asking a librarian for help an embedded librarian hopes to increase likelihood of being approached by learner s with an information or access ne ed With only one direct learner librarian interaction during the course it is difficult to may have had about approaching the librarian While the learners indicated that they did not interact with the librarian or experience within the course the librarian researcher was astutely aware of her presence within the course and found being i ntegrated into the course beneficial for a variety of reasons Librarian benefits of being embedded include d the ability to create instructional materials customized to the course content, participate in course discussions and create material quickly as needed in response to discussion topics and organically as disc ussions evolve d and see the participants research topics evolve from a list of potential ideas to the final topic In having access to the course the librarian was able to better understand the course and the ways in which library and information litera cy content could contribute.
100 Additional f indings While the research question focuse d on the embedded librarian experience from librarian researcher t itself to additional findings related only tangentially to participant experience. One of the significant findings relate d to faculty collaboration for course integrated and embedded instruction The course instructor support ed collaborations and ma de rec ommendations for future collaborations: specific courses seven people, but what about for large courses, for pharmacy for d entistry, medicine and not just online courses, but face to face courses If anything you could collaborate with the instructor a nd strengthen their materials (interview, lines 239 243). Related to faculty library collaboration is the idea of creating cust omized library instructional content. Content in the OnMed embedded librarian case study was created specifically for academic health faculty researching clinical as well as educational topics. Collaboration with a faculty member facilitates the close inte gration of library content to the needs of a course (Bozeman & Owens, 2008; Chesnut et al. ., 2009; Dugan, 2008; Hall, 2008; Matthew & Schroeder, 2006; Stewart, 2007; Tennant & Miyamoto, 2002; York & Vance, 2009) Without the collaboration between librarian and course instructor, access to the course syllabus and content may not have occurred as early in the design process, if at all. Collaboration with the course instructor and a detailed review of the course content allowed the librarian to capitalize on knowledge of both the subject matter and the course content as well as information science centric domain knowledge. An example of t he benefit of librarian/instructor collaboration is demonstrated in the post course
101 interview A discussion of levels of evidence and the hierarchical nature of publications turned into a discussion of primary versus secondary sources and lead to the instr for students. Discussion S ummary One of the defining features of this case was the close integration of the library instructional materials to the course goals and objectives. Data indicate d that the library instructional content was a valuable addition to the course and that learner content interactions occurred. The librarian researcher was well positioned to create such customized content because of her background in both educational techn ology (and familiarity with the course content) and medical education /medical literature searching practices While not all subject or liaison librarians will have as much familiarity with the course content as was demonstrated in this case, customization of online library instructional content is still possible through the use of an instructional design model, thorough needs assessment, and close collaboration with faculty. Interactions in this case were primarily categorized as learner content interaction s because the students used the instructional content developed by the librarian, which could lead to the conclusion that the most useful feature of course integrated embedded librarians is the instructional content While that is an accurate statement in this case, other research suggests that librarian presence and learner librarian interactions are defining element s of embedded librarianship (Bowler & Street, 2008) Their research was conducted with undergraduate level humanities courses, a context with learner characteristics and needs differing from the OnMed case The difference in findings between other embedded librarian studies and the OnMed evaluation study can be
102 attributed to contexts, including the small class size, learner characteristics and p rior experience with information literacy and library skills. Despite the lack of librarian presence felt in this implementation of an online embedded librarian, the positive impact on self efficacy demonstrates that it is still a valuable experience for t he students, instructor and librarian The fact that interaction can possibly be attributed to the unique features of the context and may not necessarily generaliz e to all online embedded librarian scenarios Findings and the Literature Embedded L ibrarianship Primary findings relate to the positive impact the embedded librarian had upon participant information literacy self of the embedded librarian (specifically relating to interactions with the librarian), and faculty librarian collaborations. Presence in the context of an embedded librarian refers to the prolonged availability of the librarian within the course and interac tions between the librarian and learners (Dugan, 2008; Hall, 2008; York & Vance, 2009). Interaction in the general sense of an online course refers to the various types of interactions in a distance or online course described by Michael Moore. ( Moore 1986 ) Each of these specific themes will be discussed in relation to the literature after a discussion of the broad theme of the online embedded librarianship literature and the findings of this case study review of librarians embe dded into online courses focused on linking the librarian with the CMS (course management system), careful selection of courses in which to embed the librarian,
103 active librarian participation in the course, and marketing the servi ce to other faculty members Many of these recommendations were integrated into the design of the embedded librarian implementation developed for the OnMed course, t hus providing ure analysis recommendations. In order to investigate the efficacy of various levels of course level embedding Bowler and Street (2008) designed several experimental face to face embedded librarian instances with differing levels of integration The resea rchers found that a higher level of librarian integration with more student interaction with the librarian resulted in a significant improvement in student scores on a standardized information literacy rubric The Bowler and Street findings indicate that a high level of integration and presence is preferred; however the research was conducted in a traditional face to face classroom and the extendibility of these findings to online courses has yet to be investigated Despite this difference in setting these results do not support the findings in the OnMed case, which indicate that presence is not as influential as customized instruction The Lillard et al (2009) embedded librarianship implementation served a dual purpose : to prepare library and infor mation graduate students (nascent librarians) to serve as online embedded librarians and to investigate embedded librarianship in online graduate nursing courses. The authors found that the experience was generally positive for the student embedded librari ans, graduate nursing students, and faculty Key regarding course selection and collaboration.
104 Feedback from the nursing students enrolled in courses with embedded librarians indicates that the usefulness of the embedded librarian was related to the course topic and where in the curriculum the course occurred S everal students commented that the instruction and embedded librarian experience would have been more beneficial if it had occurred earlier in their program (Lillard et al 2000) These findings could inform findings from the OnMed embedded librarian case While the EDG 6931 course occurred relatively early in the OnMed curriculum (in the second semester) and the Lillard et al. findings suggest ed experience d health care professionals could have influence d their use of the embedded librarian Their prior experience provide d them with more opportunities for library ins truction and the practice of evidence based medicine provide d opportunities to apply information literacy skills emphasized by the embedded librarian was also noted by the co urse instructor when he commented that the level of work they produced was similar to that of first semester doctoral students rather than beginning Masters students (instructor interview, 23 25; 353 358). Data from the library student reflections in the York and Vance (2009) st udy indicate d their experience as embedded librarians was directly related to the amount of communication between themselves and the faculty instructor and the freedom of communication allowed between the librarian and nursing students enrolled in the cour se Clearly in th e York and Vance instance s of embedded librarianship the were influenced by interactions with the students The issue of librarian presence within
105 th e course and interaction with the students was a key finding for the OnMed study and warrant s further investigation. Summary of the F inding s in the C ontext of the L iterature Although the description and assessment of these projects was not as thorough as the OnMed case study online embedded librarian is a valuable experience for students and can foster further collaboration between librarian and faculty. The OnMed embedded librarian project provided outcomes that mirror findings from the literature, but it also extend ed them as it is a more multifaceted, in depth analysis of embedded librarianship in a particularly unique setting of learners w ho have dual roles as graduate student and academ ic health faculty By providing a rich case study of an online embedded librarian implementation, this project can serve as a model for future investigations of other online embedded librarian contexts. Threats to V alidity While the study was designed with validity in mind by implementing methodological measures including the use of validated instruments, member checking, and an audit trail, it is impossible to protect against all threats to validity Factors such as the size and selection of the case, cour se content and specifics of the population all contribute threats to validity While there is no minimum number of participants necessary for a case study (Yin 2009), it is clear that the number of participants in this case negative ly impacted the course dynamics and interaction, which could have affected the results Additionally, the specialized population of heath care professionals influenced results and it is not clear to what extent the results would generalize to another context. One possible solut ion to these issues would be to conduct a multi case
106 study and use another larger, more typical case in addition to the OnMed case The use of other study designs including an experimental or quasi experimental approach would allow for more control of othe r potential confounding variables. Implications Overview Results and implications in this study inform practice in several areas and are Evaluation Model (Kirkpatrick & Kirkpatrick, 2006). The Kirkpatrick Model was devel training programs and includes four evaluation levels (Kirkpatrick, 1959a; 1959b; 1960a; ; l evel two assesses learn ned as a result of the training; level three ; and level four assesses outcomes and results based on the training. The four levels can be thought of in a hi erarchical manner with level one at the bottom and level four on top; F igure 5 project particularly because it is a commonly used model (Alliger & Janak,1989; Cascio,1987) tha t may resonate with stakeholders interested in the results of this study. Implications are categorized broadly into three areas: implications for the practice of librarianship, administrative and policy implications, and educational and curricular implicat ions. Implications for Professional Librarianship Findings from this and other studies support the efficacy of embedding librarians into online courses (Bowler & Street, 2008; Edwards, Kumar & Ochoa, 2010; Lillard et al. 2009; York & Vance, 2009; Shumaker & Talley, 2009) However, the findings
107 highlight the necessity of careful consideration of the course and audience prior to initiating an embedded librarian and during the designing of the embedded librarian content. Prior to planning and implemen ting an online embedded library program, librarians must carefully consider the curriculum to select appropriate courses, collaborate closely with instructional faculty, and employ an instructional design plan that includes a thorough needs assessment. It is imperative that the design of the embedded library content, including instructional materials and interactions be tied to the needs of the learners and the course. For example, an entry level online undergraduate course that was designed to be highly i nteractive may require more experienced learners. In order to maximize student impact while balancing the time investment necessary for embedded librarian projects, librarians should work closely with the course instructors to assess the needs of the course and learner characteristics prior to implementing an embedded librarian program that features interaction with the librarian In addition to careful course selection and ins tructional design, librarians embedding themselves in online courses should familiarize themselves with strategies used to increase learner interactions and best practices for establishing an online teaching presence. Implementing the recommendations above will help to make the embedded librarian design and development process more efficient and effective ; however there is still an intensive time comm itment required. There is the potential for librarians to be overwhelmed with meeting the needs of students enrolled in the course with an
108 embedded librarian and balancing other responsibilities. For example, in one instance of an embedded librarian in the HSC library the librarian is integrated into the foundational course for the online professional program in the college to which she liaises. The librarian interacts with students in the foundational course but is also responsible for supporting the needs of the entire program (approximately 600 800 students) and the college at large, which includes several o ther large programs in addition to the faculty members. Given the large number of potential users this liaison librarian supports, a high level of involvement in the course in which she is embedded can be overwhelming and lead to increased stress and poten tially decreased effectiveness The conditions in which a librarian can successfully embed in a course and the degree of integration must be determined by the librarian, library administration, and the faculty and administration in the college and/or progr am in which the embedding will occur. In addition to these recommendations for practicing librarians, the study has implications for the education of practicing and future and librarians. Embedded librarianship as demonstrated in this study (librarians embedded into courses or the curriculum) requires a set of skills that the librarian researcher but not necessarily all academic librarians, possesses including knowledge of instructional de sign models, learning theories, and educational technology. Steven Bell and John Shank describe Bell & Shank, 2004). A blended of librarianship learning Academic libraries are acknowledging this need by a ctively recruiting
109 librarians with these skills An analysis of position advertisements demonstrates that fewer than twenty four design/er and of that total there were only 10 announcements t hat were not reposting earlier advertised positions (Shank, 2006) The Shank study analyzes the knowledge, skills, and abilities described in the position descriptions to define key characteristics for librarians whose primary responsibilities focus on edu cational technology and instructional design In seeking to hire instructional technology librarians and reference and instructional librarians with some instructional design skill, academic libraries are responding to the shifting nature of academic libra rianship. Clearly ac ademic libraries recognize the importance of instructional skills, but when, how and where do librarians acquire these skills? Options for providing these instructional skills fall into two broad categories; educating future librarians and educating practicing librarians. A broad, far reaching solution involves changes to the graduate school curricula for library and information science programs. An examination of the course listings of the graduate school pro that, with the exception of media specialist certification requirements, a minority of the courses, As academic librarians are increasingly engaged in designing and delivering instruction, formal preparation in learning theories, instructional design and pedagogy/andragogy would more fully prepare librarians for their instructional roles. Whil e this is an optimal strategy for library education, curricular changes require large scale institutional adjustments and take time to plan and implement. Therefore it
110 is important to continue to provide practicing librarians with opportunities to develop and cultivate instructional knowledge and skills. This continuing education can occur at various levels including within the library, on campus, and through regional, national, and international conferences and professional organizations. Libraries sponso r and facilitate professional development on a variety of topics, which should include quality offerings designed to provide both theoretical and practical knowledge and skills relevant to instruction. Another source of professional development and suppor t that should not be overlooked includes campus based resources, instructional centers and institutes, provides faculty with training on a variety of inst ructional design and technology topics. In addition to offering training, CITT staff also provides instructional desig n and development support to produce instructional content Academic librarians should make use of these local sources of continuing educa tion and professional development, which are often provided at low or no cost to the individual, to enhance their instruction al knowledge and skills Beyond these readily available economically feasible continuing education options there are additional sources at the national and international level. Professional organizations, both library and non library organizations, are an important source of training opportunities. An example of an established, high profile program is the Asso mmersion Program. According to their website, the goal of the immersion program is to prov ide librarians with a training experience designed to supplement their teaching, instructional
111 progra m planning reflective teaching and assessment skills specifically relating to information literacy concepts : ACRL's Immersion Program provides instruction librarians with the opportunity to work intensively on all aspects of information literacy. Wheth er your institution is just beginning to think about implementing an information literacy component or whether you have a program well under way, the Immersion Program will provide your instruction librarian with the intellectual tools and practical techni ques to help your institution build or enhance its instruction program (ACRL, 2011). While anecdotal evidence from practicing librarians support the efficacy of these programs, registration fees range from $1,875 $1,975 depending upon membership status, and the high cost can be prohibitive. consider other avenues of continuing education including sessions offered at professional conferences. Conferences offer formal continuing education opportunities in the form of workshops and informal development from conference presentations. In many academic libraries attendance and participation in professional conferences is required or highly recommended and with varying amounts of financial support allotted. By coupling conference attendance with formal and informal continuing education in a conference setting, librarians maximize the time and financial investment required for those activities. Another option for academic librarians serious about gaining instructional des ign knowledge and skill involves taking credit bearing graduate level course work beyond that required for the library science degree. This can be done through a college of education for either a post graduate certificate or for an additional graduate degr ee. College s of education, including the University Of Florida College Of online targeting practitioners.
112 Findings from this case study indicate that high quality embedd ed librarian programs positively influence student outcomes, and in order to design and develop successful implementations, academic librarians need to be able to acquire and develop their instructional knowledge and skills, formally and informally. Policy and Administrative Implications This study evaluated a single use case of an embedded librarian instance While results demonstrate benefit to both the students and the instructor, the benefit was limited to a very specific group of users and the overall and intellectual investment is undetermined In order to increase the institutional impact of course integrated embedded librarians similar projects would need to be implemented in other colleges, departments, and programs. In fact wide scale campus implementation of course level embedded librarians was a recommendation resulting from the interview with the course instructor ( instructor interview, lines 236 237). While theoretically it may be beneficial to use this model to embed librarians in courses campus wide, the model may not support expansion of that level As mentioned previously in relation to this project, and in the embedded librarian literature (Bozeman & Owens 2008; Edwards, Kumar & Ochoa 2010; Dewey 2004; Duga n 2008; Freiburger & Kramer 2009; Hall 2008; Lillard et al. 2009; Kesselman & Watstein 2009; Matthew & Schroder 2006; Rudin 2008; York & Vance 2009 ; Shumaker & Talley, 2009 ; Tennant & Miyamoto, 2002 ) embedding a librarian in courses requires a significant time investment from the librarian. Each stage of the instructional design process beginning with the needs assessment to collaboration with the faculty member and the development of instructional materials is time and labor intensive, but crucial to th e success of the project In addition to investing time in developing customized
113 integrated instructional materials a truly embedded librarian will devote time to participating in the course in which they are embedded, including offering support and inter acting with the learners and/or instructor While librarians may be willing to devote the intensive amount of time to function in the library, doing so may limit thei r ability to perform additional vital functions that support the library and their career, especially in institutions where librarians are tenure track faculty There are several potential solutions to this dilemma, designed to help the individual embedded librarian and the library achieve some return on their time and invest in resources necessary to produce a quality, successful embedded librarian product. One potential solution to this problem is to create instructional materials that are reusable. In th is case study the instructional content developed by the librarian researcher was reused for a similar audience of medical educators and modified and repurposed for other audiences, thus extending the usability of instructional content and expanding the b enefit beyond the OnMed embedded librarian project However, the fact that the instructional materials were ostensibly developed for a specific group can possibly curtail the reusability of the materials so that they are only useful to reuse with a group s imilar to the one for which they were designed It may also be feasible to design and develop templates with details and examples based upon a pre determined framework, which librarians can use to facilitate and hasten the development of instructional mate rials.
114 Another solution to extend the benefit of the time investment is to turn the service focused instructional project into a research project by participating in the scholarship of teaching and learning (SoTL) In conducting SoTL research related to t he embedded librarian duties, librarians and their institutions can achieve some return on their time investment by disseminating and publicizing library research and supporting retention through the tenure and promotion process. If an institution or libra ry decides to implement course level embedded librarian programs regardless of the time involved with the endeavor, there may be other less tangible ways to recoup some of the lost investment Evaluating embedded librarian programs in these terms can be se en as a discussion of return on investment, is an example of a Kirkpatrick level four evaluation (evaluating results) (Kirkpatrick & Kirkpatrick, 2006) and has direct policy and administrative implications. Intangible benefits include supporting accreditat ion (at both the regional level i.e. SACS and the college level) and supporting patient care through improved patient outcomes The following paragraphs discuss both the issue of accreditation competencies and patient care outcomes as they relate to course level embedded librarianship in a health science center. The context of this project was focused on providing curricular library support in an academic health science center (known as the academic health center AHC) ; therefore this discussion of accredi tation standards focuses on the accreditation standards of four of the six colleges comprising the AHC, including the colleges of Dentistry, Medicine, Nursing and Pharmacy The fifth and sixth unmentioned colleges are the College of Health and Health Professions (CoPHHP) and the College of
115 Veterinary Medicine (CoVM) The CoPHHP will not be discussed because it contains several distinct programs all with varying accreditatio n requirements T he CoVM will not be discussed because the term patient care is being limited to human patient care in this study. Accreditation standards for each of the four colleges all reference the skills and abilities described by the Association of College and Research Libraries (ACRL) in their definition of Information Literacy See F igure 5 2 for a visual alignment of the ACRL Information Literacy definition with the similar terms and phrases used in accreditation. A course level embedded librarian supports students acquisition of these skills by creating targeted instructional content designed to foster information literacy and encouraging and participating in critical thinking discourse, which facilitates application and synthesis level cognitive tasks (Bloom, 1956). There is a link between the skills described in the accreditation standards and information literacy; however the phrase is not used directly in any of the standards, which can create difficulties for libraries attempting to connect their services to accreditation standards for the purposes of marketing, support and funding. Library administration must find ways to highlight the connections between information literacy and accreditation, especially to significan t stakeholders within the institution. While the patient care outcomes associated with other manifestations of embedded librarians, particularly clinical librarians are clear and well reported in the literature (Brettle et al. 2010; Swinglehurst, Pierce & Fuller, 2001; Wagner & Byrd, 2004; Weightman & Williamson, 2005; Winning & Beverley, 2003), the connection between patient care outcomes and course level embedded librarians are less clear and
116 direct but none the less appreciable Specifically in the con text of the OnMed cohort in addition to being learners and graduate students themselves, the participants are also health care professionals all of whom are directly engaged in teaching students (undergraduates and graduates) and some of whom are providin g clinical services. Clinical teaching, especially at the graduate level (during internships and residencies) directly relate to patient care outcomes ( Chaudhuri et al. 2006; C lark et al. 2010; Lin, Shabbir & So, 2010; Oo, Grayson & Ra shid, 2004; Thors e t al. 2010 ) The embedded searching abilities, which in turn supports their education and research processes. This can indirectly influence patient care through the i mprovement of their own patient care and the care of the patients for whom their residents and students will be responsible. Curricular and Educational Implications Data supporting the practice of embedding librarians in courses primarily assesses learner satisfaction and reaction (Kirkpatrick level 1) (Kirkpatrick & Kirkpatrick, 2006) but does not provide sufficient data regarding learning, behavior, or results which is especially important when assessing new library services (Barton, German & Joint 2004; McKee 2010). This case study extends the current research and as such is one of the first investigations into measuring learning outcomes (Edwards, Kumar & Ochoa 2010) and detailed participant reflections in an online embedded librarian setting While ass useful feedback, it is important to consider more in depth evaluations and studies to take into account learning outcomes and student experiences with the embedded librarian (Barton, Ge rman & Joint, 2004; Bowler & Street, 2008; McKee 2010; Weaver 2010).
117 As a result of the focus on measuring learning outcomes (Kirkpatrick level 2) (Kirkpatrick & Kirkpatri ck, 2006), the embedded librarian evaluation has potential curricular implications, especially for the College of Medicine, which is in the process of restructuring its undergraduate medical curriculum. The current medical curriculum includes a course on e vidence based medicine (EBM) and the new curriculum may include an evidence based medicine thread Librarians support both the practice of evidence based medicine (Atlas, Smigielski, Wulff & Coleman, 2003; Kronefeld et al. ., 2007; McCarthy, 1996; McGowan et al. ., 2010; Mulvaney et al. ., 2008; Rader & Gagnon, 2000; Rose, 1998; Scherrer & Dorsch, 1999 Tod et al. ., 2007) and the teaching of evidence based medicine to undergraduate medical students and residents (Atlas, Smigielski, Wulff, & Coleman, 2003; Bexo n & Falzon, 2003; Bradley, Rana, Lypson, & Hamstra, 2010; Brown & Nelson, 2003; Dorsch, Jacobson & Schemer, 2003; Kem & Weiss, 2005; Koufogiannakis et al. ., 2005; McCarthy, 1996). An embedded librarian in the evidence based medicine curriculum can have pro found implications by improving evidence based medicine competencies, which may even translate into long term behavior changes (Kirkpatrick Level 3) and results (Kirkpatrick Level 4) (Kirkpatrick & Kirkpatrick, 2006) One of the primary goals of embedding in the medical curriculum is to influence includes the following basic components: (1) correctly asking a clinical question (2) searching the literature (3) using the resul ts to answer the question. Long term goals for embedding in the medical curriculum focus on improving patient care by improving
118 idence based medicine concepts, both theoretically and in practice. Information literacy and evidence based medicine are complex concepts that are not (and should not be) distinctly measured using a standard assessment, such as the United States Medical Li censing Exam (USMLE). Therefore, e valuating the long term outcomes of librarian involvement in the medical curricula requires a multifaceted approach designed to collect a variety of data quantitative and qualitative from a variety of sources A comprehe nsive evaluation plan should collect a myriad of evidence from the learners themselves, from the instructors (didactic and clinical) and from librarians who interact with the learners. Data from the learners includes artifact analysis of various written d ocuments including those included in eportfolios citation analysis of written documents (including the culminating assignment submitted in the second year evidence based medicine cours) semi structured interviews and focus groups Another source of data from the medical students can come from the documentation required in the patient log As medical students are logging their patient care exposures they can be queried as to whether or not they used the medical evidence in their decision making with a sub sequent description of the clinical question, search strategy, search results, and how the literature influenced the clinical decision College of Medicine instructors ( both le cture/didactic and clinical) are another significant source of evaluation data. The current curriculum includes an evidence based me dicine course in the second year and the new curriculum redesign includes various tracks, with the possibility of an evidence based medicine track. Feedback from
119 the EBM course in structor gathered throug h semi structured interviews can provide and the impact of the librarian embedded into the EBM course. Similarly clinical instructors, interns, residents and other clin ical professionals working with clerkship students can be interviewed to evaluate the clinical EBM skills of medical students. Librarians can provide a final source of evaluation information not to be overlooked. Data from the librarians includes contact s tatistics recording the incidences of medical student contact with librarians, type of contact (including whether or not it was an EBM related query), duration of interaction, and other descriptive information. Additional librarian data includes interviews focus groups and field note observations from librarians supportin g medical students. While librarians have a clear influence on the evidence based medicine skills. The current curriculum includes a research track and all indications support the notion t hat the new redeveloped curricula will as well. Librarians can contribute to the success of students in the research track by supporting their acquisition and refinement of literature searching and critical analysis abilities Directions for Future Resear ch Findings from this study, particularly regarding the information literacy self efficacy assessment indicate that embedded librarians are beneficial to students and improve their information literacy self efficacy and skills, primarily through learner c ontent interactions rather than learner instructor (or learner librarian) interactions. The findings may lead to the assumption that individuals are more interested in interacting with
120 library content rather than a librarian; however these findings may be specific to the learners in this case In cases similar to this featuring non traditional learners who may have more prior experience with library research and information literacy concepts it may be preferable to focus on collaboration with faculty and the creation of customized instructional materials rather than developing rich learner librarian interactions Further study is required to test the above assertion. The development and analysis of this case uncovered several new questions particularly r evident that there is a discrepancy between the experiences of the participants and the librarian researcher which leads to several questions including: What factors contributed to the lack of interaction? felt? What could the librarian have done differently to increase interactions? How could the librarian have changed participant perceptions regarding the What could the instructor have done differently? Additional studies further explore the presence of embedded librarians in online courses These studies are based in other contexts b oth in graduate and undergraduate settin gs and were designed to determine whether the preference for learner content rather than learner librarian interactions is true of other typ es of students in other courses These studies would hopefully make recommendations for strategies to increase lear ner instructor interactions in future iterations Online interaction is related to online presen ce and communities of inquiry. The general education literature has a rich body of research describing communities of
121 inquiry and online presence Garrison, And erson and Archer (2000; 2001; 2002) are leading researchers in the area of communities of inquiry (COI). Their heavily cited 2001 article establishes a COI framework consisting of social, cognitive and teaching presence (Garrison, Anderson & Archer 2000), which is followed up by further research (Garrison, Anderson & Archer 2001; Garrison 2003) and a 2008 review of the COI research (Garrison & Arbaugh 2008) As Garrison, Anderson and Archer describe the communication (interaction and discourse) to construct meaning; and finally teaching presence describes the design and facilitation of p rocess to enhance social and cognitive presence (Garrison, Anderson & Archer 2000) Findings regarding social and teaching presence in online courses could impact the roles of librarians embedded into online courses. The nature and extent of the impact is an area in need of further study. Research concerning communities of inquiry, interaction and presence in online courses potentially dependent upon what role (s) the embedded librarian plays within the course For instance, librarians who have more freedom and flexibility with their interactions with students enrolled in the course have more potential for interactions of all types Librarians who have an instructor like role wi thin the course will interact with students in a manner similar to the course instructor while a librarian with a less formal role may interact with students in a manner that more closely resembles student to student interactions Future research could foc
122 courses and determining how students interact with librarians and the ways in which librarians affect cognitive presence. In addition to researching librarian presence, future studies could use other methodol including experimental or quasi experimental designs with a control group to provide more rigorous empirical evidence of the efficacy of the embedded librarian. By collecting a variety of data from learners, instructors, and librarians it is possible to comprehensively evaluate the impact of embedding librarians into the curriculum. Recently the HSC Library implemented a new addition to the existing librarian liaison program. While librari ans liaise to colleges and departments, there has never been a librarian liaising specifically to medical students. To fill this gap the library decided to assign one librarian to liaison with a particular medical class. This approach will aid the evaluati on process as it will allow specific librarians and interventions to be assessed and the impact tracked through the four year undergraduate medical curriculum. Conclusions As is demonstrated by the professional literature (Bozeman & Owens 2008; Edwards, Kumar & Ochoa 2010; Dewey 2004; Dugan 2008; Freiburger & Kramer 2009; Hall 2008; Lillard et al. 2009; Kesselman & Watstein 2009; Matthew & Schroder 2006; Rudin 2008; Schumaker & Tally, 20 09 ; York & Vance 2009), librarians are bein g increasingly embedded or integrated into a variety of contexts including colleges, departments, research teams, and both online and face to face courses ; it is clear that embedded librarianship is the future of academic librarianship. Embedded librarians are being used to provide contextualized instruction in the increasing number
123 of online courses being offered by institutions of higher education (Allen & Seaman, 201 1 ) Evaluation is essential because this is a new service area. The embedded librarian im plementation described in this project built upon experiences with a similar pilot project and best practices from the literature including customized instructional content, multiple modes of optional interaction with the librarian, and extensive faculty s upport and collaboration However, this project extends the literature by using a experience with the embedded librarian Both quantitative and qualitative methods were employ ed including a pre and post assessment of information literacy self efficacy, citation analysis of student submissions, and participant reflections. A post course interview with the instructor and field notes from the librarian researcher further inform ed the findings Results indicate d an increase in self efficacy and high quality annotated bibliography submissions, which are primarily attributed to viewing the instructional content rather than interacting with the librarian This project pave d the way f or future embedded librarian initiatives at the University of Florida and further collaborations between librarians and faculty as well as additional research regarding experiences with an online embedded librarian and librarian presence.
124 Table 5 1 P re and post test scores for each participant Participant Test: Mean 1 Pre test 2.214 .714 Post test 2.928 2 Pre test 3.5714 1.143 Post test 4.7143 3 Pre test 2.2143 2.643 Post test 4.8571 4 Pre test 3.357 .859 Post test 4.4286 Mean Pre test 3.058 1.17 Post test 4.232 N= 14; Key: 1= Not C onfident, 2= Somewhat Confident; 3=Neutral, 5= Confident; 5=Very C onfident, N=number of questions
125 Figure 5
1 26 Figure 5 Four Health /Medical Accrediting Agencies
127 APPENDIX A PRE/POST TEST SELF EFFICACY AND LIBRARY SKILLS INSTRUMENT 1) I can identify the most appropriate keywords or phrases for the information needed when I search a topic Not Confident (1) Confident (2) Neutral (0) Confident (3) Very Confident (4) 2) I can identify alternate terminology, such as synonym and broader or narrower terms, for the information needed. Not Confident (1) Con fident (2) Neutral (0) Confident (3) Very Confident (4) 3) I can use a thesaurus in a database to select subject terms for searching. Not Confident (1) Confident (2) Neutral (0) Confident (3) Very Confident (4) 4) I can construct a search using Boolean op erators (e.g., AND, OR, NOT). Not Confident (1) Confident (2) Neutral (0) Confident (3) Very Confident (4) 5) I can use a particular search field (e.g., title, URL, author) when searching for specific information. Not Confident (1) Confident (2) Neutral (0) Confident (3) Very Confident (4) 6) I can construct a keyword search so that my search words are found near each other, within the same paragraph of a document. Not Confident (1) Confident (2) Neutral (0) Confid ent (3) Very Confident (4) 7) I can construct a search to retrieve documents containing an exact phrase. Not Confident (1) Confident (2) Neutral (0) Confident (3) Very Confident (4) 8) I can construct a complex search using more than one Boolean operator and grouping terms together using parentheses. Not Confident (1) Confident (2) Neutral (0) Confident (3) Very Confident (4) 9) I can use truncation symbols (e.g., *, $) to find variants of search words (e.g., teach, teacher, teaching) when searching in a database. Not Confident (1) Confident (2) Neutral (0) Confident (3) Very Confident (4) 10) When subject terms relevant to a topic are shown in a database, I can search for additional information using those subject terms.
128 Not Confident (1) Confident (2) N eutral (0) Confident (3) Very Confident (4) 11) I can narrow or broaden my search to retrieve the appropriate quantity of information. Not Confident (1) Confident (2) Neutral (0) Confident (3) Very Confident (4) 12) Please use library resources to complet e the following searches and enter your results: A) What years do we have electronic access to this journal: British Journal of Educational Technology B) Database search: Using the most appropriate library database please find an article on blended learning in undergraduate medical education written in the last 2 years Use the box below to enter the citation for the article and your exact search strategy. C) Using the library catalog find an electronic book on the topic of blended learning Use the box below t o copy and paste the citation information (title, author, publication date, publication place) 13) Please rate your confidence in completing the searches in question #12 Not Confident (1) Confident (2) Neutral (0) Confident (3) Very Confident (4)
129 APPENDIX B OBJECTIVE CITATION R UBRIC SCORING SCALE Resource Points > 3 yrs old > 10 years old Max Points Dissertations (Published & unpublished) 2 .3 .2 2.5 Theses/pracitcums/action based research 1 .3 .2 1.5 Periodicals (magazines, trade journals) 0 .3 .2 .5 Scholarly Periodicals 1.5 .3 .2 2.0 Journals +.3 .3 .2 Academic/Scholarly +.2 .3 .2 Peer reviewed +1 .3 .2. Books/book chapters (not scholarly) 0 .3 .2 1.5 Books/book chapters (not scholarly) 1 .3 .2 1.5 Books/book chapters (not scholarly) 1 .3 .2 1.5 Reports (gov. agencies, foundations, associations, universities,etc.) 1 .3 .2 1.5 Conference Papers and proceedings (published and unpublished) 1 0 0 1.0 Government laws/legal cases .5 0 0 .5 ERIC ED documents 0 0 0 0 Newspapers 0 0 0 0 Web sites 0 0 0 0 Miscellaneous 0 0 0 0 Documents that fit two or more categories were included in category with higher weight.
130 APPENDIX C SUBJECTIVE RUBRIC SC ORING SCALE ID Number: ______________ Rater:___________________ Criteria Level 1 Inadequate Level 2 Marginally Adequate Level 3 Adequate Level 4 Superior Breadth of Resources number of citations variety of resources cited Student used a limited number and/or variety of resources available on topic or did not show awareness of specialized sources Limited number and variety of sources cited Reasonable number and variety of sources used for topic Exhaustive search that utilizes a comprehensive number and full range of type s of sources available for topic Depth: understanding as demonstrated through the citing of historical, theoretical & background resources Depth of understanding underdeveloped by a lack of citations from historical, theoretical background resources Depth of understanding emerging as demonstrated through the citation of a li mited number of historical, theoretical, background resources Depth of understanding developed as demonstrated through the citation of a substantial number of historical, theoretical, background resources Depth of understanding exemplary as demonstrated th rough the exhaustive citation of historical, theoretical, background resources Depth: appropriateness (quality of resource) primary resources empirical research peer reviewed seminal/landmark studies Majority of resources superficial/weak Limited number of scholarly, peer reviewed, resources/too few empirical research was superficial Majority of resources were scholarly, peer reviewed and reasonable number of empirical research studies A rich representation of quality, peer reviewed empirical research/resources very scholarly Currency: Make no consideration to the availability of resources on the topic being researched Not current Majority of references older than 10 years from date of completion A disproportionate number of unnecessarily da ted resources (majority over 5 years) The majority of resources published 5 years or less from completion of dissertation Extremely current majority of references within 5 years of dissertation completion Relevancy to the topic Majority of sources do no t relate/pertain to the topic A disproportionate number of sources do not relate/pertain to the topic Sources generally support/pertain to the topic Sources directly on target and support/pertain to the topic OVERALL SCORE _____________________ /20
131 APPENDIX D PARTICIPANT REFLECTI ON QUESTIONS Students will be asked to respond anonymously to these reflection questions. 1) What role did the librarian (who offered instructional support and assistance) play in your process of completing the annotated bibli ography assignment (describing research questions, refining your question, searching for literature, and finally evaluating and synthesizing the literature)? 2) In what ways/How did your prior experiences in clinical research and literature searching influen ce your completion of the annotated bibliography assignment and use of the embedded librarian support?
132 APPENDIX E INSTRUCTOR INTERVIEW QUESTIONS 1. What were the expectations you perceived prior to starting? 2. Expectations of the embedded librarian and the research experiences of the participants? 3. a function of their professional experience in medicine? What kind of factors may influence that? 4. Did the experience meet yo ur expectations? 5. What are the benefits and concerns for collaborating with a librarian? 6. 7. Do you perceive that the embedded librarian experience improved the quality of the annotated bibliography assignments? In what ways? 8. Did you discover that students found better resources for their assignments if they watched the lectures and instructional vi deos? 9. What would you do differently? 10. Final thoughts or comments.
133 LIST OF REFERENCES Addoyunlu, B., & Yilmaz Soylu, M. (2008). Development of a scale on learners' views on blended learning and its implementation process. Internet and Higher Education, 11 (1), 26 32. Retrieved from http://vnweb.hwwilsonweb. com/hww/jumpstart.jhtml?recid=0bc05f7a67b1790e70e3 97f5ebc28d34246f70f65fbf2466f170207470993f1ea46a631234c0f868&fmt=C Akeroyd, J. (2005). Information management and e learning: Some perspectives. Aslib Proceedings, 57 (2), 157 167. Retrieved from http://vnweb.hwwilsonweb.com/hww/jumpstart.jhtml?recid=0bc05f7a67b1790e70e3 97f5ebc28d34125f47fc d2b39c66b4a5e11da406c556490ca220f0f10369&fmt=C; Allegri, F. (1985). Course integrated instruction: Metamorphosis for the twenty first century. Medical Reference Services Quarterly, 4 47 66. Retrieved from http://vnweb.hwwilsonweb.com/hww/jumpstart.jhtml?recid=0bc05f7a67b1790e70e3 97f5ebc28d340 dc26cb7ac65c5cc1de11e49c64bc2d2823fdcccd2ab4beb&fmt=C Allen, I & Seaman, J. (2010). Learning on demand: Online education in the united states. USA: Babson Survey Research Group; The Sloan Consortium. Allen, I & Seaman, J. (2009). Learning on Demand: Onli ne Education in the United States Babson Park MA: Babson College Survey Research Group. Retrieved February 9, 2010, from www.sloanconsortium.org/publications/surve y/pdf/learningondemand.pdf Alliger, G. M., Janak, E. A. (1989). Kirkpatrick's levels of training criteria: Thirty years later. Personnel Psychology 42 (2), 331 342. "Guidelines for Distance Learning Library Services," American Library Association, Septembe r 01, 2006. http://www.ala.org/ala/mgrps/divs/acrl/standards/guidelinesdistancelearning.cfm (Accessed May 11, 2010) Document ID: 185689 "Information Literacy Competency Standards for Higher Education," American Library Association, September 01, 2006. http://www.ala.org/ala/mgrps/divs/acrl/standards/informationliteracycompetency.cfm (Accessed May 11, 2010) Document ID: 185693 Association of Research Libraries. 2005 Collaboration for distance learning information literacy instruction: A SPEC KIT. Washi ngton, DC: Association of Research Libraries, Office of Leadership and Management Services
134 Atlas, M. C., Smigielski, E. M., Wulff, J. L., & Coleman, M. T. (2003). Case studies from morning report: Librarians' role in helping residents find evidence based clinical information. Medical Reference Services Quarterly, 22 (3), 1 14. doi:10.1300/J115v22n03_01 Auerbach, J. D., Lonner, B. S., Antonacci, M. D., & Kean, K. E. (2008). Perioperative outcomes and complications related to teaching residents and fellows in scoliosis surgery. Spine, 33 (10), 1113 1118. doi:10.1097/BRS.0b013e31816f69cf uy766 Badke, W. (2009). Ramping up the one shot. Online (Weston, Conn.), 33 (2), 47 49. Retrieved from HTML: http://vnweb.hwwilsonweb.com/hww/jumpstart.jhtml?recid=0bc05f7a67b1790e70e3 97f5ebc28d347 1ce09edbf1de6275d1cb57f369da93f1ae884504ddd03c6&fmt=H PDF: http://vnweb.hwwilsonweb.co m/hww/jumpstart.jhtml?recid=0bc05f7a67b1790e70e3 97f5ebc28d3471ce09edbf1de6275d1cb57f369da93f1ae884504ddd03c6&fmt=P Bailey Hainer, B., & Horton, V. (2005). Library support for E learning in colorado. Colorado Libraries, 31 (3), 6 9. Retrieved from Full Text HTMLFull Text PDF HTML: http://vnweb.hwwilsonweb.com/hww/jumpstart.jhtml?recid=0bc05f 7a67b1790e70e3 97f5ebc28d34125f47fcd2b39c661fa04a4d34e3519d5d6d496dafa4f63a&fmt=H Baird, C. M., & Wilson, P. (2002). Distance learning librarian: Essential team member in distance learning design. Journal of Library Administration, 37 (1/2), 49. Bandura, A.J. (1977) Self efficacy: The exercise of control New York: Freeman. Bangert Drowns, R.L. (1986). Review of developments in meta analytic methods. Psychological Bulletin, 99, 388 399. Bangert Drowns, Robert L. & Rudner, Lawrence M. (1991). Meta analysi s in educational research. Practical Assessment, Research & Evaluation 2(8). Barton, J., German, R., & Joint, N. (2004). Beyond the happy sheets! evaluating learning in information skills teaching. IATUL Annual Conference Proceedings,14 1 1. Retrieved fr om https://search.ebscohost.com/login.aspx?direct=true&db=lxh&AN=15353977& site=ehost live Bauerly, R. J., & Johnson, D. T. (2005). An evaluation of journals used in doctoral marketing programs. Journal of the Academy of Marketing Science, 33 (3), 313 329. doi:10.1177/0092070304272052
135 Behr, M. D. (2004). On ramp to research: Creation of a multimedia library instruction presentation for off campus students. Journal of Library Administration, 41 (1/2), 19 30. Retrieved from http://vnweb.hwwilsonweb.com/hww/jumpstart.jhtml?recid=0bc05f7a67b1790e70e3 97f5ebc28d34e9c89366c0a401091664c9e9bda9ab99745fe2b5d570fafa&fmt=C; Beile, P. (2002). The effect of library instruction learning environments on self efficacy levels and learning outcomes of graduate students in education., 8. Beile, P. M. (2003). Effectiveness of course integrated and repeated library instruction on library skills of education students. Journal of Educational Media & Library Sciences, 40 (3), 271 277. Retrieved f rom Full Text HTML Full Text PDF HTML: http://vnweb.hwwilsonweb.com/hww/jumpstart.jhtm l?recid=0bc05f7a67b1790e70e3 97f5ebc28d34fbec02af0eab02136119210743b56c24eeebb2570b9148ba&fmt=H PDF: http://vnweb.hwwilsonweb.com/hww/jumpstart.jhtml?recid=0bc05f7a67b1790e70e3 97f5ebc28d34fbec02af0eab02136119210743b56c24eeebb2570b9148ba&fmt=P Beile, P. M., & Boote, D. N. (2004). Does the medium matter?: A comparison of a web based tutorial with face to face library instruction on education students' self efficacy levels and learning outcomes. Research Strategies, 20 (1/2), 57 68. Retrieved from http://vnweb.hwwilsonweb.com/hww/jumpstart.jhtml?recid=0bc05f7a67b1790e70e3 97f5ebc28d34e9c89366c0a401097266d4518b5aa893be1f18c31f2304c5&fmt=C; The author of this article has chosen the following Web sites: http://www.acrl.org/ala/acrl/acrlstandards/informationliteracycompetency.htm Bell, S. J. (2005). Creating community online. American Libraries, 36 (4), 68 71. Retrieved from Full Text HTMLFull Text PDF HTML: http://vnweb.hwwilsonweb.com/hww/jumpstart.jhtml?recid=0bc05f7a67b1790 e70e397f5ebc28d34125f47fcd2b39c667a1771a64439795ab50732fc721b9587&fmt= HPDF: http://vnweb.hwwilsonweb.com/hww/jumpstart.jhtml?recid=0bc05f7a67b1790 e70e397f5ebc28d34125f47fcd2b39c667a1771a64439795ab50732fc721b9587&fmt= P Bell, S. J., & Shank, J. (2004). The blended librarian: A blueprint for redefining the teaching and learning role of academic librarians. College & Research Libraries News, 65 (7), 372 375. Retrieved from Full Text HTMLFull Text PDF HTML: http://vnweb.hwwilsonweb.com/hww/jumpstart.jhtml?recid=0bc05f7a67b1790 e70e397f5ebc28d34e9c89366c0a4010948abdfbdac78 99be8a74cf20e2c7c1ac&fmt =HPDF: http://vnweb.hwwilsonweb.com/hww/jumpstart.jhtml?recid =0bc05f7a67b179 0e70e397f5ebc28d34e9c89366c0a4010948abdfbdac7899be8a74cf20e2c7c1ac&fmt =P;
136 Bird, J., & Morgan, C. (2003). Adults contemplating university study at a distance: Issues, themes and concerns. International Review of Research in Open and Distance Learning, Retrieved from Find it @ UF http://vnweb.hwwilsonweb.com/hww/jumpst art.jhtml?recid=0bc05f7a67b1790e7 0e397f5ebc28d34f09e2d096935499dc58dc3a88675bc56d44485e052db680e&fmt= C Birdsall, W. F. (2008). The chiasmus of librarianship and collaborative research for evidence based practice. Evidence Based Library and Information Prac tice,3 (2), 65 75. Retrieved from Find it @ UF http://vnweb.hwwilsonweb.com/hww/jumpst art.jhtml?recid=0bc05f7a67b1790e7 0e397f5ebc28d34246f70f65fbf2466b0d996a043b232431808c9639c4eb2cc&fmt=C; Blakeslee, S., & Johnson, K. (2002). Using HorizonLive to deliver library instruction to distance and online students. Reference Services Review, 30 (4) 324 329. Bober, C., et al. (1995). Evaluating library instruction in academic libraries: A critical review of the literature, 1980 1993. Reference Librarian, (51 52), 53 71. Bogel, G. (2008). Facets of practice. Knowledge Quest, 37 (2), 10 15. Retrieved from Full Text HTMLFull Text PDF HTML: http://vnweb.hwwilsonweb.com/hww/j umpstart.jhtml?recid=0bc05f7a67b1790 e70e397f5ebc28d34246f70f65fbf246687af4065c857e9b347a25cc77bbdc272&fmt= HPDF: http://vnweb.hwwilsonweb.com/hww/jumpstart.jhtml?recid=0bc05f7a67b1790 e70e397f5ebc28d34246f70f65fbf246687af4065c857e9b347a25cc77bbdc272&fmt= P Boote, P. & Beile, D. (2005). Scholars before researchers: On the centrality of the disse rtation literature review in research preparation Educational Researcher, 34(6), 3 15 Bordonaro, K., & Richardson, G. (2004). Scaffolding and reflection in course integrated library instruction. The Journal of Academic Librarianship, 30 (5), 391 401. Retrieved from Full Text HTML Full Text PDF HTML: http://vnweb.hwwilsonw eb.com/hww/jumpstart.jhtml?recid=0bc05f7a67b1790e70e3 97f5ebc28d34e9c89366c0a4010949f7e2c1fb3346a73250acd624440f68&fmt=H PDF: http://vnweb.hwwilsonweb.com/hww/jumpstart.jhtml?recid=0bc05f7a67b1790e70e3 97f5ebc28d34e9c89366c0a4010949f7e2c1fb3346a73250acd624440f68&fmt=P Borowski, D. W., Ratcliffe, A. A., Bharathan, B., Gunn, A., Bradburn, D. M ., Mills, S. J., . Northern Region Colorectal Cancer Audit Group (NORCCAG). (2008). Involvement of surgical trainees in surgery for colorectal cancer and their effect on outcome. Colorectal Disease : The Official Journal of the Association of Coloproct ology of Great Britain and Ireland, 10 (8), 837 845. doi:10.1111/j.1463 1318.2007.01465.x
137 Bowler, M., & Street, K. (2008). Investigating the efficacy of embedment: Experiments in information literacy integration Retrieved from https://search.ebscohost.com/login.aspx?direct=true&db=lxh&AN=35937284&s ite=ehost live Bozeman, D., & Owens, R. (2008). Providing services to o nline students: Embedded librarians and access to resources. Mississippi Libraries, 72 (3), 57 59. Retrieved from http://search.ebscohost.com/l ogin.aspx?direct=true&db=lxh&AN=34694030&site=e host live Bradley, D. R., Rana, G. K., Lypson, M. L., & Hamstra, S. J. (2010). A centralized practice based learning and improvement curriculum for residents and fellows: A collaboration of health sciences li brarians and graduate medical education administration. Journal of the Medical Library Association JMLA, 98 (2), 175 178. doi:10.3163/1536 5050.98.2.013 Brazzeal, B., & Fowler, R. (2005). Patterns of information use in graduate research in forestry: A cita tion analysis of master's theses at mississippi state university. Science & Technology Libraries, 26 (2), 91 106. doi:10.1300/J122v26n02 05 Brettle, A. (2007). Transforming the profession: Reflections on evidence based library and information practice, 4th international conference. Evidence Based Library and Information Practice, 2 (2), 123 126. Retrieved from Find it @ UF http://vnweb.hwwilsonweb.com/hww/jumpstart.jhtml?recid=0bc05f7a67b1790e7 0e397f5ebc28d343b1cd1d482ab0cfacb22ef18a541b68f61883f99df8e0a14&fmt=C; The author of this article has chosen the following Web sites: http://www.eblip4.unc.edu/ Brown, J. F., & Nelson, J. L. (2003). Integration of information literacy into a revised medical school curriculum. Medical Reference Services Quarterly, 22 (3), 63 74. doi:10.1300/J115v22n03_ 07 Brown, J.S., Collins, A. & Duguid, S. (1989). Situated cognition and the culture of learning. Educational Researcher 18(1), 32 42. Bruer, J. T. (1985). Methodological quality and citation frequency of the continuing medical education literature. Journal of Documentation, 41 165 172. Retrieved from http://vnweb.hwwilsonweb.com/hw w/jumpstart.jhtml?recid=0bc05f7a67b1790e70e3 97f5ebc28d340dc26cb7ac65c5cc008f217c6a47898a1b6cb9309b7e1ba6&fmt=C
138 Brunvand, A., & Pashkova Balkenhol, T. (2008). Undergraduate use of government information: What citation studies tell us about instruction st rategies. Portal: Libraries & the Academy, 8 (2), 197 209. Retrieved from http://search.ebscohost.com/login.aspx?direct=true&db=lxh&AN=31790012 &site=e host live Buddy, J. W. (2006). A virtual new start. School Library Media Activities Monthly, 23 (1), 55 58. Retrieved from http://vnweb.hwwilsonweb.com/hww/jumpstart.jhtml?recid=0bc05f7a67b1790e70e3 97f5ebc28d34efc82a53b652dcead348f2d94b8e9e5406a65f5fcd44ed7d&fmt=C Cain, D. L., & Lockee, B. (2002). Student support services at a distan ce: Are institutions meeting the needs of distance learners?, 16. Callison, D. (1997). Evolution of methods to measure student information use. Library & Information Science Research, 19 (4), 347 357. Retrieved from Find it @ UF http://vnweb.hwwilsonweb.com/hww/jumpstart.jhtml?recid=0bc05f7a67b1790e7 0e397f5ebc28d34c 0f2954970d7ad6c636012cfb487544d689b40eb53e5af3f&fmt=C Carnwell, R., & Harrington, C. (2001). Diagnosing student support needs for distance learning. AIR 2001 annual forum paper, 24. Casey, D. M. (2008). A journey to legitimacy: The historical development o f distance education through technology. TechTrends: Linking Research and Practice to Improve Learning, 52 (2), 45 51. Retrieved from http://dx.doi.org/10.1007/s11528 008 0135 z Cassner, M., & Adams, K. E. (2006). Assessing the professional development needs of distance librarians in academic libraries. Journal of Library Administration, 45 (1/2), 81 99. Retrieved from Find it @ UF http://vnweb.hwwilsonweb.com/hww/jumpstart.jhtml?recid=0bc05f7a67b1790e7 0e397f5ebc28d34efc82a53b652dcea2f3e84f71c240be61781b1a2b6d826d0 &fmt=C ; The author of this article has chosen the following Web sites: http://www.ala.org/acrl/resjune02.htmlhttp://www.lu.com/odli Cavanaugh, C., Dawson, K., & Black, E W. (2010). Performance narrative. (Grant Annual Report Chakraborty, M., & Victor, S. (2004). Do's and don' ts of simultaneous instruction to on campus and distance students via videoconferencing. Journal of Library Administration, 41 (1/2), 97 112. Chau dhuri, N., Grayson, A. D., Grainger, R., Mediratta, N. K., Carr, M. H., Soorae, A. S., & Page, R. D. (2006). Effect of training on patient outcomes following lobectomy. Thorax, 61 (4), 327 330. doi:10.1136/thx.2005.046847
139 Chen, K., & Lin, P. (2011). Information literacy in university library user education. Aslib Proceedings, 63 (4), 399 418. doi:10.1108/00012531111148967 Churkovick, M., Oughtred, C. (2002) Can an Online Tutorial Pass the Test for Library Instruction? An Evaluation and Comparison of Library Skills Instruction Methods for First Year Students at Deakin University Australian Academic & Research Libraries 33(1), 25 38. Cipkin, C. (2002). Using a virtual learning environment to integrate information skills into the curriculum: A subject librarian's experiences. SCONUL NEWSLETTER, 7 9. Clark, J. M. & Paivio, A. (1991). Dual coding theory and education. Educational Psychology Review, 3(3), 149 170. Clark, R.E. (1983). Reconsidering research on learning from media. Review of Educational R esearch, 53 (4), 445 459. Clark, W., Hernandez, J., McKeon, B. A., Kahn, A., Morton, C., Toomey, P., . Rosemurgy, A. (2010). Surgery residency training programmes have greater impact on outcomes after pancreaticoduodenectomy than hospital volume or surgeon frequency. HPB : The Official Journal of the International Hepato Pancreato Biliary Association, 12 (1), 68 72. doi:10.1111/j.1477 2574.2009.00130.x Clayton, P. (1995). No easy option: Case study research in libraries. Australian Academic & Researc h Libraries, 26 69 75. Retrieved from Find it @ UF http://vnweb.hwwilsonweb.com/hww/j umpstart.jhtml?recid=0bc05f7a67b1790e7 0e397f5ebc28d349180c1451b57511e799b84dc99d879be309c086b93f629d2&fmt= C Cmor, D., Chan, A., & Kong, T. (2010). Course integrated learning outcomes for library database searching: Three assessment points on the path of ev idence. Evidence Based Library and Information Practice, 5 (1), 64 81. Retrieved from http://vnweb.hwwilsonweb.com/hww/jumpstart.jhtml?recid=0bc05f7a67b1790e 70e397f5ebc28d34c9aabf518ccc3f8e102d5633b21ed22ccc839ede73a78b4a&fmt= C Cmor, D., & Marshall, V. (2006). Librarian class attendance: Methods, outcomes and opportunities. IATUL Annual Con ference Proceedings, 16 56 61. Retrieved from http://search.ebscohost.com/login.aspx?direct=true&db=lxh&AN=26225970&site=e host live Collins, A., Brown, J. S. and Newman, S. E. (1989). Cognitive apprenticeship: Teaching the crafts of reading, writing, and mathematics. In L.B. Resnick (Ed.), Knowing, learning and instruction: Essays in honour of Robert Glaser Hillsdale, NJ: LEA, pp. 453 494.
140 Collins, K. M. T., & Veal, R. E. (2004). Off anxiety as a predictor of attitudes toward the internet. Library & Information Science Research, 26 5 14. Cook, D. L. (2002). Ship to shore: An online information literacy tutorial using BlackBoard distance education software. Journal of Library Administration, 37 (1/2), 177. Cook, S. (2009). Important Events in the Development of Academic Advising in the United States. NACADA Journal 29 (2), 18 40. Cornelius, S., & Gordon, C. (2008). Providing a flexible, learner centred programme: Challenges for educators. Internet and Higher Education, 11 (1), 33 41. Retrieved from http://vnweb.hwwilsonweb.com/hww/jumpstart.jhtml?recid=0bc05f7a67b1790e70e3 97f5ebc28d34246f70f65fbf2466f170207470993f1eb0793b21a0800322&fmt=C; Craven, C. K., Goode, V., Twose, C., Zhang D., & Roderer, N. K. (2010). WelServe: The DBMS for capturing and tracking welch medical library's embedded informationist service delivery at johns hopkins medical institutions. Journal of Library Administration, 50 (4), 397 411. doi:10.1080/01930821003667112 Dare, L. A., Zapata, L. P., & Thomas, A. G. (2005). Assessing the needs of distance learners: A student affairs perspective. New Directions for Student Services, (112), 39 54. Retrieved from http://dx.doi.org/10.1002/ss.183 Davis, E. (2005). Virtual research and the online school. Colorado Libraries, 31 (3), 14. Retrieved from Full Text HTMLFull Text PDF HTML: http://vnweb.hwwilsonweb.com/hww/jumpstart.jhtml?recid=0bc05f7a67b1790e70e3 97f5ebc28d34125f47fcd2b39c661fa04a4d34e3519d8dc9a8253aaa9ba7&fmt=H D e Groote, S. L. (2008). Citation patterns of online and print journals in the digital age. Journal of the Medical Library Association, 96 (4), 362 369. Retrieved from http://search.ebscohost.com/login.aspx?direct=true&db=lxh&AN=35974523&site=e host live Dewey, B. I. ( 2004). The embedded librarian: Strategic campus collaborations. Resource Sharing & Information Networks, 17 Distance education and training council: The global leader in distance learning accreditation Retrieved 11/18/2011, 2011, from http://www.detc.org/ Dharmadhikari, V. (2 007). Education for capacity building: New strategies to include ICT. Information Studies, 13 (3), 131 138.
141 Dorsch, J. L., Jacobson, S., & Scherrer, C. S. (2003). Teaching EBM teachers: A team approach. Medical Reference Services Quarterly, 22 (2), 107 114. doi:10.1300/J115v22n02_10 Doyle, B. (2006). Crossing the digital divide. EContent, 29 (7), 36. Drori, O. (2003). Finding a paper's subject based on cited and citing papers. Journal of Information Science, 29 (6), 537 540. Retrieved from http://search.ebscohost.com/login.aspx?direct=true&db=lxh&AN=12033906&site=e host live Duffy, T.M. and Kirkley, J.R. (2004). Learner Centered th eory and practice in distance education: cases from higher education London: Lawrence Erlbaum Associates. Dugan, M. (2008). Embedded librarians in an ag econ class: Transcending the traditional. Journal of Agricultural & Food Information, 9 (4), 301 309. doi:10.1080/10496500802480342 Dunlap, S. (2002). "Watch for the Little Red Light: Delivery of Bibliographic Instruction by Unconventional Means." In The Tenth Off Campus Library Services Conference Proceedings: Cincinnati, Ohio, April 17 19, 2002 edited by Patrick B. Mahoney. Mount Pleasant, MI: Central Michigan University, 2002, 221 225. Reprinted in Journal of Library Administration 37(1/2), 279 285. Eckel, E. J. (2009). The emerging engineering scholar: A citation analysis of theses and dissertations at western michigan university. Issues in Science & Technology Librarianship, (56), 5 5. Retrieved from http://search.ebscohost.com/login.aspx ?direct=true&db=lxh&AN=44059094&site=e host live Edwards, M. Kumar, S. Ochoa, M. (2010). Assessing the Value of Onine Embedded Librarians. Public Services Quarterly 6(2/3) June 2010. Edzan, N. N. (2008). Analysing the references of final year project repo rts. Journal of Educational Media & Library Sciences, 46 (2), 211 231. Retrieved from http://search.ebscohost.com/login.aspx?direct=true&db=lxh &AN=37131377&site=e host live Ferree, N., Schaefer, N., Butson, L. C., & Tennant, M. R. (2009). Liaison librarian tiers: Levels of service. Journal of the Medical Library Association: JMLA, 97 (2), 145. Ferguson, K. S., & Ferguson, A. (2005). The remote library and point of need user education: An australian academic library perspective. Journal of Interlibrary Loan, Document Delivery & Information Supply, 15 (3), 43 60. Retrieved from http://vnweb.hwwilsonweb.com/hww/jumpstart.jhtml?recid=0bc05f7a67b1790e70e3 97f5ebc28d34125f47fcd2b39c66f9628619bc7a5f8e6b6ba412a882f675&fmt=C;
142 Finkelstein, J. (2006) Learning in real time: synchronous teaching and learning online San Francisco, CA: Jossey Bass. Fisher, M., & Baird, D. E. (2005). Online learning design that fosters student support, self regulation, and retention. Campus Wide Information Systems, 22 (2), 88 107. Retrieved from http://dx.doi.org/10.1108/10650740510587100; http://vnweb.hwwilsonweb.com /hww/jumpstart.jhtml?recid=0bc05f7a67b1790e70e397f5ebc28d34f 09e2d09693549 9d67ddaa9c1c1be22ebede34624821f966&fmt=C Fletcher, K. M. (2005). Self efficacy as an evaluation measure for programs in support of online learning literacies for undergraduates. Internet and Higher Education, 8 (4), 307 322. Retrieved from The author of this article has chosen the following Web sites: http://www.apa.org/monitor/oct04/gifts.htmlhtt p://www.abds.gov.au/ausstatshtt p://it.coe.uga.edu/itformm/paper75/paper75.html; http:// vnweb.hwwilsonweb.com/hw w/jumpstart.jhtml?recid=0bc05f7a67b1790e70e397f5ebc28d34125f47fcd2b39c664 5da4eb0c202cd2ddb3557b50fcc464d&fmt=C Foote, M. (2001). Research & librarianship. North Carolina Libraries, 59 (1), 4 25. Retrieved from Full Text PDF PDF: http://vnweb.hwwilsonweb.com/hww/jumpstart.jhtml?recid=0bc05f7a67b1790e 70e397f5ebc28d347528 5a693210cd5738676e161151d72a033d4fa7917bec47&fmt =P Forte, E., & DeDecker, S. (2005). Distance government publications instruction to library school students: Lessons learned. DttP, 33 (1), 28 31. Foster, F. & Gibbons, S. (Eds.). (2007). Studying Students. Chicago: Association of College and Research Libraries. Fourie, I. (2001). Using a variety of methods of distance teaching methods to teach online search skills to working adults. National Online 2001 Proceedings, 139 156. Freiburger, G., & Kramer, S. (2009). Embedded librarians: One library's model for decentralized service. Journal of the Medical Library Association, 97 (2), 139 142. Retrieved from http://search.ebscohost.com/login.aspx?direct=true&db=lxh&AN=40097809&site=e host live Fritts, J., & Casey, A. M. (2010). Who trains distance librarians? A study of the training and development needs of distance learning librarians. Journal of Lib rary Administration, 50 (5/6), 617 627. Retrieved from http://vnweb.hwwilsonweb.com/hww /jumpstart.jhtml?recid=0bc05f7a67b1790e70e3 97f5ebc28d34c9aabf518ccc3f8e541209cf6ab81d147eb260a68996db0e&fmt=C
143 Gandhi, S. (2003). Academic librarians and distance education challenges and opportunities. Reference and User Services Quarterly, 43 (2), 138 154 Gao, S., Yu, W., & Luo, F. (2009). Citation analysis of PhD thesis at wuhan university, china. Library Collections, Acquisitions, & Technical Services, 33 (1), 8 16. doi:10.1016/j.lcats.2009.03.001 Garnsey, M. R. (2002). What distance learners should kn ow about information retrieval on the world wide web. Distance Learning: Information Access and Services for Virtual Users, 37 (77), 19. Garrison, D. R., & Shale, D. (1987). Mapping the boundaries of distance education: Problems in defining the field. Amer ican Journal of Distance Education, 1 (1), 7 13. Garrison, D. R., Anderson, T., & Archer, W. (1999). Critical inquiry in a text based environment: Computer conferencing in higher education. The Internet and Higher Education, 2 (2 3), 87 105. Garrison, D. R ., Anderson, T., & Archer, W. (2001). Critical thinking and computer conferencing: A model and tool to assess cognitive presence. Garrison, D. R., Anderson, T., & Archer, W. (2001). Critical thinking, cognitive presence, and computer conferencing in dista nce education. American Journal of Distance Education, 15 (1), 7 23. Gibbons, S. (2005). Course management systems. Library Technology Reports, 41 (3), 7 11. Retrieved from http://search.ebscohost.com/login.aspx?direct=true&db=lxh&AN=17297353&site=e host live Glaser, B. G & Strauss, A. L. ( 1967 ) The Discovery of Grounded Theory: Strategies for Qualitative Research, Chicago, Aldine Publishing Company Glass, G.V, McGaw, B., & M.L. Smith (1981). Meta analysis in social research Beverly Hills, CA: Sage. Glynn, L. (2008). Evidence based practice outside the box. Evid ence Based Library and Information Practice, 3 (3), 1 2. Retrieved from Find it @ UF http://vnweb.hwwilsonweb.com/hww/jumpstart.jhtml?recid=0bc05f7a67b1790e70 e397f5ebc28d34246f70f65fbf2466f34c7f30b68572dd10d7bd1ff5ca200c&fmt=C Gooch, J. (1998). They blazed a trail for distance education Retrieved Februaruy 3 rd 2011, from http://www.uwex. edu/disted/gooch.cfm Grover, R. (1994). Assessing information skills instruction. Reference Librarian, (44), 173 189.
144 Grassian, E. S., & Kaplowitz, J. R. (2001). Information literacy instruction: Theory and practice. information literacy sourcebooks Nea l Schuman Publishers, Inc., 100 Newfield Ave., Edison, NJ 08837 ($55). Tel: 866 672 6657 (Toll Free); Fax: 866 209 7932 (Toll Free); e mail: orders@neal schuman.com ; Web site: http: // www.neal schuman.com. Retrieved from No Hampel, R. (2003) Theoretical perspectives and new practices in audio graphic conferencing for language learning ReCALL 15(1), 21 36. Hampel, R. (2006) Rethinking tas k design for the digital age: a framework for language teaching and learning in a synchronous online environment ReCALL 18(1), 105 121. Hannelore B. Rader. (2000). Library instruction and information literacy 1999. Reference Services Review, 28 (4), 378. Haricharan, R. N., Aprahamian, C. J., Celik, A., Harmon, C. M., Georgeson, K. E., & Barnhart, D. C. (2008). Laparoscopic pyloromyotomy: Effect of resident training on complications. Journal of Pediatric Surgery, 43 (1), 97 101. doi:10.1016/j.jped surg.2007.09.028 Haynes, A. (2002). Distance learning library services: Challenges and opportunities for an academic library system. Helliwell, M. (2007). Reflections of a practitioner in an evidence based world: 4th international evidence based library & information practice conference. Evidence Based Library and Information Practice, 2 (2), 120 122. Retrieved from http://vnweb.hwwilsonweb.com/hww/jumpstart.jhtml?recid=0bc05f7a67b1790e 70e397f5ebc28d343b1cd1d482ab0cfacb22ef18a541b68fadd86e55513af743&fmt= C Henner, T. (2002). "Bridging the Distance: Bibliographic Instruction for Remote Library Users." Medical Reference Services Quarterly 21(1), 79 85. Herther, N. K. (2009). Research evaluation and citation analysis: Key issues and implications. Electronic Library, 27 (3), 361 375. Retrieved from http://search.ebscohost.com/login.aspx?direct=true&db=lxh&AN=43810907&site=e host live Hirose, Y., & Nakazawa, N. (2007). Special feature: Evalu ation methods in library and information activities. methods of journal collection assessment using journal citation data. Journal of Information Science & Technology Association/Joho no Kagaku to Gijutsu, 57 (8), 396 403. Retrieved from http://search.ebscohost.com/login.aspx?direct=true&db=lxh&AN=31970433&site=e host live
145 Hoffert, S. P., 1997. Meta Analysis Gaining Status In Science And Policymaking The Scientist Vol. 11, n.18, p. 1,5 Hricko, M. (2002). Using the invisible web to teach information liter acy. Journal of Library Administration, 37 (3/4), 379. Hunter, J.E., & F.L. Schmidt (1990). Methods of meta analysis Newbury Park, CA: Sage. Hurst, S., & Leonard, J. (2007). Garbage in, garbage out: The effect of library instruction on the quality of students' term papers. E JASL: The Electronic Journal of Academic and Special Librarianship, Retrieved from http://vnweb.hwwilsonweb.com/hww/jumpstart.jhtml?recid=0bc05f7a67b1790e70e3 97f5ebc28d343b1cd1d482ab0cfae7db56fc517fa7cb4204626e9016b971&fmt=C Jayne, E. A. (2000). The librarian as bricoleur: Meeting the needs of distance learners. Ref erence Services for the Adult Learner: Challenging Issues for the Traditional and Technological Era, 161. Jiao, Q. G., Onwuegbuzie, A. J., & Waytowich, V. L. (2008). The relationship between citation errors and library anxiety: An empirical study of doc toral students in education. Information Processing & Management, 44 (2), 948 956. doi:10.1016/j.ipm.2007.05.007 Johnson, W. G. (2008). Educational technology and college librarianship. College & Undergraduate Libraries, 15 (4), 463 475. Retrieved from Find it @ UF http://vnweb.hwwilsonweb.com/hww/jumpstart.jhtml?recid=0bc05f7a67b1790e70 e397f5ebc28d34246f70f65fbf2466df4069b4b7e5772520036889ebcb3524&fmt=C; The author of this article has chosen the following Web sites: http://aect.org/About/Historyhttp://aect.org/About/Ethics.asphttp://ala.org/ala/a crl/acrlstandards/informationliteracycompetency.htm Jones, M. F. (2 003). Help! I'm the new distance LibrarianWhere do I begin? Journal of Library Administration, 37 (3/4), 397 410. Retrieved from Find it @ UF http://vnweb.hwwilsonweb.com/hww/jumpstart.jhtml?recid=0bc05f7a67b1790e70 e397f5ebc28d34fbec02af0eab021351732b2abd668b6f9cbe3c7ed294f719&fmt=C; The author of this article has chosen the following Web si tes: http://caspian.switchinc.org/ distlearn/ Kappus, T., & Jenks, K. (2008). Traveling the information goat trail: Serving the very distant, the involuntarily clueless and the self descri bed techno peasant. PNLA Quarterly, 73 (1), 5 9. Kearley, J. & Phillips, L. (2004) Embedding library reference services in online courses. Improving Internet Reference Services to Distance Learners 9(1/2), 65 76.
146 Kelly, R. (2001). An integrated approach to student services. Distance Education Report, 5 (6), 5. Retrieved from http://vnweb.hw wilsonweb.com/hww/jumpstart.jhtml?recid=0bc05f7a67b1790e 70e397f5ebc28d34f09e2d096935499dc1407572df1537369edfc42505249224&fmt= C Kesselman, M. A., & Watstein, S. B. (2009). Creating opportunities: Embedded librarians. Journal of Library Administration, 49 (4) 383 400. doi:10.1080/01930820902832538 Kinder, R. (2002). Instructional services for distance education. Distance Learning: Information Access and Services for Virtual Users, 37 (77), 63. Klem, M. L., & Weiss, P. M. (2005). Evidence based resources and the role of librarians in developing evidence based practice curricula. Journal of Professional Nursing : Official Journal of the American Association of Colleges of Nursing, 21 (6), 380 387. doi:10.1016/j.profnurs.2005.10.004 Kirkpatrick D. L. (1959). Tec hniques for evaluating training programs. ''Journal of American Society of Training Directors'', 13 (3): pp. 21 26. Kronenfeld, M., Stephenson, P. L., Nail Chiwetalu, B., Tweed, E. M., Sauers, E. L., McLeod, T. C., . Ratner, N. B. (2007). Review for librarians of evidence based practice in nursing and the allied health professions in the united states. Journal of the Medical Library Association : JMLA, 95 (4), 394 407. doi:10.3163/1536 5050.95.4.394 Kuhlthau, C. C. (1988). Developing a model of the library search process: Cognitive and affective aspects. Reference Quarterly, 28 (2), 232 242. Kuhlthau, C. C. (1991). Inside the search process: Information seeking from the user's perspective. Journal of the American Society for Information Science, 42 (5 ), 361 371. Kulik, J.A., & C. L.C. Kulik (1989). Meta analysis in education. International Journal of Educational Research 13, 221 340. Kurbanoglu, S. S. (2003). Self efficacy: A concept closely linked to information literacy and lifelong learning. Jour nal of Documentation, 59 (6), 635 646. Retrieved from The author of this article has chosen the following Web sites: http://www.ala.org/aas/positions/PS_infolit.htmlhttp://www.ala.org/acrl/ilstandar dlo.htmlhttp://graduateresearch.com/Kear.htm; http://vnweb.hwwilsonweb.com/hww/ jumpstart.jhtml?recid=0bc05f7a67b1790e70e397f5ebc28d34fbec02af0eab021343ff 73498afa938697c34fb450488671&fmt=C
147 Kurbanoglu, S. S., Akkoyunlu, B., & Umay, A. (2006). Developin g the information literacy self efficacy scale. Journal of Documentation, 62 (6), 730 743. Kontos, F., & Henkel, H. (2008). Live instruction for distance students: Development of synchronous online workshops. Public Services Quarterly, 4 (1), 1 14. doi:10.1080/15228950802135657 Koufogiannakis, D., & Crumley, E. (2003). Facilitating evidence based librarianship: A C anadian experience. Health Information and Libraries Journal, 20 Suppl 1 73 75. Kozma, R.B. (1991). Learning with media. Rev iew of Educational Research, 61 (2), 179 211. Kumar, K., & Sharma, J. (2007). E learning: The learner centred education for LIS professionals. Journal of Library & Information Science, 32 (1), 1 16. Retrieved from http://search.ebscohost.com/login.aspx?direct=true&db=lxh&AN=28156675&site=e host live Lillard, L. L., Norwood, S., Wise, K., Brooks, J., & Kitts, R. (2009). Embedded librarians: MLS s tudents as apprentice librarians in online courses. Journal of Library Administration, 49 (1), 11 22. doi:10.1080/01930820802310544 Lin, Y. Y., Shabbir, A., & So, J. B. (2010). Laparoscopic appendectomy by residents: Evaluating outcomes and learning curve. Surgical Endoscopy, 24 (1), 125 130. doi:10.1007/s00464 009 0691 0 Lindsay, E. B., Cummings, L., & Johnson, C. M. (2006). If you build it, will they learn? assessing online information literacy tutorials. College & Research Libraries, 67 (5), 429 445. Retrieved from HTML: http://vnweb.hwwilsonweb.com/hww/jumpstart.jhtml?recid=0bc05f 7a67b179 0e70e397f5ebc28d34efc82a53b652dcea0637c07791019fe68cc3b625b9b0106c&f mt=HPDF: ht tp://vnweb.hwwilsonweb.com/hww/jumpstart.jhtml?recid=0bc05f7a67b 1790e70e397f5ebc28d34efc82a53b652dcea0637c07791019fe68cc3b625b9b0106 c&fmt=P Locatis, C., Gaines, C., & Liu Wei Li. (2006). A blended training approach using videoconferencing for distance educ ation. Journal of the Medical Library Association, 94 (4), 464 468. Retrieved from http ://vnweb.hwwilsonweb.com/hww/jumpstart.jhtml?recid=0bc05f7a67b1790e70e3 97f5ebc28d34efc82a53b652dcea614d58bcea74efb6db5fff0d4368ede9&fmt=C Loertscher, D. V. (2009). Evidence based practice: Evolution or revolution? Evidence Based Library and Information Pr actice, 4 (2), 178 181. Retrieved from Find it @ UF
148 Love, M., & Norwood, S. (2007). Finding our way as "embedded librarians". College & Undergraduate Libraries, 14 (4), 87 93. doi:10.1080/10691310802128369 Ludwig Hardman, S., & Dunlap, J. C. (2003). Learner support services for online students: Scaffolding for success. International Review of Research in Open and Distance Learning, Retrieved from Find it @ UF http://vnweb.hwwilsonweb.com/hww/jumpstart.jhtml?recid=0bc05f7a67b1790e7 0e397f5ebc28d34f09e2d096935499dc58dc3a88675bc56b4d257f7a693435f&fmt=C Lyons, R. (2009). The call for evidence based practice: Speaking louder than words. Evidence Based Library and Information Practice, 4 (3), 63 67. Retrieved from Find it @ UF http://vnweb.hwwilsonweb.com/hww/jumpstart.jhtml?recid=0bc05f7a67b1790e7 0e397f5ebc28d3471ce09edbf1de627fa456678d103ea68f6804ca49fa7c575&fmt=C; MacDonald, D. (1998) Audio and audiographic learning: t he cornerstone of the information highway Montreal: Cheneliere McGraw hill. Machen State of the University FL. 26 August 2010 Faculty Senate Meeting http://www.president.ufl.edu/state of university/docs/State of the University 2010.pdf Main, L. (1998). WOOing the millennium. Journal of Education for Li brary & Information Science, 39 (4), 339. Retrieved from http://search.ebscohost.com/login.aspx?direct=true&db=lxh&AN=1485550&site=eh ost live M aki, P. L. (2001). From standardized tests to alternative methods: Some current resources on methods to assess learning in general education. Change 33(2), 28 31. Malikowski, S. R., Thompson, M. e., & Theis, J. G. (2006). External factors associated with adopting a CMS in resident college courses. Internet and Higher Education, 9 (3), 163 174. Retrieved from http://vnweb.hwwilsonweb.com/hww/jumpstart.jhtml?recid=0bc05f7a67b1790e70e3 97f5ebc28d34efc82a53b652dcea97126ceb95308856db2908d9a2e432a2&fmt=C; Manus S. J. B. (2009). Librarian in the classroom: An embedded approach to music information literacy for first year undergraduates. Notes, 66 (2), 249 261. Retrieved from https://search.ebscohost.com/login.aspx?direct=true &db=lxh&AN=45177684& site=ehost live ducation program is a whale of a task. Journal of General Education 53(1), 1 19.
149 Markgraf J. S. (2004). Librarian participation in the online classroom. Internet Reference Services Quarterly, 9 (1/2), 5 19. Matthew, V., & Schroeder, A. (2006). The embedded librarian program. EDUCAUSE Quarterly, 29 (4), 61 65. Mayer, R. E. (1997). Multimedia learning: Are we asking the right questions? Educational Psychologist, 32 (1), 1 19. Mayer, R. E. (2005). Cognitive theory of multimedia learning. The Cambridge Handbook of Multimedia Learning, 31 48. Means, B., Toyama, Y., Murphy, R., Bakia, M., & Jones, K. (2009). Evaluation of evidence based practices in online learning: A meta analysis and review of online learning studies. US Department of Education. Available from: ED Pubs. P.O. Box 1398, Jessup, MD 20794 1398. Tel: 877 433 7827; Fax: 301 470 1244; We b site: http://www.edpubs.org Retrieved from ERIC Mellon, C. A. (1984). Process not product in course integrated instruction: A generic model of library research. College and Research Libraries, 45 (6), 471 478. Mellon, C. A. (1986). Library anxiety: A grounded theory and its development. College & Research Libraries, 47 (2), 160 165. Mellon, C. A. (1988). Attitudes: The forgotten dimension in library instruction. Library Journal, 113 (14), 137 139. McBrien, J L., Jones, P., & Cheng, R. (2009). Virtual spaces: Employing a synchronous online classroom to facilitate student engagement in online learning. International Review of Research in Open and Distance Learning, McCarthy, L. H. (1996). Evidence based medicine: An opportunity for health sciences librarians. Medical Reference Services Quarterly, 15 (4), 63 71. doi:10.1300/J115V15N04_06 McEvoy, G. M., & Cascio, W. F. (1987). Do good or poor performers leave? A meta analysis of the relationship between per formance and turnover. Academy of Management Journal, 744 762. McGowan, J., Straus, S. E., & Tugwell, P. (2006). Canada urgently needs a national network of libraries to access evidence. Healthcare Quarterly (Toronto, Ont.), 9 (1), 72 4, 4. McKee, N. P. (2010). A multifaceted approach to the assessment and evaluation of learning commons services and resources. College & Undergraduate Libraries, 17 (2), 297 309. doi:10.1080/10691316.2010.490198
150 McLean, E., & Dew, S. H. (2006). Providing library instruction to distance learning students in the 21st century: Meeting the current and changing needs of a diverse community. Journal of Library Administration, 45 (3/4), 315 337. Retrieved from http://vnweb.hwwilsonweb.com/hww/jumpstart.jhtml?recid=0bc05f7a67b1790e70e3 97f5ebc28d34efc82a53b652dcea140bb142584f1095fc87e2972b700163&fmt=C; Mirijamdotter, A. (2010). Toward collaborative evidence based information practices: Organisation and leadership essentials. Evidence Based Library and Information Practice, 5 (1), 17 25. Retrieved from Find it @ UF http://vnweb.hwwilsonweb.com/hww/jumpstart.jhtml?recid=0bc05f7a67b1790e7 0e397f5ebc28d34c9aabf518ccc3f8e102d5633b21ed22c688634801a7ff04a&fmt=C Monoi, S., O'Hanlon, N., & Diaz, K. R. (2005). Online searching skills: Development of an inventory to assess self efficacy. The Journal of Academic Librarianship,31 (2), 98 105. Retrieved from Full Text HTMLFull Text PDF HTML: http://vnweb.hwwilsonweb.com/hww/jumpstart.jhtml?recid=0bc05f7a67b179 0e70e397f5ebc28d34125f47fcd2b39c666590da0 8d53072c6fb6c5cc633d00def&fmt =HPDF: http://vnweb.hwwilsonweb.com/hww/jumpstart.jhtml?re cid=0bc05f7a67b17 90e70e397f5ebc28d34125f47fcd2b39c666590da08d53072c6fb6c5cc633d00def&f mt=P Moore, M. G. (1989) Three types of interactions American Journal of Distance Education 3(2), 1 6. Moore, M.G. (1993) Theory of transactional distance In D. Keeg an (Ed.), Theoretical principles of distance education (pp. 22 38) New York: Routledge. Moore, M. and Kearsley, G. (2006) Distance education: A systems view New York: Wadsworth Publishing. Moore, M., Lockee, B., & Burton, J. (2002). Measuring success: Evaluation strategies for distance education. Educause Quarterly, 25 (1), 20 26. Morrison, G. R., Kemp, J. E., & Ross, S. M. (200 6 ). Designing Effective Instruction. Hoboken, NJ: J. Wiley & Sons. Mulvaney, S. A., Bickman, L., Giuse, N. B., Lambert, E. W., Sathe, N. A., & Jerome, R. N. (2008). A randomized effectiveness trial of a clinical informatics consult service: Impact on evidence based decision making and knowledge implementation. Journal of the American Medical Informatics Association : JAMIA, 15 (2) 203 211. doi:10.1197/jamia.M2461 Neely, T. Y. (2000). Aspects of Information Literacy: A Sociological and Psychological Study
151 Neuhaus, C., & Daniel, H. (2008). Data sources for performing citation analysis: An overview. Journal of Documentation, 64 (2), 193 210. doi:10.1108/00220410810858010 Nobisso, J. (2008). (Evidence based) practice makes perfect. Knowledge Quest, 37 (2), 70 73. Retrieved from Full Text HTMLFull Text PDF HTML: http://vnweb.hwwilsonweb.com/hww/jumpstart.jhtml?recid=0bc05f7a67b1790 e70e397f5ebc28d34246f70f65fbf246687af4065c857e9b30cf5840076699149&fmt= HPDF: http://vnweb.hwwilsonweb.com/hww/jumpstart.jhtml?recid=0bc05f7a67b1790 e70e397f5ebc28d34246f70f 65fbf246687af4065c857e9b30cf5840076699149&fmt= P Olatokun, W. M. (2009). Citation analysis of doctoral works submitted to the department of animal science, university of ibadan, nigeria. Library Philosophy & Practice, 11 (2), 1 15. Retrieved from http://search.ebscohost.com/login.aspx?direct=true&db=lxh&AN=47776528&site=e host live Onwuegbuzie, A. J. (1997b). Writing a research proposal: The role of library anxiety, statistics anxiety, and composition anxiety. Library and Information Science Research, 19, 5 33. Oo, A. Y., Grayson, A. D., & Rashid, A. (2004). Effect of training on outcomes following coronary artery bypass graft surgery. European Journal of Cardio Thoracic Surgery : Official Journal of the European Association for Cardio Thoracic Surgery, 25 (4), 591 596. doi:10.1016/j.ejcts.2003.12.012 Orme, W. A. (2004) A Study of the residual impact of the Texas information literacy tutorial on the information seeking ability of first year college students. College and Research Libraries 65(3), 205 215. Ory, J. C. (1992). Meta assessment: Evaluating assessment activities. Research in Higher Education, 33 (4), 467 481. Osman, G., & Herring, S. C. (2007). Interaction, facilitation, and deep learning in cross cultural chat: A case study. Internet & Higher Education, 10 (2), 125 141. doi:10.1016/j.iheduc.2007.03.004 Pace, A. K. (2001). Distance learning service: It's closer than you think. Computers in Libraries, 21 (4), 49 51. Paivio, A. (1971). Imagery and Verbal Processes. New York: Holt, Rinehart & Winston. Paivio, A. (1986). Mental Representations. New York: Oxford University Press.
152 Pajares, F. (2006). Self efficacy informatio n Retrieved 01/24/2011, from www.des.emory.edu/mfp/self efficacy.htm Paris, M. (1988). Thoughts on the case study. Journal of Education for Library and Information Science, 29 137 139. Retrieved from Find it @ UF http://vnweb.hwwilsonweb.com/hww/jumpstart.jhtml?recid=0bc 05f7a67b1790e7 0e397f5ebc28d34ddf9242c1b6f55c570cf938c17f1ce31403070d86e679397&fmt=C Patton, M. Q., & Patton, M. Q. (2002). Qualitative research and evaluation methods Thousand Oaks, Calif: Sage Publications. Perrone, V., & Perrone, V. (2000). The changing role of librarians and the online learning environment, 2000. Distance Education: An Open Question, Pival, P. R., & Tunon, J. (2001). Innovative methods for providing instruction to distance students using technology. Journal of Library Administration, 32 (1), 347. Retrieved from http://search.ebscohost.com/login.aspx?direct=true&db=lxh&AN=5088134&site=eh ost live Pollack, M. (2005). Class dismissed. (cover story). Library Journal, 130 2 6. Retrieved from http://search.ebscohost.com/login.aspx?dir ect=true&db=lxh&AN=18576231&site=e host live Powell, R. R. (1997). Basic research methods for librarians Greenwich, Conn: Ablex. Purpur, Geraldine; Ocha, Louise. "New frontiers for distance learning library services : a virtual world library at Appalachi an State University." In Bell, Lori, Trueman, Rhonda B. (Eds). Virtual worlds, real libraries : librarians and educators in Second Life and other multi user virtual environments. Medford, N.J. : Information Today, c2008. Rama, K., Ed., & Hope, A., Ed. (20 09). Quality assurance toolkit for distance higher education institutions and programmes. In (pp. 371) Commonwealth of Learning. Ramsay, K. M. & Kinne, J. (2006). The Embedded Librarian. Library Journal 131( 6), 34 45. Rader, T., & Gagnon, A. J. (2000). Expediting the transfer of evidence into practice: Building clinical partnerships. Bulletin of the Medical Library Association, 88 (3), 247 250. Reeves, L. A. (2005). Trying it on for size: Piloting synchronous online reference service with elluminate vClass. Internet Reference Services Quarterly, 10 (2), 19 33.
153 Rempel, H. G., & McMillen, P. S. (2008). Using courseware discussion boards to engage graduate students in online library workshops. Internet Reference Services Quarter ly, 13 (4), 363 380. Retrieved from http://vnweb.hwwilsonweb.com/hww/jumpstart.jhtml?recid=0bc05f7a67b1790e 70e397f5ebc28d34246f70f65fbf2466d4d3f8dff5abdc7a7f2bc1b2aed9abd4&fmt=C; Ren, W. (2000). Library instruction and college student self efficacy in electr onic information searching. The Journal of Academic Librarianship, 26 (5), 323 328. Retrieved from Full Text HTML HTML: http://vnweb.hwwilsonweb.com/hww/jumpstart.jhtml?recid=0bc05f7a67b179 0e70e397f5ebc28d342 8a64628d64647f48a6cf28b47375e55cc8d43c5c27ff5b9&fmt =H Rourke, L., Anderson, T., Garrison, D. R., & Archer, W. (1999). Assessing social presence in asynchronous text based computer conferencing. Journal of Distance Education, 14 (2), 50 71. Rudin, P. (2008) No fixed address: The evolution of outreach library services on university campuses. Reference Librarian, 49 (1), 55 75. Retrieved from http: //search.ebscohost.com/login.aspx?direct=true&db=lxh&AN=33007447&site=e host live Russell T.L. ( 1999 ) The no significant difference phenome non as reported in 355 research reports, summaries and papers: A comparative research annotated bibliography on tec hnology for distance education North Carolina State University: Office of Instructional Telecommunications. Sacchanand, C., & Jaroenpuntaruk, V. (2006). Development of a web based self training package for information retrieval using the distance educati on approach. The Electronic Library, 24 (4), 501 516. Retrieved from http://vnweb.hwwilsonweb.com/hww/jumpstart.jhtml?recid=0bc05f7a67b1790e70 e397f5ebc28d34efc82a53b652dcea6f0d15f0bd2338abbc84d1531db624c3&fmt= C; Sam, J., & Tackie, S. N. B. (2007). Citation analysis of dissertations at the department of information studies, university of ghana, legon. African Journal of Library, Archives & Information Science, 17 (2), 123 132. Retrieved from http://search.ebscohos t.com/login.aspx?direct=true&db=lxh&AN=27245781&site=e host live Searing, S. E. (2004). All in the family: Library services for LIS online education. Journal of Library Administration, 41 (3/4), 391 405. Retrieved from http://vnweb.hwwilsonweb.com/hww/jumpstart.jhtml?recid=0bc05f7a67b1790e70e3 97f5ebc28d34e9c89366c0a40109d1ef59f132f7f0ddda2b 36119f72f1aa&fmt=C;
154 Seliger, H & Shohamy, E. (1989) Second Language Research Methods Oxford: Oxford University Press. Scales, B. J., & Lindsay, E. B. (2005). Qualitative assessment of student attitudes toward information literacy. Portal: Libraries & th e Academy, 5 (4), 513 526. Schlosser, C. A., & Anderson, M. L. (1994). Distance education: Review of the literature. AECT Publication Sales, 1025 Vermont Ave., NW, Ste. 820, Washington, DC 20005 3547. Schullo, S., Venable, M., Barron, A. E., Kromrey, J. D ., Hilbelink, A. & Hohlfeld, T. (2005) Enhancing online courses with synchronous software: an analysis of strategies and interactions Paper presented at the National Educational Computing Conference, June 27 30 Philadelphia, PA. Shank, J. D. (2006). The blended librarian: A job announcement analysis of the newly emerging position of instructional design librarian. College & Research Libraries, 67 (6), 515 524. Shank, J. The blended librarian. (2005). Library Journal (1976), 130 (5), 19. Retrieved from Full Text HTMLFull Text PDF HTML: http://vnweb.hwwilsonweb.com/hww/jumpstart.jhtml ?recid=0bc05f7a67b1790 e70e397f5ebc28d34125f47fcd2b39c66e29c8be7f05e4c58d471206fd76da626&fmt= HPDF: http://vnweb.hwwilsonweb.com/hww/jumpstart.jhtml?recid=0bc05f7a67b1790 e70e397f5ebc28d34125f47fcd2b39c66e29c8be7f05e4c58d471206fd76da626&fmt= P Sherriff, G. (2010). Information use in history research: A citation analysis of master's level theses Portal: Libraries & the Academy, 10 (2), 165 183. Retrieved from http://search.ebscohost.com/login.aspx?direct=true&db=lxh&AN=49713464&site=e host live Shumaker, D., & Talley, M. (2009). Models of embedded librarianship: Final report. Special Libraries Association. Http://www.Sla.org/pdfs/EmbeddedLibrariansh ipFinalRptRev.Pdf 9 Simpson, O. (2002). Supporting students in online, open and distance learning. second edition. open and distance learning series. In (pp. 226) Sinclair, B. (2009). The blended librarian in the learning commons: New skills for the blended library. College & Research Libraries News, 70 (9), 504 7, 516. Smyth, J. B. (2001). Using a web based MOO for library instruction in distance education. Journal of Library Administration, 32 (1), 383. Retrieved from http://search.ebscohost.com/login.aspx?direct=true&db=lxh&AN=5088138&site=eh ost live
155 Sochrin, S. (2004). "Learning to Teach in a New Medium: Adapting Library Instruction to a Videoconferencing Environment." In The Eleventh Off Campus Library Services Conference Proceedings: Scottsdale, Arizona, May 5 7, 2004 edited by Patrick B. Mahoney. Mount Pleasant, MI: Central Michigan University, 2004, 321 330. Reprinted in Journal of Library Administration 41(3/4), 429 442 Stein, L. L., & Lamb, J. M. (1998). Not just another BI: Faculty librarian collaboration to guide students through the research process. Research Strategies, 16 (1), 29 39. Retrieved from http://vnweb.hwwilsonweb.com/hww/jumpstart.jhtml?recid=0bc05f7a67b1790e70e3 97f5ebc28d347 2869cb64e954b912424200bd8f9b540909cd95ceada725e&fmt=C Stewart, V. D. (2007). Embedded in the blackboard jungle: The embedded librarian program at pulaski technical college. Arkansas Libraries, 64 (3), 29 32. Retrieved from http://search.ebscohost.com/login.aspx?direct=true&db=lxh&AN=27247367&site=e host live Sun, Y. (2007). Special feature: Evaluation methods in library and information activitie s. bibliometrics and its applications as a research evaluation tool. Journal of Information Science & Technology Association/Joho no Kagaku to Gijutsu, 57 (8), 372 377. Retrieved from http://search.ebscohost.com/login.aspx?direct=true&db=lxh&AN=31970429&site=e host live Tait, A. (2000). Planning student support for open and distance learning. Open Learning, 15 (3), 287 299. Retrieved from http://vnweb.hwwilsonweb.com/hww/jumpstart.jhtml?recid=0bc05f7a67b1790e 70e397f5ebc28d34f 09e2d096935499d2df4a47ea9fcc481468b56ad509003eb&fmt= C Tait, A., Ed., & Mills, R., Ed. (2003). Rethinking learner support in distance education: Change and continuity in an international context. RoutledgeFalmer studies in distance education. In (pp. 218) Tennant, M. R., Butson, L. C., Rezeau, M. E., Tucker, P. J., Boyle, M. E., & Clayton, G. (2001). Customizing for clients: Developing a library liaison program from need to plan. Bulletin of the Medical Library Association, 89 (1), 8. Tennant, M. R., Cataldo T. T., Sherwill Navarro, P., & Jesano, R. (2006). Evaluation of a liaison librarian program: Client and liaison perspectives. Journal of the Medical Library Association, 94 (4), 402. Tennant, M. R., & Miyamoto, M. M. (2002). The role of medical libraries in undergraduate education: A case study in genetics. Journal of the Medical Library Association, 90 (2), 181.
156 Thomas, J. (2000). Never enough: Graduate student use of journals -citation analysis of social work theses. Behavioral & Social Sciences Libraria n, 19 (1), 1. Retrieved from http://search.ebscohost.com/login.aspx?direct=true&db=lxh&AN=27706867&site=e host live Thompson, G. B. (2002). Inf ormation literacy accreditation mandates: What they mean for faculty and librarians. Library Trends, 51 (2), 218 241. Thorpe, M. (2002). Rethinking learner support: The challenge of collaborative online learning. Open Learning, 17 (2), 105 119. Retrieved from Find it @ UF http://vnweb.hwwilsonweb.com/hww/jumpstart.jhtml?recid=0bc05f7a67b179 0e70 e397f5ebc28d34f09e2d096935499df4810cc99afe698f7de675892af08500&fmt=C Thors, A., Dunki Jacobs, E., Engel, A. M., McDonough, S., & Welling, R. E. (2010). Does participation in graduate medical education contribute to improved patient outcomes as outlined by surgical care improvement project guidelines? Journal of Surgical Education, 67 (1), 9 13. doi:10.1016/j.jsurg.2009.12.002 Tod, A. M., Bond, B., Leonard, N., Gilsenan, I. J., & Palfreyman, S. (2007). Exploring the contribution of the clinical librarian to facilitating evidence based nursing. Journal of Clinical Nursing, 16 (4), 621 629. doi:10.1111/j.1365 2702.2006.01726.x Todd, R. J. (2008). A question of evidence. Knowledge Quest, 37 (2), 16 21. Retrieved from Full Text HTMLFull Text PDF HTML: http://vnweb.hwwilsonweb.com/hww/jumpstart.jhtml?recid=0bc05f7a67b1790e70e3 97f5ebc28d34246f 70f65fbf246687af4065c857e9b312ac139cc8d0d5bb&fmt=HPDF: http://vnweb.hwwilsonweb.com/hw w/jumpstart.jhtml?recid=0bc05f7a67b1790e70e3 97f5ebc28d34246f70f65fbf246687af4065c857e9b312ac139cc8d0d5bb&fmt=P Toth, M. (2005). Research and writing and theses -oh my! the journey of a collaboratively taught graduate research and writing course. The Refer ence Librarian, (89/90), 81 92. Retrieved from http://vnweb.hwwilsonweb.com/hww/jumpstart.jhtml?recid=0bc05f7a67b1790e70e3 97f5ebc28d34125f47fcd2b39c6674d1fd5d1a382906a58fb5b7444e37db&fmt=C Tun on J., & Brydges, B. (2006). A study on using rubrics and citation analysis to measure the quality of doctora l dissertation reference lists from traditional and nontraditional institutions. Journal of Library Administration, 45 (3/4), 459 481. Retrieved from The author of this article has chosen the following Web sites: http://www.carnegiefoundation.org/Classification/CIHE2000/defNotes/Definitio ns./htmhttp://www.nova.edu/rpga/ factbook/2005/http://www.edschools.org/pdf/Final 313.pdf;
157 Tunon, J., & Brydges, B. (2009). Expanded assessment study examining the citation patterns from traditional and nontraditional institutions and their effect upon the quality of doctoral dissertatio n reference lists. Journal of Library Administration, 49 (1), 137 159. doi:10.1080/01930820802312912 Turner, J. L., & McCaffrey, E. (2007). Library instruction for campus based and distance faculty at Regis. Colorado Libraries, 33 (3), 32 35. Van Vuren, A. J., & Henning, J. C. (2001). User education in a flexible learning environment an opportunity to stay relevant in the 21st century. South African Journal of Library & Information Science, 67 (2), 79. Wagner, E. D. (2008). Realizing the promises of mobil e learning. Journal of Computing in Higher Education, 20 (2), 4 14. Retrieved from http://dx.doi.org/10.1007/s12528 008 9008 x Wakimoto, D. K. (2010). Information literacy instruction assessment and improvement through evidence based practice: A mixed method study. Evidence Based Library and Information Practice, 5 (1), 82 92. Retrieved from Find it @ UF http://vnweb.hwwilsonweb.com/hww/jumpstart.jhtml?recid=0bc05f7a67b1790e7 0e397f5ebc28d34c9aabf518ccc3f8e102d5633b21ed22c3e88a0fcf8e40c5f&fmt=C Watson Boone, R. (2000). Aca demic librarians as practitioner researchers. The Journal of Academic Librarianship, 26 (2), 85 93. Retrieved from Full Text HTMLFull Text PDF HTML: http://vnweb.hwwilsonweb.com/hww/jumpstart.jhtml?recid=0bc05f7a67b179 0e70e397f5ebc28d3428a64628d64647f 410c2d94a59712cd517eb058b772930da&f mt Weaver, K. D., & Pier P. M. (2010). Embedded information literacy in the basic oral communication course: From conception through assessment. Public Services Quarterly, 6 (2), 259 270. doi:10.1080/15228959.2010.497455 Webb, P. (2006). Meeting the needs of distance education students: Creating an online only library instruction course. College & Research Libraries News, 67 (9), 548 550. Weightman, A. L., & Williamson, J. (2005). The value and impact of information prov ided through library services for patient care: A systematic review. Health Information & Libraries Journal, 22 (1), 4 25. Wheeler, S. (2002). Student perceptions of learning support in distance education. Quarterly Review of Distance Education, 3 (4), 419 429. Retrieved from http://vnweb.hwwilsonweb.com/hww/jumpstart.jhtml?recid=0bc05f7a 67b1790e 70e397f5ebc28d34f09e2d096935499d7593eb230b80d71373acdc6878dadff3&fmt= C
158 Whipple, E. C., McGowan, J. J., Dixon, B., E., & Zafar, A. (2009). The selection of high impact health informatics literature: A comparison of results between the content expert and the expert searcher. Journal of the Medical Library Association, 97 (3), 212 218. Winning, M., & Beverley, C. (2003). Clinical librarianship: A systematic review of the literature. Health Information & Libraries Journal, 20 10 21. Yin, R. K. (2009). Case study research: Design and methods Los Angeles, Calif: Sage Publications. York, A. C., & Vance, J. M. (2009). Taking library instruction into the online classroom: Best practices for embedded librarians. Journal of Library Administration, 49 (1), 197 209. doi:10.1080/01930820802312995 Zhang, L. (2006). Effectively incorporating instructional media into web based information literacy. The Electronic Library, 24 (3), 294 306.
159 BIOGRAPHICAL SKETCH Mary Edwards was born in 1979 in Gainesville, F lorida, and received her primary and secondary education in Alachua, F lorida She attended college at Mercer University (Macon, G eorgia ) where she received a Bachelor of Arts in h istory, with a minor concentration in b iology in 2001. After completing her undergraduate degree Mary returned to Florida for her graduate degree South Florida (2003) in library and information science After completing her library degree Mary took the position of Distance Education Librarian in the University of Florida Health Science Center Libraries where she worked her way up from an OPS (temporary) position to a Visiting position. Most recently in January 2011 she started her new position as an Assistant University Librarian in the University of Florida George A Smathers He alth Science Center Libraries. research interests in clude instructional design, online teaching and learning, distance education, program evaluation, and new literacies including media, digital and information. Shortly after beginning her career at the University of Florida Mary made the decision to return to pursue a doctoral degree She began taking courses in the College of Education in 2006 and entered the EdD program in c urriculum and i nst ruction (with a focus on educational technology) in 2008. Upon completion of the Doctor of Education degree Mary will return to her position in the HSC Libraries working with distance education, library instruction, and program evaluation. Mary has been m arried to her husband, Eddie Edwards for 10 years They have two daughters ; Allison Elizabeth, who is eight years old and Samantha Hope, who is three years old