Citation
Cataloging Open Online Learning Design Patterns for Computer Science Courses

Material Information

Title:
Cataloging Open Online Learning Design Patterns for Computer Science Courses
Creator:
Adnan, Nor Hafizah
Place of Publication:
[Gainesville, Fla.]
Florida
Publisher:
University of Florida
Publication Date:
Language:
english
Physical Description:
1 online resource (220 p.)

Thesis/Dissertation Information

Degree:
Doctorate ( Ph.D.)
Degree Grantor:
University of Florida
Degree Disciplines:
Curriculum and Instruction
Teaching and Learning
Committee Chair:
RITZHAUPT,ALBERT D
Committee Co-Chair:
ANTONENKO,PAVLO
Committee Members:
BEAL,CAROLE R
MANLEY,ANNE CORINNE

Subjects

Subjects / Keywords:
design -- moocs
Teaching and Learning -- Dissertations, Academic -- UF
Genre:
bibliography ( marcgt )
theses ( marcgt )
government publication (state, provincial, terriorial, dependent) ( marcgt )
born-digital ( sobekcm )
Electronic Thesis or Dissertation
Curriculum and Instruction thesis, Ph.D.

Notes

Abstract:
This study was conducted for the purpose of developing a catalog of open online learning design patterns for computer science courses, a template for documenting and reusing successful design solutions. The study also sought to explore different approaches that contribute to the rich description of the catalog of design patterns. This work started with the mining of design patterns from Massive Open Online Courses. Design patterns are effective solutions to recurring problems that are useful for guiding design decisions. Reusability is the key element of design patterns, where the solutions can be used in many different contexts. Merrill's First Principles of Instruction served as a theoretical framework in this study. First principles prescribe a task-centered approach that integrates the solving of problems encountered in real-world situations with a direct instruction of problem components. The fifteen design patterns presented in this study can be used in conjunction with other few principles for teaching materials and learning activities, such as the collaboration, interaction, motivation, and navigation in designing a quality open online learning for computer science courses. Besides, this study also proposed a template to the instructional design community on how to effectively document and communicate design patterns in open education context. Designers can use this template to express their design expertise to other instructional design professionals and also make use of design patterns in practice. ( en )
General Note:
In the series University of Florida Digital Collections.
General Note:
Includes vita.
Bibliography:
Includes bibliographical references.
Source of Description:
Description based on online resource; title from PDF title page.
Source of Description:
This bibliographic record is available under the Creative Commons CC0 public domain dedication. The University of Florida Libraries, as creator of this bibliographic record, has waived all rights to it worldwide under copyright law, including all related and neighboring rights, to the extent allowed by law.
Thesis:
Thesis (Ph.D.)--University of Florida, 2017.
Local:
Adviser: RITZHAUPT,ALBERT D.
Local:
Co-adviser: ANTONENKO,PAVLO.
Statement of Responsibility:
by Nor Hafizah Adnan.

Record Information

Source Institution:
UFRGP
Rights Management:
Applicable rights reserved.
Classification:
LD1780 2017 ( lcc )

Downloads

This item has the following downloads:


Full Text

PAGE 1

1 CATALOGING OPEN ONLINE LEARNING DESIGN PATTERNS FOR COMPUTER SCIENCE COURSES By NOR HAFIZAH ADNAN A DISSERTATION PRESENTED TO THE GRADUATE SCHOOL OF THE UNIVERSITY OF FLORIDA IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY UNIVERSITY OF FLORIDA 2017

PAGE 2

2 2017 Nor Hafizah Adnan

PAGE 3

3 To my Mom and Dad

PAGE 4

4 ACKNOWLEDGMENTS I would like to acknowledge the following individuals for all of their assistance and understanding along the way. I would like to thank Albert Ritzhaupt for his availability, advice, and understanding at every step. I also thank the rest of my dissertation committee, Pasha Antonenko, Carole Beal, and Corinne Manley for their feedback on my proposal. I thank my peers in the Educational Technology department: Matthew Wilson, Robert Davis, Jiahui Wang, Shil pa Sahay, and Mehmet Celepkolu. I would also like to th ank my participants, especially the interviewees, who spent a considerable amount of time to give thoughtful responses and who were very encouraging about the study. A special thanks to those who participated in member checking. Finally, I would like to th ank my family for all of their support over the past years and for supporting me throughout my education. Last but not least, I would like to thank the Ministry of Higher Education of Malaysia and National University of Malaysia for sponsoring my PhD.

PAGE 5

5 TABLE OF CONTENTS page ACKNOWLEDGMENTS ................................ ................................ ................................ .. 4 LIST OF TABLES ................................ ................................ ................................ .......... 12 LIST OF FIGURES ................................ ................................ ................................ ........ 14 LIST OF ABBREVIATIONS ................................ ................................ ........................... 17 ABSTRACT ................................ ................................ ................................ ................... 18 CHAPTER 1 INTRODUCTION ................................ ................................ ................................ .... 20 Background ................................ ................................ ................................ ............. 20 Context ................................ ................................ ................................ ................... 27 Professional Background ................................ ................................ .................. 27 Current Challenges ................................ ................................ .......................... 28 Future Difficulty ................................ ................................ ................................ 29 Rationale for the Computer Science MOOCs ................................ ................... 29 Justification on the Reinvent New Solutions ................................ ..................... 29 Problem Statement ................................ ................................ ................................ 30 Research Questions ................................ ................................ ............................... 35 Research Design ................................ ................................ ................................ .... 35 Significance of the Study ................................ ................................ ........................ 37 Definition of Terms ................................ ................................ ................................ .. 38 Organization of Study ................................ ................................ ............................. 39 2 LITERATURE REVIEW ................................ ................................ .......................... 40 Open Education ................................ ................................ ................................ ...... 40 Open Online Learning ................................ ................................ ............................. 41 Instructional Designer ................................ ................................ ............................. 43 Subject Matter Experts ................................ ................................ ............................ 45 Epistemology of Design ................................ ................................ .......................... 46 Design Knowledge ................................ ................................ ................................ .. 49 Knowledge Management ................................ ................................ ........................ 51 Explicit Knowledge and Tacit Knowledge ................................ ............................... 53 What are Design Guidelines, Design Patterns, and Design Principles? ................. 54 Research on Design Patterns ................................ ................................ ................. 56 Design Pattern Usage ................................ ................................ ............................. 58 Design Patterns in Architecture ................................ ................................ ........ 59 Design Patterns in Software Engineering ................................ ......................... 59 Design Patterns in Education ................................ ................................ ........... 61

PAGE 6

6 Design patterns for human computer interaction ................................ ....... 62 Design patterns for collaborative learning ................................ .................. 63 Design patterns for cognitive learning ................................ ........................ 63 Design patterns for mobile learning user interface ................................ ..... 64 Design patterns for open learning ................................ .............................. 65 Methodology of Design Patterns ................................ ................................ ............. 66 Schema Theory ................................ ................................ ................................ ...... 68 Theoretical Framew ork ................................ ................................ ........................... 72 Summary ................................ ................................ ................................ ................ 74 3 METHODOLOGY ................................ ................................ ................................ ... 76 Research Design ................................ ................................ ................................ .... 76 Phase 1: Design Pattern Mining ................................ ................................ ............. 76 Self Observation ................................ ................................ ............................... 77 Subjectivity statement ................................ ................................ ................ 78 Expert Interview ................................ ................................ ................................ 80 Participants ................................ ................................ ................................ 80 Interview questions ................................ ................................ .................... 81 Procedures ................................ ................................ ................................ 82 Validity ................................ ................................ ................................ ....... 82 Ethical considerations ................................ ................................ ................ 83 Artifactual Study ................................ ................................ ............................... 84 Literature Review ................................ ................................ ............................. 87 Adaptation of Existing Published Patterns ................................ ........................ 88 Phase 2: Design Pattern Writing ................................ ................................ ............. 90 Data Analysis ................................ ................................ ................................ .......... 94 Summary ................................ ................................ ................................ ................ 96 4 RESULTS ................................ ................................ ................................ ............... 97 Background ................................ ................................ ................................ ............. 97 Study Findings ................................ ................................ ................................ ........ 97 Use of Online Affordances ................................ ................................ ................ 98 Conten t and Course Material ................................ ................................ .......... 102 Instructional Strategy and Learning Outcomes ................................ ............... 105 Knowledge Activation ................................ ................................ ..................... 108 Transfer of Learning ................................ ................................ ....................... 114 Design Patterns as the link between Theory and Practice ................................ .... 116 Catalog of Design Patterns ................................ ................................ ................... 117 Show Task Design Pattern ................................ ................................ ................... 118 Pattern Name ................................ ................................ ................................ 118 Also Known As ................................ ................................ ............................... 118 Category ................................ ................................ ................................ ......... 118 Context ................................ ................................ ................................ ........... 118 Problem ................................ ................................ ................................ .......... 118 Forces ................................ ................................ ................................ ............ 119

PAGE 7

7 Solution ................................ ................................ ................................ .......... 119 Consequences ................................ ................................ ............................... 119 Learning Scenario ................................ ................................ .......................... 120 Examples ................................ ................................ ................................ ........ 120 Related Patterns ................................ ................................ ............................. 121 Supporting Research ................................ ................................ ...................... 12 1 Task Level Design Pattern ................................ ................................ .................... 122 Pattern Name ................................ ................................ ................................ 122 Also Known As ................................ ................................ ............................... 122 Category ................................ ................................ ................................ ......... 122 Context ................................ ................................ ................................ ........... 122 Problem ................................ ................................ ................................ .......... 122 Forces ................................ ................................ ................................ ............ 123 Solution ................................ ................................ ................................ .......... 123 Consequences ................................ ................................ ............................... 123 Learning Scenario ................................ ................................ .......................... 123 Examples ................................ ................................ ................................ ........ 124 Related Patterns ................................ ................................ ............................. 125 Supporting Research ................................ ................................ ...................... 125 Problem Progression Design Pattern ................................ ................................ .... 125 Pattern Name ................................ ................................ ................................ 125 Also Known As ................................ ................................ ............................... 125 Category ................................ ................................ ................................ ......... 126 Context ................................ ................................ ................................ ........... 126 Problem ................................ ................................ ................................ .......... 126 Forces ................................ ................................ ................................ ............ 126 Solution ................................ ................................ ................................ .......... 126 Consequences ................................ ................................ ............................... 128 Learning Scenario ................................ ................................ .......................... 128 Examples ................................ ................................ ................................ ........ 128 Related Patterns ................................ ................................ ............................. 130 Supporting Research ................................ ................................ ...................... 130 Previous Exp erience Design Pattern ................................ ................................ .... 130 Pattern Name ................................ ................................ ................................ 130 Also Known As ................................ ................................ ............................... 131 Category ................................ ................................ ................................ ......... 131 Context ................................ ................................ ................................ ........... 131 Problem ................................ ................................ ................................ .......... 131 Forces ................................ ................................ ................................ ............ 131 Solution ................................ ................................ ................................ .......... 131 Consequences ................................ ................................ ............................... 132 Learning Scenario ................................ ................................ .......................... 132 Examples ................................ ................................ ................................ ........ 132 Related Patterns ................................ ................................ ............................. 134 Supporting Res earch ................................ ................................ ...................... 134 New Experience Design Pattern ................................ ................................ ........... 135

PAGE 8

8 Pattern Name ................................ ................................ ................................ 135 Also Known As ................................ ................................ ............................... 135 Category ................................ ................................ ................................ ......... 135 Context ................................ ................................ ................................ ........... 135 Problem ................................ ................................ ................................ .......... 135 Forces ................................ ................................ ................................ ............ 136 Solution ................................ ................................ ................................ .......... 136 Consequences ................................ ................................ ............................... 136 Learning Scenario ................................ ................................ .......................... 136 Examples ................................ ................................ ................................ ........ 136 Related Patterns ................................ ................................ ............................. 137 Supporting Research ................................ ................................ ...................... 137 Structure Design Pattern ................................ ................................ ....................... 138 Pattern Name ................................ ................................ ................................ 138 Also Known As ................................ ................................ ............................... 138 Category ................................ ................................ ................................ ......... 138 Context ................................ ................................ ................................ ........... 138 Problem ................................ ................................ ................................ .......... 138 Forces ................................ ................................ ................................ ............ 139 Solution ................................ ................................ ................................ .......... 139 Consequences ................................ ................................ ............................... 139 Learning Scenario ................................ ................................ .......................... 139 Examples ................................ ................................ ................................ ........ 139 Related Patterns ................................ ................................ ............................. 141 Supporting Research ................................ ................................ ...................... 141 Demonstration Consistency Design Pattern ................................ ......................... 141 Pattern Name ................................ ................................ ................................ 141 Also Known As ................................ ................................ ............................... 141 Category ................................ ................................ ................................ ......... 141 Context ................................ ................................ ................................ ........... 141 Problem ................................ ................................ ................................ .......... 141 Forces ................................ ................................ ................................ ............ 142 Solution ................................ ................................ ................................ .......... 142 Consequences ................................ ................................ ............................... 142 Learning Scenario ................................ ................................ .......................... 142 Examples ................................ ................................ ................................ ........ 143 Related Patterns ................................ ................................ ............................. 144 Supporting Research ................................ ................................ ...................... 144 Learner Guid ance Design Pattern ................................ ................................ ........ 144 Pattern Name ................................ ................................ ................................ 144 Also Known As ................................ ................................ ............................... 145 Category ................................ ................................ ................................ ......... 145 Context ................................ ................................ ................................ ........... 145 Problem ................................ ................................ ................................ .......... 145 Forces ................................ ................................ ................................ ............ 145 Solution ................................ ................................ ................................ .......... 145

PAGE 9

9 Consequences ................................ ................................ ............................... 146 Learning Scenario ................................ ................................ .......................... 146 Examples ................................ ................................ ................................ ........ 146 Related Patterns ................................ ................................ ............................. 148 Supporting Research ................................ ................................ ...................... 148 Relevant Media Design Pattern ................................ ................................ ............ 148 Pattern Name ................................ ................................ ................................ 148 Also Known As ................................ ................................ ............................... 149 Category ................................ ................................ ................................ ......... 149 Context ................................ ................................ ................................ ........... 149 Problem ................................ ................................ ................................ .......... 149 Forces ................................ ................................ ................................ ............ 149 Solution ................................ ................................ ................................ .......... 149 Consequences ................................ ................................ ............................... 150 Learning Scenario ................................ ................................ .......................... 150 Examples ................................ ................................ ................................ ........ 150 Related Patterns ................................ ................................ ............................. 151 Supporting Research ................................ ................................ ...................... 151 Practice Consistency Design Pattern ................................ ................................ .... 151 Pattern Name ................................ ................................ ................................ 151 Also Known As ................................ ................................ ............................... 151 Category ................................ ................................ ................................ ......... 151 Context ................................ ................................ ................................ ........... 151 Problem ................................ ................................ ................................ .......... 152 Forces ................................ ................................ ................................ ............ 152 Solution ................................ ................................ ................................ .......... 152 Solution ................................ ................................ ................................ .......... 152 Consequences ................................ ................................ ............................... 153 Learning Scenario ................................ ................................ .......................... 153 Examples ................................ ................................ ................................ ........ 153 Related Patterns ................................ ................................ ............................. 155 Supporting Research ................................ ................................ ...................... 155 Diminishing Coaching Design Pattern ................................ ................................ .. 155 Pattern Name ................................ ................................ ................................ 155 Also Known As ................................ ................................ ............................... 155 Category ................................ ................................ ................................ ......... 155 Context ................................ ................................ ................................ ........... 155 Problem ................................ ................................ ................................ .......... 156 Forces ................................ ................................ ................................ ............ 156 Solution ................................ ................................ ................................ .......... 156 Consequences ................................ ................................ ............................... 157 Learning Scenario ................................ ................................ .......................... 157 Examples ................................ ................................ ................................ ........ 157 Related Patterns ................................ ................................ ............................. 158 Supporting Res earch ................................ ................................ ...................... 159 Varied Problems Design Pattern ................................ ................................ ........... 159

PAGE 10

10 Pattern Name ................................ ................................ ................................ 159 Also Known As ................................ ................................ ............................... 159 Category ................................ ................................ ................................ ......... 159 Context ................................ ................................ ................................ ........... 159 Problem ................................ ................................ ................................ .......... 160 Forces ................................ ................................ ................................ ............ 160 Solution ................................ ................................ ................................ .......... 160 Consequences ................................ ................................ ............................... 160 Learning Scenario ................................ ................................ .......................... 160 Examples ................................ ................................ ................................ ........ 161 Related Patterns ................................ ................................ ............................. 162 Supporting Research ................................ ................................ ...................... 162 Watch Me Design Pattern ................................ ................................ ..................... 163 Pattern Name ................................ ................................ ................................ 163 Also Known As ................................ ................................ ............................... 163 Category ................................ ................................ ................................ ......... 163 Context ................................ ................................ ................................ ........... 163 Problem ................................ ................................ ................................ .......... 163 Forces ................................ ................................ ................................ ............ 163 Solution ................................ ................................ ................................ .......... 163 Consequences ................................ ................................ ............................... 164 Learning Scenario ................................ ................................ .......................... 164 Examples ................................ ................................ ................................ ........ 164 Related Patterns ................................ ................................ ............................. 165 Supporting Research ................................ ................................ ...................... 165 Reflection Design Pattern ................................ ................................ ..................... 165 Patter n Name ................................ ................................ ................................ 165 Also Known As ................................ ................................ ............................... 166 Category ................................ ................................ ................................ ......... 166 Context ................................ ................................ ................................ ........... 166 Problem ................................ ................................ ................................ .......... 166 Forces ................................ ................................ ................................ ............ 166 Solution ................................ ................................ ................................ .......... 166 Consequences ................................ ................................ ............................... 167 Learning Scenario ................................ ................................ .......................... 167 Examples ................................ ................................ ................................ ........ 167 Related Patterns ................................ ................................ ............................. 167 Supporting Research ................................ ................................ ...................... 168 Creation Design Pattern ................................ ................................ ........................ 1 68 Pattern Name ................................ ................................ ................................ 168 Also Known As ................................ ................................ ............................... 168 Category ................................ ................................ ................................ ......... 168 Context ................................ ................................ ................................ ........... 168 Problem ................................ ................................ ................................ .......... 168 Forces ................................ ................................ ................................ ............ 169 Solution ................................ ................................ ................................ .......... 169

PAGE 11

11 Consequences ................................ ................................ ............................... 169 Learning Scenario ................................ ................................ .......................... 169 Examples ................................ ................................ ................................ ........ 169 Related Patterns ................................ ................................ ............................. 170 Supporting Res earch ................................ ................................ ...................... 170 Structure of Open Online Learning Design Patterns ................................ ............. 170 Recommendation for Practice ................................ ................................ ........ 171 Summary ................................ ................................ ................................ .............. 173 5 DISCUSSION ................................ ................................ ................................ ....... 189 Discussion of Findings ................................ ................................ .......................... 190 Use of Online Affordances ................................ ................................ .............. 190 Learne r to learner interaction ................................ ................................ ... 190 Learner to content interaction ................................ ................................ .. 191 Learner to instructor interaction ................................ ............................... 193 Content and Course Material ................................ ................................ .......... 194 Instructional Strategy and Learning Outcomes ................................ ............... 195 Knowledge Activation ................................ ................................ ..................... 196 Transfer of Learning ................................ ................................ ....................... 197 Limitations of the Study ................................ ................................ ......................... 198 Recommendations for Future Research ................................ ............................... 201 Conclusion ................................ ................................ ................................ ............ 202 APPENDIX A RESEARCH PARTICIPATION REQUEST ................................ ........................... 204 B PRE SURVEY QUESTIONNAIRE ................................ ................................ ........ 205 C IN TERVIEW QUESTIONS ................................ ................................ .................... 206 D INTERVIEW TRANSCRIPT ................................ ................................ .................. 208 LIST OF REFERENCES ................................ ................................ ............................. 211 BIOGRAPHICAL SKETCH ................................ ................................ .......................... 220

PAGE 12

12 LIST OF TABLES Table page 2 1 Although the design science research guidelines ................................ ............... 48 3 1 Design pattern mining approaches ................................ ................................ ..... 77 3 2 MOOC providers for the design pattern mining ................................ .................. 85 3 3 First Principles of Instruction ................................ ................................ ............... 88 3 4 Adaptation of design patterns from other domains ................................ ............. 89 4 1 Transcribed interview with a participant regarding the interaction among learners part 1 ................................ ................................ ................................ .... 98 4 2 Transcribed interview with a par ticipant regarding the interaction among learners part 2 ................................ ................................ ................................ .... 99 4 3 Transcribed interviews with participants regarding the interaction o f learners with contents ................................ ................................ ................................ ..... 100 4 4 Transcribed interview with a participant regarding the interaction between learner and instructor ................................ ................................ ........................ 102 4 5 Transcribed interview with a participant regarding course effectiveness .......... 103 4 6 Transcribed interviews with participants regarding learner evaluation part 1 ... 104 4 7 Transcribed interviews with participants regarding learner evaluation part 2 ... 105 4 8 Transcribed interviews with participants regarding instructional strategy and learning outcomes part 1 ................................ ................................ .................. 107 4 9 Transcribed interviews with participants regarding instructional strategy and learning outcomes part 2 ................................ ................................ .................. 108 4 10 Transcribed interviews with participants regarding instructional strategy part 1 ................................ ................................ ................................ ....................... 109 4 11 Transcribed interviews with participants regarding instructional strategy part 2 ................................ ................................ ................................ ....................... 110 4 12 Transcribed interviews with participants regarding course completion part 1 ... 111 4 13 Transcribed interviews with participants regard ing course completion part 2 ... 112

PAGE 13

13 4 14 Transcribed interviews with participants regarding motivation .......................... 113 4 15 Transcribed interviews with participants related to transfer of learning ............ 115 4 16 Instructional sequence for teaching components of the whole task .................. 127 4 17 Suggestion for practice ................................ ................................ ..................... 153

PAGE 14

14 LIST OF FIGURES Figure page 2 1 Four essential elements of a design pattern ................................ ....................... 57 2 2 Describing design patterns ................................ ................................ ................. 60 2 3 Linking schema theory to the design pattern concept ................................ ......... 69 2 4 First principles of instruction ................................ ................................ ............... 72 3 1 The most popular topics of computer science MOOCs ................................ ...... 86 3 2 Design pattern template 1 ................................ ................................ .................. 91 3 3 Design pattern template 2 ................................ ................................ .................. 92 3 4 Design pattern template 3 ................................ ................................ .................. 93 3 5 Qualitative content analysis process ................................ ................................ .. 95 4 1 Theoretical framework and classification of design patterns ............................ 117 4 2 Elements of the visual Educational Modeling Language ................................ .. 118 4 3 An instructor video introduces a r eal world problem to the learners ................. 120 4 4 An instructor video shows learners the complete whole task that they will learn to do ................................ ................................ ................................ ........ 121 4 5 An instructor video demonstrates what learners will achieve by the end of the course ................................ ................................ ................................ ............... 121 4 6 Instructional strategies for task level ................................ ................................ 124 4 7 Introduction of the learning unit ................................ ................................ ........ 125 4 8 Quiz with a worked example approach ................................ ............................. 129 4 9 Quiz with a fading approach ................................ ................................ ............. 129 4 10 Quiz with a self explanation approach ................................ .............................. 130 4 11 Messages from the instructor for the first quiz ................................ .................. 133 4 12 The first quiz for the Intro to Computer Science course ................................ .... 133 4 13 ................. 134

PAGE 15

15 4 14 Pretest to assess lea ................................ ..................... 134 4 15 An instructor video presents the introductory materials ................................ .... 137 4 16 An instructor video demonstrates a relevant example ................................ ...... 137 4 17 Advance organizer ................................ ................................ ............................ 140 4 18 Motivational themes ................................ ................................ .......................... 140 4 19 An example to show the abstract idea of objects ................................ .............. 143 4 20 A demonstration or tutorial video to create objects ................................ ........... 144 4 21 Java interface example ................................ ................................ ..................... 147 4 22 Demonstration video ................................ ................................ ......................... 147 4 23 R eflection task ................................ ................................ ................................ .. 147 4 24 Peer interaction ................................ ................................ ................................ 148 4 25 Graphic with audio narration ................................ ................................ ............. 150 4 26 Procedure and produce output practice ................................ ............................ 154 4 27 Identify and label practice ................................ ................................ ................. 154 4 28 Worked example ................................ ................................ ............................... 158 4 29 Automat ed feedback ................................ ................................ ......................... 158 4 30 First problem in a sequence ................................ ................................ ............. 161 4 31 Second problem in a sequence ................................ ................................ ........ 162 4 32 Demonstrate a task or solve a problem ................................ ............................ 164 4 33 Receive feedback ................................ ................................ ............................. 165 4 34 Reflective thinking task ................................ ................................ ..................... 167 4 35 Creating task ................................ ................................ ................................ .... 170 4 36 Open online learning d esign pattern relationships ................................ ............ 172 4 37 The metamodel of the Show Task learning scenario ................................ ........ 174 4 38 The metamodel of the Task Level learning scenario ................................ ........ 175

PAGE 16

16 4 39 The metamodel of the Problem Progression learning scenario ........................ 176 4 40 The metamodel of the Previous Experience learning scenario ......................... 177 4 41 The metamodel of the New Experience learning scenario ............................... 178 4 42 The metamodel of the Structure learning scenario ................................ ........... 179 4 43 The metamodel of the Demonstration Consistency learning scenario .............. 180 4 44 The metamodel of the Learner Gu idance learning scenario ............................. 181 4 45 The metamodel of the Relevant Media learning scenario ................................ 182 4 46 The metamodel of the Practice Consistency learning scenario ........................ 183 4 47 The metamodel of the Diminishing Coaching learning scenario ....................... 184 4 48 The metamodel of the Varied Problems learning scenario ............................... 185 4 49 The metamodel of the Watch Me learning scenario ................................ ......... 186 4 50 The metamodel of the Reflection learning scenario ................................ .......... 187 4 51 The metamodel of the Creation learning scenario ................................ ............ 188

PAGE 17

17 LIST OF ABBREVIATIONS MOOC Massive Open Online Course OER Open Educational Resources

PAGE 18

18 Abstract of Dissertation Presented to the Graduate School of the University of Florida in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy CATALOGING OPEN ONLINE LEARNING DESIGN PATTERNS FOR COMPUTER SCIENCE COURSES By Nor Hafizah Adnan December 2017 Chair: Albert Ritzhaupt Major: Curriculum and Instruction This study was conducted for the purpose of developing a catalog of open online learning design patterns for computer science courses a template for documenting and reusing successful design solutions. The study also sought to explore different approaches that contribute to the rich description of the catalog of design patterns. This work started with the mining of design patterns from Massive Open Online Courses (MOOCs). Design patterns are effective solutions to recurring problems that are usefu l for guiding design decisions. Reusability is the key element of design patterns, where the solutions can be used in many different contexts. study. First principles prescribe a task centered approach t hat integrates the solving of problems encountered in real world situations with a direct instruction of problem components. The fifteen design patterns presented in this study can be used in conjunction with other few principles for teaching materials and learning activities, such as the collaboration, interaction, motivation, and navigation in designing a quality open online learning for computer science courses. Besides, this study also proposed a template to the instructional design community on how to effectively document and

PAGE 19

19 communicate design patterns in an open education context. Designers can use this template to express their design expertise to other instructional design professionals and also make use of design patterns in practice.

PAGE 20

20 CHAPTER 1 I NTRODUCTION This study was conducted for the purpose of developing a catalog of open online learning design patterns for computer science courses, a template for documenting and reusing successful design solutions. The study also sought to explore differen t approaches that contribute to the rich description of the catalog of design patterns. This work started with the mining of design patterns from Massive Open Online Courses (MOOCs). Design patterns are effective solutions to recurring problems that are us eful for guiding design decisions. Reusability is the key element of design patterns, where the solutions can be used in many different contexts. Open online learning is a revolution in higher education that provides free online courses to anyone with acc ess to the Internet (Wiley, 2015). Open online courses are typically designed around a self guided format without personalized, one to one support from instructors that assumes learners can regulate their own learning (Milligan & Littlejohn, 2016). Due to open access and massive target audience, open online courses require different learning designs from those online courses that have small student numbers (Wiley, 2015). Background Design patterns in instructional design are general reusable solutions to a commonly occurring problem within a given context. A design pattern is a description or template for how to solve a problem that can be used in many different situations. Design patterns are formalized best practices that the instructional designer can us e to solve common problems when designing an instructional system. One example of a commonly used documentation format is the one used by Erich Gamma, Richard Helm,

PAGE 21

21 their famous book Design Patterns Elements of Reusable Object Oriented Software Christopher Alexander (1979) introduced the concept of design patterns to facilitate rigorous discourse in architecture, building, and planning. Alexander in his seminal book The Timeless Way of Building (1979) defined a pattern as an instruction that shows how the arrangement of elements can be used repeatedly to resolve the problem, provided the context is relevant. Design links theory and practice, connecting scientific activ ities and creativity to deal with the uncertainty and complexity of open ended, ill structured problems as found in the creation of learning technologies. Design knowledge differs from other types of knowledge. Design knowledge is a meta knowledge that lea ns more toward the Making tacit design knowledge explicit is extremely difficult, and conveying such knowledge is particularly challenging. According to Hoadley and Cox (2009 p. 19): to get a better grip on what experienced designers know, in whatever sense of the word come up with effective, reproducible ways of getting novices to a similar stage, such that they understand the general ideas that all expert designers sha re, and develop their own unique ways of understa nding and applying those ideas. Experts can be considered unique since it is not easy to explain exactly what is on their minds. It is hard to determine how things have been done previously and why they have been done in some certain way, as well as how to reuse those solutions. The fiel d of knowledge engineering addresses exactly this issue (Gomez Perez, Fernndez Lpez, & Corcho, 2004). Building knowledge based systems could be done by assembling reusable components rather than constructing them from scratch, enabling system developers to reuse and share knowledge components, such as declarative

PAGE 22

22 knowledge, problem solving approaches, and reasoning activities using a common vocabulary among systems (Gomez Perez et al., 2004). Thus, system developers could concentrate on the creation of sp ecialized knowledge of their systems. In their book, Gomez Perez et al. (2004) focused on the ontology learning methods to reduce the effort of the knowledge acquisition process, preserve the original ontologies, and evaluate the ontology content. In the s tudy conducted by Chi, Feltovich, and Glaser (1981), the problem schemata of experts were clearly different from those of novices. In this study, experts categorized the problems based on the abstract physics principles, while novices characterized the pro blems according to the particular features (Chi et al., 1981). The categorization of problems based on the relevant principles will prompt the activation of problem schemata or particular knowledge structures that will determine which efficient solution to be used. The evidence from research on expert novice comparisons indicated that the experts relied mostly on the problem structures for determining the similarity of solutions, whereas the novices depended mainly on the surface structures (Hardiman, Duf resne, & Mestre, 1989). On the other hand, novices who took advantage of the principles had a tendency to categorize problems in the same way experts categorized them, and scored higher in problem solving. Due to these reasons, Hardiman et al. (1989) sugge sted to structure the information presented to novices in a way that assists them in organizing knowledge by principles, which in turn can lead to better understanding.

PAGE 23

23 Groen and Patel (1985) carried out research in medical education and found experts used strong methods to address a specific type of problem, depending on a detailed and structured knowledge base. A main difference between experts and novices in solving problems was that the former tended to employ a form of forward reasoning of hypothetico deductive thinking, while the latter engaged in backward reasoning (Groen & Patel, 1985). The forward reasoning approach involves applying a set of if then production rules to a problem, generating a diagnosis without a hypothesis. In order to apply if the n rules, the clinician drew from his structured knowledge base. As a result, any comparison of expert and novice problem solving models should consider Overall, the studies of experts conclude tha t experts predominantly recognize the types of problems and already know the solution methods. Thus, there is a growing demand for documenting and communicating design knowledge in a systematic and structured manner. In the design literature, several metho ds are commonly used for capturing and sharing design knowledge, and one of them is through patterns. Design patterns received considerable attention over the last three decades in various fields as a means for disseminating design knowledge to novice de signers. Design patterns are reusable solutions to recurring problems that can be used in different contexts. Design patterns were first applied to the field of architecture in order to provide solutions to common problems encountered in the modern archite ctural design, such as the communities and neighborhood design (Alexander, 1979). The objective was to present this knowledge in a comprehensible and consistent form that could be reused by architects. Alexander (1979) proposed design patterns in a narrati ve

PAGE 24

24 form, providing textual descriptions and pictures that illustrate a design problem and its part rule, which expresses a relation between a certain context, a problem, and The Timeless Way of Building that was published in 1979 consists of 253 architecture design patterns. Pattern documentation typically contains the causes of problems in a particular situation, and how the essential eleme nts of the pattern relate to each other in order to provide the solution. For instance, Alexander et al. (1977) through the pattern of view of thermal considerations, eithe r to direct light all around the room or avoid the room overheats on summer afternoons. Instead of suggesting the designer to put how many windows in the room, a pattern would guide the designer in the decision making process by proposing a set of solution s for the particular situation. The purpose of the pattern is to guide a designer, rather than prescribe. Thus, Alexander et al. (1977) suggested enough windows to allow the natural light to come in and keep the room west axi s sets up a building to keep the heat in during winter, and to keep the heat out during the summer. This makes buildings more Place the most important rooms along the south edge of the building, and spread the building out along the east west axis. Fine tune the arrangement so that the proper rooms are exposed to the south east and the south west sun. For example: give the common area a full southern exposure, bedrooms south east, porch south west. For most climates, this means the shape of the building is elongated east west. ( Alexander et al. 1977, p. 617) Alexander et al. (1977) considered this suggestion as the best solution because daylighting creates a positive home environ ment and makes the home a more enjoyable

PAGE 25

25 place. Also, Alexander et al. (1977) described the applicability of the pattern due to a broad range of housing types. The situations in which a pattern can be applied is known detache d, semi detached, townhouses, and apartments. In another example, Alexander et al. (1977) considered both bus stops and waiting area ion of this pattern was to create a positive waiting (for provide the waiting areas with newspaper, coffee, pool tables, and something that draws people in who are not atmosphere will come naturally if the waiting area provides some places that are quiet, protected, and do not draw out the an The greatest impact of design patterns can be seen within the field of software engineering. According to the data from the Scopus publication database, design pattern was among the topics that had the h ighest number of papers published in the area of software engineering (Garousi & Mntyl 2016). Expert designers generally do not solve each problem from the first principles. When they found some solutions that worked really well, they began to use them repeatedly. Such experience makes them experts and could be documented as design patterns. Gamma, Helms, Johnson, and Vlissides (1995) introduced a template for describing and organizing design patterns in object oriented software design using a consistent format that includes four main elements, namely, pattern name, problem, solution, and consequences. Further, Gamma et al. (1995) proposed a catalog of object oriented software design patterns

PAGE 26

26 that contains 23 patterns. Similarly, Chung, Hong, Lin, Prabak er, Landay, and Liu (2004) described that design patterns are different from other design formats, such as guidelines and heuristics in capturing design knowledge. The difference between design patterns and guidelines are: (1) D esign patterns provide abstract solutions to the more general problems, rather than presenting specific solutions ( Hoadley & Cox, 2009), (2) Design patterns provide examples of actual designs, helping designers produce new solutions (Chung et al., 2004), and (3) Design patterns are interrelated with others in a hierarchy structure, allowing designers to address both high level and low level problems ( Gamma et al., 1995) The concept of design patterns can be applied not only to support architects and software engineers, but als o can be used to guide instructional designers in designing online courses. Design patterns have been used in the field of education as a way to capture, share and reuse effective learning solutions, as well as to maintain an up to date record of best prac tices. Retalis, Georgiakakis, and Dimitriadis (2006) believed that design patterns do not describe a concrete design or specific design solutions, but they are just a template for how to solve complex problems which applies to different, but related situat ions, providing an abstract description of a design problem and the way to solve it. Kolfschoten, Lukosch, Verbraeck, Valentin, and de Vreede (2010) found that design patterns are a practical way to transfer knowledge, presenting ready made solutions to d esigners. The study by Kolfschoten et al. (2010) also discovered that design patterns do not only increase understanding of the design process among novices, but also increase the efficiency, flexibility, and reusability of the design effort.

PAGE 27

27 Kohls and Utt echt (2009) conducted a case study on the mining, writing, and application of patterns for interactive educational graphics. Further, Kohls and Uttecht (2009) claimed that the most significant attribute of design patterns is they document actual design ins tead of abstract solutions. Design patterns have been useful in many fields, ranging from architecture and software engineering to education and others. Indeed, there was some work on them in online course development, but there is a need for a comprehens ive catalogue that can be used and shared within the instructional design community. Context The purpose of this study was to develop a comprehensive catalog of open online learning design patterns for computer science courses as an effective means of capt uring and sharing successful solutions for recurring problems. The other goal of this study was to provide a template that designers can use to express their design expertise to other instructional design professionals. This solution should provide the ins tructional design community with a necessary template on how to effectively document and communicate design patterns in the field of open online learning. The use of design patterns is significant as it allows the reusability of expert knowledge within ope n online education. As a scholar, I hope to answer certain questions related to the adoption of the design pattern concept in this study. Professional Background My current role is that of a faculty member at a public research university in Malaysia that h as recently implemented and integrated massive open online courses with on campus courses. The first stage of the initiative began with compulsory courses and learners from public universities in Malaysia participated in those courses. Coming

PAGE 28

28 from a comput er science background, I am always interested in the design and development of technology enhanced learning environments. I constantly seek to learn about new approaches to create effective and engaging online learning environments towards meaningful learn ing among learners. Through the design and development of the catalog of design patterns, I have had the opportunity to speak with expert instructional designers who have designed, developed, and delivered open online learning for computer science courses. Also, I have explored different approaches that contributed to the rich description of the catalog of design patterns, from the self observation, to the analysis of the functionality of computer science MOOCs, to the review of the literature on pedagogica l strategies, and the study of existing published patterns in other related areas. Current Challenges One of the most emerging challenges as a novice instructional designer is to effectively design a large number of high quality courses. In order to creat e dynamic and innovative online learning environments, it is critical to allow the reuse of expert knowledge that a novice instructional designer can adopt when they design, develop, and deliver massive open online courses. Since the university at which I am employed has implemented a national MOOC program, instructional designers set about finding the right approaches to provide learners with a transformative learning experience, as well as to increase the accessibility and quality of higher education with in Malaysia. I now seek to understand the approach of open online education more holistically from the perspective of an expert instructional designer.

PAGE 29

29 Future Difficulty Designing a high quality MOOC requires a number of roles that are generally filled b y an instructional designer, including course developer, instructor, and subject matter expert. Thus, another overarching challenge is my desire to assist novice instructional designers through providing a catalog of design patterns in designing effective learning modules that can be reused in different contexts, creating an engaging community experience. Rationale for the Computer Science MOOCs Coursera, edX, and Udacity are among outstanding MOOC platforms that provide high quality computer science and technology education. Almost every topic in computer science, such as game development, Java/Python programming, and machine learning is covered in those MOOC platforms. S ince MOOCs are generally offered by the world leading universities, the courses are the same ones as given on those prestigious institutions and are taught by the same professors. Lectures on these MOOC platforms are structured quite different from regular online courses. The lectures are comprised of multiple short videos, typically 3 5 minutes in length with a series of quizzes. This approach keeps the learner engaged and interested in the topic at hand. Some MOOCs are created in collaboration with indust ry partners such as AT&T, Amazon, Facebook, and Google. These companies have identified relevant skills that can be used in a job setting and built a pipeline of high potential employees. Learners are required to complete a series of projects and the feed back received on those projects is exceptional. Justification on the R einvent N ew S olutions

PAGE 30

30 Instructional designers can use design patterns to solve problems that arise as they designing their solutions Design Patterns are reusable solutions to recurrin g problems within a given context. It is important to note that more than likely, someone has already solved the problems that instructional designers encounter when designing open online courses. Thus, there is no reason for instructional designers reinve nt the wheel each time they design a new open online course. The more they use design patterns, the easier it will be to solve new problems. Problem Statement Design patterns provide a format for capturing and communicating design knowledge among practitioners. The use of design patterns that are often cited include: act as a design tool, provide for accurate and concise communication among designers, and convey expert knowledge to novices. In the meantime, the emergence of Massive Open Online Courses (MOOCs) has generated a new and broad interest in open online learning. This growing trend in higher education has emphasized the capabilities and challenges relate d to the design of such learning settings (Chapman, Goodman, Jawitz, & Deacon, 2016). The instructional design community, including practitioners and researchers have dealt with the design challenges d ue to the continuous development of open online courses Many different approaches are available for the functionality, pedagogy, delivery, and support of open online learning. Some have been successful and others have ended in failure. Thus, there is an urgent need to effectively document and communicate desi gn knowledge in this field. Open online learning has its roots in the traditional online and distance learning, and this study was expanding the effort through design patterns for solving problems that are generated by teaching and learning at such a massi ve scale.

PAGE 31

31 The enrollment of MOOCs has grown exponentially and the enrolled students in some courses had exceeded 230,000 (Jordan, 2014). Coursera, edX, and Udacity have appeared as the largest MOOCs platforms that offered more than 1,000 online courses (L in, Lin, & Hung, 2015). Despite gaining popularity, issues have arisen regarding open online learning, for instance, high dropout rates, low completion rates, and low student engagement (Chen & Chen, 2015; Ferguson, Clow, Beale, Cooper, Morris, Bayne, & Wo odgate, 2015). Some of the possible explanations for such student behavior were poor course content design (Margaryan, Bianco, & Littlejohn, 2015) and confusing learning interfaces (Lin et al., 2015). Although MOOCs have opened up education to millions of learners, these massive courses have been plagued by extremely high dropout rates. Findings of Stich and Reeves (2017) revealed that MOOC dropout rates as high as 90%. Online learning usually requires learners to be independent and able to deal with techno confusion and frustration is one of the reasons for high dropout rates. Liyanagunawardena, Adams, and Williams (2013) focused on the completion rates, progression, and retention of MOOC s, which could provide a better More research is required, which focuses on individual learners as MOOCs embrace the diversity of learner experiences through the co ncept of openness. Jordan (2014) claimed the vast majority of learners who registered for the MOOCs failed to complete the course. Chapman, Goodman, Jawitz, and Deacon (2016) reported that MOOC poor completion rates typically below 10% of signups. Similarl y, Stich and Reeves (2017) stated that MOOC completion rates are between 5% and 12%. Some learners may

PAGE 32

32 enroll without fully understanding the course requirements, such as prerequisites for those that are more advanced, and for those that are introductory b ut still require certain background knowledge or skill level (Stich & Reeves, 2017). Other learners may have unrealistic expectations of what they can achieve throughout the course, and take much not satisfy all individual needs due to the diversity of learner backgrounds. Milligan and Littlejohn (2014) highlighted the importance for learners to monitor, control or self regulate their own learning due to the non existenc e of face to face interaction with MOOC instructors and other learners. Open entry encourages informal enrollment, particularly professionals for advancing a current job (Stich & Reeves, 2017), for instance to learn more about the MOOCs format as a way to produce their own courses, and some because of curiosity and enjoyment, as well as personal challenge rather than to gain understanding of the subject itself (Breslow, Pritchard, DeBoer, Stump, Ho, & Seaton, 2013). In their paper on instructional design qu ality of MOOCs, Margaryan, Bianco, and Littlejohn (2015) discovered that most course designs concentrate on the presentation of learning material instead of an interaction or feedback involving massive numbers of participants. Learner retention is an effec tive way to measure a success of MOOC as those who persevere in the course have an opportunity of obtaining the educational benefits. Hone and El Said (2016) conducted a survey to explore the factors affecting MOOC retention. The study revealed two signifi cant predictors of retention in MOOC, namely the course content and interaction with the instructors.

PAGE 33

33 Dating back to 2009, Fini claimed that MOOCs are nothing more than just traditional face to face courses, but using technology to support teaching and lea rning at scale, such as video recorded lectures. The findings of t his study enlightened how higher education institutions misuse a MOOC format as they relied upon traditional teaching methods that failed to leverage the benefits of an open platform. The au thor suggested to investigate the profile of the participants of MOOCs for further research as those associated with learning outcomes and retention. According to Gibbons (2010), one of the main criticisms of technology based learning was the lack of human interaction in certain aspects of a course design that were required to stimulate student engagement, the key to academic motivation, persistence, and course completion (Devlin, Feldhaus, & Bentrem, 2013). The design of online courses directly influences learning outcomes (Kauffman, 2015) and some of them already addressed problems related to course content design and the learning interface. Ruey (2010) investigated whether learners benefit from an online course based on constructivist instructional strate gies, for instance, virtual group projects, peer moderated discussions, and chat room meetings. The findings of the study revealed that this approach increased self directed learning skills and interaction among learners through peer collaboration, as well (Ruey, 2010). Instead of focusing on grades, learners became more interested in their learning, but constructive feedback and appropriate facilitation from the instructor were significant to achieve online course quality. In some cases, the same project found it difficult to replicate their previous successful experience. For instance, Carnegie Learning produced the well known

PAGE 34

34 intelligent tutoring system for high school mathematics based on h uman cognition, Cognitive Tutor Algebra, which was then followed by Cognitive Tutor Geometry. However, the findings showed that Cognitive Tutor Geometry was not as effective and successful as Cognitive Tutor Algebra (Bibi, 2010; Cen, Koedinger, & Junker, 2 007; Pane, McCaffrey, Slaughter, Steele, & Ikemoto, 2010; Steenbergen Hu & Cooper, 2013). Thus, codification of best design solutions into a set of best practices is important to transfer the past related experience of managing similar situations in the fu ture. Ideally, the development of open online courses should be pedagogically driven, instead of technology driven. When designing debates for an online course, for instance, the instructional designers should be mindful of the pedagogical issues, includin g reflection, interpersonal, flexibility, and teamwork skill development (Retalis et al., 2006). However, the design of usable and pedagogically effective learning environments is very challenging as it requires a significant amount of expertise and creati vity, which could be a complicated task for novice designers who lack of experience (Frizell & Hubscher, 2002). Past experiences and best practices are bound up with tacit knowledge and difficult to transfer to another person, but sometimes could be shared through design guidelines or observation and practice with expert instructional designers. Design patterns capture effective and successful design solutions that can be reused in different contexts, as well as could mediate and transfer design knowledge o f experts to novices. The practical use of design patterns in open online learning specifically on computer science courses is yet to be formalized if compared to other domains, particularly architecture and software engineering. Instructional designers of

PAGE 35

35 online courses, especially novices could exploit design expertise and past experience of others to avoid reinventing solutions in their design effort (Retalis et al., 2006). Hence, there is a demand for a more structured method for documenting and describ ing design patterns best practices in designing open online courses. Also, it is significant in making design patterns explicit and available to the instructional design community and become a common practice. Research Questions The primary research questi ons addressed by this study were: 1. To what extent do the design patterns exist within the Massive Open Online Courses (MOOCs) in computer science? 2. How is a catalog of design patterns for open online learning constructed? Research Design This research was designed as a two phase study. Phase 1 focused on the design pattern mining and described how to derive patterns for open online learning. Ideally, the best source would have been successful and high quality open online courses, specifically MOOCs. MOOC s are primarily offered by elite universities such as Stanford, Harvard, Berkeley, and MIT, and are taught by the same professors that teach in those prominent campuses. MOOCs provide an opportunity for instructional designers to learn best practices from others. Computer science is one of the most popular subjects on MOOCs, and dozens of computer science related topics are available for the undergraduate and graduate levels. There are many instructional strategies designed to help learners understand the c ore concepts of computer science. After identifying the platforms, design patterns of the most popular topics of computer science MOOCs were mined. In this study, design

PAGE 36

36 patterns of MOOCs were mined through five methods: (1) Self observation, (2) Expert in terview, (3) Analysis of the functionality of computer science MOOCs, (4) Review of the literature on pedagogical strategies, and (5) Learn from existing published patterns in other related areas. In particular, I searched for the recurring and meaningful design patterns. Instead of providing facts, design patterns describe the design experience in a more descriptive way. I focused on the practicality of design patterns since the goal of design patt erns was the description of reusable solutions to recurring problems. Each design pattern experience involved interviewing instructional designers to explore their experiences, expertise, and knowledge in designing open online courses. Themes emerged from qualitative analysis of the interviews were recorded. Information from the pattern mining approaches were used to inform the catalog of design patterns. Design patt erns consist of instructional strategies generalized from a number of successful design cases or best practices. During Phase 2, design patterns were described and organized in a standard format. Design patterns are a tool for documenting and reusing prev ious solutions. Cataloging design patterns was performed in a formal way. This study used a template that was modified from the Gamma, Helms, Johnson, and Vlissides (1995) and Alexander (1979) pattern structures to describe and organize design patterns. Ea ch design pattern was named, explained and described systematically. A qualitative approach was used to analyze the data. More details on the specific qualitative

PAGE 37

37 technique is given in Chapter 3. The process of integrating the findings and the catalog of d esign patterns is provided in Chapter 4. Significance of the Study Design patterns have appeared as a way to capture the design experience and knowledge of experts and deliver design solutions to others, including inexperienced ones. Design patterns do not describe a particular concrete design. Instead, the design solutions are just a template that can be used in many different situations, providing an abstract description of a design problem and how to solve it. Having the capability of capturing and commu nicating design expertise not only allows for the reuse of design knowledge, but also can save cost and resources, as well as helping to get the right design faster. Furthermore, only a few articles that discuss ways of identifying design patterns have app eared in the literature. This study presented a set of instructional design knowledge that was based on general reusable solutions for recurring problems to design open online courses. The results of this study could provide both expert and novice instruct ional designers a method to create high quality designs, as well as suggested functionalities for more advanced and productive design approaches. In other words, this study had important implications for practice. The catalog of design patterns developed i n this study could assist instructional designers to design better quality of open online courses, enabling them to learn from past successes and failures of others, instead of reinventing solutions that others have struggled to develop. However, instructi onal designers need to be creative because each design problem has both common and new/unique parts. The catalog of design patterns helps instructional designers to solve the common problems and release resources to solve the new problems.

PAGE 38

38 Definition of T erms Design p attern : A general reusable solution to a recurring problem within a particular context, as well as an approach for capturing, representing, and sharing design knowledge ( Alexander, 1979) Design pattern c atalog : A documented collection of relat ed design pa tterns with a consistent format (Gamma et al., 1995) Explicit knowledge : Codified or written knowledge and usually represented by scientific literature ( Yoshikawa, 1993) Instructional d esigner : An individual that is responsible for designing instruction, performing and organizing work plan, and managing the overall aspects of the instructional design process ( Koszalka et al. 2013) Massive Open Online Course (MOOC) : An online course that is des igned for open access through the web and unlimited participation (Milligan & Littlejohn, 2016) Open online l earning : A free access and delivery of educational content and instruction through the use of computer and communication technologies (Stockley, 20 03) Pattern l anguage : A structure for design patterns within a particular domain that creates common understanding among practitioners, researchers and learners in exploring and sharing their ideas about successful teaching and learning ( Alexander, 1979) Tacit knowledge : Knowledge acquired through practice and direct experience, highly pragmatic and context specific, and typically shared through interactive conversation ( Yoshikawa, 1993)

PAGE 39

39 Organization of Study This study is organized into another five chapters and appendices. Chapter 2 introduces the history of open education, open online learning, instructional designers, knowledge management, explicit and tacit knowledge, presents the theoretical framework of the study, and reviews of the relevant literature on design pattern research in education. Chapter 3 provides the research methodology. Chapter 4 discusses the analysis of the analysis of data mining, interview transcripts and preselection survey results, findings, and the catalog of design patterns for open online learning. Chapter 5 presents the summary of the study, discussion, and implications of the findings, and also recommendations for the future research. Finally, the appendices include interview qu estions, examples of the interview transcript, survey instrument, and list of references.

PAGE 40

40 CHAPTER 2 LITERATURE REVIEW Advances in technology have provided new opportunities for radically new models of online education (Milligan & Littlejohn, 2016). Sin ce 2012, many universities have begun venturing into MOOCs as ways to provide cost effective access to education (Margaryan, Bianco, & Littlejohn, 2015). There are various different approaches to the design of open online courses, some have been very succe ssful and others are not. Apparently from the literature, the instructional designer community is struggling with the design issues due to the accelerated expansion in open online education (Wang & Baker, 2015; White, Davis, Dickens, Leon, & Sanchez Vera, 2015; Zheng, Rosson, Shih, and Carroll, 2015). It is interesting to see researchers expanding their research on traditional online education into open online learning, and sharing design knowledge could solve particular challenges associated with teaching and learning at scale. Open Education The concept of open education dates back to 1969 when the Open University of the UK was established, and admitting students in 1971 (Wiley, 2015). In the context of anyone to enroll in courses irrespective of their educational background. In recent hing materials available at no cost to the public (Wiley, 2015). This open license gives a person or organization permission to retain, reuse, revise, remix, and redistribute all the materials used in teaching without payment and copyright infringement. Ca nvas, Moodle, and

PAGE 41

41 Sakai are some examples of learning infrastructure technologies that are created under open licenses. The conception of open licensing in supporting open education brought up a new initiative in 2001 (Hewlett Foundation, 2016). In 2002, William and Flora Hewlett Foundation started investing in OER to provide high quality materials for teaching, learning, and research that are free and accessible to people a round the globe. Hewlett Foundation specifically defines OER as: Open Educational Resources are teaching, learning, and research resources that reside in the public domain or have been released under an intellectual property license that permits their fre e use and repurposing by others. Open Educational Resources include full courses, course materials, modules, textbooks, streaming videos, tests, software, and any other tools, materials, or techniques used to support access to knowledge (Hewlett Foundatio n, 2016) The evolution of OER can be traced back to the mid 1990s, when the idea of open learning objects spread through education communities. Wiley (2002) defined learning objects as reusable digital resources deliver over the Internet to support learn ing, such as video, audio, image, and animation. The main characteristics of learning objects are interoperable with any system or delivery tool, reusable in various learning events, accessible, and manageable media contents. The movement continued to the open courseware and open textbook initiatives in the early 2000s (Hanley, 2015). Open Online Learning Online learning is the delivery of an educational program, training or learning through electronic medium, which also known as e learning, distance learni ng or online training (Stockley, 2003). Campus universities have largely used online learning as a complement to face to face instruction while open universities tend to apply models of

PAGE 42

42 distance education for the delivery of digital content (Milligan & Li ttlejohn, 2016). At the same time, open education is seen as a solution for expanding access to affordable learning (Hanley, 2015). Massive Open Online Courses (MOOCs) are a unique form of online learning that allow for the capacity of courses to enroll la rge numbers of learners to participate at no or minimum cost with an adequate Internet connection. MOOCs are a part of the ongoing evolution of open education initiatives for serving growing and diverse learners through the use of technology to mediate ped agogical interactions, as well as to create and distribute educational contents. MOOCs, however, are particularly different from other conventional online participation and open enrollment (Hanley, 2015), also to track vast quantities of to open access for learners regardless of their academic qualifications (Milligan & Littlejohn, 2016 ), while materials for the course are mostly free and accessible to all a start and a finish date to address particular learning objectives, followed by a sequence of activities organized by an instructor while offering a coherent resource set. To provide scalable solutions and cost effective access, MOOCs are typically designed based upon self regulation that assumes learners are able to control their own learning, ins tead of just depending on instructor guidance (Milligan & Littlejohn, 2016). and methods of a ssessment. Content based MOOCs or xMOOCs are typically designed

PAGE 43

43 around a modular video lectures format, followed by additional readings, assignments, multiple choice questions, as well as peer graded and auto graded quizzes for of knowledge (Margaryan et al., 2015) Also, online discussion forums on xMOOCs enable active participation among learners, allowing the exchange of knowledge and ideas to create global learning communities (Hanley, 2015). In contrast, connectivist MOOCs o r cMOOCs focus on an overarching instructional goal and are less directive with respect to process. These cMOOCs give more attention on exploration and discussion, instead of focusing on instructor provided content (Margaryan et al., 2015) Using a variety of media resources or text based, instructors may pose weekly questions to the learners, while learners usually develop their knowledge through collaborative learning with peers. Instructional Designer Instructional designers often engage in the analysi s, design, development, implementation, and evaluation of instruction to provide successful learning experiences for learners in a wide variety of settings, ranging from K 12 environments to higher education. Koszalka, Russ Eft, and Reiser (2013) suggested instructional designers show their competencies by applying systemic thinking practices and selecting a sound instructional design and development tools in order to maintain the quality of instructional solutions a profession, it cons ists of a series of well defined competencies, and an active group of practitioners who work in increasingly complex and sophisticated (Richey, Klein, & Tracey, 2011, p. 1). T he International Board of Standards for Training, Performance and I nstruction (ibstpi) published the most complete and recent version of statements defining a competent instructional designer ( Koszalka et al. 2013). According to ibstpi an

PAGE 44

44 instructional designer must be knowledgeable in the design sciences (i.e., assessm ent, message visual), project management, production processes, collaborative activities, as well as emerging technologies to prepare themselves in facilitating different types of learning ( Koszalka et al 2013) However, it is important to note that tho se knowledge and skills are useful to support instructional designers with the competencies when designing and creating instruction, not to turn themselves into information technology or production specialists. Ritzhaupt and Kumar (2015) interviewed higher education instructional designers from across the United States to discover the essential knowledge and skills for success in their roles. All participants were reported to have graduate degrees in education with a concentration in either instructional de sign, instructional technology, instructional systems design, educational technology, learning technologies or multimedia design. Most participants stated that those academic degrees had prepared them well for the jobs and provided them with the knowledge of instructional design models and processes, learning environments, multimedia development, and communication design, in which they were able to apply in their current tasks. In general, an instructional designer is described as having mastery of the le arning theories, instructional design models, also possesses soft and hard skills (Ritzhaupt & Martin, 2014). Also, instructional designers should always be willing to learn on the job and keeping abreast of emerging technologies. Instructional designers a re interacting one to one with subject matter experts and working closely with a technical team, such as programmers, graphic artists, animators and audiovisual

PAGE 45

45 subject matter experts, instructional designers can create an instructional blueprint to be successfully implemented by the technical members. Although the instructional designer is a highly demanding job, Retalis et al. (2006) believed that novices are coming in to the field do not need a lot of experience to be hired if they could exploit design knowledge and expertise of others to avoid reinventing solutions in their design effort. Subject Matter Experts W hile the role of instructional designers is to design cou rses the role of s ubject matter experts is to provide expertise in a defined area. After analyzing the learner needs instructional designers prepare the course specifications, define the learning scope and objectives, as well as decide on the assessment method. Subject matter experts are typically not the ones creating a course, but they share their knowledge, propose the contents that should be covered, identify the resources, and ensure the accuracy of the learning materials prepared by the instructiona l design er (Keppell, 2000). Subject matter experts have a tendency to forget about sharing important information when the content is too common to them. Thus, instructional designers should catch the situation, reminding subject matter experts to provide e xamples and elaboration to the learners. Subject matter experts should work together with instructional designers to create a reliable and well integrated instructional system. In other words, the collaboration between subject matter experts and instructio nal designers is necessary to produce a course with accurate content, as well as to provide an assessment that is fair, consistent, and free of bias. In some cases, an instructional designer need to be a subject matter expert. There are two major componen ts of course design: the human part and the technical part. The human component requires subject matter experts to work closely

PAGE 46

46 with instructional designers to successfully produce the desired or intended result. The design and development of instructional system require a team with a diverse range of skills to successfully complete all aspects of a module. The technical component of course design can be guided by the well defined processes in terms of translating the content into a form that embodies the s ound educational design (Keppell, 2000). The design and development of instructional systems often require instructional designers to assist subject matter experts in creating suitable teaching and learning resources. Subject matter experts are well versed in their area of expertise, but often are not familiar with the learning process. Similarly, instructional designers know the science of learning very well, but are not always familiar with the subject matter. Creating a common language between subject ma tter experts and instructional designers is necessary, particularly when important issues arise. According to Keppell (2000), the knowledge map could assist to focus the attention of the subject matter experts and instructional designers on the most crucia l elements of the content. Epistemology of Design Design is a broad concept and can be classified into three stages: (1) Design as activity, (2) Design as planning, and (3) Design as epistemology. The classification of design consists of a conceptual struc ture that can define the epistemology of design, linking relevant scientific endeavors with engineering activities (Mahdjoubi, 2003). Design as activity is related to the pre execution or conceptualization stages for creating new products that can be organ (Mahdjoubi, 2003). Applied art/industrial design, architecture, engineering, and fine art are some examples of the academic disciplines and professional fields for design as activity. Fine art is mainly

PAGE 47

47 psychological and physical characteristics of human beings. Design as planning is regarded as the pre execution or conceptualization stages of planning, strategizing, and decision making, as well as systematic thinking prior to execution (Mahdjoubi, 2003). In other words, design as planning is a visualization for subsequent implementation. This design stage requires an interdisciplinary approach in a wide range of fields, including art, business, industry, management, and military. While design as activity is associated with professi onal endeavors such as architecture or engineering, design as planning is more towards integrating the managerial and strategic aspects of design. Design as epistemology is related to the synthetic methodologies of execution (Mahdjoubi, 2003). The epistem ology of design can also be referred to as science of design. Since the ancient Greeks, the main goal of science has been the exploration of truth. Design, on the other hand, is mostly about synthesis and how things should be. One can think of epistemology of design as a method for art, change, and strategy, while science is based on analytical research. Apparently, the discussion on epistemology of design and analytic methodology can be linked with the study of knowledge management. Niehaves (2007) howeve r, had a different viewpoint about epistemology. While design practice seeks to apply existing knowledge, design science research is

PAGE 48

48 concerned mainly with adding new knowledge to the body of knowledge. While Hevner, March, Park, and Sudha (2004) argued that design science research is another paradigm besides interpretivism and positivism, Niehaves (2007) argued that epistemological assumptions are fundamental to the design science research and greatly impact the execution and evaluation of such research. Table 2 1 Although the design science research guidelines (Hevner et al., 2004) Guideline Descriptio n Guideline 1: Design as an artifact Design science research must produce a viable artifact in the form of a construct, model, method, or instantiation. Guideline 2: Problem relevance The objective of design science research is to develop technology based solutions to important and relevant business problems. Guideline 3: Design evaluation The utility, quality, and efficacy of a design artifact must be demonstrated rigorously by means of well executed evaluation methods Guideline 4: Research contribution Effective design science research must provide clear and verifiable contributions in the areas of the design artefact, design foundations, and/or desig n methodologies. Guideline 5: Research rigor Design science research relies upon the application of rigorous methods in both the construction and evaluation of the design artifact. Guideline 6: Design as a search process The search for an effective arti fact requires utilizing available means to reach desired ends while satisfying laws in the problem environment. Guideline 7: Communication of research Design science research must be presented effectively both to technology oriented and management orient ed audiences.

PAGE 49

49 Hevner et al. (2004) developed a set of guidelines on how to conduct and evaluate design science research as in Table 2 1 The design science research guidelines were proposed with the aim of achieving and evaluating design knowledge, for instance, design models, methods, processes, theories, or executions in a consistent way (Hevner et al., 2004). Although the design science research guidelines were developed specifically for the information system discipline, they are applicable to relate d design disciplines. Hevner et al. (2004) believed that a researcher can make a contribution to the body of design knowledge by applying these guidelines in a rigorous manner. Design Knowledge There are many definitions available for design given by diffe rent authors. Carrara, Kalay, and Novembri (1992) described design as a creative process that specifies the required actions to be taken in order to achieve particular goals. According to Koh, Ha, Kim, Rho, and Lee (2003), d esign is not only creating a new product or par t of it, but also synthesizing pre existing designs in an efficient way. Richey et al. (2011) claimed that a design contains factual knowledge that related to many topics, for instance the definitions of a learning hierarchy and mental model. Yokigawa (1993) stated that a design process starts with vague or unclear descriptions of the design object until they gra dually become more comprehensive. Yokigawa (1993) further This indicated that a design process is a typical ill defined and ill structured p. 133) Since there is no formal me thod or procedure for such effort, the design is basically exploratory in nature actions are hypothesized and the results are evaluated against the predefined goals (Carrara et al., 1992) Designers, of course, depend on

PAGE 50

50 their knowledge to effectively ac complish this exploratory process. The subjective including past experience, perceptions, belief, preferences, and feelings (Carrara et al., 1992) The objective sources of design knowledge are the attainment of truth based on scientific research and theory for solving problems (Richey et al., 2011). There are two key principles in the design literature: (1) Good design is generally iterative, and (2) The design can be enha nced through multiple iterations with each iteration continuously being improved until an optimal solution is reached. In the context of learning technology, iteration involves some form of feedback, such as course evaluation, learner assessment, or quiz t o achieve improved design solutions. Hoadley and Cox (2009) suggested two main themes of the design: (1) Observing good and bad examples; and (2) Putting design method into appropriate practice. Given that design novice designers should consider applying those fundamental design principles in managing the complexity of problems in the creation of learning technologies. Also, an effective design is not easy to get right the first time. Compared to experienced designers, novices are often overwhelmed with many design options. Experienced designers sometimes come out with good designs. Design knowledge is different from other types of knowledge, and thus it should be delivered differ ently from other disciplines. In order to convey design knowledge, it is important to understand how novices can acquire it (Hoadley & Cox, 2009). Indeed, it is challenging to find an easy way for novices to understand the right design solution. Experts ar e unique and have different approaches. Although experts familiar with the

PAGE 51

51 subject, they have difficulty in communicating design knowledge to novices. In short, there is no optimal solution to solve design problems. For instance, one can document the recur all time (Brewster, 2004), however, it is not easy to replicate his success, not only the novices, but also the experts. Design knowledge is hard to identify, but it does exist and can be shared within the design community, so novices can develop their own understanding by applying those solutions. Knowledge Management Managing knowledge is important in any discipline. Knowledge management is a process to improve the performanc e of organizations through the use of information and knowledge (Sallis & Jones, 2002) Bhusry and Ranjan (2012) defined k nowledge management as the management of knowledge related to organizational goals and objectives. The successful management of knowle dge helps organizations to deliver value added products and services, thus it is crucial to promote the deployment and sharing of knowledge within the organization (Bhusry & Ranjan, 2012). Knowledge management plays a fundamental role as a survival strateg y for any type of organization, including educational institutions as a way to strengthen their performance (Sallis & Jones, 2002). Sallis and Jones (2002) claimed that sound knowledge management not only could enhance the effectiveness and efficiency of business or technology companies, but also could allow educational institutions to improve the learning of their learners and staff. Knowledge is dynamic and ever changing, what was previously important could be outdated and obsolete (Koh et al., 2003). Fu rther, Sallis and Jones (2002) claimed,

PAGE 52

52 knowledge management in organizations are intended to develop intelligence and achieve improved performance and high productivity level. other reason for managing knowledge is to retain knowledge and experience when designe rs resign or retire. For instance, some instructional designers had worked for almost five years in the same educational institution and in other cases, for over 10 years. Resigning designers are sometimes replaced with novices. The novice instructional de signers need more time to build up experience, and thus there is a demand to capture, store and reuse expert knowledge. The motivations for knowledge reuse are to allow others search for previous solutions when working on a similar design problem, and enab le existing designers understand the rational behind the decisions made earlier. Besides, instructional designers could validate or justify their design decisions based on previous experience (Ahmed, 2005). Due to these reasons, it is important for the edu cational institution to have a retrieval system with a high flexibility mechanism (Koh et al., 2003) to support the knowledge reuse. The goal of a knowledge management system is to capture, store, structure, and share organizational knowledge. Bhusry and R anjan (2012) stated that the development of institutional repository or known as a database is crucial to capture, store, index, maintain, and distribute all of the knowledge bases in digital formats. Hence, teaching materials such as lecture notes, presen tation slides, question banks, simulation games, role plays, videos, and audios can be made available on the Internet, and further can be utilized by faculty in course preparation. A central repository not only allows easy

PAGE 53

53 access to knowledge, but also imp roved validity of knowledge and easy to trace the source of knowledge. Overall, t he development of a knowledge management system can facilitate the promotion of institutional value in the competitive academic society ( Bhusry & Ranjan, 2012) Explicit Knowledge and Tacit Knowledge Within knowledge management, two types of knowledge are often discussed, namely explicit knowledge and tacit knowledge (Koh et al., 2003). Explicit knowledge or abases, documents, and notes. symbols figures, and Yoshikawa, 1993, p. 133). S cientific knowledge mostly falls into this category. On the other hand, t codified knowledge rooted in attitudes, background, cultural beliefs, experience, practice, and values (Richey, Klein, & Tracey, 2011). a form of knowledge that is explicitly or implicitly recognized by human and used for reasoning, but very difficult to Expertise and skill are basically composed of tacit knowledge. In practice, all knowledge is a combination of both explicit and tacit elements. The transfer processes for explicit and tacit knowledge are different, in terms of their conditions, methods, pace, and supporting mechanisms. Since explicit knowledge is formally documented, it can be easily captured, stored, and retrieved. This knowledge can be handled efficiently by knowl edge management system, facilitating the process of reviewing, storing, modification, and retrieval of documents. Conversely, tacit knowledge is very hard to capture and convey since it resides in the mind of the

PAGE 54

54 practitioner. Tacit knowledge is transferre d mostly through socialization, mentoring, observations, and face to face interactions. Compared to explicit knowledge, tacit knowledge is regarded as the most valuable asset to the organization, and less focus on it could reduce the ability to sustain co mpetitive advantage (Koh et al., 2003). To handle tacit knowledge is indeed challenging even with the help of knowledge management systems. It would be near impossible to express our intuitive understanding gained through years of practice and experience. An expert instructional designer, for instance, will solve a design problem based on his experience. It would be very hard for the designer to convey his What are Design Guidelines, Design Patterns, and Design Principles? Design knowledge can be documented in many different ways. Design guidelines, design patterns, and design principles are widely known forms of design guidance that may help solve some design problems. Novices often h ave difficulties in understanding and applying design knowledge to a particular problem. Selecting the most appropriate alternative is often based on evaluating certain advantages and disadvantages of the various design options. For each design problem, th ere are various alternative solutions for designers to choose from. A good design solution is about balancing between needs and constraints. Design guidelines are the most concrete form of design guidance. Design guidelines are written at a low level of abstraction, so they can be applied in a specific context. Stewart and Travis (2003) defined guidelines as suggestions of good practice that depend on the author as an expert on a particular topic. In practice, design guidelines require design trade offs as they may conflict with each other. However,

PAGE 55

55 design guidelines are difficult to apply in other different contexts as they provide detailed instruction (Gould & Lewis, 1985). Design patterns are descriptive, normative, and communicative, and are designed to provide general solutions to recurring problems that can be reused in different learning contexts (Kolfschoten et al., 2010). Pescio (1997, p. 130) suggested, how), and most people can also make goo d choices between alternatives (say design problems, identify the solutions that worked well, and capture their essence in a pattern (Pescio, 1997). Design patterns provide examples of how the solution approach can be used to assist novice designers in interpreting the guidance. It is important to consider various relations between single design patterns, instead of isolated solutions. However, there is no standard procedure to extract design patterns as they are fuzzy and hard to capture (Kohls & Uttecht, 2009). The recorded patterns usually differ not only in content, but also in style, depending on the designers who design the pattern. Some designers may concentrate on didactics issues, others may be more focus on st ructures. Design patterns are useful in translating requirements to specific design solution, while design guidelines are more useful to describe requirements. Apart from that, design patterns focus on the effectiveness of design solutions in a specific co ntext, but design guidelines generally assume an absolute validity. Design principles describe a rule to be followed to achieve a particular situation (Hoadley & Cox, 2009). Compared to design guidelines and design patterns, design principles are the most abstract form of design guidance. If patterns represent the

PAGE 56

56 Design principles are design goals that are written at a high level of abstraction. Those principles can be applie d in any context, but do not provide concrete design guidance. For instance, Gould and Lewis (1985) in their study suggested general design principles for system usability that include involvement with users, empirical measurement, and iterative design. De signers have to interpret, refine, and extend design principles in order to use them in a specific context. However, the practical use of design principles in the design process can be increased by specifying general principles at a low level of abstractio n. The formal example of design principle is the International Standards principle often provides no guidance toward a design that respects the principle itself. This m akes principles driven design somewhat hard to practice successfully unless you Research on Design Patterns Design is about finding solutions to problems within predefined constraints, yet designers usually reinvent new sol utions (Hoadley & Cox, 2009). Thus, it is difficult to determine how things have been done previously, and why they have been done in some certain way, and how to reuse solutions. Design patterns are just like theories that are subjected to empirical tests (Kolfschoten et al., 2010). The purpose of most design pattern research is to provide a method for capturing and communicating design knowledge in a field. The concept of design patterns came from architecture (Alexander, 1979) and Alexander, Ishikawa, S ilverstein, Jacobson, Fiksdahl King, and Angel (1977 p. x ) said, Each pattern describes a problem which occurs over and over again in our environment, and then describes the core of the solution to that problem,

PAGE 57

57 in such a way you can use this solution a m illion times over, without ev er doing it the same way twice After almost two decades, design patterns have been successfull y used in software engineering. Gamma et al. (1995) adopted the idea of design patterns and applied it to the ob ject oriented softwa re design. Gamma et al. (1995) described design patterns in their influential book, Design Patterns Elements of Reusable Object Oriented Software as, Design patterns are not about designs such as linked lists and hash tables that can be encoded in classe s and reused as is. Nor are they complex, domain specific designs for an entire application or subsystem. The design patterns in this book are descriptions of communicating objects and classes that are customized to solve a general problem in a particular context ( Gamma et al., 1995, p. 3) pattern is not a design. Instead, a design pattern is a template for how to solve complex problems that applies to different, but relate d, situations. A design pattern can be pattern as shown in Figure 2 1. Figure 2 1 Four essential elements of a design pattern Expert designers can facilitate design knowledge by assisting novices to identify and apply patterns. Novices can establish a way to classify problems and identify important patterns for certain types of solutions. As mentioned earlier, patterns contain

PAGE 58

58 ge neralizable solutions to common problems, hence expert designers can use pattern templates to assist novices establish ways of recognizing particular problems and their general solutions. These patterns, however, should be more abstract instead of specific solutions. Novices usually start to understand how the patterns work by recognizing the recurring solutions in a particular context, trying to apply those solutions in some cases, and noticing the relevance of each pattern (Hoadley & Cox, 2009). Design Pa ttern Usage The main benefits of design pattern usage are often cited are: (1) Provide a template to effectively document and communicate design expertise to other designer s (2) Disseminate design expertise, best practices and knowledge to novices and (3 ) Serve as a teaching and learning tool. According to Alexander (1979) and Gamma et al. (1995), reusability of solutions is the essential element of design patterns. There are two commonly used terms that are synonyms for design patterns: (1) A catalog of design patterns a set of pattern that has a relatively low level of structure and classification (Gamma et al., 1995), and (2) A pattern language coherent and interrelated design r, 1979). A pattern language is a hierarchy of design patterns that is structured by scope, and the relations between the individual patterns are clearly marked (Borchers, 2001). Alexander (1979) developed a hierarchical pattern language, known as a hiera rchy of problems to show the implementation order of related design patterns from top to down (Borchers, 2001). This approach assists designers to discover an appropriate solution to a particular design problem and indicate useful combinations with related patterns. Design patterns have been used to capture and share design knowledge between practitioners (Chung et al., 2004), and have been disseminated to novice designers in

PAGE 59

59 many domains, such as architecture, software engineering, human computer interacti on, and education. Design Patterns in Architecture The objective of design patterns in this field was to provide design solutions to recurring problems encountered in the modern architectural design, and to present this knowledge in a comprehensible and consistent form that could be reused by architects. Alexander (1979) asserted that design patterns are a narrative form, involving textual descriptions and pictures that illustrate a design problem and its solution. In particular, architecture design patte rns consist of 253 patterns, supporting architects in designing the modern architectural structures, such as the communities and neighborhood design. The development of those architecture design patterns was based on observation, analysis, and abstraction of implemented design solutions. Design Patterns in Software Engineering T he concept of design patterns is very well known in object oriented software design. T here are four main elements of a design pattern: pattern name, problem, solution, and consequenc es (Gamma et al., 1995) The pattern name is a description of a problem, its solutions and consequences. The problem is a description when to apply the pattern that describes the problem, its context, and a list of conditions. The solution describe s the r elationships, responsibilities, and collaborations of the elements that make up the design. T he consequences are the outcomes and trade offs from the pattern application. Figure 2 2 shows a template for describing and organizing design patterns in object o riented software design using a consistent format (Gamma et al., 1995). The catalog of design patterns by Gamma et al. (1995) contains 23 design patterns.

PAGE 60

60 Figure 2 2 Describing design patterns (Gamma et al., 1995)

PAGE 61

61 Design Patterns in Education Instead of providing design support to architects, software engineers, and manufacturers, design patterns are also a promising approach for instructional designers to express their design expertise to other instructional design profes sionals. In creating design patterns, focus is given to the forces acting on the problem and the reason for selecting a specific solution. Some patterns are thoroughly explored and elaborated through several years of research, while others are still new. A lexander, the founder of design patterns denoted asterixes (*) to indicate the difference between mature and immature patterns (Baggetun, Rusman, & Poggi, 2004). The classification of design patterns into mature and immature patterns fits nicely with instr uctional design, in which there are already established knowledge about instructional solutions, and also immature ones that need to be explored through further research. Design patterns have been used in the field of education as a means to capture, reuse and share design expertise and best practices. Design patterns are general reusable solutions to commonly recurring problems within a given context, based on the collaboration between experts and professionals in the education community. The objectives o f design patterns in education are threefold (Bokhorst, Moskaliuk, & Cress, 2014; Borchers, 2001; Kolfschoten, Lukosch, Verbraeck, Valentin, & de Vreede, 2010): (1) As a teaching tool to help students in obtaining design knowledge and skills, (2) To captur e knowledge from learning theories, instructional design models, expert best practices, and experiences for assisting student learning, and (3) To improve the quality of instructional systems. Design patterns are relatively new in the context of open onlin e learning, and available patterns currently focus on blended learning (Derntl & Motschnig Pitrik, 2005),

PAGE 62

62 cognitive learning (Kolfschoten, Lukosch, Verbraeck, Valentin, & Vreede, 2010), collaborative learning (Persico, Pozzi, & Sarti, 2009; Zitter, Kinkhor st, Simons, & Cate, 2009), e learning (Retalis et al., 2006), and learning management systems (McAndrew, Goodyear, & Dalziel, 2006). Besides, there are also design patterns that focus on courseware production (Yang, Moore, & Burton, 1995), human computer i nteraction (Kohls & Uttecht, 2009), learning object design (Chikh, 2014), and mobile learning user interface (Al Samarraie & Ahmad, 2016). Yet, none of them published a comprehensive catalog of design patterns for open online learning specifically in compu ter science. Design patterns for human computer i nteraction Kohls and Uttecht (2009) presented a case study on the mining, writing, and application of patterns for interactive educational graphics. The authors emphasized on the pattern mining and explained how to obtain patterns from experience and analysis. Drawing on schema theory, the authors conducted an empirical study to investigate whether different people perceive the same patterns. Kohls and Uttecht claimed that patterns are vague and hard to captu re because individuals may have different patterns in their minds. Consequently, the captured patterns differ not only in content, but in writing style as well, depending on the patent authors. Kohls and Uttecht asserted that a significant attribute of des ign patterns is that they document real design, rather than concepts evolve from theories. Kohls and Uttecht made the important distinction between: (1) The patterns in the real world, (2) The patterns in the human mind, and (3) The documented patterns. Ho wever, the authors stated that the most challenging task is to capture patterns in a standard way since there is no agreed procedure to deal with the pattern mining.

PAGE 63

63 Design patterns for c ollaborative l earning Zitter et al. (2009) suggested a theoretical framework that is based on graphical and textual representations, known as a task conceptualization, which adds new insights into the process of creating design patterns. A task conceptualization consists of elements of activity theory, boundary objects, c ollaboration scripts, and scaffolding to facilitate the process of redesign e learning environments. The authors selected the authentic task as a core concept and adopted the concept of the role and event to highlight the practical and concrete aspects of organizing meetings and sessions for learners. The authors also chose the concept of boundary object to add a particular analytical viewpoint as well as the concept of scaffolding. The properly designed roles, events and boundary objects provide learners w ith a necessary support to perform a task. The task conceptualization helps to develop valuable design patterns, specifically when redesigning online courses that involves theories and approaches, in which the task conceptualization turns out to be generic enough to handle the concepts that already in used. Design p atterns for c ognitive l earning Kolfschoten et al. (2010) developed design patterns to teach undergraduate students in designing simulation and computer programming related courses. Design ready made thors argued that design patterns do not only increase understanding of the design process among novices, but also increase the efficiency, flexibility, and reusability of the design effort. The authors analyzed the concept of design patterns in relation t o the c ognitive l oad t heory to explain how information can be presented to learners in ways that enables them to use the human brain capacity

PAGE 64

64 optimally for learning and comprehension. The authors found that novices working without design patterns really st ruggled than those working with design patterns. Their findings suggested that design patterns allow novices to build solutions that have an acceptable quality, as well as have the potential to reduce the design and development cost. Apart from that, it se ems design patterns are more helpful in establishing a nonlinear knowledge representation and transferring that knowledge to new situations. Another reason for the impact of design patterns on knowledge transfer could be the duces cognitive load. The authors concluded that design patterns are useful for novice learners, while practice problems are better for more experienced learners. The results clearly showed that experienced learners did not benefit from the learning effect between their own mental patterns and the design patterns offered to them, resulting in benefit from the use of design patterns as a baseline to which they can compare their own solutions. Design patterns for mobile learning user i nterface Al Samarraie and Ahmad (2016) developed design patterns for mobile learning user interface as a solution to design problems an d improve user experience. The authors established 71 design patterns and divided them into seven categories: dealing and social activities. The authors then investi gated the relationships between the design patterns and the learning activities. Their findings provided evidence that the use of design patterns differed between right and left hand dominant participants with regard to reading, information retrieval, and information browsing tasks. These results could

PAGE 65

65 guide designers to consider suitable design patterns for promoting learning activities based on hand dominance. Design patterns for open l earning Warburton and Mor (2015) constructed and evaluated the MO OCs design patterns by means of a shepherding process through three intensive workshops. The design patterns were developed from shared narratives of successful practice with ve dimensions of MOOCs design patterns are: orientation, participation, learning, community, and management, with 20 design patterns, such as Induction, Know Your Audience, Sharing Wall, See Do Share, Six Minute Video, Chatflow, Bring Them Along, Provocati ve Question, and Sparking Forum Participation. These 20 design patterns can be used to scaffold both novice and expert developers to develop a MOOC and propose the integration of design patterns into a simple iterative design cycle. For instance, Know You r Audience design pattern concentrates on a course design, so instructors can get to know each of their learners better. The design pattern for Know Your Audience as below: Pattern name: Know Your Audience Problem space: The open nature of MOOCs means tha t the barriers to sign up are low and therefore virtually anyone can become a participant. Yet when we design a course we often have a particular type of audience in mind. With a the target audience? learners are and what they bring to the learning journey. Although this MOOCs design pattern mapping is an example of open online learning design patterns, they are too gen eric and do not focus completely on the design,

PAGE 66

66 development, and delivering of MOOCs, particularly computer science. Furthermore, the authors only proposed three elements of a design pattern: pattern name, problem space, and solution statement. Methodo logy of Design Patterns The methodology used for the research of design patterns is governed by such a philosophy and involved a few stages, including pattern mining, pattern writing, pattern application, and pattern evaluation (Avgeriou, Papasalouros, Ret alis, & Skordalakis, 2003; Derntl & Motschnig Pitrik, 2005; Frizell & Hubscher, 2002; Kohls & Uttecht, 2009; Persico, Pozzi, & Sarti, 2009; Retalis et al., 2006). Kohls and Uttecht (2009) conducted pattern mining and described how to derive design patterns for interactive educational graphics from experience and analysis. However, the authors informed that the main challenge of design patterns is to find the right patterns and how t o properly capture them. There are also no agreed set of standards, procedures and guidelines to define, analyze, organize and evaluate such design patterns. The authors asserted that design patterns are always work in progress in which they are not writte n down at once and forever. Every comment on the patterns introduces a new perspective and each successful application brings in a new variation or strengthens a pattern. While every failure reflects constraints on the pattern and overall, a clear understa nding about its context and applicability can be achieved. Instead of focusing on pattern mining, their study also aimed to record patterns that are knowledge about the actual patterns precisely, which known as pattern writing. In contrast, Rusman, van Bruggen, Corvers, Sloep and Koper (2009) focused on the application of design patterns. The authors performed a case study in authentic

PAGE 67

67 educational settings, and concluded that design patterns can be applied to a new context, provided that it meets the design conditions, allowing evaluation and refinement of the pattern. Evaluation in online learning is important as it provides the feedback that helps students learn and determine how well the design patterns can improve student learning. According to Inventado and Scupelli (2015), limited evaluation of design patterns for online courses to some extent explain why design solutions do well in certain learning contexts, but failed in other contexts, for instance, different presentation platform desktop vs. mobile, different subject content, background knowledge of the student, and student motivation. Design patterns are captured to help other instructional designers learn from good design and reusable solutions. Thus, it is important to verify and evaluate whether design patterns could really help both novice and expert designers in their tasks. Kohls and Scheiter (2008) explained that the quality of design patterns depends on the va lidity contributors in this context were referring to the expert designers. The verification and evaluation process then should only involve expert designers since they will be a ble to provide guidance to the pattern writer in order to refine and perfect each design pattern. In reality, design patterns are a team effort and not created by a single person. Kohls and Scheiter (2008) further added that there are a few different metho ds to provide evidence for the proposed design patterns: experiments, interviews, observation, and workshop. Design patterns have been useful in other fields, ranging from architecture and software engineering to manufacturing and others. Indeed, there wa s some work on

PAGE 68

68 them in online course development, but there was a need for a comprehensive catalog that can be used and shared within the instructional designer community for designing open online courses. Schema Theory Expert designers construct cognitive schemas to solve recurring problems and this mental pattern is known as problem schemas (Kolfschoten et al., 2010). Expert designers use previously acquired schemas to perform particular tasks, but this is not the case with novices. Trying to understand a subject matter without prior schema is extremely difficult as the novices need to construct schemas of the concept to be learned without explanation. Thus, design patterns can be used to assist novices in acquiring design knowledge and analogical reasonin g skills from more experienced designers. Design patterns help novices remove cognitive barriers that impede the acquisition of design knowledge and domain expertise (Kolfschoten et al., 2010). Briefly, design patterns are schemas that novices can activate to better understand real things. The seminal work of Pollock, Chandler, and Sweller (2002) discovered that an isolated interacting elements approach had a significant effect on novice learners, rather than experienced learners. The isolated interacting elements approach suggests to present the information in smaller steps without indicating the manner in which they interact, before providing the full interacting material (Pollock et al., 2002). For instance, the novices learn how to perform a particular task without explanation at first and when this is captured in their schema, they can easily understand the logic behind the approach. Design patterns basically do the same thing, divide a complex structure into smaller components and provide explanation o n how to combine, as well as how to use these components. By pre structuring the information, it can be absorbed in parts

PAGE 69

69 (involving less cognitive effort) that are meaningfully related, which facilitates schema building (Kolfschoten et al., 2010). Convers ely, experts with already available and automated schemas tend to have a reverse effect as this method provides redundant information, and thus increase cognitive load. an auto mated schema for this problem that tells them immediately, without conscious account for performance differences between experts and novices. A problem schema enables probl em solvers to group problems in each category that need similar solutions (Sweller et al., 1998). Connecting design pattern concept to the schema theory can resolve problems involving in identifying patterns, as well as locating the right level of abstract ion and granularity (Kohls & Uttecht, 2009). This relation can be best explained in Figure 2 3. Based on schema theory, multiple examples are required for novices to induce a schema, store the schema in long term memory, and then transfer it when solving a new problem. Concrete examples are important as they help a novice designer to see the design in action. Figure 2 3 Linking schema theory to the design pattern concept (Kohls and Uttecht, 2009)

PAGE 70

70 Similarly, the study by Kohl s and Scheiter (2008) also made a connection between design patterns and schema theory. A schema theory explains human memory and the capability to recognize the use of schema efficiently. A schema is a mental is used by the brain to recognize experiences and knowledge. Design patterns are similar to the concept of schema, the documented design patterns help designers learn and assimilate or accommodate new schema. Apart from that, van der Veer and Melguizo (20 02) claimed pattern languages comprehensive description on how patterns are acquired and represented in the Kohls and Uttecht (2009) brought up a question of how our minds capture the real world patterns. Although pattern languages are considered as a mental model, it does not inform us of how the mental patterns are constructed. This is where the schema theory comes into the picture. Schema theory provides a clear explanation of how patterns are acquired and represented in the human mind. Interestingly, Kohls and Uttecht (2009) discussed two complementary processes that related to schema theory: accommodation and assimilation. Structures that are perceived i n the real world are assimilated into the mental structures, and strengthened the pattern in mind. On the other hand, the mental structures will be accommodated if perceived structures do not match the pattern in mind. Experts have constructed cognitive st rategies for problem solving, especially recurring ones, which is known as a problem solving schema. Similar to the concept of patterns, a problem solving schema enables individuals to organize problems into groups and each group requires the same generic solutions to solve

PAGE 71

71 problems. In addition, the invariant parts of a problem can also be captured by a problem solving schema the schema is being filled with the problem instantiation. Kohls and Uttecht (2009) made a good point about schema theor y that describes how schemata are obtained from various examples through abstraction, and existing schemata are restructured and refined during learning. Schemata can be classified into two different types: implicit mental patterns and explicit mental patt erns. Individuals have problem solving schemata in their head with unconscious mental representations or implicitly express the context, force or the problem. This implicit knowledge can be formalized by the explicit patterns. Since the documented patterns usually refer to the real world solutions, people misunderstood that these patterns are always reliable. In reality, induction is a mental process and this suggests that pattern induction is the way people generate ideas about patterns. Thus, the document ed patterns may be incorrect or incomplete there are good design patterns that can lead to success, and how the misuse of bad design patterns can lead to problem. To reiterate, the purpose of design patterns is to capture design knowledge, expertise, an d best practices, and therefore the main purpose of documented patterns is to allow other people learn from a good design. In order to accept a pattern and recognize it as a good design, one needs to understand the problem it solves. By understanding the p roblem, people are not only justifying a pattern, but also become aware of the problems that current designs or practices may cause for them. Design patterns do not only provide good solutions to common problems, but also shed light on hidden problems.

PAGE 72

72 The oretical Framework Schema theory has influenced instructional design in a variety of ways. One of Principles of Instruction a technique for learners to impose struc ture on what they learn in solving problems. Many instructional models proposed that the most effective learning are those related to a real world problem. However, most instructional practices ignore the activation of prior knowledge, application of skill s, and integration of of skills into real world activities as they focused mainly on the demonstration of skills ( Merill, 2002) Merill (2002) identified five principles of instruction that have been included in a variety of design theories and models: (1) The problem centered principle, (2) The activation principle, (3) The demonstration principle, (4) The application principle, and (5) The integration principle. Figure 2 of Instruction. According to Merr ill (2002, p. 44): (1) Learning is promoted when learners are engaged in solving real world problems, (2) Learning is promoted when existing knowledge is activated as a foundation for new knowledge, (3) Learning is promoted when new knowledge is demonstra ted to the learner, (4) Learning is promoted when new knowledge is applied by the learner, and (5) Learning is promoted when new knowledge is integ Figure 2 4 First principles of instruction (Merrill, 2002)

PAGE 73

73 First principles of instruction are more into creating learning environments instead of describing how learners obtain knowledge and skill from these environments. First principles are prescriptive (design o riented) rather than descriptive (learning oriented), make it appropriate to be applied to two or more different specific situations and also implemented in any types of system or instructional architecture In order to demonstrate how design patterns sol ve design problems, f irst p rinciples ( Merill, 2002) were used in this study as a framework to identify patterns. The development of skills in a digital age is needed and the challenge is even greater when dealing with massive numbers as in MOOCs. T he prope rties of the first principles are most appropriate for acquiring complex computer science skills, and providing support or structure needed to ensure conceptual and deep learning in MOOCs. Margaryan, Bianco, and Littlejohn (2015) conducted a study to ident ify the extent Margaryan et al. (2015) used rst principles of instruction to determine the quality of 76 randomly selected MOOCs. They analyzed the instructional design qualit y of MOOCs found that the majority of MOOCs scored highly on the organization and presentation of course material, but scored poorly on most instructional design princi ples. Gardner (2011a) through a systematic review of 22 contemporary instructional principles of instruction were abstracted from key instructional design theories and mo dels that consist of interrelated prescriptive criteria for effective teaching and instruction. Of the 22 seven instructional theories reviewed, seven theories emphasized

PAGE 74

74 all five principles, 10 theories mentioned four principles, four theories highlighte d three principles, and one theory emphasized two of the principles (Gardner, 2011a). Interestingly, each principle was supported by most of the instructional theories of the theories reviewed, while Applicatio n was mentioned by 100%. The principle of Problem or Task centered was mentioned by 81% of the theories in the review, Activation was mentioned by 54%, and In other research, Gardner (2011b) observed how award winning professors in higher education used the Merrill's First Principles of Instruction in real settings. This study confirmed the presence of first principles among the recognized instructors in their teaching, and linked the use of t hese principles to effective instruction in higher education. The study also emphasized that the effectiveness of these principles can be enhanced through positive motivational strategies and characteristics. Gardner (2011b) proposed future studies to iden tify the use of first principles in specific learning contexts, for instance in an online environment in higher education. It is worth noting that the existence of first principles in several different settings in higher education showed the ubiquitous nat ure of these principles, which suggested that they can be employed, regardless of program or practice (Gardner, 2011b). Summary Although there is limited research published on design patterns in education, particularly in open online learning, the existing literature builds upon the foundations of the large body of work on architecture, software engineering, manufacturing, and other design pattern research. Due to an increased need for learning at such massive scale and reduce the cost of learning support, MOOCs are likely to grow in number as they

PAGE 75

75 attract millions of learners. Since MOOCs are the latest trend in online learning, many higher education institutions consider offering them. Even though the use of design patterns has been shown to be an effectiv e method to support instructional designers in designing instructional solutions, there is still a need to conduct further research on the mining, writing, and evaluation of design patterns in open online education.

PAGE 76

76 CHAPTER 3 METHODOLOGY The purpose of t his study was to develop a comprehensive catalog of design patterns for open online courses by capturing, and sharing successful solutions for recurring problems in the context of Massive Open Online Courses (MOOCs). The study also sought to explore differ ent approaches involved in developing the catalog of open online learning design patterns for computer science courses. Research Design As stated in Chapter 1, this study investigated the primary research questions: 1. To what extent do the design patter ns exist within the Massive Open Online Courses (MOOCs) in computer science? 2. How is a catalog of design patterns for open online learning constructed? This research was designed as a two phase study. The first research question was answered through P hase 1. The second research question was answered through Phase 2. Phase 1: Design Pattern Mining The purpose of this phase was to mine design patterns of computer science MOOCs. Various techniques for pattern mining have been proposed in the literature. Retalis, Georgiakakis, and Dimitriadis (2006) suggested that design patterns can be tasks, (3) Thorough analysis of e learning system functionalities, (4) Literature review of pedagogical strategies, (5) Analysis of learner log files, and (6) Learn from existing published patterns. I considered all approaches suggested by the authors in the design pattern mining except the analysis of learner log files as this was unav ailable to me. Table 3 1 shows the approaches to mine the design patterns of computer science

PAGE 77

77 MOOCs in this study. I revisited those approaches in several phases of the complete lifecycle to collect further information that could strengthen the proposed de sign patterns. Table 3 1 Design pattern mining approaches Design Pattern Mining Strategy Approach 1. Self observation Performed self observation or introspection in defining patterns. 2. Expert interview Interviewed instructional designers of computer science MOOCs to obtain in depth information experiences, expertise, and knowledge in designing open online courses. 3. Artifactual study Registered for the computer science MOOCs, went through them, analyzed the functionality, and coded the design patterns. 4. Literature review Reviewed the literature that emphasis on the pedagogical strategies of MOOCs and also to identify common proble ms of open online courses. 5. Adaptation of existing published patterns Learned from existing published design patterns in distance learning, mobile learning, online learning, human computer interaction, and other related areas. Self Observation Retali s et al. (2006) proposed observation of learner tasks in eliciting design patterns for e learning systems. Instead of observing things external to me, I chose self observation or introspection as the first step in defining patterns. Our perspectives influe nce how we interpret a pattern and people may focus on different aspects because

PAGE 78

78 they have different experiences. It was significant to search for design patterns that could be shared and accepted by other designers. In particular, I mined for effective so lutions to recurring problems that I believe were meaningful from my own experience and wrote down the patterns. However, the most challenging task in the pattern definition was the identification of a pattern and its relationship with others. I then self evaluated the quality of each pattern and revised them when necessary. Subjectivity s tatement Software Engineering, both from the University of Malaya, Malaysia. Prior to joining the faculty at the National University of Malaysia, I was a certified network engineer and my primary responsibilities were designing, installing, configuring, maintaining, and troubleshooting of network systems and technology integration functions. Working in an ICT industry for several years taught me much about the fields of software and network engineering, and has allowed me to find the best ways to leverage technology to motivate learners and positively influence their learning My current research focus on the application of software design principles to instructional design aimed to provide instructional designers a method to create high quality designs. When I started working on my final undergraduate project, I developed a patient information system. It was very complicated because I had to design and develop a system from scratch. The same thing happened when I was doing my master thesis support system based on object oriented approach. It had been

PAGE 79

79 software design, Design Patterns Elements of Reusable Object Oriented Software The book made me aware of the design problems I ha d gone through. Little did I know that this concept would lead me to my present research focus. Through the development of open online learning design patterns, I have had the opportunity to speak with instructional designers who have designed and develop ed MOOCs. The only source of knowledge is experience Albert Einstein. This quote made me think more about the importance of documenting design experience in open online courses. Instead of rediscovering a solution, instructional designers could reuse t he experience of others in solving a problem. In their seminal work on object oriented software design patterns, Gamma et al. (1995) said: The purpose of this book is to record experience in designing object oriented software as design patterns. Each desi gn pattern systematically names, explain, and evaluates an important and recurring design in object oriented systems. Our goal is to capture design experience in a form t hat people can use effectively. ( G amma et al., 1995, p. 2) Although Gamma et al. (199 5) discussed about patterns in object oriented software, what they said is also true for open online learning design patterns. Patterns can be constructed at a considerably higher level than source code. Thus, the solutions in this study were expressed in terms of best practices instead of objects and interfaces, but the most essential part of both types of design patterns are a solution to a known problem in a context. I found recurring patterns of instructions in many MOOC platforms. Those patterns resolv e particular design problems and eventually reusable that could minimize redesign. I included patterns that have been applied more than once in different MOOC platforms. I only described design patterns that I believe successful based on my experience. Yet this study documented only a part of what expert instructional designers might know.

PAGE 80

80 Expert Interview The purpose of the interviews was to explore the experience and expertise of instructional designers in designing, developing, and delivering computer science MOOCs for use in Phase 2 Design Pattern Writing. Design patterns can be employed to encapsulate expert knowledge, making it easier to use and organize solutions. The expert interview took more time than self observation because instead of mining my own experience, I conducted in depth interviews with expert instructional designers while I observed them engaged with their MOOCs and asked them to describe their design decisions. I posed specific probe questions to assist experts in describing their tasks. This was known as Cognitive Task Analysis (CTA), a method to capture data for supporting the development of a catalog of design patterns and analytic strategies. I specifically adopted the Critical Decision Method (CDM), a semi structured interview (Hutchins, Pirolli, & Card, 2004). Participants Participants for this study were recruited from computer science MOOCs in spring and summer 2017. Potential participants w ere approached by sending invitations through their emails. To ensure high response rates, a personalized email was sent to educational background and experience. The su rvey instrument was constructed within Qualtrics using the unmodified University of Florida template. There were 1 1 items in the survey questionnaire that took three minutes to complete. The questionnaire used for this survey is located in Appendix B Participants selected in this study met the following inclusion criteria as follows:

PAGE 81

81 The participant is an instructional designer. Possible roles or titles may include professor, course developer, course manager, instructor, graduate teaching assistant, teaching assistant, instructional designer, and others. A participant is included if his/her primary role focuses on the design, development, and/or delivery of computer science MOOCs. This role may include full time or part time employment. The partici is formal and professional in nature. The participant is available for online interviews. Four professionals responded to the survey, and, of those, two people met the inclusion criteria a nd were invited to participate in the study based on their background and experience. Both participants were male and they are 25 34 years old. On average, their total course enrollment ranged from 5,000 to 400,000 students. One participant held a doctoral degree, while t One participant had the title of professor and worked at a public university, also as a course instructor at a MOOC provider Another participant had the title of course developer and worked at a MOOC provide r. Following the interviews, the participants were requested to verify the initial understandings of the responses through member checking (Johnson & Onwuegbuzie, 2004). Interview q uestions As part of the design pattern mining, a semi structured interview protocol of 15 relevant open ended questions was developed. The questions were aimed at capturing the experience, expertise, and knowledge of instructional designers in designing, developing, and delivering computer science MOOCs. These interview questions were presented in two parts. The first part asked about the instructional perspective. For

PAGE 82

82 web in your MOOC? If so, what ways do you incorporate existing web content i nto your participants went through their MOOCs a step at a time, and I asked some questions. Appendix C along with the selected transcript of the intervi ews in Appendix D Procedures The interviews were conducted individually via Skype, an instant messaging tool that enables video conference calls, and recorded for later transcription. The interview questions were emailed to the participants in advance so they had a chance to review the details ahead of time. During the interviews, all participants were asked the same questions in the same order. Validity The interviews were semi structured, with a mixture of open ended questions to mine design patterns of computer science MOOCs. Instead of allowing the researcher to clarify questions, a semi structured interview a lso enables the researcher to probe for additional information. The first draft of the research instrument was carefully reviewed by the chair of the dissertation committee and three expert peers before its administration to the study participants. Based o revised accordingly and ready for a final draft. Validity in qualitative research can be achieved through member checking (Johnson & Onwuegbuzie, 2004). Member checking is usually done at the end of the interview t o ensure the quality of research (Savenye & Robinson, 2004). Prior to the interviews, I asked the participants if they were willing to participate in member

PAGE 83

83 checking. The interpretations of data or my initial understandings of the major points from the int erviews were sent to each of these individuals to ensure that their voices were represented in the findings. Ethical c onsiderations ethical issues that may arise during the stu Prior to the Skype interviews, each participant was sent a copy of the consent form through email. The form indicated that participating in the study is voluntary, thus will not expose the participants to und ue risk. I would keep the consent forms on file for three years following the completion of the research. Beginning to conduct the interviews, permission for the audio and video recording was obtained from the participants. The recordings would be destroye d after two years subsequent to the completion of this study. At the start of each interview, the participants were explained the aim of the study, which was to collect information for cataloging design patterns through exploring their experience, experti se, and knowledge when designing open online courses. The expected length of the interview was informed to the participants, and each interview the introduction, questio ns, and closing remarks of the interviews were read from the pre prepared texts so that each participant was subjected to the same situation. During the interviews, I avoided leading questions and sharing personal impressions, as well as protected sensitiv e information, such as name, organization, and email address.

PAGE 84

84 Artifactual Study Patterns are not only created through cognitive processes, but to some extent they can be detected from patterns in the data or systems (Retalis et al., 2006). Alexander examined existing buildings to create most of his architectural patterns (Kohl, 2013). Many pattern authors in software engineering also used an artifactual approach to contrast and compare similar systems to develop design patte rns (Kohl, 2013). In the field of open online learning, design patterns should guide the design of self regulated learning environments, where one can be self directed enough to learn and stay motivated, as well as make the selection or decision on valuabl e information in virtual settings (Lin & Cranton, 2015). The design of engaging and motivating open online learning environments is a challenging task that involves a significant amount of experience, expertise, and knowledge. Following self observation a nd expert interview, I selected the artifactual study method analysis of the functionality of computer science MOOCs as a means to observe their behaviors for mining design patterns. Ideally, the best source for design pattern mining would have been succ essful and high quality open online courses. MOOCs are primarily offered by prestigious universities and colleges, such as Harvard, MIT, and Stanford, and are taught by the same professors that teach in those prominent campuses. MOOCs provide an opportunit y for instructional designers to learn best practices from others. edX, for instance, has many good instructional practices, including instructional videos with a pause button, practice exercises, online quizzes, automated grading, final projects, and disc ussion forums. Given the current appearance of MOOCs, there is a clear trend that computer science is one of the most popular subjects, and dozens of computer science related

PAGE 85

85 topics are available for the undergraduate and graduate levels. Most of the out standing MOOC platforms, such as Coursera, edX, and Udacity focus on computer science courses. There are many instructional strategies designed to help learners understand the core concepts of computer science, for instance, video lectures, subtitles or tr anscript, lecture notes, assignments and solutions, exams and solutions, and other micro strategies, such as group discussions, stimulus response video, interactive feedback, and online forums. Udacity has recently concentrated exclusively on computer scie nce and technology courses, providing multiple short video lectures with a series of quizzes to keep the student engaged. edX also has listed computer science courses, such as Introduction to Linux, Introduction to Java Programming, as well as Introduction to Computer Science and Programming u sing Python among t en most popular courses of 2016, based on total enrollments and learner ratings. Coursera top universities and o rganizations to offer courses online. Convenience sampling was used to select the MOOC platforms or providers. Table 3 2 shows the MOOC providers that were chosen in this study. Table 3 2 MOOC providers for the design patter n mining MOOC Platform URL 1. Coursera https://www.coursera.com 2. edX https://www.edx.org 3. Udacity https://www.udacity.com After identifying the MOOC providers, design patterns of the most popular topics of computer science MOOCs were mined. Figure 3 1 shows the ten topics of computer science MOOCs offered by Coursera, edX, and Udacity that were selected in this study.

PAGE 86

86 Figur e 3 1 The most popular topics of computer science MOOCs In analyzing the functionality of computer science MOOCs, I examined thoroughly the features provided by the three course providers. This approach aimed to analyze the behavior of each computer scien ce MOOC. I compared the three computer science MOOC providers on specific features, for instance, how they assisted learners in becoming familiar and comfortable with the technologies used. I also identified what types of learner to learner and learner to content interactions were available within each MOOC. In a self directed learning environment, it was important to identify how learners measured or tracked their personal learning progress. Further, I was interested to find out how learners receive feedba ck on their learning. For instance, one of the MOOCs provided automated feedback on most programming exercises and all other types of quizzes frequently. Although MOOCs are free, learners of some particular courses received personalized feedback from instr uctors on their project or cumulative assessment at the end of the course with substantial fees. Most MOOC providers may charge fees for assessment. The effective solutions or recurring patterns found in these MOOCs were recorded as inputs to the developme nt of a catalog of design patterns.

PAGE 87

87 Literature Review There were a number of techniques for pattern mining that have been proposed in the literature, and one of them was the literature review. The advantages of using different pattern mining approaches tha t were often cited are: (1) Different approaches may produce different results even mining from the same source, (2) Assist patent authors to understand both the similarities and differences between those approaches, and (3) Help to discover challenging is sues in pattern prospecting. Following self observation, artifactual study, and expert interview, I reviewed the literature that emphasizes the pedagogical strategies of MOOCs and also identified common problems of open online courses. Since many design pa tterns were mined in this study, it was important to organize them. Table 3 3 shows the classification of design patterns based on the review of the literature on pedagogical strategies. st Principles of Instruction because they were related to teaching and learning activities as discussed in Chapter 2. differently. Some of them consider that a problem is a lesson plan for a lear ning activity that uses some form of simulation based method, while others believe that it only means participating in real wide range of activities which learners first encounter part of the whole task activity and the task is part of the real world problems to be solved by following instructions. The main idea of the problem centered instruction is the components of the task are learned in isolation prior to introducing the real world ta sk to the learners.

PAGE 88

88 Table 3 3 First Principles of Instruction Dimension Design Pattern Description 1. Problem Show Task Task Level Problem Progression Learning is promoted when learners are engaged in solving real world problems. 2. Activation Prior Knowledge Existing Experience Structure Learning is promoted when relevant previous experience is activated. 3. Demonstration Demo Consistency Learner Guidance Relevant Media Learning is promoted when the instruction demonstrates what is to be learned rather than merely telling information about what is to be learned. 4. Application Practice Consistency Diminishing Coaching Varied Problems Learning is promoted when learners are required to use their new knowledge or skill to solve problems. 5. Integration Watch Me Reflection Creation Learning is promoted when learners are encouraged to integrate (transfer) the new knowledge or skill into their everyday life. Adaptation of Existing Published Patterns oriented software design patterns (1995), many published design patterns were available nowadays. I reused des ign expertise and exploited knowledge of patterns in other related areas as part of pattern mining. The reusable design elements were derived from expert experience and backed by theory, but could be immediately applied in new situations. Sharing and reusi ng good design practices could save time, and resources, as well as helping to get the right design faster. Table 3 4

PAGE 89

89 shows the adaptation of existing published patterns. In general, I mined the most dominant patterns in those areas. Table 3 4 Adaptation of design patterns from other domains Pattern Author Domain Pattern Name 1. Al Samarraie & Ahmad (2016) Mobile Learning User Interface Social activities 2. Warburton & Mor (2015) Open Learning SHARING WALL FISHBOWL SPARKING FORUM PARTICIPATION 3. Kohl (2013) E Learning Prepared Example 4. Zitter et al. (2009) Collaborative Learning Connect to an outside online community Introduce primary boundary objects at the start of a project 5. Persico et al. (2009) Collaborative Learning Knowledge building bricks 6. Retalis et al. (2006) Collaborative Learning ANNOTATION_ON_POSTED_MESSAGES 7. McAndrew et al. (2006) Learning Management Systems Discussion group 8. Derntl & Motschnig Pitrik (2005) Blended Learning Generic Evaluation 9. Eckstein (2000) Pedagogy Ask your neighbor Challenge Wrap up 10. Bergin (2000) Computer Science Education Early Bird Spiral Student Design Sprint

PAGE 90

90 Phase 2: Design Pattern Writing The purpose of design pattern writing was two fold: (1) To provide a standard template for instructional designers to express their design expertise to other instructional design professionals, and (2) To describe and organize design patterns for open onli ne learning using a standard template. Design patterns are written at a high level of abstraction in which the problems are easy to recognize instead of being written in a specific way. The underlying idea behind design patterns is to guide rather than pre scribe, a feature that makes them potentially a useful tool for designing effective learning courses (Rohse & Anderson, 2006). Design patterns are intentionally incomplete as they share expertise, but do not limit creativity (Gamma et al., 1995). For inst ance, an instructional designer that encounters an unfamiliar topic can still recognize the problem description as stated in the design pattern, but, the specific activity will be influenced by their expertise, experience, and knowledge. Implementing desig n pattern requires the instructional designer to engage with the pattern and interpret the pedagogic descriptions to their own needs. As the instructional designer finds potential ideas and new directions, the nature of their topic changes, so do the subse quent activities. Further information and examples instructional designer the opportunity to create activities for the topic at hand. Besides providing facts, design patterns al so describe the design knowledge in a more descriptive way. I focused on the practicality of design patterns since the goal of design patterns was the description of reusable solutions to recurring problems. During Phase 2, design patterns were described a nd organized in a standard format. To reiterate, design patterns are a tool for documenting and reusing previous solutions. A

PAGE 91

91 template to describe and organize design patterns for open online learning was modified from the Gamma et al. (1995) and Alexander (1979) pattern structures. In order to reuse design patterns, it is important to effectively document them using a consistent format that helps novice designers in scanning and selecting patterns. Each design pattern was named, explained, and described sy stematically. Figure 3 2 Figure 3 3, and Figure 3 4 illustrate a template to describe open online learning design patterns and the explanation for each section. There are eight sections in the template of design patterns: Pattern Name, Also Known As, Prob lem, Context, Solution, Related Patterns, Examples, and References. Pattern Name The pattern name provides a shared vocabulary for both the pattern writer and the content expert that enables unambiguous communication (Kohls & Uttecht, 2009). Hence, it is important to give a descriptive and unique name for identifying and referring the pattern, with the purpose to describe a design problem, its solutions and also consequences. According to Gamma et al. (1995), naming a pattern immediately increases design v ocabulary. Likewise, Derntl and Motschnig Pitrik (2005) pointed out that the pattern name is a meaningful descriptor for the pattern to convey its essence. Kolfschoten et al. (2010) suggested that patterns should have catchy names to describe and remember them. Also Known As It refers to act of giving other names for the pattern, if necessary. Problem This section provides a brief description of the design problem at hand. Gamma et al. (1995) described problems as motivations, the scenarios that help us to better understand the abstract description of the pattern and then illustrate how the elements of the pattern could solve the problem. In contrast, Alexander (1979) emphasized on the forces, such as issues or concerns that are acting on the Figure 3 2 Design pattern template 1

PAGE 92

92 problem, and the rational for selecting a particular solution. Context This section explains a context for the problem in order to avoid an over generalized solution (Rohse & Anderson, 2006). Also, a context can be defined as a description of the indicators/factors that influence the use and implementation of the solution (Z itter et al., 2009). Describing the context is crucial as it helps to communicate the nature of the problem and its solution. Designers can adapt the invariant aspects captured by the pattern to a variety of design problems within a similar context (Bianco Robinson, Metcher, & Hendy, 2011). Gamma et al. (1995) referred to this as applicability, provides situations for which the pattern could be suitable. Further, they emphasize that it is important to give examples of poor designs, which the pattern can so lve, and an explanation of the sources of the problem. Solution This section refers to a description of the solution proposed by this pattern that addresses the problem and context, including the best practices that show how the problem can be solved (Zitt er et al., 2009). However, Gamma et al. (1995) and Derntl and Motschnig Pitrik (2005) described solutions as intents, a brief statement pertaining to the scenario or situation the pattern addresses. Related Patterns It refers to a list of closely related patterns as references that support the solution. Examples This section gives e xamples of putting the pattern into practice, especially pictures to illustrate the use of pattern in actual courses and to clarify the invariant aspects of different cases (Bok horst, et al., 2014). Gamma et al. (1995) suggested to provide at least two concrete examples found in different contexts. Thus, each example helps the pattern reader to understand the core and meaning of a pattern. Similarly, Kohls and Uttecht (2009) cons idered this section as indispensable since examples are the most powerful medium to convey the message of a pattern and provide an implicit way of giving evidence for the Figure 3 3 Design pattern template 2

PAGE 93

93 pattern. According to schema theory, a learner requires multiple examples to induce a schema. References The supporting research presents a list of references to literature or related work in order to support the use of pattern. Figure 3 4 Design pattern template 3 The design pattern template provides a standard structure to the information, making design patterns easier to learn, share, and use. In particular, each pattern allows designers to express their design expertise into pre purpose of stimulating abstraction. The design patterns then should be presented in the same narrative structure and ordered hierarchically, that contained those eight sections and elements. However, this design pattern could only make sense when it is used with the other related design pat terns. In other words, each design pattern was not isolated, but it was interrelated to other design patterns in the catalog. Implementing design patterns not only requires the designer to engage with the pattern, but also to interpret the narratives consi sting of pedagogical descriptions and pictures to meet their needs. E xamples and references to related work give the instructional designer ideas to design their courses. Design patterns are an effective way to express design expertise and make explicit th e mental patterns that exist within expert designers in solving recurring problems. Ideally, the instructional design patterns should focus on the problems of the end users, rather than the problems of the designers (Frizell & Hubscher, 2002). The learner participating in the learning experience is considered the end user.

PAGE 94

94 Data Analysis Following mining, the patterns were further examined through content analysis. Considering the purpose of this study, the qualitative content analysis was selected as an ap propriate analysis approach to review and interpret patterns. Content analysis is a method for providing new insights, knowledge, representation of facts, and a practical guide to action, as well as for making valid inferences from data to their contexts ( Krippendorff, 2012). Four main stages were involved in analyzing the data as proposed by Bengtsson (2016) : (1) Decontextualization, (2) Recontextualization, (3) Categorization, and (4) Compilation. To ensure the trustworthiness of the analysis, each stage was visited several times during data analysis. Figure 3 5 illustrates the process of the qualitative content analysis from data collection to presentation of the result. In qualitative content analysis, data are presented in the forms of themes and words (Bengtsson, 2016), enabling the researcher to interpret results and draw conclusions. The results of these analyses are detailed in Chapter 4. I read through the transcribed interviews repeatedly to familiarize myself with the data and to get a general i dea of the whole text. The transcripts were broken down into smaller meaning units and resulted in 25 relevant units considered for analysis. I the context. Berg (2001) r ecognized this procedure as an open coding process. In the assembled into groups. To ensure the reliability of the analysis, I used a coding list to reduce a cognitive c hange throughout the analysis process. Codes were generated inductively, which the list was created during the analysis process and changed as the study progresses.

PAGE 95

95 Figure 3 5 Qualitative content analysis process I used NVivo11 to facilitate and speed up the coding process by finding codes and grouping data into their own categories. After the meaning units have been reviewed if all aspects of the content have been covered in relation to the purpose of the study. I read again the original transcripts together with the list of meaning units. I made sure that the important information was included in the analysis while t he unimportant ones were excluded. Meaning units were condensed prior to creating categories, which insignificant words were removed without losing the content. The coded data were basically divided based on the interview questions. Categories and themes w Compare the original data Coding inductively Find the underlying meaning Condense the meaning units Stage 1. Decontextualization Identify meaning units Stage 2. Recontextualization Include content exclude worthless Stage 3. Categorization Identify homogenous groups Stage 4. Compilation Draw conclusions, interpret result DATA ANALYSIS RESULT PRESENTATION DATA COLLECTION

PAGE 96

96 Krippendorff (2004), the data should not be in between categories and themes or fit into more than one group. Theme refers to the underlying meaning of the texts in order to answer a research q uestion (Bengtsson, 2016). Summary The use of design patterns has evolved to solve problems often encountered in architecture, prominently in software engineering, and recently in the instructional design communities. Design patterns in this study consiste d of reusable solutions generalized from a number of successful design cases and best practices. This study proposed a template to the instructional design community on how to effectively document and communicate design patterns in an open online learning context. Instructional designers can use this template to express their design expertise to other instructional design professionals and also make use of design patterns in practice.

PAGE 97

97 CHAPTER 4 RESULTS The purpose of this research study was to develop a c atalog of open online learning design patterns for computer science courses, a template for documenting and reusing successful design solutions. The study also sought to explore different approaches that contribute to the rich description of the catalog of design patterns. The research questions of this study were: 1. To what extent do the design patterns exist within the Massive Open Online Courses in computer science? 2. How is a catalog of design patterns for open online learning constructed? During in depth interviews, study participants described their experiences and perceptions in designing, developing, and delivering computer science MOOCs. The research findings were also based on the analysis of the following data sources: self observatio n, functionality of computer science MOOCs, literature on pedagogical strategies, and existing published patterns in other related areas. Background The interview participants of this study were comprised of two instructional designers from the leading MO OC providers. They aged between 25 to 34 years old and both were male. One participant holds a Doctor of Philosophy (PhD) and one Study Findings This chapter consists of two parts. The first part is the study findings from expert interviews, and the second part is the overall findings in the form of the catalog of open online learning design patterns for computer science courses.

PAGE 98

98 Use of Online Affordances In order to understand the interactivity, communication, and co llaboration within MOOCs, I asked participants questions about what types of interactions are available in their MOOCs and how they facilitate those interactions. One participant explained the learner to learner interaction in his MOOCs: Our class is very Every week, we assign students three or four assignments from their t for any actual grading. We just used a grade to put them in a position of approaching things from that angle. So that is the main thing for learner to learner forums. We also post ex emplary assignments for a chance at peer feedback, so students kind of comparing what they did to the best in the class. Another participant mentioned that besides discussion forums for communication, learners can even connect to one another in person thr ough Udacity Connect. Learner to learner interaction is the most common type of communication that exists in open online education (Gameel, 2017). This interaction refers to the exchange of information and ideas among learners geographically dispersed, eit her through asynchronous (discussion forums) or synchronous communication (chats). Table 4 1 and Table 4 2 illustrate the content analysis process from condensed data to category. Table 4 1 Transc r ibed interview with a participant regarding the interaction among learners part 1 Condensed Meaning Unit Code Category We use a peer feedback system named Peer Feedback. Every week, we assign students three or four assignments from their peers, for them to review, grade based on a rubric, but we did not use it for Interactivity, communication, and collaboration Learner t o learner interaction

PAGE 99

99 Table 4 2 Transcribed interview with a participant regarding the interaction among learners part 2 Condensed Meaning Unit Code Category any actual grading. We also post exemplary assignments at Peer Feedback, so students can compare what they did to the best in the class. Instead of the peer feedback, we also have forums for learner to learner interaction. Interactivity, communication, and co llaboration Learner to learner interaction Prior work found that many learners struggle with motivation and self regulated learning in MOOCs. Hence, I asked participants how learners measure or track their personal learning progress. One participant we display their progress on the ways of tracking personal learning progress: One is progress through the material, just kind of how far they are. And one a percent through this course. And last time you were here, you were on this gh this lesson. So they visualize that kind of progress in that way. Also, we create them quizzes so they know how of the material. They also have tests and projects usual kind of tr appings. For learner to content interaction, it is mostly based around online recorded d that they used video for quiz introductions, quiz solutions, tutorials, intros, outros, and explanations, and text for supporting information. While interactive programming quizzes are used for student self assessment. The content analysis process from c ondensed data to category as in Table 4 3

PAGE 100

100 Table 4 3 Transc r ibed interviews with participants regarding the interaction of learners with contents Condensed Meaning Unit Code Category First is progress through the material of how far they are. Second is h ow well they understand it. In terms of how far they are, you have got a percent through this course and last time you were here, you were on this lesson, and you are halfway through this lesson. So they visu alize that kind of progress. Also, we create them quizzes so they know how well they understand the material, and demonstrate their understanding. They also have tests and projects. Our three kinds of video are one week home head chats, which is I am just standing in a studio, it is white behind, I am talking directly to the camera. We filmed in my class, at my house, and at our office. We also do what we call tablet recording, which is basically I will have a PowerPoint presentation. I will point or motion to things, and they will see my hands interact with the content. That is usually for diagrams. Track personal learning progress Media and technologies Learner to conten t interaction Learner to content interaction Learners usually received automated feedback on most programming exercises and all other types of quizzes frequently. Learners who were enrolled in the Nanodegree received personalized feedback from a grader on their project or

PAGE 101

101 cumulative assessment at the end of the course. Due to open access and massive enrollments, instructors could interact with learners in a way that does not require them to provide individual messages to each learner, such as a broadcast method. Although little learner to instructor interaction happens in MOOCs, one participant explained that they interact with learners through free online collaboration platforms that have been integrated int o the MOOC. So the two main ways are our forum and Slack. The place we mostly direct things are on the forum. We have a team of TAs, who are available to answer questions. Their main responsibility is actually creating ble to answer questions on Piazza. as the main tool. I also have three times weekly office hours. So we also e have those three times a week. Realistically, students learn pretty quickly always kind of at my computer. So the office hours be ing scheduled kind of disappear pretty quickly because Piazza is a free Q&A web service that helps learners interact with professors, teaching assistants, and peers. Piazza is designed with a wiki and forum style formats that can be integrated into most Learning Management Systems (LMSs). S ome key features of Piazza including anonymous posting to promote learner participation, highlighting questions and posts that require immediate action, and creating online polls. While Slack is a collaboration tool with basic features such as messaging, v ideo calling, file sharing, and archiving. Founded by Steward Butterfield in 2013, Slack provides many Internet Relay Chat (IRC) features, particularly persistent chat rooms organized by discussion topic. Table 4 4 illustrates the content analysis process from condensed data to category.

PAGE 102

102 Table 4 4 Transc r ibed interview with a participant regarding the interaction between learner and instructor Condensed Meaning Unit Code Category The two main ways are forum and Slack. The place we mostly direct things are on the forum. We have a team of TAs, who are available to answer questions. Their main responsibility is actually creating assignments, but they are also available to answer questions on Piazza. And I do as well. We also communicate via Slack, just becaus e it is easier on me via chat. Realistically, students learn pretty quickly that if they message me on Slack as I am usually there. Interactivity, communication, and collaboration Learner to instructor interaction Content and Course Material MOOC can be effective at communicating difficult material and abstract concepts of computer science. Hence, it is important to evaluate the courses since well designed MOOCs could be as effective for learning as traditional classes. I asked participants to des cribe the process of ensuring the effectiveness of MOOCs before they are released. One participant explained the process: We used to have a process by which we would do beta testers, things like that. We do beta test, we just do it very early on. We discov film because the cost of re to justify the cost. So instead we feel more into peer reviewing scripts and reviewing things in ad reviewed them after it is produced. We will work on a review with my crew as we go along. Our process is all about partnering a professor wi th instructional designers who experts in doing online courses. And so they

PAGE 103

103 seeing at the end that oh, sorry, this is awful, go back and do everything again. Beta testing or pre launch testing is an important function of quality control and a critical step before a software product is released (Sekhon & Hartley, 2014). Beta testing then can be used as one of the MOOC quality assurance approaches to help uncover content errors, level of user involvement, software bugs, and usability (Sekhon & Hartley, 2014). Generally, learners will participate in testing of the MOOC materials before they go live to improve the quality of courses by catching and pointing out mistakes in video lectures and quizzes P rofessors will also peer review the materials to assure the quality of the content. The content analysis process from condensed data to category as in Table 4 5 Table 4 5 Transc r ibed interview with a participant regarding course effectiveness Condensed Meaning Unit Code Category We used to have a beta test and we did it very early on. We discovered it is basically once you filmed, you are very unlikely to go back and re film because the cost of re filming is so high. Instead, we feel more into peer revi ewing scripts and reviewing things in advance before they are filmed. So we never really reviewed them after it is produced. We will work on a review with my crew as we go along. Our process is all about partnering a professor wi th instructional designers who experts in doing online courses. It is ongoing expertise as opposed to waiting to film it all and seeing at the end, go back and do everything again. Quality control MOOC testing

PAGE 104

104 I also asked the participants a question on how they obtain learner feedback during and after the course. One participant explained two main ways to obtain learner feedback: a starter course, quarter course, mid course, and end of course survey. That we give in the weeks. So our semester is 17 weeks. We give the surveys at week 1, 5, 9, and 17. We used to give one at 13, but then we information At the end of course survey, we ask different questions so questions like how would you rate this class against others from the strengths and weaknesses? As well as open ended questions, you know that are meant to let a student improve mid semester. He also indicated that they provided a folder called Feedback Box that allows learners to post any feedback to the i nstructors. students will email and say hey, I think X could use improvement. And the professor tried to improve X, and they found out oh all the other students something. Why was it changed? And so when they post on Piazza, we get an email, we can hear from other students whether they support it, Table 4 6 and Table 4 7 illustrate the content analysis process from condensed data to category. Table 4 6 Transcribed interviews with participants regarding learner evaluation part 1 Condensed Meaning Unit Code Category We have a folder called Feedback Box, where any time you have any feedback for us, please just post there. I like that method because we get to see other get an email, we can hear from other students whet her, they support it, do not like it, or not. Learner feedback MOOC evaluation

PAGE 105

105 Table 4 7 Transc r ibed interviews with participants regarding learner evaluation part 2 Condensed Meaning Unit Code Category We have four surveys for each course of the semester. It is a starter course, quarter course, mid course, and end of course survey. Our semester is 17 weeks. We give the surveys at week 1, 5, 9, and 17. We used to give one at 13, but then we just kind of realized that at week 13, we are not gaining any real new information. In the end of course survey, we ask different questions so they are a little bit more substantive. So we dropped the three quarters course survey. On those surveys, they answer those questions like how would you rate this class ag ainst others from the strengths and weaknesses? As well as open ended questions, you know that are meant to let a student improve mid semester. Learner survey MOOC evaluation Instructional Strategy and Learning Outcomes Learning outcomes in a MOOC platform may not be similar to those in traditional or regular online education. Understanding the factors influencing learning outcomes, particularly learning activities and teaching context are significant as they are important steps t owards designing effective open online courses. I asked participants which approaches to instruction have proven to be the most effective in the implementation of the MOOCs. One participant mentioned about interactive and constant activities for learners. PowerP oint and more friend sitting next to you explaining things. I used to teach science. Hands on

PAGE 106

106 is the explained: I would say that the most important thing is actually a match between content and instruction. So it seems in instances where professors try to content. And so for example, in my HCI focus on those video lessons. We go on with this. In my Educational where you w ork closely with a particular TA. And so I think the number one point is, the most important thing is the match the instruction to the learning goals and to the assessments that are going to be used. I guess matching assessments to learning goals and their learning goals to instruction style. undergrad class, I have rapid feedback the instantaneous assessment of we can do when we can do it. I have seen professors try to do that in material where it doesn a lot of effectivenes s. works for your material. Just do that. Other than that, a lot of it is fostering student community. Fostering student activity, responsiveness, just because students feel tha playlist with quizzes next to it. Most MOOCs provide an automated grading method for quizzes, such as true/false, multiple choice, and short answer sets. To investigate the types of assessment used for gr ading, I asked participants a question about how they assess the learning outcomes or results of those that participate in the MOOC: Technically the tests are automated, so the tests are all multiple choice. boils down. And that actually gets us an objective and outlet variable. human created the output there is not quite as reliable as you know, the objective true or false. So we also use those.

PAGE 107

107 engagement numbers... number of learner s progressing into the He further we measure the number of students who get jobs after completing the Table 4 8 and Table 4 9 Table 4 8 Transc r ibed interviews with participants regardi ng instructional strategy and learning outcomes part 1 Condensed Meaning Unit Code Category The most important thing is actually a match between content and instruction. So it seems in instances where professors try to teach, I have heard that X works best, but X does not work best for their content. For example, in my HCI class, it is very video heavy. In my Educational Technology class, there is basically no video whatsoever. It is all project based, it is all open ended. It is based on more of a mentorship model, where you work closely with a particular TA. So I think the most important thing i s to match the instruction to the learning goals and to the assessments that are going to be used. Technically the tests are automated, so the tests are all multiple choice. It is basically 100 true or false questions. And that actually gets us an object ive and outlet variable from the course. Other than that, it is the assignments. The assignments do not change that much semester to semester, but because they are human created, the output there is not quite as reliable as the objective true or false. Eff ective instruction Automated grading Practice consistency Learning outcomes

PAGE 108

108 Table 4 9 Transcribed interviews with participants regarding instructional strategy and learning outcomes part 2 Condensed Meaning Unit Code Category The rapid feedback is probably the most effective thing. We cannot really do it in my HCI class because it is all open ended. But my other class, I have rapid feedback, the instantaneous assessment of where you are, when you can move on. Other than that, a lot of it is fostering student community. Fostering student activity, responsiveness, just because students feel that they are actually in touch with people, not just a playlist with quizzes next to it. Rapid feedback Learning outcomes Knowledge Activation Activation involves more than just enabling learners to recall prior knowledge or provide relevant experience. To stimulate the development of the schemes and mental models, learning activities should be designed to help learners integ rate the new knowledge and skill into new experience. Hence, I asked participants what design features do they use to promote the activation of prior knowledge. a ing them Talk about X, Y, and Z. And then at the end yo about X, Y, and Z, and remember we said this, and go look at this. We designed the entire course that way. So unit one is like the introduction about all of this. Unit five is, in this entire course, we talked about this. Within each unit, we do that too. He also explained that the first lesson in each unit is the introduction to the unit:

PAGE 109

109 lesson, the first video talked about this... you know, talked about this lesson. We probably use this a lot to connect to other areas of the course material, as well as things that students might psychology we talked about last lesson. And notice also that what we instead of teaching the content. We also have we have various checkpoints that basically say try and see how this applies in the real world, you know, so we try to have them connect it out to something beyond the course. And our assignments often do that too. Basically say, think of an interface that you use in everyday life, analyze it from the perspective of cognition. And so it kind of connects what they know or did before starting the course. practice and students constantly work with s in Table 4 10 and Table 4 11 Table 4 10 Transc r ibed interviews with participants regarding instructional strategy part 1 Condensed Meaning Unit Code Category The main thing is about giving a good speech. Tell them what you are going to tell them. So it is basically today, I am going to talk to you about X, Y, and Z. Talk about X, Y, and Z. Then at the end you would say, today we talked about X, Y, and Z, and remember we said this, and go look at this. We designed the entire course that way. So unit on e is like the introduction part of the entire course. We say in this entire course, we are going to talk about all of this. Unit five is, in this entire course, we talked about this. Within each unit, we do that too. Approach to instruction Prior k nowledge

PAGE 110

110 Table 4 11 Transcribed interviews with participants regarding instructional strategy part 2 Condensed Meaning Unit Code Category The first lesson in each unit is the introduction to this unit. Here is what we are going to talk about in this unit. Within each lesson, the first video here is what we will talk about in this lesson, here is what we talked about this, talked about this lesson. We probably use this a lot to connect to other areas of the course material, as well as things t hat students might have elsewhere. So we will say in the introduction to the lesson, in this lesson we are going to talk about representations. What you will find is that these heavily leverage the limitations of human psychology we talked about last less on. And notice also that what we talked about in this lesson, you are going to see again in the next lesson. So it kind of half set aside places where we are talking about the content instead of teaching the content. We also have various checkpoints that b asically say try and see how this applies in the real world, so we try to have them connect it out to something beyond the course. And our assignments often do that too. Basically say, think of an interface that you use in everyday life, analyze it from th e perspective of cognition. And so it kind of connects what they know or did before starting the course. Approach to instruction Structure

PAGE 111

111 Further, I asked participants about what knowledge, skills, and dispositions are needed for the successful completion of the MOOC. One participant mentioned about basic web development skills, good attitude, and willingness to try things. Another participant described: rse, they need very little s towards a curious mind set. By giving an example of his Human Computer Interaction (HCI) course, he indicated: Then to complete the class with everything that you need to development is a kind of two general assessments of skills. You need to know the ability to analyzing a use r interface and reassign from the perspectives of principals of teach. And that means different things, so that means understanding how known theories and ideas apply different places. And it means understa nding how to go through a process of designing something, testing with the user, looking at the results, revising it. So both kind s of the mechanics of being able to do it, and then the workflow of doing it, basically doing it well. Table 4 12 and Table 4 13 illustrate the content analysis process from condensed data to category. Table 4 12 Transc r ibed interviews with participants regarding course completion part 1 Condensed Meaning Unit Code Category I would say that generally when they are entering a course, they need very little prior knowledge. We cannot require hard computer science knowledge because there are people who are not taking computer science course s I would say that they need to have a disposition towards a curious mindset. Successful co urse completion Prior knowledge

PAGE 112

112 Table 4 13 Transcribed interviews with participants regarding course completion part 2 Condensed Meaning Unit Code Category Then to complete the class with everything that you need to development is a kind of two general assessments of skills. You need to know the ability to analyzing a user interface and reassign from the perspectives of principals of teach. I guess that is th e overall skill. So that means understanding how known theories and ideas apply different places. And it means understanding how to go through a process of designing something, testing with the user, looking at the results, revising it. Successful course completion New experience Given that MOOCs are becoming increasingly popular worldwide, factors So the only ones who are taking the course are the ones who are interested in the course. need that credit. This is a link to a student run review set, where students available, so you can see what everyone is saying. Based on th at, I would speculate that the reasons why students take this course. One is that want to take another one with me. One is that the topic is very applicable, com puter science required for it. siest, but probably the bottom 30 percent in terms of difficulty. And so I think there are students who take it because they want to finis h and they will take the easy class in every session. And then the other is, all our students do a report that they find this kind of development to whatever they do professionally. So working in computer science, anyone could benefit from this perspective So I think those are the four main reasons I see.

PAGE 113

113 Another participant believed that the reasons learners participated in the MOOC were to improve their web development skills and build websites faster. The content analysis process from condensed data to category as in Table 4 14 Table 4 14 Transc r ibed interviews with participants regarding motivation Condensed Meaning Unit Code Category So the only ones who are taking the course are the ones who are interested in the course. This is a link to a student run review set, where students have reviews on one of the courses in the program. And it i s public ly available, so you can see what everyone is saying. Based on that, I would speculate that the reasons why students take this course. One is that they have taken my earlier courses and they have liked them. And they just want to take another one with me. One is that the topic is very applicable, no matter what you are going into. So it is very accessible, since there is no computer science required for it. It is also weighted as one of the easiest, but probably the bottom 30 percent in terms of difficulty. So I think there are students who take it because they want to finish and they will take the easy class in every session. And then the other is, all our stu dents do a report that they find this kind of development to whatever they do professionally. Working in computer science, anyone could benefit from this perspective. Personal interest Prior knowledge

PAGE 114

114 Transfer of Learning previously acquired knowledge and skills to new contexts. Course design features are among the important factors for promoting transfer to real world situations. One participant explained, We have lots of checkpoints and our assignments are based on that. The projects are both things that students are encouraged to take or that they care about in the real world. Men tally encourage students that you know, to basically say if you have a UI design thing you have to do for work, feel free to do it for this class. We like that, because then you get feedback from the creator, you get feedback from your peers, and you also get to use it for work. So we promote a lot of those connections out to students to use in the real world. And this class an online course, so of course you are. So in The emerging factors in MOOC design were further explored by a probing question to understand how they provide adequate practice for learners to apply new knowledge or s kills for a variety of problems. One participant emphasized the value of learning activities. demonstrating that you have the capacity to master the material if that makes sense. the entire course is basically a series of challenges where students are given real websites and Table 4 1 5 illustrates the content analysis process from condensed data to category.

PAGE 115

115 Table 4 1 5 Transc r ibed interviews with participants related to transfer of learning Condensed Meaning Unit Code Category We have lots of checkpoints and our assignments are based on that. The projects are both things that students are encouraged to take or that they care about in the real world. Mentally encourage students that you know, to basically say if you have a UI des ign thing you have to do for work, feel free to do it for this class. We like that, because then you get feedback from the creator, you get feedback from your peers, and you also get to use it for work. So we promote a lot of those connections out to stude nts to use in the real world. And this class it is very easy, just because it is human interaction, everyone is interacting with computers all the time. I think those mostly come through the assignments right now. I do not know if I would quite call it a dequate for mastering material as adequate for demonstrating that you have the capacity to master the material if that makes sense. Real world problem Mastering material Creation Demonstrate new skill or knowledge The first research question was answered through the design pattern mining. To reiterate, the design pattern mining involved five main approaches, including expert interviews. Interview participants contributed differing amounts of information to the five

PAGE 116

116 themes that comprised the narrative. One participant talked at length on two or three themes, and another participant made nearly equal contributions across all five themes. ond research question was answered through the catalog of open online learning design patterns for computer science courses. Design Patterns as the link between Theory and Practice The goal of this section was to structure the information presented in the catalog of design patterns in a way that assists instructional designers in organizing knowledge by principles. Since there were many instructional design patterns that have been identified in this study, it was important to organize them. Figure 4 1 provi des a theoretical framework for this study and the classification of design patterns. This figure classifies design patterns so they can be referred to families of related patterns. The classification helps designers learn the design patterns in the catalo g faster, as well as it can direct efforts to find new design patterns, if any. Blue oval represents the dimension or category of patterns, while the white oval represents the design patterns. Taken together, there are five dimensions of patterns with 15 design patterns for open online learning. The catalog of design patterns is a set of pattern that has a relatively low level of structure and classification. A pattern language then is the coherent and interrelated design patterns that can be used to solve dimensions and design patterns were adapted from (2012) and these principles were a fundamental principle for the dimensions of patterns and design pattern s.

PAGE 117

117 Figure 4 1 Theoretical framework and classification of design patterns Catalog of Design Patterns The Educational Modeling Language (EML) is the most commonly used modeling language in instructional design. In an attempt to visualize the learning scenario of each design pattern, this study adopted the EML of the Retbi, Merrouch, Idrissi, and Bennani (2012). Figure 4 2 depicts the elements of the visual EML. Action is the notion of sequence, Activity is connected to FlowConnector Branch is the Pattern mining approache s : 1. Self observation 2. Expert interview 3 Artifactual study 4. Literature review 5. Adaptation of existing published patterns ACTIVATION DEMONSTRATION APPLICATION INTEGRATION PROBLEM Show task Task level Problem progression Watch me Reflection Creation Previous experience Structure Practice consistency Diminishing coaching Demonstration consistency Learner guidance Relevant media New experience Varied problems Pattern searching Pattern mining Pattern writing refining search Pattern writing Instruction

PAGE 118

118 condition for the action sequence, FlowConnector illustrates the adaptive action sequence, and Synch ronization facilitates the implementation of actions. Figure 4 2 Elements of the visual Educational Modeling Language ( Retbi et al. 2012) Show Task Design Pattern Pattern Name Show Task Also Known As Learning Objective Category Problem Context I ntroducing a real world problem or whole task to the learners. The instruction shows learners the task that they will learn to do or the problem that they will learn to solve when they finish a lesson Problem Instructors state learning objectives at the beginning of a lesson. L earning objectives are often written in a form of observable activities, for instan will be able to Acti on Activity FlowConnector Branch Synchronization

PAGE 119

119 objective statements and what they will be able to do once they successfu lly complete a unit of learning. Forces Instead of stating abstract learning objectives, learners can understand the whole task better when beginning a lesson by showing them how to solve a real world and authentic problem. Solution Demonstrate the fir st and complete whole task in a sequence that outlines the learning objective for the lesson. This demonstration should be the easiest task in a sequence. In practice, the first problem in a sequence should be the easiest one so the learner progressively constructs a knowledge base by first understanding a basic concept and skill before moving to a more advanced problem. Furthermore, demonstration of the specific steps to solve a real world problem or whole task provides a better orientation to the instru ctional materials as opposed to abstract objectives. The components of the whole task are demonstrated at a high level so as not to overwhelm learners with too many details. For instance, after identifying a whole task, the initial instruction would be a f ully worked example to show learners the steps in a problem solving process, and thus reduce the cognitive load on their working memory. Consequences Worked examples help learners focus on the essential parts of the problems. Learners first study a worke d example, then they solve a problem during independence practice. Providing learners with an effective cognitive support helps them to solve problems faster and avoid frustration.

PAGE 120

120 Learning Scenario Figure 4 37 illustrates t he metamodel of the Show Task learning scenario Examples The instruction for an open online course starts by demonstrating a complete whole task of a real world problem, but the easiest version in a sequence. This initial demonstration provides an overview of all the whole task components that forms the learning objective for the lesson. Refer to the examples, the abstract learning objective for this course is to introduce the fundamental ideas in computing, and to teach learners how to read and write computer program s. The instructor starts the lesson by introducing a real world problem in the context of building a we b search engine as in Figure 4 3 Within the online content, an instru ctor video such as in Figure 4 4 can be used to show learners the complete whole ta sk that they will learn to do or the problem that they will learn to solve. Building a search engine involves three main parts: finding data, building an index, and ranking pages. Further, the instructor demonstrates what learners will be able to do as a r esult of completing a c ourse or lesson as in Figure 4 5 Figure 4 3 An instructor video introduces a real world problem to the learners

PAGE 121

121 Figure 4 4 An instructor video shows learners the complete whole task that they will learn to do Figure 4 5 An instructor video demonstrates what learners will achieve by the end of the course Related Patterns Show Task is often implemented with Task Level and Problem Progression. Supporting Research 1. Merrill, M. D. (2002). First principles of instruction. Educational Technology Research and Development, 50 (3), 43 59.

PAGE 122

122 2. Silber K. H., & Foshay, W. R. (Eds.). (2010). Handbook of improving performance in the workplace San Francisco, CA: Pfeiffer 3. Sweller, J., van Merrienboer, J. J. G., & Paas, F. G. W. C (1998). Cognitive architecture and instructional design. Educational Psychology Review, 10 (3), 251 296. Task Level Design Pattern Pattern Name Task Level Also Known As Problem Level Category Problem Context Effective learning requires learners to solve an authentic, real world problem or whole task that allows them to explore and construct relevant concepts. Thus, it is important for learners to engage in the four levels of performance: the problem, the task, the operation, and the action level. Acquiring knowledge and skill in the context of the whole task helps learners to structure mental models about how these individual components are integrated into a complete performance. Problem Most instructions foc us on teaching the prerequisites and decontextualized skills prior to introducing the real world problems to the learners, which can greatly reduce their motivation to learn the materials. Learners do not feel a sense of ownership if the problems to be sol ved are uninteresting, irrelevant, and unappealing. The whole task or problem can be taught by breaking them into components. However, teaching

PAGE 123

123 components of the problem in isolation do not really help learners learn to solve the real world problems in a m eaningful way. Forces Learners should not be engaged only in the operation or action level, but also at the problem or task level in order to promote learning. Solution The instruction for open online courses should consider teaching the whole task, rath er than individual task components, such as isolated actions or operations. This instructional approach or known as the contextualization of basic skills can be used to create explicit connections between the problem, the tasks for solving the problem, the operations that consist of the required tasks, and the actions that involve the operations. For example, website development skills are taught with direct reference to topics covered (how the web works, databases, building a basic blog, APIs, etc.) in a w eb development course. The application of skill components should take place after the demonstration of the first whole task in a sequence. New material then can be presented in small steps and practice can be provided after each step, but again this compo nent should be related to the whole task. The practice is used to assess the knowledge or skill that learners have acquired at the end of the lesson. Consequences Following the contextualized instruction, learners will be likely to transfer the skills to solve a real world problem when the instruction is connected to those topic areas rather than taught abstractly. Learning Scenario Figure 4 38 illustrates t he metamodel of the Task Level learning scenario

PAGE 124

124 Examples The whole task instruction consists of demonstrating a complete task to learners and allow them to practice the task as a single unit. Figure 4 6 and Figure 4 7 show an open online course Introduction to Computer Science that provides learners with key computer science concepts and they will learn Python, a powerful programming language. Computer science is about how to solve a real world problem such as building a web search engine by breaking them into smaller components and precisely describing a sequence of steps to solve these components. Theoretically, t he three main components for building a web search engine are: (1) Finding data by building a web crawler learners will learn this topic in learning unit 1 3, (2) Building an index to ensure fast response times to search queries learne rs will learn this t opic in learning unit 4 6 and (3) Ranking pages to get the best web pages for any given query learners will learn this topic in learning unit 7 8 Thus, acquiring knowledge and skill in the context of whole tasks enables learners to form mental models of how those individual skills can be integrated into a complete performance. Figure 4 6 Instructional strategies for task level

PAGE 125

125 Figure 4 7 Introduction of the learning unit Related Patterns Task Level is often implemented with S how Task and Problem Progression. Supporting Research 1. Johnson, E. B. (2002). to stay. Thousand Oaks, CA: Corwin Press. 2. Merrill, M. D. (2002). First principles of instruction. Educational Technology Research and Development, 50 (3), 43 59. 3. Perin, D. (2011). Facilitating student learning through contextualization: A review of evidence. Community College Review, 39 (3), 268 295. 4. Simpson, M. L., & Nist, S. L. (2002). Encouragin g active reading at the college level. In C. C. Block & M. Pressley (Eds.), Comprehension instruction: Researc h based best practices (pp. 365 381). New York, NY: Guilford Press. Problem Progression Design Pattern Pattern Name Problem Progression Also Know n As Practice, Test or Quiz

PAGE 126

126 Category Problem Context After each learning module, a learner takes a quiz to ensure their readiness to continue. Effective learning requires a progression of problems the problems start levels of expertise increase, instructional strategies should consider the alteration in the cognitive load to facilitate the transition from novice to expert. Problem Learners have to solve the whole task or some of the problems that are very complex. H owever, solving a single problem and receiving little or no guidance are likely not an effective way to help learners acquire problem solving skills. Forces increases. The expertise reversal effect refers to the reversal of the effectiveness of instructional strategies on learners as levels of knowledge in a domain change. Different le vels of instructional guidance can result in different learning outcomes, depending on Solution The research of cognitive load theory has shown that for novice learners with low prior knowledge, instruction consists of worked exa mples is more effective and efficient with lower mental effort than instruction consists of problem solving. Scaffolding is certainly suitable in such conditions to provide novices with sufficient guidance. Also, learners must begin with a less complex pro blem in order to master a complex problem.

PAGE 127

127 Learners are then given a more complex problem when the first problem is mastered. improve until they can solve more complex prob lems. Initially, novice learners should be provided with complete worked examples as full guidance. Learners study the problem by working through every step of the sequence in the worked examples. Then, the prompt can be gradually faded and replaced with c ompletion tasks, for instance, ask learners to complete remaining steps of the Java code partially guided. Learners in the fading condition usually perform better on transfer tasks than learners who receive fully worked examples when solving practice tasks Finally, as learners reach higher levels of knowledge in the subject area, problem solving practice with no guidance or self explanation can be used. An active processing of worked examples can be encouraged by self explanation. At this stage, the learne r should independently provide the proper solution. Providing self explanation prompts also benefited transfer. Although self explaining increases the mental effort, to some extent it results in active processing of instructional materials, leading to grea ter learning outcomes. Table below suggests a possible sequence for problem progression. Table 4 1 6 Instructional sequence for teaching components of the whole task Task 1 Task 2 Task 3 Task 4 Lesson 1 Show Practice Practice Practice Lesson 2 Show Show Practice Practice Lesson 3 Show Show Practice Lesson 4 Show Practice Practice A single demonstration of some components (lesson 1) is sufficient because they are easy to understand. Further demonstrations are required for comprehension

PAGE 128

128 because some components (lesson 2) are more complex. No demonstration in the early sequence, some components (lesson 3) are demonstrated for the later tasks. Although some components (lesson 4) are very complex, a single instance is sufficient for expert learners. Consequences Cognitive load on learners varies with different levels of expertise and has a great impact on learning performances. In the early stage of learning, cognitive load is high due to few schemas or knowledge structures are available. Studying worked examples that provi de the clearest example can be better than solving the equivalent tasks. However, it is appropriate to increase learner control and to decrease instructor control as levels of expertise increase. During the intermediate stage when schemas are formed and working memory capacity is freed, germane load can be increased thro ugh completion tasks. In the final stage of learning, there must be sufficient working memory capacity to allow for more problem solving practice with no guidance or self explanation. Overall, this instructional approach prepares learners to transition fro m early to final stage. Learning Scenario Figure 4 39 illustrates t he metamodel of the Problem Progression learning scenario Examples The extraneous cognitive load can be reduced by providing worked examples in the beginning, following by completion task s and then full tasks. The examples show quizzes through a progression of increasingly complex problems, presenting different forms of instructional strategies. Problems must first start with a less complex in order to

PAGE 129

129 rove as in Figure 4 8 Worked examples are necessary to minimize extraneous cognitive load because this strategy provides novice learners with information needed to encourage comprehensive knowledge. Most research also suggested applying fading as in Figure 4 9 when teaching with worked examples in actual settings. While self explanation prompts as in Figure 4 1 0 could foster conceptual knowledge. Figure 4 8 Quiz with a worked example approach Figure 4 9 Quiz with a fading approach

PAGE 130

130 Figure 4 10 Quiz with a self explanation approach Related Patterns Problem Progression is often implemented with Show Task and Task Level. Supporting Research 1. Atkinson, R. K., & Alexander. R. (2007). Intera ctive example based learning environments: Using interactive elements to encourage effective processing of worked examples. Educational Psychology Review, 19 (3), 375 386. 2. Hilbert, T. S., Renkl, A., Schworm, S., Kessler, S., & Reiss, K. (2008). Learning to teach with worked out examples: A computer based learning environment for teachers. Journal of Computer Assisted Learning, 24 (4), 316 332. 3. Merrill, M. D. (2002). First principles of instruction. Educational Technology Research and Development, 50 (3) 43 59. 4. Salden, R. J. C. M., Koedinger, K. R., Renkl, A., Aleven, V., & McLaren, B. M. (2010). Accounting for beneficial effects of worked examples in tutored problem solving. Educational Psychology Review, 22 (4), 379 392. 5. Sweller, J., van Merrienb oer, J. J. G., & Paas, F. G. W. C. (1998). Cognitive architecture and instructional design. Educational Psychology Review, 10 (3), 251 296. Previous Experience Design Pattern Pattern Name Previous Experience

PAGE 131

131 Also Known As Prior Knowledge Category Activation Context expected to be broad and greatly variable. When learners have relevant prior knowledge, the instruction should activate their existing experience to link previously learned information to the new knowledge. Problem One of the challenges of knowledge to novel situations. Some instructions require learners to complete a pretest to assess the level of their prior knowledge at the beginning of each learning unit. This instruction is sometimes no t effective as learners do not understand the new material to be learned without engaging in any learning activity. Forces When learners understand some of the new material to be learned, their prior knowledge can be activated through a relevant learning activity to demonstrate what they already know about the topic. Solution Instruction should direct learners to recall, relate, describe or apply relevant prior knowledge that can be served as a foundation for the new knowledge to be learned Activating pri or knowledge can be done through starting a lesson by introducing a new topic and demonstrating relevant examples. Learners then can be instructed to

PAGE 132

132 complete multiple choice questions or short answer to probe their prior knowledge. It is important to incl ude questions that most learners can answer correctly as a starting point for activating prior knowledge. Also, l earners should know that they will not be graded for the assessment. The information from the assessments should be shared with the learners as appropriate. Consequences The activation of prior knowledge helps learners connect previously learned information to the new learning material. Learning Scenario Figure 4 40 illustrates t he metamodel of the Previous Experience learning scenario Example s The prior knowledge that learners bring to a learning environment i s a critical factor influencing new learning. Figure 4 1 1 shows messages from the instructor to remind that the first quiz is intended to check their understanding and they will not be g raded for the assessment. Figure 4 1 2 illustrates the first quiz that relates to the learning goal of the lesson. For the purpose of activating prior knowledge, it is important to provide questions that most learners can answer correctly. Then, the instruc tor a program 1 3 Following the demonstration, learners can take the quiz as many times as they want without penalty. Figure 4 1 4 Some q have been covered. Such questions ensure the information presented to learners is adequate in building the necessary knowledge for them to understand the new material

PAGE 133

133 better. There are solution videos to the quiz and if learners get the quiz right, they can still refer to the helpful information and tips or even new material in the videos. While interactive environments may increase essential cognitive load, it is not sufficien t for learners to just actively process the lear ning material, rather it is important for them to process information with a focus on central concepts and principles Figure 4 11 Messages from the instructor for the first quiz Figure 4 12 The first quiz for the Intro to Computer Science course

PAGE 134

134 Figure 4 13 a Program Figure 4 14 Related Patterns Previous Experience is often implemented with New Experience and Structure. Supporting Research 1. Hone, K. S., & El Said, G. R. (2016). Exploring the factors affecting MOOC retention: A survey study. Computers & Education, 98 157 168.

PAGE 135

135 2. Kalyuga, S. (2012). Role of prior knowledge in learning processes In N. M. Seel (Ed.), Encyclopedia of the Sciences of Learning (pp. 2886 2888 ). Heidelberg, Germany: Springer Verlag. 3. Margaryan, A., Bianco, M., & Littlejohn, A. (2015). Instructional quality of massive open online courses (MOOCs). Computers & Education, 80 77 83. 4. Merrill, M. D. (2002). First principles of instruction. Educational Technology Research and Development, 50 (3), 43 59. New Experience Design Pattern Pattern Name New Experience Also Known As Provide Experience Categor y Activation Context The instruction should provide learners with relevant knowledge to serve as a foundation for a new experience. The instruction should also help learners feel confidence in their ability to acquire a new knowledge and skill, as well as to realize its relevance. Problem Most instruction begins with abstract representations, but without providing relevant experience to the learners. Learners feel overwhelmed when the new and unfamiliar material is introduced immediately without sufficien t support. As prior they tend to memorize the new material.

PAGE 136

136 Forces In opposition to a traditional classroom setting, open online learning assumes learners can self regu late their own learning. Thus, p roviding relevant experience to the learners before beginning instruction on any new material is necessary. Solution Learner engagement can be achieved by clearly communicating the benefit of the learning goals to the learn ers, and helping them connect their interests and personal goals to the learning goals. In a traditional classroom setting, instructors help learners practice new information by asking a large number of questions. An open online course does not have the ca pacity to support such interaction due to the large numbers of geographically dispersed learners for synchronous sessions. Thus, i t should become a common practice to provide learners with a relevant experience before introducing a new learning material by presenting a learning goal and giving more examples. Consequences Preparing learners prior to introducing a new experience can avoid them to memorize the new material. Learners can actively engage in a course and stay on track. Learning Scenario Figure 4 41 illustrates t he metamodel of the New Experience learning scenario Examples Figure 4 15 shows that learners are presented with the introductory material, such as the definition of the design principles and the explanation of complex concepts prior to independent practice. The instructor further demonstrates another example that relevant to the new topic as in Figure 4 16 The use of multiple explanations and

PAGE 137

137 examples will provide learners with a brief introduction to the new concepts to be learned. Figure 4 15 An instructor video presents the introductory materials Figure 4 16 An instructor video demonstrates a relevant example Related Patterns New Experience is often implemented with Previous Experience and Structure. Supporting Research 1. Ho ne, K. S., & El Said, G. R. (2016). Exploring the factors affecting MOOC retention: A survey study. Computers & Education, 98 157 168.

PAGE 138

138 2. Kalyuga, S. (2012). Role of prior knowledge in learning processes In N. M. Seel (Ed.), Encyclopedia of the Sciences of Learning (pp. 2886 2888 ). Heidelberg, Germany: Springer Verlag. 3. Margaryan, A., Bianco, M., & Littlejohn, A. (2015). Instructional quality of massive open online courses (MOOCs). Computers & Education, 80 77 83. 4. Merrill, M. D. (2002). First princ iples of instruction. Educational Technology Research and Development, 50 (3), 43 59. Structure Design Pattern Pattern Name Structure Also Known As Schema Category Activation Context The instruction should encourage learners to activate their mental models for organizing a new knowledge when they have relevant mental models. Otherwise, instruction should provide a structure that enables learners to build appropriate mental models for a new knowledge. Problem Learners have to build a new set of interrelated skills based on their previously acquired mental models in order to solve complex problems. Too often learners activate inappropriate mental models that increase their mental efforts and results in errors when trying to solve new problems.

PAGE 139

139 Forces Activation of mental models helps learners to stimulate their mental models that can be modified to integrate the existing knowledge into the new knowledge. Solution The instruction should encourage learners to recall a structure for organizing the newly a cquired knowledge and skill. Graphic organizers such as an advance organizer can be used to help learners organize the interrelationships among knowledge components. A dvance organizers are also useful for activating and building schema and help learners re member, retain, and understand the new material. While m otivational themes can be used for organizing structure as they help learners to activate appropriate mental models, provided that the themes relevant to the new content. Consequences The activation o f mental models can increase cognitive load in learners that enable them to acquire the necessary knowledge and skill. Learners then can build the necessary organizational schema for the new knowledge. Learning Scenario Figure 4 42 illustrates t he metamode l of the Structure learning scenario Examples Graphic organizers can be presented to the learners before a new content is introduced. Figure 4 17 shows an advance organizer to connect general knowledge to the new information. This graphic organizer is not only presenting the overview of the learning unit, but also sharing ideas and techniques of the new material to be learned. Encouraging learners to recall previous relevant experience can activate appropriate mental models to facilitate the acquisition of new skills. Mental models can be modified

PAGE 140

140 to allow learners integrate the previously acquired knowledge to the new knowledge. a framework for organizing their new contents. Figure 4 18 show s motivational themes to serve as an organizing structure. These motivational themes activate appropriate mental models to promote instructional effectiveness and result in the increasing of cognitive load necessary to acquire the new knowledge. Figure 4 17 Advance organizer Figure 4 18 Motivational themes

PAGE 141

141 Related Patterns Structure is often implemented with Previous Experience and New Experience. Supporting Research 1. Hone, K. S., & El Said, G. R. (2016). Exploring the factors affecting MOOC ret ention: A survey study. Computers & Education, 98 157 168. 2. Kalyuga, S. (2012). Role of prior knowledge in learning processes In N. M. Seel (Ed.), Encyclopedia of the Sciences of Learning (pp. 2886 2888 ). Heidelberg, Germany: Springer Verlag. 3. Margaryan, A., Bianco, M., & Littlejohn, A. (2015). Instructional quality of massive open online courses (MOOCs). Computers & Education, 80 77 83. 4. Merrill, M. D. (2002). First principles of instruction. Educational Technology Research and Development, 50 (3), 43 59. Demonstration Consistency Design Pattern Pattern Name Demonstration Consistency Also Known As Show Me Category Demonstration Context Each different types of problems require different knowledge structures and different component skills. Hence, the demonstration of the content to be taught should be consistent with the intended learning outcomes for effective learning. Problem Too often the demonstration is inconsistent with the type of skills to be learned. Also, the most common instruction is the presentation of information and is followed by memorization. The memorization of information based on repetition is ineffective as

PAGE 142

142 learners do not have the opportunity to apply what has been learned. Too much solving skills. Forces Effective learning occurs when the demonstration of the skills being promoted is consistent with the content to be taught. Solution The instruction of an open online course should demonstrate what is to be learned. For better understanding of abstract ideas, examples and non examples can be used to teach learners the concepts, pri nciples, and theories. Demonstrations are useful to show the sequence of procedures to the learners. Visualization is a representation of a set of information, situation or object such as an image or chart, which is practical to visualize processes. Modeli ng can be used to describe the interactions between objects and the overall behavior of a particular system, such as sequence diagrams or use case diagrams. Overall, the instruction should consistent with the desired learned performance in order to promote the development of the cognitive structures. Consequences Specific techniques such as modeling, simulations, and visualizations to demonstrate what is being taught are more effective than just presenting information to learners. Learning Scenario Figure 4 43 illustrates t he metamodel of the Demonstration Consistency learning scenario

PAGE 143

143 Examples One of the best ways to teach abstract concepts and complex ideas is by showing an example. Figure 4 19 illustrates an example to help learners understand the abstract idea of objects in the context of object oriented programming. A n interactive software program could be a rational tool to support examples and to foster understanding among learners. However, several limitations must be taken into account in interpreting the technique associated with cognitive load theory. E xamples may not work for all types of learners, especially the advanced ones and the methods in which a particular learner processes the e xamples can have a major impact on the effectiveness of the learning experience. While a demonstration or tutorial video is an instructi onal program that provides step by step information to complete a task. Figure 4 20 shows a video to demonstrate how to create objects on the BlueJ programming environment. This demonstration video is aimed to help learners familiarize with the capabilities of the BlueJ environment. BlueJ is a Java development environment specifically designed for teaching at an introductor y level. Figure 4 19 An example to show the abstract idea of objects

PAGE 144

144 Figure 4 20 A demonstration or tutorial video to create objects Related Patterns Demonstration Consistency is often implemented with Learner Guidance and Relevant Media. Supporting Research 1. Merrill, M. D. (2002). First principles of instruction. Educational Technology Research and Development, 50 (3), 43 59. 2. Moller, L., Huett, J. B., & Harvey, D. M. (2009). Learning and instructional technologies for the 21st centur y: Visions of the future New York, NY: Springer. 3. Morrison, G. R., Ross, S. M., Kalman, H. K., & Kemp, J. E. (2013). Designing effective instruction Hoboken, NJ: John Wiley & Sons. 4. Reigeluth, C. M., & Carr Chellman, A. A. (Eds.). (2009). Instruction al design theories and models New York, NY: Routledge. 5. Retalis, S., Georgiakakis, P., & Dimitriadis, Y. (2006). Eliciting design patterns for e learning systems. Computer Science Education, 16 (2), 105 118. Learner Guidance Design Pattern Pattern Name L earner Guidance

PAGE 145

145 Also Known As Help Me Category Demonstration Context Provide multiple means of representations for approaching different contents, so learners have multiple ways of understanding the information presented. Problem Some instruction does not provide learners with multiple means of representation. Transfer of learning does not occur with a single representation as learners cannot make connections between concepts presented. Forces Transfer is promoted when the instruction provides learners focusing focus on the relevant information. Solution The instruction should provide learners with appropriate levels of guidance. Enabling learners relate t heir personal experiences and perceptions to the current topic, such as through reflections and discussions are an effective way to provide relevance. This guidance helps learners relate the previously acquired knowledge and skill structures with the curre nt information. Instead of passively observing the demonstration, learning from demonstrations can be improved when learners actively engage in peer discussion and peer demonstration. Learners can head to the forums for interaction with the open online cou rse community. Another form of guidance is to

PAGE 146

146 provide learners with multiple means of representations as learners differ in the ways they perceive the information presented. Learners can relate the content or information to specific instances as a result o f the demonstrations. Thus, providing options for representation is essential in open online courses. Consequences Presenting the relevant content helps learners develop into self regulated and motivated learners. Transfer of learning occurs when multiple means of representations are used as learners are allowed to make connections between concepts. Through multiple representations learners can develop the mental models necessary for describing a more detailed and accurate description of the information Learning Scenario Figure 4 44 illustrates t he metamodel of the Learner Guidance learning scenario Examples New information can be presented to learners in varied ways. Figure 4 2 1 and Figure 4 22 are part of a MOOC, Introduction to Java Programming about basic of Java interfaces and on how to implement interfaces in class. Figure 4 21 illustrate s a Java programming tutorial while Figure 4 22 presents a Java interface example with a live practical demo that are very useful for beginners Figure 4 23 is another type of learner guidance a reflection task to enable learners relate their personal experiences to the topic. Learners are able to critically review their own learning process through self reflection. Also, learning can be enhanced through pe er interaction in a discussion forum as in Figure 4 24 Discussions provide opportunities for learners to articulate and defend their positions, accept different opinions, foster intellectual agility and evaluate artifacts

PAGE 147

147 Figure 4 21 Java interface example Figure 4 22 Demonstration video Figure 4 23 Reflection task

PAGE 148

148 Figure 4 24 Peer interaction Related Patterns Learner Guidance is often implemented with Demonstration Consistency and Relevant Media. Supporting Research 1. Merrill, M. D. (2002). First principles of instruction. Educational Technology Research and Development, 50 (3), 43 59. 2. Moller, L., Huett, J. B., & Harvey, D. M. (2009). Learning and instructional technologies for the 21st century: Visions of the fut ure New York, NY: Springer. 3. Morrison, G. R., Ross, S. M., Kalman, H. K., & Kemp, J. E. (2013). Designing effective instruction Hoboken, NJ: John Wiley & Sons. 4. Reigeluth, C. M., & Carr Chellman, A. A. (Eds.). (2009). Instructional design theories an d models New York, NY: Routledge. 5. Retalis, S., Georgiakakis, P., & Dimitriadis, Y. (2006). Eliciting design patterns for e learning systems. Computer Science Education, 16 (2), 105 118. Relevant Media Design Pattern Pattern Name Relevant Media

PAGE 149

149 Also Known As Instructional Media Category Demonstration Context Integrating and infusing social media, online instructional tools, video, and digital content in ways that connect and engage learners with course content. Problem Some instruction may use irrelevant audio, text, and visuals to deliver digital contents. Using this combination of multimedia can potentially hurt learning as it competes for the learner attention and result in the increasing of cognitive load. Forces The effective use of digital media such as audio, graphics, and text can create a highly engaging course and thus optimize learning. Solution Digital content can be delivered through tablet recording and presentation software, in which the instructor will po int or motion to things, and learners can see the I nstructional videos with graphics should be explained solely by audio, rather than by both audio and text in order to facilitate the transfer of learn ing. For instance, using audio narration for explaining a complex graphic in a topic when the elements of instructional media consist of on screen graphics Graphics then should be selected based on the type of contents. In contrast, videos presented with narration of on screen text are more effective than videos presented with narration alone when there is no graphic on the screen. Although

PAGE 150

150 graphics such as charts, diagrams, and photographs can improve learning, it is important to choose the ones that are congruent with the learning goal. Consequences Learning is effective when the combination of multimedia does not compete for the learner attention. Learning Scenario Figure 4 45 illustrates t he metamodel of the Relevant Media learning scenario Examples Working memory is a limited resource that is responsible for information processing. The use of graphics with audio can maximize the capacity of working memory. Figure 4 25 shows a video with audio narration for explaining a graphic in a topic that is comp lex and unfamiliar to the learners. Adding a graphic to the content can improve learning, provided that the illustration is congruent with the instructional message. Placing related text to the graphic close to each other on the screen also can help to imp rove learning. Figure 4 25 Graphic with audio narration

PAGE 151

151 Related Patterns Relevant Media is often implemented with Demonstration Consistency and Learner Guidance. Supporting Research 1. Mayer, R. E. (2001). Multimedia learning New York: Cambridge University Press. 2. Mayer, R. E., & Moreno, R. (2003). Nine ways to reduce cognitive load in multimedia l earning. Educational Psychologist, 38 (1), 43 52. 3. Merrill, M. D. (2002). First principles of instruction. Educational Technology Research and Develo pment, 50 (3), 43 59. 4. Morrison, G. R., Ross, S. M., Kalman, H. K., & Kemp, J. E. (2013). Designing effective instruction Hoboken, NJ: John Wiley & Sons. 5. Reigeluth, C. M., & Carr Chellman, A. A. (Eds.). (2009). Instructional design theories and model s New York, NY: Routledge. Practice Consistency Design Pattern Pattern Name Practice Consistency Also Known As Practice Alignment Category Application Context After observing effective demonstrations, learners should get chances to engage in application of the newly acquired skill and knowledge. The i nstruction should include application (practice) and assessment (tests) that consistent with the implied learning objectives.

PAGE 152

152 Problem Learning is ineffective when l earners engage in practice that is inconsist ent with the intended goals of the instruction Forces Learners can have a meaningful interaction with the content by getting a chance to actively apply and practice their newly acquired knowledge and skills. Aligning practice with learning goals can help enhance the learner content interaction. Solution Different types of skills require different practice. Instruction then should provide practice that is consistent with the types of skills to be acquired. However, it is important to ensure the consistency of the p ractice with the desired instructional goal. Some of the practice that can be offered to the learners are as follows: Learners can have a meaningful interaction with the content by getting a chance to actively apply and practice their newly acquire d knowledge and skills. Aligning practice with learning goals can help enhance the learner content interaction. Solution Different types of skills require different practice. Instruction then should provide practice that is consistent with the types of sk ills to be acquired For instance, a common p ractice for the concept classification is recall and recognition. Th is memory retrieval is a process of remembering information stored in a long term memory. Examining a worked example is another practice for the concept classification. However, it is important to ensure the consistency of the p ractice with the desire d instructional goal. Table 4 17 depicts some of the practice that can be offered to the learners in an open online course.

PAGE 153

153 Table 4 1 7 Suggestion for practice Types of Skill Practice example Concept classification Recall and recognition the more learners practice a piece of information, the more they likely remember it. Review learners examine a work ed example. Describe learners identify, sort, rank, and label each component after review a video presenting a task and the core ideas. Procedure Do perform a procedure using a method and tool to produce outputs. Predicting consequences Predict learners assume the outcome of a process. Discuss learners head to the online forums for discussion with the online course community. Reflect and share learners take a moment to reflect on what have been learned and share their reflections wi th the rest of the online forums. Consequences The interaction occurs in an open online course is much different than the interaction in a traditional classroom due to the instructional media used in the virtual platform. Effective design of a learner to content interaction can have a positive impact o Learning Scenario Figure 4 46 illustrates the metamodel of the Practice Consistency learning scenario. Examples The learner content interaction occurs when learners engage with the course content and participate in i ndependent practice. The continuous and extensive interaction with the content leads to higher levels of learning and greater satisfaction with the course. One way to improve the learner content interaction is through the

PAGE 154

154 consistency of practice and postte st with the desired learning goal. Figure 4 2 6 illustrates a practice that requests learners to do the procedure and produce an output. Figure 4 2 7 shows a practice that requests learners to identify and match some methods with their classes after reviewin g a worked example. Figure 4 26 Procedure and produce output practice Figure 4 27 Identify and label practice

PAGE 155

155 Related Patterns Practice Consistency is often implemented with Diminishing Coaching and Varied Problems. Supporting Research 1. Merrill, M. D. (2002). First principles of instruction. Educational Technology Research and Development, 50 (3), 43 59. 2. Morrison, G. R., Ross, S. M., Kalman, H. K., & Kemp, J. E. (2013). Designing effective instruction Hoboken, NJ: John Wiley & Sons 3. Retalis, S., Georgiakakis, P., & Dimitriadis, Y. (2006). Eliciting design patterns for e learning systems. Computer Science Education, 16 (2), 105 118. 4. Reigeluth, C. M., & Carr Chellman, A. A. (Eds.). (2009). Instructional design theories and models New York, NY: Routledge. Diminishing Coaching Design Pattern Pattern Name Diminishing Coaching Also Known As Scaffolding Category Application Context Coaching is a fundamental role in knowledge transfer and retention. Worked examples for instance prov ide learners with ready made solutions to problems and step by step instructions that assist learners construct deep understanding of the content areas. Providing one to one video solution and discussion on quizzes are some formance and persistence.

PAGE 156

156 Problem An instructor is the focal point in the traditional classroom and does most of the teaching, coaching, questioning, and responding to learners. Due to the lack of face to face interaction with the live instructor, on line learners often feel disconnected, uninterested, and unmotivated to complete the task. Some learners are unable to demonstrate their ability when learning complex topics in open online environments without scaffolding. They also fail to increase unders tanding of the complex concepts taught. Forces When learners are having difficulty with a problem solving task, the instruction should allow learners to access help or provide them with gradually diminished coaching. Instead of right wrong feedback, instr uction should provide practice with corrective feedback to learners, as well as an indication of progress. Solution Scaffolding is an effective instruction with the idea of guiding learners early in the learning process, but this support will gradually dim inish as learners gain independence and the coaching is gradually diminished as learners become more competent. Work examples enable learners to observe a demonstration of a correct solution to the problem. By observing a worked example, learners watch the expert video, in which they can see the instructor using their hands interact with the content, or even using their hands to point or motion to things. Learners are encouraged to pause the video and examine the information in the video. On the other hand, feedback is one of the most important types of learner guidance that enables learners to reflect on their learning. Making mistakes is a natural part of learning. Learners make mistakes and

PAGE 157

157 they learn best when they observe solutions or answers on how to identify the mistake, and how to avoid the mistake in the future. Within a MOOC, learners receive feedback in a form of error detection and correction through automated feedback. Overall, learning is promoted when learners are provided with coaching and ap propriate feedback during problem solving tasks. Consequences Learners can actively apply their new knowledge through an application that involve interactions with the content, and other learners. When learners practice and apply their newly acquired know ledge and skill, they are provided with sufficient guidance and feedback on performance. Scaffolding such as w orked examples provide s learners with step by step instructions and ready made solutions in solving problems to help learners develop a deeper understanding of the instruction. Learning Scenario Figure 4 47 illustrates t he metamodel of the Diminishing Coaching learning scenario Examples Presenting learners with appropriate guidance such as scaffolding helps to reduce the cognitive load of the learners as they encounter the problems to be solved. Figure 4 28 shows a worked example, as a coaching tool that provides learners the first experience with the material. Learners can reflect on their prior experiences with the content and enable them to construct meaning from the instruction. Figure 4 29 illustrates the automated feedback when learners make a mistake during a quiz.

PAGE 158

158 Figure 4 28 Worked example Figure 4 29 Automat ed feedback Related Patterns Diminishing Coaching is often implemented with Practice Consistency and Varied Problems.

PAGE 159

159 Supporting Research 1. Atkinson, R. K., & Alexander. R. (2007). Interactive example based learning environments: Using interactive elements to encourage effective processing of worked examples. Educational Psychology Review, 19 (3), 375 386. 2. Hilbert, T. S., Renkl, A., Schworm, S., Kessler, S., & Reiss, K. (2008). Learning to teach with worked out examples: A computer based learning environment for teachers. Journal of Computer Assisted Learning, 24 (4), 316 332. 3. Merrill, M. D. (2002). First principles of instruction. Educational Technology Research and Development, 50 (3), 43 59. 4. Salden, R. J. C. M., Koedinger, K. R., Renkl, A. Aleven, V., McLaren, B. M. (2010). Accounting for beneficial effects of worked examples in tutored problem solving. Educational Psychology Review, 22 (4), 379 392. 5. Reigeluth, C. M., & Carr Chellman, A. A. (Eds.). (2009). Instructional design theories and models New York, NY: Routledge. 6. Sweller, J., van Merrienboer, J. J. G., & Paas, F. G. W. C. (1998). Cognitive architecture and instructional design. Educational Psychology Review, 10 (3), 251 296. Varied Problems Design Pattern Pattern Name Varied P roblems Also Known As Variability of Practice Category Application Context Learners should be provided with a sequence of varied problems for effective learning. Presenting learners with a range of divergent examples provides multiple opportunities for them to apply new knowledge or skill to a variety of problems.

PAGE 160

160 Problem Some instructions provide learners with a single problem. Solving a single problem enables learners use only one perspective in perceiving a problem. Apparently, learners have not learn ed a cognitive skill much just by applying knowledge to a single problem. A n instructional strategy that consists of a single problem is far less effective than a varied sequence of problems Forces Learners should apply their new knowledge or skill to multiple problems Solution Learners are presented with a sequence of varie d problems, but the problems should not be too similar to the demonstrated examples in order to avoid learners engage in limited reconstruction of mental models. Learners should be provided with new and challenging varied problems for application. P resenti ng learners with a range of divergent examples can help them to construct an adequate mental model to solve multiple problems Overall, variability of practice enables generalization of cognitive schemas and enhances learning transfer. Consequences When le arners are provided only with a single problem, they may not have developed the nuanced mental model required to deal with more complex and varied problems In contrast, learners are able to continually improve their mental model when they attempt to solve a sequence of varied problems Learning Scenario Figure 4 48 illustrates t he metamodel of the Varied Problems learning scenario

PAGE 161

161 Examples Learners are given a series of problems to complete as an independent practice. A worked example is an instructional strategy that was derived from cognitive load theory and most notably, used to teach cognitive procedures such as science, technology, engineering and mathematics which have a great deal in common when it comes to learning requirements. Worked example s s how every step of the problem solving process while a learner studies the problem by working through every step of the example. After learners observe a worked example, they will solve a sequence of mixture problems as in Figure 4 30 and Figure 4 31 The f igures below show quizzes for a fundamental data types lesson in the course Intro duction to Java Programming. Figure 4 30 illustrates the practice for number types, while Figure 4 31 shows the practice of arithmetic operations. Having learners applies ne w knowledge allows them to experience how experts approach and solve multiple problems, hence learners will be prepared to face the real world problems. Figure 4 30 First problem in a sequence

PAGE 162

162 Figure 4 31 Second problem in a sequence Related Patterns Varied Problems is often implemented with Practice Consistency and Diminishing Coaching. Supporting Research 1. Hilbert, T. S., Renkl, A., Schworm, S., Kessler, S., & Reiss, K. (2008). Learning to teach with worked out examples: A computer based learning environment for teachers. Journal of Computer Assisted Learning, 24 (4), 316 332. 2. Merrill, M. D. (2002). First principles of instruction. Educational Technology Research and Development, 50 (3), 43 59. 3. Reigeluth, C. M., & Carr Chellman, A. A. (Eds.). (2009). Instructional design theories and models New York, NY: Routledge. 4. Salden, R. J. C. M., Koedinger, K. R., Renkl, A., Aleven, V., McLaren, B. M. (2010). Accounting for beneficial effects of worked examples in tutored problem solving. Educational Psychology Review, 22 (4), 379 392. 5. Sweller, J., van Merrienboer, J. J. G., & Paas, F. G. W. C. (1998). Cognitive architecture and instructional design. Educational Psychology Review, 10 (3), 251 296. 6. van Merrienboer, J. J. G. (2012). Variability of practice In N. M. Seel (Ed.), Encyclopedia of the Sciences of Learning (pp. 3389 3390 ). Heidelberg, Germany: Springer Verlag.

PAGE 163

163 Watch Me Design Pattern Pattern Name Watch Me Also Known As See What I Can Do Category Integration Context Instruction should encourage learners to demonstrate their newly acquired knowledge and skills Problem While animation, games, and multimedia may grab attention, these are not the motivational factors of instructional products. In other words, t ho se interactive media have a short term effect on motivation. Forces Learning is promoted when learners demonstrate how to solve a problem or perform a task. Also, the most inspiring activity is when learners can observe their own learning progress. Solut ion Carefully designed instruction should increase learner motivation. One of the most motivating event is when learners realize they can accomplish a task or solve a problem that they could not perform before. Learners deeply desire to demonstrate their newly acquired skills when they obtain new abilities. Thus, i nstruction should give a chance for learners to apply their ideas, demonstrate their solutions, and receive

PAGE 164

164 feedback from the instructor. Other than that, e ducational games, for instance, present ing learners with the increasing skill level in which they can observe their own learning progress. Consequences When learners are able to demonstrate improvement in skills, they have incorporated new knowledge into their lives. Learning Scenario Figure 4 49 illustrates t he metamodel of the Watch Me learning scenario Examples Demonstration is an opportunity for learners to defend ideas and publicize their solutions as in Figure 4 32 Providing social learning environments such as class forums is important as learners are able to see other points of view. After learners solve a problem or perform a task, they should receive feedback from the instructor as in Figure 4 33 Providing some explanations could guide learners during learning. Figure 4 32 Demonstrate a task or solve a problem

PAGE 165

165 Figure 4 33 Receive f eedback Related Patterns Watch Me is often implemented with Reflection and Creation. Supporting Research 1. Hilbert, T. S., Renkl, A., Schworm, S., Kessler, S., & Reiss, K. (2008). Learni ng to teach with worked out examples: A computer based learning environment for teachers. Journal of Computer Assisted Learning, 24(4), 316 332. 2. Merrill, M. D. (2002). First principles of instruction. Educational Technology Research and Development, 50( 3), 43 59. 3. Reigeluth, C. M., & Carr Chellman, A. A. (Eds.). (2009). Instructional design theories and models. New York, NY: Routledge. 4. Sweller, J., van Merrienboer, J. J. G., & Paas, F. G. W. C. (1998). Cognitive architecture and instructional design Educational Psychology Review, 10(3), 251 296. 5. van Merrienboer, J. J. G. (2012). Variability of practice In N. M. Seel (Ed.), Encyclopedia of the Sciences of Learning (pp. 3389 3390 ). Heidelberg, Germany: Springer Verlag. Reflection Design Pattern Pattern Name Reflection

PAGE 166

166 Also Known As Reflective Thinking Category Integration Context Instruction should allow learners to reflect on what they have learned and share with peers. Problem Too often instruction requires learners to perform a task or solve a problem. Reflection is another important component of instruction, giving a chance for learners to review their learning processes. Forces Instead of just turning in their task performanc e or problem solutions to the instructor, effective instruction requires learners to reflect on and share with peers. Solution Reflection is a critical thinking process, making judgments about what has happened through the process of analyzing. In order t o support reflective thinking, open online courses could provide an opportunity for learners to reflect after completing the lessons in each unit. During explorations, it is important to provide feedback to guide Providing auth entic tasks that involve ill structured data could encourage reflective thinking among learners. Also, learning environments that are flexible could prompt learners to discover what they believe is important.

PAGE 167

167 Consequences When learners are able to ref lect on their new knowledge, they have integrated instruction into their lives. Learning Scenario Figure 4 50 illustrates t he metamodel of the Reflection learning scenario Examples Reflection is a significant activity in problem solving models for collab orative problem solving. Figure 4 34 shows a reflection task and learners are required to share their reflective thinking with peers over the forums. By actively participating in reflective thinking, learners generally control and aware of their learning. Reflective thinking allows learners to bridge the gap assess between what they need to know and what they already know during the learning process. Figure 4 34 Reflective thinking task Related Patterns Reflection is often implemented with Watch Me and Creation.

PAGE 168

168 Supporting Research 1. M errill, M. D. (2002). First principles of instruction. Educational Technology Research and Development, 50(3), 43 59. 2. Moller, L., Huett, J. B., & Harvey, D. M. (2009). Learning and instructional technologies for the 21st century: Visions of the future. New York, NY: Springer. 3. Morrison, G. R., Ross, S. M., Kalman, H. K., & Kemp, J. E. (2013). Designing effective instruction. Hoboken, NJ: John Wiley & Sons. 4. Reigeluth, C. M., & Carr Chellman, A. A. (Eds.). (2009). Instructional design theories and mo dels. New York, NY: Routledge. 5. Retalis, S., Georgiakakis, P., & Dimitriadis, Y. (2006). Eliciting design patterns for e learning systems. Computer Science Education, 16(2), 105 118. Creation Design Pattern Pattern Name Creation Also Known As Invention Category Integration Context Instruction should provide an opportunity for learners to create, explore, and invent personal ways to apply their newly acquired skills. Problem M ost instruction focuses on applying a procedure and analyzing how the parts relate to one another. However, creating individual adaptations of the new ability is one of the most effective instructions

PAGE 169

169 Forces Encouraging learners to incorporate the new knowledge or skill into their everyday life. Solution Creating requires lear ners to put all of the pieces together in order to form a functional whole. In other words, creating allows learners to reorganize elements into a new structure or pattern. This mental function is the most difficult process in the revised taxonomy of the c ognitive domain. For instance, learners will be able to create a learning portfolio as part of the learning objectives. Another learning objective is learners will be able to design efficient project workflow. Consequences Learning is promoted when a learner moves beyond the instructional environment, and modifies the new knowledge and skill to make it personal. Learning Scenario Figure 4 51 illustrates t he metamodel of the Creation learning scenario Examples Creating is the most complex level in th depends on the analysis as a fundamental process which similar to the distinction between critical thinking and creative thinking. Figure 4 35 shows the most complex type of cognitive thinking creating task To rei terate, creating is putting parts together to structure a functional whole or rearrange parts in a new and personal way This process can be achieved through generating, planning, and producing elements into a new structure or pattern. In this quiz, learne rs are asked to plan their action on how they would proceed to develop a user interface for a software application.

PAGE 170

170 Figure 4 35 Creating task Related Patterns Creation is often implemented with Watch Me and Reflection. Supporting Research 1. Anderson, L. W., & Krathwohl, D. R. (Eds.). (2001). A taxonomy for learning, Boston, MA: Allyn & Bacon. 2. Bloom, B. S., & Krathwohl, D. R. (1956). Taxonomy of educational objectives: The classification of educational goals New York, NY: Longmans. 3. Merrill, M. D. (2002). First principles of instruction. Educational Technology Research and Development, 50(3), 43 59. 4. Reigeluth, C. M., & Carr Chellman, A. A. (Eds.). (2009 ). Instructional design theories and models New York, NY: Routledge. Structure of Open Online Learning Design Patterns The practicality of design patterns is important since the goal of design patterns is the description of reusable solutions to recurring problems. Each instructional design pattern is not isolated, but it is interrelated to other design patterns in the catalog. Figure 4 36

PAGE 171

171 real world problems, learning is promoted when learners are first ly given an overview of the whole task, often includes the learning objective and benefits to indicate what the learners will be able to do after completing a course or a module (Merrill, 2002). The centered learning experience. However, it is still incomplete to create a working blueprint for a comprehensive task centered le arning. Instead, just like other established design patterns, this design pattern can only make sense when it is used with the other related Re commendation for Practice In getting started with a MOOC, the instructional designer needs usable mechanisms to activate prior knowledge of the learner and get them familiar with the new learning environment. The design scenarios as follows, Name: Gettin g started in a computer science MOOC. Context: Introduction to Human Computer Interaction (HCI) is an introductory course on HCI, covering the principles, techniques, and open areas of development in HCI Challenge: A first time learner enrolled in an int ermediate computer science MOOC. T his course does not have significant prerequisites before participation The timeline for the course is approximately 16 weeks. Patterns Used: Show Task, Task Level, Problem Progression, Prior Knowledge, Existing Experienc e, Structure Proposed Solution: will present the learner with a short audio/slide presentation about the direct the learner to t he advance organizer about HCI in order to provide learners with a conceptual model that can facilitate the acquisition of

PAGE 172

172 about Interacting and Interfaces covered in the introductio will show a new whole task to the learner. After a brief explanation of the on the first quiz. The learner will solve a progression of problems that are explicitly Figure 4 36 Open online learning d esign pattern relationships influences influences influences influences influences influences has patterns Activation Integration Demonstration Application Problem Show Task Task Level Problem Progression Watch Me Reflection Previous Experience New Experience Structure Practice Consistency Diminishing Coaching Varied Problems Demonstration Consistency Learner Guidance Relevant Media has patterns has patterns has patterns has patterns Creation

PAGE 173

173 Summary This study proposed five dim ensions of design patterns with 15 design patterns that captured and presented solutions to problems in which the instructional designers can use when designing open online learning for computer science courses. The catalog of design patterns was developed from the self observation, expert interview, analysis of the functionality of computer science MOOCs, review of the literature on pedagogical strategies, and learn from existing published patterns in other related were used to organize the c atalog of open online learning design patterns. This study used a template that was modified from the Gamma et al. (1995) and Alexander (1979) pattern structures to describe and organize design patterns. Each design pattern was named, explained and describ ed systematically. The full list of design patterns was presented in this chapter that indicated the pattern name, also known as, category, context, problem, forces, solution, consequences, learning scenario, examples, related patterns, and references.

PAGE 174

174 Figure 4 37 The metamodel of the Show Task learning scenario Show Task Demonstrate the topic components for the task Present topic components specific to the task Learners apply previously learned topic components Show another new whole task or problem Show a new whole task or problem

PAGE 175

175 Figure 4 38 The metamodel of the Task Level learning scenario Task Level Demonstrate a successful task completion Present a particular complex problem or whole task Provide information for each skill component Demonstrate the skill components Learners apply previously learned skill components Show the application of these skill components to the task formance

PAGE 176

176 Figure 4 39 The metamodel of the Problem Progression learning scenario Learning objectives Lessons Questions Worked example Fading Self explanation Answers Instructional strategies Problem Progression Scores

PAGE 177

177 Figure 4 40 The metamodel of the Previous Experience learning scenario Previous Experience Present a new material and demonstrate examples Learners to recall, describe or apply relevant prior knowledge Show the application of these skill components to the task

PAGE 178

178 Figure 4 41 The metamodel of the New Experience learning scenario New Experience Present the introductory material Demonstrate relevant examples of the new material Learners to recall, describe or apply relevant prior knowledge Show the application of these skill components to the task

PAGE 179

179 Figure 4 42 The metamodel of the Structure learning scenario Structure Show an advance organizer or motivational themes Present a new material Learners apply previously learned skill components Show the application of these skill components to the task

PAGE 180

180 Figure 4 43 The metamodel of the Demonstration Consistency learning scenario Demonstration Consistency Show an example of the related concept Demonstrate a procedure to complete a task Learners apply previously learned skill components Show the application of these skill components to the task Present an overview of a new topic

PAGE 181

181 Figure 4 44 The metamodel of the Learner Guidance learning scenario Learner Guidance Sh ow a work example to introduce a skill component Demonstrate a procedure to complete a task Learners reflect previously learned skill components the reflection task

PAGE 182

182 Figure 4 45 The metamodel of the Relevant Media learning sc enario Relevant Media Present an overview of the skill components Demonstrate the complex content with relevant media Learners apply previously learned skill components Show the application of these skill components to the task

PAGE 183

183 Figure 4 46 The metamodel of the Practice Consistency learning scenario Practice Consistency Present an overview of the skill components Demonstrate a worked example of a new method Learners identify previously learned skill components Show the application of these skill components to the task

PAGE 184

184 Figure 4 47 The metamodel of the Diminishing Coaching learning scenario Learning objectives Lessons Questions W orked example Fading Self explanation Results Learner guidance Diminishing Coaching New lesson Questions Automated feedback Results

PAGE 185

185 Figure 4 48 The metamodel of the Varied Problems learning scenario Instructional goals Lessons Problems Practice 1 Practice 2 Practice 3 Answers Guidance and feedback Varied Problems Scores

PAGE 186

186 Figure 4 49 The metamodel of the Watch Me learning scenario Instructional goals Lessons Problems Quiz 1 Quiz 2 Quiz 3 Answers Guidance and feedback Watch Me The Scores

PAGE 187

187 Figure 4 50 The metamodel of the Reflection learning scenario Reflection Show a work example to introduce a skill component Demonstrate a procedure to complete a task Learners reflect previously learned skill components Share the reflection with peers in the forums

PAGE 188

188 Figure 4 51 The metamodel of the Creation learning scenario Instructional goals Lessons Quiz Practice 1 : Generate Practice 2 : Plan Practice 3 : Produce Answers Guidance and feedback Creation Scores

PAGE 189

189 CHAPTER 5 DISCUSSION Massive Open Online Courses (MOOCs) have revolutionized higher education by delivering high quality content to a large amount of l earners via the Internet (Staubitz, Petrick, Bauer, Renz, & Meinel, 2016). All of the leading MOOC providers such as Coursera, edX, and Udacity have built their courses through partnerships with prominent universities, for instance Massachusetts Institute of Technology, Harvard University, and Stanford University. Institutions around the world are increasingly looking for ways in blending MOOC content and materials into their on campus teaching or hosting their own MOOCs ( Bonk, Lee, Reeves, & Reynolds 2015). Existing research on major MOOC providers has found that more than half of total enrollment was in computer science courses ( Straumsheim 2015). Identifying the key design features for these computer science MOOCs is somehow difficult as MOOCs dep end on an evolving set of practices. Hence, there is a demand for a more structured method for documenting and describing best practices in designing open online courses. To address this gap in the literature, this study sought to answer the following rese arch questions: 1. To what extent do the design patterns exist within the Massive Open Online Courses in computer science? 2. How is a catalog of design patterns for open online learning constructed? The first research question of this study was answe red through the design pattern mining. In this study, design patterns of MOOCs were mined through five methods: (1) Self observation, (2) Expert interview, (3) Analysis of the functionality of computer science MOOCs, (4) Review of the literature on pedagog ical strategies, and

PAGE 190

190 (5) Learn from existing published patterns in other related areas. The second research question was answered through the design pattern writing. Discussion of Findings The results presented in Chapter 4 address a qualitative content a nalysis, from self observation to existing published patterns in other related areas, as well as the catalog of design patterns. The discussion provided in this chapter will interpret the findings as they are relevant to the research questions, and also th e theoretical framework employed in their pursuit. Use of Online Affordances Learner to l earne r i nteraction The findings of this study indicated that a learner to learner interaction within MOOCs occur through peer assessment and discussion forum s The findings by enabling them to evaluate the work of peers. Learners could also develop critical thinking skills through the process of assessing other learners. Peer assessmen t has been successfully implemented in traditional classrooms and regular online learning ( van Zundert, Sluijsmans, & van Merrienboer, 2010 ). However, more evidence is needed to determine the successful implementation of peer assessment in MOOCs (Staubitz, Petrick, Bauer, Renz, & Meinel, 2016). It is not surprising that assigning learners randomly for peer feedback can be complicated as some learners may not have the ability or knowledge in giving an (Staubitz et al., 2016). Also, it is In the analysis of instructional design quality of 76 randomly selected MOOCs, Margaryan

PAGE 191

191 et al. (2015), found that there were no c ollaborative activities for 68 MOOCs. Collective knowledge in MOOCs refers to the extent to which the learning activities require learners learn from each other. Instead of merely consuming knowledge, learners should contribute to collective knowledge and build on the et al., 2015). In contrast to the face to face classroom where learners are expected to interact before, during, and at the end of each lesson, online learners and instructors are represented by on screen text. Th e text based interaction occurs in the online learning community can be misinterpreted by learners due to the lack of visual expressiveness (McInnerney & Roberts, 2004). Discussion forums are spaces where MOOC learners from all over the world ask for help, comment on the content of the course, post questions, provide suggestions, reflect on what have been learned, and share ideas. Learners rely on each other to respond to questions or comments when instructors offer no moderation in the forums. Collaborati on and social interaction is essential for successful participation in online learning tasks. Hence, this study proposed the use of discussion forum in practice consistency and learner guidance design patterns. Besides discussion forums for communication, one participant mentioned that learners can connect to one another in person through Udacity Connect. Unfortunately, MOOCs are not always open as in the sense of Open Educational Resources (OER) since learners can only add Udacity Connect if they pay $99 a month Learner to c ontent i nteraction Statements from participants of this study would seem to imply that a learner to content interaction occurs through instructional videos. Most c omputer science MOOCs

PAGE 192

192 use videos for intros, outros, explanations, tutor ials, quiz introductions, and quiz solutions. While interactive programming quizzes are used for the student self assessment. These instructional video forms were proposed in the catalog of design patterns to establish a learner to content interaction. It is interesting to note that learners received automated feedback on most programming exercises and all other types of quizzes frequently. Automated grading options were provided for simple testing, for instance multiple choice, true/false, and short answer questions. Learners should be encouraged to analyze, apply, assess, evaluate, solve, synthesize, and reflect on what they learn as they interact with the content ( Khan, 2016) During the learner to content interaction, learners process the information onc e they access the learning materials, and transform it from short term to long term memory ( Khan, 2016) Some open platforms allow limited access to reuse their material without permission and others may restrict the reuse of material. One participant in this study informed that his computer science MOOC, titled Introduction to Human Computer Interaction, provided more open ended assignments for assessing student learning. Examples of open ended assignments are essay questions to analyze existing interface as well as surveys and interviews with real people. Although automated grading tools work best with computer science courses, the assessment is difficult for written essays. Some MOOC providers may charge a range of fees, especially for the open ended as sessment as they cannot be graded by an automated grader. Grading and providing feedback on open ended assignments in MOOCs has been the topic of a number of recent articles. edX announced to use automated essay scoring for supporting the assessment of wri tten work (Markoff, 2013). Automated essay

PAGE 193

193 scoring was discovered more than a decade ago, and can provide rapid feedback as reliable as those from human raters for different types of essays (Balfour, 2013). The tool can also help learners improve their wri ting by giving them categorical feedback. However, this grading mechanism would be available outside their MOOC environment. On the other hand, Coursera decided to use peer evaluation to assess open ended assignments, in which learners are taught on a part icular scoring rubric for an assignment using practice essays prior to the peer review process (Balfour, 2013). It would be valuable to consider a design pattern for an automated essay grading tool in the future. Learner to instructor i nteraction Nanodegree received personalized feedback from an instructor on their project or cumulative assessment at the end of the course. MOOCs are not as open as their name N anodegree is a n online certification for entry level programming and analyst positions that can be completed in less than a year for $200 a month. On the other hand, Udacity Connect provides face to face learning, interactive feedback, goal setting, group accountability and monitoring that can be added to the Nanodegree program. These open platforms certainly cannot replace the formal and credit based education, but can be used to make it more effective. Low instructor involvement after the course start s was a major cl aim related to MOOCs (Balfour, 2013). One participant explained that they interact with learners through discussion forums and free online collaboration platforms that have been integrated into the MOOC. Moderating each individual post may become overwhelm ing due to the large number of comments and discussion posts. Thus, the discussion or

PAGE 194

194 comments were moderated and directed at all learners rather than to individuals. Another participant informed that they used paid teaching assistants to re spond to each l or questions. The teaching assistants were also responsible to identify common areas of concern by a number of learners, and respond to these comments and questions. Based on the findings of this study, to develop conceptual and deep lear ning, the intervention by subject matter experts were required in order to provide feedback, enlighten misconceptions or misunderstandings, and even clarify arguments (Memmel, Wolpers, Condotta, Niemann, & Schirru, 2010). It might make sense to address thi s critical learner to instructor interaction in the catalog of design patterns. Content and Course Material Based on the findings of this study, it was surprising to know that b eta testing was not performed anymore before a MOOC was released. Beta testing is a great way to ensure the effectiveness of an online course, and one of the critical steps to uncover content errors, usability, software bugs, and level of user involvement (Sekhon & Hartley, 2014). Coursera for instance recruited beta testers to spot any errors before launch their courses (Coursera, 2016). Beta testers refer to learners who enroll a course for the first time, and they are provided with a list of questions to consider while reviewing a course. They have to provide invaluable feedback t o course instructors after reviewing the community is limited to those who have tested a computer science course. It would be wonderful if all MOOC providers would consider beta testing be fore they go live. Alternatively, participants in this study mentioned they were more into peer reviewing scripts and partnering a professor wit h instructional designers who experts in

PAGE 195

195 doing online courses before the video production. It was interesting to note that one participant confirmed three to four cycles of learner evaluation were planned through the course to obtain learner feedback before and after the course. As with any software engineering process, understanding learner requirements is sign ificant to design and develop an effective MOOC. Furthermore, another participant indicated they encouraged learners to post any feedback to the instructors. The instructors and instructional designers of MOOCs then adapted or changed course materials as a result of the learner feedback. Instructional Strategy and Learning Outcomes actually a match between the instruction and the learning goals... as well as the assessments that a Learning is promoted when the demonstration is consistent with the learning goal onsistency: Learning is promoted when the practice is consistent catalog of design patterns. While successful completion has been a subject of interest to researc hers, understanding the factors influencing learning outcomes, particularly learning activities and teaching context are significant as they are important steps towards designing high quality open online courses. Participants in this study clearly indicate d that providing immediate feedback and fostering learner community are among the most effective methods. Rapid feedback on computer marked assignments can be extremely valuable

PAGE 196

196 for formative assessment, which allowing learners to determine if they underst ood the concepts covered in the MOOCs Knowledge Activation One participant clarified an even more basic concern when responding to a question about knowledge activation. For him, activation is more than just requires learners to recall prior knowledge or provide relevant experience. He pointed out the importance of learning activities that can be designed to stimulate the development of esson in each unit is the introduction to the unit as a way to connect the lesson to other areas of the course material, as well as how learners can apply those in the real ners were also asked to recall what they learned in the the First Principles of Inst Previous experience: Learning is promoted when learners are directed to recall, relate, describe, or apply knowledge from 46). According to Sa ndeen (2013), only 10% of the learners who enroll in the prominent MOOCs actually complete the course. motivation have been explored for many years (Barak, Wateed, & Haick, 2016). The result of this study was not surprising, s ince it suggested that those who lost interest in So the only ones who

PAGE 197

197 students who take it because they need tha s where learners give reviews of the courses in the program and it is public ly available. Based on the review, one participant considered two possible explanations why learners enrolled in his course. First, the motivation to learn was based on their previous successful experience they took his earlier courses and they liked him, so they just wanted to take another course with him. Barak et al. (2016) refers this i (p. 50). Second, the motivation to learn was based on their desire to stay updated and informed about the latest technology. This motivation to learn refers to perso nal 2016). Thus, it would be beneficial to consider a design pattern for a student review set in the future. Transfer of Learning Course d esign features are among t h e important factors for promoting transfer to real world situations. The findings suggested that the participants used lots of checkpoints and their assignments were based on that Further, they encouraged learners to complete projects that related to the real course is basically a series of challenges where students are given real websites and those connections out to students to use in the rea l E ncourage learners to do such activities would benefit them, in terms of getting feedback from the instructors, peers, and they could also get to use it for their work. As pointed out in the literature, by reparing learners to apply the knowledge or

PAGE 198

198 skills gained in a learning environment to another context (DuFour, DuFour, Eaker, & Many, 2010) What is perhaps more interesting is that one participant emphasized the value of learning activities and assignme nts when responding to a question about how they provide adequate practice for learners to apply new knowledge or skills for a variety of for demonstrating that you ha ve the capacity to master the material if that makes not the case with most academic learning contexts. Learners are usually taught a broad set of knowledge and skills that they may apply in countless ways Limitations of the Study The literature almost conclusively suggested that design patterns should be written collaboratively. Drafted patterns are typically shared, analyzed, evaluated, and In ot her words, design patterns are a team effort and not created by a single person. Thus, the catalog of design patterns developed in this study should be analyzed and evaluated through a professional dialogue within the instructional design community. Also, there was some overlap between the pattern elements, and this suggested some revision of the design patterns with a group of practitioners are needed. External evaluation and feedback received during an extended process of collaboration can be used to refi ne design patterns, using varying perspectives from different practitioners. Brainstorming with researchers and practitioners involved in designing, developing, and delivering MOOCs is also necessary. In order to get useful information about learner

PAGE 199

199 needs reports, etc. are also significant. perceptions of their instructional design experience subjective and not necessarily verifiable by other sources of data. Furthermore, participants may not necessarily be able to provide the optimal method to design, develop, and deliver computer science MOOCs. However, underst anding the experiences of these expert instructional designers may well lead to valuable insights into ways to enhance open online education, as well as more structured informal educational experiences for novice instructional designers. The main challeng e of cataloging design patterns was no agreed set of guidelines procedures and standards to define, organize analyze, and evaluate patterns. Design patterns are not written down at once and forever in which they are always work in progress ( Kohls & Utt echt, 2009) Every feedback on the patterns bring s in a new perspective and every successful application strengthens a pattern or introduces a new variation. On the other hand, each failure reflects constraints on the pattern and therefore a clear underst anding about its context as well as applicability can be realized. The target group for this study was instructional designers. Some instructional designers might argue that design patterns limit their creativity by following the prescribed and proposed so lutions. It might be true as design patterns describe design increase complexity, especially for novice designers who lack experience in designing

PAGE 200

200 open online courses. Whil e design patterns developed in this study provided a detailed description of a problem and its solution in a generic form that can be understood by instructional designers, as well as professors, instructors, and teaching assistants, it must be noted that some of the problems and solutions were either too general or too specific. Cataloging design patterns is important as it provides a common vocabulary for practitioners to communicate, document, and explore design alternatives. The first draft of the catal og was only a start, it was an effort to document the expertise of practitioners in open online learning. The catalog just documented the most common and existing design patterns that expert instructional designers used in designing open online courses. Th e underlying idea behind design patterns is to guide rather than instruct, a feature that makes them potentially a useful tool for designing effective learning courses. The catalog of design patterns did not provide a rigorous method, nor did it present a complete set of patterns that offers step by step instructions for designing an open online course. This study just documented the most common and related design patterns. Also, the design patterns were written at a higher level of abstraction, making a de sign seems less complex. Hence, creativity is absolutely needed in using the catalog of design patterns. Design scenarios represent the final stage of the design pattern development activity (Warburton & Mor, 2015). This study, however, only proposed one s imple design scenario. Design scenarios usually use as design challenges to test design patterns. Future studies should include more authentic scenarios for open online learning activities.

PAGE 201

201 Finally, I did not receive the degree of response to the interview request. Over 30 instructional designers were contacted for expert interviews, only two participated and completed the interviews. The number of interview participants was relatively small and more experts would probably improve the results of this study. Although these responses contributed important aspects to the findings, they were not adequate to address some of the open topics. Recommendations for Future Research The catalog of open online learning design patterns developed in this study should be f urther evaluated and refined through shepherding and pattern writing workshops. Design patterns are documented to allow other people learn from a good design (Kohls & Scheiter, 2008). Thus, it is important to evaluate if the catalog of design patterns coul d actually assist both novice and expert instructional designers in designing open online courses. The evaluation process should involve expert instructional designers and subject matter experts, allowing better evaluations of the resulting design patterns (Kohls & Scheiter, 2008). In order to refine and perfect each design pattern, experts could provide guidance to the pattern author by giving specific suggestions on what could be done to improve it. The iterative and constructive feedback received during shepherding and pattern writing workshops not only could refine design patterns, but also could finalize design patterns for public release. Other than that, it is important to conduct experiments to investigate the effect of pattern application. This eff ort should be performed to demonstrate the use of design patterns by the application to the design of new open online courses.

PAGE 202

202 Following evaluation and application, the catalog of open online learning design patterns should be published in an online reposi tory for dissemination purposes. An online repository, for instance, in the form of Wiki could serve as a hub to foster collaboration between instructional designers and practitioners, allowing them to access, share, capture, modify, and apply design patte rn knowledge and solutions to a specific instructional problem. Many people are familiar with the Wiki functionalities. Thus, instructional designers can easily share the results of pattern writing and evaluations of pattern applications, as well as provid e recommendations and discuss issues related to design patterns. Eventually, I hope to replicate this type of study in computer science cMOOCs. cMOOC is another form of massive open online learning based on collaboration and networking, fundamentally diffe rent from xMOOCs but more appropriate to address the needs of learners in a digital age. Since the focus of xMOOCs is more on the individual learner, the courses are didactic in nature in which the delivery of materials is via multimedia and videos, along with interactive and automated assessment to provide feedback on learning. On the other hand, cMOOCs emphasis on learning in a social context, enabling participants to create their own personal learning environment through the use of social media. Conclusi on The use of design patterns has evolved to solve problems often encountered in architecture, prominently in software engineering and human computer interaction, and recently in the instructional design communities. Design patterns consist of reusable sol utions generalized from a number of successful design cases and best practices. This study developed 15 design patterns that captured and presented solutions to

PAGE 203

203 problems in which the instructional designers could use when designing open online learning for computer science courses. The catalog of design patterns was based on the self observation, thorough analysis of open online courses functionalities, literature review on pedagogical strategies, expert interviews, and existing published patterns in other related areas. Further research is needed to consider the analysis of learner log files in order to refine the design patterns. study. First principles prescribe a task ce ntered approach that integrates the solving of problems encountered in real world situations with a direct instruction of problem components. The fifteen design patterns presented in this study can be used in conjunction with other few principles for teach ing materials and learning activities, such as the collaboration, interaction, motivation, and navigation in designing a quality open online learning for computer science courses. Besides, this study also proposed a template to the instructional design com munity on how to effectively document and communicate design patterns in open education context. Designers can use this template to express their design expertise to other instructional design professionals and also make use of design patterns in practice.

PAGE 204

204 APPENDIX A RESEARCH PARTICIPATION REQUEST Study Title: Cataloging Design Patterns for Open Online Learning in Computer Science Courses Are you a current or former instructional designer? Have you created computer science modules for open online learning in the higher education environment? If your answer to each of the above questions is yes, Nor Hafizah Adnan from University of Florida is conducting research to exp l ore the experience and expertise of instructional designers in designing massive open online courses. The intention is to develop a catalog of design patterns for open online learning, a template for documenting and reusing successful design solutions. Complete the initial Member Consent below to take part in the study. You will be s ent a link to take a short (approximately 10 minutes) online survey to determine your eligibility for the study. If you qualify, you will be contacted to schedule a web based interview to discuss your instructional design experiences. You will also be em ailed the official consent form. The time required for participants in the online interview is approximately 60 minutes. If you have additional questions please contact Nor Hafizah Adnan at hafizah@ufl.edu

PAGE 205

205 APPENDIX B PRE SURVEY QUESTIONNAIRE 1. What is your gender? 2. What is your age range? 3. What is your highest degree earned? 4. What is your current title at the institution? 5. Do you have experience designing, developing, and delivering a Massive Open Online Course (MOOC) in th Questions 6, 7, 8, and 9. 6. At which MOOC platform or provider have you designed? 7. What course have you designed? 8. Who is the target audience for the MOOC? 9. On average, how many learners participate in the M OOC? 10. If you wish to help further and participate in a web based interview, please provide the dates and times. 11. Please provide your email address.

PAGE 206

206 APPENDIX C INTERVIEW QUESTIONS 1. Do you use any existing content from the web in your MOOC? If so, what ways do you incorporate existing web content into your MOOC design? 2. What types of learner to learner and learner to content interactions (interactivity, communication, and collaboration) are available within your MOOC? Probe: How do you facilitate the lea rner to learner and learner to content interactions within your MOOC? 3. What media and technologies are being used in your MOOC and for what purpose (e.g., video > content presentation)? Probes: How do you assist learners in becoming familiar and comforta ble with the technologies used and/or operations of MOOC features? How do learners receive support when they might need assistance within the MOOC? 4. Did you evaluate the effectiveness of your MOOC before it went live? If so, describe the process? 5. How do lea rners measure or track personal learning progress? Probes: How do you help learners demonstrate their newly acquired skills? How do learners reflect within the course on what they have learned? 6. How do learners receive feedback on their learning (e.g., auto mated grading)? Probe: Do they receive frequent feedback on the strengths and weaknesses in their demonstrated learning? 7. How do you obtain learner feedback during the course? How do you obtain learner feedback after the course?

PAGE 207

207 8. How do you assess the learn ing outcomes or results of those that participate in the MOOC? 9. In your experience, which approaches to instruction have proven to be the most effective in the implementation of the MOOC? Probes: Which factors affected your decision of adopting a particular approach for your instruction? Which factors affected your decision of not adopting a particular approach for your instruction? What types of approaches will you use in the future? Now I'd like to go through your MOOC a step at a time and ask some questio ns. 10. What knowledge, skills, and dispositions are needed for the successful completion of the MOOC? 11. Probes: What are the important steps in your instructional sequence? Do you provide di fferent kinds of practice for different instructional goals? 12. Identify the lesson in your MOOC that performed the best. Probes: What do you think worked about this particular lesson in the MOOC? How do you ensure that the demonstration (if any) is consiste nt with the learning goal? 13. What motivates learners to participate in the MOOC? 14. What design features do you use to promote the activation of prior knowledge? 15. What design features do you use to promote transfer to real world situations? Probe: How do you go about providing adequate practice for learners to use their new knowledge or skill for a variety of problems?

PAGE 208

208 APPENDIX D INTERVIEW TRANSCRIPT Question (Q): Your participation in this study is completely voluntary and responses will remain anonymous You have the right to withdraw from the study at any time if you feel uncomfortable. You do not have to answer any questions you do not want to answer. Thank you for agreeing to participate. [Q1 Do you use any existing content from the web in your MOOC? ] Answer (A): Yes, I do. Q: If so, what ways do you incorporate existing web content into your MOOC design? A: I incorporate other sites from the web as examples or documentation. [Q2 What types of learner to learner and learner to content interactions (interactivity, communication, and collaboration) are available within your MOOC? Probe: How do you facili tate the learner to learner a nd learner to content interactions within your MOOC?] A: Well, the students have discussion forums and Slack for communi cation. Students can even connect to one another in person through Udacity Connect. [Q3 What media and technologies are being used in your MOOC and for what purpose (e.g., video > content presentation)? Probes: How do you assist learners in becoming famil iar and comfortable with the technologies used and/or operations of MOOC features? How do learners receive support when they might need assistance within the MOOC?] A: We use video for quiz introductions, quiz solutions, tutorials, intros, outros, and expl anations, and text for supporting information. While interactive programming quizzes for student self assessment. [Q4 Did you evaluate the effectiveness of your MOOC before it went live? If so, describe the process?] [Q5 How do learners m easure or track personal learning progress? Probes: How do you help learners demonstrate their newly acquired skills? How do learners reflect within the course on what they have learned?] A: We display their progress on the site. Students demonstrate skill s by building projects.

PAGE 209

209 [Q6 How do learners receive feedback on their learning (e.g., automated grading)? Probe: Do they receive frequent feedback on the strengths and weaknesses in their demonstrated learning?] A: Well, students receive automated feedback on most programming exercises and all other types of quizzes frequently. Students who are enrolled in the Nanodegree get personalized feedback from a grader on their project (cumulative assessment) at the end of the course. [Q7 How do you obtain learner f eedback during the course? How do you obtain learner feedback after the course?] A: We watch the forums, talk to students and watch engagement numbers (number of learners progressing into the course). [Q8 How do you assess the learning outcomes or results of those that participate in the MOOC?] A: We measure the number of students who get jobs after completing the Nanodegree. [Q9 In your experience, which approaches to instruction have proven to be the most effective in the implementation of the MOOC?] A: Interactive and constant activities for students. Q: Which factors affected your decision of adopting a particular approach for your instruction? Which factors affected your decision of not adopting a particular approach for your instruction? What type s of approaches will you use in the future? A: Engaging presentation of the content, less P ower P oint and more friend sitting next to you explaining things. I used to teach science. Hands on is the best, so I made my classes as hands on as possible. Q: Now I'd like to go through your MOOC a step at a time and ask some questions. [Q10 What knowledge, skills, and dispositions are needed for the successful completion of the MOOC?] A: Basic web development skills, good attitude, and willingness to try things. [ What are the important steps in your instructional sequence? Do you provide different kinds of practice for different instructional goals?] A: Cannot NDA.

PAGE 210

210 [Q12 Identify the lesson in your MOOC that performed the best. Probes: What do you think worked about this particular lesson in the MOOC? How do you ensure that the demonstration (if any) is consistent with the learning goal?] [Q13 What mot ivates learners to participate in the MOOC?] A: Improve web deve lopment skills and build websites faster [Q14 What design features do you use to promote the activation of prior knowledge?] A: Practice. Students constantly work with sites. [Q15 What design features do you use to promote transfer to real world situations? Probe: How do you go about providing adequate practice for learners to use their new knowledge or skill for a variety of problems?] A: The entire course is basically a series of challenges where students are given real websites and are asked to debug them.

PAGE 211

211 LIST OF REFERENCES Alexander, C., Ishikawa, S., Silverstein, M., Jacobson, M., Fiksdahl King, I., & Angel, S. (1977). A Pattern Language New York: Oxford University Press. Alexander, C. (1979). The timeless way of building New York: Oxford University Press. Ahmed, S. (2005). Encouraging reuse of design knowledge: A method to index knowledge. Design Studies, 26 565 592. Retrieved from http://orbit.dtu.dk/en/publications/encouraging reuse of design knowledge(c51c4b9d cb20 49d6 8dbf 5c42b587b63c).html Andre, T. (1997). Selected microinstructional methods to facilitate knowledge construction: Implications for instructional design. In R. D. Tennyson, F. Schott, N. Seel, & S. Dijkstra, Instructional design: International perspective: Theory, research, and models (Vol. 1) (pp. 243 267). Mahwah, NJ: Lawrence Erlbaum Associates. Avgeriou, P., Papasalouros, A., Retalis, S., & Skordalakis, M. (2003). Towards a pattern language for learning management systems. Educational Technology & Society 6 (2), 11 24. Retrie ved from http://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=13FA0453706712F955 F3A6A6AFDCAEF9?doi=10.1.1.126. 4853&rep=rep1&type=pdf Baggetun, R., Rusman, E., & Poggi, C. (Eds.). (2004). Design p atterns f or c ollaborative l earning: From p ractice t o t heory a nd b ack. Proceedings of the ED Lugano, Switzerland. Retrieved from http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.103.4274&rep=rep1&ty pe=pdf Balfour, S. P. (2013). Assessing writing in MOOCs: Automated essay scoring and calibrated peer review. Research & Practice in Assessment, 8 40 48. Retrieved from https://pdfs.semanticscholar.org/2c03/81a33a503179eaa74630576cc7be5dc23e b4.pdf Barak, M., Wateed, A., & Haick, H. (2016). Motivation to learn in massive open online courses: Examining aspects of language and social engagement. Computers & Education, 94, 49 60. doi: https://doi.org/10.1016/j.compedu. 2015.11.010 Bibi, T. (2010). Analysis of cognitive tutor geometry curriculum (Doctoral dissertation). Available from Iowa State University Digital Repository Graduate Theses and Dissertations. Retrieved from http://lib.dr.iastate.edu/cgi/viewcontent.cgi?article=2582&context=etd Bokhorst, F., Moskaliuk, J., & Cress, U. (2014). How patterns support computer mediated exchange of knowledge in use. Computers & Education, 71 153 164. d oi: 10.1016/j.compedu.2013.09.021

PAGE 212

212 Bonk, C. J., Lee, M. M., Reeves, T. C., & Reynolds, T. H. (2015). MOOCs and open education aroun d the world. New York, NY: Routledge. Borchers, J. (2001). A pattern approach to interaction design New York, NY: John Wiley & Sons. Brandl, D. (2007). Design patterns for flexible manufacturing Research Triangle Park, NC: Instrumentation, Systems and Au tomation Society. Breslow, L. B., Pritchard, D. E., DeBoer, J., Stump, G. S., Ho, A. D., & Seaton, D. T. MOOC. Research & Practice in Assessment, 8 13 25. Retrieved from http://www.rpajournal.com/dev/wp content/uploads/2013/05/SF2.pdf Cen, H., Koedinger, K. R., & Junker, B. (2007). Is over practice necessary? Improving learning efficiency with the co gnitive tutor t hrough educational data mining. In Luckin, R., Koedi nger, K. R. & Greer, J. (Eds.). Proceedings of 13th International Conference on Artificial Intelligence in Education (AIED2007), 511 518. Amsterdam, IOS Press. Retrieved from https://dl.acm.org/citation.cfm?id=1563681 Chapman, S. A., Goodman, S., Jawitz, J., & Deacon, A. (2016). A strategy for monitoring and evaluating massive open online courses. Evaluation and Program Pla nning, 57 55 63. doi: https://doi.org/10.1016/j.evalprogplan.2016.04.006 Chen, Y. H., & Chen, P. J. (2015). MOOC study group: Facilitation strategies, influential factors, and student perceiv ed gains. Computers & Education, 86 55 70. Chi, M. T., Feltovich, P. J., & Glaser, R. (1981). Categorization and representation of physics problems by experts and novices. Cognitive science 5 (2), 121 152. Retrieved from https://pdfs.semanticscholar.org/16ef/4cc3a80ee7ba8f59e0a55b2ef134c31e18b3 .pdf Chung, E., Hong, J., Lin, J., Prabaker, M., Landay, J., & Liu, A. (2004). Development and evaluation of emerging desi gn patterns for ubiquitous computing. Proceedings of the 2004 Conference on Designing Interactive Systems: Processes, Practices, Methods, and Techniques, Cambridge, MA (pp. 233 242). doi: 10.1145/1013115. 1013148 Coursera. (2016, March 8). Introduction to the beta tester community. Retrieved from http://www.coursera.community/introduction to the beta tester community/ Devlin, T. J., Feldhaus, C. R., & Bentrem, K. M. (2013). The evolving classroom: A study of traditional and technology based instruction in a stem classroom. Journal of Technology Education, 25 (1), 34. Retrieved from http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.904.9040

PAGE 213

213 DuFour, R., DuFour, R., Eaker, R., & Many, T. (2010). Learning by doing: A handbook for professional learning communities at work Bloomington, IN: Sol ution Tree. Gardner, H. (1999). Multiple approaches to understanding. In C. M. Reigeluth (Ed.), Instructional design theories and models: A new paradigm of instructional theory ( Vol. II) (pp. 69 89). Mahwah, NJ: Lawrence Erlbaum Associates. Gardner, J. (20 systematic review. Midwest Journal of Educational Communication and Technology, 5 (1), 8 17. Retrieved from https://www.academia.edu/353 8793/Gardner_J._2011_._Investigating_theoretica l_support_for_first_principles_of_instruction_A_systematic_review._Midwest_Jou rnal_of_Educational_Communication_and_Technology._5_1_pp._8 17 Gardner, J. (2011b). How award winning professors in higher educatio n use Merrill's rst principles of instruction. International Journal of Instructional Technology and Distance Learning, 8 (5), 3 16. Retrieved from https://www.academia.edu/3538810/Gardner_J._2011_._How_award winning_professors_in_higher_education_use_Merrill_s_first _principles_of_instr uction._International_Journal_of_Instructional_Technology_and_Distance_Learni ng_8_5_p._3 16 Gay, L. R., Mills, G. E., & Airasian, P. (2009). Educat ional research competencies for analysis and applications (9 th ed.). Columbus, OH: Merril l Pearson. Gamma, E., Helms, R., Johnson, R., & Vlissides, J. (1995). Design patterns: Elements of reusable object oriented software Reading, MA: Addison Wesley Professional. Gameel, G. G. (2017). Learner satisfaction with massive open online courses. Am erican Journal of Distance Education, 31(2), 98 111. d oi: https://doi.org/10.1080/08923647.2017.1300462 Gibbons, A. S. (2010). A contextual framework for identifying instructional design patterns. In P. Goodyear & S. Retalis (Eds.), Technology enhanced learning: Design patterns and pattern languages (pp. 29 47). Rotterdam, The Netherlands: Sense Publisher. G omez Perez, A., Fernndez Lpez, M., & Corcho, O. (2006). Ontological Engineering: with examples from the areas of Knowledge Management, e Commerce and the Semantic Web London, UK: Springer Science & Business Media. Gould, J. D., & Lewis, C. ( 1985). Desig n for usability: Key principles and what designers think. Communications of the ACM, 28(3), 300 311. doi: 10.1145/3166.3170 Greene, J. C., Caracelli, V. J., & Graham, W. F. (1989). Toward a conceptual framewor k for mixed method evaluation designs. Educational Evaluation and Policy

PAGE 214

214 Analysis, 11, 255 274. Retrieved from https://www.jstor.org/stable/1163620?seq=1#page_scan_tab_con tents Groen, G. J., & Patel, V. L. (1985). Medical problem solving: some questionable ssumptions. Medical Education 19 (2), 95 100. Retrieved from https://www.ncbi.nlm.nih.gov/pubmed/3982318 Ferguson, R. Clow, D. Beale, R. Cooper, A. J. Morris, N. P. Bayne, S., & Woodgate, A. (2015). Moving through MOOCS: Pedagogy, learning design and patterns of engagement. In Design for Te aching and Learning in a Networked World. Lecture Notes in Computer Science, vol 9307. Springer, Cham. doi: 10.1007/978 3 319 24258 3_6 Fini, Antonio. (2009). The technological dimension of a massive open online course: The case of the CCK08 course tools. The International Review of Research in Open and Distributed Learning, 10 (5). Retrieved from http://www.irrodl.org/index.php /irrodl/article/view/643/1402 Frizell, S. S., & Hubscher, R. (2002 June 24 29 ). Supporting the application of design patterns in web course design. Proceedings of the 14th World Conference on Educational Multimedia, Hypermedia & Telecommunications Denver, Colorado Retrieved from https://eric.ed.gov/?id=ED477010 Hanley, G. L. (2015). MOOCs, MERLOT, and open educational services. In C. J. Bonk, M. M. Lee, T. C. Reeves & T. H. Reynolds (Eds.), MOOCs and o pen education around the world New York, NY: Routledge. Hardiman, P. T., Dufresne, R., & Mestre, J. P. (1989). The relation between problem categorization and problem solving among experts and novices. Memory & Cognition 17 (5), 627 638. Retrieved from https://link.springer.com/article/10.3758/BF03197085 Hevner, A. R., March, T. S., Park, J., & Sudha, R. (2004). Design science in information systems research. MIS Quarterly, 28 (1), 75 105. R etrieved from https://dl.acm.org/citation.cfm?id=2017217 Hoadley, C., & Cox, C. (2009). What is design knowledg e and how do we teach it. In C. DiGiano, S. Goldman, & M. Chorost (Eds.), Educating lear ning technology designers: Guiding and inspiring creators of innovative educational tools (pp. 19 35). New York, NY: Routledge. Retrieved from https://steinhardt.nyu.edu/scms Admin/uploads/004/301/l3d%20chp2.pdf Hone, K. S., & El Said, G. R. (2016). Exploring the fact ors affecting MOOC retention: A survey study. Computers & Education, 98 157 168. doi: http://dx.doi.org/10.1016/j.compedu.2016.03.016 Hutchins, S. G., Pirolli, P. L., & Card, S. K. (2004). A new perspective on use of the critical decision method with intelligence analysts. Ninth International Command

PAGE 215

215 and Control Research and Technology Symp osium San Diego, CA: Space and Naval Warfare Systems Center. Retrieved from http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.196.535&rep=rep1&typ e=pdf Inventado, P. S., & Scupelli, P. (2015). Design patterns for open online teaching. eLearning Papers, 42, 14 27. Retrieved from https://www.openeduca tioneuropa.eu/en/paper/design patterns open online teaching and learning Johnson, R. B., & Onwuegbuzie, A. J. (2004). Mixed methods research: A research paradigm whose time has come. Educational Researcher, 33 (7), 14 26. Retrieved from http://journals.sagepub.com/doi/abs/10.3102/0013189X033007014 Jonassen, D. (1999). Designing constructivist learning environments. In C. M. Reigeluth (Ed.), Instructional design theories and models: A new paradigm of instructional theory (Vol. II) (pp. 215 239). Mahwah, NJ: Lawrence Erlbaum Associates. Jordan, K. (2014). Initial trends in enrolment and completion of massive open online courses. The International Review of Research in Open And Distance Learning, 15 (1), 134 160. Retrieved from http://www.irrodl.org/index.php/irrodl/article/view/1651 Kauffman, H. (2015). A review of predictive factors of student success in and satisfaction with online learning. Research in Learning Technology, 23 26507 doi: https://doi.org/10.3402/rlt.v23.26507 Keppe ll, M. (2000, December 9 14). Principles at the heart of an instructional designer: Subject matter expert interaction. Proceedings of the 17th Annual Conference for Australian Society in Learning in Tertiary Education Southern Cross University, Coffs Harb our. Khan, B. H. (2016). Revolutionizing modern education through meaningful e learning implementation. Hershey, PA: IGI Global. Kohls, C., & Scheiter, K. (2008 October 18 20 ). The relation between design patterns and schema theory. Proceedings of the 15t h Conference on Pattern Languages of Programs Nashville, Tennessee. doi: 10.1145/1753196.1753214 Kohls, C., & Uttecht, J. G. (2009). Lessons learnt in mining and writing design patterns for educational interactive graphics. Computers in Human Behavior, 25 1040 1055 doi: 10.1016/j.chb.2009.01.004 Koszalka, T., Russ Eft, D., & Reise r, R. (2013). Instructional design competencies: The standards (4 th ed.). Charlotte, NC: Information Age Publishing.

PAGE 216

216 Krippendorff, K. (2012). Content analysis: An introduction to its methodology. Thousand Oaks, CA: Sage Publications. Retrieved from https://www.ocac.cl/wp content/uploads/2015/01/Klaus Krippendorff Content analysis. An introduction to its methodology.pd f Lin, L., & Cranton, P. (2015). Informal and self directed learning in the age of massive open online courses (MOOCs). In O. Mejiuni, P. Cranton, & O. Tw (Eds.), Measuring and analyzing informal learning in the digital age (pp. 91 104). Hershey, PA: IG I Global. Retrieved from http://courseweb.unt.edu/llin/LinPublications/bookChapters/2015LinCranton_MO OCS_chapter7.pdf Lin, Y. L., Lin, H. W., & Hung, T. T. (2015). Value hierarchy for massive open online courses. Computers in Human Behavior, 53 408 418. d oi: https://doi.org/10.1016/j.chb.2015.07.006 Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic Inquiry Newbury Park, CA: Sage. Liyanagunawardena, T., Adams, A., & Williams, S. (2013). MOOCs: a systematic study of the published literature 2008 2012. International Review of Research in Open and Distance Learning, 14 (3), 202 227. Mahdjoub i, D. (2003, June). Epistemology of design Paper presented at the 7th World Conference on Integrated Design and Process Technology, Austin, Texas. Margaryan, A., Bianco, M., & Littlejohn, A. (2015). Instructional quality of massive open online courses (MO OCs). Computers & Education, 80 77 83. doi: http://dx.doi.org/10.1016/j.compedu.2014.08.005 Markoff, J. (2013, April 4). Essay grading software offers professors a break. The New York Times. Re trieved from http://www.nytimes.com/2013/04/05/science/new test for computers grading essays at college level.html Memmel, M., Wolpers, M., Condotta, M., Niemann, K., & Schirru, R. (2010). Introducing a social backbone to support access to digital resources. Proceedings of the 5th European Conference on Technology Enhanced Learning Barcelona, Spain (pp. 151 166 ). Retrieved from https://dl.acm.org/citation.cfm?id=1889308 Milligan, C., & Littlejohn, A. (2016). How health professionals regulate their learning in massive open online courses. Internet and Higher Education, 31 113 121. doi: 10.1016/j.iheduc.2016.07.005 MOOC Design Pattern (2015). Retrieved from http://www.moocdesign.cde.london.ac.uk/ McA ndrew, P., Goodyear, P., & Dalziel, J. (2006). Patterns, designs and activities: Unifying descriptions of learning structures. International Journal of Learning Technology, 2 (2), 216 242. doi: 10.1504/IJLT.2006.010632

PAGE 217

217 McCarthy, B., & McCarthy, D. (2006). Teaching around the 4MAT cycle: Designing instruction for diverse learners with diverse learning styles. Thousand Oaks, CA: Corwin Press. McInnerney, J. M., & Roberts, T. S. (2004). Online Lea rning: social interaction and the creation of a sense of community. Educational Technology & Society, 7(3), 73 81. Retrieved from http://citeseerx.ist.psu.edu/ viewdoc/download?doi=10.1.1.99.9614&rep=rep1&typ e=pdf Merrill, M. D. (2002). First principles of instruction. Educational Technology Research and Development, 50(3), 43 59. Retrieved from http://mdavidmerrill.com/Papers/firstprinciplesbymerrill.pdf Milligan, C., & Littlejohn, A. (2014). Supporting professi onal learning in a massive open online course. The International Review of Research in Open and Distance Learning, 15 (5), 197 213. doi: http://dx.doi.org/10.19173/irrodl.v15i5.1855 Nelson, L. M. (1999). Collaborative problem solving. In C. M. Reigeluth (Ed.), Instructional design theories and models: A new paradigm of instruct ional theory (Vol. II) (pp.241 267). Mahwah, NJ: Lawrence Erlbaum Associates. Niehaves, B. (2007). On epistemological diversity in design science new vistas for a design oriented IS research Paper presented at the 28th International Conference on Information Systems, Montreal, Canada. Retrieved from http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.231.2753&rep=rep1&ty pe=pdf Pane, J. F., McCaffr ey, D. F., Slaughter, M. E., Steele, J. L., & Ikemoto, G. S. (2010). An experiment to evaluate the efficacy of cognitive tutor geometry. Journal of Research on Educational Effectiveness, 3 (3), 254 281. Retrieved from https://www.learntechlib.org/p/107052/ Pollock, E., Chandler, P., & Sweller, J. (2002). Assimilating complex information. Learning and Instruction, 12 61 86. doi: https://doi.org/10. 1016/S0959 4752(01)00016 0 Retalis, S., Georgiakakis, P., & Dimitriadis, Y. (2006). Eliciting design patterns for e learning systems. Computer Science Education,16 (2), 105 118. d oi: https://doi.org/10. 1080/08993400600773323 Retbi, A., Merrouch, F., Idrissi, M. K., & Bennani, S. (2012). Towards a visual educational modeling language for effective learning. International Journal of Computer Science Issues, 9 (3), 382 390. Retrieved from https://www.researchgate.net/profile/Samir_Bennani2/publication/268269631_To wards_ a_visual_Educational_Modeling_Language_for_effective_learning/links/5 52d13fa0cf21acb09212a82.pdf

PAGE 218

218 Richey, R., Klein, J., & Tracey, M. (2011). The instructional design knowledge base: Theory, research, and practice New York, NY: Routledge. Ruey, S. (2010). A case study of constructivist instructional strategies for adult online learning. British Journal of Educational Technology, 41(5), 706 720. doi: 10.1111/j.1467 8535.2009.00965.x Rusman, E., va n Bruggen, J., Corvers, R., Sloep, P. & Koper, R. (2009). From pattern to practice: Evaluation of a design pattern fostering trust in virtual teams. Computers in Human Behavior, 25 1010 1019 d oi: htt ps://doi.org/10.1016/j.chb.2009.01.002 Sallis, E., & Jones, G. ( 2002). Knowledge management in education: Enhancing learning & education. New York, NY: Routledge. Research & Practice in Assessm ent, 8 (1), 5 12. Retrieved from http://www.rpajournal.com/dev/wp content/uploads/2013/05/SF1.pdf Savenye, W. C., & Robinson, R. S. (2004). Qualitative research issues and methods: An introduction for educational technologists. In D. H. Jonassen (Ed.), Handbook of research on educational communications and technology (2nd ed., pp. 1045 1071). Mahwah, New Jersey: Lawrence Erlbaum. Schank, R. C., Berman, T. R. & Macperson, K. A. (1999). Learning by doing. In C. M. Reigeluth (Ed.), Instructional design theories and models: A new paradigm of instructional theory (Vol. II) (pp. 161 181). Mahwah, NJ: Lawrence Erlbaum Associates. Sekhon, M., & Hartley, D. (2016). Basics of e learnin g revisited Alexandria, VA: American Society for Training & Development. Song, L., Singleton, E. S., Hill, J. R., & Koh, M. H. (2004). Improving online learning: student perceptions of useful and challenging characteristics. Internet and Higher Education 7 (1), 59 70. Retrieved from https://www.learntechlib.org/p/102596/ Steenbergen Hu, S., & Cooper, H. (2013). A meta analysis of the effectiveness of intelligent tutoring systems on K tical learning. Journal of Educational Psychology, 105 (4), 970 987 doi: 10.1037/a0032447 Stich A. E., Reeves T. D. (2017). Massive open online courses and underserved students in the United States. Internet an d Higher Education 32, 58 71. d oi: 10.1016/j.iheduc.2016.09.001 Staubitz, T., Petrick, D., Bauer, M., Renz, J., & Meinel C. (2016). Improving the peer assessment experience on MOOC platforms. Proceedings of the 3rd ACM Conference on Learning Scale New York, NY (pp. 389 398). doi: 10.1145/2876034.2876043

PAGE 219

219 Sweller, J., van Merrienboer, J. J. G., & Paas, F. (1998). Cognitive architecture and in structional design. Educational Psychology Review, 10 (3), 251 296. Retrieved from http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.89.9802&rep=rep1&typ e=pdf Schwartz, D. L., Brophy, S., Lin, X., & Bransford, J. D. (1999). Software for managing complex learning: Examples from an educational psychology course Educational Technology Research and Development, 47 (2), 39 59. Retrieved from https://link.springer.com/article/10.1007/BF02299464 van Diggelen, W., & Overdijk, M. (2009). Grounded design: Design patterns as the link between theory and practice. Computers in Human Beh a vior, 25 1056 1066. doi: https://doi.org/10.1016/j.chb.2009.01.005 van der Veer, G. C. & Melguizo, M. C. (2002). Mental models. In J.A. Jacko & A. Sears (Eds.) The human computer interaction handbook: Fundamentals, evolving technologies and emerging applications Lawrence Erlbaum & Associates. van Gog, T., & Paas, F. (2008). Data collection and analysis. In J. M. Spector, M. D. Merrill, J. V. Merrienboer & M. P. Driscoll (Eds.), Handbook of r esearch on educational communications and technology New York, NY: Lawrence Erlbaum Associates. van Zundert, M., Sluijsmans, D., & van Merrienboer, J. (2010). Effective peer assessment processes: Research findings and future directions. Learning and Instr uction, 20, 270 279. doi: https://doi.org/10.1016/j.learninstruc.2009.08.004 Wiley, D. (2015). The MOOC misstep and the open education. In J. M. Curtis, M. M. Lee, T. C. Reeves & T. H. Reynold s (Eds.), MOOCs and open education around the world New York, NY: Taylor & Francis. Zitter, I., Kinkhorst, G., Simons, R., & Cate, O. (2009). In search of common ground: A task conceptualization to facilitate the design of (e) le arning environments with design patterns. Computers in Human Behavior, 25 999 1009. doi: https://doi.org/10.1016/j.chb.2009.01.001

PAGE 220

220 BIOGRAPHICAL SKETCH Nor Hafizah Adnan grew up in Kuala Lumpur Mala ysia S he earned her c omputer s cience, with a major in m anagement i nformation s ystem c omputer s cience, with a concentration in s oftware e ngineering, both from the University of Malaya, Malaysia. Prior to joining the faculty at the National University of Malaysia, she was a certified network engineer and worked in the informatio n communications and technology industry for several years. She currently works as an assistant professor at the N ational University of Malaysia.