<%BANNER%>

Using Immersion and Information Visualization to Analyze Human-Virtual Human Interactions

Permanent Link: http://ufdc.ufl.edu/UFE0024140/00001

Material Information

Title: Using Immersion and Information Visualization to Analyze Human-Virtual Human Interactions
Physical Description: 1 online resource (205 p.)
Language: english
Creator: Raij, Andrew
Publisher: University of Florida
Place of Publication: Gainesville, Fla.
Publication Date: 2009

Subjects

Subjects / Keywords: action, after, agents, computer, conversational, education, embodied, environments, human, humans, immersion, information, interaction, medicine, perspective, reality, review, simulation, taking, training, virtual, visualization
Computer and Information Science and Engineering -- Dissertations, Academic -- UF
Genre: Computer Engineering thesis, Ph.D.
bibliography   ( marcgt )
theses   ( marcgt )
government publication (state, provincial, terriorial, dependent)   ( marcgt )
born-digital   ( sobekcm )
Electronic Thesis or Dissertation

Notes

Abstract: We identify verbal and nonverbal communication as the primary way in which humans try to interact with virtual human interfaces, and then use this result to develop three different approaches for analyzing interactions between humans and virtual humans. Each of these approaches is applied to analyzing interactions with virtual humans for training domain-specific interpersonal skills. In providing new approaches to analyzing virtual humans interfaces, we advance the state-of-the-art in facilitating and training interpersonal interactions with virtual human interfaces. Verbal and nonverbal communication is identified as the primary way users try to interact with virtual humans by comparing interactions with virtual humans to similar interactions with real humans. In two user studies (n=82), participants elicited the same information from a virtual and real human using verbal communication. However, participant nonverbal behavior indicated participants were less engaged, insincere, and demonstrated a poorer attitude towards the virtual human. These behavioral differences likely stemmed from the participants' difficulty understanding the virtual human's limited expressive behavior. The Interpersonal Scenario Visualizer (IPSViz) was then developed to enable review, analysis, and evaluation of the communication between a human and a virtual human. IPSViz generates visualizations of a human-virtual human interaction by capturing, logging, and processing the human and virtual human's verbal and nonverbal behavior. A user study (n=27) shows that conducting an interaction with a virtual human and then reviewing that interaction with IPSViz elicits self-reflection on interpersonal skills, including verbal and nonverbal behavior, rapport-building, and communicating clearly under stress. The next system, the Virtual Social Perspective-taking (VSP) system, enables review, analysis, and evaluation of an interaction with a virtual human from the perspective of the virtual human. The VSP system records a virtual human patient's experience of talking to a medical student, and then uses the recording to transport the medical student into the patient's body and relive the conversation through her eyes. The student relives the conversation to better understand the virtual human patient's perspective and learn to address her (and future real patients') fears. The results of a pilot study (n = 16) indicate that VSP encourages reflection on the perspectives of others and elicits self-directed change of behavior in future social interactions. The last system, IPSVizN, enables review, analysis, and evaluation of trends and outliers in human-virtual human interactions. IPSVizN processes groups of human-virtual human interaction logs to generate summary visualizations of the interactions. An evaluation of IPSVizN with representative end-users found that participants were able to rapidly (within minutes) identify trends and outliers in overall group interpersonal skills, including verbal behavior, organization, completeness, empathy, and communicating under stress. Identifying these trends and outliers without IPSVizN would have required hours of manual effort.
General Note: In the series University of Florida Digital Collections.
General Note: Includes vita.
Bibliography: Includes bibliographical references.
Source of Description: Description based on online resource; title from PDF title page.
Source of Description: This bibliographic record is available under the Creative Commons CC0 public domain dedication. The University of Florida Libraries, as creator of this bibliographic record, has waived all rights to it worldwide under copyright law, including all related and neighboring rights, to the extent allowed by law.
Statement of Responsibility: by Andrew Raij.
Thesis: Thesis (Ph.D.)--University of Florida, 2009.
Local: Adviser: Lok, Benjamin C.
Electronic Access: RESTRICTED TO UF STUDENTS, STAFF, FACULTY, AND ON-CAMPUS USE UNTIL 2011-05-31

Record Information

Source Institution: UFRGP
Rights Management: Applicable rights reserved.
Classification: lcc - LD1780 2009
System ID: UFE0024140:00001

Permanent Link: http://ufdc.ufl.edu/UFE0024140/00001

Material Information

Title: Using Immersion and Information Visualization to Analyze Human-Virtual Human Interactions
Physical Description: 1 online resource (205 p.)
Language: english
Creator: Raij, Andrew
Publisher: University of Florida
Place of Publication: Gainesville, Fla.
Publication Date: 2009

Subjects

Subjects / Keywords: action, after, agents, computer, conversational, education, embodied, environments, human, humans, immersion, information, interaction, medicine, perspective, reality, review, simulation, taking, training, virtual, visualization
Computer and Information Science and Engineering -- Dissertations, Academic -- UF
Genre: Computer Engineering thesis, Ph.D.
bibliography   ( marcgt )
theses   ( marcgt )
government publication (state, provincial, terriorial, dependent)   ( marcgt )
born-digital   ( sobekcm )
Electronic Thesis or Dissertation

Notes

Abstract: We identify verbal and nonverbal communication as the primary way in which humans try to interact with virtual human interfaces, and then use this result to develop three different approaches for analyzing interactions between humans and virtual humans. Each of these approaches is applied to analyzing interactions with virtual humans for training domain-specific interpersonal skills. In providing new approaches to analyzing virtual humans interfaces, we advance the state-of-the-art in facilitating and training interpersonal interactions with virtual human interfaces. Verbal and nonverbal communication is identified as the primary way users try to interact with virtual humans by comparing interactions with virtual humans to similar interactions with real humans. In two user studies (n=82), participants elicited the same information from a virtual and real human using verbal communication. However, participant nonverbal behavior indicated participants were less engaged, insincere, and demonstrated a poorer attitude towards the virtual human. These behavioral differences likely stemmed from the participants' difficulty understanding the virtual human's limited expressive behavior. The Interpersonal Scenario Visualizer (IPSViz) was then developed to enable review, analysis, and evaluation of the communication between a human and a virtual human. IPSViz generates visualizations of a human-virtual human interaction by capturing, logging, and processing the human and virtual human's verbal and nonverbal behavior. A user study (n=27) shows that conducting an interaction with a virtual human and then reviewing that interaction with IPSViz elicits self-reflection on interpersonal skills, including verbal and nonverbal behavior, rapport-building, and communicating clearly under stress. The next system, the Virtual Social Perspective-taking (VSP) system, enables review, analysis, and evaluation of an interaction with a virtual human from the perspective of the virtual human. The VSP system records a virtual human patient's experience of talking to a medical student, and then uses the recording to transport the medical student into the patient's body and relive the conversation through her eyes. The student relives the conversation to better understand the virtual human patient's perspective and learn to address her (and future real patients') fears. The results of a pilot study (n = 16) indicate that VSP encourages reflection on the perspectives of others and elicits self-directed change of behavior in future social interactions. The last system, IPSVizN, enables review, analysis, and evaluation of trends and outliers in human-virtual human interactions. IPSVizN processes groups of human-virtual human interaction logs to generate summary visualizations of the interactions. An evaluation of IPSVizN with representative end-users found that participants were able to rapidly (within minutes) identify trends and outliers in overall group interpersonal skills, including verbal behavior, organization, completeness, empathy, and communicating under stress. Identifying these trends and outliers without IPSVizN would have required hours of manual effort.
General Note: In the series University of Florida Digital Collections.
General Note: Includes vita.
Bibliography: Includes bibliographical references.
Source of Description: Description based on online resource; title from PDF title page.
Source of Description: This bibliographic record is available under the Creative Commons CC0 public domain dedication. The University of Florida Libraries, as creator of this bibliographic record, has waived all rights to it worldwide under copyright law, including all related and neighboring rights, to the extent allowed by law.
Statement of Responsibility: by Andrew Raij.
Thesis: Thesis (Ph.D.)--University of Florida, 2009.
Local: Adviser: Lok, Benjamin C.
Electronic Access: RESTRICTED TO UF STUDENTS, STAFF, FACULTY, AND ON-CAMPUS USE UNTIL 2011-05-31

Record Information

Source Institution: UFRGP
Rights Management: Applicable rights reserved.
Classification: lcc - LD1780 2009
System ID: UFE0024140:00001


This item has the following downloads:


Full Text

PAGE 1

1

PAGE 2

2

PAGE 3

3

PAGE 4

Ithankmyresearchadvisorandsupervisorycommitteechair,Dr.BenjaminLok,forgivingmetheopportunitytosucceed;forhisadvice,direction,supportandpatience;andforthehonorofbeingapartoftheVirtualExperiencesResearchGroup(VERG).ThepeopleinVERGhavebeenamazingcolleaguesandfriends.Iamproudtohaveworkedwithallofthem,andIwanttoespeciallyhighlightKyleJohnsen,AaronKotranza,BrentRossen,JohnQuarles,XiyongWang,RobertDickerson,HaroldRodriguez,CyrusHarrison,JoonHaoChuah,LoisCao,andDr.YongHoHwang.IalsowanttothankDr.D.ScottLind,Dr.AmyStevens,Dr.AdelineDeladisma,Dr.DianeBeck,Dr.JuanCendan,Dr.MarcCohen,Dr.PeggyWagner,andallmyothercollaboratorsinmedicalandpharmacyeducation.Theirperspectiveshaveinuencedmyworkgreatly,andithasbeenenlighteningworkingwiththem.IthankDr.RichardFerdig,Dr.PaulFishwick,Dr.ChrisJermaine,andDr.JereyHoforservingonmysupervisorycommitteeandhelpingmeseebroaderperspectivesonmywork.IthanktheUniversityofFloridaalumniwhosupportedmyworkthroughaUniversityofFloridaGraduateAlumniFellowship.Lastly,Ithankmyfamily:mybrotherIrwin;mysisterMadeline;mybrother-in-lawTodd;theBrandonkids;myparentsJoseandElisa;myin-lawsShelly,Barbara,andBrett;mywifeEmily,andourdogRuby.IthankallofthemfortheirloveandsupportthroughoutthiscrazyjourneytomyPh.D.Theymadethisallpossible. 4

PAGE 5

page ACKNOWLEDGMENTS ................................. 4 LISTOFTABLES ..................................... 11 LISTOFFIGURES .................................... 12 ABSTRACT ........................................ 14 CHAPTER 1INTRODUCTION .................................. 16 1.1DrivingIssues .................................. 17 1.1.1TrainingInterpersonalSkillswithVirtualHumans .......... 17 1.1.2After-ActionReviews .......................... 18 1.1.3MedicalInterviewTraining ....................... 19 1.2ThesisStatement ................................ 20 1.3OverviewofApproach ............................. 20 1.3.1Human-VirtualHumanInteractionHuman-HumanInteraction? 20 1.3.2CaptureofVerbalandNonverbalBehavior .............. 21 1.3.3AnalysisofDomain-SpecicInterpersonalSkillswithInformationVisualizationTechniques ........................ 21 1.3.3.1IPSViz ............................. 21 1.3.3.2IPSVizN 22 1.3.4AnalysisofDomain-SpecicInterpersonalSkillswithImmersiveVirtualRealityTechniques ....................... 23 1.4Innovations ................................... 23 1.4.1InterpersonalSimulationsReal-worldInterpersonalInteractions 24 1.4.2VerbalandNonverbalCommunication ................. 25 1.4.3Capture,Process,andDisplay ..................... 25 1.4.4InsightthroughAfter-ActionReview ................. 26 1.4.5After-ActionReview+VirtualHumanExperiences=Self-directedChange .................................. 27 2PREVIOUSWORK ................................. 28 2.1VirtualHumans ................................. 28 2.1.1WhatIsaVirtualHuman? ....................... 28 2.1.2EectiveVirtualHumans ........................ 28 2.1.3VirtualHumanApplications ...................... 30 2.1.4VirtualHumansasSurrogatesforRealHumans ........... 30 2.2Metacognition:Monitoring,Reection,Evaluation,andChange ...... 31 2.3AfterActionReview .............................. 33 2.3.1FundamentalsofAfterActionReview ................. 33 5

PAGE 6

...................... 36 2.3.2.1ApplicationsofAARsystems ................ 36 2.3.2.2CommoncharacteristicsofAARsystems .......... 37 2.4VisualizationofTemporalData ........................ 40 2.4.1InstantandIntervalVisualizations ................... 41 2.4.2PeriodicVisualizations ......................... 42 2.4.3BranchingTimeVisualizations ..................... 42 2.4.4Spatio-TemporalVisualizations .................... 42 2.4.5InteractionwithVisualizations ..................... 43 2.4.6CoordinatedVisualization ....................... 44 2.5VisualizationofInterpersonalCommunication ................ 44 2.5.1TextualCommunication ........................ 45 2.5.2SpokenCommunication ......................... 46 2.5.3NonverbalCommunication ....................... 47 2.6IPSViz,IPSVizN,andVSP .......................... 48 3COMPARINGINTERPERSONALINTERACTIONSWITHAVIRTUALHUMANTOTHOSEWITHAREALHUMAN ....................... 49 3.1Introduction ................................... 50 3.2RelatedWork .................................. 52 3.3VirtualPatientSystem ............................. 53 3.4StudyI:Design ................................. 53 3.4.1Measures ................................. 54 3.4.1.1Elicitingcriticalinformation ................. 54 3.4.1.2Interactionbehavior ..................... 54 3.4.1.3Perceptionsoftheinteraction ................ 55 3.4.2ParticipantBackground ......................... 55 3.4.3Procedure ................................ 55 3.4.3.1Pre-experience ........................ 56 3.4.3.2Experience .......................... 56 3.4.3.3Post-experience ........................ 58 3.5StudyI:ResultsandAnalysis ......................... 58 3.5.1StatisticalAnalysisandNomenclature ................. 58 3.5.2ContentMeasures ............................ 59 3.5.2.1Elicitingcriticalinformation ................. 59 3.5.2.2Educationalgoals ....................... 61 3.5.2.3Empathy ........................... 61 3.5.3Behavior ................................. 62 3.5.3.1Empathy ........................... 62 3.5.3.2Conversationalbehavior ................... 63 3.5.4Expressiveness:VirtualHumanvs.StandardizedPatient ...... 64 3.5.4.1Dierences ........................... 64 3.5.4.2Similarities .......................... 65 3.5.4.3Eectsofexpressiveness ................... 65 6

PAGE 7

............................... 66 3.5.6Post-StudyReections ......................... 67 3.5.7Summary ................................. 69 3.6StudyII:MotivationandDesign ........................ 69 3.6.1Measures ................................. 70 3.6.1.1Content ............................ 70 3.6.1.2Rapport-buildingbehavior .................. 70 3.6.2Procedure ................................ 71 3.7StudyII:ResultsandAnalysis ......................... 72 3.7.1Inter-observerReliability ........................ 72 3.7.2ContentMeasures ............................ 73 3.7.2.1Elicitingcriticalinformation ................. 73 3.7.2.2Patienthistory ........................ 73 3.7.3Rapport-BuildingBehavior ....................... 74 3.7.3.1Processandetiquette ..................... 75 3.7.3.2Empathy ........................... 76 3.7.3.3Nonverbalcommunication .................. 76 3.7.4ExpressivenessoftheVirtualHuman ................. 77 3.7.5Summary ................................. 77 3.8Conclusions ................................... 77 3.9FutureWork ................................... 78 4IPSVIZ:HUMAN-VIRTUALHUMANEXPERIENCESFORSELF-REFLECTIONANDSELF-DIRECTEDCHANGE ......................... 85 4.1Introduction ................................... 86 4.1.1AARofaHuman-VirtualHumanInteraction ............. 86 4.1.2InterpersonalScenarioVisualizer(IPSViz) .............. 87 4.1.3EvaluatingAARforH-VHExperiences ................ 88 4.2InterpersonalScenarioVisualizer ....................... 88 4.2.1ExpandingAARtoH-VHInteractions ................ 88 4.2.2OverviewofIPSViz ........................... 89 4.2.3AHuman-VirtualHumanExperience ................. 89 4.3CapturingHuman-VirtualHumanCommunication ............. 90 4.3.1SystemInputs .............................. 90 4.3.2SystemOutputs ............................. 91 4.3.3SummaryofCapturedData ...................... 91 4.4FilteringandProcessing ............................ 92 4.5Spatial,Temporal,andSocialVisualization .................. 94 4.5.1SpatialVisualization .......................... 94 4.5.2TemporalVisualization ......................... 95 4.5.3SocialVisualization ........................... 96 4.5.3.1Verbalcommunication .................... 96 4.5.3.2Nonverbalcommunication .................. 96 4.6StudyDesign .................................. 97 7

PAGE 8

............................... 97 4.6.2StudyProcedure ............................. 98 4.6.3Measures ................................. 99 4.6.3.1Friendly,natural,andemotionalexpression ......... 100 4.6.3.2Stateofthevirtualhuman .................. 100 4.6.3.3Medicalinterviewcommunication .............. 100 4.7StudyResultsandDiscussion ......................... 101 4.7.1After-ActionReviewImpactsSelf-Perceptions ............ 101 4.7.2ChangingCommunicationwithRealHumans ............. 102 4.7.3AAR,VHSkinTone,andDisplaytype ................ 103 4.7.4ScopeofPerceptions .......................... 104 4.7.4.1Verbalcommunication .................... 105 4.7.4.2Nonverbalcommunication .................. 107 4.7.4.3Establishingarelationship .................. 109 4.7.4.4Reactionsunderstress .................... 111 4.7.5Summary ................................. 111 4.8ImpactofInteractionRealismonAAREectiveness ............. 112 4.8.1Realismvs.AAREectiveness:ParticipantComments ....... 113 4.8.2Realismvs.AAREectiveness:StatisticalAnalysis ......... 115 4.8.2.1OperationalizationofVHinteractionrealismandAAReectiveness .......................... 116 4.8.2.2H1:VHinteractionrealismispositivelycorrelatedwithAAReectiveness ....................... 117 4.8.2.3H2:VHinteractionrealismisafactorindeterminingAAReectiveness .......................... 119 4.9ConclusionsandFutureWork ......................... 121 5VSP:VIRTUALEXPERIENCESFORSOCIALPERSPECTIVE-TAKING .. 128 5.1Introduction ................................... 128 5.1.1VirtualSocialPerspective-Taking ................... 129 5.1.2DrivingApplication:MedicalInterviewTraining ........... 129 5.1.3Evaluation ................................ 130 5.2PreviousWork ................................. 131 5.2.1SocialVEsCanBenetfromVSP ................... 131 5.2.2AvatarsAectHumanBehavior .................... 131 5.3VSPForMedicalInterviewTraining ..................... 132 5.3.1SimulatingaDoctor-PatientBreastInteraction ............ 132 5.3.2RecordinganInteractionforVSP ................... 133 5.3.3Sensing .................................. 134 5.3.3.1Seeingandhearing ...................... 134 5.3.3.2Touchandproprioceptivefeel ................ 135 5.3.4RemindersofVirtualIdentity ..................... 136 5.3.4.1Greenscreenvirtualmirror ................. 137 5.3.5ReenactingBehavior .......................... 138 8

PAGE 9

............................ 138 5.3.5.2Moving ............................ 138 5.3.5.3Relivingaclinicalbreastexam ............... 139 5.4EvaluationofaVSPforMedicalInterviewTraining ............. 139 5.4.1StudyDesign .............................. 140 5.4.2Population ................................ 140 5.4.3Procedure ................................ 140 5.4.3.1Patientinteractions ...................... 141 5.4.3.2VSPexperience ........................ 143 5.4.4Measures ................................. 143 5.4.4.1Empathicbehaviorandperspective-taking ......... 144 5.4.4.2Copresence .......................... 144 5.4.5Results .................................. 145 5.4.5.1VSPselicitreectionandself-directedchange ....... 145 5.4.5.2VSPaordssocialprocessesandinteraction ........ 145 5.4.5.3Observations ......................... 146 5.4.6Discussion ................................ 147 5.4.6.1VSPelicitssocialprocesses ................. 147 5.4.6.2VSPmotivatesself-directedchange ............. 147 5.4.6.3Seeingtheworldthroughunfamiliareyes .......... 148 5.5ConclusionsandFutureWork ......................... 148 6IPSVIZN:AGGREGATEAFTER-ACTIONREVIEWFORVIRTUALHUMANEXPERIENCES ................................... 158 6.1Introduction ................................... 158 6.1.1Motivation:FeedbackandEvaluationofInterpersonalSkills .... 159 6.1.2Motivation:VirtualHumansforInterpersonalSkillTraining .... 160 6.1.3TheInterpersonalScenarioVisualizer ................. 160 6.1.4IPSVizN 161 6.2AnalysisTasks ................................. 161 6.3VirtualHumanPatients ............................ 164 6.3.1Motivation ................................ 164 6.3.2Interaction ................................ 165 6.4PreparingforVisualization ........................... 166 6.4.1Capture ................................. 166 6.4.2FilterandProcess ............................ 167 6.5IdentifyingandComparingTrendsandOutlierswithVisualization ..... 167 6.5.1OverviewsofVerbalBehavior ..................... 168 6.5.2SupportingDiscoveryandComparisonwithInteraction ....... 169 6.5.2.1Filteringtorevealtrendsandoutliers ............ 169 6.5.2.2Rescalingtimeforidenticationandcomparison ...... 169 6.5.2.3Identifyingoutlierswithselectionqueries .......... 170 6.5.2.4Semanticzoomingtoreviewspecicmoments ....... 170 6.6PreliminaryEvaluation ............................. 171 9

PAGE 10

.................................. 171 6.6.2Questionnaire .............................. 172 6.6.3Results .................................. 172 6.6.3.1Trends ............................. 173 6.6.3.2Outliers ............................ 174 6.6.3.3Errorsinthedata ....................... 174 6.6.3.4Suggestedimprovements ................... 176 6.7ConclusionsandFutureWork ......................... 176 7SUMMARYANDFUTUREWORK ........................ 186 7.1LessonsfromIPSViz .............................. 186 7.2LessonsfromtheVSPSystem ......................... 186 7.3LessonsfromIPSVizN 187 7.4FutureWork ................................... 187 REFERENCES ....................................... 189 BIOGRAPHICALSKETCH ................................ 205 10

PAGE 11

Table page 3-1Pearsoncorrelationbetweenobservers. ....................... 80 5-1Empathy(top)andperspective-taking(bottom)questionnaires. ......... 150 11

PAGE 12

Figure page 3-1Realinterpersonalinteraction(left)andequivalentvirtualinterpersonalinteraction(right). ......................................... 81 3-2Systemoverview ................................... 81 3-3StudyprocedureforgroupsSPandVH ....................... 82 3-4VIC(left)andDIANA(right)inthevirtualexamroom ............. 82 3-5Perceivededucationalvalueofrealandvirtualinteractions ............ 83 3-6Perceivedexpressivenessofvirtualhumanandstandardizedpatient ....... 83 3-7Elicitinginformationscoreforbothgroupsandstudies. .............. 83 3-8Expertratingsofempathyondescriptivescales .................. 84 3-9Nonverbalbehavior,attitude,andattentiveness .................. 84 4-1ScreenshotofIPSViz. ................................. 123 4-2Userinteractswithavirtualhuman,thenreviewsinteractionwithIPSViz. ... 124 4-3H-VHinteractioniscaptured,ltered,processed,andvisualizedforreview,evaluationandfeedback. ..................................... 125 4-4ExamplestudentbodyleanthroughoutanH-VHinteraction .......... 125 4-5VisualizationofvetopicsignalsshowingwhenstudentandVHdiscussedimportanttopics. ......................................... 125 4-6User(woodenposingdoll),VHandvirtualenvironmentrenderedin3D. A )ReviewfromperspectiveofVH. B )Interactionaugmentedwithgazeinformation. .... 126 4-7StudyProcedure ................................... 126 4-8InteractionofVHskintoneandafter-actionreviewonshowinginterestintheVH. .......................................... 126 4-9VHinteractionrealismversuschangeinself-ratedrapport-buildingscores .... 127 5-1Medicalstudentconverseswithvirtualpatient(Top),thenrelivesconversationasthepatient(Bottom). ............................... 151 5-2LoggingofpatientinteractionforVSP ....................... 152 5-3DuringVSP,thestudentseestheexamroom,hisavatar(Amanda'sbody),andvideoofhimselftalkingtoAmandafromAmanda'sperspective. ......... 153 12

PAGE 13

........................... 154 5-5Transformingthegreenscreenintoavirtualmirror. ................ 155 5-6VHspeechloggedforlaterdisplayintheVSPexperience. ............ 156 5-7Studyprocedure:borderedimagesindicaterole(patientordoctor)participantplayedateachstage. ................................. 157 5-8Empathyandperspective-takingimprovedafterVSP,indicatingparticipantstriedtoimproveempathyandperspective-takinginthe2ndpatientinteraction. 157 6-1Interactiontimelineshowsuseoftopicsovertimewithrespecttoeachinteraction. 179 6-2Topictimelineshowsuseoftopicsovertimeforallusers. ............. 180 6-3Topichistogramshowsfrequencyoftopicuseforaselectedgroupofusers. ... 181 6-4Interactiontimelinevisualization(Figure 6-1 withaseriesofltersapplied):A)Discussionofhistoryofpresentillness(hpi)only.B)Reviewinghpi,butfurtherlteringtoonlyreviewtheclinicianssubgroup.C)Reviewinghpionlyforthestudentssubgroup. .................................. 182 6-5Useoftopicsonanormalizedtimeline ....................... 183 6-6Queryingtimespansbyselection ........................... 184 6-7Reviewingvideoofpatientchallenges.A)Interactiontimelinelteredbypatientchallenges.B)Uponselectingchallenge,correspondingvideoappearsforreview. 185 13

PAGE 14

Weidentifyverbalandnonverbalcommunicationastheprimarywayinwhichhumanstrytointeractwithvirtualhumaninterfaces,andthenusethisresulttodevelopthreedierentapproachesforanalyzinginteractionsbetweenhumansandvirtualhumans.Eachoftheseapproachesisappliedtoanalyzinginteractionswithvirtualhumansfortrainingdomain-specicinterpersonalskills.Inprovidingnewapproachestoanalyzingvirtualhumansinterfaces,weadvancethestate-of-the-artinfacilitatingandtraininginterpersonalinteractionswithvirtualhumaninterfaces. Verbalandnonverbalcommunicationisidentiedastheprimarywayuserstrytointeractwithvirtualhumansbycomparinginteractionswithvirtualhumanstosimilarinteractionswithrealhumans.Intwouserstudies(n=82),participantselicitedthesameinformationfromavirtualandrealhumanusingverbalcommunication.However,participantnonverbalbehaviorindicatedparticipantswerelessengaged,insincere,anddemonstratedapoorerattitudetowardsthevirtualhuman.Thesebehavioraldierenceslikelystemmedfromtheparticipants'dicultyunderstandingthevirtualhumanslimitedexpressivebehavior. TheInterpersonalScenarioVisualizer(IPSViz)wasthendevelopedtoenablereview,analysis,andevaluationofthecommunicationbetweenahumanandavirtualhuman.IPSVizgeneratesvisualizationsofahuman-virtualhumaninteractionbycapturing,logging,andprocessingthehumanandvirtualhuman'sverbalandnonverbalbehavior. 14

PAGE 15

Thenextsystem,theVirtualSocialPerspective-taking(VSP)system,enablesreview,analysis,andevaluationofaninteractionwithavirtualhumanfromtheperspectiveofthevirtualhuman.TheVSPsystemrecordsavirtualhumanpatientsexperienceoftalkingtoamedicalstudent,andthenusestherecordingtotransportthemedicalstudentintothepatientsbodyandrelivetheconversationthroughhereyes.Thestudentrelivestheconversationtobetterunderstandthevirtualhumanpatientsperspectiveandlearntoaddressherandfuturerealpatients-fears.Theresultsofapilotstudy(n=16)indicatethatVSPencouragesreectionontheperspectivesofothersandelicitsself-directedchangeofbehaviorinfuturesocialinteractions. Thelastsystem,IPSVizN,enablesreview,analysis,andevaluationoftrendsandoutliersinhuman-virtualhumaninteractions.IPSVizNprocessesgroupsofhuman-virtualhumaninteractionlogstogeneratesummaryvisualizationsoftheinteractions.AnevaluationofIPSVizNwithrepresentativeend-usersfoundthatparticipantswereabletorapidly(withinminutes)identifytrendsandoutliersinoverallgroupinterpersonalskills,includingverbalbehavior,organization,completeness,empathy,andcommunicatingunderstress.IdentifyingthesetrendsandoutlierswithoutIPSVizNwouldhaverequiredhoursofmanualeort. 15

PAGE 16

Animmersivevirtualhuman,orembodiedconversationalagent[ 33 ],isacomputationalprocessthatexhibitsahuman-likeuserinterface.Human-likeinterfaceshavetheapproximatesizeandappearanceofarealhuman.Theyaordhuman-computerinteractionsimilarto,butnotnecessarilythesameas,human-humaninteraction. Thecapacityofvirtualhumanstoaordinteractionsimilartohuman-humaninteractionhasdriventhedevelopmentofinterpersonalsimulators[ 86 ].Interpersonalsimulatorssimulateinterpersonalscenarios(interactionswithpeople)byreplacingrealhumanswithvirtualhumans.Usersinteractwiththevirtualhumanstotrainforsimilarinteractionswithrealhumans.Interpersonalsimulatorshavebeendevelopedtotrainavarietyofdomain-specicinterpersonalskills,includingmilitaryleadership[ 79 ],medicalpractice[ 87 89 115 ],psychologypractice[ 155 ],culturalcompetency[ 10 40 ],lawenforcement[ 59 ],andautismeducation[ 178 ]. Thisworkproposes,implements,andevaluatesthreedierentapproachestoanalyzingvirtualhumaninterfacesforinterpersonalskillstraining.Eachapproachisbuiltonthepremisethat,asinhuman-humaninteractions,theprimarymethodsofinteractionbetweenusersandvirtualhumansareverbalandnonverbalbehavior.Thus,toanalyzevirtualhumaninterfaces,onemustanalyzetheverbalandnonverbalbehaviorofusersandvirtualhumans. Threeproof-of-conceptsystems,IPSViz,VSP,andIPSVizN,weredevelopedtoanalyzeuserandvirtualhumanbehaviors.Eachsystemdemonstratesoneofthreeanalysisapproachesproposedinthiswork. 1. 2. 16

PAGE 17

Theseapproachesprovideinsightsintouserinteractionwithvirtualhumansandallowusersofinterpersonalsimulatorstogetrapidfeedbackondomain-specicinterpersonalskillsthatwouldbedicultorimpossibletoreceivethroughhuman-humaninteractions.Thus,indevelopingandevaluatingapproachesforanalyzinginteractionswithvirtualhumans,thisdissertationprovidesinnovationstoboththestudyofvirtualhumanuserinterfacesandtheirapplicationininterpersonalsimulators. 1.1.1TrainingInterpersonalSkillswithVirtualHumans 17

PAGE 18

17 18 38 41 66 72 ].Theirsuccessinrealandothervirtualexperiencesmotivatestheiruseforhuman-virtualhumanexperiences. Furthermore,after-actionreviewsforhuman-virtualhumaninteractionshavethepotentialtoprovidenovelfeedbackthatwouldbedicultorimpossibletoprovidethroughhuman-humaninteractions.Unlikehuman-humaninteractions,thecomputationalnatureofhuman-virtualhumaninteractionsmeanstheycanbeeasilyloggedtoprovideadetaileddescriptionoftheinteraction.Thedescriptionoftheinteractioncanthenbeprocessedtoprovidenovelfeedbacktotheuser. Thisdissertationleveragesinformationvisualizationandimmersivevirtualrealitytechniquestoenablerapid,focused,analysisofhuman-virtualhumaninteractionlogs.Theseanalysisapproachesaddressthechallengesinanalyzingreal-worldroleplayexperiences,aswellprovidenewbenetsuniquetointerpersonalsimulations: 18

PAGE 19

19

PAGE 20

Inapplyingnewanalysisapproachestoareal-worldapplicationofvirtualhumans,thisdissertationadvancesthestate-of-the-artinmedicalinterviewtraining,interpersonalskillstraining,andvirtualhumaninterfaces. 1.3.1Human-VirtualHumanInteractionHuman-HumanInteraction? 3 ).Thegoalofthestudyistoidentifytheextenttowhichinteractionswithvirtualhumansaresimilartointeractionswithrealhumans.Establishingthissimilaritysupportsthethesisstatementintwoways. First,establishingthissimilaritywouldmeanthatdomain-specicinterpersonalskillsexhibitedinhuman-humaninteractionsareexhibitedinhuman-virtualhumaninteractions.Thisiscrucialtosupportingthethesisstatementbecauseitisunlikelyreviewinghuman-virtualhumaninteractionswould\elicitreectionson,changeperceptionsof,andmotivatechangeindomain-specicinterpersonalskills"ifdomain-specicinterpersonalskillsarenotexhibitedinhuman-virtualhumaninteractions. Second,thecomparisonwouldidentifythedomain-specicinterpersonalskillsthatareimportantinthemedicaleducationscenariosusedthroughoutthisdissertation.Once 20

PAGE 21

4 ).Thepipelineisdevelopedbasedontheprinciplethatverbalandnonverbalbehaviorsaresignals,time-varyingcharacteristicsofahuman-virtualhumaninteraction.Treatingtheinteractionasasetofsignalsallowsthepipelinetoprocessandltertheinteractiontocomputetheinformationneededforvisualizationandimmersivereviewfromthevirtualhuman'srst-personperspective. 4 ),generatesoverviewanddetailvisualizationsoftheverbalandnonverbalbehaviorinahuman-virtualhumaninteraction.Withaglance,theoverviewvisualizationsprovidearapidsenseofhowdomain-specicinterpersonalskillsareusedinaninteraction.Specicmedicaldomainskillsthatarepresentedwithoverviewvisualizationsincludeusergazebehavior,theorganizationandthoroughnessoftheuser'squestions,andtheuser'sinterpersonaldistancetothevirtualhuman. 21

PAGE 22

ToevaluateIPSViz,healthprofessionalsinteractwithavirtualhumanpatientandthenconductanafter-actionreview(AAR)oftheirinteractionusingIPSViz.Pre-andpost-AARsurveysareadministeredtoparticipantstomeasurethechangeintheirperceptionsoftheirinterpersonalskillsduetousingIPSViz.Inaddition,participantsareinterviewedtocollectsubjectivefeedbackabouttheAAR.Ifthereisasignicantchangeinperceptionsofone'sdomain-specicinterpersonalskills,and/orparticipantsdiscusslessonslearnedfromthereviewsession,thenitisclearparticipantsself-reectedon,changedperceptionsof,andconsideredchangesindomain-specicinterpersonalskills. 6 ).However,IPSVizNaggregatesinteractionsignalsfrommanyhuman-virtualhumaninteractions,ratherthanjustone,topresentoverviewsanddetailsofgroupinterpersonalskills.Thesevisualizationsenableuserstorapidlyidentifytrendsandoutliersinthedomain-specicinterpersonalskillsofgroups. ApilotstudyevaluatesthebenetsofIPSVizNforagroupofhealthprofessionseducators.Thehealthprofessionseducatorsarerepresentativeofaclassofuserswhodonotinteractdirectlywiththevirtualhuman,butinsteaddirectstudentstoconducthuman-virtualhumaninteractions.Theseusersneedtoreectonthehuman-virtualhumaninteractionstheydesignanddirect,identifytrendsandoutliersinthecommunicationskillsoftheirstudents(perceptions),andmakechangesintheirstudent'sfutureinteractionswithvirtualandrealhumans.Inthestudy,multipleversionsofIPSVizNaredistributedtoparticipants,withfeedbackcollectedfromparticipantsaftereachversion.Eachversionincorporatesthepreviousversion'sfeedbacktoimprovethe 22

PAGE 23

5 ).TheVSPsystemallowsauserwearingahead-mounteddisplaytoseewhatavirtualhumansaw,aswellashearwhatthevirtualhumanheardbyplayingbackaudiooffromthelogs.Inaddition,theVSPsystempromptstheusertoactoutthevirtualhuman'sbehaviors(speechandmovement).Thisimmersioninthevirtualhuman'srst-personexperienceisaimedathelpingtheuserbetterunderstandthevirtualhuman'sperspective. TheVSPsystemisevaluatedbymeasuringhowvirtualsocialperspective-takingchangesauser'sperceptionsof1)avirtualhumanpatientinteractiontheuserconductedand2)afuturevirtualhumanpatientinteractiontheuserconductsaftertheVSPinteraction.First,theuserinteractswithavirtualhumanpatient,Then,theuserreviewsthatinteractionusingtheVSPsystem.Finally,theuserconductsanothervirtualhumanpatientinteraction.AsurveyisgivenaftereachpatientinteractionandtheVSPinteraction.Thisstudydesignallowsdeterminingifimmersioninavirtualhuman'srst-personperspectiveelicitsself-reectionon,changesself-perceptionsof,andmotivateschangeindomain-specicinterpersonalskills. 23

PAGE 24

Thereal-worldcounterpartwasamedicalinterview,aninterpersonalinteractionwheredoctorsmustcommunicateeectivelywithpatientstodiagnoseandtreatthem.Interviewsbetweenmedicalstudentsandanactorplayingapatientwerecomparedtointerviewsbetweenmedicalstudentsandasimilarvirtualhumanpatient.Thecomparisonindicatedthatthesimulatedinterpersonalinteractionsapproximatedreal-worldinterpersonalinteractions.Thisapproximateequivalencebolsterstheargumentthatinterpersonalsimulationscanbeusedforteaching,training,andevaluatingreal-worldinterpersonalskills. 24

PAGE 25

WhileparticipantsalsousednonverbalbehaviorwiththeVH,thatbehaviorcommunicateddierentmessagesthannonverbalbehaviorwiththerealhuman.Withthevirtualhuman,usersnoddedless,hadamoredistantposture,andusedamoremonotonevoice.Thesenonverbalsignalscommunicatedapoorattitude,lowinterest,andinsinceritytowardsthevirtualhuman. Theextensiveuseofverbalandnonverbalcommunicationwithvirtualhumans,anditssimilaritytothatwithrealhumans,impliesthatanalyzinghuman-virtualhumancommunicationprovidesinsightintohuman-humancommunication.Thisfurthermotivatesthecomputationalanalysisofhuman-virtualhumancommunication.Whilestandard,video-basedanalysisofhuman-humancommunicationispossible,sensor-driven,naturalvirtualhumaninterfacesaordamoresophisticatedapproach. 25

PAGE 26

IPSVizandVSPalloweduserswhointeractedwithavirtualhumantogaininsightintotheirowncommunicationskills.WithIPSViz,usersself-reectedontheircommunicationwithavirtualhumanandidentiedwaystheymightchangebehaviorwithrealhumans.Specicchangesusersproposedincludeimprovinggazebehavior,conductingamorefocusedverbalinteraction,andbeingmoreempathetictothevirtualhuman.AfterusingtheVSPsystem,usersindicatedtheybetterunderstandthevirtualhuman's 26

PAGE 27

IPSVizNalloweduserswhomonitorhuman-virtualhumaninteractionstoidentifytrendsandoutliersinthecommunicationskillsofgroups.Trendsdiscoveredamongagroupofmedicalprofessionalswhointeractedwithavirtualhumanpatientincludemoreexibilityamongmoreskilledmedicalprofessionalthanlessskilledcounterpartsandatendencyforskilledphysicianstodiscussfamilyhistoryless.Outliersinthesamegrouprepresentedmedicalprofessionalswhonevershowedempathy,weretoobrief,and/ordidnotdiscussimportanttopicswiththevirtualhuman.Fromthesetrendsandoutliers,after-actionreviewparticipantssuggestedremedialtrainingfortheoutliersandconsideredhowtousethetooltodeepentheirunderstandingofwhatcommunicationskillsleadtosuccessfulinteractionwithpatients. 87 ],butcanalsoencourageself-reectionontheseinteractions.Thisself-reectioncanleadtoself-directedchangeininterpersonalinteractionswithrealhumans.Byinteractingwithavirtualhuman,andconductinganafter-actionreviewofthatexperience,interpersonalinteractionswithrealhumanscanbeimproved. 27

PAGE 28

Thisdissertationisinuencedby,andprovidesinnovationsbackto,fourareasofresearch:virtualhumans(Section 2.1 ),after-actionreviews(Section 2.3 ),temporalvisualizations(Section 2.4 ),andvisualizationofinterpersonalcommunication(Section 2.5 ). 33 ],arecomputationalprocessesthatexhibitahuman-likeinterface.Ahuman-likeinterfacemeansthatthevirtualhumanapproximatestheappearance,perception,andbehaviorofarealhuman.Theextenttowhichvirtualhumansmusthavethesecharacteristics,aswellasthebestwaystosimulatethesecharacteristics,remainopenresearchproblems[ 189 ].Thisdissertationfocusesonadierentproblem:buildingeectivevirtualhumanexperiencesforinterpersonalskillstraining. 13 ]suggestvirtualhumans\shouldmoveorrespondlikeahuman"and\mustexist,work,actandreactwithina3Dvirtualenvironment."AlessiandHuang[ 6 ]expandtheserulesforpsychologyapplications.Theysuggestvirtualhumansshouldbesocial,emotionallyexpressive,andinteractive.Virtualhumansshould\captureandassessa 28

PAGE 29

ThorissonandCassell[ 181 ]agreethathuman-likeemotionalexpressionisimportant,butnon-verbalbehaviorsthatsupportconversation,e.g.,gesturingatobjectsandlookingattheusertoindicateattention,aremoresignicant.Casselletal.[ 33 ]focusonmodelingconversationalinteractiontocreatebelievable,functionalvirtualhumans.Blascovich[ 25 ]theorizesvirtualhumanswithhigherbehavioralrealismaremorepersuasive.Vinayagamoorthyetal.[ 190 ]suggestavirtualhuman'sexpressionsandbehaviorshouldbeappropriatefortheapplication'scontext. Thereisalsosignicantevidencethatmakingvirtualhumansmorehuman-likedoesnotnecessarilymakethemmoreeectiveoracceptabletousers.ThisnotionwasrstproposedforrobotsbyMori[ 126 ]astheUncannyValleytheory.TheUncannyValleytheoryarguesthatwhenavirtualhumanreachesacertainthresholdofhuman-realism,reactionstothevirtualhumanbecomenegative.Vinayagamoorthyetal.[ 188 ]foundstudyparticipantsexposedtodierenttypesofvirtualhumansweremoreforgivingofawsinlessrealistic,cartoon-likecharacters.Steptoeetal.[ 172 ]foundasimilartradeobetweenimprovingthedelityofavirtualhuman'seyegazebehaviorandtheabilityofuserstoaccuratelyjudgeavirtualhuman'scommunication. Otherworkindicatesvirtualhumansshouldhavehuman-likecharacteristicswithconsistentlevelsofdelity.Garauetal.[ 65 ]foundthatthequalityofcommunicationbetweenavirtualhumanandrealhumanwasratedhigherwhenthevirtualhuman'sappearanceandbehaviorwereatsimilarlevelsofrealism.Similarly,Bailensonetal.[ 16 ]foundthatcopresence(thesenseofbeingwithanotherpersoninavirtualenvironment)withavirtualhumanwaslowestwhenavirtualhuman'sbehavioraland 29

PAGE 30

190 ]tosimilarconclusions. 79 ].TheJustVRsystem[ 115 ]allowsamedicaltraineetoworkwithavirtualassistanttotreatavirtualvictim.Balcisoyetal.[ 19 ]createdasystemwhereusersplaychessagainstavirtualhumanaugmentedtotherealworld.TheVirtualClassroom[ 155 ]usesvirtualteachersandstudentstoassessattentionandsocialanxietydisorders.Babuetal.[ 10 ]usevirtualhumanstoteachSouthIndiangreetingetiquette,andDeatonetal.[ 40 ]usesvirtualhumanstoteachculturalcompetencytosoldiersworkinginforeignlands.Franketal.[ 59 ]usevirtualhumanstoteachlawenforcementpersonnelhowtointeractwiththementallyunstable.TheVirtualExperiencesResearchGroupattheUniversityofFloridausesvirtualhumanstotrainmedicalstudentsoncommunicatingwithpatients[ 87 89 ].Ryokaietal.[ 159 ]usesvirtualhumansasstorytellingpartnersforyoungchildren.Thorisson's[ 180 ]interactiveguide,Gandalf,givestoursofthesolarsystem.TheHumanModelingandSimulationGroupattheUniversityofPennsylvaniausesvirtualhumansfortaskanalysisandassemblyvalidation[ 12 ].Tartaroetal.[ 178 ]areusinginteractionswithvirtualhumanstohelppeoplewithautismspectrumdisordersdevelopinterpersonalskills.Thecommonthemeinalmostalltheseapplicationsistheuseofvirtualhumansassurrogatesforrealhumans. 30

PAGE 31

14 15 ]haveshownthatpeoplemanagepersonalspacewithvirtualhumansinthesamemannerastheydowithrealhumans.Studyparticipantskeptmoredistancefromahigh-statusvirtualhumanthanfromanunknownvirtualhumanandmoredistancefromavirtualhumanthaninanimatevirtualobjects.Femaleparticipantskeptmoredistancefromvirtualhumansthatmaintainedeyecontactthanwithvirtualhumansthatdidnot. Zanbakaetal.[ 201 ]haveshownvirtualentities(humanoranimal-like)canbeaseectiveasrealpeopleatpersuasion.Ininteractionswithbothrealandvirtualspeakers,persuasionwasstrongerwhenparticipantslistenedtoaspeakeroftheopposite-sex.Inadierentstudy,Zanbakaetel[ 202 ]foundthat,aswithrealhumans,thepresenceofavirtualhumanlowersperformanceonnovelorcomplextasks. Pertaubetal.[ 139 ]notedparticipantswithafearofpublicspeakingreportedsimilaranxietieswhenspeakingtoanaudienceofvirtualpeople.Garauetal.[ 64 ]showedthatpeoplerepresentedbyavatarscommunicatebetterwhentheavatarsemployrealistic,task-appropriateeye-gaze. Thesecharacteristicshavebeenobservedforotherformsofcomputationalagents.Forexample,Nassetal.[ 130 ]exploredtheaectivepowerofcomputersandintelligentagents.Theirworkhasshownpeopleascribehumancharacteristicstocomputers,suchashelpfulness,usability,andfriendliness. 9 27 55 154 203 ]istheactofself-regulatinglearningby: 1. 2. 31

PAGE 32

Goodmetacognitiveskillsarehypothesizedtobeanindicatorofacademic,professional,andpersonalsuccess.Metacognitionisassociatedwithpositiveeducationaloutcomes,self-condence,andhappiness[ 153 ].Crucialtothemetacognitiveprocessisself-awareness.Apersonmustunderstandherselftoself-monitor,self-reect,andself-evaluateaccuratelyandwithcondence.Increasingself-awarenesscansignicantlyimprovemetacognition[ 39 48 184 ].Thus,toolsusedtofacilitatemetacognitionplaceaheavyemphasisonself-awareness. Themostcommonandmostresearchedtoolusedformetacognitionisvideo.Reviewingvideofacilitatesmetacognitionbyallowinguserstoseeinteractionsfromtheperspectiveofanexternalobserver.Videoreviewhashadpositiveimpactsinmanydomains.Forexample,videoreviewcanimproveself-perceptionsofperformanceonaspeechtherapyexercise[ 122 ],decreaseaggressioninadolescents[ 133 ],improvesocialskillsofchildrenwithautism[ 179 ],medicalinterviewskillsofmedicalstudents[ 136 157 ],andself-understandingforpeoplewhoaresociallyanxious[ 149 ].Inallofthesestudies,thevideoreviewmadepeoplereevaluatetheirself-perceptionsandchangeactionsandstrategiesinfuturesituations. Reectivewritinghasalsobeenusedextensivelyformetacognition.TheLearningAssessmentJournal[ 31 ]isasetofforms,eachgearedtowardsadierentpedagogicalgoal,whichguidesandstructuresstudentwriting.Inastudy,thestructuredwritingforcedstudentstoreectonthemselvesandtheirlearningprocess.Actionlogs[ 129 ]aredailydiariesstudentswritetoself-reectontheday'slessons.Teacherscanalsoreadactionlogstofacilitatetheirownself-reectiononhowtoimproveteachingandlessons.McCrindleetal.[ 120 ]testedself-reectivewritinginarst-yearuniversitybiologyclass.Participantswereeithertaskedwithwritingasinglereportattheendofthesemester(thecontrolgroup)orwritingjournalentriesperiodicallythroughoutthecourse.Thosewhowrote 32

PAGE 33

Lastly,graphinghasbeenusedtofacilitatemetacognition.Graphsaretypicallyusedtopresentobjectiveperformancedataandmilestones.Toimproveself-awareness,peoplecomparetheobjectivegraphstotheirownself-biasedperceptionsoftheirperformance.Theyalsousegraphstotrackperformanceovertimeandseeiftheymeetmilestones.Reviewinggraphshashelpedstudentswithlearningdisabilitiesperformcognitivetasksbetterandbemoreproductive[ 45 ],improvedmotorskills,self-esteem,andself-awarenessoflearning[ 95 ],andhelpedteachersbecomemoreself-awareoftheirteachingmistakes[ 92 ]. ThissectionplacesthisdissertationinthecontextofexistingknowledgeaboutAARs.BoththetheoryandpracticeofAARsarediscussed,withafocusonAARsforvirtualexperiences.AlthoughvirtualexperienceshaveincorporatedAARspreviously,toourknowledge,thisworkrepresentstherstuseofAARsforvirtualhumanexperiences.Virtualhumanexperiencesarecharacterizedbysignicantinterpersonalcommunication,andthusrequiredierentapproachestoAARthanothervirtualandrealexperiences.ThisdissertationidentiesseveralapproachestoAARsofVHexperiences,expandingwhatisknownaboutAARsandidentifyingtheirusefulnessforvirtualhumanexperiences. 17 ].SuccessfulAARsencourage 33

PAGE 34

41 ].AARshavebeensoeectiveinthemilitarythatotherorganizations(e.g.,businesses,not-for-prots,andgovernmentagencies)incorporateAARsintotheiremployeetrainingandevaluationprograms[ 18 38 66 72 ]. Oneaimofthisdissertationistodemonstrateeectiveafteractionreviewsforvirtualhumanexperiences.Hence,thissectionexploresthecharacteristicsthatmakeAARseective.Furthermore,thissectionalsoidentiescorrespondingchallengesthatmustbeovercometoextendvirtualhumanexperienceswithafter-actionreviews.Thecharacteristicsandchallengesidentiedbelowarederivedfromameta-analysisofthefollowingresourcesonafteractionreview[ 17 18 38 41 66 72 ]. 34

PAGE 35

TheseguidelinesforsuccessfulAARshighlightseveralchallengesfordevelopingAARsforvirtualhumanexperiences. Thesechallengesareaddressedinthisdissertationthroughthedesignandevaluationofthreeafteractionreviewsystemsforvirtualhumanexperiences,IPSViz,IPSVizN,andVSP. 35

PAGE 36

83 ],direct-re,troop-to-troopcombatexercises[ 112 ],large-scalebattleandwarsimulation[ 109 119 137 169 ],ightsimulations[ 54 71 152 ],teamcommunication[ 102 ],anddismountedinfantrysimulationsinvirtualenvironments[ 97 ].Inadditiontotrainingexercises,militarieshaveusedAARsystemstoreviewrealmilitaryengagementsthathavealreadyoccurred[ 125 ].EmergencyresponseAARsystemshavebeenbuiltforreviewingchemicalleakandspillcleanupandrescueoperations[ 85 128 ],aswellasforcommunicationandcoordinationamongreghters[ 67 ].Thesesystemssupportreviewoflive,virtual,andmixedrealitytrainingexercises. AARsystemsarealsoseeingincreaseduseinthemedicaldomain.Laparoscopicandothersurgicalsimulatorsoftenincludetoolsforreviewingtrainingexercises[ 63 ].TheRoterInteractionAnalysisSystem(RIAS)[ 157 ]allowsmedicalstudentsandpractitionerstoreviewandratepatient-doctorinteractions.Quarlesetal.[ 142 ]addedAARfunctionalitytoamixedrealitytrainingsystemforanesthesiamachines.ThisdissertationpresentsanAARsystemforpatient-doctorinteractions,wherethepatientisavirtualhuman. Finally,AARsystemshavebeenusedtoreviewuserexperiencesinvirtualexperimentalvirtualenvironmentsandgames.Phloem[ 127 ]allowspsychologyandvirtualenvironmentsresearcherstoreviewscienticexperimentsconductedinvirtualenvironments.Steedetal.[ 171 ]builtanAARsystemtostudybreaksinpresence.(Abreakinpresenceisamomentwhenauserinavirtualenvironmentbecomesawareofbeinginarealenvironment.) 36

PAGE 37

94 ]instrumentvideogamestocollectuserinteractiondata,andthenconductvisualization-assistedAARsofthedatatolearnhowtoimprovegames. Inaddition,reviewsystemshavebeenbuiltintocorevirtualreality(VR)toolsandplatforms,suchastrackingsystems,toolkits,andgameengines[ 58 105 121 131 ].ThesecoreVRtoolsprovidebasicAARfunctionalitytotheapplicationsbuiltonthesetools,aswellasbettertoolsfordiagnostics(i.e.debugging). Manytypesofsensorsareusedforcapture.Themostcommonsensoristhevideocamera,whichprovidesavideorecordingofanevent[ 83 94 157 ].3Dsensorsareusedtotrackthepositionsandorientationsofvehicles,people,andtools(e.g.,tanks,soldiers,andweapons)[ 83 97 170 ].Physiologicalsensors(e.g.,heartrateandgalvanicskinresponsemonitors)captureinformationaboutuserstresslevelsoremotions[ 83 ].Microphones,speechrecognitionsoftware,e-maillogs,andtext-messageloggerscapturecommunicationbetweeneventparticipants[ 102 ]. Forlarge-scaledistributedevents,AARdataiscapturedbyaheterogeneous,distributed,sensornetwork.Signicantinfrastructureisneededtoaggregate,distribute,andsynchronizetheheterogeneous,distributeddataforvisualization,playback,andevaluation[ 56 57 163 ]. Severalapproachestorecordingavirtualexperiences(VE)forlaterreviewhavebeenproposed.InHart'simage-basedapproach[ 74 ],acubeisalignedwiththeVEuser'sviewpoint.Eachfaceofthecuberepresentsthenearplaneofacamerafrustum 37

PAGE 38

69 ]proposetemporallinksthatdenearelationshipbetweenpresentandpasteventsinaVE.Deningthetemporal,spatialandpresentationalaspectsofthisrelationshipenablesaddingpasteventsinaVEintoaVEoccurringinthepresent.Friedmanetal.[ 60 ]proposestandardizingtherecordingofVEsforanalyzingpresence.Guidelinesarederivedbyanalyzingarealpresencedataset.Animportantdistinctionismadebetweentemporaldata(e.g.,systemevents,tracking,videorecordings)andnontemporaldata(e.g.,questionnaires).TheauthorsproposebuildingtoolsthataggregateVEdatatoreplayorsummarizeVEs. AARsystemspresentedinthisdissertationaggregatetemporaldataassociatedwithaVHexperience(e.g.,trackingdata,audio,video,speechrecognitiondata,eventlogs,andVHbehavior)toproducevisualizationsoftheinteraction.ThevisualizationscharacterizethecommunicationbetweenaVHandaperson.CharacterizingthiscommunicationenablesAARofVHexperiencesforcommunicationskillstraining. 56 57 97 109 119 127 137 142 163 169 { 171 ].Thisprovidescontextandhelpsdemonstratehowspatialcharacteristicsledtospecicoutcomes.Timelines(eventsandtrainingvariablesgraphedversustime)showhoweventsunfoldandthecausalrelationshipsbetweenevents(e.g.,howdidaneventattimetcausealatereventattimet+n)[ 102 109 119 127 137 157 169 ].Playbackisaformofvisualizationsthatdisplaysonesampleatatime,butinrapidsuccession(animation).Commonformsofplaybackarevideo,2Danimation,and3Danimation.Bywatchingaplaybackofanevent,AARparticipantslearnhoweventoutcomesunfoldovertime.Asplayingbackanentireinteractionforreviewcanbetime-consuming,AARsoftenindexrecordedevents 38

PAGE 39

173 ]-everydaylifemomentswherethereisanopportunitytoteachalesson-canbehighlightedontranscriptsandtimelines,allowinguserstoreviewthosemomentswhicharemostimportant. MuchofthedataassociatedwithAARsareinherentlytemporal,inthattheydescribehowthecapturedeventevolvesovertime.Thevisualizationoftemporalinformationisanimportantsub-classofvisualizationthatisdiscussedinmoredetailinSection 2.4 157 ].Inaddition,evaluationisincreasinglybeingdonecomputationally.Computationalapproachestoevaluationaimtoautomaticallyidentifyimportantmomentstoreview,determinecausality,andprovidetargeted,pedagogically-focusedfeedbacktoparticipants. TheAutomatedCommunicationsAnalysisSystem(AutoCAS)[ 102 ]processesverbalhuman-humanteamcommunicationtoprovideneededcommunicationskillsfeedback[ 96 ].Speechrecognitionsoftwaretranscribescommunicationbetweenteammates,enablingdisplayofcommunicationtranscripts,categorizationofspeechutterances,andatimelinethatshowswhendierentparticipantsarespeaking.Althoughthespeechrecognitionprocessproducesnoisytranscriptions,thesystemhelpsparticipantsunderstandhowpoorteamcommunicationcanleadtoundesiredoutcomes. MaoandGratch[ 68 117 118 ]havedevelopedasystemtodeterminecausalityfromlogsofanevent,andthenprovidethisdatabacktoeventparticipantsinanAAR.Givenanoutcome,thesystembacktracesthrougheventlogstotheindividualsorconditionsthatledtoanoutcome.IdentifyingcausalityenablesAARsystemstoautomaticallyidentify\teachablemoments"forreview. 39

PAGE 40

152 ]isanintelligenttutoringsystemforahelicoptersimulator.Thetutoranalyzesuserbehaviorandprovidesfeedbackbothduringandafterthesimulation.Afterthesimulation,AIS-IFTsummarizesusertaskperformanceandprovidesreplayfunctionality.TheAARistailoredbothtothesimulation'spedagogicalgoalsandtotheuser'shistorywiththesimulator.Forexample,studentsdonotreviewskillsduringtheAARwhichtheirhistoryindicatesthey'vealreadymastered. TheDARWARSproject[ 114 ]automaticallyanalyzessimulationlogstodetermineifparticipantsmetthepedagogicalgoalsofthesimulation.Pedagogyisdenedwiththreecomponents:missionobjectives,constraints,andmeasuresofsuccess.Thesearedenedatmultiplescales,enablingAARsforindividuals,smallteamsandlargeteams.TrackingmeasuresovertimeallowsAARstoshowtraineeprogressandpredicttraineeperformanceintherealworld. 2 3 7 37 164 ].Aswithmostvisualizations,temporalvisualizationsaimtohelpusersanswerquestionsordiscoverunknownpropertiesoftheunderlyingdatasetthatwouldbetime-consuming,dicultorimpossibletodetermineotherwise.Visualizationhasbeenusedtoanalyzeavarietyoftemporaldatasetsincludinghealth[ 70 140 ]andjuveniledetentionrecords[ 140 ],website[ 82 ]andapplication[ 76 ]logs,serversecuritylogs[ 177 ],photohistories[ 84 ],migrationpaths[ 174 ],hotelvisitationrecords[ 193 ],lmboxocerecords[ 29 ],andmusiclisteninglogs[ 29 ].Temporalvisualizationsareusedinthisdissertationtohelpusersgaininsightintohuman-virtualhumaninteractions. Therestofthissectiondiscussestemporalvisualizationsfromseveralperspectives.First,fourtypesoftemporalvisualiztionsaredescribed:Instant,Interval,Period,and 40

PAGE 41

2 2 164 ].Then,aspecialclassofvisualizationthatcombinestemporaldatawithspatialdataisdiscussed.Lastly,therolesofinteractionandcoordinationinvisualizationaredescribed. 70 76 98 140 182 ].Intervalsareoftenrepresentedgraphicallywithhorizontalbarswhoseleftandrightboundariescorrespondtotheedgesofthetimeintervalt1andt2.Instantsareoftendepictedwithpointsorglyphs,iconsorpictorialimagesthatdepictthemeaningofthedataunderlyingthecorrespondinginstant[ 140 ].Multipletimeseriescanbeplottedonthesametimelinetofacilitatecomparisonandhighlightrelationshipsbetweendatasets. Stackedplots[ 29 75 ]areaspecialtypeoftimelinevisualizationthatuseareatodepictthevalueofatemporalvariable.Stackedplotsaretypicallyusedtodepictrelativequantitiesorpercentages.Variablesaredisplayedontheplotas2Dregions.Theregionsarestackedoneontopoftheother(assumingtimeisthehorizontalaxis),andthesizeofeachregionshrinksandgrowsverticallydependingonthevalueoftheunderlyingvariableatdierentinstantsoftime.Thisallowsforrapidcomparisonoftherelative\strength"orcontributionofasetofvariables. Anotherformofinstantvisualizationisthesnapshot.Snapshotsdisplaythestateofavariableataspecicmomentintimetusingasingleimage.Thedierencebetweenasnapshotandatimelineisthatatimelinewillusuallydisplaymultipleinstantsforavariable,whereasasnapshotonlydepictsthesingleinstantalone[ 49 ].Often,snapshots 41

PAGE 42

1 ]. 21 113 185 193 ].AGregoriancalendarisatablewhereeachrowrepresentsaweek,eachcolumnrepresentstherepetitionsofeachdayoftheweek,andeachcellrepresentsaspecicday.VariationsontheGregoriancalendar(sometimescalledreruns[ 193 ])changetheperiodoftheroworcolumn(e.g.from7to14days),orchangethelevelofdetailby,forexample,zoomingintoaweekanddepictingcellsashours,orzoomingouttoayearanddepictingeachmonthasacell.Whentemporaldataisplottedonacalendar,theverticalandhorizontalalignmentofperiodshighlightcyclicalpatternsinthedata. Anothercommoncyclicalvisualizationisthespiral[ 22 30 194 ].Spiralscanbethoughtofasatimelinebentintoaseriesofconcentriccircles,whereeachcirclerepresentsaperiodofacycle.Justaswithtimelines,dots,glyphs,andbars(instantsandintervals)canbeplotonthecircles.Theradialalignmentofthecircleshighlightscyclicalpatternsinthedata. 3 ].PlanningLines[ 4 ]depictuncertaintiesinintervalsusinghorizontalbarsmarkedwithuncertainstartandendtimes.VizTree[ 106 ]depictsalternativetimelinesasatreestructure. 7 ].Spatio-temporalvisualizationsdisplaydatawithbothspatialand 42

PAGE 43

174 ]),oranimationcanbeusedtodepicthowcharacteristicschangeacrossageographicregionovertime[ 111 ].Otherspatio-temporalvisualizationsdepictspaceastwodimensionsofacubeandtimeasthe3rddimension.Space-timecubes[ 110 ]areanalogoustotakingtheframesinavideoandstackingthemontopofeachothertoformacube. 161 ]isaphilosophyproposedbyShneidermandescribinghowvisualizationtoolsshouldbestructuredandwhatkindsofinteractionfacilitiessuchtoolsshouldinclude.Thisphilosophyextendstotemporalvisualizationtools.First,usersshouldbepresentedwithavisualoverviewofatemporaldataset.Agoodoverviewallowsuserstogetthegistofthedata,oroveralltrendsinthedata,withaglance.Inaddition,agoodoverviewalsohelpsusersidentifyfeaturesinthedatatoexploremoreclosely.Featurescanbeexploredindetailbyaddingpanningandzoomingfacilitiestotheinterface[ 3 98 182 ].Filteringisalsousefulforremovinguninterestingdatafromthedisplay.Forexample,Timeboxes[ 80 ]allowuserstoselectaseriesofoverlappingboxes,andinformationthatonlymeetstheparametersoftheboxaredisplayed.Similarinprincipletoltering,queriescanbespeciedtosearchfororextracttemporalpatternsofinterestfromadataset.Timeboxescanbedrawnoveratemporalpatterntondothersimilarpatternsinadataset[ 28 80 ].Anotherqueryapproachprovidesuserswithtoolstodrawatemporalpatterntosearchforinthedataset[ 53 192 ].Afterzooming,panning,ltering,andquerying,usersshouldbeabletointeractivelydisplaydetailsondemand,specicinformationaboutindividualdatapoints. Onechallengewiththe\Overviewrst,zoomandlter,thendetailsondemand"philosophyisthatuserscanlosetrackofcontextastheydrilldownintothedetails.Toresolvethisissue,visualizationtoolsincorporate\focus+context"visualizations 43

PAGE 44

62 ].Focusvisualizationsshowthedetailsthattheuserisinterestedin.Onthesamescreen,contextvisualizationsdisplaytheoverviewandhighlighttherangeofthefocusvisualizationontheoverview.Onecommonapproachto\focus+context"istheapplicationofspatialdistortionstothevisualization,suchasalogarithmicscaleonalineplotorash-eyelensdistortiononaverylargegraphstructure[ 61 ]. 76 162 ].Thesetoolslinkthevisualizationstogethersuchthatinteractionwithonevisualizationaectstheothers.Forexample,usersmightselectaspecicinstantinatimeline,andthenagraphicaldepictionofthatinstantinanotherwindowishighlighted.Similarly,usersmightselectasetofinterestingpointsonalineplot,andrelatedspatialdatamightbehighlightedonamapinanotherwindow.Thislinkingofmultiplevisualizationsthroughinteractionisknownascoordinatedvisualization[ 132 ].Bycoordinatingdierentvisualizationsofthesamedataset,usersareabletocombineinformationfromeachvisualizationtogaininsightintounderlyingcomplexrelationshipsinthedata. 34 78 124 ].Forexample,Mehrabian[ 123 124 ]hasshownthatpeoplesubconsciouslyuseposture,gaze,anddistancetoexpresslikinganddominance.Someresearchersestimatethatnonverbalbehaviorexpressesasmuchas93%ofourfeelingsandattitudes[ 124 ](theremaining7%isencodedinverbalbehavior).Thus,nonverbalbehaviorprovidesawindowintoaperson'sfeelingsandattitudes. 44

PAGE 45

187 ]plotsthemajorthemesinaseriesofe-mailsagainstthemonthsoftheyear.Morefrequent,distinctivethemesarerenderedlarger.Ataglance,userscandeterminethethemesofe-maildiscussions,andhowdiscussionsdierfrompersontoperson.PeopleGarden[ 198 ]usesagrowingowermetaphortodescribepostingbehavioronamessageboard.Theheightandnumberofpetalsontheowercorrelatetotimeasamemberofthemessageofboardandposts,respectively.AggregatingowerscreatesaPeopleGarden,whichvisualizesamessageboard'soverallactivity.ThreadArcs[ 93 ]depictse-mailthreadsaschronologicallysortednodeswitharcsowingtoreplies.Filterscolornodesandarcstohighlightcharacteristicsofthee-mailthread(Whoisparticipatingintheconversation?Isthethreadabusinessconversationorpersonalconversation?).Netscan[ 168 ]alsovisualizese-mailthreads.Atreedepictsthehierarchicalrelationshipbetweene-mailsandreplies,allowingaquickreviewofwhiche-mailsledtothemost(andleast)discussion.Adirectedgraphdepictswhichthreadparticipants(nodes)replytoeachother(arrows),allowingaquickdeterminationofwhocommunicateswithwho.A3Dbarchartrenderseachthreadparticipantonthevertical,anddaysonthehorizontal.Foreachparticipant-daypair,abarisrendereddepictingthenumberofpoststheusermadethatday.Thishighlightstemporalpatternsinuserinteractionwithathread.Babble[ 52 ]depictsreal-timeparticipationinachatroomusingacirclemetaphor.Circlesrepresentchatrooms 45

PAGE 46

186 ]usescirclestorepresentusersinachatroom.Thecolorofthecircleisusedtorepresentidentity,thebrightnessandsizeofthecircleindicatesactivity(recentposting),andtheproximityofcirclesenablesdiscoveryofcommunitiesofusersdiscussingsimilartopics(muchlikesmallgroupsformatparties).UserscanalsoreviewtheConversationLandscape,ahistoryofeachuser'spostingstodiscoverwhoisdominantinaconversation,whousersinteractwith,andwhetherauser'spostsoccurredneartheuserorfaraway.IPSVizandIPSVizNprovidesimilarvisualizationsofconversationhistory,topicsandactivitylevel. 22 ],areectionofhiddenaspectsoftheirsocialinteractionincluding.TheConversationClock[ 22 ]depictsthehistoryofeveryone'sspeechtohighlight\turntaking,interruption,conversationaldominance,silence,agreement,auralback-channels,mimicry,timespans,rhythmandow."Othersystemsdepictthemajortopicsoftheconversation[ 23 ]andtherole,participation,anddominanceofparticipants[ 11 46 ].Alallahetal.[ 5 ]createvisualizationsofface-to-facemeetingstohighlightthedecision-makingprocess.Atimelinedemonstrateswheneachparticipantsspeaks,whenturnschange,whichdecisioneachparticipantarguesfor(oragainst),andthenalvoteandoutcomeofthedecisionmakingprocess.Previousworkhashighlightedthatreal-timeexposuretoasocialmirrorcanchangeone'sinteractionswithothersinaface-to-face 46

PAGE 47

46 ].IPSViz,IPSVizN,andVSPhaveasimilaraim.Theypresentone'scommunicationwithavirtualhumantoencourageself-reectionandself-directedchangeininterpersonalcommunicationskills. 150 ].Anotherworklooksathowheadgazediersinvirtualenvironmentstodeterminewhataspectsofavirtualenvironmentcreatebreaksinpresence[ 171 ]. Gazevisualizationstakemanyforms.Fixationmaps[ 196 ]aggregatexationpoints-placeswhereusergazedwells-ontoasurface.Higherareasonthesurfacerepresentwhereusersfocusmoreinastimuliimage.Thissurfacecanbemodulatedwiththestimuliimagetocallattentiontoareasoflowandhighgaze(e.g.,bybrighteningareasofhighgaze).Ramlolletal.[ 148 ]createheatmapsfromxationpointstounderstandgazeon3Dobjects.Ovalshavebeenusedtovisualizeclustersofxationpointsandregions-of-interest-placeswheretheusersgaze(butdonotnecessarilydwell)[ 160 ].Timeplotsare2Dstaticgraphsdepictinggazepaths,thepathsbetweenxationpoints.Ifxationpointsarerepresentedascoordinates(x,y)inastimuliimage,atimeplotignoresonedimension(sayx)andplotstheother(y)againsttime[ 143 ].Anotherapproachtoreviewingpathsbetweengazepointsisbyplayingbackasequenceofgazepointsasamovie[ 20 ]. Someprojectsprovideabstractvisualizationsofnonverbalbehaviorandtheirunderlyingaectivemeaning.VirtualWave[ 191 ]mapsuserheadmotionandbreathingbehaviortoaseriesof3dblocksarrangedonaplane.Theblocksexpandandcontract 47

PAGE 48

32 73 116 ],moreabstractvisualizationshavealsobeenused.Sometext-basedsystemsdisplayimagesoffacialexpressionstorepresentanoverallemotionfromaconversationpartner[ 26 107 ].Othersmapaecttodistinctcolors[ 108 ],fontproperties(e.g.,bold,italics,type,color,size)[ 91 ],andtextanimation[ 175 ]. 1. Theinterpersonalscenariovisualizer(IPSViz)generatesvisualizationsoftheverbalandnonverbalcommunicationbetweenahumanandavirtualhuman.UsingIPSViztoconductanAARofone'sinteractionwithaVHencouragesreectiononone'sinterpersonalskillsandself-directedchange. 2. TheVirtualSocial-PerspectiveTakingsystem(VSP)leveragesvirtualexperiencestoenablerelivingone'sinteractionwithaVHthroughtheVH'seyes.VSPallowsuserstoreectontheperspectivesofothers.Lessonslearnedthroughperspective-takinginVSPelicitschangesinbehaviorinfutureinteractionswithrealhumans. 3. IPSVizNpresentsaggregatevisualizationsofinterpersonalcommunicationamonglargegroupsofVHexperiences.ThisenablesuserstoidentifytrendsinoutliersininterpersonalcommunicationamongVHexperiencesandVHusers. Beforediscussingthesesystems,thenextchapterreviewsmyworkidentifyingthesimilaritiesanddierencesbetweeninteractionswithavirtualhumanandinteractionswitharealhuman.Bycomparinginteractionswithavirtualhumantothosewitharealhuman,thisworkidentiestheverbalandnonverbalcharacteristicsofVHexperiencesthatmustbeincorporatedintoanyAARsystemforVHexperiences. 48

PAGE 49

Thischapterdescribestwostudiesaimedatidentifyingthesimilaritiesanddierencesbetweeninterpersonalinteractionswithavirtualhumanandthosewitharealhuman.TherststudywaspublishedintheproceedingsoftheIEEEconferenceonVirtualReality2006[ 145 ]andanextendedversiondiscussingbothstudieswaspublishedinIEEETransactionsonVisualizationandComputerGraphics[ 144 ]. 1. ByestablishingsimilaritiesbetweenVHinteractionsandrealhuman(RH)interactionsondomain-specicinterpersonalskills,thischaptermotivatesexploringafter-actionreviews(AARs)forVHinteractions.AARsareeectivepedagogicaltoolsinRHinteractions.ThesimilaritiesbetweenRHandVHinteractionsimpliesAARmightalsobeaneectivepedagogicaltoolforVHinteractions. 2. IntheprocessofestablishingsimilaritiesanddierencesbetweenRHandVHinteractions,thischapteridentiesspecicdomain-orientedinterpersonalskillsthatareexhibitedintheRHandVHinteractions.Thesearetheskillsthatmustbecaptured,processed,anddisplayedforafter-actionreview. 49

PAGE 50

44 87 88 ]allowsmedicalstudentstopracticeadicultinterpersonalsituation-themedicalinterview-throughinteractionwithavirtualhuman.Justasightsimulatorshelppilotsimproveightskills,interpersonalsimulatorsliketheVirtualPatientsystemhavethepotentialtohelpusersimproveinterpersonalcommunicationskills.Thispaperexploresthepotentialofinterpersonalsimulatorsbycomparinginteractionswithavirtualpatient,avirtualhumanthatsimulatesapatient,tointeractionswithastandardizedpatient,arealhumanthatsimulatesapatient. Standardizedpatientsareusedextensivelyinmedicalschoolsworldwide.Nexttoseeingarealpatient,theyarethemosteectivewaytotrainmedicalstudentsonpatientinteraction.Asthestandardizedpatientinteractionisavalidatedsimulationofarealmedicalinterview,itistheidealgoldstandardtocomparethevirtualpatientinteractionto.Toourknowledge,nootherworkhasbeenpublishedwhereaninterpersonalsimulatorisformallycomparedtoavalidatedreal-worldcounterpart.Thiscomparisoniskeytolearninghowtobuildandevaluateeectiveinterpersonalsimulators. Thispaperdescribestwostudiesthatcomparestandardizedpatientinteractionstovirtualpatientinteractions.Participantsweremedicalstudentswhointerviewedeithera)astandardizedpatienttrainedtosimulatethesymptomsofappendicitis(Figure 3-1 -left)orb)avirtualhumanprogrammedtosimulatethesamesymptoms(Figure 3-1 -right).Theinteractionswerethencomparedonthecontentoftheinterview,thebehaviorofparticipants,andtheauthenticityoftheinteraction. 145 ],foundthatinteractionswiththestandardizedpatientandvirtualhumanweresimilarongatheringcriticalinformationfromthepatientandothercontentmeasures.Subtledierenceswerefoundonbehaviorsrelatedtorapportwiththepatient.Participantsappearedlessengagedandinsincerewiththevirtualhuman.Dierencesonrapport-buildingbehaviorsstemmedfromthevirtualhuman'slimited 50

PAGE 51

BuildingonStudyI,StudyII(n=58)soughtto1)furthercharacterizehowbehaviorchangeswithvirtualhumansusingnewmeasuresand2)strengthenthemainndingsofStudyI.StudyIIdieredfromStudyIinthefollowingways: StudyIIstrengthenedStudyI'sresultsandalsoprovidednewinsightintorapport-buildingbehavior.Participants'nonverbalbehaviorcommunicatedlowerinterestintheinteractionandapoorerattitudetowardthevirtualhuman.Somenewbehavioralmeasuresweretoosubjectivetoyieldusefulinformation.Thishighlightedtheneedformoreobjective,physically-basedmeasuresofhumanbehaviorinfuturestudies.Overall,thestudiesprovidekeyinsightsintotheconstructionandevaluationofeectiveinterpersonalsimulators: 51

PAGE 52

51 ]comparedthereactionsofacrophobes(personswithafearofheights)invirtualandrealenvironments.Theauthorsfoundthatexposuretherapyinthevirtualenvironmentwasaseectiveastherapyintherealone.Rothbaumetal.[ 158 ]foundsimilarresultsfortreatingthefearofying.Experiencingavirtualairplanewasjustaseectiveasexperiencingarealoneinreducingyinganxiety. Othershavelookedathumanperceptionofrealandvirtualstimuli.Billger[ 24 ]examinedtheperceptionofcolorinvirtualandrealenvironments.Wuilleminetal.[ 197 ]lookedattheperceptionofvirtualandrealspherespresentedvisuallyandwithhaptics.Virtualspherespresentedvisuallywereperceivedaslargerthanrealspheresofthesamesize. Heldaletal.[ 77 ]studiedcollaborationinrealandvirtualenvironments.ParticipantscollaboratedonbuildingaRubik'scuberealorsharedvirtualenvironments.Performanceinsymmetricimmersiveenvironments(e.g.,bothparticipantscollaboratingthroughanimmersiveprojectionsystem)approachedperformanceinrealenvironments.Performanceinasymmetricenvironments(e.g.,HMDvs.immersiveprojection)waspoorer. Slateretal.[ 166 ]lookedatthebehaviorofsmallgroupsinrealandvirtualenvironments.Participantsviewedimmersedpeersasleadersinthevirtualscenariobutnotintherealone.Groupaccordwashigherintherealenvironment. Usohetal.[ 183 ]examinedparticipantresponsesonpresencequestionnairesafterexperiencingarealenvironmentorasimilarvirtualenvironment.Participantsindicatedtheyfeltjustaspresentinthevirtualenvironmentasintherealenvironment.Thissurprisingresultshowsthatsubjectivequestionnairesshouldnotbeusedtocomparedierentenvironments. 52

PAGE 53

87 ](Figure 3-2 )allowspracticeofmedicalinterviewskills.Studentscangatherthekeyfactsofapatient'sconditionandarriveatadierentialdiagnosis,alistofconditionsthepatientmayhave.Theyalsocanpracticecommunicatingclearlywiththepatientandaddressingtheirfears.Thesecommunicationskillsdonotjustimprovewithclinicalexperience.Theyshouldbetaughtandpracticed[ 103 ]. Aspartofthestudies,thesystemwasinstalledinarealmedicalexamroom.Avirtualexamroomwasprojectedonawall.DIANA,avirtualhumanwithseverestomachpain,wasinthevirtualroom.DIANA'sappearanceandresponsesaremodeledafterarealstandardizedpatient,Maria,trainedtoexhibitseverestomachpain.ModelingDIANAafterMariaallowedparticipantstointeractwithsimilarpatientsintherealandvirtualexperiences.Anothervirtualhuman,VIC,servedasaninstructorthattutorsstudentsonhowtointeractwithDIANA.Commercialspeechrecognitionsoftwareandasimplealgorithmforparsingutterances[ 44 ]enabledtalkingtoVICandDIANAnaturallywithinthescopeofthescenario.Thestudent'shandwastracked,allowingDIANA'spaintobelocalizedwithpointinggestures.Thestudent'sheadwasalsotrackedtorenderthescenefromherperspectiveandallowthevirtualhumantomaintaineyecontact. 53

PAGE 54

3.4.1.1Elicitingcriticalinformation 1. Whendidthepainstart? 2. Whereisthepainlocated? 3. Whatdoesthepainfeellike? 4. Isthepatientnauseous? 5. Hasthepatientvomited? 6. Doesthepatienthaveanappetite? 7. Hasthepatienthadanyunusualbowelmovements? 8. Isthepatientsexuallyactive? 9. Whenwasthepatient'slastperiod? 10. Hasthepatienthadanyunusualvaginaldischarge? 11. Doesthepatienthaveafever? 12. Hasthepatienthadanyunusualurinarysymptoms? Studentsthatelicitedsevenofthetwelveitemsreceivedapassinggrade.Eachgroupwasgradedbytwoparties: 134 ]observedspontaneousdisuencies(falsestarts,hesitations,etc.)occurlessinmachine-humaninteractionthaninhuman-humaninteraction.Therefore,interactionswereassessedontheconversationow.Conversationowwasgradedbycountingthenumberofconrmatorywords,like\ok"and\mmhmm,"usedintheinterview.Suchphrasesare 54

PAGE 55

Theinteractionswerealsoanalyzedforempatheticbehavior.Empathizingwiththepatientisakeycomponentofbuildingrapport.Empathyletsthepatientknowthedoctorunderstandshersituation[ 35 ].Empatheticbehaviorisalsoanindicatoroftheparticipant'semotionalinvolvementintheinteraction.Participants'empatheticactions(e.g.,saying\Iknowithurts,"acknowledgingthepatient'sfears,etc.)werecounted. 195 ]isavalidatedsurveyusedtoevaluatestandardizedpatients.Togatherperceptionsofthevirtualandrealinteractions,participantslledoutamodiedMaSPfocusingonauthenticityandbehavior.QuestionsontheMaSPincludewhetherthepatientischallenging/testing,thepatientmaintainsappropriateeyecontactandthesimulatedpatientcouldbearealpatient. GroupSP(n=8):Eight2nd-yearmedicalstudents(fourmaleandfourfemale)fromtheUniversityofFloridainterviewedthestandardizedpatient(SP).Onaverage,thisgroupinterviewedsixteenSPspriortothisstudy. 3-3 summarizestheStudyIprocedure. 55

PAGE 56

56

PAGE 57

3-4 showsthevirtualscenepresentedbythesystem.Thevirtualinstructor,VIC,stoodinthebackgroundandthevirtualpatient,DIANA,layontheexaminationbedintheforeground. Participantswereinstructedtosay\hello"toVICtobegintheinteraction.VICrespondedbyguidingparticipantsthroughashorttutorialoninteractingwiththesystem.Afterthetutorial,VICtoldtheparticipantshehadtenminutestocompletetheinterview.VICthenlefttheroomsothattheparticipantandDIANAcouldhaveprivacy.ThetenminutetimerbeganassoonasVICleft.Atthe8-minutemark,VICinformedtheparticipantthattwominutesremainedoverthesystemspeaker(withoutreenteringtheroom).Aftertenminutes,VICreturnedtotheroomandendedtheinteraction.Hethenaskedtheparticipantforadiagnosis.Aftertheparticipantstatedtheirdiagnosis,VICthankedtheparticipantandaskedhertoleavetheroom.AsinGroupSP,theamountoftimespentinterviewingthevirtualhumanvariedfromparticipanttoparticipant.Participantswhonishedearlywereallowedtoleavetheroomandmoveontothepost-experiencesurveys. OnemightbeconcernedthatthepresenceofavirtualinstructorcouldleadGroupVHtobelievetheywerebeingobserved.Actually,itisstandardpracticeforinstructorstoobserveinteractionswithstandardizedpatientsviaclosed-circuitcamera.Thispracticewasfollowedforbothgroups,andallparticipantswereawarethattheywerebeingobserved. 57

PAGE 58

GroupSP:ParticipantsrelatedtheirperceptionsofthestandardizedpatientbyllingouttheMaSPsurvey(Section 3.4.1 ). 135 ],andweplantousetheminfuturework. Throughoutthissection,statisticsarepresentedoftheformMSwhereMisameanandSisastandarddeviation.Unlessotherwisenoted,MSPrepresentsafractionoftheparticipantsinGroupSP,andMVPhasthesamemeaningforGroupVP. 58

PAGE 59

Toassessthevariabilitybetweentheexpert'sobservations,thetotalscoreonthecriticalinformationmetricwascorrelatedpair-wiseacrossobservers.ThelowestPearsoncorrelationbetweentheexpertobserverswasr=0:86.Allcorrelationsweresignicantatp=0:006orlower.Thissignicant,largepositivecorrelationindicatestherewaslittleinter-observervariationonthecriticalinformationmetric.Observerscoreswerecombinedbyaveragingduetothelowinter-observervariation.Meanvalues(MSPandMVP)representthefractionofparticipantsthatelicitedacriticalpieceofinformation. Accordingtotheexpertobservers,participantsaskedthevirtualhumanandstandardizedpatientthesamecriticalquestions.Asthevirtualhuman'sresponseswerebasedonthestandardizedpatient'sresponses,theanswerstothecriticalquestionswerealsosimilarinbothgroups.Nodierencewasfoundforbotheasilydiscussedinformation(\Thepainissharpandstabbing",MSP=10,MVH=0:80:35,p=0:12)andsensitiveinformation(\Iamsexuallyactive",MSP=0:540:5,MVH=0:450:52,p=0:72).Finalscoresonelicitingthe12criticalitems(MSP=6:31:7,MVH=5:52:1,p=0:37)and 59

PAGE 60

Innormalstandardizedpatientinteractions,gradesareusuallygivenbythestandardizedpatientinsteadofexpertobservers.Thereforestudentswerealsogradedbythepatient(virtualorreal)theyspoketo.Accordingtothepatientgrades,thelocation/progressionofthepain(MSP=0:250:46,MVH=10,p=1E6),thefactthatthepatientisnauseated(MSP=0:880:35,MVH=0:250:44,p=0:0023)andthefactthatthepatientissexuallyactive(MSP=0:880:35,MVH=0:440:51,p=0:042)werenotaskedwiththesamefrequencyinbothinteractions. Althoughdierenceswerefoundonthepatientgrades,wedefertotheexpertgradesbecausetheyaremorereliableandconsistent.Standardizedpatientgradingisnotalwaysreliablebecausestandardizedpatientsarehuman.Whetherconsciouslyorsubconsciously,theytakeothersubjectivefactorsintoaccountwhengrading.Also,standardizedpatientsgradeduringshortbreaksinbetweeninteractionswithmedicalstudents.Theydonothavemuchtimetoconsidergradesbecauseanotherstudentiswaitingoutsideforthenextinteraction.Themedicalexperts,ontheotherhand,watchedtheinteractionsonvideoafterwards.Theyhadampletimetoreviewthevideoandmakesuretheirgradeswereaccurate.Therewasalsoahighdegreeofconsistencybetweenexpertgrades,lendingmorestrengthtotheirobservations.TheSP'sgrades,however,werenotcorrelatedtoanyoftheexperts(atbest,r=0:041,p=0:923).Thehigherreliabilityandconsistencyoftheexpertgradesindicatesthattherealandvirtualinteractionsweresimilaronelicitingcriticalinformation. Itshouldbenotedthatthevirtualhumanistheonlytrueobjectivegrader.Thisisbecausethevirtualhumangradedparticipantsbyloggingwhateverinformationsherevealedtothem.Thevirtualhumancannottakeintoaccountotherfactorswhengrading.Incontrasttothestandardizedpatient'sgrades,thevirtualhuman'sgradestendtomatch 60

PAGE 61

3-5 -1=CompleteDisagreement,5=CompleteAgreement).Onestudentsaid,\Ithoughtitwasreallyinteresting,itwaschallenginganditwasgoodtorefreshmymemoryonalotofcommunicationandinterviewingskills."Anotherstudentnotedthatthesystemallowsonetopracticetheprocessofinterviewingapatientwithoutfeelingnervous:\Itwasalotlesspressurethanarealperson,evenastandardizedpatient.Intherewiththevirtualpatient,Iwasn'tworriedaboutlookingnaturalandcondent...lookingnaturaltotherealpatient.Iwasouttheretakingtimetryingtogureoutwhat'swrongwiththepatient."Thevirtualscenariowasavaluableeducationalexperience. Empathyencouragespatientstoshareinformation.Byexpressingempathy,participantswereworkingtowardstheirtaskofelicitingcriticalinformationfromthe 61

PAGE 62

3.5.3.1Empathy Ontheotherhand,GroupVHusedamorerehearsed,roboticempathy.Theyrespondedtothevirtualhuman'scryforhelp,buttheirlackofemotionalexpressionandmonotonevoicemadetheseempatheticresponsesappearinsincere.Ofcourse,participantscouldnottouchthevirtualhumanassheoccupiesthevirtualspacebeyondtheprojectiononthewall.However,noparticipanteventriedtotouchtheimageofthevirtualhumanonthewall.Indebriengs,oneparticipantfromGroupVHsaid:\I'm(normally)reallyengagingwithmypatients.Eventhoughitwasveryreal,itwasverycoldandarticial.Icouldn'tgetveryinvolved."Thiscommenthintsthatthepoorexpressivenessofthevirtualhumanledparticipantstoadapttheirconversationstyle. ItwasalsoclearthatsomestudentsinGroupVHbotheredtouseempathybecausetheyarerequiredtoininterviewswithstandardizedpatients.They'retraining(andfearofabadgrade)compelledthemtouseempathy.However,theydidnothavetoappearsinceresincethevirtualhumanwasnotcapableofevaluatingandrespondingtosincerity(orthelackthereof). Fromanevaluationandtrainingstandpoint,thesestudentsgamedthesystem.Theyknewtheycouldbehaveimproperlywiththevirtualhumanwithoutbeingpenalizedforit.Forthissystemtobeeective,itmustbeabletodetectwhenstudentsgamethesystemandrespondappropriately.Thevirtualhumanshouldmakeacommentorchange 62

PAGE 63

63

PAGE 64

3.5.4.1Dierences Thevirtualhumanwasmuchlessexpressive.Hervoicehadaregularvolumeandtone.Herfacedidnotconveytherightlevelofpain.Sheoccasionallyshiftedherbodyormovedherhands,butherfacialexpressionsdidnotchangeaccordingly.Besideslookingattheparticipant,thevirtualhumanusednootherexplicitbehaviorstoindicatelistening.Occasionaldelaysinspeechrecognitionproduceddelaysinthevirtualhuman'sresponses.Participantsofteninterpretedthistomeanthevirtualhumanwasnotasengagedintheconversation.FeedbackontheMaSPshowedthestandardizedpatientcommunicatedhowshefeltbetterthanthevirtualhumanandappearedtobeabetterlistener(Figure 3-6 ). Previousresultsdidnotindicatethevirtualhuman'slackofvocalexpressivenesswasamajordeciencyofthesystem[ 88 ],andnosignicantdierencewasfoundbetweensynthesizedspeechandmorerealisticrecordedspeech[ 43 ].Therefore,noeortwasputintoimprovingDIANA'svoice.However,thiscomparisonwasbetweendierentspeechmodesofthevirtualsystem.Whencomparedagainstthevoiceofthestandardized 64

PAGE 65

Dierencesinexpressivenesswerealsopronouncedbecausethevirtualhumananimationtoolsmadeitdiculttocreatesophisticatedexpressivebehaviorswithinareasonableamountoftime.Thelargedierenceinexpressivenesssuggeststhateortmustbeinvestedbeforefuturestudiesareconducted. 3.5.5 )andimpressionsoftheinteractionoverall(MSP=9:50:53,MVH=6:62:0onascaleof1to10,p=1E4)werelowerinGroupVH.Furthermore,althoughparticipantsaskedthevirtualhumanandstandardizedpatientsimilarquestions,somebehavior(Section 3.5.3 )wasdierentwiththevirtualhuman.GroupVHaskedquestionsinamoredirect,rapid-refashion,andchangestoconversationowwereobserved.Empathywasexpressed,buttheempathywasnotassincereasthatseenintherealscenario.Participantssuggestedthatthevirtualhumanbemoreexpressive:\Iwouldsuggesttohavemoreemotionsintothem.Maybeiftherewasmorefeelings,moreemotionalexpression."Theeectivenessofvirtualhumansarestronglyimpactedbytheirexpressiveness. 65

PAGE 66

3.4.1.3 )isastandardizedsurveylledoutbymedicalstudentstoassessstandardizedpatientauthenticityfromalocalandglobalperspective.Globalmeasureslookatoverallimpressionsoftheinteraction,likewhetherthestandardizedpatientactslikearealpatient.Localmeasureslookatspeciccomponentsoftheinteraction,likewhetherthepatientexpressedpainrealistically.ParticipantslledouttheMaSPafterinteractingwiththevirtualhumanandstandardizedpatient. Global(big-picture)measuresindicatedthevirtualinteractionwaslessauthenticthantherealinteraction.Thevirtualhumanappearedlessauthentic(MSP=50:0,MVH=3:80:58,p=9E6)andwaslesslesslikelytobeconsideredarealpatient(MSP=4:80:46,MVH=3:81:1,p=0:008).Also,thevirtualencounterwaslesssimilartootherstandardizedpatientencounters(MSP=4:51:1,MVH=2:50:94,p=2:00E4). However,local(subcomponent)measuresmostlyindicatethevirtualandrealscenarioswerenotdierentonauthenticity.Nodierenceswerefoundonwhetherthepatientsimulatedphysicalcomplaintsunrealistically(MSP=1:81:4,MVH=2:61:0,p=0:096),whetherthepatientansweredquestionsinanaturalmanner(MSP=21:4,MVH=2:91:2,p=0:13),andwhetherthepatientappearedtowithholdinformationunnecessarily(MSP=4:11:2,MVH=3:41:2,p=0:23).Asingledierencewasfoundonwhetherthepatient'sappearancetstherole(MSP=50:0,MVH=4:30:47,p=4:0E04) Giventhedierencesonbehaviorandexpressiveness,itissurprisingthatthevirtualandrealinteractionwereconsideredsimilaronlocalauthenticitymeasures.Onewouldexpectanyrealinteractiontoalwaysbeconsideredmoreauthenticthanitsvirtualcounterpart.Wehypothesizetherealandvirtualinteractionsweresimilarlyauthentic 66

PAGE 67

183 ]conclusionthatpeopleapplydierentstandardstorealandvirtualenvironmentsonpresencequestionnaires. 67

PAGE 68

43 ].Therefore,thetwogroupswerecombinedtogetherinouranalysis.However,asvirtualhumanexpressivenesshasbeenidentiedasaectingresults,recordedprofessionaltalentshouldalwaysbeused. Nevertheless,webelievethatstudents'experiencewiththerealinteractionpartiallytransferstothevirtualanddecreasesanyconfoundingeects.Thistransferoccursbecauseofthevariouswaysthevirtualinteractionmimicstherealinteraction.Studentsexperiencethevirtualinteractioninthesamemedicalexamroomsasintherealinteraction.Theprojectedvirtualexamroomismodeledaftertherealroom.Ithasthesamecolorwalls,thesamedimensions,thesamekindofpatientbedandsoon.Thevirtualhumanmimicsreal-worldsymptomsofappendicitis.Eventhestudyprocedurewasmodeledaftertheparticipants'normalexperienceswithstandardizedpatients. StudyI'sresultsalsoprovideevidencethatexperiencetransfersfromtherealtothevirtualinteraction.ThefactthatstudentsaskedthesamequestionsinbothrealandvirtualinteractionsimpliesthatGroupVHbroughttheirexperienceswiththemintothevirtualinterview.Furthermore,in[ 89 ],astrongcorrelationwasfoundininteractionskillsbetweenSPandVHinterviews.StudentswhodowellinVHinteractionsalsodowellinSPinteractions.LikewisestudentswhodopoorlyinVHinteractionsdopoorlyinSPinteractions.Thiscouldnotbepossibleunlessexperiencetransfersfromtherealtovirtual. 68

PAGE 69

AsinStudyI,studentsweresplitintotwogroups.GroupVH(nVH=33)interviewedthevirtualhumanandGroupSP(nSP=25)interviewedastandardizedpatient. 69

PAGE 70

3.4.1 ).Reachingadiagnosismaybeeasierwithadetailedpictureofapatient'smedicalhistory.Therefore,participantsweregradedonwhethertheyelicitedinformationonvehistorycategories:socialhistory,familyhistory,historyofpresentillness,medicalhistoryandreviewofsystems.Patienthistoryinformationisnotcriticaltoreachingadiagnosis,butitcanbehelpfulinnarrowingdownthelistoftopicstoaskabout. 70

PAGE 71

123 ].Expertsgradedparticipantsonnonverbalcommunicationbecauseitcontributessignicantlytorapportwiththepatient. Asourstudywasintegratedintothecourse,wefollowedthisprocedureprecisely,withonlyslightdeviations.AppointmentsweremadesothatGroupVHcoulddospeechtrainingbeforetheinteractions.Duringthestudy,GroupVHparticipantswereassignedoneroomwithavirtualhumanwithsymptomsofappendicitisandoneroomwithastandardizedpatientwithdierentsymptoms.Likewise,GroupSPparticipantswereassignedoneroomwithastandardizedpatientwithsymptomsofappendicitisandoneroomwithastandardizedpatientexperiencingdierentsymptoms. Duetothestrictcourseschedule,andadesiretomakethevirtualinteractionliketherealone,VICwasremovedfromthevirtualinteraction.Asaresult,thestart,2-minutewarningandendsoundswereallplayedfromthefacility'sspeakers.Also, 71

PAGE 72

Followingthecourseschedulemadeitdiculttocollectdatafromstudents.TherewasnotimeforparticipantstollouttheMaSPsurveyusedinStudyI,norwereparticipantsdebriefedforcomments.Ontheotherhand,anadvantageoffollowingthecourseschedulewasthatstudyparticipationwasamorefamiliarexperiencetostudents. Table 3-1 showsthepair-wisePearsoncorrelationoftheobserversonthethreesummarymeasures.TableentriescontainanXwherenocorrelationwasfoundduetoinsucientobserveroverlap.Mostobserversarereasonablycorrelatedwitheachother(r>0:4)andhavealessthan5%(<0:05)chanceofbeingcorrelatedduetochance.Thisimpliestheobserversratedtheinteractionssimilarly. 72

PAGE 73

3.7.2.1Elicitingcriticalinformation 3-7 comparesoverallperformanceonelicitingcriticalinformationinStudyIandStudyII.Notonlywasoverallperformancesimilaracrossgroups,itwasalsosimilaracrossstudies.Theconsistencyonelicitingcriticalinformationacrossstudiesstrengthenstheassertionthatcontentwassimilarintherealandvirtualinteractions. Asingledierencewasfoundonwhetherthestudentelicitedthelocationofthepain(MSP=0:750:36,MVH=0:910:16,p=0:02).Systemlogsshowthevirtualhumanoftenrevealedthepainlocationevenwhennotdirectlyaskedaboutit.Thiswasduetoerrorsinmatchingnoisyinputspeechtoresponsesinthevirtualhuman'sdatabase.Aspartoffuturework,thesystem'sresponsematchingthresholdswillbetunedtoreducefalsepositives.Thiswillhelppreventthesystemfromrevealinginformationthathasnotbeenaskedfor. 73

PAGE 74

Dierencesingatheringpatienthistoryhighlighthowpeopleadapttheirbehaviortothelimitationsofthevirtualhuman.Amedicalstudentwouldnormallyusemultiplefollowupquestionstoexplorethesetopics.Followupquestionsaredicultforthevirtualhumantohandlebecausetheyrequireknowledgeofcontext.AsinStudyI,participantsdiscoveredthattheycannotaskthevirtualhumancontext-dependentquestions,andtheyadaptedtheirbehaviorappropriately.Aspartoffuturework,weplanontrackingcontextoverthecourseoftheinterview.Thiswillallowthevirtualhumantodeterminethatfollowupquestionsrefertopreviousquestions. Itshouldbenotedthatnodierenceswerefoundonwhetherparticipantsaskedaboutsocialhistory(MSP=0:430:41,MVH=0:330:37,p=0:33).Thisislikelybecausesocialhistoryquestions(e.g.,\Doyoudrinkalcohol?")haveveryfewfollowupquestions.Also,participantsmayhaveavoidedsocialhistorybecauseitisasensitivesubject.Approximately60%ofparticipantsdidnotasksocialhistoryquestions. Dierencesonpatienthistorydonotnecessarilyindicatethatthecontentoftheinteractionwasdierentoverall.Gatheringcriticalinformationisamuchmoreimportantpartoftheinterviewthangatheringpatienthistoryandshouldbeweightedstrongerintheoverallassessmentoftheinterviewcontent.Despitedierencesongatheringpatienthistory,theoverallcontentofthevirtualandrealinterviewswassimilar. 74

PAGE 75

Mostprocessandetiquetteguidelineswerefollowedinthevirtualandrealinteractions.Participantsintroducedthemselves(MSP=0:7930:36,MVH=0:680:39,p=0:28),exploredtheirpatient'sconcerns(MSP=0:870:30,MVH=0:860:18,p=0:96)andendedtheinterviewappropriately(MSP=0:540:43,MVH=0:580:34,p=0:69).Theseresultsaresurprisingbecausethevirtualhumandoesnot\care"whethertheseguidelinesarefollowed.Thevirtualhumandoesnotactdierentlyifparticipantsendtheinterviewinappropriately.Clearlyparticipantsappliedrulesfromthereal-worldtothisvirtualinterpersonalinteraction. Whenprocessandetiquettewasabandoned,itwasbecausethevirtualhumancouldnothandlethemproperly.Forexample,GroupVHconductedtheinterviewinalesslogicalandorderlyfashion(MSP=0:870:25,MVH=0:530:35,p=0:0001).Participantsdidnothavealogicalorderlyconversationwiththevirtualhumanbecausethevirtualhumanisincapableofhavingaconversationinalogicalorder.Forexample,astudentmaybediscussingheadacheswiththevirtualhuman.Ifspeechrecognitionmisinterpretsthenextquestiontobeaboutfever,thevirtualhumanwillsuddenlychangethetopicandrespondaboutherfever.Thisunexpectedtopicchangeshowsparticipantsthatthesystemdoesnotcareabouttheorderofquestions.Therefore,participantsdonotbotherinteractingwiththesysteminanylogicalorder. Asmentionedpreviously,futureversionsofthesystemwilladdressthisbytrackingthecurrenttopic,orcontext,oftheconversation.Thiswillallowthevirtualhumantodeterminewhenaquerychangesthetopicandifthechangeintopicislogical.Ifthetopicchangeisunexpected,thevirtualhumancanasktheusertorepeatthequestiontoconrm. 75

PAGE 76

Surprisingly,specic,descriptiveratingsofempatheticbehaviorinthevirtualandrealinteractionswerenotdierent.Forexample,Figure 3-8 showsthatbothgroupswereratedsimilarlyondescriptivescaleslike\good/bad,"\weak/strong,"and\active/passive."Thisisinstarkcontrasttotheoverallsensethatempathybehaviorwaspoorerwiththevirtualhuman.Wehypothesizenodierenceswerefoundonthesescalesbecausetheyaretoosubjective.Theexpertraterscouldnotobjectivelyrateabstractconceptslike\weak/strong."Aspartoffuturework,weareexploringtheuseofobjectivebehavioralmeasurestoaugmentthesesubjectivemeasures. 3-9 -1=VeryPoor,4=VeryGood).Theyusedlessforwardbodyleanandnoddedless.Thesebehaviorswereinappropriatebecausetheyareassociatedwithlowerinterestandapoorerattitude[ 123 ].Notsurprisingly,GroupVHappearedlessattentiveandhadalesspositiveattitudewiththevirtualhuman(Figure 3-9 ). Itshouldbenotedthatexpertratingsofparticipants'eyecontactweresimilarwiththevirtualandrealhuman.ThiswasalsoseeninStudyI,whereparticipantsindicatedthevirtualhumanandstandardizedpatientmaintainedgoodeyecontact.Thevirtualhumanconstantlylookedattheparticipantthroughouttheinterview,inuencing 76

PAGE 77

Clearly,expressivenessmustbeimprovedfurther.Thevirtualhumanshoulduseeverydayconversationalidiosyncracies,likestoppingtothinkandsaying\um"and\uh."Herfaceshouldconveymorepain.Herbodyshouldbelessrigid,yetstillenoughtoconveythepainthatmovingcreates.Herresponsestoqueriesshouldbeimmediate.Thislistisonlyasmallsampleoftheexpressiveabilitiesthatmustbeimproved. 77

PAGE 78

Thestudiesalsoshowthatinteractionauthenticityandparticipantempathycannotbeassessedeasily.Globalmeasuresshowedtherealscenariowasmoreauthentic,butlocalmeasuressuggest-onacomponentlevel-thevirtualscenariowassimilartotherealscenario.Asimilarcontradictionwasseeninexpertratingsofempathy.Whileparticipantsappearedtouseinsincereempathywiththevirtualhuman,subjective,descriptiveratingsofempathyfoundnodierences.Theseresultsleadtothefollowingguidelinesforconstructingandevaluatingeectiveinterpersonalsimulators. 78

PAGE 79

Mostcriticaltounderstandingwhyparticipantbehaviorchangeswithvirtualhumansisthedevelopmentofobjective,physicalmeasuresofbehavior.Sensors,likethemicrophoneandreectivemarkersusersalreadywear,willbeusedtocharacterizephysicalbehavior.Thefollowingsubsetofbehaviorsthatimpactperceivedrapportwiththepatientwillbetracked. Tohelpwithinterpretingthisbehavioraldata,atoolforvisualizinginteractionsbetweenrealandvirtualhumansisbeingdeveloped.Visualizationwillprovideafocusinglensthroughwhichwecananalyzethecollectedbehavioraldata. 79

PAGE 80

Pearsoncorrelationbetweenobservers. Yes/NoObjective Observer O1O2O3O4O5 O1 10.7950.6540.822X O2 0.79510.8620.597X O3 0.6540.86210.6000.754 O4 0.8220.5970.60010.166 O5 XX0.7540.1661 Observer O1O2O3O4O5 O1 -9E-63E-52E-5X O2 9E-6-1E-100.040X O3 3E-51E-10-0.0020.083 O4 2E-50.0400.002-0.754 O5 XX0.0830.754Empathy Observer O1O2O3O4O5 O1 10.820.5840.839X O2 0.8210.4840.738X O3 0.5840.48410.4720.911 O4 0.8390.7380.47210.467 O5 XX0.9110.4671 Observer O1O2O3O4O5 O1 -3E-63E-47E-6X O2 3E-6-0.0040.006X O3 3E-40.004-0.0170.012 O4 7E-60.0060.017-0.351 O5 XX0.0120.351NonverbalCommunication Observer O1O2O3O4O5 O1 10.4240.7550.83X O2 0.42410.3070.454X O3 0.7550.30710.6700.592 O4 0.830.4540.67010.990 O5 XX0.5920.9901 Observer O1O2O3O4O5 O1 -0.0492E-71E-5X O2 0.049-0.0830.138X O3 2E-70.083-2E-40.215 O4 1E-50.1382E-4-2E-4 O5 XX0.2152E-4-

PAGE 81

Realinterpersonalinteraction(left)andequivalentvirtualinterpersonalinteraction(right). Figure3-2. Systemoverview 81

PAGE 82

StudyprocedureforgroupsSPandVH Figure3-4. VIC(left)andDIANA(right)inthevirtualexamroom 82

PAGE 83

Perceivededucationalvalueofrealandvirtualinteractions Figure3-6. Perceivedexpressivenessofvirtualhumanandstandardizedpatient Figure3-7. Elicitinginformationscoreforbothgroupsandstudies. 83

PAGE 84

Expertratingsofempathyondescriptivescales Figure3-9. Nonverbalbehavior,attitude,andattentiveness 84

PAGE 85

Thischapterdescribes,IPSViz,adesktop-basedafter-actionreview(AAR)systemforinterpersonalinteractionswithavirtualhuman(VH).IPSVizleveragesvisualizationtoenablereviewofinterpersonalcommunicationwithavirtualhuman.ApilotstudyreviewstheimpactofIPSVizonusersofVHexperiences.ThisworkwaspublishedintheproceedingsoftheIEEEconferenceonVirtualReality2008[ 147 ]. 85

PAGE 86

147 ],reviewingone'sinteractionwithavirtualhumanwasfoundtochangeself-perceptionsofcommunicationskills.TheeectwasstrongenoughthatstudyparticipantsclaimedtheywouldchangebehaviorwithrealhumansasaresultoftheAAR.Suchself-directedchangewouldbeexpectedforinteractionswithrealhumans,butwassurprisingforinteractionswithvirtualhumans.State-of-the-artvirtualhumanscanbesimilartorealhumans,buttheyclearlydonotmeetthevisualandbehavioralrealismofrealhumans[ 144 ].Thus,itwouldbeunexpectedforuserstoreviewtheirinteractionwithavirtualhumanandbetterunderstandtheirinteractionswithrealhumans.Thisarticlebuildsonthisunexpectedresultbyexploringtherelationshipbetweentherealismofthevirtualhumanandtheimpactsofafter-actionreview(AAR).Thisexplorationledtotwonewdiscoveries:1)aninitialdenitionofthescopeofimpactsanAARofanH-VHexperiencecanhaveonauser,and2)evidencethatsuchimpactsareattenuated,butnoteliminated,byvirtualhumanswithlowinteractionrealism. 79 ],lawenforcement[ 59 ],culturalcompetency[ 10 40 ],medicalinterviewanddiagnosis[ 89 115 144 ],andinteractionbetweenchildren[ 159 ].AstrainingforH-Hsocialsituationsrequiresafter-actionreviews(AAR),sodoestrainingwithH-VHinteractions. After-actionreviewsplayanimportantroleincommunicationskillstraining.Communicationskillstrainingisamajorcomponentofbusiness,military,andmedicaleducation.Inbusiness,military,medicaleducation,andothereldswherecommunicationskillsarecrucialtosuccess,instructorsteachstudentscommunicationskillsusinglectures,role-play,andsituationalimmersionwithinstructor-observation.AcriticaleducationalcomponentofthesemethodsistheAAR.InAARs,studentsreviewtheirsocialinteraction 86

PAGE 87

Recently,VHshavebeenappliedtosimulatingandeducatingcommunicationskills.Thus,weproposeaugmentingH-VHinteractionswithAARtoimprovecommunicationskillseducationwithVHs.Oneofthispaper'saimsistoprovidetoolsthatsupportAARsofH-VHinteractions.SuchtoolsallowausertointeractwithaVH,andthenreviewtheircommunicationwiththeVHminutesafterwardtolearnhowtoimprovefuturecommunicationwithrealhumans. 1. AnH-VHinteractioniscomposedofsocial,temporal,andspatialcharacteristics.Thus,usersshouldbeabletoexplorethesecharacteristicsinIPSViz.IPSVizleveragesvisualizationtoexplorethesecharacteristics. 2. AnH-VHinteractioncanbecharacterizedasasetofsignals.Thus,interactionsignalsarecaptured,logged,andprocessedtoproduceAARvisualizations. 3. AnH-VHinteractioniscomplex.Thus,usersgaininsightintothiscomplexitybyreviewingmultiplevisualizations,suchasaudio,video,text,andgraphs. ToenableAARforcommunicationskillseducation,IPSVizgeneratesvisualizationsofH-VHinteractions.NovelvisualizationsaregeneratedbyleveragingthemanyinteractionsignalsthatarecapturedinanH-VHinteraction.GivenanH-VHinteraction,AARisfacilitatedthroughthefollowingvisualizationtypes: 87

PAGE 88

ThestudyfoundthattheH-VHexperiencedoesnotendwhentheuserleavesthevirtualenvironment.Rather,theimpactofthevirtualexperiencecontinuesintotheafter-actionreview.ThroughAAR,self-reection,changesinself-perceptions,andself-directedchangearepossiblewithhuman-virtualhumanexperiences.Furthermore,human-virtualhumanexperienceselicitthemostimpactonuserswhenthey1)arecoupledwithafter-actionreviewstotheexperience,and2)striveforhighVHinteractionrealism. 4.2.1ExpandingAARtoH-VHInteractions 88

PAGE 89

4-3 ). Togeneratevisualizationsofaninteraction,theinteractioniscapturedfromavarietyofsensors(Section 4.3 ).Fromasignalanalysisperspective,captureisequivalenttosamplingtheinteractionasifitwereasetofcontinuoussignals.Theseinteractionsignalscharacterizetheinteractionbetweenahumanandvirtualhuman. Beforegeneratingvisualizations,interactionsignalsmayundergolteringandprocessing(Section 4.4 ).Inbothstages,achainofdigitalltersisappliedtooneormoresignalstoderivenewsignals.Filteringandprocessingareseparatedintotwostagesaseachsolvesadierentproblem.Filteringcompensatesforerrorscausedbysamplingacontinuoussignalwithreal-worldsensors(e.g.,discretizationerrorandnoise).Processingmanipulatesandcombinessignalstoprovidenewinformationabouttheinteraction. Afterlteringandprocessing,interactionsignalsaremappedtothevisual(orotherperceptual)domaintoproducevisualizations(Section 4.5 ).ThevisualizationsallowuserstogainnewinsightintoH-VHcommunication. 88 ](Figure 4-2 ).ThisexperiencewaschosentoguidethediscussionbecauseHPstudents1)takethisinteractionseriously[ 89 ],and2)needreview,evaluation,andfeedbacktoimprovetheircommunicationwithpatients[ 35 157 ]. AtypicalinteractionbetweenaHPstudentandaVHbeginswiththeVHcomplainingofamedicalproblem(e.g.,pain).Thestudent'sgoalistodeterminewhattheproblemis(diagnosis)andtreatit.Eectivediagnosisandtreatmentrequiresgatheringaccurate 89

PAGE 90

Duringtheinterview,theVHwillalsoaskthestudentquestionstolearnwhatishappeningtoherandwhy.TheVHmayalsoaskquestionsinthehopethatthestudentcanrelieveheranxietyaboutthemedicalproblem.TypicalquestionsaVHcanaskinclude\DoyouknowwhatIhave?"and\Doyouthinkthiscouldbecancer?"StudentsshouldanswerthesequestionscarefullytorelievetheVH'sanxietyandbuildrapportwiththeVH. Naturalspeech:Studentswearawirelessmicrophoneontheirhead.ThisenablestalkingtotheVHusingnaturalspeech.Speechrecognitionsoftware(DragonNaturallySpeaking9Pro)extractsthewordsspokenbytheuserfrommicrophoneinput.Boththespeechwaveformandthespeechrecognitionoutputarelogged. 90

PAGE 91

Naturalspeechandanimation:WhenastudentspeakstotheVH,thespeechrecognitionsoftwareinterpretsherwords.AkeywordmatchingalgorithmmatchesspeechrecognitionoutputtoquestionsintheVH'sresponsedatabase.Ifamatchisfoundinthedatabase,theVHexecutesacorrespondingvocalandgestureresponse.TheVHrespondsappropriatelytothequestion60to70%ofthetime.TheVH'sgestureandspeechresponsesarelogged. Usingthehead-trackinginputs,thesystemrenderstheVEfromthestudent'sperspective.ThisallowsthestudenttoperceivetheVH'sgazebehavioraccurately.TheheadtrackingdataalsoallowstheVHtorespondwhenthestudententerstheroom.IntheHMDcondition,head-trackingenableslookingaroundthevirtualroom.Life-size,user-perspectiverenderingandVHresponsivebehaviorscreateahighlyimmersiveexperience.3Denvironmentparameters,renderingparameters,andVHgazearelogged. 91

PAGE 92

Notethatthislistofdataisnotmeanttobeexhaustive,norcompletelyrepresentativeforallVHapplications.Rather,thetypesofdatacapturedwerechosenbecauseoftheimportanceof: 78 124 ], 35 167 ],and 10 36 40 79 89 115 159 ]. 92

PAGE 93

Toovercometheseissues,anapproximatebodyleansignalL(t)iscomputedbycombiningheadandchairtrackingdata.First,headandchairdataislteredtocompensatefortrackerjitter.Thenthechairpositionissubtractedfromtheheadpositiontocomputeahead-chairvector.Thehead-chairvectorisasubstituteforamoreaccuratebodyleanvectorthatrunsalongthestudent'sspine.L(t)issettotheanglebetweenthehead-chairvectorandtheup-vector. 4-4 didnotchangehisbodyleanexceptforleaningforwardandthenbackaround3:12. 93

PAGE 94

Astheitalicizedwordsaboveshow,thesequestionsarespatialandtemporalinnature.Thespatialquestionsfocusonwhereobjectsareandhowtheyarerelatedtoeachotherinthespaceofthe3Dworld.Thetemporalquestionsfocusonwheneventshappenandhowlongtheyhappenfor.Furthermore,thesequestionsfocusonhowthestudentbehavedsociallywiththevirtualhuman.IPSVizgeneratesvisualizationsthatarespatial,temporal,andsocialtohelpstudentsgaininsightintotheircommunicationwiththeVH. 4-6A ).SeeingtheinteractionthroughtheVH'seyesisapowerfulwayofdemonstratingtostudentshowtheirnonverbalbehaviorisperceivedbytheirpatients. 94

PAGE 95

4-6B ).Thegazetargetisatexturethatisprojectedwherevertheuserwaslookingduringtheinteraction.Thegazetargetallowsstudentstobecomeawareofwheretheirattentionwasactuallyfocused,asopposedtowheretheythoughtitwasfocused. 4-1 -bottom).Additionally,eventscanbeselectedfromatranscript(Figure 4-1 -right)toreviewthem. 161 ]:\Overviewrst,zoomandlter,thendetails-on-demand."Thetopic(Figure 4-5 )andbodyleanplots(Figure 4-4 )demonstratetheuseofscalabletimelines. 95

PAGE 96

4-1 -right),graphicallywithatopicplot(Figure 4-5 ),andaurallybyplayingrecordedaudiooftheinteraction. 8 ].Hence,gazeishighlightedinIPSVizbyrenderingthegazetargetandheadposeofthe3dmodelofthestudent.Inaddition,renderingthescenefromtheVH'sviewpointallowsstudentstoreviewtheirheadgazebehaviorfromthepatient'sperspective,andlearnthatpatientsnoticeheadgazebehaviorandknowwhentheyarenotpayingattention. 124 ].ThestudentcanreviewtheirpostureinIPSVizby 96

PAGE 97

78 ].Playingaudiooftheinteractionallowsreviewofparalanguage. Inaddition,twoseparatestudieswereconductedconcurrentlytoevaluatetheeectofVHskintone(lightvs.dark)andsystemdisplaytype(head-mountedvs.projectiondisplay)ontheH-VHinteraction.VHskintoneanddisplaytypearebetween-subjectsfactorsthatwerevariedrandomlyamongtheN=27participants.Thebetween-subjectsresultsfromthesestudieswillbereportedinfuturepublications.WhilenocrossinteractionsbetweenVHskintone,displaytype,andAARwereexpected,thesefactorsweretakenintoaccountintheanalysisasaprecaution. 97

PAGE 98

4-7 summarizesthestudyprocedure. Inthemaininteraction,theVHwasEDNA(ElderlyDIANA),a55-year-oldwomanwhofoundamassinherbreast.ParticipantswereinstructedtointerviewEDNAasifshewerearealpatient.Participantsweretoldtheywouldnotreceiveanyassistanceandtheproctordidnotentertheroomwiththestudent.Themaininteractionwasbetweenveandtenminuteslong. InterviewingapatientwhofoundamassinherbreastisachallengingsituationforanHPstudent.WechosetoamplifythischallengeandincreasetherealismofthescenariobyhavingtheVHchallengethestudentatthreeseparatepointsintheinterview. 98

PAGE 99

4.6.3 InthemainAAR,studentsusedIPSViztoreviewthemainVHinteraction.First,studentsreviewedtheinteractionontheirown.TheywereencouragedtouseIPSVizanywaytheywanted,includingchanging3Dviewpoints,reviewingplots,andskippingtoimportantevents.Thestudyproctordidnotinteractwithparticipantsduringthistime.Whenthestudentwasdone,thestudyproctorconductedashort,guidedreviewoftheinteraction.Theguidedreviewdirectedstudentstoreviewtheirreactionstothethreechallengestatements(asdescribedpreviously).ThemainAARtookapproximately10minutes. 99

PAGE 100

144 ].ToassessifparticipantsrecognizedthisimproperbehaviorduringAAR,participantsevaluatedthemselvesonnaturalness,expressionofemotion,andfriendliness.Five\friendliness"scaleswereused-pleasant,cruel,cold,unfriendly,andunlikeable(sevenpointLikertScales,1=Notatall,4=Neutral,and7=Very).Thefriendlinessmeasureshavehighinternalconsistencyforrealhumaninteractions(>0:9)[ 47 ]. Questionsfromthesurveyaredividedintothreecategories,informationgathering,buildingrapportwiththeVH,andproceduralaspectsoftheinterview.Sampleinformationgatheringquestionsare\Ratehowwellyoufoundoutallcomplaints"and\Ratehowwellyouelicitedthestoryandmeaningaswellasbiomedicalfacts."Samplerapport-buildingquestionsare\Didyoulegitimizethepatient'sideasandfeelings?"and\Didyouuseappropriateeyecontact?"Sampleprocedurequestionsare\Didyouusethepatient'snameappropriately?"and\Ratehowwellyoubeganwithopen-endedquestionsand 100

PAGE 101

4.7.1After-ActionReviewImpactsSelf-Perceptions SurveyresultsshowthatafteractionreviewchangedparticipantperceptionsoftheirinteractionwiththeVH.Participantsindicatedtheywerelessfriendly(F1;19=17:7,p<:001,Pre-AAR:M=5:5;SE=:19,Post-AAR:M=4:6;SE=:25)andlessnatural(F1;19=17:1,p<:001,Pre-AAR:M=3:8;SE=:27,Post-AAR:M=2:7;SE=:28)towardsthevirtualhumanthantheyremembered.On15separatemeasuresofrapport-building(e.g.,non-verbalbehavior,listening,andsensitivity)usersindicatedtheirrapportwiththeVHwasworseaftertheAAR(F1;19=18:4,p<:001,Pre-AAR:M=2:54;SE=:11,Post-AAR:M=2:18;SE=:11).TheAARalsochangedthewayparticipantsinterpretedthestateoftheVH.TheVHwasperceivedasbeingmorescaredafterAAR(F1;19=4:1,p<:06,Pre-AAR:M=69;SE=5:0,Post-AAR:M=76;SE=2:8).Nodierenceswerefoundonratingsoftheuser'semotionalexpressionandonperceptionsoftheVH'sfriendliness. TheeectofAARonperceptionsofinformationgatheringandproceduralskillswasnotasclear.AfterAAR,participantsratedthemselvesloweron\ndingoutallthepatient'scomplaints"(F1;19=18:3,p<:001,Pre-AAR:M=2:4;SE=:15,Post-AAR:M=2:0;SE=:12)and\beginningwithopen-endedquestionsandmovingtoclosed-endedquestions"(F1;19=4:87,p<:04,Pre-AAR:M=1:8;SE=:18,Post-AAR:M=1:5;SE=:17).Incontrast,dierenceson\gatheringthepatient'sstoryaswellasbiomedicalfacts"and\usingthepatient'snameappropriately,"weresmallandnotstatisticallysignicant. 101

PAGE 102

StudentsmentionedspecicimproperbehaviorsthatIPSVizhelpedthembecomeawareof. StudentsreportedtheywouldchangebehaviorinfutureinterviewswithrealpatientsbasedontheAARwithIPSViz. 102

PAGE 103

ThenotionthatanH-VHexperienceshouldimpactfutureinteractionswithrealhumansisasurprisingresultofthestudy.Giventhetechnologicallimitationsofthesystem(e.g.,speechrecognitionandunderstanding),itwasnotclearthatuserswouldseetheVHexperienceascorrelatingtointeractionswithrealpatients.AARservedasacatalysttoconnectinginterviewswiththeVHtointerviewswithrealhumans. 4.7.3AAR,VHSkinTone,andDisplaytype DisplaytypedidnotaecttheAARresponses,butanintriguingcross-interactionbetweenVHskintoneandAARwasfound.AsshowninFigure 4-8 ,thedierenceduetoAARon\showinginterestintheVH"(F1;19=6:9,p=:02)wasaectedbytheVH'sskintone.AfterAAR,studentswhotalkedtoadark-skinnedVHloweredtheirscoreonshowinginteresttotheVH.Studentswhointeractedwithalight-skinnedVHdidnotchangetheirscore. ThecrossinteractionbetweentheVHSkinandAARconditionsshowsthatparticipantsconsideredtheskintoneoftheVHwhenevaluatingtheirbehaviorwithIPSViz.Theyevaluatedthemselvesdierentlyonthismeasuredependingonwhethertheyinteractedwithadarkorlight-skinnedVH.Thisimpliesparticipants(consciouslyorsubconsciously)perceivedthattheskincolorofthedark-skinnedVHbiasedtheirinteraction.Thus,theVHexperienceservedasabiasdetector.ByinteractingwithaVHandconductinganAAR,participantsdetected1)abias,and2)thatthisbiasaectstheirbehavior. 103

PAGE 104

Thesectionisdividedintofourparts,eachdiscussingadierentgroupofcommunicationskillsthatusershighlightedduringtheAAR.Allofthesecommunicationskillsarenotjustimportantinthismedicaltrainingscenario,butforanysocialcontext.Theyare: Inidentifyinguserreectiononthesebroadcommunicationskills,weshowthatinteractingwithaVHandconductinganAARofthatinteractionenablesevaluatingarangeofone'scommunicationskills.Furthermore,astheseskillsareusefulinmanysocialcontexts,thelessonslearnedfromtheVHinteractionmayapplyoutsidethesocialscenariotheVHinteractionsimulates. Acommonthemeinthesectionstofollowisthatuserswerewillingtocriticizetheircommunicationwithavirtualhumanasifitwascommunicationwitharealhuman.Thismeansuserssawvirtualhumansasequivalent,orasanacceptableproxiesfor,realhumans.Thenotionthatavirtualhumanisanacceptableproxyforarealhuman 104

PAGE 105

Inthisstudy,theAARhelpedparticipantsnotetheirdicultywithverbalcommunication.Oneparticipantdirectlystated,\[Idid]so-soonthequestionanswering/askingpart."Threespecicaspectsofverbalcommunicationwerehighlightedasneedingimprovement,thephrasingofquestions,thethoroughnessoftheinterview,anditsorganization. Thetopicplotplayedaroleinhelpingstudentsevaluatethoroughness.Thetopicplotgraphicallydepictedwhenimportantscenariotopics(e.g.,breastmass,familymedical 105

PAGE 106

Severalparticipantsnotedtheirinterviewswereorganizedpoorly.Oneparticipantstated,\InoticedIwasjumpingaround-askingquestionsaboutthebreastlump,andthenIwouldtrygotopastmedicalhistory,andthenIwould,like,trytogoback." Again,thetopicplotplayedanimportantroleinshapingparticipantperceptionsoftheirorganization.Forsome,thetopicplotintroducedtheideaofintervieworganization:\Therewerethingsafterlookingatthereviewtool.especiallylikethelittleplotwiththeowofconversation.IguessIneverthoughtabouttheowofconversationthroughtheinterview."Forthosewhoalreadyunderstoodtheconceptofintervieworganization,thetopicplotgavethemabetterunderstandingofwhatanorganizedinterviewlookslike:\IthinkoneofthethingsIdidlikeisthe[topicplot's]colorcodingofwhatparticularpartsofthehistorywereaskedatdierenttimes.Iguessyouwouldsometimeswantallthecolorstobeinasimilarplace,ifyouweredoingaprettysystematicapproach."Oneparticipantcomparedwhatsheconsideredtheidealorganizationofaninterviewtowhatheobservedonthetopicplotofhisinterview:\Normallyyou'regoingtodoyourchiefcomplaint,history,blahblah.Butshowingthingsbeingallaround[onthetopicplot],painhere,symptomshere,painbackagain.Thatgetsyoukindofseeinghowtheinterview 106

PAGE 107

8 124 ]. Participantsevaluatedgazebehaviorbyobservingboththegazetargetandtheavatar'sheadorientation:\Ididn'trealizehowmuchIwaskindoflookingaround.Like,IswitchedittoherpointofviewandIwaskindoflookingallaroundinsteadoflookingather...[This]willmakemebemoreconsciousofwhereI'mlookingwhenI'mtalkingtoapatient.BecauseIthoughtIhadfairlygoodeyecontactasfaraspayingattentiontoher,butthebullseyewasgoingallaround.Isawthat." ThecommentsalsoshowthatuserscaninterpretAARvisualizationsincorrectly.Inseveralofthequotesabove,end-usersmisinterpretedthemeaningofthegazetarget.Althoughthegazetargetdemonstratedheadgaze,severalend-usersthoughtthegaze 107

PAGE 108

MisinterpretationoftheAARvisualizationspresentsachallenge.Ifend-usersmisunderstandthevisualization,theymaybeencouragedtochangebehaviorthatdoesnotneedtobechanged(orviceversa).Eortsshouldbemadetodisambiguatethemeaningofvisualizationssothatend-usersdonotchangebehaviortheydonotneedto.Nonetheless,itisstillencouragingthatthegazetargetimpactedend-userperceptions.ThisshowsthatAARshavethecapacitytochangeperceptionsofbehavior.CapitalizingonthiscapacitytochangeperceptionsaccuratelyisanimportantstepindevelopingeectiveAARtoolsforH-VHexperiences. 78 ].Thus,evaluatingone'sparalanguageisimportantforlearninghowonecommunicateswithothers. DuringAAR,thefollowingformsofparalanguagewereidentiedasneedingimprovement. ParalanguageplayedalargeroleincommunicatingdiscomfortandnervousnesstotheVH.Forexample,longpauses(orquiet)communicateddiscomfortandlackoffriendliness: 108

PAGE 109

Reviewingtheuser'savatarfromthepatient'sperspectivealsohelpedparticipantsevaluatetheirbodypostureandmotion:\It'sinterestingseeingwhereyourheadstartsoryourbodyposition{thatyoudon'trealizewhileyou'reinteractingwiththepatient.LikeIthinkthatmightbeoneofmyfavoritefeaturesof[IPSViz],gettingtoseehowIwaspositioned,wheremyheadwasturned.AndIlikebeingabletoseeitfromthepatient'sperspective,too,whatitlookslike,becausethat'saneattoolandsomethingyoudon'tthinkabout."Thiscommenthighlightstheimportanceofreviewinganexperiencefromtheperspectiveofaconversationpartner.Itenablesseeinghownonverbalbehaviormightbeinterpretedbyaconversationpartner. 109

PAGE 110

42 90 101 ].Manyparticipantssaidtheywerenotasempathicastheythought:\Therewasdenitelyalackofempathyandactuallyrelatingtothepatient."Oneparticipantsaidhemissedopportunitiestobeempathic:\IfeellikeIlearned,um,likehowto,youknow,ifthepersonbringsupitcouldpossiblybecancer,thenIshouldprobablyaddressthat.IthinkIwastoofocusedontryingtogetwhatkindofsymptomsshehas."Thestudentnotonlyrecognizedthathewasn'tempathic,butthathisstrongdesiretodiagnosethepatientpreventedhimfromrecognizingthepatient'sneedforempathy. 110

PAGE 111

DuringtheAAR,participantsnotedtheirreactionstothevirtualhumanwhileunderstress:\Ihavealottoimproveon.I'marst-yearPAstudentsoI'vehadlimitedpatientexperience.Especiallygoingintosomethingwhereyoudon'treallyhaveanyideaofwhatyou'regoingtogetinto.Soitwasalittlebitofastruggle.AndthenassoonasIgotustered,ofcourse,itincreasedjustasmyownawkwardnessalreadyatdoing[apatient]historyaswell.That'sgoodtoseeinthisstudy." Acommonthemeinparticipantdiscussionsaboutstresswasthatthestressstemmedfrominteractingwiththevirtualhuman:\Alsoitshowsthat,Iguessunderalittlebitofstress...you'reintherewithallthecamerasandwhatnot,Iguessyoucanseehowthingsdon'talwaysowtalkingwiththepatient." InVHexperiences,stresslevelisparticularlyhighbecauseofthetechnologicallimitationsoftoday'sVHs.WithourVH,theVHdoesnotalwaysrespondtoquestionscorrectlyordisplayemotionsproperly.Furthermore,theinteractionrequiresencumbrancessuchastrackingequipment.Thesetechnologicallimitationsfrustrateandstressend-users.Whilethecausesofthisstressaredierentfromthecausesofstressinasimilarreal-worldscenario,multipleparticipantssaiditwasbenecialtoreviewhowtheyreactunderstresswiththeVH.ThisindicatesparticipantssawarelationshipbetweentheirbehaviorunderstresswithaVH,andtheirbehaviorunderstresswitharealhuman. 111

PAGE 112

Thescopeofreal-worldinterpersonalskillsobservableinanH-VHexperiencearebroadenoughforuseinevaluatingreal-worldinterpersonalskills.Participantsobservedandreectedontheirverbalandnonverbalbehavior,theirabilitytobuildarelationshipwithacommunicationpartner,andhowwelltheycommunicatedunderstress.TheseskillsareimportantinH-Hinteractions.Thus,notonlycanreviewinganinteractionwithavirtualhumanprovideinsightintoone'sinteractionswithrealhumans,butthatinsightpertainsdirectlytoskillsthatmatterinH-Hexperiences.

PAGE 113

113

PAGE 114

Sinceparticipantssawtheinteractionwiththevirtualhumanasunrepresentativeofreal-worldinterpersonalskills,participantsdismissedsomeofthefeedbackfromtheAARasirrelevantforinteractionswithrealhumans.26outof27participantssawthemselvesinteractpoorlywiththevirtualhuman,butdismissedthatfeedbackpartiallyorentirelyduetotheVH'slackofhumanrealism.Forexample,oneparticipantsawthevirtualhumanasmorelikeaprogramthanahuman:\Itreateditasaprograminsteadoftryingtogettothediagnosis.[Idid]notreallytreatthepatientasapersonasIusuallywouldtreatifarealpersoncomestotheclinicinsteadofasimulatedprogram."Tothisparticipant,hisbehaviorwith\aprogram"isnotareectionofhisbehaviorwithrealpeople.OneparticipantindicatedtheVH'slackofrealismkepthimfromusingaspecictheinterpersonalskillofempathy:\WhenIwentinthereIdidn'tthinksomuchaboutusingempathy,becausethisisacomputerpatient.InoticedthatwhenIlookedatitagain,that'sverydierentthanhowIamwitharealpatient."AnotherparticipantdismissedhismistakeswiththevirtualhumanbecauseoftheVH'sdicultyansweringhisquestions:\Idon'tguessIwoulddoanythinganydierent[inafutureinteractionwithapatient].Becausethat'snotreallyhowIwouldgointoaroom.It'sdicultwhentheydon'tunderstandwhatyou'resaying.Youhavetoaskthem[questionsin]certainways."ThisparticipantfeltheshouldnotchangehisbehaviorinfutureinteractionswithrealpatientsasaresultoftheAAR,becausetheAARprovidedhimwithfeedbackbasedonanunrealisticsituation. DespitetheapparenteectofVHinteractionrealism,allparticipantsself-reectedontheirinteractionwiththevirtualhumanandmanyfoundthattheinteractionwiththevirtualhumanprovidedsomeindicationsoftheirinterpersonalskillswithrealhumans. 114

PAGE 115

ThefocusofparticipantcommentsonVHinteractionrealismsupportstheargumentthatinteractionrealismismoreimportantthanotherformsofrealisminH-VHexperiences[ 33 189 ].Otherformsofrealism,suchasvisualrealismandscenariorealism,werenotmentionedbyparticipants.Thisindicatesotherformsofrealismwereaminimalfactor(incomparisontointeractionrealism),ornotafactoratall,indeterminingtheeectivenessoftheAAR. 115

PAGE 116

Thesehypothesesareconrmedinthefollowingsubsections,bolsteringtheargumentthatVHinteractionrealismhasalargerinuenceoverthesuccessofanH-VHinteractionthanotherformsofrealism,aswellasmotivatingtheneedfornewapproachestocreatingvirtualhumansthatinteractmorelikerealhumansthantoday'sstate-of-the-art. AsitisdiculttoalgorithmicallydetermineifaVHrespondedproperlytoaperson'sspeech,Rwascalculatedbyexternalobservers.ObserversrsttranscribedeachH-VHinteraction.Then,foreachparticipantutteranceintheinteraction,thevirtualhuman'scorrespondingresponseswerecategorizedaslogicalorillogicalfortheutterance.Forexample,iftheparticipantsaid,\Doyouhaveanyallergies?"andthepatientresponded,\Mymotherhadbreastcancer,"thiswouldbecategorizedasillogicalbecausethe 116

PAGE 117

4.6.3 )theAAR.Ifself-ratingschange,theimplicationwouldbethattheAARwaseectiveaselicitingself-reectiononanH-VHinteraction,andchangesinself-perceptionsofinterpersonalskills. UsingRandPostAARPreAAR,twotypesofstatisticaltestswereconductedtoconrmhypothesesH1andH2,correlations(Pearson's)anddierencetests(ANOVA),respectively.ThecorrelationtestsshowthatVHinteractionrealismpredictsAAReectivenessonprovidingfeedbackonrapport-building.ThedierencetestsshowVHinteractionrealismaectsthefeedbackfromanAARonrapport-building 1. Itwouldindicatethereisarelationshipbetweenthesevariables,bolsteringtheargumentforvirtualhumansthatinteractmorelikerealhumans. 2. ItwouldenablepredictionofAAReectivenessbasedontherealismoftheVHinteraction.IfVHinteractionrealismislow,itmaynotevenbeworthwhiletoconductanAARoftheinteraction.(Note:EvenifVHinteractionrealismislow,itmaystillbeworthconductingtheH-VHinteractionitself.ThebenetsoftheH-VHexperiencedonotcomefromtheAARalone.Therearebenetsinconductingtheexperienceitself[ 86 ].) 3. ItwouldenablemeasuringVHinteractionrealismindirectly.Realismisanabstract,subjectiveconceptwhichismorediculttomeasurethanAAReectivness.IfAAR 117

PAGE 118

Twocorrelationtestsshowtheeectivenessofreviewingrapport-buildingskillsarepositivelycorrelatedwithVHinteractionrealism(asmeasuredbythespeechunderstandingsuccessrate).Thus,ifVHinteractionrealismistoolow,itwilllikelybedicultforuserstogetfeedbackfromaninteractionwithavirtualhumanonthecrucialskillofrapport-building.Inthemedicalcontextofthisstudy,feedbackonrapport-buildingskillswereparticularlycrucialtothestudyparticipantsbecausebetterrapportwithpatientstypicallyleadstobetterhealthoutcomesforpatientsandlessmalpracticelawsuits.Goodrapportlikelyalsoprovidessimilarbenetsinotherinterpersonalscenarios.Thus,improvingVHinteractionrealismmayenableusersofVHexperiencestogeteectivefeedbackonrapport-buildingskills. TherstcorrelationtestwasmadebetweentheVHspeechsuccessrateRandthechangeinself-ratingsduetoAARPostAARPreAAR.Anegativecorrelationwasexpected,meaningparticipantswhoencounteramorerealisticVH(aVHthatspeaksmorelikearealhuman)wouldratethemselvespooreronrapport-buildingpost-AARthanthosewhosawalessrealisticVH.Anegativecorrelationwasfoundonameasureofrapportbuilding,whethertheparticipant\expressedsupportandpartnership"totheVH(Two-tailed,Pearson'sr=-0.382,p=0.049).Inotherwords,asVHinteractionrealismincreased,participantratingsofthemselvesonsupportandpartnershiptrendedlowerpost-AAR.ThismeansparticipantswhosawmorerealisticVHsweremorewillingtotrustthefeedbackprovidebytheAARontheiruseofrapport-building. ThesecondcorrelationtestwasmadebetweentheVHspeechaccuracyrateandtheabs(PostAARPreAAR),themagnitudeofthedierencebetweenparticipantself-ratingsbeforeandaftertheAAR.Apositivecorrelationwasexpected,meaningparticipantswhoencounteramorerealisticVHwouldchangetheirself-ratingsonrapport-buildingmorethanthosewhosawalessrealisticVH.Apositivecorrelationwasfoundontwomeasures 118

PAGE 119

Althoughunrelatedtorapport-building,acorrelationwasfoundthatindicatesfrustrationwiththevirtualhumanwasrelatedtothevirtualhuman'sinteractionrealism.AnegativecorrelationwasfoundbetweenVHspeechaccuracyandtheextenttowhichparticipantsthoughtheywereemotionalwheninteractingwiththeVH(r=-0.362,p=0.032).Inotherwords,astheinteractionrealismoftheVHincreased,participantstendedtonotchangeself-ratingsoftheiremotionalexpression.ThisislikelybecauseparticipantswhotalkedtolessrealisticVH'sweremorelikelytobefrustratedbytheVH,andthereforemorelikelytoexpressthatfrustrationemotionallywhentalkingtotheVH. Insummary,itappearsVHinteractionrealismiscorrelatedspecicallytohowoneevaluatestheirrapport-buildingwithavirtualhuman.ThelessrealistictheinteractionwithVH,thelesswillinguserswillbetochangeevaluationsoftheirrapport-buildingwithaVH.Ifweaimtoimproverapport-buildingskillsusingH-VHexperiencescombinedwithafter-actionreviews,thentheinteractionrealismofthevirtualhumanwillprovideagoodpredictorofthesuccessoftheAAR. 119

PAGE 120

Thegroupsweredeterminedbybinningspeechsuccessratesbystandarddeviationsfromthemeanspeechsuccessrate.Letribethespeechsuccessrateforparticipanti'sinteractionwiththeVH,mbethemeanspeechsuccessrate,andbethestandarddeviationofthespeechsuccessrate.Thenriisbinnedintooneofthefollowingfourspeechsuccessrategroups,wherenisthenumberofparticipantsineachgroup: TheANOVAshowsVHinteractionrealismaectedparticipantratingsofthemselvesonrapport(Figure 4-9 ).Thespeechsuccessrategrouphadasignicanteectforseveralmeasuresofrapport,includingasummarymeasurecomposedfrom15measuresofrapport.Furthermore,changesinself-ratingsweresmallerforthosewhointeractedwithalessrealisticVHandvice-versa.Thus,themorerealistictheinteractionpresentedbytheVH,themoreparticipantschangedtheirself-ratingsofrapport-building. AlthoughVHinteractionrealismaectedAAReectiveness,evenparticipantsonwhointeractedwithaVHwhodemonstratedlowinteractionrealismstillchangedtheirratingsduetoAAR(Section 4.7.1 ).Thisindicatesallparticipantsattributedsomevalidityortrusttotheafter-actionreview.OnequestionthisworkdoesnotansweriswhetherthereissomeminimumVHspeechaccuracybelowwhichuserscompletelydismisstheAARasinvalid.Clearly,lowinteractionrealismattenuatestheimpactoftheAARofanH-VHexperience.Theextentofthatattenuationeectisunansweredhere. 120

PAGE 121

Towhatextent(ifatall)canwechangeoraectpeopleusinganinteractionwithavirtualhuman?ThisworkarguesthatwecanchangepeoplebycombiningH-VHinteractionswithAAR.GuidelinesforAARforH-VHinteractionsandIPSViz,atoolthatgeneratesvisualizationsforAAR,werepresented.Inaddition,astudydemonstratedthatcombiningH-VHinteractionwithAAR,atleastintheshortterm,changesthewayauserseeshimself(self-identityandself-awareness).Moreimportantly,usercommentsindicatethischangemayextendintofutureinteractionswithrealhumans. OneissuethatemergedfromthisstudyisthattheimpactoftheAARcanbeattenuatedbylowVHbehavioralrealism.LowrealismledparticipantstobelievethatthebehaviortheyobservedduringtheAARdoesnotaccuratelyreecttheirbehaviorwithrealhumans.Otherkindsofrealism,suchasscenarioorvisualrealism,didnotappeartohaveaneectonparticipantperceptionspost-AAR.FuturestudiesmustexploretheeectofVHrealismclosertodetermineatwhatthresholdVHrealismcompletelynegatestheimpactoftheAAR. ThenextstepistolearnifusingIPSVizresultsinmeasurableimprovementinH-VHandeventuallyH-Hcommunicationskills.ThiswillbeevaluatedthroughrepeatedexposurestoVHsfollowedbyAARwithIPSViz.Thisexperience-feedbackloopwillenableconclusivelyevaluatingifstudentcommentsofAARchangingtheirperspectiveisarealizablegoal. 121

PAGE 122

IPSVizrepresentsanimportantprogressionintheapplicationofVEs.Traditionally,theimpactofavirtualexperiencehasbeenviewedasaproductoftheexperienceitself-theVEiswhatimpactstheuserandmakestheexperiencevaluable.IPSVizextendsthistoincludereviewandself-reection.Byenablingreviewandself-reection,theVHexperiencecontinuestoimpactthestudentbeyondtheconclusionoftheexperienceitself.Thisimpactisdierentthanthatprovidedbytheexperienceitself.WithAAR,usershaveanopportunityforself-reection,insight,andself-directedchangeforreal-worldsocialinteractions. 122

PAGE 123

ScreenshotofIPSViz.

PAGE 124

Userinteractswithavirtualhuman,thenreviewsinteractionwithIPSViz. 124

PAGE 125

H-VHinteractioniscaptured,ltered,processed,andvisualizedforreview,evaluationandfeedback. Figure4-4. ExamplestudentbodyleanthroughoutanH-VHinteraction Figure4-5. VisualizationofvetopicsignalsshowingwhenstudentandVHdiscussedimportanttopics. 125

PAGE 126

B User(woodenposingdoll),VHandvirtualenvironmentrenderedin3D. A )ReviewfromperspectiveofVH. B )Interactionaugmentedwithgazeinformation. Figure4-7. StudyProcedure Figure4-8. InteractionofVHskintoneandafter-actionreviewonshowinginterestintheVH. 126

PAGE 127

VHinteractionrealismversuschangeinself-ratedrapport-buildingscores 127

PAGE 128

ThischapterdescribestheVSPsystem,animmersiveafter-actionreview(AAR)systemforinterpersonalinteractionswithavirtualhuman(VH).VSPleveragesvirtualrealitytechniquestoenablerelivinganinteractionwithaVHfromtheVH'sperspective.ApilotstudyreviewstheimpactofVSPonusersofVHexperiences.ThisworkwillappearattheIEEEconferenceonVirtualReality2009[ 146 ]. 128

PAGE 129

42 90 101 ]. Weproposevirtualsocialperspective-taking(VSP).Virtualexperiencesareuniquelycapableoffacilitatingsocialperspective-takingbecausetheycanrenderanexperiencefromtheperspectiveofanotherperson.Renderingfromtheperspectiveofanotherpersonenablesrelivingtheexperienceofanotherperson:seeingwhatanothersaw,hearingwhatanotherheard,touchingwhatanothertouched,sayingwhatanothersaid,movingasanothermoved,and-throughnarrativeanddrama-feelingtheemotionsanotherfelt. Relivingtheexperienceofanotherpersonhasthegoalofhelpingusersreecton,understand,andlearnfromotherpeople'sexperiences.ThisgoalisrealizedbyusingVSPtotransportmedicalstudentsintothepotentiallyunfamiliarexperienceofbeingapatient.Thepatientisa34-year-oldCaucasianwomanwhoisafraidshehasbreastcancerandisundergoingaphysicalbreastexam.Relivingthispatient'sunfamiliarexperience-particularlyunfamiliarformalemedicalstudents-improvesstudentunderstandingofthepatient'sperspectiveandstudentbehaviorinfuturepatientinteractions. 129

PAGE 130

99 ],capableofinteractingwiththestudentthroughspeechandtouch,thusenablingthestudenttointerviewandexamineAmandamuchlikeareal-worlddoctor-patientinteraction.Camerasandlogsrecordtheinteractionfromthepatient'sperspectiveforVSP(Figure 5-2 ). Immediatelyfollowingthedoctor-patientinteraction,themedicalstudentparticipatesinVSPbyrelivingtherecordedinteractionasAmanda(Figure 5-1 ).ThestudentwearsanHMD,isseatedonaphysicalexambedwhereAmandasat,andhisbodyisreplacedwithAmanda'savatar.ThreebasicprinciplesareappliedtoallowtheusertorelivetheinteractionasAmanda. 1. Thestudent'ssensesarereplacedwithrecordingsofAmanda'ssenses.ThestudentseesandhearshimselfaskingquestionsandconductingaphysicalbreastexamfromAmanda'sperspective. 2. ThestudentisremindedthatheisAmanda.Whenlookingathisbodyorinavirtualmirror,thestudentseesAmanda'savatar. 3. ThestudentreenactsAmanda'sbehavior.TopromptthestudenttoreenactAmanda'sbehavior,thestudent'savatarisupdatedwithAmanda'smovements,andthetextofAmanda'sspeechisdisplayedintheHMD. 130

PAGE 131

5.2.1SocialVEsCanBenetfromVSP 79 176 ],culturalcompetency[ 10 40 ],medicaldiagnosis[ 89 ],anddealingwiththementallyill[ 59 ].Thesevirtualexperiencessimulatesocialinteractionsinwhichtheusercanbenetgreatlyfromengaginginsocialperspective-taking.socialperspective-takinghasbeenshowntoaidinresolvingconicts[ 42 ],promotingcooperation[ 90 ],andreducingbias[ 156 ].Associalperspective-takingsignicantlyimpactssocialinteractions,wehypothesizethatVSPcanimprovethebehaviorofusersinvirtualandrealsocialinteractions.Thispaperfocusesonvirtualsocialinteractions,evaluatingifVSPimprovesusers'empathicbehaviorinaninteractionwithavirtualhuman. 199 ]).Whengivenanavatarofapersondissimilartohimself,theusertakesonthebehaviorsheexpectsfromastereotypicalmemberofthisdissimilargroup(e.g.,talleravatarsresultinmorecondentbehavior;attractiveavatarsresultinmoreintimatebehavior[ 200 ]). Whilepreviousworkdemonstratesthatplacingapersoninthebodyofamemberofatargetgroupcausesthepersontobetteridentifywiththetargetgroup,VSPexperiencesgoonestepfurtherbyplacingapersoninthespecicexperienceofamemberofatargetgroup.InaVSPexperience,theuserbecomesthetarget,aspecicmemberofagroupofpeople,andrelivesaspecicexperienceinthetarget'slife.Relivingofaspecic 131

PAGE 132

5.3.1 ).ThisinteractionisrecordedfromAmanda'sperspective(Section 5.3.2 )sothatthestudentcanrelivetheinteractionasAmandaexperiencedit(Sections 5.3.3 5.3.4 5.3.5 ). 99 ].TheusabilityofaMRHpatientforpracticingandevaluatingmedicalstudents'physicalexamskillshasbeendemonstrated[ 100 ]. AMRHconsistsofalife-sizedvirtualhumanwhichisregisteredtoatangibleinterfacerepresentingthevirtualhuman'sbody.UsersareabletocommunicatewiththeMRHthroughnaturalspeechandtouchingofthetangibleinterface;theMRHcommunicatesthroughpre-recordedspeech,gestures(keyframe-basedanimations),gaze-behavior,andfacialexpressions. AMRHpatientwascreatedtoallowmedicalstudentstopracticebreasthistory-takingandclinicalbreastexams.ThetangibleinterfacetotheMRHpatientisamannequinandphysicalbreastmodelinstrumentedwith64sensorsthatdetectusertouch. 132

PAGE 133

99 ].Theuserwearsaheadmounteddisplay(HMD)toviewtheMRHandamicrophonetospeaktotheMRH.Theuser'sheadpositionistrackedusingtwoOptitrakNaturalPointcamerasandheadorientationistrackedbyanIntersenceInertiaCube2.Theuserisabletoseehistouchingofthetangibleinterfaceinthevirtualworld.Thevideostreamfromawebcammountedabovethemannequinisprocessedtoincorporatetheuser'shandsandthephysicalbreastintothevirtualscene.Anadditionalwebcamtracksaphysicalhospitalgownattachedtotheinstrumentedmannequin,allowingmanipulationofthegownasanadditionalinteractioninput.Asimulationmoduletakestheinputsofusertouch,gownmanipulation,anduserspeechandmatchestheseinputstoadatabaseoftheMRH'sverbalandgesturalresponses. 5-2 ).TwovideocamerasareneededbecauseAmandahastwoperspectivesduringtheinteraction,oneforwhensheissittinguponthebedandanotherforwhensheislyingdown.ThecamerasrecordthestudentasheinteractswithAmanda,includingaudio,sothatthestudentcanseeandhearhisconversationwithAmandaintheVSPexperience. 133

PAGE 134

5-4 ). WeproposeimmersingthemedicalstudentwhilereplayingtherecordingsoftheinteractiontoenablesensingwhatAmandasensedandreenactingAmanda'sbehavior.IntheVSP,thestudentseeswhatshesaw,hearswhatthesheheard,toucheswhatshetouched,andfeelsproprioceptivelywhatshefelt. 5.3.3.1Seeingandhearing 134

PAGE 135

5-3 ). HeadtrackinggivesthestudentcontroloverAmanda'sviewpoint.Thestudentcanlookaroundtheroomandathisavatar(Amanda),aswellasmovehisviewpointbymovinghishead.AsAmanda'sheadpositionwasmostlyxedintheoriginalinteraction,thestudentdoesnotneedtomovehisheadmuchintheVSPexperience.Thus,headpositionistrackedwithminimalinfrastructurethatstillprovidesthevisualeectsassociatedwithamovingviewpoint(e.g.,motionparallax). Headpositionistrackedin2Dusingasingleinfrared-viewingwebcamandasingleinfrared-reectivedotontheuser'sHMD.Thewebcamispositionedparalleltothestudent'sfacesothat2Dtrackingdatacanbemappedtoverticalandhorizontalviewpointmotion.HeadorientationistrackedusingtheZ800'sorientationsensor(33hzupdaterate). ToallowthestudenttoseehimselfasAmandadid,therecordedvideoofthestudentisprojectedontoavirtualplane.Thevirtualplane'spositionwasupdatedtoreecttheapproximatepositionofthestudentinthepatientinteraction.Thevideoisprojectedontheplanefromapproximatelythesameviewpointfromwhichitwasrecorded.AsthevideowasrecordedfromAmanda'sapproximateviewpoint,andthestudentwatchesthevideofromAmanda'sapproximateviewpointduringVSP,thevideoappearsreasonablyclosetowhatAmandasawduringthepatientinteraction.TherecordedaudiooftheinteractionisalsoplayedbackintheVSPexperience.Thisallowstheusertoevaluatehowbothhischoiceofwordsandnonverbalaspectsofspeech,liketoneofvoiceandprosody,wereperceivedbyAmanda. 135

PAGE 136

ThebedalsoprovidessubtleproprioceptivecuestomatchtheproprioceptivecuesAmandaexperienced.Whilesittingup,thestudent'slegshangothebed,andthebed'sheightprovidesthefeelingofbeinghigherupandlookingdownontherecordingvideoofthestudent.Whilelyingdown,thestudentfeelsthephysicalsensationoflyingdown(e.g.,gravityactingonthebodydierentlythanwhensitting,bloodowchangesinthebody,andarmsandlegspositionedlikeAmanda).Thisenablesthestudenttoexperiencethepatient'ssusceptiblepositionduringthephysicalexamportionoftheinteraction(Section 5.3.5.3 ). 165 ],actorsneedavatarswhenrehearsingaplayinaVE[ 151 ],andavatarappearancecanaectone'sbehavior[ 200 ].Thus,throughouttheVSPexperience,thestudentisremindedthatheisplayingAmanda'srole. ThestudentisgivenAmanda'savatar,whichhecanseeinplaceofhisownbodyintheHMD.AstheHMDhadalimitedeldofview,virtualmirrorsarealsoaddedtothescene.Amirrorontheceilingallowstheusertoseehisavatar(Amanda)whilelyingdown(Figure 5-4 ).Agreenscreenbehindthevideorecordingofthestudentistransformedintoamirror,allowingthestudenttoseehisavatarwhilesittingup(Figure 5-5 ).Thestudent'strackedheadmovementsarelinkedtohisavatar.Whenthestudentmoveshishead,heseeshisavatar'sheadmoveinthevirtualmirrors. 136

PAGE 137

199 ].Greenscreenshavebeenusedtocreateanaugmentedrealitymirrortoaugmentauser'sreectionwithcustomizedgarments[ 50 ]. Thegreenscreenistransformedintoavirtualmirrorintwopasses,amirrorpassandagreen-screenpass.Inthemirrorpass,theviewpointoftheuserisreectedaboutthemirrorplane,theplaneontowhichthegreenscreenvideoisprojected.Thesceneisthenrenderedfromthereectedviewpointintoamirrortexture.Aclippingplaneisplacedatthevideoplanetopreventgeometrybehinditfrombeingrenderedintothemirrortexture.Inthegreen-screenpass,greenscreentexelsarereplacedbytexelsfromthemirrortexture.Afragmentshaderrstsegmentsgreentexelsfromthevideousingasimplecolorsegmentationtechnique.Thetechniquesegmentstexelswhichconsistofmuchmoregreenthanredorblue(2GRB,whereisanexperimentally-determinedthreshold).Segmentedgreentexelsarereplacedwiththemirrortexture.Whilemoresophisticatedgreenscreen(chroma-key)techniquesexist,thissimpleapproachwassucientforthisscenarioandlightingconditions. Subtlecuesreinforcethenotionthatthegreenscreenmirrorisamirror.Whenthestudentmoveshishead,thestudentseeshisavatar'sheadmovinginthemirror,andthemirrorimagemovestoreectmotionparallax.Asthegreenscreenisrecordedbythecamerasduringthepatientinteraction,theparallaxcueisstrengthenedbythestudent'spartialocclusionofthegreenscreen.TheparallaxcueandheadmotionalsoreinforcethenotionthattheuserisincontrolofAmanda'sviewpointandavatar. 137

PAGE 138

5-6 ).ThestudentspeaksAmanda'slines,andthenahumanoperatorresumesplaybackofthemedicalstudent'spart.ThelogscontainthetextofAmanda'sspeechwithtimestamps,enablingpausingandresumingplayback,anddisplayingAmanda'slineswhenneeded. Duringthepatientinteraction,thestudentinstructedAmandatoperformthesemovements.Thus,duringtheVSPexperience,theplaybackoftheseinstructionsprompts 138

PAGE 139

VisualcuesalsoremindthestudenttomoveasAmandamoved.WhenoneofAmanda'smovementsisencounteredinthelogs,thestudent'savatarisautomaticallyupdatedtoreectthemovement.Thestudentcanobserveavatarmovementsinthevirtualmirrorstoseehowtomimicthem.Forthelyingdownmotion,anadditionalvisualreminderisprovided;thestudent'sviewpointisautomaticallyanimatedtothelyingdownposition. 104 141 ].Withvisualpseudo-hapticcues,thestudent'sbodyisnotactuallytouched.Instead,theuserseesthevisualimpactofthattouchonhisavatar'sbreast.Toenableseeingtheimpactofhistouch,therecordedvideoofthestudent'shandstouchingAmanda'sbreastsisprojectedontothestudent'savatar.DuetothelimitedeldofviewoftheHMD,thestudentcouldnotseetheprojectionwhilelyingdown.Instead,thevirtualmirrorontheceilingallowsthestudenttoseeareectionofhisavatarwithhisrecordedhandstouchinghisavatar'sbreast(Figure 5-4 ). 139

PAGE 140

Patientinteractionsweremonitoredbyoneoftheinvestigators,whotriggeredpatientresponsesifspeechrecognitionfailedrepeatedlytounderstandtheparticipants'speech,ortomovetheconversationforwardiftheparticipantaskedaboutatopic(e.g.,mentalhealth)forwhichthepatientdidnothaveresponses. 5.4.4 ). 5-7 .ParticipantscompletedapatientinteractionMRH1,followedbyaVSPexperiencetoreviewtheinteraction.AftertheVSPexperience,theparticipantcompletedasecondpatientinteractionMRH2toassessiftheVSPcausedtheparticipanttoengageinreectionandself-directedchangeofhisempathicbehavior. 140

PAGE 141

5.3 .ToencourageparticipantstoreenactAmanda'sactions,participantswereinstructedtomimicAmandaphysically(e.g.,\liedownwhenthepatientdoes")andverbally(e.g.,\readthepatient'slineswhentheyappearonscreen"). Aftertheparticipantconversedwiththepatientfor10minutes,theparticipantbeganaclinicalbreastexamofthepatient.Thebreastexambeganwiththeparticipantaskingthepatienttoremovehergown(bareherbreasts)inordertovisuallyinspectherbreastsforabnormalities.Whenthepatientwasaskedtoremovehergown,sheexpressedthatshewasnervousandshyaboutbeingnakedinfrontofthedoctor. 141

PAGE 142

Thethirdempatheticmomentoccurredastheparticipantsexplainedhisndingsandnextstepsinthepatient'streatment.Bothpatientshadamass,sothendingsoftheparticipantshouldbethatamasshadbeenfoundinthepatient'sbreast.ThenextstepintreatmentwasforMs.Jonestoreceiveamammogram(becauseshehadneverreceivedone)andforMs.Jacobstohaveabiopsyofthemass(becausesherecentlyhadamammogramwhichfailedtohighlightthemass).Eachpatientexpressedafearofthetreatment: Bothofthepatients'storiesoffeardemonstrateanintertwiningofincorrectmedicalknowledgewithcomplexpsychologicalissuesoffearandloss.Respondingtothesefearsappropriatelyrequiresunderstandingthepatient'sperspective.Theparticipantshouldhandlethissituationbybothempathizingandeducating,withoutdemeaningthepatientorherconcerns.Anyimprovementinempathyfromthersttosecondpatientinteractionisseenasbeingduetothesocialperspective-takingaordedbytheVSPexperienceadministeredbetweenthetwopatientinteractions. 142

PAGE 143

5.3.5 Aftertheseinstructions,theparticipantspent10minutesrelivingthepatientinteractionasAmanda.ToaccustomtheparticipanttotheVSPexperience,thestudentrelivedtherstminute(greeting,patient'scomplaint)ofMRH1.ThentheparticipantrelivedtheempatheticmomentsofMRH1.Ratherthanreliveeachempatheticmomentinisolation,theparticipantrelivedaminutelongperiodstarting30secondsbeforetheempatheticmoment.Relivingthisfullminuteprovidedconversationalcontextfortheempatheticmoment. 143

PAGE 144

5-1 )consistingoften5-pointLikertitemsadaptedfromempathysubscalesofvalidatedinstrumentsusedinmedicaleducation:theJeersonScaleofPhysicianEmpathy[ 81 ]andthe4-HabitsCodingScheme[ 101 ].Reliabilityofthisquestionnairewashigh(Cronbach's=0:73). Perspective-takingwasmeasuredwithaquestionnaire(Table 5-1 )offour5-pointLikertitemsalsoadaptedfrom[ 81 101 ].Theperspective-takingquestionnairehashighinternalconsistency(Cronbach's=0:91). 16 ].ThisinstrumentwasalsoadaptedtomeasurecopresenceintheVSPexperience.Specically,three5-pointLikertitemswereadapted:(1)Duringmytimeintheroleofthepatient,Imadeeyecontactwiththedoctor.(2)DuringmytimeintheroleofthepatientIfeltlikeIwastalkingtoanotherperson.(3)Ifeltthatthedoctorwasawareofmypresence. ReliabilityoftheVSPcopresenceinstrumentwashigh(Cronbach's=0:83).Enforcingthattheconstructthisinstrumentmeasuresisindeedcopresence,alargecorrelationwasfoundbetweenthisinstrumentandthecopresenceinstrumentusedinthepostpatient-interactionquestionnaire.Post-VSPcopresenceandPost-MRH1copresenceweresignicantlycorrelated(Pearson'sr(13)=0:65;p<0:05).Post-VSPcopresenceandPost-MRH2copresencewerealsosignicantlycorrelated(Pearson'sr(13)=0:81;p<0:0005). 144

PAGE 145

5.4.5.1VSPselicitreectionandself-directedchange ThereectionaordedbyVSPallowedparticipantstoengageinself-directedchangeoftheirbehaviorinthesecondpatientinteraction.Thisisevidencedbyaself-reportedimprovementinparticipants'useofempathyandperspectivetakinginthesecondpatientinteraction.Participantsratedthemselvesasimprovingsignicantlyintheiruseofempathy(Post-VSP:3.490.89;Post-MRH2:3.880.44;F(1,14)=7.0,p<0.05)andtrendedtowardsanimprovementinperspectivetaking(Post-VSP:3.670.61;Post-MRH2:3.950.45;F(1,14)=3.7,p=0.076).Theself-ratedimprovementinempathyandperspective-takingindicatesthatparticipantstriedtoimprovetheiruseofempathyandperspective-takinginthesecondpatientinteraction.Thisself-directedchangeisbroughtaboutbythereectionaordedbytakingthepatient'sperspectiveduringtheVSPexperience. 145

PAGE 146

Elevenof14participantsreportedmakingeyecontactwiththedoctor.Eyecontactisasocialprocessandimportantcomponentofnonverbalcommunication.Althoughtheparticipantcouldnotactuallymakeeyecontactwiththedoctor(thedoctor'sfacewascoveredbyanHMD),theresponsesindicateparticipantsweresociallypresentwiththedoctorandattemptedeyecontacttocommunicatenonverballyandactoutthesocialconventionofmakingeyecontactwhenspeakingandlisteningastheywouldinaconversationwithpatients. AsignicantmajorityofparticipantsfelttheyweretalkingtoanotherpersonduringtheVSPexperience,indicatingthatVSPsupportssocialinteractionthroughverbalcommunicationaswell.8of14participantsfeltthedoctorwasawareoftheirpresence. ParticipantscommentedonwhattheylearnedfromVSPandwhataspectoftheirpatientinteractionVSPmotivatedthemtoimprove.Theareasmostcommonlyidentiedasneedingimprovementwereaneedtobetterexpressempathy(3participants),aneedtodisplaymorecondence(3participants),andaneedtobetteraddresspatient'sconcerns,includingthepatient'sfears(4participants).Acommonrealizationofparticipantswasthattheylookdierentinapatientinterviewthantheythought,e.g.,\WhatIthoughtIlookedlikeininterviewingpatientsismuchdierentthanIactuallydo."ParticipantsalsocommentedhowtheVSPexperiencewouldchangetheirfutureinteractionswith 146

PAGE 147

5.4.6.1VSPelicitssocialprocesses VSPissuccessfulatfacilitatingsocialperspective-takingandself-directedchangepreciselybecauseitelicitssocialprocesses.Withoutsocialprocesses,suchascopresenceandconversationalinteraction,participantswouldhavedicultyunderstandingwhatitwaslikeforthepatienttointeractsociallywiththeparticipant. ThesocialprocesseselicitedbyVSPalsoshowthatparticipantsfeltimmersedintheexperience.DespitethearticialnatureofthepatientandVSPinteractions,participantswerewillingtoimmersethemselvessociallyintheVSPinteraction. 147

PAGE 148

147 ]bydemonstratingthatchangeinbehaviorwithVHscanbeelicitedthroughAARs. RelivingtheinteractionastheVHeectivelyallowedseeingwhatitisliketotalktooneself.Thisprovidedhealthcarestudentswithanimportanteducationalexperienceofreectingonhowtheyareperceivedbypatients,andonwhatitwasliketobethepatient.Thisledstudentstoself-identifyskillsinneedofimprovementandchangebehavior.Theseresponsesarediculttorealizewithcurrentapproaches,suchasgroupandinstructorvideoreview.ThusthisworkhighlightstheexpansionofVRtosupporttheeducationofsocialperspective-taking. WhileourresultsindicateVSPelicitedperspective-taking,wehypothesizemoreimpactfulperspective-takingcouldbeelicitedbyallowingtheusertothinkanddecidehowtoreenacttheVH'sbehavior.Currently,usersaretoldwhattosayandshownhowto 148

PAGE 149

WealsoproposeusingVSPtoallowthestudenttorelivetheVH'sinteractionwithanempathyexpert(e.g.,anoncologistwithidentiedexpertexperienceininteractingwithcancerpatients).Weexpectthistoimprovesocialperspective-takingbyallowingthestudenttocontrastapatient'sperspectiveoftheexpertwiththepatient'sperspectiveofhimself. Lastly,webelievethenextstepforVSPisenablingVSPofhuman-human(H-H)socialinteractions.ImplementingthethreeVSPguidelinesforH-Hinteractionswouldbechallenging.Ifsuccessful,userswouldbeabletoreliveandlearnfromtheexperiencesofthepeopletheyinteractwitheveryday. 149

PAGE 150

Ididnotpayattentiontothepatient'semotionswheninterviewingandexaminingher Ididnotfeelthatitwasimportantformetogainanunderstandingofhowthepatient feltabouthermother'sexperiencewithcancer IfeltthatImadeanemotionalconnectionwiththepatient Iencouragedthepatienttoexpressheremotions Iacceptedand/orvalidatedthepatient'sfeelings Idisplayedlittleinterestorconcerntothepatient Imadelittleornoattempttoexplorethepatient'sfeelings Ilegitimizedthepatient'sideasandfeelings Idemonstratedappropriatenon-verbalbehavior Itriedtoputmyselfinthepatient'spositiontohelpunderstandherfear Icouldnotunderstandwhythepatientwasfearful Itwasdicultformetoviewthingsfromthepatient'sperspective Iunderstoodthepatient'sperspective Table5-1. Empathy(top)andperspective-taking(bottom)questionnaires. 150

PAGE 151

Medicalstudentconverseswithvirtualpatient(Top),thenrelivesconversationasthepatient(Bottom). 151

PAGE 152

LoggingofpatientinteractionforVSP 152

PAGE 153

DuringVSP,thestudentseestheexamroom,hisavatar(Amanda'sbody),andvideoofhimselftalkingtoAmandafromAmanda'sperspective. 153

PAGE 154

IntheVSPexperience,thestudent,playingtheroleofAmanda,relivesthebreastexamheperformedonAmanda. 154

PAGE 155

Transformingthegreenscreenintoavirtualmirror. 155

PAGE 156

VHspeechloggedforlaterdisplayintheVSPexperience. 156

PAGE 157

Studyprocedure:borderedimagesindicaterole(patientordoctor)participantplayedateachstage. Figure5-8. Empathyandperspective-takingimprovedafterVSP,indicatingparticipantstriedtoimproveempathyandperspective-takinginthe2ndpatientinteraction. 157

PAGE 158

ThischapterdescribesIPSVizN,adesktop-based,after-actionreview(AAR)systemforgroupsofinteractionswithvirtualhumans(VHs).IPSVizNleveragesvisualizationtopresentoverviewsanddetailsofaggregateH-VHexperiencedata.Thevisualizationsallowuserstouncovertrendsandoutliersininterpersonalcommunicationbetweenhumansandvirtualhumans.ApreliminaryevaluationexploresthebenetsofconductingaggregateAARsofH-VHexperiences. 138 ].TheutilityofIPSVizNisdemonstratedbyusingittoanalyzeinterpersonalsimulations-interactionsbetweenhumansandvirtualhumansforinterpersonalskillstraining. 158

PAGE 159

Thistraditionalapproachhasseverallimitationsforevaluatinginterpersonalskills,including1)thetimeinvestmentrequiredforevaluation,2)thelackofavailabilityofinteractionparticipants,and3)the\hiddennature"ofimportantinterpersonalskills. 159

PAGE 160

89 145 ].Byextension,evaluatinginteractionswithavirtualhumanshouldprovideinsightintointeractionswithrealhumans. Inaddition,virtualhumanscanaddressthemultiplechallengesposedbyevaluatinginteractionswithrealhumans: 147 ]wasproposedtosupportevaluationofaninteractionwithavirtualhuman.IPSViztransformsthelogofaninteractionbetweenahumanandvirtualhumanintoasetofinteractivevisualizations. 160

PAGE 161

Inauserstudy,IPSVizusersreviewedtheirinteractionwithavirtualhumantolearnabouttheirreal-worldinterpersonalskills.AfterusingIPSViztoreviewtheirowninteractionwithavirtualhuman,participantsreectedonandchangedself-evaluationsoftheirinterpersonalskills.SeveralparticipantshighlightedwaystheywouldchangebehaviorinfutureinteractionswithrealhumansasaresultofthereviewwithIPSViz.Theskillshighlightedbystudentswereskillsthattheyhadnotconsideredpreviouslyintheirinteractionswithrealhumans.Thus,thestudyconrmsthebenetsgainedfromusinganinteractionwithavirtualhumanasthebasisforinterpersonalskillsfeedback:rapid,frequentreviewof\hidden"interpersonalskills(verbalandnonverbalcommunication)inwaysthatwouldbedicultorimpossibletodothroughinteractionswithrealhumans. 1. Discoveringtrendsandoutliers(deviationsfromtrends)ingroups 161

PAGE 162

Comparinganinteractionorgroupofinteractionstoanotherinteractionorgroupofinteractions Theseanalysistasksaredicultortime-consumingtoperformusingexistingmethodsforthefollowingreasons: Forexample,inatypical,10-week,medicalcommunicationskillscourse,180studentswouldconducteight10-minuteinterviewswithvirtualhumanpatients.Thatamountsto240hoursoflinearvideoreviewtotal,or24hoursofvideoreviewaweek,thattheinstructorwouldhavetoconducttogainaholisticsenseofwhomthegoodandbadstudentsare(outliers)andoveralltrendsinstudentperformance.Performanceinamedicalinterviewismeasuredintermsofthefollowingtechnical(e.g.diagnosis,treatment,andphysicalexamination)andinterpersonalmedicalskills[ 35 167 ]: 162

PAGE 163

Toproperlyevaluatetheclassof180studentsontheseinterpersonalskills,theinstructorneedstotrack,summarize,andcomparethestudents'andvirtualhumans'behaviorsacross240hoursofvideo.Thesebehaviorsinclude: IPSVizNsolvesthechallengesinherentintrackingandidentifyingtrendsandoutliersininterpersonalskillsacrosslargenumbersofH-VHinteractions.ThechallengesareaddressedbycapturingtheverbalandnonverbalbehaviorusedinH-VHinteractionsandusinginformationvisualizationtechniquestoprovideuserswithrapidinsightintothem. 1. 2. 163

PAGE 164

Toprovideaninitialevaluationofthisapproach,IPSVizNwasappliedtoasimpliedversionofthemedicaleducationscenariodescribedpreviously(evaluatingaclassofstudents).Interactionsbetween29medicalprofessionalsandavirtualhumancomplainingofbreastpainwerecaptured.Behaviorscapturedincludedthewordsspokenbytheuserandvirtualhuman,meta-topicswhichsummarizethesewords,andnonverbalbehavior(toneofvoice,posture,andfacialexpressions)fromvideooftheinteractions.Thesebehaviorshadspecicrelevancetothismedicalscenario,andallowedforanalysisoftrendsandoutliersininformationgathering,rapport,empathy,anddiagnosisandtreatment.UsingIPSVizN,visualizationsofthecapturedbehaviorswerethenprovidedtomedicaleducators.Byreviewingandinteractingwiththevisualizations,theeducatorsidentiedtrendsandoutliersinuserbehaviorthatwouldbedicultorimpossibletoidentifyotherwise. 6.3.1Motivation 89 ]forexploringtheunderlyingquestionsposedbythischapter: 89 ].Furthermore,interactionswithVHpatientsaresimilartointeractionswithsimulatedpatients(humanactors).Lastly,workapplyingafter-actionreviewstoH-VHexperiences[ 146 147 ]hasfoundthatusercanself-evaluatetheirinterpersonalskillsbyreviewingtheirinteractionwithavirtualhumanpatient.Theseresultsprovideevidenceandmotivationforevaluatinggroupinterpersonalskillsthroughinteractionswithvirtualhumans. 164

PAGE 165

161 ],provideguidelinesforgaininginsightintolargedatasetsthroughinformationvisualization.Inaddition,severalcompellingvisualizationsofinterpersonalcommunicationhavebeendeveloped[ 5 20 22 46 93 160 186 187 191 ].Lastly,previousworkhasshownvisualizationsofindividualinteractionswithVHpatientscanbevisualizedtoprovideinsightintointerpersonalskills[ 147 ].Thesevisualizationsprovidedinsightthatweredicultorimpossibletoreceivethroughpracticeinteractionswithrealhumans. 100 ].Afterthephysicalexam,theMPdiscussedherndingsandnextstepswithAmanda.Atseveralpointsthroughouttheinteraction,AmandachallengedtheMPbyaskingdicultquestionsormakingtroublingstatements,e.g.,\Doyouthinkthiscouldbecancer?"or\Idon'tknow,doIhavetogetamammogram?"Thepurposeofthesechallenges 165

PAGE 166

TheMP'sverbalandnonverbalinteractionwithAmandaisfacilitatedusingsensors.MicrophonesandspeechrecognitioncapturetheMP'sspeech.ThetextofthespeechisusedtosearchadatabaseofresponsesthatAmandacanvocallyandgesturallyrespondwith.Whenanappropriateresponseisfound,thecorrespondingaudioisplayedwithappropriatelip-synching.TheMP'sheadistracked,allowingAmandatolookattheMP.TheMPcantouchAmandathroughaphysicalcorrelate,amannequinoutttedwithforcesensorsandthebreastsimulator.AllverbalandnonverbalinteractionwasloggedforlateraggregatevisualizationinIPSVizN. Thebreast-cancerscenarioprovidesopportunitiesformedicalprofessionalstodemonstrateavarietyofmedicalcommunicationskills,includingcompletenessandorganizationofdiscussion,empathy,rapport-building,andreactionsunderpressure.TheseopportunitiesarealsologgedbythesystemsothattheycanbereviewedinIPSVizN. 166

PAGE 167

167

PAGE 168

Thespatialorganizationorlayoutofthetopicdotsvisuallyaggregatesthedatasuchthatusersrequirelittlecognitiveeorttoidentifyandcomparetrendsandoutliersinverbalbehavior.ThreetypesoflayoutsareavailableinIPSVizN,eachofwhichallowsuserstoidentifydierentrelationshipsintheinteractions'verbalcommunication: 6-1 )laysouttopicdotsbyinteraction(verticalaxis)andtime(horizontalaxis).Eachhorizontallineoftheplotrepresentsthetopicuseinaspecicinteractionwithavirtualhuman.Theverticalalignmentofeachinteraction'stopicuseallowsidenticationoftrendsandoutliersininteractionthoroughness,interactionorganization,andlengthofinteraction,aswellascomparisonoftopicuseindierentinteractions. 6-2 )laysouttopicdotsbytopic(verticalaxis)andtime(horizontalaxis).EachhorizontallineoftheplotrepresentstheuseofaspecictopicacrossallH-VHinteractions.Thisplotshowsgrosstrendsandoutliersintopicuse(thoroughnessandorganization),independentofindividualinteractions. 6-3 laysouttopicdotsbyfrequencycount(verticalaxis)andtopic(horizontalaxis).Theheightofeachverticallineoftheplotindicatestherelativefrequencyofthattopic'suseamongtheH-VHinteractions,allowingquickidenticationofbothdominantandrarelydiscussedtopics. 168

PAGE 169

6-4 providesanexampleoflteringbytopicandsubgroup. Theuserinterfaceforlteringconsistsoftwomenus,alegendoftopicsandalistofsubgroupswithinthedata.Thetopiclegend,displayedjustabovetheplots,bothdenesthemappingofcoloreddotstotopicsandalsoservesasalteringmenu.Selectingatopicnamefromthelegendfadesdotsnotcorrespondingtotheselectedtopicoutofview.Likewise,theinteractiongrouplist,displayedtotherightoftheplots,bothdenesthesubgroupsinthedatasetandalsoservesasalteringmenu.Selectingagroupnamefromthelistfadesdotsnotcorrespondingtotheselectedgroupoutofview. 169

PAGE 170

6-1 ,onecanseethattheshortestinteractiontookabout3.5minutesandthelongestabout18minutes. 6-5 )scaleaddressesthisproblembyscalingeachinteraction'stimedimensionsuchthatallinteractionsappeartohaveequivalenttimelengths.Thiseectivelystretchesshorterinteractionsspatiallytoenableeasieridenticationandcomparisonoftopicdiscussionamongallinteractions. Figure 6-6 demonstratesamedical-domain-orienteduseofselectionqueries.Aneducatorqueriesasetofhistoryofpresentillness(HPI)dotsneartheendoftheinterviews.Inthemedicaldomain,thedotsrepresentoutliersbecausemedicalprofessionalsshoulddiscussHPIwiththepatientearlier.DiscussingHPIlaterintheinterviewisasignofconductingadisorganizedmedicalinterview.Thequeryreturnsthelistofpotentiallydisorganizeduserstotheeducator,andtheeducatorcanusethislisttoinvestigatethepotentiallydisorganizedusersfurther. 170

PAGE 171

138 ],changingtherepresentationofthedataasitllsmorescreenspace,isappliedheretohelptheuserreviewspecicmomentsintheinteraction. Toenabletargetedreviewofspecicmoments,videoisassociatedwitheachtopicdot(Figure 6-7 ).Asthevideoandtopicdataissynchronizedintimeduringcapture,eachtopicdotislinkedtoitscorrespondingmomentinitsassociatedvideo.Double-clickingonadotsemanticallyzoomsintothedotanddisplaysavideowindow.Videoplaybackstartsattheselectedmoment.Clickingagainstopsplaybackandhidesthevideowindow.Thus,theusercaneasilyselectandreviewspecicmomentsofinterestfordetailedreview. Theparticipantswerecomputerscientists(9),medicaleducators(4),andamedicalstudent.TheyeitherwatchedapresentationandlivedemoofIPSVizNorweresentaweblinktotryIPSVizNontheirowntime. 171

PAGE 172

1. Inwhatorderdiduserstendtoconducttheirinterview?Doesanysubgrouptendtoconducttheirinterviewinadierentorderthantheentiregroup? 2. Whatarethedierencesbetweenstudentsandclinicians?Isonegroupmoreorganized?Isonegroupmorethoroughandcomplete? 3. Whatcanyouidentifywithrespecttoindividualstudents(e.g.,beststudent?worststudent?studentwhospentthemosttime?etc.) 4. Whatcanyouidentifywithrespecttosubgroups? 5. Usetheplotterfreely.Ifyoumakeanynewobservations,pleasewritethemhere. Thefeedbackportionofthequestionnaireasked: 1. Abouthowlongdidittakeyoutoanswerthequestionsabove? 2. Whichquestionswerediculttoanswer?Whichtookthelongesttoanswer?Why? 3. Wasthereanythingyouwantedtodothatyoucouldn'tdo? 4. Howwouldyouimprovethetopicplotter? 5. Ifyouhaveanyotherfeedback,pleasewriteithere. 172

PAGE 173

Participantscomparedtrendsacrossgroups.Theynotedthatcliniciansdiscussedfamilyhistorylessandfocusedmostlyonthehistoryofpresentillness.Students,ontheotherhand,stucktothestandardorganizationofthemedicalinterview,proceedingfromtopictotopicinthespeciedorder.Someparticipantsexpectedthisdierencebetweencliniciansandstudentsbutdidnothaveanypriorevidencetoconrmit.Clinicians,theytoldus,havetheexperiencetoknowexactlywhatquestionstoaskthepatient,whereasinexperiencedstudentsneedtoconductorganizedinterviewstomakesuretheygetalltheinformationneededtodiagnosethepatient.Thus,IPSVizNallowedparticipantstoconrmahypothesisaboutgrouptrends,andparticipantsindicatedthistrendwiththevirtualhumanreectsatrendwithhumansaswell. ReviewwithIPSVizNencouragedgenerationandtestingofnewhypothesesabouttheinteractionsaswell.Oneparticipanthypothesizedarelationshipbetweendiscussionofsocialhistoryanduseofempathy.AfterusingIPSVizN,theparticipantdeterminednosuchrelationshipexists. 173

PAGE 174

Althoughshedidnotdiscussanyonespecicindividual,oneparticipantnotedIPVSizNcouldhighlightconcernsaboutanindividual.WithIPSVizN,userscouldidentify\thosewhonevershowempathy,thosewhoaretoobrief,toolong,[and]thosewhonevermentionmammograms(nextsteps).TheseusersmaynotrepresentthebestorworstMPs,buttheymightbeforgettingoneimportantdetailintheirinteractionswithpatients.Furthermore,thesemistakescouldbeasignofalargerproblemwithaperson'sinteractionwithpatientsthatshouldbestudiedfurtherwithothertools. Onereasonoutlierswerediculttondwasthattheinterfacedidnothighlightthetoolsforidentifyingoutliers.Oneofthemaintoolsforidentifyingoutliers,theselectionqueries,requiredclicking-and-dragging.Althoughinstructionswereprovided,unlesstheusertrieditexplicitly,theymaynothaveknownitevenexisted.Addingaselectionquerymodewhichisactivatedbyabuttonmaycallmoreattentiontothisfeature. Additionally,outliersmaynothavebeendramaticenoughtostandoutfromthelargetrendsintopicdiscussion.Replacingrowsoftopicdotsthatrepresenttrendswithasingleclusterdotmightallownon-trenddata(outliers)tostandoutmorefromthedataset. 174

PAGE 175

Indeed,despitetheerrorsinthetopiclabels,participantswereabletoidentifytrendsinthedata,andtheyappearedtotrustthesetrendswererealratherthanaresultoferror.Participantstrustedthetrendsbecausethetrendsttheirexpectedmodelofwhathappensinapatient-doctorinterview.Forexample,theparticipantssaidcleartrendsinthediscussionofhistoryofpresentillnesswerethesameasthatexpectedinasimilarinteractionwitharealpatient.Thus,theinformationprovidedbyIPSVizNseemedtrustworthy.Overall,theerrorrateaectsuserobservations,butnotenoughtooutweighthebenetsofthesystem. Nonetheless,futureversionsshouldincorporatesometransparencytotheprocessoflabelingthedata.Addingvideoplaybacktotheinterfacepartiallyaddressesthisissue,asuserscanwatchvideoassociatedwithadottoseeifthedotislabeledwiththecorrecttopic.Historicaltrendsintheerrorratescouldalsobedisplayedintheinterface.Inthelongterm,relabelingtoolscouldbeaddedtotheinterfacesothatuserscouldrelabeltopicdotswhenvideoreviewrevealstheyarelabeledincorrectly. 175

PAGE 176

176

PAGE 177

AnevaluationofIPSVizNwithrepresentativeend-usershighlightedseveralusecasesforanalyzinggroupsofH-VHinteractions.Theseincludeidentifyingunusualorinterestingcaseswithinagroup(outliers),identifyingcommoncaseswithinagroup(trends),comparingtrendsandoutliersbetweenindividualsand/orsubgroups,andgeneratingandconrminghypothesesaboutH-VHinteractions. TheevaluationalsoshowsonecangeneratevisualizationsofrecordedH-VHinteractionsthatprovideinsightsthatwouldbedicultorimpossibletolearnfromreal-worldinteractions.End-userswereabletorapidly(withinminutes)identifytrendsinoverallgroupinterpersonalskills,includingverbalbehavior,organization,completeness,empathy,andcommunicatingunderstress.IdentifyingthesetrendswithoutIPSVizNwouldhaverequiredhoursofmanualeort,includingvideoreview,transcription,andmeta-analysisoftranscripts(topiclabeling). Lastly,theevaluationindicatesonecanevaluateagroup'sreal-worldinterpersonalskillsbyevaluatingthatgroup'sinteractionswithvirtualhumans.End-usersidentiedtrendsininterpersonalbehaviorwithavirtualhuman,andcomparedtrendsamongsubgroupsofinteractions(e.g.,cliniciansvs.students).Someobservedtrendstend-userexpectationsofinterpersonalbehaviorinasimilarreal-worldinteraction,providingstrongevidencethatIPSVizNdisplaysreal-worldinterpersonalskillstoend-users. ThenextstepistointegratevirtualhumanpatientexperiencesandIPSVizNintoamedicalcommunicationcurriculum.Thiswillallowstudyinghowend-users(medicaleducators,students,andVHresearchers)useIPSVizNinamorerealisticscenariowheremuchlargergroupsofinteractionswillneedtobeanalyzed.Inthisscenario,NwillgrowexponentiallyasstudentsperformmoreandmoreinteractionswithVHpatientsoverthe 177

PAGE 178

178

PAGE 179

Interactiontimelineshowsuseoftopicsovertimewithrespecttoeachinteraction. 179

PAGE 180

Topictimelineshowsuseoftopicsovertimeforallusers. 180

PAGE 181

Topichistogramshowsfrequencyoftopicuseforaselectedgroupofusers. 181

PAGE 182

B C Interactiontimelinevisualization(Figure 6-1 withaseriesofltersapplied):A)Discussionofhistoryofpresentillness(hpi)only.B)Reviewinghpi,butfurtherlteringtoonlyreviewtheclinicianssubgroup.C)Reviewinghpionlyforthestudentssubgroup. 182

PAGE 183

Useoftopicsonanormalizedtimeline 183

PAGE 184

Queryingtimespansbyselection 184

PAGE 185

B Reviewingvideoofpatientchallenges.A)Interactiontimelinelteredbypatientchallenges.B)Uponselectingchallenge,correspondingvideoappearsforreview. 185

PAGE 186

Weproposedafter-actionreviews(AARs)forhuman-virtualhuman(H-VH)interactions.Threeproof-of-conceptAARsystemswerepresented:theInterpersonalScenarioVisualizer(IPSViz),theVSPsystem,andIPSVizN.Thesethreesystemsuseinformationvisualizationandimmersivevirtualrealitytechniquestoprovideinsightintodomain-orientedinterpersonalskills.AseriesofuserstudiesfoundthatreviewingH-VHinteractionswithIPSViz,theVSPsystem,andIPSVizNelicitsreectionon,changesperceptionsof,andmotivateschangeindomain-specicinterpersonalskills. 186

PAGE 187

Byproposingafter-actionreviewsforhuman-virtualhumanexperiences,thisdissertationexpandsthecapabilitiesofhuman-virtualhumanexperiences.Addingafter-actionreviewstohuman-virtualhumanexperienceselicitsreectionon,changesperceptionsof,andmotivateschangeindomain-specicinterpersonalskills. AlsoimportantisdeterminingiftheAARschangeuserbehaviorindesiredways.ThisdissertationshowsthatAARsimpactpeople,butitisnotcleariftheAARsimpactpeopletomeetdesiredgoals.Thenextstepistoexploretheparameter-spaceofAARstoidentifyhowtoimpactspecicskillsthroughAARofH-VHinteractions.OnepossibleapproachwouldbetoincorporateexpertknowledgeintotheAARsystem. 187

PAGE 188

Lastly,thesuccessofvisualizationandimmersivevirtualrealitytechniquesinAARsofhuman-virtualhumaninteractionsmotivatestheiruseforAARsofhuman-humaninteractions.ItissignicantlyeasiertocreatesuchAARsforH-VHinteractionsbecausemuchoftheinteraction(thevirtualhumanandtheenvironment)iscomputergenerated.WithH-Hinteractions,allaspectsoftheinteractionarereal,signicantlyincreasingthechallengesofcaptureandprocessing.Ifthesechallengescanbeovercome,thenwewouldbeabletoprovidepeoplewithtoolstoreview,analyze,andlearnfromtheireverydaylives. 188

PAGE 189

[1] W.AcevedoandP.Masuoka.Time-seriesanimationtechniquesforvisualizingurbangrowth.ComputersandGeosciences,23(4):423{435,1997. [2] W.Aigner,S.Miksch,W.Muller,H.Schumann,andC.Tominski.Visualizingtime-orienteddataAsystematicview.Computers&Graphics,31(3):401{409,2007. [3] W.Aigner,S.Miksch,W.Muller,H.Schumann,andC.Tominski.VisualMethodsforAnalyzingTime-OrientedData.IEEETRANSACTIONSONVISUALIZATIONANDCOMPUTERGRAPHICS,pages47{60,2008. [4] W.Aigner,S.Miksch,B.Thurnher,andS.Bi.PlanningLines:NovelGlyphsforRepresentingTemporalUncertaintiesandtheirEvaluation.InTheNinthInternationalConferenceonInformationVisualization,pages457{463,2005. [5] F.S.Alallah,M.Nezhadasl,P.Irani,andD.Jin.Visualizingthedecision-makingprocessinaface-to-facemeeting.In11thInternationalConferenceInformationVisualization(IV'07),pages168{176,2007. [6] N.E.AlessiandM.P.Huang.Evolutionofthevirtualhuman:Fromtermtopotentialapplicationinpsychiatry.CyberPsychology&Behavior,3(3):321{326,2000. [7] N.Andrienko,G.Andrienko,andP.Gatalsky.Exploratoryspatio-temporalvisualization:ananalyticalreview.JournalofVisualLanguagesandComputing,14(6):503{541,2003. [8] M.ArgyleandM.Cook.GazeandMutualGaze.CambridgeUniversityPress,UK,1976. [9] J.AstingtonandD.Olson.MetacognitiveandMetalinguisticLanguage:LearningtoTalkaboutThought.AppliedPsychology,39(1):77{87,1990. [10] S.Babu,T.Barnes,andL.Hodges.Canimmersivevirtualhumansteachsocialconversationalprotocols?InIEEEVirtualReality2007,pages215{218,Charlotte,NC,March2007. [11] K.Bachour,F.Kaplan,andP.Dillenbourg.Reect:Aninteractivetableforregulatingface-to-facecollaborativelearning.InProceedingsoftheEuropeanConferenceonTechnologyEnhancedLearning2008,2008. [12] N.I.Badler,C.A.Erignac,andY.Liu.Virtualhumansforvalidatingmaintenanceprocedures.Comm.oftheACM,45(7):56{63,2002. [13] N.I.Badler,C.B.Phillips,andB.L.Webber.SimulatingHumans:ComputerGraphics,Animation,andControl.OxfordUniv.Press,NY,1993. 189

PAGE 190

J.N.Bailenson,E.Aharoni,A.C.Beall,R.E.Guadagno,A.Dimov,andJ.Blascovich.Comparingbehavioralandself-reportmeasuresofembodiedagents'socialpresenceinimmersivevirtualenvironments.InThe7thAnnualInternationalWorkshoponPresence,Valencia,Spain,2004. [15] J.N.Bailenson,J.Blascovich,A.C.Beall,andJ.M.Loomis.Equilibriumtheoryrevisited:Mutualgazeandpersonalspaceinvirtualenvironments.PRESENCE:TeleoperatorsandVirtualEnvironments,10(6):583{598,2001. [16] J.N.Bailenson,K.Swinth,C.Hoyt,S.Persky,A.Dimov,andJ.Blascovich.Theindependentandinteractiveeectsofembodied-agentappearanceandbehavioronself-report,cognitive,andbehavioralmarkersofcopresenceinimmersivevirtualenvironments.Presence:Teleoper.VirtualEnviron.,14(4):379{393,2005. [17] L.Baird,J.Henderson,andS.Watts.LearningfromAction:AnAnalysisoftheCenterforArmyLessonsLearned(CALL).HumanResourceManagement,36(4):385{395,1997. [18] L.Baird,P.Holland,andS.Deacon.Learningfromaction:Embeddingmorelearningintotheperformancefastenoughtomakeadierence.OrganizationalDynamics,27(4):19{32,1999. [19] S.Balcisoy,R.Torre,M.Ponder,P.Fua,andD.Thalmann.Augmentedrealityforrealandvirtualhumans.InComputerGraphicsInternational,pages303{308,2000. [20] D.BauerandJ.Hollan.IRYS:avisualizationtoolfortemporalanalysisofmultimodalinteraction.InProceedingsofthe5thInternationalConferenceonMultimodalInterfaces,pages285{288,2003. [21] B.Bederson,A.Clamage,M.Czerwinski,andG.Robertson.DateLens:AsheyecalendarinterfaceforPDAs.ACMTransactionsonComputer-HumanInteraction(TOCHI),11(1):90{119,2004. [22] T.BergstromandK.Karahalios.SeeingMore:VisualizingAudioCues.ProceedingsofINTERACT2007,2007. [23] T.BergstromandK.Karahalios.Conversationclusters:human-computerdialogfortopicextraction.InCHI'08ExtendedAbstractsonHumanfactorsinComputingSystems,pages2829{2834,2008. [24] M.Billger.Colourappearanceinvirtualreality:Acomparisonbetweenafull-scaleroomandavirtualrealitysimulation.InThe9thCongressoftheInternationalColourAssociation,2001. [25] J.Blascovich.Atheoreticalmodelofsocialinuenceforincreasingtheutilityofcollaborativevirtualenvironments.InCVE'02:Proceedingsofthe4thInternationalConferenceonCollaborativeVirtualEnvironments,pages25{30,NewYork,NY,USA,2002.ACM. 190

PAGE 191

A.Boucouvalas.RealTimeText-to-EmotionEngineforExpressiveInternetCommunications.EmergingCommunication,5:305{320,2003. [27] E.BoydandA.Fales.ReectiveLearning:KeytoLearningfromExperience.JournalofHumanisticPsychology,23(2):99,1983. [28] P.Buono,A.Aris,C.Plaisant,A.Khella,andB.Shneiderman.Interactivepatternsearchintimeseries.InProc.SPIE,volume5669,pages175{186,2005. [29] L.ByronandM.Wattenberg.StackedGraphs{Geometry&Aesthetics.IEEETransactionsonVisualizationandComputerGraphics,14(6):1245{1252,2008. [30] J.V.CarlisandJ.A.Konstan.Interactivevisualizationofserialperiodicdata.InUIST'98:Proceedingsofthe11thannualACMsymposiumonUserinterfacesoftwareandtechnology,pages29{38,NewYork,NY,USA,1998.ACM. [31] S.Carroll,S.Beyerlein,M.Ford,andD.Apple.TheLearningAssessmentJournalasatoolforstructuredreectioninprocesseducation.InProceedingsofFrontiersinEducationConference(FIE'96),volume1,1996. [32] J.Cassell,Y.Nakano,T.Bickmore,C.Sidner,andC.Rich.Non-verbalcuesfordiscoursestructure.InProceedingsofthe39thAnnualMeetingonAssociationforComputationalLinguistics,pages114{123.AssociationforComputationalLinguisticsMorristown,NJ,USA,2001. [33] J.Cassell,J.Sullivan,S.Prevost,andE.Churchill,editors.EmbodiedConversationalAgents.MITPress,Cambridge,MA,2000. [34] G.Collier.EmotionalExpression.LawrenceErlbaumAssociates,London,1985. [35] J.L.CoulehanandM.R.Block.TheMedicalInterview:MasteringSkillsforClinicalPractice.F.A.DavisCo.,Philadelphia,3rdedition,1997. [36] S.Craig,B.Gholson,andD.Driscoll.Animatedpedagogicalagentsinmultimediaeducationalenvironments:Eectsofagentproperties,picturefeatures,andredundancy.JournalofEducationalPsychology,94(2):428{434,2002. [37] C.Daassi,L.Nigay,M.Fauvet,L.CLIPS-IMAG,andF.Grenoble.Ataxonomyoftemporaldatavisualizationtechniques.Information-Interaction-Intelligence,5(2):41,2006. [38] M.DarlingandC.Parry.After-ActionReviews:LinkingReectionandPlanninginaLearningPractice.Reections:TheSoLJournal,3(2):64{72,2001. [39] D.Davis,P.Mazmanian,M.Fordis,R.VanHarrison,K.Thorpe,andL.Perrier.AccuracyofPhysicianSelf-assessmentComparedWithObservedMeasuresofCompetence:ASystematicReview.JAMA,296(9):1094,2006. 191

PAGE 192

E.Deaton,C.Barba,T.Santarelli,L.Rosenzweig,V.Souders,C.McCollum,J.Seip,W.Knerr,andJ.Singer.Virtualenvironmentculturaltrainingforoperationalreadiness(vector).VirtualReality,8(3):156{167,2005. [41] DepartmentoftheArmy,Washington,DC.TrainingCircular25-20:ALeader'sGuideToAfter-ActionReviews,September1993. [42] M.DeutschandE.NationalAssociationforMediationin.Educatingforapeacefulworld.AmericanPsychologist,48:510{510,1993. [43] R.Dickerson,K.Johnsen,A.Raij,B.Lok,T.Bernard,A.Stevens,andD.S.Lind.Virtualpatients:Assessmentofsynthesizedversusrecordedspeech.InMedicineMeetsVirtualReality14,LongBeach,CA,2006. [44] R.Dickerson,K.Johnsen,A.Raij,B.Lok,J.Hernandez,andA.Stevens.Evaluatingascript-basedapproachforsimulatingpatient-doctorinteraction.InTheInternationalConferenceonHuman-ComputerInterfaceAdvancesforModelingandSimulation,NewOrleans,LA,2005. [45] S.DiGangi.Self-GraphingofOn-TaskBehavior:EnhancingtheReactiveEectsofSelf-MonitoringonOn-TaskBehaviorandAcademicPerformance.LearningDisabilityQuarterly,14(3):221{30,1991. [46] J.DiMicco,K.Hollenbach,A.Pandolfo,andW.Bender.TheImpactofIncreasedAwarenessWhileFace-to-Face.Human-ComputerInteraction,22(1-2):47{96,2007. [47] J.Dovidio,K.Kawakami,andS.Gaertner.Implicitandexplicitprejudiceandinterracialinteraction.JournalofPersonalityandSocialPsychology,82(1):62{68,2002. [48] F.DuyandE.Holmboe.Self-assessmentinLifelongLearningandImprovingPerformanceinPractice:PhysicianKnowThyself.JAMA,296(9):1137,2006. [49] E.Chi,J.Pitkow,J.Mackinlay,P.Pirolli,R.Gossweiler,SKCard.VisualizingtheEvolutionofWebEcologies.InConferenceonHumanFactorsinComputingSystems(CHI98),pages400{407,1998. [50] P.Eisert,J.Rurainsky,andP.Fechteler.Virtualmirror:Real-timetrackingofshoesinaugmentedrealityenvironments.IEEEInternationalConferenceonImageProcessing2007,2:557{560,2007. [51] P.Emmelkamp,M.Krijn,A.Hulsbosch,S.deVries,M.Schuemie,andC.vanderMast.Virtualrealitytreatmentversusexposureinvivo:acomparativeevaluationinacrophobia.BehaviourResearchandTherapy,40:509{516,2002. [52] T.Erickson,D.Smith,W.Kellogg,M.La,J.Richards,andE.Bradner.SociallyTranslucentSystems:SocialProxies,persistentconversation,andthedesignofbabble.InProceedingsoftheSIGCHIConferenceonHumanFactorsinComputingSystems:TheCHIistheLimit,pages72{79,1999. 192

PAGE 193

J.Fails,A.Karlson,L.Shahamat,andB.Shneiderman.AVisualInterfaceforMultivariateTemporalData:FindingPatternsofEventsoverTime.InProceedingsofIEEESymposiumonVisualAnalyticsScienceandTechnology(VAST'06),2006. [54] FifthDimensionTechnologies.FlightVisualizationSystem,August2008.FifthDimensionTechnologies. [55] J.Flavell.MetacognitionandCognitiveMonitoring:ANewAreaofCognitive-DevelopmentalInquiry.AmericanPsychologist,34(10):906{11,1979. [56] R.Ford,V.Lamar,R.Giuli,andS.Oberg.TheJointTrainingExperimentationProgramApproachtoaDistributedAfterActionReview(DAAR).04E-SIW-063,SimulationInteroperabilityWorkshop,2004. [57] R.Ford,J.Shockley,M.Beebe,M.Faust,G.Lucha,M.Johnson,andJ.Bernatz.TheJointTrainingExperimentationProgram:LessonsLearnedfromtheFirstDemonstration.SimulationInteroperabilityWorkshop,2003. [58] Forterra.On-lineinteractivevirtualenvironment(olive):Amulti-user,virtualworldsoftwareplatform.Technicalreport,ForterraSystems,December2006. [59] G.Frank,C.Guinn,R.Hubal,P.Pope,M.Stanford,andD.Lamm-Weisel.Just-talk:Anapplicationofresponsivevirtualhumantechnology.InInProc.oftheInterservice/IndustryTraining,SimulationandEducationConference,Orlando,FL,December2002. [60] D.Friedman,A.Brogni,A.Antley,C.Guger,A.Steed,andM.Slater.Standardizingdataanalysisinpresenceexperiments.Presence:TeleoperatorsandVirtualEnvironments,15(5):599{610,October2006. [61] G.Furnas.Generalizedsheyeviews.InProceedingsoftheSIGCHIconferenceonHumanfactorsincomputingsystems,pages16{23.ACMNewYork,NY,USA,1986. [62] G.Furnas.TheFISHEYEView:ANewLookatStructuredFiles.ReadingsinInformationVisualization:UsingVisiontoThink,pages312{330,1999. [63] A.Gande.InstructorStationforvirtuallaparoscopicsurgery.PhDthesis,Mastersthesis,UniversityofTexasatArlington,August2003. [64] M.Garau,M.Slater,S.Bee,andM.A.Sasse.Theimpactofeyegazeoncommunicationusinghumanoidavatars.InTheSIGCHIConferenceonHumanFactorsinComputingSystems,Seattle,WA,2001. [65] M.Garau,M.Slater,V.Vinayagamoorthy,A.Brogni,A.Steed,andM.Sasse.Theimpactofavatarrealismandeyegazecontrolonperceivedqualityofcommunicationinasharedimmersivevirtualenvironment.Proc.oftheConferenceonHumanFactorsinComputingSystems,pages529{536,2003. [66] D.Garvin.Buildingalearningorganization.HarvBusRev,71(4):78{91,1993. 193

PAGE 194

R.Granlund.Web-basedmicro-worldsimulationforemergencymanagementtraining.FutureGenerationComputerSystems,17(5):561{572,2001. [68] J.GratchandW.Mao.AutomatingAfterActionReview:AttributingBlameorCreditinTeamTraining.ConferenceonBehaviorRepresentationinModelingandSimulation,2003. [69] C.Greenhalgh,M.Flintham,J.Purbrick,andS.Benford.Applicationsoftemporallinks:Recordingandreplayingvirtualenvironments.InIEEEVirtualReality2002,March2002. [70] D.Gresh,D.Rabenhorst,A.Shabo,andS.Slavin.Prima:Acasestudyofusinginformationvisualizationtechniquesforpatientrecordanalysis.InProceedingsofIEEEVisualization,2002. [71] M.R.Guibert,C.A.Pelz,E.N.Bachelder,andK.Barrera.Nextgenerationvisualizationtoolformissionplanning,brieng,andafteractionreview.InIMAGE2007Conference,Scottsdale,Arizona,July2007. [72] F.Gulliver.Post-projectappraisalspay.HarvardBusinessReview,65(2):128{32,1987. [73] A.Guye-Vuilleme,T.Capin,S.Pandzic,N.Thalmann,andD.Thalmann.Nonverbalcommunicationinterfaceforcollaborativevirtualenvironments.VirtualReality,4(1):49{59,1999. [74] J.C.Hart.Onrecordingvirtualenvironments.InIEEESymposiumonResearchFrontiersinVirtualReality,page8083,October1993. [75] S.Havre,E.Hetzler,P.Whitney,andL.Nowell.ThemeRiver:VisualizingThematicChangesinLargeDocumentCollections.IEEETRANSACTIONSONVISUALIZATIONANDCOMPUTERGRAPHICS,pages9{20,2002. [76] J.Heer,J.Mackinlay,C.Stolte,andM.Agrawala.GraphicalHistoriesforVisualization:SupportingAnalysis,Communication,andEvaluation.IEEETransactionsonVisualizationandComputerGraphics,14(6):1189{1196,2008. [77] I.Heldal,R.Schroeder,A.Steed,A.S.Axelsson,M.Spant,andJ.Widestrom.Immersivenessandsymmetryincopresentscenarios.InIEEEVirtualReality2005,pages171{178,2005. [78] M.HicksonIII,D.W.Stacks,andN.-J.Moore.NonverbalCommunication:StudiesandApplications.RoxburyPublishingCompany,LosAngeles,CA,2004. [79] R.W.HillJr.,J.Gratch,S.Marsella,J.Rickel,W.Swartout,andD.Traum.Virtualhumansinthemissionrehearsalexercisesystem.KynstlicheIntelligenz(KIJournal),17(4),2003. 194

PAGE 195

H.HochheiserandB.Shneiderman.Dynamicquerytoolsfortimeseriesdatasets:Timeboxwidgetsforinteractiveexploration.InformationVisualization,3(1):1{18,2004. [81] M.Hojat,J.S.Gonnella,T.J.Nasca,S.Mangione,M.Vergare,andM.Magee.Physicianempathy:Denition,components,measurement,andrelationshiptogenderandspecialty.AmericanJournalofPsychiatry,159(9):1563,2002. [82] J.HongandJ.Landay.WebQuilt:aframeworkforcapturingandvisualizingthewebexperience.InProceedingsofthe10thinternationalconferenceonWorldWideWeb,pages717{724.ACMNewYork,NY,USA,2001. [83] A.Hoover,E.Muth,andF.Switzer.Clemsonuniversitymoutproject. [84] D.Huynh,S.Drucker,P.Baudisch,andC.Wong.Timequilt:scalingupzoomablephotobrowsersforlarge,unstructuredphotocollections.InConferenceonHumanFactorsinComputingSystems,pages1937{1940.ACMNewYork,NY,USA,2005. [85] J.JenvaldandM.Morin.Simulation-SupportedLiveTrainingforEmergencyResponseinHazardousEnvironments.Simulation&Gaming,35(3):363,2004. [86] K.Johnsen.DesignandValidationofaVirtualHumanSystemforInterpersonalSkillsEducation.Dissertation,UniversityofFlorida,2008. [87] K.Johnsen,R.Dickerson,A.Raij,C.Harrison,B.Lok,A.Stevens,andD.S.Lind.Evolvinganimmersivemedicalcommunicationskillstrainer.Presence:TeleoperatorsandVirtualEnvironments,15(1):33{46,February2006. [88] K.Johnsen,R.Dickerson,A.Raij,B.Lok,J.Jackson,M.Shin,J.Hernandez,A.Stevens,andD.S.Lind.Experiencesinusingimmersivevirtualcharacterstoeducatemedicalcommunicationskills.InIEEEVirtualReality2005,Bonn,Germany,2005. [89] K.Johnsen,A.Raij,B.Lok,A.Stevens,andD.S.Lind.Thevalidityofavirtualhumanexperienceforinterpersonalskillseducation.InProceedingsoftheSIGCHIConferenceonHumanFactorsinComputingSystems,2007. [90] D.W.Johnson.Cooperativenessandsocialperspectivetaking.JournalofPersonalityandSocialPsychology,31:241{244,1975. [91] A.KalraandK.Karahalios.TextTone:ExpressingEmotionThroughText.INTERACT2005,3585:966,2005. [92] C.KellerandM.Duy.ISaidThat?TEACHINGExceptionalChildren,page37,2005. [93] B.Kerr.ThreadArcs:AnEmailThreadVisualization.InProceedingsofthe2003IEEESymposiumonInformationVisualization,page27,2003. 195

PAGE 196

J.Kim,D.Gunn,E.Schuh,B.Phillips,R.Pagulayan,andD.Wixon.Trackingreal-timeuserexperience(TRUE):acomprehensiveinstrumentationsolutionforcomplexsystems.InProceedingsoftheSIGCHIConferenceonHumanFactorsinComputingSystems.ACMNewYork,NY,USA,2008. [95] A.KitsantasandB.Zimmerman.Enhancingself-regulationofpractice:theinuenceofgraphingandself-evaluativestandards.MetacognitionandLearning,1(3):201{212,2006. [96] B.KnerrandD.Lampton.AnAssessmentoftheVirtual{IntegratedMOUTTrainingSystem(V-IMTS).TechnicalReportADA438315,UnitedStatesArmyResearchInstitutefortheBehavioralandSocialSciences,2005. [97] B.W.Knerr,D.R.Lampton,G.A.Martin,D.A.Washburn,andD.Cope.Developinganafteractionreviewsystemforvirtualdismountedinfantrysimulations.InTheInterserviceIndustryTraining,Simulation&EducationConference(I/ITSEC),2002. [98] T.KoetterandM.Theus.Fortune:Asystemforinteractivestatisticalgraphicsfortimeseries. [99] A.KotranzaandB.Lok.Virtualhuman+tangibleinterface=mixedrealityhuman.aninitialexplorationwithavirtualbreastexampatient.InIEEEVirtualReality2008,pages99{107,Reno,NV,March2008. [100] A.Kotranza,B.Lok,A.Deladisma,C.M.Pugh,andD.Lind.Mixedrealityhumans:Evaluatingbehavior,usability,andacceptability.ToAppearinIEEETransactionsonVisualizationandComputerGraphics,2009. [101] E.Krupat,R.Frankel,T.Stein,andJ.Irish.Thefourhabitscodingscheme:Validationofaninstrumenttoassessclinicianscommunicationbehavior.PatientEducationandCounseling,62(1):38{45,2006. [102] D.Lampton,J.Cohn,M.Endsley,J.Freeman,M.Gately,andG.Martin.MeasuringSituationAwrenessforDismountedInfantrySquads.TheInterservice/IndustryTraining,Simulation&EducationConference(I/ITSEC),2005(-1),2005. [103] F.Lang,R.McCord,L.Harvill,andD.S.Anderson.Communicationassessmentusingthecommongroundinstrument:Psychometricproperties.FamilyMedicine,36(3):189{198,2004. [104] A.Lecuyer,S.Coquillart,A.Kheddar,P.Richard,andP.Coiet.Pseudo-hapticfeedback:canisometricinputdevicessimulateforcefeedback?ProceedingsofIEEEVirtualReality2000,pages83{90,2000. 196

PAGE 197

J.Leigh,A.E.Johnson,T.A.DeFanti,M.Brown,andetal.Areviewoftele-immersiveapplicationsinthecaveresearchnetwork.InIEEEVirtualReality1999,pages180{187,Houston,TX,March1999. [106] J.Lin,E.Keogh,andS.Lonardi.Visualizinganddiscoveringnon-trivialpatternsinlargetimeseriesdatabases.InformationVisualization,4(2):61{82,2005. [107] H.Liu,H.Lieberman,andT.Selker.AutomaticAectiveFeedbackinanEmailBrowser.ProceedingsoftheSIGCHIConferenceonHumanFactorsinComputingSystems,2003. [108] H.Liu,T.Selker,andH.Lieberman.Visualizingtheaectivestructureofatextdocument.ACMCHI'03ExtendedAbstractsonHumanFactorsinComputingSystems,April2003. [109] LockheedMartinInformationSystems.AdvancedDistributedSimulationTechnologyII(ADSTII)CombatSyntheticTrainingAssessmentRange(CSTAR)FASReport,1997. [110] A.MacEachren.HowMapsWork:Representation,Visualization,andDesign.GuilfordPressNewYork,1995. [111] A.MacEachren,F.Boscoe,D.Haug,andL.Pickle.Geographicvisualization:designingmanipulablemapsforexploringtemporallyvaryinggeoreferencedstatistics.InProceedingsofIEEESymposiumonInformationVisualization1998,pages87{94,1998. [112] M.Macedonia.GamesSoldiersPlay.IEEESpectrum,39(3):32{37,2002. [113] J.Mackinlay,G.Robertson,andR.DeLine.Developingcalendarvisualizersfortheinformationvisualizer.InProceedingsofthe7thannualACMsymposiumonUserinterfacesoftwareandtechnology,pages109{118.ACMNewYork,NY,USA,1994. [114] J.MacMillan,A.Alexander,S.Weil,B.Littleton,I.Aptima,M.Woburn,B.Roberts,W.Ferguson,J.Berliner,R.Tomlinson,etal.DARWARS:AnArchitectureThatSupportsEectiveExperientialTraining.Technicalreport,BBNTechnologies,2005. [115] A.Manganas,M.Tsiknakis,E.Leisch,M.Ponder,T.Molet,B.Herbelin,N.Magnenat-Thalmann,D.Thalmann,M.Fato,andA.Schenone.Thejustvrtool:Aninnovativeapproachtotrainingpersonnelforemergencysituationsusingvirtualrealitytechniques.JournalonInformationTechnologyinHealthcare,2(6):399{412,2004. [116] T.Manninen.InteractionFormsandCommunicativeActionsinMultiplayerGames.GameStudies,3(1):2003,2003. [117] W.MaoandJ.Gratch.TheSocialCreditAssignmentProblem.ProceedingsoftheFourthInternationalWorkingConferenceonIntelligentVirtualAgents,2003. 197

PAGE 198

W.MaoandJ.Gratch.SocialCausalityandResponsibility:ModelingandEvaluation.ProceedingsoftheFifthInternationalConferenceonIntelligentVirtualAgents,2005. [119] P.May.ManagementOverviewoftheIntelligenceandElectronicWarfareTacticalProciencyTrainer.2002SpringSimulationInteroperabilityWorkshop,2002. [120] A.McCrindleandC.Christensen.Theimpactoflearningjournalsonmetacognitiveandcognitiveprocessesandlearningperformance.LearningandInstruction,5(2):167{185,1995. [121] P.McDowell,R.Darken,J.Sullivan,andE.Johnson.Delta3D:ACompleteOpenSourceGameandSimulationEngineforBuildingMilitaryTrainingSystems.TheInterservice/IndustryTraining,Simulation&EducationConference(I/ITSEC),2005,2005. [122] M.McGovern.Theuseofvideointheself-evaluationofspeechtherapystudents.InternationalJournalofLanguage&CommunicationDisorders,20(3):297{300,1985. [123] A.Mehrabian.NonverbalCommunication.Aldine-Atherton,Chicago,IL,1972. [124] A.Mehrabian.Silentmessages:Implicitcommunicationofemotionsandattitudes.Wadsworth,Belmont,California,2ndedition,1981. [125] D.MillerandJ.Thorpe.SIMNET:theadventofsimulatornetworking.ProceedingsoftheIEEE,83(8):1114{1123,1995. [126] M.Mori.Theuncannyvalley.Energy,7(4):33{35,1970.TranslatedbyKarlF.MacDormanandTakashiMinato. [127] J.Morie,K.Iyer,D.Luigi,J.Williams,A.Dozois,andA.Rizzo.DevelopmentofaDataManagementToolforInvestigatingMultivariateSpaceandFreeWillExperiencesinVirtualReality.AppliedPsychophysiologyandBiofeedback,30(3):319{331,2005. [128] M.Morin,J.Jenvald,andM.Thorstensson.Computer-supportedvisualizationofrescueoperations.SafetyScience,35(1-3):3{27,2000. [129] T.Murphey.Whydon'tteacherslearnwhatlearnerslearn?TakingtheguessworkoutwithActionLogging.InEnglishTeachingForum,volume21,pages6{10,1993. [130] C.NassandY.Moon.Machinesandmindlessness:Socialresponsestocomputers.JournalofSocialIssues,56(1):81{103,2000. [131] NaturalPoint.Optitracksoftware. 198

PAGE 199

C.L.North.AUserInterfaceforCoordinatingVisualizationsBasedonRelationalSchemata:Snap-togetherVisualization.PhDthesis,UniversityofMaryland,May2000. [133] M.OReilly,M.OHalloran,J.Sigafoos,G.Lancioni,V.Green,C.Edrisinha,H.Cannella,andM.Olive.Evaluationofvideofeedbackandself-managementtodecreaseschoolyardaggressionandincreasepro-socialbehaviourintwostudentswithbehaviouraldisorders.EducationalPsychology,25(2):199{206,2005. [134] S.L.Oviatt,P.R.Cohen,M.Wang,andJ.Gaston.Asimulation-basedresearchstrategyfordesigningcomplexnlsystems.InARPAHumanLanguageTechnologyWorkshop,Princeton,NJ,1993. [135] D.F.Parkhurst.Statisticalsignicancetests:Equivalenceandreversetestsshouldreducemisinterpretation.BioScience,51(12):1051{1057,2001. [136] S.Paul,K.Dawson,J.Lanphear,andM.Cheema.Videorecordingfeedback:afeasibleandeectiveapproachtoteachinghistory-takingandphysicalexaminationskillsinundergraduatepaediatricmedicine.MedicalEducation,32(3):332{336,1998. [137] E.PayneandK.Dietrick.WARSIMEnterstheSceneinArmyTraining.CrossTalkTheJournalofDefenseSoftwareEngineering,18(9):12{13,2005. [138] K.PerlinandD.Fox.Pad:analternativeapproachtothecomputerinterface.InProceedingsofthe20thannualconferenceonComputergraphicsandinteractivetechniques,pages57{64.ACMPressNewYork,NY,USA,1993. [139] D.-P.Pertaub,M.Slater,andC.Barker.Anexperimentonpublicspeakinganxietyinresponsetothreedierenttypesofvirtualaudience.Presence:TeleoperatorsandVirtualEnvironments,11(1):68{78,2002. [140] C.Plaisant,B.Milash,A.Rose,S.Wido,andB.Shneiderman.LifeLines:visualizingpersonalhistories.Proc.oftheSIGCHIConferenceonHumanFactorsinComputingSystems:CommonGround,1996. [141] A.Pusch,O.Martin,andS.Coquillart.Hemphand-displacement-basedpseudo-haptics:Astudyofaforceeldapplication.ProceedingsofIEEE3DUserInterfaces2008,pages59{66,2008. [142] J.Quarles,S.Lampotang,I.Fischler,P.Fishwick,andB.Lok.Amixedrealitysystemforenablingcollocatedafteractionreview.In7thIEEEandACMInternationalSymposiumonMixedandAugmentedReality-ISMAR2008,Cambridge,UK,September2008. [143] K.Raiha,A.Aula,P.Majaranta,H.Rantala,andK.Koivunen.Staticvisualizationoftemporaleye-trackingdata.LectureNotesinComputerScience,3585:946{949,2005. 199

PAGE 200

A.Raij,K.Johnsen,R.Dickerson,B.Lok,M.Cohen,M.Duerson,R.Pauley,A.Stevens,P.Wagner,andD.S.Lind.Comparinginterpersonalinteractionswithavirtualhumantothosewitharealhuman.IEEETransactionsonVisualizationandComputerGraphics,13(3),May2007. [145] A.Raij,K.Johnsen,R.Dickerson,B.Lok,M.Cohen,A.Stevens,T.Bernard,C.Oxendine,P.Wagner,andD.S.Lind.Interpersonalscenarios:Virtualreal?InIEEEVirtualReality2006,pages59{66,2006. [146] A.Raij,A.Kotranza,D.S.Lind,andB.Lok.Virtualexperiencesforsocialperspective-taking.InToappearinProceedingsofIEEEVirtualReality2009,Lafayette,Louisiana,March2009. [147] A.RaijandB.Lok.Ipsviz:Anafter-actionreviewtoolforhuman-virtualhumanexperiences.InIEEEVirtualReality2008,Reno,NV,March2008. [148] R.Ramloll,C.Trepagnier,M.Sebrechts,andJ.Beedasy.Gazedatavisualizationtools:opportunitiesandchallenges.InformationVisualisation,2004.IV2004.Proceedings.EighthInternationalConferenceon,pages173{180,July2004. [149] R.RapeeandK.Hayman.Theeectsofvideofeedbackontheself-evaluationofperformanceinsociallyanxioussubjects.BehaviourResearchandTherapy,34(4):315{322,1996. [150] R.Reeder,P.Pirolli,andS.Card.WebEyeMapperandWebLogger:toolsforanalyzingeyetrackingdatacollectedinweb-usestudies.InConferenceonHumanFactorsinComputingSystems,pages19{20,2001. [151] C.Reeve.Presenceinvirtualtheater.Presence:Teleoperators&VirtualEnvironments,9(2):209{213,2000. [152] E.Remolina,S.Ramachandran,R.Stottler,andW.Howse.IntelligentSimulation-BasedTutorforFlightTraining.TheInterservice/IndustryTraining,Simulation&EducationConference(I/ITSEC),2004(-1),2004. [153] S.Rezvan.TheeectsofmetacognitivetrainingontheacademicachievementandhappinessofEsfahanUniversityconditionalstudents.CounsellingPsychologyQuarterly,19(4):415{428,2006. [154] D.RIDLEY.Self-RegulatedLearning:theInteractiveInuenceofMetacognitiveAwarenessandGoal-Setting.JournalofExperimentalEducation,60(4):293{306,1992. [155] A.A.Rizzo,D.Klimchuk,R.Mitura,T.Bowerly,J.G.Buckwalter,K.Kerns,K.Randall,R.Adams,P.Finn,I.Tarnanas,C.Sirbu,T.H.Ollendick,andS.-C.Yeh.Avirtualrealityscenarioforallseasons:Thevirtualclassroom.In11thInternationalConferenceonHuman-ComputerInteraction,LasVegas,NV,July2005. 200

PAGE 201

M.Rokeach.TheOpenandClosedMind:InvestigationsIntotheNatureofBeliefSystemsandPersonalitySystems.BasicBooks,1960. [157] D.Roter,S.Larson,H.Shinitzky,R.Cherno,J.Serwint,G.Adamo,andL.Wissow.Useofaninnovativevideofeedbacktechniquetoenhancecommunicationskillstraining.MedicalEducation,38:145{157(13),February2004. [158] B.O.Rothbaum,L.Hodges,S.Smith,J.H.Lee,andL.Price.Acontrolledstudyofvirtualrealityexposuretherapyforthefearofying.JournalofConsultingandClinicalPsychology,68(6):1020{26,2000. [159] K.Ryokai,C.Vaucelle,andJ.Cassell.Virtualpeersaspartnersinstorytellingandliteracylearning.JournalofComputerAssistedLearning,19(2):195{208,2003. [160] A.SantellaandD.DeCarlo.Robustclusteringofeyemovementrecordingsforquanticationofvisualinterest.InProceedingsofthe2004SymposiumonEyeTrackingResearch&Applications,pages27{34,2004. [161] B.Shneiderman.Theeyeshaveit:Ataskbydatatypetaxonomyforinformationvisualizations.InIEEEVisualLanguages,pages336{343,CollegePark,Maryland20742,U.S.A.,1996. [162] B.Shneiderman.Inventingdiscoverytools:combininginformationvisualizationwithdatamining.InformationVisualization,1(1):5{12,2002. [163] J.Shockley,R.Ford,M.Beebe,G.Lucha,andM.Johnson.TheJointTrainingExperimentationProgram:HotwashfromtheSecondDemonstration.04SSIW-132,SimulationInteroperabilityWorkshop,April,2004. [164] S.SilvaandT.Catarci.Visualizationoflineartime-orienteddata:asurvey.InProceedingsoftheFirstInternationalConferenceonWebInformationSystemsEngineering,volume1,pages310{319,2000. [165] E.Simon.MaskingUnmasked:FourApproachestoBasicActing.PalgraveMacmillan,2003. [166] M.Slater,A.Sadagic,M.Usoh,andR.Schroeder.Small-groupbehaviorinavirtualandrealenvironment:Acomparativestudy.Presence:TeleoperatorsandVirtualEnvironments,9(1):37{51,2000. [167] P.D.Sloane,R.Beck,V.Kowlowitz,A.M.Blotzer,L.Wang,L.Akins,F.White-Chu,andC.M.Mitchell.Behavioralcodingforevaluationofmedicalstudentcommunication:Claricationorobfuscation?AcademicMedicine,79(2):162{70,February2004. [168] M.SmithandA.Fiore.Visualizationcomponentsforpersistentconversations.InProceedingsoftheSIGCHIConferenceonHumanFactorsinComputingSystems,pages136{143,2001. 201

PAGE 202

R.Smith.IEWTPTSystemDesignReview:SoftwareDesign.USArmySimulationTrainingandInstrumentationCommand,2001. [170] R.SmithandG.Allen.Afteractionreviewinmilitarytrainingsimulations.InWSC'94:Proceedingsofthe26thConferenceonWinterSimulation,pages845{849,SanDiego,CA,USA,1994. [171] A.Steed,A.Brogni,andV.Vinayagamoorthy.Breaksinpresenceasusabilitycriteria.InProc.ofHCIInternational2005,The11thInternationalConferenceonHumanComputerInteraction,July2005. [172] W.SteptoeandA.Steed.High-delityavatareye-representation.IEEEVirtualReality2008,pages111{114,March2008. [173] V.Stevens.Makingthemostofateachablemoment:asmokeless-tobaccocessationinterventioninthedentaloce.AmericanJournalofPublicHealth,85(2):231{235,1995. [174] S.StoevandW.Straer.Acasestudyoninteractiveexplorationandguidanceaidsforvisualizinghistoricaldata.InProceedingsoftheIEEEConferenceonVisualization,pages485{488.IEEEComputerSocietyWashington,DC,USA,2001. [175] C.Strapparava,A.Valitutti,andO.Stock.DanceswithWords.InProceedingsoftheInternationalJointConferenceonArticialIntelligence,pages1719{1724,2007. [176] W.Swartout,J.Gratch,R.Hill,E.Hovy,S.Marsella,J.Rickel,D.Traum,andT.UniversityOfSouthernCaliforniaMarinaDelReyCaInstForCreative.Towardvirtualhumans.AIMagazine,27(2):96,2006. [177] T.TakadaandH.Koike.Tudumi:InformationVisualizationSystemforMonitoringandAuditingComputerLogs.InProceedingsontheIEEEConferenceonInformationVisualization,pages570{576,2002. [178] A.TartaroandJ.Cassell.Authorablevirtualpeersforautismspectrumdisorders.InWorkshoponLanguage-EnabledEducationalTechnologyatthe17thEuropeanConferenceonArticialIntelligence(ECAI06),2006. [179] K.ThiemannandH.Goldstein.Socialstories,writtentextcues,andvideofeedback:Eectsonsocialcommunicationofchildrenwithautism.JournalofAppliedBehaviorAnalysis,34(4):425{446,2001. [180] K.R.Thorisson.Gandalf:Anembodiedhumanoidcapableofreal-timemultimodaldialoguewithpeople.In1stACMInternationalConferenceonAutonomousAgents,pages536{7,MarinadelRey,CA,1997. [181] K.R.ThorissonandJ.Cassell.Whyputanagentinabody:Theimportanceofcommunicativefeedbackinhuman-humanoiddialogue.InLifelikeComputerCharacters'96,Snowbird,UT,1996. 202

PAGE 203

A.UnwinandG.Wills.ExploringTimeSeriesGraphically.StatisticalComputingandGraphicsNewsletter,2:13{15,1999. [183] M.Usoh,E.Catena,S.Arman,andM.Slater.Usingpresencequestionnairesinreality.Presence:TeleoperatorsandVirtualEnvironments,9(5):497{503,2000. [184] A.VanBurenandS.Nowicki.AwarenessofInterpersonalstyleandself-evaluation.TheJournalofsocialpsychology,137(4):429{434,1997. [185] J.vanWijkandE.vanSelow.Clusterandcalendarbasedvisualizationoftimeseriesdata.InProceedingsofthe1999IEEESymposiumonInformationVisualization,page4.IEEEComputerSocietyWashington,DC,USA,1999. [186] F.ViegasandJ.Donath.Chatcircles.InProceedingsoftheSIGCHIConferenceonHumanFactorsinComputingSystems:TheCHIistheLimit,pages9{16.ACMNewYork,NY,USA,1999. [187] F.B.Viegas,S.Golder,andJ.Donath.Visualizingemailcontent:portrayingrelationshipsfromconversationalhistories.InCHI'06:ProceedingsoftheSIGCHIConferenceonHumanFactorsinComputingSystems,pages979{988,2006. [188] V.Vinayagamoorthy,A.Brogni,M.Gillies,M.Slater,andA.Steed.Aninvestigationofpresenceresponseacrossvariationsinvisualrealism.The7thAnnualInternationalPresenceWorkshop,pages148{155,2004. [189] V.Vinayagamoorthy,M.Gillies,A.Steed,E.Tanguy,X.Pan,C.Loscos,andM.Slater.Buildingexpressionintovirtualcharacters.InEurographics2006StateoftheArtReports,Vienna,Austria,2006. [190] V.Vinayagamoorthy,A.Steed,andM.Slater.Buildingcharacters:Lessonsdrawnfromvirtualenvironments.InTowardSocialMechanismsofAndroidScience:ACogSci2005Workshop,pages119{26,Stresa,Italy,2005. [191] T.Watanabe,M.Ogikubo,andY.Ishil.Visualizationofrespirationintheembodiedvirtualcommunicationsystemanditsevaluation.InternationalJournalofHuman-ComputerInteraction,17(1):p89{102,March2004. [192] M.Wattenberg.Sketchingagraphtoqueryatime-seriesdatabase.InCHI'01:CHI'01extendedabstractsonHumanfactorsincomputingsystems,pages381{382,NewYork,NY,USA,2001.ACM. [193] C.Weaver,D.Fyfe,A.Robinson,D.Holdsworth,D.Peuquet,andA.MacEachren.VisualAnalysisofHistoricHotelVisitationPatterns.InProceedingsoftheIEEESymposiumonVisualAnalyticsScienceandTechnology,2006. [194] M.Weber,M.Alexa,andW.Muller.VisualizingTime-SeriesonSpirals.InProceedingsoftheIEEESymposiumonInformationVisualization,page7,2001. 203

PAGE 204

L.A.Wind,J.vanDalen,A.M.M.Muijtjens,andJ.-J.Rethans.Assessingsimulatedpatientsinaneducationalsetting:themasp(maastrichtassessmentofsimulatedpatients).MedicalEducation,38(1):39{44,2004. [196] D.Wooding.FixationMaps:Quantifyingeye-movementtraces.InProceedingsofthe2002SymposiumonEyeTrackingResearch&Applications,pages31{36,2002. [197] D.Wuillemin,G.vanDoorn,B.Richardson,andM.Symmons.Hapticandvisualsizejudgementsinvirtualandrealenvironments.InThe1stJointEurohapticsConferenceandSymposiumonHapticInterfacesforVirtualEnvironmentandTeleoperatorSystems,pages86{89,2005. [198] R.XiongandJ.Donath.Peoplegarden:creatingdataportraitsforusers.InUIST'99:Proceedingsofthe12thAnnualACMSymposiumonUserInterfaceSoftwareandTechnology,pages37{44,1999. [199] N.YeeandJ.Bailenson.Walkamileindigitalshoes:Theimpactofembodiedperspective-takingonthereductionofnegativestereotypinginimmersivevirtualenvironments.ProceedingsofPRESENCE,page2426,2006. [200] N.YeeandJ.Bailenson.Theproteuseect:Theeectoftransformedself-representationonbehavior.HumanCommunicationResearch,33(3):271{290,2007. [201] C.Zanbaka,P.Goolkasian,andL.F.Hodges.Canavirtualcatpersuadeyou?theroleofgenderandrealisminspeakerpersuasiveness.InConferenceonHumanFactorsinComputingSystems,pages1153{1162,Montreal,Quebec,Canada,2006. [202] C.Zanbaka,A.Ulinski,P.Goolkasian,andL.F.Hodges.Eectsofvirtualhumanpresenceontaskperformance.InTheInternationalConferenceonArticialRealityandTelexistence,pages174{181,2004. [203] B.Zimmerman.Self-RegulatedLearningandAcademicAchievement:AnOverview.EducationalPsychologist,25(1):3{17,1990. 204

PAGE 205

AndrewBrianRaijwasbornin1979inMiamiBeach,Florida.AndrewwasraisedinSouthFlorida,wherehegraduatedcumlaudefromtheRansomEvergladesSchoolin1997.In2001,hereceivedabachelorofsciencedegreeincomputerscienceandaminorinEnglishfromNorthwesternUniversity.AfterNorthwestern,hewasagraduatestudentincomputerscienceattheUniversityofNorthCarolinaatChapelHill(UNC-CH).AtUNC-CH,heconductedresearchintheareasofcomputergraphics,computervision,andprojector-camerasystemsunderthesupervisionofHenryFuchs,MarcPollefeys,andHermanTowles.Aftercompletingamaster'sdegreeincomputerscienceatUNC-CHin2003,AndrewbeganaPh.D.programattheUniversityofFlorida(UF)incomputerengineeringunderthesupervisionofDr.BenjaminLok.HewasawardedaUFAlumniFellowshiptosupporthisresearch,whichfocusesonvirtualhumaninterfacesandtheiremerginguseforinterpersonalskillstraining.Hisdissertationexploresleveraginginteractionlogs,visualization,andvirtualenvironmentstocreateenhancedafter-actionreviewsofinteractionsbetweenhumansandvirtualhumans.Hisworkhasreceivedsignicantrecognitioninboththeeldsofcomputerscienceandmedicinewith14articlespublishedinleadingjournalsandconferences,includingafeaturedarticleintheMay2007issueofIEEETransactionsonVisualizationsandComputerGraphics.HeisamemberoftheIEEEandACM,andservedasapanelistatIEEEVirtualReality2008.HeisalsoamemberoftheDeltaEpsilonIotaAcademicHonorSociety,AlphaLambdaDeltaAcademicFraternity,andtheNationalSocietyofCollegiateScholars.OnSeptember21,2008,Andrewmarriedhiscollegesweetheart,Emily.AndrewandEmilycurrentlyresideinGainesville,FloridawiththeirdogRuby.UponcompletionofhisPh.D.program,AndrewwillcontinuehisresearchasapostdoctoralresearcherattheUniversityofFlorida. 205