Real-time motion data sequencing and aggregation in virtual environments/live performance

MISSING IMAGE

Material Information

Title:
Real-time motion data sequencing and aggregation in virtual environments/live performance
Physical Description:
Project in lieu of thesis
Creator:
Yudin, Anton
Publisher:
College of Fine Arts, University of Florida
Place of Publication:
Gainesville, Fla.
Publication Date:

Notes

Abstract:
This paper briefly describes an open framework that can route, sequence, and aggregate data from different motion capture systems. The project uses open standards like J2EE, JSON, and XML to achieve these goals. The framework includes a custom protocol designed to send and receive motion data, a router that manages connections from different sources and consumers of motion data, a 3D environment that is used as a client for the system, and a web-based user interface to configure the router of motion data. The protocol and the router were optimized to be used in a distributed environment with multiple simultaneous clients and sources. The system allows dancers and performing artists to collaborate in distributed performances around the world utilizing computer networks.The working system has been successfully demonstrated during several conferences and shows, including eComm-2010, SIGGRAPH-2010, and iDMAA 2010.
General Note:
Digital Arts and Sciences terminal project

Record Information

Source Institution:
University of Florida Institutional Repository
Holding Location:
University of Florida
Rights Management:
All rights reserved by the source institution and holding location.
System ID:
AA00004960:00001


This item is only available as the following downloads:


Full Text

PAGE 1

REAL-TIMEMOTIONDATASEQUENCINGANDAGGREGATIONINVIRTUALENVIRONMENTS/LIVEPERFORMANCEByANTONYUDINSUPERVISORYCOMMITTEE:OLIVERIO,JAMESCHARLES,CHAIRBARMPOUTIS,ANGELOS,MEMBERDEVANE,BENJAMIN,MEMBERAPROJECTINLIEUOFTHESISPRESENTEDTOTHECOLLEGEOFFINEARTSOFTHEUNIVERSITYOFFLORIDAINPARTIALFULFILLMENTOFTHEREQUIREMENTSFORTHEDEGREEOFMASTEROFARTSUNIVERSITYOFFLORIDA2011

PAGE 2

SummaryofProjectOptioninLieuofThesisPresentedtotheCollegeOfFineArtsoftheUniversityOfFloridainPartialFulllmentoftheRequirementsfortheDegreeofMasterOfArtsREAL-TIMEMOTIONDATASEQUENCINGANDAGGREGATIONINVIRTUALENVIRONMENTS/LIVEPERFORMANCEByAntonYudinAugust2011Chair:Oliverio,JamesCharlesMajor:DigitalArtsandScienceThispaperbrieydescribesanopenframeworkthatcanroute,sequence,andaggregatedatafromdierentmotioncapturesystems.TheprojectusesopenstandardslikeJ2EE,JSON,andXMLtoachievethesegoals.Theframeworkincludesacustomprotocoldesignedtosendandreceivemotiondata,arouterthatmanagesconnectionsfromdierentsourcesandcon-sumersofmotiondata,a3Denvironmentthatisusedasaclientforthesystem,andaweb-baseduserinterfacetoconguretherouterofmotiondata.Theprotocolandtherouterwereoptimizedtobeusedinadis-tributedenvironmentwithmultiplesimultaneousclientsandsources. Thesystemallowsdancersandperformingartiststocollaborateindistributedperformancesaroundtheworldutilizingcomputernetworks. Theworkingsystemhasbeensuccessfullydemonstratedduringseveralconferencesandshows,includingeComm-2010,SIGGRAPH-2010,andiDMAA2010.

PAGE 3

MORPHEUSHelphim,Trinity.Neoallowshimselftobehelpedintooneofthechairs.MORPHEUSDoyourememberwhenIaskedyouaboutanapparatusthatcouldturnavirtualrealityintoreality?Neonods.MORPHEUSIt'srighthere.HetouchesNeo'shead.MORPHEUSAndit'saccessedhere. LarryandAndyWachowskiTheMatrix1999 1

PAGE 4

Contents1Introduction41.1MotionData........................... 4 1.23D/VirtualenvironmentsandVirtualReality......... 5 1.3HumanInterfaceDevices..................... 5 1.3.1TypesofHumanInterfaceDevices........... 5 1.3.2DierentwaystoconnecttoaHumanInterfaceDevice 6 1.3.3Availableuniversalcommunicationprotocolsandin-terfaces.......................... 6 2Motivation72.1Openenvironmentforscienceandartsexperimentswithmo-tiondata............................. 7 2.2Requirementsfora3Denvironment.............. 8 2.3Requirementsforaprotocol................... 8 2.3.1Simplicity......................... 9 2.3.2Extensibility....................... 9 2.4RequirementsforUserInterface................. 9 3Implementation113.1Details............................... 11 3.1.13DEnvironment..................... 11 3.1.2CustomJSON/XMLbasedprotocol.......... 11 3.1.3BasicMessages...................... 12 3.1.4SerializationandDeserialization............ 13 3.1.5MessageRouting..................... 14 3.1.6Scripting......................... 14 3.1.7WebBasedTechnologies................. 17 3.2Challenges............................. 18 3.2.1Complexityoftheproject................ 18 3.2.2Protocolperformanceissues............... 18 4Applications214.1History.............................. 21 2

PAGE 5

4.2InternationalSymposiumonMixedandAugmentedRealityISMAR,Orlando,USA-2009................. 22 4.3EmergingCommunicationsConferenceandAwardseComm,SanFrancisco,California,USA-2010............. 22 4.4SIGGRAPH,LosAngeles,California,USA-2010....... 22 4.5TheInternationalDigitalMediaArtsAssociationConference,Vancouver,Canada-2010.................... 22 4.6DigitalArtsFestival,Redmond,Washington,USA-2011.. 23 5Acknowledgments30 3

PAGE 6

Chapter1Introduction Theemergentinterestingeographicallydistributedperformancehascreatedgreaterdemandforaccessiblesoftwaresolutionsthatworkinheterogeneousenvironmentswithoutintroducingunwarrantedofadditionaltechnicalcom-plexitytotheoverallsystem. Thegoalofthisprojectistocreateadistributed,scalable,andopenframeworkthatallowsresearchersandartiststoexperimentina3Denvi-ronmentwithmotiondataobtainedfromdierenthumaninterfacedevices.Thisframeworkisdevidedintofourmodules:acustomprotocolforsendingandreceivingmotiondata,arouterthatmanagesconnections,a3Denvironmentthatisusedtovisualizemotiondata,andaweb-baseduserinterfacetoconguretherouter.Allfourmodulescanbeextendedorsubstitutedindividuallywithoutcompromisingotherpartsofthesystem.Theprotocolisbasedonmeta-languagesXMLandJSON.TherouterisbuiltasawebservletandismanagedbyaJ2EEwebcontainer.TheuserinterfaceusesaRESTfularchitecture.1.1MotionDataMotiondatacanberepresentedbyasequenceofstates"ofa3Dobject.Forexample,motiondataobtainedfromahumanbodymovinginaroommightberepresentedbyasequenceof3Dtransformationsforeachbodyparthead,hands,legs,etc..Thisdatamightbeusedtorecreatethemovementsofthepersononthescreenusinganavatar,"triggermusicalsoundsorvisualeects,oranalyzeproblematicpatternsinawalkthatmightbesymptomsofaparticulardisease.Becausetherearevirtuallyunlimitedwaysthatmotiondatamaybeused,insteadofcreatingafocusedenduserapplication,itwouldbemorevaluabletocreateaframeworkthatcanbemodiedandextendedtoaccomplishdierentgoalsindierentapplications. 4

PAGE 7

1.23D/VirtualenvironmentsandVirtualRealityToday,3Dgraphics,3Dgames,VirtualWorldsandVirtualRealityarepop-ulartopic,thoughtheactualtermsaresometimesmisused.3Dgraphicsisatechnologyforrendering3Dscenesbuildings,trees,people,cars,furnitureona2Dscreen[1].Thereareseveralwaystoaccom-plishthisandfromthesoftwaredevelopmentperspectiveitmeansusinga3DapplicationprogramminginterfaceAPItoalibrarythatimplementsthebasicconcepts-renderashape,applyatexture,lightthescene.ThemostpopularAPIsareOpenGLandDirectX[2].3Dgamesareprogramsthatuse3Dgraphicsandletaplayerinteractwith3Dscenes.3Dgamescoverabroadrangeofgenresthatincludesaction,ightsimulator,sport,puzzles,role-playing,cardandchessgames.Dierentgenresputadierentemphasisonthequalityofthe3Dgraphics.Forexample,3Daction,sport,andsimulatorgamesareknowntohavehighresolutionandhighdetailgraphics.VirtualWorldsarespecialsystemsusuallydistributedandaccessibleonlinethatletusersengageindierentactivitiesinasharedsimulatedenvironment.ThemaindierencebetweenVirtualWorldsand3DgamesisthatusuallyinVirtualWorldstherearenogoalsandusersdonotwin".Usersmightcommunicateorparticipateinavirtualactivities,yettheydonotscorepoints.VirtualWorldsoftenareusedtotrainacertaingroupofpeopleskillsthataredicultorexpensivetotrainintherealworld.OnedenitionofVirtualRealitymightbeasimulated3Dscenewheretheuserinteractswithimaginaryobjectsandtheinteractionmechanismtendstoresembletherealmechanismascloselyaspossible.ModerninputandoutputdevicessometimescalledhumaninterfacedevicesthatareoftenusedinVirtualRealityincludegloves,motioncapturestages,hapticmice,andvideogoggles.TherearedierentprotocolsandAPIsthatletdierentpartsofthesystemcommunicatewiththesedevices.1.3HumanInterfaceDevices1.3.1TypesofHumanInterfaceDevicesHumanInterfaceDevicesmightbedividedintotwomaincategories:inputandoutputdevices.Someofthedevicesmightincludebothinputandoutputcharacteristics.Agoodexampleofaninputdeviceisastandardcomputermouse.Amouseinputdevicedetectsmovementsandbuttonspresses,encodestheseeventsusingaprotocol,andsendsthisdatatothecomputer.Videogogglesareagoodexampleofanoutputdevice.Videogogglesreceiveasignalsentfromthecomputeranddisplaytheimagefortheuser.Agoodexampleofahybridinputandoutputhumaninterfacedevice 5

PAGE 8

isaHapticMouse.Thisdevicebehaveslikearegularmousethatsendsmovementeventstothecomputerandalsosendsfeedbackfromthecom-putertotheuser.Thiskindofdevicesletstheusertouch"avirtualshapeorfeel"dierenttypesofforcessimulatedbyaprogram.Motioncapturesystemsareanotherexampleofahumaninterfacede-vice.Thistypeofaninputdevicedetects3Dpositionsofdierentpartsoftheuser'sbodyinreal-timeandsendsthisdatatothecomputer.Therearedierentwaystoaccomplishthis.Somemotioncapturesystemsusewearablesensorsthatconstantlysendmotiondatatothecomputer.Othermotioncapturesystemsusemarkersandcamerasthatreconstructtheposi-tionsofthehumanbody.Evenmoreadvancedsystemsdonotrequireanytypeofspecialclothormarkersandreconstructthepositionsofthehumanbodybyanalyzingpure2Dimagestakenfromseveralcameras.1.3.2DierentwaystoconnecttoaHumanInterfaceDeviceThereareseveralwaystoconnectacomputertoahumaninterfacedevice.ThemostcommonwaysincludeRS232,USB,andEthernetconnections.Somemorecomplexsystems,likemotioncapturesystems,mightusepro-prietarynon-standardconnections.1.3.3Availableuniversalcommunicationprotocolsandin-terfacesThereareseveralprotocolsthatareusedtodaytoconnecttohumaninter-facedevices.ThemostcommonisMIDI.MIDIwascreatedin1982asauniversalpro-tocolthatconnectsdigitalmusicalinstruments[3].Thisprotocolsupportsupto16virtualchannelsthatcanbeusedtosendandreceivesimplemes-sages,likekeyboardpressevents.EventhoughMIDIsupportsextensions,itisconsideredobsoletebecauseofitslimitations.Anotherpopularprotocolthatisusedtosendeventsfromdierentin-putdevicesisOpenSoundControlOSC.UnlikeMIDI,OSCisbasedonTCP/IP.Itsupportsfourtypesofdatathatcanbeusedtoconstructames-sage:integers,oats,c-stylestrings,andblobs.OSCprotocolwasinventedmainlytoovercomethelimitationsoftheMIDIprotocol.UnlikeMIDI,thisprotocolusestextualnamesfortheparametersinsteadofnumbers.Becauseofthisfeature,OSCprotocolcanbeeasilyextended[4].ExtendingMIDIprotocolrequirestheuseofSYSEXmessagesthatarenotsupportedbystandarddevices. 6

PAGE 9

Chapter2MotivationTherearedierentpackagesthatworkwithdierentsourcesofmotiondata.First,eachmotioncapturesystemcomeswithitsownsoftwaredevelopmentkitSDKthatallowsadevelopertoreceivedatafromthesystemusingproprietaryandclosedprotocols.AnotherwayistouseapackagelikeMotionBuilderthatworkswithdierentmotioncapturesystems[5].Un-fortunately,thosekindsofsystemsarerelativelyexpansiveforresearchersandartists,anddiculttoextend.AnextensiontoMotionBuilder,forexample,requiresC++orPythonprogrammingskills. Additionally,inthecontextofadistributedperformance,dierentper-formingartistsmightwishtousedierentmotioncapturesystemsthatarenotcompatiblewitheachother.Inthiscasetheywouldneedamethodthatworkswithdierenttypesofmotiondatasourcesandallowstheartiststointuitivelycollaborateinacommonvirtualenvironment. Thisprojecttriestosolvethoseissuesbysettingcertainrequirenmentsthatarediscussedbelow.2.1Openenvironmentforscienceandartsexper-imentswithmotiondataLackofanopenandhighlycustomizableenvironmenttoexperimentwithreal-timemotiondatawasthemainmotivationforbuildingthesystem.Onecouldarguethattherearealotof3Dengines,environments,andpackages,yettherearenoopenandintegratedenvironmentsthatallowtheresearchertoexperimentwithmotiondata.Forexample,thereareseveralproprietaryandopensourcegameengineavailabletoday.Hereisashortlistofsuchengines.Opensourceengines: Blender IdTech1,2,3,ioquake 7

PAGE 10

OpenSceneGraph Panda3D OGRE JMonkeyEngine Unity3D UnrealEngineProprietaryengines: Unity3D UnrealEngineHowever,noneofthesepackagescanbeeasilyconnectedtoamotioncapturesystem.Adeveloperwouldneedtowriteacustomplug-infortheenginetobeabletousethemotiondatainreal-time.SomeoftheenginesUnity3Dforexamplerequireaspecial,oftenexpensivelicensethatallowsadevelopertocreatecustomplug-ins.2.2Requirementsfora3DenvironmentAsystemthatcanbeusedforexperimentsaddsextrarequirements: Lowlevelaccessto3DgraphicsAPIslikeOpenGL Scriptabilityusingasimpleandwellknownlanguage Supportformultipleplatforms2.3RequirementsforaprotocolThereareseveralpossiblewaystoconnectfroma3Denvironmenttoamotioncapturesystem.Onewayistocreateaplug-inthatusesamotioncapturesystemsSDK.Usingthisapproach,thesystemwouldgeneratetheminimumoverheadassociatedwithretrievingmotiondatafromamotioncapturesystem.ThemainproblemwiththisapproachisthatanupdatetothemotioncapturesystemSDKoraswitchtoanothermotioncapturesystemwouldrequireaplug-inupdateandpotentiallyrebuildingthecodethatusestheplug-in.Theplug-insystemwouldhavetobesophisticatedenoughtosupportnewplug-ins/APIswithouttheneedtorebuildthecode.Thischallengeaddssignicantlymorecomplexitytothesystem. 8

PAGE 11

Theotherwayistocreateanetworkingprotocolthatwouldallowa3Denvironmenttoconnecttoandreceivedatafromamotioncapturesystemthroughanetworkconnection.Anewmodulemightbeintroducedtothesystemthatwillbehaveasabridge"betweenamotioncapturesystemandthe3Denvironment.Whilethisapproachaddsoverheadtothecommuni-cation,italsoaddsexibilityinswitchingbetweendierentmotioncapturesystems,changingAPIsasaddingnewfunctionalitydoesnotrequirere-buildinglegacycode,anddistributionofthemotioncapturesystemsandenvironmentsthatuseit.Thissecondapproachwaschosenasthebasisofthisprojectbecauseofitssuperiorexibility.2.3.1SimplicityOneoftherequirementsfortheprotocolissimplicity.Theprotocolmustbesimpleenoughtobeeasilyadaptedtodierentdevelopmentenviron-mentslikeC/C++,Java,C#,Python,andJavaScript.Forexample,abinaryprotocolwouldcreatedicultiesforscriptlanguageslikeJavaScript.JavaScript,bydefault,doesnothaveaconvenientwaytoworkwithbinarydata.Atextbasedprotocol,ontheotherhand,canbeeasilyparsedinanylanguagethathasbasicstringbasedfunctions.Additionally,thereareseveraltextbasedmetalanguagesthatcanbeusedasafoundationfortheprotocol.ThethreemostfrequentlyusedareCSV,XML,andJSON.2.3.2ExtensibilityAnotherrequirementfortheprotocolisextensibility.Addingnewelementstotheprotocolshouldnotbreaktheexistingcode.Achievingthisextensibil-ityusingacustombinaryprotocolrequiresasignicantamountofcoding.UsinganexistingmetalanguagelikeXMLorJSON,ontheotherhand,del-egatesthistothenumerouslibrariesalreadyavailable.Additionally,mostofthemodernlanguageslikeJava,C#,PythonandJavaScripthavebuilt-instandardfunctionstoworkwiththeselanguages.2.4RequirementsforUserInterfaceAdeveloperhasmultipleoptionswhenitcomestotheuserinterface.Theuserinterfacecanbeastandardconventionalinterfacebasedonawt/swingJavaorWindowsFormsC#oritcanbeawebbasedinterface.ThewebinterfacecanbeaRESTful-basedinterfaceorastandardHTMLbasedinterface.Thestandaloneapplication,withaconventionaluserinterface,hasallofthebenetsofanapplicationthathasaccesstoanoperating/window-ingsystemoftheuser`scomputer.Thewebinterface,ontheotherhand, 9

PAGE 12

doesnotrequirethattheuserinstallanyadditionalsoftwareandallowsinstantaneousaccesstothesystem.ARESTful-basedwebuserinterfacewaschosenbecauseofthedis-tributednatureoftheprojectitself. 10

PAGE 13

Chapter3Implementation3.1Details3.1.13DEnvironmentBeforemakinganaldecisionabouta3Dframeworkthatmightbeusedforthisproject,theauthorlookedatseveralexistingoptionsincluding:Ogre3D,JMonkeyEngine2/3,Unity3D,andIdTech1/2/3.Alloftheseframeworksareavailableforfreeandmostofthemcomewithanopensourcecodewhichisveryimportantforavoidinglimitationswhileexperimentingwithdierenttechnologies.Unity3Dwaseliminatedasachoicebecauseofitsdependencyonthedevelopmentenvironment.AdditionallyUnity3Drequiresthepurchaseofaspeciallicensetohaveplug-insupport,whichwouldbeessentialforworkingwithmotioncapturesystems.IdTechengineslookedoverlycomplexandwouldhaverequiredalotoftimetolearnhowtousethem.Ogre3Dlookedpromising,yet,becauseitisconsideredjustarenderingengineanddoesnotprovideaudioandphysicssupport,itwasnotchosenasaplatform.JMonkeyEngine,ontheotherhand,hadbuilt-inphysicssupport,3Daudio,andwaswritteninJavawhichmakesitatrulymulti-platformsolution.Additionally,becauseJMonkeyEngineisaJavabasedproject,addingfeatureslikenetworkingandscriptingisatrivialtask.3.1.2CustomJSON/XMLbasedprotocolTheprotocolforthissystemisbasedontheconceptofdierentstates"ofdierentsources".Forexample,achairina3Dspaceisconsideredasourceandits3Dpositionisconsideredastate.Ifachairismovedfromonepositiontoanother,itsstateischanged.EachstateisrepresentedasanXMLorJSONmessage[6].Eachmessagehasareferencetoitssource.Thisletsthesystemdierentiatemessagesfromdierentsourcesinonestreamofdata.Additionally,messagesmight 11

PAGE 14

becombinedincontainers.Thatletsthesystemhavemorethanonestate"persource".Forexample,acontainermessagemightinclude3Dpositionsforleftandrighthandsofaperson.Inthiscase,thecontainermessagehastheperson"asthesource,whilemessagesthatrepresent3Dpositionsoftheleftandrighthandshavelefthand"andrighthand"astheirsourcesrespectively.3.1.3BasicMessagesThebasicmessagesinclude: ButtonState TransformationState TransformationContainerState MessageState ContainerStateButtonStateisamessagethatrepresentsastateofabutton.Thebuttonmightbepressedorunpressed.Theinterfaceofthisstateincludesonlyonemethod: Listing3.1:ButtonStatemethods public boolean isPressed; TransformationStatecontainsinformationabout2D/3Dtransformationsforanobject.Thetransformationmightberepresentedasanarrayofangles,translationsandscalingorasatransformationmatrix. Listing3.2:TransformationStatemethods public double []getAngles; public double []getTranslation; public double []getScaling; public double [][]getTransformation; TransformationContainerStateisacompactversionofalistofTransfor-mationStatemessages.Forexample,ifa3Dobjectcontainsseveralparts",andeachparthasitsowntransformationparameters,thenallofthedatamayberepresentedasonemessage.TransformationContainerStatehasanarrayofnamesandasequenceofallnumbersofallmatrices. Listing3.3:TransformationContainerStatemethods public int getDimension; public String[]getIdentities; public double []getValues; 12

PAGE 15

MessageStatecontainsatextualmessageassociatedwithasource. Listing3.4:MessageStatemethods public MessagegetContent; ContainerStatecontainsalistofstatesthatareembeddedinsidethismessage. Listing3.5:ContainerStatemethods public ListgetStates; 3.1.4SerializationandDeserializationTheprocessofconvertingjavaobjectsthatrepresentsstates"tothestreathemofbytesserializationandtheprocessofconvertingastreamofbytesbacktothejavaobjectsdeserializationisdelegatedtowellknownlibraries:JavaArchitectureforXMLBindingandJacksonJSONjavalibrary.AllstateclassesareannotatedusingJAXBandJacksonannotations[7].Forexample,todeserializeastreamofbytestojavaobjectsonewouldusethiscode: Listing3.6:De-serializationofastreamofbytes JAXBContextcontext=JAXBContext.newInstancemanifold.states.jaxb.ButtonState. class ,manifold.states.jaxb.ContainerState. class ,manifold.states.jaxb.MessageState. class ,manifold.states.jaxb.TransformationState. class ,manifold.states.jaxb.TransformationContainerState. class ;ContainerStatecontainer=ContainerStatecontext.createUnmarshaller.unmarshalinputStream; TheinputmessageinXMLformat: Listing3.7:XMLmessage UpperChestMidTorsoLowerTorsoLeftUpperLegLeftLowerLegLeftFootRightUpperLegRightLowerLegRightFootNeckHeadRightClavicleRightShoulder 13

PAGE 16

RightUpperArmRightLowerArmRightHandLeftClavicleLeftShoulderLeftUpperArmLeftLowerArmLeftHand
393)]TJ/F34 8.9664 Tf 7.3552 0 Td[(918578791639663)]TJ/F34 8.9664 Tf 7.2085 0 Td[(67)]TJ/F34 8.9664 Tf 7.2085 0 Td[(802899612060001000194)]TJ/F34 8.9664 Tf 7.3552 0 Td[(9796378978189)]TJ/F34 8.9664 Tf 7.2085 0 Td[(86)]TJ/F34 8.9664 Tf 7.2085 0 Td[(6572799941056000100046)]TJ/F34 8.9664 Tf 7.3552 0 Td[(998517099844)]TJ/F34 8.9664 Tf 7.2085 0 Td[(31)]TJ/F34 8.9664 Tf 7.2085 0 Td[(562952998...0001000)]TJ/F34 8.9664 Tf 7.3553 0 Td[(719)]TJ/F34 8.9664 Tf 7.3553 0 Td[(5993542256245935)]TJ/F34 8.9664 Tf 7.3553 0 Td[(554)]TJ/F34 8.9664 Tf 7.3552 0 Td[(647762)]TJ/F34 8.9664 Tf 7.2085 0 Td[(2311540001000)]TJ/F34 8.9664 Tf 7.3553 0 Td[(719)]TJ/F34 8.9664 Tf 7.3553 0 Td[(599354)]TJ/F34 8.9664 Tf 7.2085 0 Td[(63256245935)]TJ/F34 8.9664 Tf 7.3553 0 Td[(725)]TJ/F34 8.9664 Tf 7.3553 0 Td[(647762)]TJ/F34 8.9664 Tf 7.2085 0 Td[(2311580001000 ExampleofaJSONmessage: Listing3.8:JSONMessage f transformationContainer :f source :f Source :f identity : client )]TJ/F34 8.9664 Tf 6.9286 0 Td[(0 gg, values :[)]TJ/F34 8.9664 Tf 7.9663 0 Td[(0.306,)]TJ/F34 8.9664 Tf 7.9662 0 Td[(0.951,)]TJ/F34 8.9664 Tf 7.9662 0 Td[(0.029,0.0,0.0,0.0,0,0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0], dimension :4, originated :1309836222186, identity : main identities :[ UpperChest ]gg 3.1.5MessageRoutingMessageroutingisbuiltaroundRouter"interface.TheRouterinterfacecontainsalistofConsumersandProviders.EachProvidermightcallabroadcast"methodtosendmessages.EachConsumerthatisroutedwithabroadcastingProviderreceivesmessagesintheprocess"method.Thedefaultimplementationoftherouterinterfaceisprovidedinlisting5.1.Theclassdiagramoftheroutingpackageisshowningure3.1.3.1.6ScriptingCreatinga3Dapplicationusuallyisaprocessthatrequiresknowledgeof3DAPI'slikeOpenGLorDirect3D.3Dgameenginessignicantlysimplifythisprocess.Goingfurther,scriptingmakestheprocessofcreatinga3D 14

PAGE 17

applicationevensimplerisscripting.Mostofthemodern3Dgameenginessupportsomesortofscripting.Yettheopensourcefreegameenginethatwaschosenforthisprojectdoesnotsupportscriptingbydefault.Tosolvethisproblemandmakethesystemeasiertouse,JavaScriptingAPIwasintegratedwiththeengine.Thatallowstheusertousebeanshell,browserjs,ejs,freemarker,groovy,jacl,jaskell,java,javascript,jawk,jelly,jep,jexl,jruby,jst,judo,juel,jython,list,ognl,pnuts,scheme,velocity,xpath,andxsltasascriptinglanguagetocustomizeandextendthesystem.Thescriptingisresponsibleforallinteractioninthesystem.Forex-ample,thematricesthatarereceivedfromthemotioncapturesystemareappliedusingJavaScriptlanguage.Thatmakesiteasytoadjustthelogicwithouttheneedtorecompilethejavacodeandcanbedonewithoutin-stallinganyadditionalsoftware.Anexampleofthecodethatappliestransformationmatricesreceivedfromthemotioncapturesystemisprovidedinlisting3.9. Listing3.9:Motiondataappliedtoaskeleton 1 this .update=functiontransformationsf2 if this .control== null f3logger.warning error : MotionModel control is null ;4 return ;5g6varbone= this .control.getSkeleton.getBone Head ;7 if !bonef8logger.warning error : MotionModel does not have bone Head ;9 return ;10g11bone= this .control.getSkeleton.getBone root ;12 if bone!= null f13bone.setBindTransforms14 this .zeroVector,15 this .zeroQuaternion,16 this .identityVector17;18g19 if transformations!= null f20 this .ready= true ;2122varq= new Quaternion;23varvector1= new Vector3f1,1,1;24varvector= new Vector3f;2526 for variintransformationsf27vartransformation=transformations[i];28q.fromRotationMatrix29transformation.getMatrix[0][0],30transformation.getMatrix[0][1],31transformation.getMatrix[0][2],32transformation.getMatrix[1][0],33transformation.getMatrix[1][1], 15

PAGE 18

34transformation.getMatrix[1][2],35transformation.getMatrix[2][0],36transformation.getMatrix[2][1],37transformation.getMatrix[2][2]38;39bone= this .control.getSkeleton.getBonetransformation.getName;40 if !bonef41logger.warning error : cannot find bone : [ +bone+ ] ;42 continue ;43g44bone.setBindTransforms45 this .zeroVector,46 this .zeroQuaternion,47 this .identityVector48;49bone.setUserControl true ;50vector.set51transformation.getMatrix[0][3],52transformation.getMatrix[1][3],53transformation.getMatrix[2][3]54;55bone.setUserTransforms56vector,57q,58vector159;60varcollider= this .colliders[ +transformation.getName];61 if colliderf62collider.physicsNode.setLocalRotationq;63collider.physicsNode.setLocalTranslation64 new Vector3f65transformation.getMatrix[0][3],66transformation.getMatrix[1][3],67transformation.getMatrix[2][3]6869;70g71 if 72 +transformation.getName== MidTorso 73f7475varmatrix= new Matrix4f;76matrix.setRotationQuaternionq;77matrix.setTranslation78 new Vector3f79transformation.getMatrix[0][3]+80 this .getLocalTranslation.get0,81transformation.getMatrix[1][3]+82 this .getLocalTranslation.get1,83transformation.getMatrix[2][3]+84 this .getLocalTranslation.get28586;87 16

PAGE 19

88varrotation= new Quaternion;89 // ROTATION 90rotation.fromAnglesMath.PI,Math.PI,0;9192q.multLocalrotation;9394matrix.setRotationQuaternionq;9596matrix=matrix.mult this .cameraShift;9798 if this .secondView!= null f99varsecondViewCamera= this .secondView.getCamera;100secondViewCamera.setLocationmatrix.toTranslationVector;101g102103 if 104 this .cameraControlEnabled105f106camera.setRotationmatrix.toRotationQuat;107camera.setLocationmatrix.toTranslationVector;108g109g110g111g else 112logger.warning transformations for [ + this .name+ ] is null ;113g 3.1.7WebBasedTechnologiesInsteadofusingastandardgraphicaluserinterface,thesystemusesawebbaseduserinterface.ThismakesitpossibletousethesystemfromdierentcomputersconnectedthoughaTCP/IPnetwork.Additionally,thereisnoneedtoinstalladditionalsoftwaretousethesystem.Java2EnterpriseEditionJ2EEenvironmentwaschosentomakethesystemscalableandportable.J2EEbringsseveralbenetstotheoverallarchitectureofthesystemsuchasaexiblesecuritymodel,scalability,andfreedomofchoiceinservers,tools,andcomponents"[8].ThecorecomponentofthesystemisaStateRoutingServletclassthatroutesthestatemessagesfromdierentpartsofthesystem.Forexample,severalmotioncapturesystemssendtheirdatatotheStateRoutingServletwherethisdataisaggregatedandpreparedforclientsthatneednotknowanythingaboutspecicmotioncapturesystems.Theclientsaccessmotiondatausinglogicnamesoraliases.TheuserinterfacetocontroltheStateRoutingServletisimplementedus-ingRESTfularchitecture.TheWebclientiswritteninJavaScriptlanguage.TheserverpartisrepresentedasaJAX-RSendpoint.JBossandTomcatareusedasanenterprise/webcontainer.Theoveralldeploymentdiagramisshownin3.2. 17

PAGE 20

Figure3.1:RoutingClassDiagram 3.2Challenges3.2.1ComplexityoftheprojectOneofthechallengesforthesystemwaslackofspecicrequirenmentsmainlybecauseoftheresearchnatureoftheproject.Therstversionofthesystemdidnotsupportseveralsourcesofthemotiondataworkingsimul-taneously,datacompression,andmessagerouting.Allthosefeatureswereaddedlaterandrequiredsignicatrefactoringofthedesignofthesystemaswellasthecodeitself.3.2.2ProtocolperformanceissuesSupportingslowconnectionsandlimitingrequiredbandwidthwereanotherchallengesfortheprotocoldesign.Thoseissuesweresolvedbyusingastandarddeate"compressionsupportedbymostoftheHTTPclientsandservers.Table3.1containsaveragepacketsizesfora21-boneskeleton. 18

PAGE 21

Figure3.2:DeploymentDiagram 19

PAGE 22

CSV JSON XML frametype initial live initial live initial live nocompression 3258 3254 2458 5764 1619 1863 deatecompression 399 1064 537 2381 435 834 Table3.1:PacketSizes Usinga24framespersecondframerate,onestreamofmotiondatarequiresapproximatelya60KB/1secconnection.AdditionaloptimizationmightbeachievedbyusingabinaryversionoftheXML[9]. 20

PAGE 23

Chapter4Applications4.1HistoryTheideaofadistributedperformanceisnotnew.Forexample,KitGallowayandSherrieRabinowitzworkedonashowin1975-1977calledTheSatelliteArtsproject".Theexperimentersusedsatellitetelecommunicationasawaytotransmitvideoofseveraldancerslocatedindierentplacesaroundtheworldthatperformedsimultaneously[10]. In2001,theDigitalWorldsInstituteattheUniversityOfFloridapro-ducedanimpressiveandtechnicallycomplexdemostrationattheGlobalSu-percomputingConferencecalledDancingBeyondBoundaries".Itallowedanimationartistis,dancers,musicians,computerscientists,andengineerslocatedindierentcitiesinNorthandSouthAmericatoperformtogethersynchronouslyutilizingtheinternet[11]. Anotherexampleofamorerecentuseoftelecommunicationtoallowdancerstoperforminasharedvirtualenvironmentisastudyconductedin2007byateamofresearchersandartistsfromUniversityofIllinois,Urbana,ILUniversityofCalifornia,Berkeley,CA,andMillsCollege,Oakland,CA[12].Thestudyusedacustombuiltmotioncapturesystemandacommu-nicationmechanismthatallowedseveraldancerstocollaborateinasharedvirtualenvironment.AnotherwaytoallowartiststoperforminasharedvirtualenvironmentwaspresentedbytheDigitalWorldsInstitutein2007duringINGENU-ITY2007"[13].Theartistsusedseveralvideostreamsandanetworkbasedmetronome-liketechnologycalledNetroNome"tosynchronizetheperfor-mance.Thisprojecttriestoextendthepossibilitiesofasharedvirtualenvi-ronmentusedforreal-timeperformancebyprovidinganopenframework.Examplesofapplicationsoftheframeworkthatwasdevelopedasaresultofthisprojectareprovidedbelow. 21

PAGE 24

4.2InternationalSymposiumonMixedandAug-mentedRealityISMAR,Orlando,USA-2009TherstpublicdemonstrationoftheveryrstversionofthesystemwasdoneduringtheInternationalSymposiumonMixedandAugmentedRealityin2009,Orlando,USA.Thedemoincludeda3Dscenewitharoomandadoor.A3Davatarlocatedinsidetheroomwascontrolinreal-timebyapersonfromUniversityOfFlorida.The3Davatarwasabletoopen"thedoorbyslowlyapproachingitwiththelefthand.Thesystemusedprimitivecollisiondetectionalgorithmtotriggertheeventthatopenedthedoor.Thedemoandthesystematthismomentwereveryprimitive,yetitshowedrstchallengesthatthesystemwasfacing:collisiondetectionsystemneededtobeimproved,theprotocolneededtoberedesignedtosupportreconnectionofthesourcesandclientswithoutbreakingthecommunication.4.3EmergingCommunicationsConferenceandAwardseComm,SanFrancisco,California,USA-2010In2010,AttheEmergingCommunicationsConferenceandAwards,SanFrancisco,USA,thesystemincludedanewmodulethatallowedcollisioneventstobedeliveredtoexternalapplicationsusingtheOpenSoundControlprotocol.4.4SIGGRAPH,LosAngeles,California,USA-2010AttheSIGGRAPH,2010thesystemstartedtosupportarstpersonview.VGAgoggleswereusedtodisplayinreal-timethe3Denvironmenttothepersonthatwasinsidethemotioncapturesystem.4.5TheInternationalDigitalMediaArtsAssoci-ationConference,Vancouver,Canada-2010DuringtheInternationalDigitalMediaArtsAssociationConferenceinVan-couverin2010,thesystemworkedsimultaneouslywith4motiondatasourcesdistributedaroundtheworld:Vancouver,Canada,Tokyo,Japan,NewYork,NewYork,andGainesville,Florida.Thesystemwastrackingcollisionsbetweentheavatarsandcommuni-catingtheseeventstoaPureDataPDthatgeneratedsoundeectsusing 22

PAGE 25

5.1channelsaudiosystem.OSCwasusedastheprotocoltocommunicatewithPD.4.6DigitalArtsFestival,Redmond,Washington,USA-2011 23

PAGE 26

Figure4.1:EmergingCommunicationsConferenceAndAwards,SanFran-cisco,California,USA-2010,April21 24

PAGE 27

Figure4.2:SIGGRAPH,LosAngeles,California,USA-2010 25

PAGE 28

Figure4.3:TheInternationalDigitalMediaArtsAssociationConference,Rehearsal,Vancouver,Canada-2010 26

PAGE 29

Figure4.4:TheInternationalDigitalMediaArtsAssociationConference,Rehearsal,Vancouver,Canada-2010 27

PAGE 30

Figure4.5:TheInternationalDigitalMediaArtsAssociationConference,Show,Vancouver,Canada-2010 28

PAGE 31

Figure4.6:DigitalArtsFestival,Redmond,Washington,USA-2011 29

PAGE 32

Chapter5AcknowledgmentsIwouldliketothankDr.JamesOliverioforallofhishelpandsupport.IwouldliketothankDr.JamesOliverio,AngelosBarmpoutis,andBenDeVaneforservinginthecommitteeandprovidingthefeedback.AlsoIwouldliketothankArturoSinclairforthecollaborationonthisproject,PartickPaganoforthecollaborationandencouragement,AngelosBarmpoutisforadvisingmetouse*TeXforwritingthispaper,JosephMurphyfortheinvaluableadviceonchoosingtherightsoftwarepackagesandforthefeedback,GarrettStrobelforreviewingmypaper,andtheDigitalWorldsInstitutefortheresourcesandfriendlyenvironment. 30

PAGE 33

ListofTables3.1PacketSizes............................ 20 31

PAGE 34

ListofFigures3.1RoutingClassDiagram..................... 18 3.2DeploymentDiagram....................... 19 4.1EmergingCommunicationsConferenceAndAwards,SanFran-cisco,California,USA-2010,April21............. 24 4.2SIGGRAPH,LosAngeles,California,USA-2010....... 25 4.3TheInternationalDigitalMediaArtsAssociationConference,Rehearsal,Vancouver,Canada-2010.............. 26 4.4TheInternationalDigitalMediaArtsAssociationConference,Rehearsal,Vancouver,Canada-2010.............. 27 4.5TheInternationalDigitalMediaArtsAssociationConference,Show,Vancouver,Canada-2010................ 28 4.6DigitalArtsFestival,Redmond,Washington,USA-2011.. 29 32

PAGE 35

Listings3.1ButtonStatemethods....................... 12 3.2TransformationStatemethods.................. 12 3.3TransformationContainerStatemethods............ 12 3.4MessageStatemethods...................... 13 3.5ContainerStatemethods..................... 13 3.6De-serializationofastreamofbytes.............. 13 3.7XMLmessage........................... 13 3.8JSONMessage.......................... 14 3.9Motiondataappliedtoaskeleton................ 15 5.1Routerimplementation...................... 36 33

PAGE 36

Bibliography [1] T.MitraandT.Chiueh.Three-dimensionalcomputergraphicsarchi-tecture.CurrentScience,78:838,April2000. [2] M.Valient.Introductionto3dgameengines.April2001. [3] Thecompletemidi1.0detailedspecication.http://www.midi.org/techspecs/midispec.php,1995-2011. [4] D.Phillips.Anintroductiontoosc.http://www.linuxjournal.com/content/introduction-osc,November2008. [5] Motionbuilder-real-time3dcharacteranimationsoftware-autodesk.www.autodesk.com/motionbuilder,2011. [6] Introducingjson.http://www.json.org/. [7] Jsr-000222javatmarchitectureforxmlbindingjaxb.http://jcp.org/aboutJava/communityprocess/mrel/jsr222/index.html,December2009. [8] DesigningEnterpriseApplicationswithJ2EEPlatform,page10.Addison-Wesley,secondedition,2002. [9] Ecientxmlinterchangeworkinggroup.http://www.w3.org/XML/EXI/,April2011. [10] Rhizome,aestheticresearchintelecommunications.http://rhizome.org/editorial/2007/jun/11/aesthetic-research-in-telecommunications/,2011. [11] Dancingbeyondboundaries.http://www.digitalworlds.u.edu/research/DBB/default.asp,November2001. [12] Newdigitaloptionsingeographicallydistributeddancecollaborationswithteeve:Tele-immersiveenvironmentsforeverybody.2007. 34

PAGE 37

[13] Incommontime:Maximumimpact.http://www.digitalworlds.u.edu/research/maximumimpact/default.asp,July2007. 35

PAGE 38

Listing5.1:Routerimplementation 1 public class DefaultRouter implements Routerf23 protected DefaultRouterf4g56 public static RouternewInstancef7 return new DefaultRouter;8g91011 private final Listconsumers= new ArrayList;1213 public void registerConsumerConsumerconsumerf14 synchronized providersf15consumers.addconsumer;16g17g1819 public void unregisterConsumerConsumerconsumerf20 synchronized providersf21 if !consumers.containsconsumer22 throw new IllegalArgumentException23 consumer is not registered 24;2526ListregisteredRoutes= new ArrayList;2728 for ProviderStateproviderState:providerStates.valuesf29 for Consumerc:providerState.getProcessors.keySetf30 if c==consumer31registeredRoutes.addproviderState.getProvider;32g33g3435 for Providerprovider:registeredRoutesf36removeRouteprovider,consumer;37g3839consumers.removeconsumer;4041logger.info consumer [ +consumer+ ] has been unregistered ;42g43g444546 public ListgetAllConsumersf47Listresult= new ArrayList;4849 synchronized providersf50result.addAllconsumers;51g52 36

PAGE 39

53 return result;54g555657 private final Mapproviders= new HashMap;5859 public void registerProviderProviderproviderf6061 synchronized providersf62 if providers.containsKeyprovider.getIdentity63 throw new IllegalArgumentException provider is already registered ;64providers.putprovider.getIdentity,provider;65g6667logger.info provider [ +provider+ ] has been registered ;68g6970 public void unregisterProviderProviderproviderf7172 synchronized providersf7374 if !providers.containsKeyprovider.getIdentity75 throw new IllegalArgumentException provider is not registered ;7677ProviderStateproviderState=providerStates.getprovider;7879 if providerState!= null f80 for Processorprocessor:providerState.getProcessors.values81provider.removeProcessorprocessor;82g8384providers.removeprovider.getIdentity;85providerStates.removeprovider;86g8788logger.info provider [ +provider+ ] has been unregistered ;89g909192 public ListgetAllProvidersf93Listresult= new ArrayList;9495 synchronized providersf96 for Providerprovider:providers.values97result.addprovider;98g99100 return result;101g102103 public ProvidergetProviderByIdentityStringidentityf104105 synchronized providersf106 return providers.getidentity; 37

PAGE 40

107g108109g110111112 private MapproviderStates=113 new HashMap;114115116 public void addRouteProviderprovider, final Consumerconsumerf117118 if provider== null 119 throw new IllegalArgumentException cannot create route with null provider ;120121 if consumer== null 122 throw new IllegalArgumentException cannot create route with null consumer ;123124 synchronized providersf125126ProviderStateproviderState=providerStates.getprovider;127128 if providerState== null f129providerState= new ProviderStateprovider;130providerStates.putprovider,providerState;131g132133 if providerState.getProcessors.containsKeyconsumer134 throw new IllegalArgumentException route already exists ;135136Processorprocessor= new Processorf137 public void processState[]states,Providerproviderf138consumer.processstates,provider;139g140g;141142providerState.getProcessors.putconsumer,processor;143144provider.addProcessorprocessor;145g146147logger.info route [ +provider+ )]TJ/F35 8.9664 Tf 5.8569 0 Td[(> +consumer+ ] has been added ;148g149150 public void removeRouteProviderprovider,Consumerconsumerf151152 synchronized providersf153154ProviderStateproviderState=providerStates.getprovider;155156 if providerState== null 157 throw new IllegalArgumentException route does not exists 1 ;158159Processorprocessor=providerState.getProcessors.getconsumer;160 38

PAGE 41

161 if processor== null 162 throw new IllegalArgumentException route does not exists 2 ;163164provider.removeProcessorprocessor;165166providerState.getProcessors.removeconsumer;167168g169170logger.info route [ +provider+ )]TJ/F35 8.9664 Tf 5.8569 0 Td[(> +consumer+ ] has been removed ;171g172173 public Map>getRoutesf174Map>result= new HashMap>;175176 synchronized providersf177 for Map.Entryentry:providerStates.entrySetf178Listconsumers= new ArrayList;179consumers.addAllentry.getValue.getProcessors.keySet;180result.putentry.getKey,consumers;181g182g183184 return result;185g186187 protected class ProviderStatef188189 private final Providerprovider;190191 public ProviderStateProviderproviderf192 this .provider=provider;193g194195 public ProvidergetProviderf196 return provider;197g198199200 private final Mapprocessors=201 new HashMap;202203 public MapgetProcessorsf204 return processors;205g206g207208g 39