<%BANNER%>

Information Geometry for Shape Analysis

Permanent Link: http://ufdc.ufl.edu/UFE0022484/00001

Material Information

Title: Information Geometry for Shape Analysis Probabilistic Models for Shape Matching and Indexing
Physical Description: 1 online resource (129 p.)
Language: english
Creator: Peter, Adrian
Publisher: University of Florida
Place of Publication: Gainesville, Fla.
Publication Date: 2008

Subjects

Subjects / Keywords: analysis, assignment, charvat, density, description, divergence, estimation, fisher, gaussian, geometry, havrda, hellinger, information, length, linear, matching, metric, minimum, mixture, model, rao, root, selection, shape, square, wavelets
Electrical and Computer Engineering -- Dissertations, Academic -- UF
Genre: Electrical and Computer Engineering thesis, Ph.D.
bibliography   ( marcgt )
theses   ( marcgt )
government publication (state, provincial, terriorial, dependent)   ( marcgt )
born-digital   ( sobekcm )
Electronic Thesis or Dissertation

Notes

Abstract: The study of shape analysis is a core field in computer vision. It is a fundamental building block of higher level cognitive tasks such as recognition. This research introduces novel approaches to basic shape analysis tasks, including shape matching and defining metrics for shape similarity. Our investigations into these methods yielded two supporting statistical tools that have general applicability outside the realm of shape analysis: a new wavelet density estimation procedure and a geometrically motivated model selection criterion to select the wavelet's multiscale decomposition levels. All of the derived techniques are theoretically grounded in the framework of information geometry. Information geometry is an emerging math discipline that applies differential geometry to space of probability distributions. This work will for the first time illustrate a systematic approach to applying information geometry to shape analysis. Our basic approach to shape analysis is simple: represent shapes as probability densities, then use the intrinsic geometry of the space of densities to establish geodesics between shapes. We can obtain valid intermediate densities (shapes) by walking along the geodesics and the length of the geodesic immediately gives us a similarity measure between shapes. We always assume an unstructured, point-set representation for the underlying shape. Hence, unlike many contemporary methods, there are no topological restrictions (like requiring shapes to be closed curves) on our shape models. We illustrate these concepts by using two types of models to represent the densities: Gaussian mixture models and wavelet densities. Our development of a wavelet-density, shape model also resulted in a new density estimation procedure. We expand the square root of the density in a multiscale, wavelet basis and then obtain a bona fide density by squaring the expansion. This new method estimates the coefficients of the wavelet expansion using a constrained maximum likelihood objective. It is shown that under this representation wavelet densities are essentially points on a unit hypersphere. The choice of the number of decomposition levels density estimation is determined using a model selection framework. We use the geometry of the space to apply the MDL criterion that selects the best model among set of competing ones by judiciously balancing a model's accuracy and complexity.
General Note: In the series University of Florida Digital Collections.
General Note: Includes vita.
Bibliography: Includes bibliographical references.
Source of Description: Description based on online resource; title from PDF title page.
Source of Description: This bibliographic record is available under the Creative Commons CC0 public domain dedication. The University of Florida Libraries, as creator of this bibliographic record, has waived all rights to it worldwide under copyright law, including all related and neighboring rights, to the extent allowed by law.
Statement of Responsibility: by Adrian Peter.
Thesis: Thesis (Ph.D.)--University of Florida, 2008.
Local: Adviser: Rangarajan, Anand.
Electronic Access: RESTRICTED TO UF STUDENTS, STAFF, FACULTY, AND ON-CAMPUS USE UNTIL 2009-08-31

Record Information

Source Institution: UFRGP
Rights Management: Applicable rights reserved.
Classification: lcc - LD1780 2008
System ID: UFE0022484:00001

Permanent Link: http://ufdc.ufl.edu/UFE0022484/00001

Material Information

Title: Information Geometry for Shape Analysis Probabilistic Models for Shape Matching and Indexing
Physical Description: 1 online resource (129 p.)
Language: english
Creator: Peter, Adrian
Publisher: University of Florida
Place of Publication: Gainesville, Fla.
Publication Date: 2008

Subjects

Subjects / Keywords: analysis, assignment, charvat, density, description, divergence, estimation, fisher, gaussian, geometry, havrda, hellinger, information, length, linear, matching, metric, minimum, mixture, model, rao, root, selection, shape, square, wavelets
Electrical and Computer Engineering -- Dissertations, Academic -- UF
Genre: Electrical and Computer Engineering thesis, Ph.D.
bibliography   ( marcgt )
theses   ( marcgt )
government publication (state, provincial, terriorial, dependent)   ( marcgt )
born-digital   ( sobekcm )
Electronic Thesis or Dissertation

Notes

Abstract: The study of shape analysis is a core field in computer vision. It is a fundamental building block of higher level cognitive tasks such as recognition. This research introduces novel approaches to basic shape analysis tasks, including shape matching and defining metrics for shape similarity. Our investigations into these methods yielded two supporting statistical tools that have general applicability outside the realm of shape analysis: a new wavelet density estimation procedure and a geometrically motivated model selection criterion to select the wavelet's multiscale decomposition levels. All of the derived techniques are theoretically grounded in the framework of information geometry. Information geometry is an emerging math discipline that applies differential geometry to space of probability distributions. This work will for the first time illustrate a systematic approach to applying information geometry to shape analysis. Our basic approach to shape analysis is simple: represent shapes as probability densities, then use the intrinsic geometry of the space of densities to establish geodesics between shapes. We can obtain valid intermediate densities (shapes) by walking along the geodesics and the length of the geodesic immediately gives us a similarity measure between shapes. We always assume an unstructured, point-set representation for the underlying shape. Hence, unlike many contemporary methods, there are no topological restrictions (like requiring shapes to be closed curves) on our shape models. We illustrate these concepts by using two types of models to represent the densities: Gaussian mixture models and wavelet densities. Our development of a wavelet-density, shape model also resulted in a new density estimation procedure. We expand the square root of the density in a multiscale, wavelet basis and then obtain a bona fide density by squaring the expansion. This new method estimates the coefficients of the wavelet expansion using a constrained maximum likelihood objective. It is shown that under this representation wavelet densities are essentially points on a unit hypersphere. The choice of the number of decomposition levels density estimation is determined using a model selection framework. We use the geometry of the space to apply the MDL criterion that selects the best model among set of competing ones by judiciously balancing a model's accuracy and complexity.
General Note: In the series University of Florida Digital Collections.
General Note: Includes vita.
Bibliography: Includes bibliographical references.
Source of Description: Description based on online resource; title from PDF title page.
Source of Description: This bibliographic record is available under the Creative Commons CC0 public domain dedication. The University of Florida Libraries, as creator of this bibliographic record, has waived all rights to it worldwide under copyright law, including all related and neighboring rights, to the extent allowed by law.
Statement of Responsibility: by Adrian Peter.
Thesis: Thesis (Ph.D.)--University of Florida, 2008.
Local: Adviser: Rangarajan, Anand.
Electronic Access: RESTRICTED TO UF STUDENTS, STAFF, FACULTY, AND ON-CAMPUS USE UNTIL 2009-08-31

Record Information

Source Institution: UFRGP
Rights Management: Applicable rights reserved.
Classification: lcc - LD1780 2008
System ID: UFE0022484:00001


This item has the following downloads:


Full Text

PAGE 1

INFORMATIONGEOMETRYFORSHAPEANALYSIS:PROBABILISTICMODELSFORSHAPEMATCHINGANDINDEXINGByADRIANM.PETERADISSERTATIONPRESENTEDTOTHEGRADUATESCHOOLOFTHEUNIVERSITYOFFLORIDAINPARTIALFULFILLMENTOFTHEREQUIREMENTSFORTHEDEGREEOFDOCTOROFPHILOSOPHYUNIVERSITYOFFLORIDA2008 1

PAGE 2

c2008AdrianM.Peter 2

PAGE 3

ToJancy Manywomendonoblethings,butyousurpassthemall.Prov.31:29 3

PAGE 4

ACKNOWLEDGMENTS Sodonotfear,forIamwithyou;donotbedismayed,forIamyourGod.Iwillstrengthenyouandhelpyou;Iwillupholdyouwithmyrighteousrighthand.Isaiah41:10JesusChrist,myLordandSavior,isthesourceofmystrengthandgenitorofinspirationineveryaspectofmylife.ThroughthecourseofthisPhDIhaveexperiencedthecounselofGodguidingmethroughallthelatenights,momentsofconfusion,doubtandfrustration.ThankyouforbelievinginmewhenIoftendidn't.ThankyouforshiningthelightwhenIwasintheabyss.ThankyouforansweringprayerwhenIcouldn'tevenspeakthewords.Thankyou!TherearenotenoughwordstodescribethegratitudeIhaveformywife,Jancy.Sheletmefollowmyheartevenwhenitmeantriskingeverything.Thankyouforencouragingmeduringthetoughdaysandputtingthe"smackdown"whenlazinesscreptin.YouareamazingandIloveyou.Elijahthankyouforyourgenuinesmileandbeingthebestsonanyonecouldhavewishedfor.Iamindebtedtomyparents(DadandAmma)forallthesacricestheyhadtomakejusttogiveustheopportunitytomakeitinabetterplace.Thankyoufortemperingyourdreamssoanother'scouldtakeight.Thankyoutomysister,niece,aunts,uncles,cousinsandin-lawswithoutyoursupportandinspirationIwouldnothavestayedthecourse.Tomyfriends,thankyoufortheconstantprayers.Tomyadvisor,AnandRangarajan,thankyouforbeingmoreanddemandingmore.Youwentaboveandbeyondwhatyouhadto:openingupyourhomeandfamily,constantlyansweringthelatenightcallsandchallengingmyintellecttoreachnewheights.Iamhappytohavegainedalifelongfriend.IwanttothankDr.Vemuriforpushingmetobebetterandnotacceptinganythingless.Dr.Principe,thankyouforworkingwithAnandandItomakethePhDpossible.ThanksgoouttoDr.Hoforallthedierentialgeometrylessonsandforbeinggenuine.Finally,mysinceregratitudegoestoDr.Tenaliforreleasingmypassionformath. 4

PAGE 5

TABLEOFCONTENTS page ACKNOWLEDGMENTS ................................. 4 LISTOFTABLES ..................................... 7 LISTOFFIGURES .................................... 8 ABSTRACT ........................................ 10 CHAPTER 1INFORMATIONGEOMETRYFORVISIONANDLEARNING ........ 12 1.1Introduction ................................... 12 1.2BackgroundonInformationGeometry .................... 15 2LANDMARKSHAPEANALYSIS:UNIFYINGSHAPEREPRESENTATIONANDDEFORMATION ............................... 19 2.1Introduction .................................. 19 2.2TheRepresentationModel:FromLandmarkstoMixtures .......... 24 2.3TheDeformationModel:RiemannianMetricsfromInformationMatricesofMixtures ................................... 26 2.4ExperimentalResultsandAnalysis ...................... 34 2.5Discussion .................................... 39 3WAVELETDENSITYESTIMATION ....................... 49 3.1Introduction ................................... 49 3.2WaveletTheoryanditsApplicationtoDensityEstimation ......... 52 3.3MaximumLikelihoodforWaveletDensityEstimation ............ 57 3.4ExperimentalResults .............................. 62 3.5Discussion .................................... 68 4SLIDINGWAVELETSFORINDEXINGANDRETRIEVAL .......... 74 4.1Introduction ................................... 74 4.2ShapeL'neRouge ............................... 77 4.3Experiments ................................... 84 4.4Discussion .................................... 88 5MDLFORWAVELETDENSITYESTIMATION ................. 92 5.1Introduction .................................. 92 5.2MotivationandRelatedWork ......................... 93 5.3MDLandtheGeometryofSquare-RootWaveletDensities ......... 98 5.4Experiments ................................... 102 5

PAGE 6

5.5Discussion .................................... 104 6CONCLUSIONS ................................... 115 6.1Contributions .................................. 115 6.2FutureWork ................................... 117 REFERENCES ....................................... 119 BIOGRAPHICALSKETCH ................................ 129 6

PAGE 7

LISTOFTABLES Table page 2-1Pairwiseshapedistances. ............................... 46 3-11Ddensityestimation ................................ 69 3-22Ddensityestimation ................................ 69 3-3Mutualinformationregistration ........................... 70 3-4Hellingerdivergenceshapealignment ........................ 70 4-1MPEG-7recognitionrate .............................. 91 5-1ModelselectionusingDaubechiesfamily ...................... 106 5-2ModelselectionusingSymletfamily ......................... 107 5-3ModelselectionusingCoietfamily ......................... 108 7

PAGE 8

LISTOFFIGURES Figure page 1-1Informationgeometryofparametricdensities ..................... 18 2-1Examplesoftheprobabilisticrepresentationmodel ................ 40 2-2Intrinsicversusextrinsic ............................... 41 2-3Bendingofstraightlinewith21landmarks ..................... 41 2-4Intermediatelandmarktrajectories ......................... 42 2-5Rotationofsquarerepresentedwithfourlandmarks ................ 42 2-6Fishshapeswithdieringtopologies ........................ 43 2-7-orderentropymetricdeformationanalysis .................... 43 2-8Landmarkdieomorphismsdeformationanalysis .................. 44 2-9Corpuscallosumshapes ............................... 44 2-10Hierarchicalclusteringwithdierentmetrics .................... 45 3-11DDensityestimationcomparison ........................... 71 3-22DDensityestimationcomparison ......................... 72 3-3Registrationusingmutualinformation ....................... 72 3-4Jointdensityestimationfromtwoimages ...................... 73 3-5Fishpointsets .................................... 73 3-62Ddensityestimatedfromshpointset ...................... 73 4-1EstimatedwaveletdensitiesfromMPEG-7data .................. 89 4-2Hypersphereofdensities ............................... 89 4-3Localnon-rigideectsandtheneedforlinearassignment ............. 90 4-4Eectsofonlinearassignment .......................... 90 4-5D2shapedistributionsusingwaveletdensitiesestimators ............. 91 5-1Surfaceareaofunithypersphere ........................... 109 5-2Riemannianvolumecomparisons,ln(VS)versusln(V^) .............. 109 5-3Multiresolutionanalysisfordensityestimation ................... 110 8

PAGE 9

5-4ModelselectionusingMDL-3versusMDL-2 .................... 111 5-5ModelselectionusingMDL-3versusAIC ...................... 112 5-6ModelselectionusingMDL-3versusBICandMSE ................ 113 5-7Modelselectionon2Ddensities ........................... 114 9

PAGE 10

AbstractofDissertationPresentedtotheGraduateSchooloftheUniversityofFloridainPartialFulllmentoftheRequirementsfortheDegreeofDoctorofPhilosophyINFORMATIONGEOMETRYFORSHAPEANALYSIS:PROBABILISTICMODELSFORSHAPEMATCHINGANDINDEXINGByAdrianM.PeterAugust2008Chair:AnandRangarajanMajor:ElectricalandComputerEngineeringThestudyofshapeanalysisisacoreeldincomputervision.Itisafundamentalbuildingblockofhigherlevelcognitivetaskssuchasrecognition.Thisresearchintroducesnovelapproachestobasicshapeanalysistasks,includingshapematchinganddeningmetricsforshapesimilarity.Ourinvestigationsintothesemethodsyieldedtwosupportingstatisticaltoolsthathavegeneralapplicabilityoutsidetherealmofshapeanalysis:anewwaveletdensityestimationprocedureandageometricallymotivatedmodelselectioncriteriontoselectthewavelet'smultiscaledecompositionlevels.Allofthederivedtechniquesaretheoreticallygroundedintheframeworkofinformationgeometry.Informationgeometryisanemergingmathdisciplinethatappliesdierentialgeometrytospaceofprobabilitydistributions.Thisworkwillforthersttimeillustrateasystematicapproachtoapplyinginformationgeometrytoshapeanalysis.Ourbasicapproachtoshapeanalysisissimple:representshapesasprobabilitydensities,thenusetheintrinsicgeometryofthespaceofdensitiestoestablishgeodesicsbetweenshapes.Wecanobtainvalidintermediatedensities(shapes)bywalkingalongthegeodesicsandthelengthofthegeodesicimmediatelygivesusasimilaritymeasurebetweenshapes.Wealwaysassumeanunstructured,point-setrepresentationfortheunderlyingshape.Hence,unlikemanycontemporarymethods,therearenotopologicalrestrictions(likerequiringshapestobeclosedcurves)onourshapemodels.Weillustrate 10

PAGE 11

theseconceptsbyusingtwotypesofmodelstorepresentthedensities:Gaussianmixturemodelsandwaveletdensities.Ourdevelopmentofawavelet-density,shapemodelalsoresultedinanewdensityestimationprocedure.Weexpandthesquarerootofthedensityinamultiscale,waveletbasisandthenobtainabonadedensitybysquaringtheexpansion.Thisnewmethodestimatesthecoecientsofthewaveletexpansionusingaconstrainedmaximumlikelihoodobjective.Itisshownthatunderthisrepresentationwaveletdensitiesareessentiallypointsonaunithypersphere.Thechoiceofthenumberofdecompositionlevelsdensityestimationisdeterminedusingamodelselectionframework.WeusethegeometryofthespacetoapplytheMDLcriterionthatselectsthebestmodelamongsetofcompetingonesbyjudiciouslybalancingamodel'saccuracyandcomplexity. 11

PAGE 12

CHAPTER1INFORMATIONGEOMETRYFORVISIONANDLEARNING 1.1IntroductionRecognitionisafundamentalandkeycomponentofcognitivedevelopment;arelativelyeasytaskforhumansyetonethatcontinuestobediculttomodelconsistentlyandconsequentlyresultsindisparatealgorithmicsolutions.Thefocusofthisdissertationistodevelopaprobabilisticframeworkforrecognition,onethatcouplesboththeshaperepresentationanditsmeasureofsimilaritytoothershapes.Themodelsweproposeprovideconsiderablymoreexibilityandprincipledalgorithmicsolutionsascomparedtothecurrentstateoftheart.Ourapproachesareuniedunderthetheoreticalumbrellaofinformationgeometryanemergingmathdisciplinethatappliesdierentialgeometrytoprobabilitydistributions.Theneedforrobustobjectrecognitionappearsinamyriadofapplications,acrossmultipledisciplines.Inremotesensing,itisthemaincomponentinautomatictargetrecognition(ATR)systems[ 1 ].Formedicalimagingandmoregeneralcontentbasedimageretrieval(CBIR)systems[ 2 4 ],ecientrecognitionisvitaltoprovidingfastandaccuratematchestouser-speciedqueries.Inbiologicalapplications,objectrecognitionisusedtostudymorphologicaleectsandcellstructures[ 5 7 ].Andautonomousnavigationwouldbeimpossibleforvehiclesandrobotswithoutrobustrecognitionsystems[ 8 12 ].Itcanbearguedthattheformulationoftherecognitionproblembeginswiththerepresentationmodel[ 13 ].Themodelchosentorepresentobjectsisparamounttoallsubsequentanalysisoneseekstoextractfromthedata.Impactingnotionsofsimilaritybetweenobjects,neighborhoodrelationships(hencecategorizations),datastructuresforstorageandretrieval,andnallyeectingthealgorithmicimplementationofanalysistechniques.Asabrieftaxonomy,objectrepresentationsrangefromunstructuredpoint-sets[ 14 15 ],weightedgraphs[ 16 ]andincludecurves[ 17 ],surfaces[ 18 ]andothergeometricmodels.Aderivativeofthesegeometricschemes,hierarchicalmodels[ 19 20 ]haverecently 12

PAGE 13

gainedinpopularity.Theyseektoachievearepresentationthatsimultaneouslycaptureslocalandglobalpropertiesofobjects.However,inherenttothesemodelsistheneedtodeneandcomputealocaltopology(neighborhoodrelationship)whichisusedtoconstructtheglobalshape.Thoughsomehavedemonstratedhighrecognitionrates,thiscomesattheexpenseofsacricingrepresentationcapabilities,restrictingtheglobaltopologyoftheobjectstobesilhouettesorclosedcurves/surfaces.Formanyinterestingandcomplexproblemsineldssuchasmolecularbiologyorremotesensing,thiscanseverelyconstrainthedescriptivecapabilitiesandoftenrenderthetechniquevoid.Whatisneededisaexible(withouttopologicalrestrictions)representationmodelcapableofcapturinganobject'slocalandglobalproperties.Ourapproachpreciselyaddressesthesedesiresby:(1)eliminatingtopologyconstraintsbysimplyrepresentingobjectsasprobabilitydensitiesestimatedfromunstructuredpointsetsand(2)endowingthesedensitieswithlocalandglobaldescriptivepowerbyexpandingtheminamultiscalewaveletbasisorGaussianmixturemodel(GMM).Anaturalby-productofthisrepresentationisthatthedensitiesvisuallyresembletheshapes(readilyapparentin2D,seeChapter2andChapter4).Inadditiontothesepowerfuldescriptivecapabilities,ageometricanalysisofthespaceofprobabilitydensities(andhencetheshapes)yieldsageodesicdistancebetweenobjects.Thismarryingoftheobjectrepresentationanditssimilaritymeasureistheoreticallymotivatedbyinformationgeometry(see 1.2 ).Thisworkmakesseminalcontributionsforapplyinginformationgeometrytoshapeanalysisforthersttimeillustratingtheintrinsicgeometriesassociatedwithrepresentingshapesasprobabilitydensities.Therestofthedissertationisorganizedasfollows.Theremainderofthischapterprovidesabriefintroductiontoinformationgeometry.Thebasicscoveredinthissectionshouldsuceforfollowingthedevelopmentsinsubsequentchapters.Sinceourapplicationsrangefromshape/imagematchingtodensityestimationandtomodel 13

PAGE 14

selection,wehavechosentocoverrelatedandpriorworkstoeachoftheseareasintheirdedicatedchapters.InChapter2wedescribetheuseofparametricmodelsforshaperepresentationanddeformation.Inparticular,weillustratehowGMMscanbeusedtorepresentlandmarkshapesandthenusetheintrinsicgeometryofthespaceofmixturestoestablishawarpbetweentheshapesandcomputeasimilaritymeasure.Theframeworkforthersttimeuniesshaperepresentationanddeformation.Anothersalientresultinthischapteristhedevelopmentofanewclosed-formRiemannianmetricforGMMs.Experimentalresultsareshownonavarietyof2Dlandmarkshapecongurationswithcomparativeanalysistoseveralothermethods.Ourinvestigationsintotheuseofparametricmixturemodelsprovidedthemotivationnecessarytorethinktherepresentationofthedensity;ratherthanusemixturemodels,weexpandthedensityinamultiscale,orthonormalwaveletbasis.Inordertoeectivelyusethismultiscalerepresentationforshapeanalysis,werstdevelopanewmethoddensityestimationtechniquetocalculatethecoecientsofthewaveletbasisexpansion.ThisisthesubjectmatterofChapter3.Ourframeworkisbasedonaconstrained,maximumlikelihood(ML)objectivewhichdirectlyestimatesp pandobtainsthedesireddensityasp p2.WegoontoshowhowtheMLobjectivecanbeminimizedbyamodiedNewton'smethod.Thisalgorithmisonceagainmotivatedinthecontextofinformationgeometry.Chapter4utilizesthewaveletdensityestimationfromChapter3todevelopanewshapeindexingandretrievalframework.Thecoreideaistorepresentpoint-setshapesasthesquarerootofprobabilitydensitiesexpandedinawaveletbasis.Wethenusethisrepresentationtodevelopanaturalsimilaritymetricthatrespectsthegeometryofthesedistributions,i.e.underthewaveletexpansiondistributionsarepointsonaunithypersphereandthedistancebetweendistributionsisgivenbytheseparatingarclength.Theprocessusesalinearassignmentsolverfornon-rigidalignmentbetweendensitiespriortomatching;thishastheconnotationofslidingwaveletcoecientsakintothesliding 14

PAGE 15

blockpuzzleL'neRouge.Giventhisrationalization,thetechniqueiscalledShapeL'neRouge.WeillustratetheutilityofthisframeworkbymatchingshapesfromtheMPEG-7datasetandprovidecomparisonstoothersimilaritymeasures,suchasEuclideandistanceshapedistributions.Chapter5returnstothewaveletdensityestimatorofChapter3toaddressthequestion:Howdoesoneselectthelevelsofthemultiscalewaveletdecompositioninthecontextofdensityestimation?Thisisourmostcurrentresearchfocus.Weanalyzethisissuebydevelopingtherazor[ 21 ],acriterionforselectingthebestmodelfromagroupofcompetingformulations,onthehypersphereofwaveletdensities.Wewillemploytheconstructsofinformationgeometrytocharacterizethespaceofthedistributionsunderourwaveletmodelandsubsequentlyillustratehowthehighlystructuredspaceofourmodelswillenableclosed-formcomputationsofrazorwhichhavepreviouslyeludedalmostallothermodels.Finally,Chapter6summarizesourcontributionsandoutlinesdirectionsforfutureresearch. 1.2BackgroundonInformationGeometryMyresearchandmathematicalinterestsareintheemergingdisciplineofinformationgeometry.Informationgeometryutilizesdierentialgeometrytoperformanalysisonthespaceofprobabilitydistributions[ 22 23 ].Thedistributionsthemselvesaretreatedasindividualelementsonamanifoldandhenceahostoftheoriesfromdierentialgeometrynowbecomerelevant.ItwasRao[ 24 ]whorstestablishedthattheFisherinformationmatrix(FIM)satisesthepropertiesofametriconaRiemannianmanifold,forthisreasontheFIMisreferredtoastheFisher-Raometricwheneveritisusedinthisgeometricmanner.TheFisherinformationmatrixarisesfrommulti-parameterdensities,wherethe(i;j)entryofthematrixisgivenby gij()=p(xj)@ @ilogp(xj)@ @jlogp(xj)dx:(1) 15

PAGE 16

TheFisher-Raometrictensor( 1 )isanintrinsicmeasure,allowingustoanalyzeanite,n-dimensionalstatisticalmanifoldMwithoutconsideringhowMsitsinanR2n+1space[ 25 ].Inthisparametric,statisticalmanifold,p2Misaprobabilitydensitywithitslocalcoordinatesdenedbythemodelparameters.Forexample,abivariateGaussiandensitycanberepresentedasasinglepointon4-dimensionalmanifoldwithcoordinates=(1=1;2=2;3=1;4=2)T,whereasusualtheserepresentthemeanandstandarddeviationofthedensity.(Thesuperscriptlabelingofcoordinatesisusedtobeconsistentwithdierentialgeometryreferences.)Dierentvaluesfor(1;2;1;2)Tindexdierenttwo-dimensionalGaussiandistributions.Manyofthecommonmetricsonprobabilitydensities(e.g.Kullback-Leibler,Jensen-Shannon,etc.)canbewrittenintermsoftheFisher-Raometricgiventhatthedensitiesareclose[ 23 ].Forexample,theKullback-Leibler(KL)divergencebetweentwoparametricdensitieswithparametersand+respectively,isproportionalto D(p(xj+)jjp(xj))1 2()Tg:(1)Inotherwords,theKLdivergenceisequalto,withinaconstant,aquadraticformwiththeFisherinformationmatrixgplayingtheroleoftheHessian.Inshort,theFIMplaysthegeometricroleofametrictensorandcansubsequentlybeusedtomeasuredistancesbetweendistributions.(Itisworthnotingthatitispossibletodeneotherprobabilistic,Riemannianmetricsonthespaceofdistributions[ 26 28 ].)Informationgeometryincorporatesseveralotherdierentialgeometryconceptsinthesettingofprobabilitydistributionsanddensities.Besideshavingametric,wealsorequiretheconstructofconnectionstomovefromonetangentspacetoanother.TheconnectionsarefacilitatedbycomputingChristoelsymbolsoftherstkind,k;ijdef=1 2n@gik @j+@gkj @i@gij @ko,whichrelyonthepartialderivativesofthemetrictensor.ItisalsopossibletocomputeChristoelsymbolsofthesecondkindwhichinvolvetheinverseofthemetrictensor.Sinceallanalysisisintrinsic,i.e.onthesurfaceofthe 16

PAGE 17

manifold,ndingtheshortestdistancebetweenpointsonthemanifoldamountstondingageodesicbetweenthem.Asdiscussedinsequel,probabilitydensitieswillrepresentshapes(moregenerallyobjects)andhencecorrespondtopointsontheprobabilisticmanifold.Figure 1-1 illustratesthisoverallidea.Severalshapesrepresentedusingdensities(mixturesorwaveletexpansions)areconsideredaspointsonmanifoldandtheparametersofdensitiesservingastheindexingcoordinates.Wecanusethemetrictensortondageodesicbetweenshapesandhenceperformstatisticalanalysis(likendingmeansandvariances)onthespaceofprobabilitydensities.Onceageodesicisfoundbetweendensities,walkingalongthegeodesicwillgiveusintermediatedensitieswhichalsocorrespondtointermediateshapes!Thegeodesiclengthwillgiveusanintrinsicshapedistance. 1.2.1RecentApplicationsTheapplicationofinformationgeometryhassteadily(albeitslowly)gainedtractioninthecomputervisionandmachinelearningcommunities.Formachinelearning,ithasbeenappliedtoparametertuninginneuralnetworks[ 29 ]andforimprovinglearningratesthroughtheuseofthenaturalgradient[ 30 ].In[ 21 ]theauthorshaveusedittoprovideamoreintuitive,geometricexplanationofmodelselectioncriteriasuchastheminimumdescriptionlength(MDL)criterion.Morerecently,ithasbeenappliedtometriclearningfortextclassication[ 31 ].Toourknowledge,thereareonlyafewrecentusesoftheFisher-Raometricforcomputervisionrelatedanalyses(excludingourown).Maybank[ 32 33 ],utilizesFisherinformationtoanalyzeprojectivetransformationsoftheline.Mioetal.[ 34 ],applynon-parametricFisher-Raometricsforimagesegmentation.Lengletetal.[ 35 ]successfullydemonstratedtheuseoftheFisher-Raometriconmultivariatenormaldensitiesintheanalysisofdiusiontensorimagingdata.Finally,Srivastavaetal.[ 36 ]havestudiedapplicationsofthenon-parametricFisher-Raometrictocurve-basedshapeclassication. 17

PAGE 18

Figure1-1.Informationgeometryofparametricdensities.ProbabilitydensitiesareindexedbytheirparameterswhichserveascoordinatesforpointsonaRiemannianmanifold(cartoondepictioninabovegure).Usingthemetrictensorgi;jitispossibletoobtainageodesicbetweenthedensities;thusgivingusaintrinsicdistancebetweendensitiesandintermediatedensitiesalongthegeodesic.Theoverarchingthemeofourproposalistorepresentobjects(point-setshapes)asdensitiesandthenconductanalysis(recognitionandregistration)thatrespectsthegeometryofthemanifold. 18

PAGE 19

CHAPTER2LANDMARKSHAPEANALYSIS:UNIFYINGSHAPEREPRESENTATIONANDDEFORMATION 2.1IntroductionShapeanalysisisakeyingredienttomanycomputervisionandmedicalimagingapplicationsthatseektostudytheintimaterelationshipbetweentheformandfunctionofnatural,cultural,medicalandbiologicalstructures.Inparticular,landmark-baseddeformablemodelshavebeenwidelyused[ 37 ]inquantiedstudiesrequiringsizeandshapesimilaritycomparisons.Shapecomparisonacrosssubjectsandmodalitiesrequirethecomputationofsimilaritymeasureswhichinturnrelyuponnon-rigiddeformationparameterizations.Almostallofthepreviousworkinthisareausesseparatemodelsforshaperepresentationanddeformation.Ourprincipalgoalistoshowthatshaperepresentationsbegetshapedeformationparameterizations[ 28 38 ].Thisunexpectedunicationdirectlyleadstoashapecomparisonmeasure.Abrief,cross-cuttingsurveyofexistingworkinshapeanalysisillustratesseveraltaxonomiesandsummaries.ShapedeformationparameterizationsrangefromProcrusteanmetrics[ 39 ]tospline-basedmodels[ 40 41 ],andfromPCA-basedmodesofdeformation[ 42 ]tolandmarkdieomorphisms[ 43 44 ].Shaperepresentationsrangefromunstructuredpoint-sets[ 14 45 ]toweightedgraphs[ 16 ]andincludecurves[ 46 ],surfaces[ 18 ]andothergeometricmodels.Theseadvanceshavebeeninstrumentalinsolidifyingtheshapeanalysislandscape.However,onecommonalityinvirtuallyallofthispreviousworkistheuseofseparatemodelsforshaperepresentationanddeformation.Forexample,thisdecouplingbetweenshaperepresentationanddeformationisevidentinthespline-based,planar Contentforthischapterhasbeenreprintedwithpermissionfrom:A.PeterandA.Rangarajan,InformationGeometryforLandmarkShapeAnalysis:UnifyingShapeRepresentationandDefor-mation,IEEETransactionsonPatternAnalysisandMachineIntelligence,(Accepted,awaitingpublicationdate.),2008. 19

PAGE 20

landmarkmatchingmodel E(f)=KXa=1kvaf(ua)k2+kLfk2(2)Minimizing( 2 )resultsinanon-rigidmappingfthattakeslandmarksuaontova.However,themappingfdidnotcomeaboutfromthelandmarks,whicharejustpointsinR2;furthermoretheclassofadmissiblemapsiscontrolledbyourchoiceofthedierentialoperatorL.Theframeworkpresentedheredirectlyaddressesthisissueofdecouplingtherepresentationfromdeformationyieldingamodelthatenablesustowarplandmarkswithouttheuseofsplines.Inourapproach,weuseprobabilisticmodelsforshaperepresentation.Specically,Gaussianmixturemodels(GMM)areusedtorepresentunstructuredlandmarksforapairofshapes.Sincethetwodensityfunctionsarefromthesameparameterizedfamilyofdensities,weshowhowaRiemannianmetricarisingfromtheirinformationmatrixcanbeusedtoconstructageodesicbetweentheshapes.WerstdiscusstheFisher-RaometricwhichisactuallytheFisherinformationmatrixoftheGMM.TomotivatetheuseoftheFisher-Raometric,assumeforthemomentthatadeformationappliedtoasetoflandmarkscreatesaslightlywarpedset.Thenewsetoflandmarkscanalsobemodeledusinganothermixturemodel.Inthelimitofinnitesimaldeformations,theKullback-Leibler(KL)distancebetweenthetwodensitiesisaquadraticformwiththeFisherinformationmatrixplayingtheroleofthemetrictensor.Usingthisfact,wecancomputeageodesicdistancebetweentwomixturemodels(withthesamenumberofparameters).AlogicalquestionaroseoutofourinvestigationswiththeFisherinformationmatrix:MustwealwayschoosetheFisher-RaoRiemannianmetricwhentryingtoestablishdistancesbetweenparametric,probabilisticmodels?(Rememberinthiscontexttheparametricmodelsareusedtorepresentshapes.)Themetric'scloseconnectionstoShannonentropyandtheconcomitantuseofFisherinformationinparameterestimation 20

PAGE 21

havecementeditastheincumbentinformationmeasure.Ithasalsobeenproliferatedbyresearcheortsininformationgeometry,whereonecanshowitsproportionalitytopopulardivergencemeasuressuchasKullback-Leibler.However,thealgebraicformoftheFisher-Raometrictensormakesitverydiculttousewhenappliedtomulti-parameterspaceslikemixturemodels.Forinstance,itisnotpossibletoderiveclosed-formsolutionsforthemetrictensororitsderivative.Toaddressmanyofthesecomputationalinecienciesthatarisewhenusingthestandardinformationmetric,weintroduceanewRiemannianmetricbasedonthegeneralizednotionofa-entropyfunctional.Wetakeonthechallengeofimproving(computationally)theinitialFisher-basedmodelbyincorporatingthenotionofgeneralizedinformationmetricsasrstshownbyBurbeaandRao[ 26 ].Therichdierentialgeometricconnectionsassociatedwithrepresentingshapesasmixturemodelsenablesaexibleshapeanalysisframework.Inthisapproach,severalofthedrawbacksoftenassociatedwithcontemporarymethodsareremedied,i.e.shapematchingunderthismodel: Providesauniedmodelforshaperepresentationanddeformationnosplinemodelneededfordeforminglandmarkshapes. Doesnotplaceconstraintsonshapetopology,i.e.shapesarenotrequiredtobesimplecurves. Allowsmixturemodelrepresentationsofshapestobeanalyzedonthemanifoldofdensities,thusrespectingthenaturalgeometryassociatedwiththerepresentation. UtilizesageneralizedmethodtodevelopnewinformationmetricsthenewmetricwedevelophassignicantcomputationalsavingsovertheFisher-Raometricandforthersttimeprovidesaclosed-formmetricforparametricGaussianmixtures.Webegininthenextsection( 2.1.1 )byprovidingfurthermotivationforourapproachandcoverafewrelatedmethods.Section 2.2 ,discussestheprobabilisticrepresentationmodelforlandmarkshapes.WeshowhowitispossibletogofromalandmarkrepresentationtooneusingGMMs.Welookattheunderlyingassumptionsandtheirconsequenceswhichplayavitalroleininterpretingtheanalysis.Section 2.3 illustratesthetheoryandintuitionbehindhowonedirectlyobtainsadeformationmodelfromtherepresentation. 21

PAGE 22

Itprovidesabriefsummaryofthenecessaryinformationgeometrybackgroundneededtounderstandallsubsequentanalysis.WeillustrateconnectionsbetweentheFisherinformationanditsuseasaRiemannianmetrictocomputeashortestpathbetweentwodensities.WethenmotivategeneralizationsbydiscussingBurbeaandRao'sworkonobtainingdierentialmetricsusingthe-entropyfunctionalinparametricprobabilityspaces.Theuseofaspecic-functionleadstoan-orderentropyrstintroducedbyHavrdaandCharvt[ 47 ].Thiscaninturnbeutilizedtodevelopanewmetric(-orderentropymetric)thatleadstoclosed-formsolutionsfortheChristoelsymbolswhenusingaGaussianmixturemodels(GMM)forcouplingshaperepresentationanddeformation.ThisenablesalmostanorderofmagnitudeperformanceincreaseovertheFisher-Raobasedsolution.Section 2.4 validatestheFisher-Raoand-orderentropymetricsbyusingthemtocomputeshapedistancesbetweencorpuscallosumdataandprovidesextensivecomparativeanalysiswithseveralotherpopularlandmark-basedshapedistances. 2.1.1MotivationandRelatedWorkThereareanumberofadvantageswhenmixturemodelsareusedtorepresentshapelandmarksorshapepoint-setsingeneral.Therstisthealleviationofthecorrespondenceproblem.Otherbenetsofthemixturerepresentationincludetheinherentrobustnesstonoiseandlocalizationerroroftheshapefeaturesandlandmarks.Ashapedistanceisobtainedbycomputingadistancebetweenprobabilitydensityfunctions.And,inamannerthatishighlyreminiscenttocomparingdistancetransformsofshapes,theprobabilitydensityfunctionscanbecomparedateverypointinR2fortwodimensionalshapes.Intheliterature,wendseveralinstancesofusingdivergencemeasures[ 45 48 49 ]andclosed-formL2distancesbetweenmixturemodels[ 50 ]asstand-insforshapedistancemeasures.Inallofthesepreviousapproaches,theobjectivefunctionminimizedisacombinationofadistancemeasurebetweenmixturedensitiesandasplineregularizationofthenon-rigidwarping.Thesplinedrivennon-rigidwarpingattemptstomakeashapemixturedensityascloseaspossibletoaxedshapemixturedensity.Theseapproaches 22

PAGE 23

canbesuccinctlysummarizedasminimizing E(f)=D(p(xj(1));p(xj(2)(f)))+kLfk2(2)where(1)isthesetofparametersoftherst(xed)shape'smixturemodeland(2)(f)isthesetof(warped)parametersofthesecondshape'smixturemodel.Thechoiceofsplineathin-platesplineorWendlandsplineforexampleisdeterminedbythechoiceofthedierentialoperatorL.Thissetofapproachesaimstodiscoverthebestnon-rigidwarpingfunctionf(whosespatialsmoothnesspropertiesaredeterminedbythechoiceofL)thattakesp(xj(2)(f))ascloseaspossibletop(xj(1)).(Whenadieomorphismissought,thesecondtermismodiedtoaccommodateaninnitesimalgeneratorofagroupoftransformations.)Aspreviouslymentioned,themixturedensitydistancemeasurecanbeadivergencemeasurelikethepopularKullback-Leibler(orJensen-Shannon)measures[ 51 ]oramorestraightforwardclosed-formL2distance.Andwhenweexaminethisnotionofshapedistancefromawiderperspective,thesedistancesarenotthatdierentfromthoseobtainedusingdistancetransforms[ 52 ]ordistributionfunctions[ 53 ].Turningourfocustothespline-basedregularizationterm,weobserveaninterestingdisconnectespeciallyfromthevantagepointofinformationgeometry.Inequation( 2 ),wehavethecombinationofadistancemeasureDbetweentwomixturedensityfunctionsandaspline-basedregularizationtermkLfk2.Thesetwotermsareindependentofeachotherandthisisreectedinthefactthatwecanchooseanydistancemeasure(Kullback-Leibler,L2etc.)andanyspline(thin-plate,Wendland,Gaussianradialbasisetc.)resultinginacross-productofchoices.Thisdecouplingofshaperepresentation(mixturemodelinthiscase)andshapedeformationisalsopresentinother(non-probabilistic)landmarkdieomorphismframeworks[ 43 44 ].Forexample,in[ 43 ],thelandmarkdieomorphismobjectivefunctiontakestheform E(f(t);ftg)=KXa=1kda dtft((t))k2dt+kLftk2dt(2) 23

PAGE 24

whereft((t))isavelocityeldand(t)isthesetoflandmarkpositionsattimet.But,andinapreviewofourcentralidea,thereisastrongrelationshipbetweenthetwoterms.ThedistancemeasureDgivesusascalarmeasureofthesimilaritybetweentwomixturedensitiesandtheregularizationoperatorLforcesftobespatiallysmoothinordertogeneratetransformationsclosetoidentity.Fromtheinformationgeometryperspective,thereisageodesicpathfromthesecondshape'sprobabilitydensityp(xj(2))totherstshape'sprobabilitydensityp(xj(1)).Whycan'tweunifythetwotermsdistancemeasureandspatialsmoothnessanddirectlyndthegeodesiconasuitablydenedprobabilisticmanifoldthatgivestheshortestpossiblepathbetweenp(xj(1))andp(xj(2))?Ifthiscanbeachieved,therewouldbenoreasontohavetwoseparateterms,oneforashapedistancemeasureandoneforaregularizationofthenon-rigiddeformation.Instead,bycomputingageodesicbetweenthetwoprobabilitydensities,allwewouldneedtodoismovefromp(xj(1))top(xj(2))ontheshortestpathconnectingthetwoshapes.Thisgivesthedistancemeasure(lengthofgeodesic)andthewarp(intermediatepointsalongthegeodesic)allwithouttheneedforaspline-basedspatialmappingregularizationterm.ThedistancemeasureDwouldbemodiedtobeageodesicobjectivefunctionservingthedualroleofshapedistanceandshaperegularization. 2.2TheRepresentationModel:FromLandmarkstoMixturesInthissectionwedescribetheuseofprobabilisticmodels,specicallymixturemodels,forshaperepresentation.Supposewearegiventwoplanarshapes,S1andS2,consistingofKlandmarks S1=fu1;u2;:::;uKg;S2=fv1;v2;:::;vKg(2)whereua=[u1a;u2a]T;va=[v1a;v2a]T2R2;8a2f1;:::;Kg.TypicalshapematchingrepresentationmodelsconsiderthelandmarksasacollectionofpointsinR2orasavectorinR2K.Aconsequencewiththeserepresentationsisthatifonewishestoperformdeformationanalysisbetweentheshapes,aseparatemodelneedstobeimposed,e.g. 24

PAGE 25

thin-platesplines[ 54 ]orlandmarkdieomorphisms[ 43 ],toestablishamapfromoneshapetotheother.(Inlandmarkmatching,thecorrespondencebetweentheshapesisassumedtobeknown.)In 2.3 ,weshowhowtheprobabilisticshaperepresentationwepresentinthecurrentsectionprovidesanintrinsicwarpingbetweentheshapesthusunifyingbothshaperepresentationanddeformation.Mixturemodelrepresentationshavebeenusedtosolveavarietyofshapeanalysisproblems,e.g.[ 49 55 ].Weselectthemostfrequentlyusedmixture-modeltorepresentourshapesbyusingaK-componentGaussianmixturemodel(GMM)wheretheshapelandmarksarethecenters(i.e.theathlandmarkpositionservesastheathmeanforaspecicbi-variatecomponentoftheGMM).Thisparametric,GMMrepresentationfortheshapesisgivenby[ 56 ] p(xj)=1 22KKXa=1expfkxak2 22g(2)whereisthesetconsistingofalllandmarks,a=[(2a1);(2a)]T,x=[x(1);x(2)]T2R2andequalweightpriorsareassignedtoallcomponents,i.e1 K.(Note:theplanarlandmarksuaorvaaremappedtothecorrespondingGMMcomponentmeana.)Thoughweonlydiscussplanarshapes,itismathematicallystraightforwardtoextendto3D.Also,thenumberoflandmarkscanbeselectedeithermanuallyorthroughtheuseofmodelselection[ 57 ],dependingontheapplication.Thevariance2cancaptureuncertaintiesthatariseinlandmarkplacementand/ornaturalvariabilityacrossapopulationofshapes.Incorporatingfullcomponent-wiseellipticalcovariancematricesprovidestheexibilitytomodelstructurallycomplicatedshapes.Theequalweightingonthecomponent-wisepriorsisacceptableintheabsenceofanyaprioriknowledge.Figure 2-1 illustratesthisrepresentationmodelforthreedierentvaluesof2.Theinputshapesconsistsof63landmarksdrawnbyanexpertfromMRIimagesofthecorpuscallosumand233landmarksmanuallyextractedfromimageofsh.Thevarianceisafreeparameterinourshapematchingalgorithmand 25

PAGE 26

inpracticeitisselectedtocontrolthesizeoftheneighborhoodofinuencefornearbylandmarks.Asevidentinthegure,anotherinterpretationisthatlargervariancesblurlocationsofhighcurvaturepresentinthecopruscallosumcurves.Thus,dependingontheapplicationwecandial-inthesensitivitiestodierenttypesoflocaldeformations.Eventhoughitmayseemthatas2increaseswelosedetailedresemblancetotheoriginalshape,itisstillvalidtocomparetwoshapeswithlargevariancesincetheirrepresentationasmixturesisstilluniquewithrespecttothelocationsoftheGMMcomponents.Alsorecallthatthevarianceallowsustohandleerrorsinthelandmarklocations.Duetothesedesirableproperties,thechoiceofthevarianceiscurrentlyafreeparameterinouralgorithmandisisotropicacrossallcomponentsoftheGMM.SofarwehaveonlyfocusedontheuseofGMMsforlandmarks.However,theyarealsowellsuitedfordensepointcloudrepresentationofshapes.Insuchapplications,themeanandcovariancematrixcanbedirectlyestimatedfromthedataviastandardparameterestimationtechniques.Therealadvantageinrepresentingashapeusingaparametricdensityisthatitallowsustoperformrichgeometricanalysisonthedensity'sparameterspace.Thenextsectioncovershowthisinterpretationinthetheoreticalsettingofinformationgeometryallowsustousethesamerepresentationmodeltodeformshapes. 2.3TheDeformationModel:RiemannianMetricsfromInformationMatricesofMixturesWenowaddresstheissueofhowthesamelandmarkshaperepresentationgivenby( 2 )canalsobeusedtoenablethecomputationofdeformationsbetweenshapes.TheoverarchingideawillbetousetheparametricmodeltocalculatetheinformationmatrixwhichisaRiemannianmetricontheparameterspaceofdensities.Ifanytwoshapesarerepresentedusingthesamefamilyofparametricdensities,themetrictensorwillallowustotakeawalkbetweenthem.Thenextsectionexpandsonouruseoftheterminologyintrinsicandextrinsictodescribetheanalysisunderourprobabilisticframework.WethenusetheFisher-Raometrictomotivatesomekeyideasfrominformationgeometry 26

PAGE 27

usedinsubsequentparts.Immediatelyfollowing,wediscusshowtoapplythethepopularFisher-Raometrictoshapematchinganddevelopthefullyintrinsicdeformationframework.Next,weshowhowitispossibletoderiveotherinformationmatricesstartingfromthenotionofageneralizedentropy.Thelastsubsectionputsforthapossiblesolutiononhowmovementoflandmarksontheintrinsicspacecanbeusedtodrivetheextrinsicspacedeformation,anecessityforapplyingthesemethodstoapplicationssuchasshaperegistration. 2.3.1IntrinsicVersusExtrinsicAnalysisInthecontextofusingmixturemodelstorepresentanddeformshapes,wewilloftenusethewordsintrinsicandextrinsic.Thesetermsareanalogoustotheiruseindierentialgeometrywhereintrinsicdescribesanalysisstrictlyderivedfromthesurfacepropertiesofthemanifoldandextrinsicreferstotheuseofthespaceambienttothemanifold.Inthepresentframework,theKlandmarksofasingleshapecorrespondtothecentersofaK-componentGMMwhichinturngivethecoordinatesofasinglepointonthemanifoldofmixturedensities.Similarly,anothershapewithKlandmarkswillalsohavethesameinterpretationasapointonthemanifold.Thusourtechnique,asdescribedinthenextsection,enablesyoutodirectlyusethisrepresentationtoobtainawarpfromoneshapeontotheothershapewithoutrequiringonetoarbitrarilyintroduceadeformablemodelsuchasaspline.Sincewealwaysstayonthemanifoldandusetheintrinsicpropertyofthemetrictensortoobtainourpathbetweendensities,whichisalsothewarpbetweenshapes,werefertothisasintrinsicanalysis.Ourreferencetowarpingoftheextrinsicspacearisesfromthefactthatoftenshapedataarerealizedaspointsets,notjustlandmarks.Forapairofpoint-setshapes,landmarkscanbeextractedbyavarietyofmethodssuchasmanualassignmentorclustering.Oncewehavelandmarksrepresentationsoftheshapes,anintrinsicwarpcanbeestablishedasdescribedabove.However,thiswarponlydescribesthemovementofthelandmarksfromoneshapeontotheother.Howdoesonemovetheshapepoints, 27

PAGE 28

i.e.theextrinsicspaceconsistingofpointssurroundingthelandmarks,basedonthemovementofthelandmarks?Thoughnotthefocalpoint,forcompletenessweprovideonepossiblesolutionin 2.3.4 .Towarptheseextrinsicpointsitwillbenecessarytointroduceaexternalregularizerbuttheformulationiscommensuratewithourtheme,usingtheGMMtodrivethewarping.Figure 2-2 illustratesashshapeconsistingofseveralthousandpoints(lightgray)fromwhichwehaveextracted233landmarks(blackpoints)theextrinsicpointsaresurroundingthelandmarkswhilethelandmarksareusedastheintrinsiccoordinates.Echoingourclaim:forlandmarkmatchingourframeworkiscompletelyintrinsic,providingapath(consequentlyawarp)fromonelandmarkshapeontoanotherwithouttheneedofasplineregularizer.Onlyiftheapplicationdictatestheneedtowarptheextrinsicspacedoweemploytheuseofasplinemodelandeventhen,thewarpsarestilldrivenbymovementalongtheintrinsicpathdeterminedbytheintermediatelandmarkshapes. 2.3.2Fisher-RaoMetricforIntrinsicShapeMatchingTodiscoverthedesiredgeodesicbetweentwoGMMrepresentedlandmarkshapes( 2 ),wecanusetheFisher-Raometric( 1 )toformulateanenergybetweenthemas s=10gij_i_jdt(2)wherethestandardEinsteinsummationconvention(wheresummationsymbolsaredropped)isassumedand_i=di dtistheparametertimederivative.Technically( 2 )integratesthesquareoftheinnitesimallengthelement,buthasthesameminimizeras10q gij_i_jdt[ 58 ](whichisthetruegeodesicdistance).Notewehaveintroducedageodesiccurveparametertwheret2[0;1].Thegeodesicpathisdenoted(t)andatt=0 28

PAGE 29

andatt=1wehavetheendpointsofourpathonthemanifold,forinstance (0)def=2666666666666666664(1)(0)(2)(0)(3)(0)(4)(0)...(2K1)(0)(2K)(0)3777777777777777775=2666666666666666664u(1)1u(2)1u(1)2u(2)2...u(1)Ku(2)K3777777777777777775:(2)(1)isdenedsimilarlyandasshowntheyrepresentthelandmarksofthereferenceandtargetshaperespectively.Thefunctional( 2 )isminimizedusingstandardcalculusofvariationstechniquesleadingtothefollowingEuler-Lagrangeequations E k=2gkii+@gij @k@gik @j@gkj @i_i_j=0:(2)Thiscanberewritteninthemorestandardform gkii+k;ij_i_j=0(2)ThisisasystemofsecondorderODEsandnotanalyticallysolvablewhenusingGMMs.Onecanusegradientdescenttondalocalsolutiontothesystemwithupdateequations k+1(t)=k(t)(+1)E k(t);8t(2)whererepresentstheiterationstepandthestepsize.Itisworthnotingthatonecanapplyotheroptimizationtechniquestominimize( 2 ).Tothisend,in[ 59 ],theauthorshaveproposedaneleganttechniquebasedonnumericalapproximationsandlocaleigenvalueanalysisofthemetrictensor.Theirproposedmethodworkswellforshapeswithasmallnumberoflandmarksbutthespeedofconvergencecandegradeconsiderablywhenthecardinalityofthelandmarksislarge.Thisduetorequirementofrepeatedlycomputingeigenvaluesoflargematrices.Alternatemethods,e.g.quasi-Newton 29

PAGE 30

algorithms,canprovideacceleratedconvergencewhileavoidingexpensivematrixmanipulations.InthenextsectionweinvestigateageneralclassofinformationmatriceswhichalsosatisfythepropertyofbeingRiemannianmetrics.ThustheanalysispresentedabovetondthegeodesicbetweentwoshapesholdsandsimplyrequiresreplacingtheFisher-Raometrictensorbythenewgi;j. 2.3.3BeyondFisher-Rao:-Entropyand-OrderEntropyMetricsRao'sseminalworkandtheFisherinformationmatrix'srelationshiptotheShannonentropyhaveentrencheditasthemetrictensorofchoicewhentryingtoestablishadistancemetricbetweentwoparametricmodels.However,BurbeaandRaowentontoshowthatthenotionofdistancesbetweenparametricmodelscanbeextendedtoalargeclassofgeneralizedmetrics[ 26 ].Theydenedthegeneralized-entropyfunctional H(p)=(p)dx(2)whereisthemeasurablespace(forourpurposesR2),andisaC2-convexfunctiondenedonR+[0;1).(Forreadabilitywewillregularlyreplacep(xj)withp.)ThemetricontheparameterspaceisobtainedbyndingtheHessianof( 2 )alongadirectioninitstangentspace.Thedirectionalderivativeof( 2 )inthedirectionofisgivenby DH=d dtH(p+t)jt=0;t2R=0(p)dx;(2)whichwedierentiateoncemoretogettheHessian D2H=00(p)2dx:(2)Assumingsucientregularitypropertieson=f1;:::;ng,thedirectioninthetangentspaceofthisparametersetcanbeobtainedbytakingthetotaldierentialofp(xj)w.r.t dp()=nXk=1@p @kdk.(2) 30

PAGE 31

ThisresultsintheHessianbeingdenedas H(p)=00(p)[dp()]2dx;(2)wherewehavereplacedwithdp,anddirectlyleadstothefollowingdierentialmetricsatisfyingRiemannianmetricproperties ds2()=H(p)=nXi;j=1gi;jdidj,(2)where gi;j=00(p)(@p @i)(@p @j)dx.(2)(Wereferthereaderto[ 26 ]formoredetailedderivationsoftheaboveequations.)Themetrictensorin( 2 )iscalledthe-entropymatrix.Byletting (p)=plogp,(2)equation( 2 )becomesthefamiliarShannonentropyand( 2 )yieldstheFisherinformationmatrix.OnemajordrawbackofusingtheFisher-Raometricisthatthecomputationofgeodesicsisveryinecientastheyrequirenumericalcalculationoftheintegralin( 1 ).WenowdiscussanalternativechoiceofthatdirectlyleadstoanewRiemannianmetricandenablesustoderiveclosed-formsolutionsfor( 2 ).Ourdesiretondacomputationallyecientinformationmetricwasmotivatedbynoticingthatiftheintegralofthemetriccouldbereducedtojustacorrelationbetweenthepartialsofthedensityw.r.tiandj,i.e.@p @i@p @jdx,thentheGMMwouldreducetoseparableonedimensionalGaussianintegralsforwhichtheclosed-fromsolutionexists.Intheframeworkofgeneralizedentropies,thisideatranslatedtoselectingasuchthat00becomesaconstantin( 2 ).In[ 47 ],HavrdaandCharvtintroducedthenotionofa-order 31

PAGE 32

entropyusingtheconvexfunction (p)=(1)1(pp),6=1.(2)Aslim!1(p),( 2 )tendsto( 2 ).Toobtainourdesiredform,weset=2whichresultsin1 200=1.(Theone-halfscalingfactordoesnotimpactthemetricproperties.)Thus,thenewmetricisdenedas gi;j=(@p @i)(@p @j)dx(2)andwerefertoitasthe-orderentropymetrictensor.ThereaderisreferredtotheAppendixin[ 28 ]whereweprovidesomeclosed-formsolutionstothe-orderentropymetrictensorandthenecessaryderivativecalculationsneededtocompute( 2 ).Thoughwewerecomputationallymotivatedinderivingthismetric,itwillbeshownviaexperimentalresultsthatithasshapediscriminabilitypropertiessimilartothatoftheFisher-Raoandothershapedistances.Derivingthenewmetricalsoopensthedoorforfurtherresearchintoapplicationsofthemetrictootherengineeringsolutions.Underthisgeneralizedframework,thereareopportunitiestodiscoverotherapplication-specicinformationmatricesthatretainRiemannianmetricproperties. 2.3.4ExtrinsicDeformationTheprevioussectionsillustratedthederivationsoftheprobabilisticRiemannianmetricswhichledtoacompletelyintrinsicmodelforestablishingthegeodesicbetweentwolandmarkshapesonastatisticalmanifold.Oncethegeodesichasbeenfound,traversingthispathyieldsanewsetof'sateachdiscretizedlocationoftwhichinturnrepresentsanintermediate,intrinsicallydeformedlandmarkshape.Wewouldalsoliketousetheresultsofourintrinsicmodeltogobackandwarptheextrinsicspace.Noticethattheintrinsicdeformationofthelandmarksonlyrequiredour'stobeparametrizedbytime.Deformationoftheambientspacex2R2,i.e.ourshapepoints, 32

PAGE 33

canbeaccomplishedviaastraightforwardincorporationofthetimeparameterontoourextrinsicspace,i.e. p(x(t)j(t))=1 KKXa=11 22expf1 22kx(t)a(t)k2g:(2)Wewanttodeformthex(t)'sofextrinsicspacethroughthevelocitiesinducedbytheintrinsicgeodesicandsimultaneouslypreservethelikelihood,i.e.p(x(t)j(t))=p(x(t+t)j(t+t)),ofalltheseambientpointsrelativetoourintrinsic's.InsteadofenforcingthisconditiononL=p(x(t)j(t)),weusethenegativelog-likelihoodlogLofthemixtureandsetthetotalderivativewithrespecttothetimeparametertozero: dlogL dt=(r1logL)T_1+(r2logL)T_2+@logL @x1(t)u+@logL @x2(t)v=0 (2) whereu(t)=dx1 dtandv(t)=dx2 dtrepresenttheprobabilisticoweldinducedbyourparametricmodel.Thenotationr1isusedtoreectthepartialderivativew.r.t.therstcoordinatelocationofeachoftheKcomponentsofthemixturedensityandsimilarlyr2arethepartialsw.r.t.thesecondcoordinateforeachoftheKcomponents.Notethatthisformulationisanalogoustotheonewendinopticalowproblems[ 60 ].Similartoopticalow,weintroduceathin-platesplineregularizertosmooththeoweld (r2u)2+(r2v)2dx:(2)WenotethatisalsopossibletousethequadraticvariationinsteadoftheLaplacianastheregularizer.Ontheinteriorofthegrid,bothofthesesatisfythesamebiharmonicbutthequadraticvariationyieldssmootherowsneartheboundaries.Theoverallextrinsicspacedeformationcanbemodeledusingthefollowingenergyfunctional E(u;v)=(r2u)2+(r2v)2+dlogL dt2!dx(2) 33

PAGE 34

whereisaregularizationparameterthatweighstheerrorintheextrinsicmotionrelativetothedeparturefromsmoothness.TheminimaloweldsareobtainedviatheEuler-Lagrangeequationof( 2 ).Asformulated,themappingfoundthroughthethin-plateregularizerisnotguaranteedtobedieomorphic.Thiscanbeenforcedifnecessaryandiscurrentlyunderinvestigationforfuturework.Inthissection,wehaveshownthatselectingtherepresentationmodel( 2 )immediatelygavethelikelihoodpreservingdatatermusedtodrivethewarpingofextrinsicshapepointsthuscontinuingourthemeofuniedshaperepresentationanddeformation. 2.4ExperimentalResultsandAnalysisEventhoughwecannotvisualizetheabstractstatisticalmanifoldonwhichweimposeourtwometrics,wehavefoundithelpfultostudytheresultinggeodesicsofbasictransformationsonsimpleshapes(Figures 2-3 and 2-5 ).Inallgures,thedashed,straightlinerepresentstheinitializationpathandthesolidbell-shapedcurveshowsthenalgeodesicbetweenshapes.Figure 2-3 showsastraight-lineshapeconsistingof21landmarksthathasbeenslightlycollapsedlikeahinge.Noticethattheresultinggeodesicisbentindicatingthecurvednatureofthestatisticalmanifolds.EventhoughthebendinginFigure 2-3 (B)isnotasvisuallyobvious,acloserlookatthelandmarktrajectoriesforacoupleoftheshape'slandmarks(Figure 2-4 )illustrateshowtheintermediatelandmarkpositionshavere-positionedthemselvesfromtheiruniforminitialization.Itisthevelocityeldresultingfromtheseintermediatelandmarksthatenablesasmoothmappingfromoneshapetoanother[ 45 ].Figure 2-5 illustratesgeodesicsobtainedfrommatchingafour-landmarksquaretoonethathasbeenrotated210clockwise.ThegeodesicsobtainedbytheFisher-Raometricareagainsmoothlycurved,illustratingthehyperbolicnatureofthemanifoldwiththisspeciedinformationmatrix[ 61 ]whereasthe-orderentropymetricdisplayssharper,abruptvariations.Inbothcases,weobtainedwell-behavedgeodesicswithcurvedgeometry. 34

PAGE 35

Aswehavenoted,oneofthestrengthsofthisframeworkisthatitdoesnottopologicallyconstraintheshapes,allowingustoobtainwarpsandsimilaritymeasuresbetweenshapesthatexhibitfeaturessuchasinteriorstructuresanddisconnectedcomponents.ToshowcasethisdesirablefeaturewematchedsixshshapesshowninFigure 2-6 .Foreachverticalpairofsh,weextractedequalnumberoflandmarks.ThelandmarklocationsforeachshserveasthemeansoftheaGaussianmixture.Sinceeachshhasnowbeenconvertedtoitsmixturedensityrepresentation,wecanapplyourframeworktondgeodesicsbetweenthepairs.Oncethegeodesicisfound,wecanobtainthewarpthattakesoneshapeontoanotherbytakingintermediatepoints(eachofwhichisavalidmixturedensity)alongthegeodesic.Weareabletoaccomplishthiswithouttheuseofasplinemodelbecausetheshapes,underthedensityrepresentation,areonthemanifoldofmixturedensities;obtainingintermediateshapesamountstotreatingthemeancomponentsoftheintermediatemixturesasthelandmarksoftheshapes.InFigure 2-7 ,weshoweightintermediateshapesforeachmatchingpairfromFigure 2-6 .Thegeodesicswerecomputedwiththe-orderentropymetric.Wecomparethesedeformationstoonesproducedusingthelandmarkdieomorphismtechnique[ 43 ].Thisisafairlyrecenttechniquewiththemetricarisingfromtheminimumenergyofttingiteratedsplinestotheinnitesimalvelocityvectorsthatdieomorphicalytakeoneshapeontotheother.Itisworthnotingthatin[ 43 ],theauthorsimplementedadiscreteapproximationtotheirproposedenergyfunctional.Inordertoavoidanynumericalapproximationissuesandexperimentalvariability,ourimplementationobtainsagradientdescentsolutiondirectlyontheanalyticEuler-Lagrangeequationsfortheirfunctional.Noticethattheintermediatedeformations,incomparisontoourmethod,areverysimilar;however,thekeydierentiatoristhatlandmarkdieomorphismsrequiretheuseofsplinestoobtaintheseintermediatewarpswhereasourmethoddoesnot.(Note:Weselectedtheparameterinlandmarkdieomorphismssuchthatitwouldyieldintermediatedeformationssimilartotheonesobtainedwithourmethodforaparticularvalueof. 35

PAGE 36

Forbothmethods,varyingtheirrespectiveparameterscanyielddierentintermediatedeformations.)Forapplicationsinmedicalimaging,wehaveevaluatedboththeFisher-Raoand-orderentropymetricsonrealdataconsistingofninecorporacallosawith63-landmarkseachasshowninFigure 2-9 .TheselandmarkswereacquiredviamanualmarkingbyanexpertfromdierentMRIscans.Aswithalllandmarkmatchingalgorithms,correspondencebetweenshapesisknown.Weperformedpairwisematchingofallshapesinordertostudythediscriminatingcapabilitiesofthemetrics.SinceboththeFisher-Raoand-orderentropymetricareobtainedfromGMMs,wetestedbothmetricswiththreedierentvaluesofthefreeparameter2.Inadditiontothetwoproposedmetrics,weperformedcomparativeanalysiswithseveralotherstandardlandmarkdistancesandsimilaritymeasures.ThedistancemetricsincludedareProcrustes[ 39 62 ],symmetrizedHausdor[ 63 ]andlandmarkdieomorphisms[ 43 ].Thersttwodistancemetricshaveestablishedthemselvesasastapleforshapecomparisonwhilethethirdismorerecentandwasusedinthepreviousdiscussionfordeformationanalysis.Theshapesimilaritymeasures(whicharenotmetrics)incorporatedinthestudyusethebendingenergyofspline-basedmodelstomapthesourcelandmarkstothetarget.Weusedtwosplinemodels:theubiquitousthin-platespline(TPS)[ 40 ]whichhasbasisfunctionsofinnitesupportandthemorerecentlyintroducedWendlandspline[ 64 ]whichhascompactlysupportedbases.Forthesakeforbrevity,wewillrefertoallmeasuresasmetricsordistanceswiththeunderstandingthatthebendingenergiesdonotsatisfytherequiredpropertiesofatruemetric.TheresultsofpairwisematchingofallnineshapesislistedinTable 2-1 ,containingtheactualpairwisedistances.Thedistancesshowaglobaltrendamongallofthemetrics.Forexample,shape1and8havethesmallestdistanceunderallthemetricsexcept-orderentropymetricwith2=0:1andthethin-platesplinebendingenergy.However,shape8isthesecondbestmatchunderboththese,clearlyillustratingasimilar 36

PAGE 37

performancetotheothers.Also,almostallthemetricsrankpair(4,7)astheworstmatch.ThesinglediscrepancycomesfromtheHausdormetric.However,itlists(4,7)asthepenultimatelyworstmatchwhichgloballyisinoverallagreementwiththeothers.Wethenusedeachofthesemetricstoperformhierarchicalclustering,Figure 2-10 ,onthenineshapes.Figure 2-10 clearlyshowsaglobaltrendinthegroupingsamongthedierentmetrics.Onecaninterpretthisagreementasareectionofobvioussimilaritiesordissimilaritiesamongtheshapes.Theinterestingpropertiesuniquetoeachofthesemetricsariseinthedierencesthatareapparentinthelocaltrend.Weattributeamajorityoftheselocalrankdierencesduetotheinherentsensitivitiesofeachmetric.Thesesensitivitiesareadirectconsequencesofhowtheyareformulated.Forexample,itiswellknownthattheHausdormetricisbiasedtooutliersduetothemax-minoperationsinitsdenition.Thebendingenergyofthesplinemodelsisinvarianttoanetransformationsbetweenshapesanditsincreaseisareectionofhowoneshapehastobebenttotheother.Thedierencesamongthesplinemodelscanbeattributedtothecompact(Wendland)versusinnite(TPS)supportofthebasisfunctions.Wereferthereadertotheaforementionedreferencesformorethroughdiscussionsoftherespectivemetricsandtheirformulations.Theseresultsclearlyvalidatethetwonewmetricsasashapedistances.Thechoiceof2=f0:1;0:5;1:5gimpactedthelocalrankingsamongthetwometrics.AsFigure 2-1 illustrated,2givesustheabilitytodial-inthelocalcurvatureshapefeatures.Whenmatchingshapes,selectingalargevalueof2impliesthatwedonotwantthematchinginuencedbylocalized,highcurvaturepointsontheshape.Similarly,alowvalueof2reectsourdesiretoincorporatesuchfeatures.Asaillustrationofthis,considertherstthreedendrogramsinthetoprowofFigure 2-10 .ThersttwodendrogramswerecomputedusingtheFisher-Raometricwithvarianceparameter2=f0:1;0:5gresultinginshape6beingrankedasthenextbestmatchtopair(1,8).Whenweset2=1:5,shape3nowbecomesthenextbestmatchto(1,8).Hence,wesee2impactsthe 37

PAGE 38

shapedistance.However,itaectsitinsuchawaythatisdiscerniblynaturalmeaningthattherankingwasnotdrasticallychangedwhichwouldnotcoincidewithourvisualintuition.ThedierencesbetweenFisher-Raoand-orderentropymetricarisefromthestructuraldierencesintheirrespectivemetrictensorsgi;j.Theo-diagonalcomponents(correspondingtointra-landmarks)ofthe-orderentropymetrictensorarezero.Thisdecouplesthecorrelationbetweenalandmark'sownxandycoordinates,thoughcorrelationsexistwiththecoordinatesofotherlandmarks.Intuitivelythischangesthecurvatureofthemanifoldandshowsupvisuallyintheshapeofthegeodesic[ 28 ]whichinturnimpactsthedistancemeasure.The-orderentropymetricprovidedhugecomputationalbenetsovertheFisher-Raometric.TheFisher-RaometricrequiresanextraO(N2)computationoftheintegraloverR2wherewehaveassumedanNpointdiscretizationofthex-andy-axes.Thiscomputationmustberepeatedateachpointalongtheevolvinggeodesicandforeverypairoflandmarks.ThederivativesofthemetrictensorwhichareneededforgeodesiccomputationrequirethesameO(N2)computationforeverylandmarktripleandateachpointontheevolvinggeodesic.Sinceournew-entropymetrictensorandderivativesareinclosed-form,thisextraO(N2)computationisnotrequired.Pleasenotethatthesituationonlyworsensin3DwhereO(N3)computationswillberequiredfortheFisher-Raometric(andderivatives)whileournewmetric(andderivatives)remaininclosed-form.Itremainstobeseenifotherclosed-forminformationmetricscanbederivedwhicharemeaningfulintheshapematchingcontext.ThecomparativeanalysiswithothermetricsillustratedtheutilityofFisher-Raoand-orderentropymetricsasviableshapedistancemeasures.Inadditiontotheirdiscriminatingcapabilities,thesetwometricshaveseveralotheradvantagesoverthepresentcontemporaries.Therepresentationmodelbasedondensitiesisinherentlymorerobusttonoiseanduncertaintiesinthelandmarkpositions.Inadditionweshowcasedtheabilityofthesemetricstodeformshapeswithvarioustopologiesthusenabling 38

PAGE 39

landmarkanalysisforanatomicalformswithinteriorpointsordisjointparts.Mostimportantly,thedeformationisdirectlyobtainedfromtheshaperepresentation,eliminatinganarbitrarysplinetermfoundinsomeformulations.Therobustnessandexibilityofthismodel,hasgoodpotentialforcomputationalmedicalapplicationssuchascomputer-aideddiagnosisandbiologicalgrowthanalysis.Asageneralshapesimilaritymeasure,ourmetricsareyetanothertoolforgeneralshaperecognitionproblems. 2.5DiscussionWehavepresentedauniedframeworkforshaperepresentationanddeformation.Previousapproachestreatrepresentationanddeformationastwodistinctproblems.OurrepresentationoflandmarkshapesusingmixturemodelsenablesimmediateapplicationofinformationmatricesasRiemannianmetrictensorstoestablishanintrinsicgeodesicbetweenshapepairs.Tothisend,wediscussedtwosuchmetrics:theFisher-Raometricandthenew-orderentropymetric.Toourknowledge,thisisthersttimetheseinformationgeometricprincipleshavebeenappliedtoshapeanalysis.Inourframework,shapesmodeledasdensitiesliveonastatisticalmanifoldandintrinsicdistancesbetweenthemarereadilyobtainedbycomputingthegeodesicconnectingtwoshapes.Ourdevelopmentofthe-orderentropywasprimarilymotivatedbythecomputationalburdensofworkingwiththeFisher-Raometric.GiventhatourparameterspacecomesfromGaussianmixturemodels,theFisher-Raometricsuersseriouscomputationalinecienciesasitisnotpossibletogetclosed-formsolutionstothemetrictensorortheChristoelsymbols.Thenew-orderentropymetric,with=2,enablesustoobtainclosed-formsolutionstothemetrictensoranditsderivativesandthereforealleviatesthiscomputationalburden.Wealsoillustratedhowtoleveragetheintrinsicgeodesicpathfromthetwometricstodeformtheextrinsicspace,importanttoapplicationssuchasregistration.Ourtechniqueswereappliedtomatchingcorpuscallosumlandmarkshapes,illustratingtheusefulnessofthisframeworkforshapediscriminationanddeformationanalysis.Testresultsshowtheapplicabilityofthenewmetricstoshapematching, 39

PAGE 40

ABCD EFGH Figure2-1.Examplesoftheprobabilisticrepresentationmodel.A)Originalshapeconsistingof63landmarks(K=63).B-D)OverheadviewofK-componentGMMusing2=0:1,2=0:5,and2=1:5respectively.E)Originalshapeconsistingof233landmarks(K=233).F-H)OverheadviewofK-componentGMMusing2=0:001,2=0:01,and2=0:025respectively. providingdiscriminabilitysimilartoseveralothermetrics.Admittedlywearestillintheearlystagesofworkingwiththesemetricsandhaveyettoperformstatisticalcomparisonsonthecomputedshapegeodesicdistances.Thesemetricsalsodonotsuerfromtopologicalconstraintsontheshapestructure(thusenablingtheirapplicabilitytoalargeclassofimageanalysisandothershapeanalysisapplications).Ourintrinsic,coupledrepresentationanddeformationframeworkisnotonlylimitedtolandmarkshapeanalysiswherecorrespondenceisassumedtobeknown.Theultimatepracticalityandutilityofthisapproachwillberealizeduponextensionofthesetechniquestounlabeledpointsetswherecorrespondenceisunknown.Existingsolutionstothismoredicultproblemhaveonlybeenformulatedviamodelsthatdecoupletheshaperepresentationanddeformation,e.g.[ 14 ].Thoughthemetricspresentedinthisworkresultfromsecondorderanalysisofthegeneralizedentropy,itispossibletoextendtheframeworktoincorporateotherprobabilistic,Riemannianmetrics.Forexample,onecanperformintrinsicanalysisonthemanifoldofvonMisesmixturedensitieswhichisparticularlyusefulforunitvectordata[ 65 ]. 40

PAGE 41

Figure2-2.Intrinsicversusextrinsic.Theoriginalshdataconsistsof49Kpoints(duetoimageresolutiontheseshowupaslightgrayoutline,seezoomedineyeforclearerdepiction).The233landmarksareillustratedbysolidblackpoints.Thelandmarksareusedforintrinsicanalysissincetheyareusedasthemeansof233componentGMM.See 2.3.4 formethodtomovetheextrinsicpoints(surroundingthelandmarks)basedonthelandmarkmovement. AB Figure2-3.Bendingofstraightlinewith21landmarks.Thedashedlineistheinitializationandthesolidlinethenalgeodesic.A)CurvatureofspaceunderFisherinformationmetricevidentinnalgeodesic.B)Thespaceunder-orderentropymetricisnotasvisuallycurvedforthistransformation. 41

PAGE 42

Figure2-4.Intermediatelandmarktrajectoriesunderthe-orderentropymetrictensor.ThesearethesecondandthirdlandmarksfromthemiddleinFigure 2-3 (B).Thetrajectoriesshowthateventhoughthenalgeodesiclookssimilartothestraightlineinitialization,theintermediatelandmarkpositionshavechangedwhichresultsindierentvelocitiesalongthegeodesic. AB Figure2-5.Rotationofsquarerepresentedwithfourlandmarks.Thedashedlineistheinitializationandthesolidlinethenalgeodesic.Thecircularlandmarksarethestartingshapeandsquarelandmarkstherotatedshape.A)Fisherinformationmetricpathiscurvedsmoothly.B)-entropymetricpathhassharpcorners. 42

PAGE 43

ABC DEF Figure2-6.Fishshapeswithdieringtopologies.Foreachverticalpairweextractedequalnumberoflandmarks:(A)&(D)233,(B)&(E)253,and(E)&(D)214. Figure2-7.DeformationanalysisusingshfromFig. 2-6 using-orderentropymetric.Toprowshowsintermediatewarpsbetween(A)and(D),2=0:5.Middlerowshowsintermediatewarpsbetween(B)and(E),2=0:25.Bottomrowshowsintermediatewarpsbetween(C)and(F),2=0:25.Thedeformationsdonotrequireasplinemodel. 43

PAGE 44

Figure2-8.DeformationanalysisusingshfromFig. 2-6 usinglandmarkdieomorphisms[ 43 ].Allshapescomputedwith=10.Toprowshowsintermediatewarpsbetween(A)and(D).Middlerowshowsintermediatewarpsbetween(B)and(E).Bottomrowshowsintermediatewarpsbetween(C)and(F).Thesedeformationsrequireasplinemodel. Figure2-9.Ninecorpuscallosumshapesusedforpairwisematching,63landmarkspershape. 44

PAGE 45

Figure2-10.Hierarchicalclusteringwithdierentmetrics.NoticethatvaryingontheFisher-Raoand-OrderEntropymetricdoesnotsignicantlyimpactglobalgroupingoftheshapes(seerstthreecolumnsofrowsoneandtwo).Almostallthemetricsagreethatshape1and8arethebestmatch,whileshape4isthemostdissimilar. 45

PAGE 46

Table2-1.Pairwiseshapedistances.Allofthecorporacallosawerematchedwitheachother.Fisher-Raoand-OrderEntropymetricswerecomputedwiththreedierentvaluesof2=f0:1;0:5;1:5gtoassesstheimpactofthefreeparameteronshapedistance.Shapes1and8havethesmallestdistanceunderalmostallthedistances,while4versus7istheworst.(Seetextformorediscussions.) Pairs Fisher-Rao(102) -OrderEntropy(103) Dieomorphism(102) Procrustes(102) Hausdor(102) Wendland(102) TPS(102) 2=:1 2=:5 2=1:5 2=:1 2=:5 2=1:5 1vs.2 142.25 27.17 5.85 4.67 4.64 0.54 45.05 11.73 27.15 128.39 7.72 1vs.3 62.22 14.59 3.80 2.06 2.66 0.40 17.72 7.74 11.83 45.08 1.47 1vs.4 375.07 87.04 20.31 13.73 16.29 2.27 114.17 18.95 50.29 203.60 10.60 1vs.5 119.75 26.72 6.79 4.09 5.07 0.80 42.80 11.49 25.52 131.52 8.28 1vs.6 54.15 9.83 2.02 2.15 2.22 0.26 17.97 7.19 13.85 65.04 4.77 1vs.7 206.41 52.81 14.76 7.63 10.96 1.88 81.49 16.53 57.06 227.29 13.28 1vs.8 24.07 3.08 0.53 1.05 0.62 0.06 8.20 4.73 5.69 50.89 3.05 1vs.9 161.57 32.19 7.36 6.65 8.05 1.07 58.49 13.27 26.54 192.29 12.92 2vs.3 106.46 20.92 5.86 3.65 3.82 0.65 39.63 11.21 17.32 123.01 6.48 2vs.4 571.37 136.56 29.39 19.65 23.83 3.02 182.93 23.54 117.74 351.38 17.62 2vs.5 367.50 86.10 21.29 11.00 14.41 2.16 123.99 19.72 72.76 312.08 16.74 2vs.6 73.74 15.88 4.44 2.52 3.24 0.55 34.84 10.31 15.19 110.61 5.55 2vs.7 150.02 44.22 15.18 5.03 8.86 1.96 80.38 16.47 71.46 254.76 11.72 46

PAGE 47

Table2-1.Continued Pairs Fisher-Rao(102) -OrderEntropy(103) Dieomorphism(102) Procrustes(102) Hausdor(102) Wendland(102) TPS(102) 2=:1 2=:5 2=1:5 2=:1 2=:5 2=1:5 2vs.8 136.85 27.96 6.39 3.95 4.56 0.60 53.75 12.68 23.35 169.20 9.42 2vs.9 94.52 20.60 5.02 3.74 5.59 0.87 43.67 11.53 28.81 147.21 10.52 3vs.4 610.51 153.60 38.20 21.85 28.07 4.27 201.91 25.17 93.71 348.10 11.13 3vs.5 231.03 53.58 12.41 6.92 8.57 1.12 67.43 14.55 33.53 153.21 6.80 3vs.6 34.58 6.21 1.18 1.28 1.16 0.11 9.54 5.17 7.41 28.71 2.74 3vs.7 92.02 21.34 5.44 3.67 4.68 0.75 39.61 11.28 19.74 100.58 6.74 3vs.8 59.26 13.33 3.27 1.86 2.24 0.32 18.32 7.69 12.11 47.59 2.06 3vs.9 119.42 22.62 4.71 5.18 5.75 0.69 40.40 10.96 29.79 116.41 9.39 4vs.5 208.30 59.56 19.19 7.54 13.18 2.70 92.67 17.85 32.92 200.05 12.84 4vs.6 435.13 110.01 27.50 15.85 21.45 3.32 147.27 21.96 64.50 311.83 23.36 4vs.7 682.10 193.47 54.20 25.14 37.77 6.59 229.74 28.60 104.18 499.73 34.32 4vs.8 325.84 79.77 19.83 11.48 14.97 2.30 105.93 18.71 61.59 224.42 16.66 4vs.9 512.94 132.76 33.98 18.76 26.97 4.17 172.52 23.82 72.78 374.71 25.14 5vs.6 163.69 37.42 8.72 4.56 5.68 0.76 56.41 13.01 28.47 157.91 10.74 5vs.7 311.52 78.63 19.60 8.88 12.34 1.79 91.71 17.46 74.11 233.17 13.85 47

PAGE 48

Table2-1.Continued Pairs Fisher-Rao(102) -OrderEntropy(103) Dieomorphism(102) Procrustes(102) Hausdor(102) Wendland(102) TPS(102) 2=:1 2=:5 2=1:5 2=:1 2=:5 2=1:5 5vs.8 86.32 20.26 5.17 2.58 3.50 0.56 31.57 9.99 20.78 89.38 4.07 5vs.9 270.52 63.21 16.13 7.75 11.11 1.63 81.30 16.31 42.61 224.62 12.78 6vs.7 82.06 22.30 6.80 2.58 4.06 0.79 38.31 11.13 23.74 105.01 5.70 6vs.8 28.72 5.96 1.21 0.86 1.14 0.14 13.81 6.22 7.76 40.29 3.33 6vs.9 43.65 10.11 2.71 1.90 2.80 0.44 21.81 8.04 12.75 59.11 4.05 7vs.8 145.55 40.08 11.83 4.62 7.85 1.45 67.70 14.50 38.37 151.59 6.87 7vs.9 85.01 21.31 6.45 2.62 3.98 0.70 31.97 10.11 28.22 95.40 5.19 8vs.9 103.71 23.95 5.87 3.68 5.62 0.80 47.65 11.84 20.90 126.93 9.24 48

PAGE 49

CHAPTER3WAVELETDENSITYESTIMATION 3.1IntroductionDensityestimationisawell-studiedeld,encompassingamyriadoftechniquesandtheoreticalformulationsallwiththecommongoalofutilizingtheobserveddataX=fxigNi=1todiscoverthebestapproximationtotheunderlyingdensitythatgeneratedthem.MethodsrangefromsimplehistogrammingtomorestatisticallyecientkernelbasedParzenwindowtechniques[ 66 67 ].Withinthelast20years,thewidespreaduseofwaveletanalysisinappliedmathematicsandengineeringhasalsomadeitswayintostatisticalapplications.Theuseofwaveletsasadensityestimatorwasrstexploredin[ 68 ].Waveletbaseshavethedesirablepropertyofbeingabletoapproximatealargeclassoffunctions(L2).Specicallyfordensityestimation,waveletanalysisisoftenperformedonnormedspacesthathavesomenotionofregularitylikeBesov,HlderandSobolev.Fromanempiricalpointofview,theutilityofrepresentingadensityinawaveletbasiscomesfromthefactthattheyareabletoachievegoodglobalapproximationpropertiesduetotheirlocallycompactnature-akeypropertywhenitcomestomodelingdensitiesthatcontainbumpsand/orabruptvariations.Itiswellknownthatwaveletsarelocalizedinbothtimeandfrequencyandthiscompactnessishighlydesirableindensityestimationaswell.Thebasicideabehindwaveletdensityestimation(forone-dimensionaldata)istorepresentthedensitypasalinearcombinationofwaveletbases p(x)=Xj0;kj0;kj0;k(x)+1Xjj0;kj;kj;k(x)(3) Contentforthischapterhasbeenreprintedwithpermissionfrom:A.PeterandA.Rangara-jan,MaximumLikelihoodWaveletDensityEstimationwithApplicationstoImageandShapeMatching,IEEETransactionsonImageProcessing,vol.17,no.4,pp.458-468,April2008. 49

PAGE 50

wherex2R,(x)and(x)arethescaling(a.k.a.father)andwavelet(a.k.a.mother)basisfunctionsrespectively,andj0;kandj;karescalingandwaveletbasisfunctioncoecients;thej-indexrepresentsthecurrentlevelandthek-indextheintegertranslationvalue.(Thetranslationrangeofkcanbecomputedfromthespanofthedataandbasisfunctionsupportsize[ 69 ].)Ourgoalthenistoestimatethecoecientsofthewaveletexpansionandobtainanestimator^pofthedensity.Thisshouldbeaccomplishedinamannerthatretainsthepropertiesofthetruedensitynotablythedensityshouldbenon-negativeandintegratetoone.Typically,waveletdensityestimators(WDE)areclassiedaslinearandnon-linear.Thetermlinearestimatordenotesthefactthatthecoecientsareobtainedviaaprojectionofthedensity'sdistributionontothespacespannedbythewaveletbasis.Non-linearestimatorsthresholdtheestimatedcoecients,bothgloballyandlocally,toobtainoptimalconvergencetothetruedensity.Theseestimators,especiallythosewiththresholding,oftencannotguaranteethattheresulting^pfromtheestimationprocesssatisestheaforementionedproperties.Toguaranteetheseproperties,onetypicallyresortstoestimatingp pas p p(x)=Xj0;kj0;kj0;k(x)+1Xjj0;kj;kj;k(x)(3)whichdirectlygivesp=p p2.Previousworkonwaveletdensityestimationofp p[ 70 71 ],stayswithintheprojectionparadigmoftryingtoestimatethecoecientsasaninnerproductwiththecorresponding(orthogonal)basis.Assuch,theyhavetodirectlyaddresstheestimationofthescalarproductinvolvingthesquarerootestimateandtheappropriatebasisfunction,e.g.estimatingcoecientj0;krequiresndinganacceptablesubstituteforRdp p(x)jo;k(x)dx.Wewillshowhowtocompletelyavoidthisparadigmbycastingtheestimationprocessinamaximumlikelihoodframework.Thewaveletcoecientsofthep pexpansionareobtainedbyminimizingthenegativeloglikelihoodovertheobservedsampleswithrespecttothecoecients.Moreover,asymptoticanalysiswillillustratearemarkablepropertyoftheFisherinformationmatrixofthedensityunder 50

PAGE 51

thisrepresentationwhichis4I,whereIistheidentitymatrixandthisisleveragedintheoptimization.ThishighlystructuredmatrixisalsotheasymptoticHessianofourmaximumlikelihoodobjectivefunctionattheoptimalsolutionpoint.Thistightcouplinginourestimationframeworkwillleadtoaveryecient,modiedNewton'smethodforcomputingthewaveletcoecients.Wemaintainallthedesirablepropertiesthatinherentlycomewithestimatingp p,whilecircumventingtheneedtoestablishthepreviouslymentionedsubstitute.Theestimationprocedureisdevelopedforbothoneandtwodimensionaldensities.Forvalidation,wedemonstratetwocomputervisionapplicationsthroughpoint-setshaperegistrationandmutualinformation(MI)basedaneregistrationofmedicalimagery[ 72 ].Givenapairofimages,oneimage,designatedthesource,isconsideredassimilarunderanetransformationstoapre-speciedtargetimage.MI-basedimagealignmenttriestomaximizethemutualinformationbetweentheimagepair,whichhopefullyoccurswhentheyareoptimallyaligned.Thealgorithmrequiresadensityestimationstepthatcomputesthejointdensitybetweentheimagepairs.Thisdensityisthenusedtocalculatethemutualinformation.Forshaperegistration,weadoptthecorrespondence-freeapproachaspresentedin[ 49 ]butreplacetheperformancecriterionwiththeHellingerdivergence[ 73 ].Forbothapplications,wereplacethetypicaldensityestimator,usuallya2Dhistogram,kernelestimatorormixturemodel,withourwaveletdensityestimatorandanalyzeitsviability.Wealsoprovidecomprehensiveresultsofthewaveletdensityestimator'sabilitytomodeltrueanalytic1Dand2DdensitiessuchasGaussianmixturemodels.Therestofthischapterisorganizedinthefollowingmanner.In 3.2 ,webrieyrecapmultiresolutionwaveletanalysisandprovideamoreindepthdiscussionconcerningwaveletdensityestimation.Section 3.3 discussesourmaximumlikelihoodframeworkforestimatingthewaveletcoecientsforp p.ItgoesontodetailthemodiedNewton'smethodusedtoecientlycomputethecoecients.In 3.4 ,ourmethodisvalidatedin 51

PAGE 52

theapplicationsettingofimageandshaperegistrationandweshowcasethecapabilitytomodelknowndensitiesandcompareperformanceagainstxedandvariablebandwidthkerneldensityestimators. 3.2WaveletTheoryanditsApplicationtoDensityEstimationWenowprovideabasicintroductiontotheideasofwaveletmultiresolutiontheoryandmoveontodiscussinghowtheseconceptsarecarriedoutinwavelet-baseddensityestimation.Wewillbefocusedthroughoutonclearlycommunicatingtheconceptualaspectsofthetheory,divertingmuchofthemathematicalmachinerytotheappropriatelycitedreferences. 3.2.1MultiresolutionForanyfunctionf2L2andastartingresolutionlevelj0,representationinthewaveletbasisisgivenby f(x)=Xj0;kj0;kj0;k(x)+1Xjj0;kj;kj;k(x);(3)where jo;k(x)=2j0=2(2j0xk);j;k(x)=2j=2(2jxk);(3)arescaledandtranslatedversionofthefather(x)andmother(x)wavelets.ThekeyideabehindmultiresolutiontheoryisasequenceofnestedsubspacesVjj2Zsuchthat V2V1V0V1V2(3)andwhichsatisfythepropertiesTVj=f0gand SVj=L2(completeness).Theresolutionincreasesasj!1anddecreasesasj!(somereferencesshowthisorderreversedduetothefacttheyinvertthescale[ 74 ]).Atanyparticularlevelj+1,wehavethefollowingrelationship VjMWj=Vj+1(3) 52

PAGE 53

whereWjisaspaceorthogonaltoVj,i.e.VjTWj=f0g.Thefatherwavelet(x)anditsintegertranslationsformabasisforV0.Themotherwavelet(x)anditsintegertranslatesspanW0.Thesespacesdecomposethefunctionintoitssmoothanddetailparts;thisisakintoviewingthefunctionatdierentscalesandateachscalehavingalowpassandhighpassversionofthefunction.Wewillassume(x),(x)andtheirscaledandtranslatedversionsformorthogonalbasesfortheirrespectivespaces.Undertheseassumptions,thestandardwaytocalculatethecoecientsfor( 3 )isbyusingtheinnerproductofthespace,e.g.thecoecientj0;kisobtainedby j0;k==f(x)j0;k(x)dx(3)wherewehaveusedtheL2innerproduct.Mostoftheexistingwavelet-baseddensityestimationtechniquesexploitthisprojectionparadigmtoestimatethecoecients.Replacingthegeneralfunctionfbyadensityp,thecoecientsfor( 3 )canbecalculatedas j0;k=p(x)j0;k(x)dx=E[j0;k(x)](3)whereEistheexpectationoperator.GivenNsamples,thisisapproximatedasthesampleaverage j0;k=1 NNXi=1j0;k(xi):(3)Manydensityestimationtechniques,includingours,requireevaluating(x)and(x)atvariousdomainpointsintheirsupportregion.However,mostfatherandmotherwaveletsdonothaveananalyticclosed-formexpression.Thestrategyistousetheclosecouplingbetweenthescalingfunctionandwaveletnd(x)bynumericallysolvingthedilationequationandthendirectlyobtain(x)bysolvingthewaveletequation.Thedilationequationisgivenby (x)=2Xkl(k)(2xk)(3) 53

PAGE 54

wherel(k)arethelowpassltercoecientsassociatedwithaparticularscalingfunctionfamily.Thiscanbenumericallysolvedusingiterativeproceduressuchasthecascadealgorithm[ 75 ].Uponsolvingfor(x),onecanimmediatelygettheassociatedwaveletfunctionbyusingthehighpassltercoecientsh(k)andsolvingthewaveletequation (x)=2Xkh(k)(2xk):(3)Thenumericalversionsof(x)and(x)willhavevaluesatdomainpointsthatareintegermultiplesof1 2M(whereMcontrolsthediscretizationlevel).Ifanxvaluelandsinbetweenthesegridpoints,itisastraightforwardprocesstointerpolateandgetthedesiredvalue.Wehaveadoptedacubicsplineinterpolationstrategytoobtaintheintermediatevalues. 3.2.2WaveletDensityEstimationApracticalconsiderationofusingwaveletsfordensityestimationrequirescarefulconsiderationofseveralissues.First,onemustdecidethefamilyofwaveletsthatwillbeusedasthebasis.Thoughthisissuehasreceivedconsiderablylessattentionintheoreticalliterature,apragmaticsolutionsuggeststhatthechoiceiscloselytiedtoproblemdomain.However,onealmostalwaysassumesthatthebasessatisfythedesirableorthonormalitypropertyandarecompactlysupported.Also,multiresolutionanalysisforfunctionapproximationrequirestheuseofbasesthathaveacoupledscalingandwaveletfunctionrelationship.ThisrestrictsourchoiceofbasisfunctionstofamiliessuchasHaar,Daubechies,CoietsandSymlets.Itisworthmentioningthatthesebasisfunctionsarenotperfectlysymmetric(exceptforHaar);infactinclassicalwaveletanalysis,symmetryisadetrimenttoperfectreconstructionofthesignal[ 74 ].ExactsymmetryisnotacriticalrequirementfordensityestimationandfamilieslikeSymletsexhibitcharacteristicsclosetosymmetryforhigherordervanishingmoments.Throughoutweassumethatourbasesareorthogonal,compactlysupportedandhavebothascalingandwaveletfunction.Second,andperhapsmostimportantly,wemustaddressecientcomputationofthebasiscoecientsandtheimpacttheyhaveonthepropertiesoftheestimated 54

PAGE 55

density.Ourmaincontributionistofurtheradvancethetheoreticalframeworkfortheestimationofthesecoecients,whilemaintainingtheimportantpropertiesofabonadedensitynon-negativeinitssupportandintegratingtoone.Finally,itisnecessarytoconsiderthepracticalissueofselectingthebasistruncationparameters.Noticethatin( 3 ),convergenceofthewaveletrepresentationtothetruefunctionassumesastartingresolutionlevelj0andusesaninnitenumberofdetailresolutionlevels.Inpracticalcomputation,itisnecessarytodevelopaprincipledwayofchoosingj0andalsoastoppinglevelj1aswecannothaveaninniteexpansion.Theseissuesarenecessarilyaddressedbymodelselectionmethodssuchascrossvalidation.Modelselectionisnotthefocusofourcurrentwork.Itispossibletoadoptanyoftheexistingmodelselectionmethods,assummarizedin[ 76 ],toappropriatelychoosetheseparametersandincorporateitintoourframework.Returningtothesecondpoint,classicalwaveletdensityestimation[ 77 78 ]doesnottrytoexplicitlyensurethatthedensityisnon-negativeandusuallysuersfromnegativevaluesinthetailsofthedensity.Forexample,intheworkofDonohoet.al.[ 77 ],thisartifactisintroducedbythenecessitytothresholdthecoecients.Non-linearthresholdingobtainsbetterconvergenceandachievestheoptimalminimaxrateundertheglobalintegratedmeansquarederror(IMSE)E[k^ppk2]measure,whereEistheexpectationoperator.Thoughthesearefavorableproperties,itisstillsomewhatunsettlingtohaveadensitywithnegativevalues.Also,thresholdingleadstotheproblemofhavingtorenormalizethecoecientstomaintaintheintegrablepropertyoftheestimateddensity.Undertheusualnon-linearestimationprocess,thisisnotastraightforwardprocedureandmayrequirefurtherintegrationtoworkoutthenormalizingconstant.Nextwewillshowhowitispossibletoincorporatethebenetsofthresholdingthecoecientswhilemaintainingtheintegrityoftheestimateddensity.Thepreferredwaytomaintainthesepropertiesistoestimatethesquarerootofthedensityp pratherthanp.Estimatingp phasseveraladvantages:(i)non-negativityis 55

PAGE 56

guaranteedbythefactp=p p2(ii)integrabilitytooneiseasytomaintaineveninthepresenceofthresholding,and(iii)thesquarerootisavariancestabilizingtransform[ 79 ].Thepresentworkalsofallsintothiscategoryoftechniquesthatestimatep p.Toourknowledgethereareonlytwopreviousworks[ 70 71 ]thatestimatethesquarerootofthedensityusingawaveletbasisexpansion.Webeginbyrepresentingthesquarerootofthedensityusing p p(x)=Xj0;kj0;kj0;k(x)+j1Xjj0;kj;kj;k(x):(3)Imposingourintegrationcondition,Rdp p2dx=1,impliesthat Xj0;k2j0;k+j1Xjj0;k2j;k=1:(3)Noticethatif( 3 )isnotonebutsomearbitraryconstantD,suchaswhenathresholdingschemechangestheweights,itispossibletoperformastraightforwardrenormalizationbymerelydividingthecoecientsbyp D.Intheprevioustwoworks[ 70 71 ],estimationofj0;kandj;kismotivatedbyapplyingthepreviouslydiscussedprojectionmethod.Wearenowworkingwithp p,however,whichchanges( 3 )to j0;k=p p(x)j0;k(x)dx=p(x) p p(x)j0;k(x)dx=Ej0;k(x) p p(x):(3)Thej;kcoecientsaredenedbyanalogy.In[ 70 ],theauthorsproposeasuitablesubstitutetotheempiricalestimator1 NPNi=1j0;k(x)=p p(x),butthecoecientestimationissensitivetothepre-estimatorofp(x).IntheworkbyPenevandDechevsky[ 71 ],thecoecientcomputationisbasedonorderstatisticsofthesampledata.Asweillustrateinthenextsection,themethodwepresentavoidstheseissuesbycastingthedensityestimationprobleminamaximumlikelihoodsetting.Themaximumlikelihoodmodel 56

PAGE 57

alsoensurestheasymptoticconsistencyofourestimatedcoecients.Tocompletethespectrumofnon-negativedensityestimationtechniques,itisworthmentioningthatWalter[ 80 ]presentsanalternativeusingacleverconstructionofnon-negativewavelets;explorationofthismethod,however,isbeyondthepresentscope. 3.3MaximumLikelihoodforWaveletDensityEstimationWenowdiscusshowtocastwaveletdensityestimationinamaximumlikelihoodframework.Oftenmaximumlikelihoodisdesignatedaparametrictechniqueandreservedforsituationswhereweareabletoassumeafunctionalformforthedensity.Thus,currentresearchtypicallycategorizeswaveletdensityestimationasanon-parametricestimationproblem.Treatingthecoecientsastheparameterswewishtoestimate,however,allowsustomovetheproblemintotheparametricrealm.Thisinterpretationisalsopossibleforotherformulationssuchaskerneldensityestimation,wheremaximumlikelihoodisappliedtoestimatethekernelbandwidthparameter.Forestimatingthewaveletcoecients,adoptingthemaximumlikelihoodprocedureswillleadtoaconstrainedoptimizationproblem.Wetheninvestigatetheconnectionsbetweenestimatingp pandtheFisherinformationofthedensity.Exploringthisconnectionwillallowustomakesimplifyingassumptionsabouttheoptimizationproblem,resultinginanecientmodiedNewton'smethodwithgoodconvergenceproperties.Wewillpresentderivationsforboth1Dand2Ddensityestimation,extensionstohigherdimensionswouldfollowasimilarpath. 3.3.11DConstrainedMaximumLikelihoodLetX=fxigNi=1;xi2RrepresentNi.i.d.samplesfromwhichwewillestimatetheparametersofthedensity.Asisoftencustomary,wewillchoosetominimizethenegativeloglikelihoodratherthanmaximizetheloglikelihood.Thenegativeloglikelihoodobjectiveisgivenby logp(X;fj0;k;j;kg)=1 NlogQNi=1hp p(xi)i2=1 NPNi=1loghPj0;kj0;kj0;k(xi)+Pj1jj0;kj;kj;k(xi)i2:(3) 57

PAGE 58

Tryingtominimize( 3 )directlyw.r.t.j0;kandj;kwouldresultinadensityestimatorwhichdoesnotintegratetoone.Toenforcethisconditionwerequirethefollowingequalityconstraint h(fj0;k;j;kg)="Xj0;k2j0;k+Xjj0;k2j;k#1=0:(3)ThisconstraintcanbeincorporatedviaaLagrangeparametertoobtainthefollowingconstrainedobjectivefunction L(X;fj0;k;j;kg;)=logp(X;fj0;k;j;kg)+h(fj0;k;j;kg)(3)Theconstraint( 3 )dictatesthatthesolutionto( 3 )livesonaunithypersphere;itcanbesolvedusingstandardconstrainedoptimizationtechniques.Beforepresentingourparticularsolution,weexplorethepropertiesoftheFisherinformationmatrixassociatedwiththisproblem. 3.3.2TheManyFacesofFisherInformationTheclassicformoftheFisherinformationmatrixisgivenby guv()=p(x;)@ @ulogp(x;)@ @vlogp(x;)dx;(3)wherethe(u;v)indexpairdenotestherow,columnentryofthematrixandconsequentlytheappropriateparameterpair.Intuitively,onecanthinkoftheFisherinformationasameasureoftheamountofinformationpresentinthedataaboutaparameter;forwaveletdensityestimation=fj0;k;j;kgandthe(u;v)-indexingisadjustedtobeassociatedwiththeappropriatelevel,translationindexpair,i.e.fu=(j;k);v=(l;m)gwherejandlarethelevelindicesandkandmarethetranslationindices.Forthecurrentsetting,whereweareestimatingp p,theFisherinformationhasamorepertinentform ~guv=@p p(xj) @u@p p(xj) @vdx)guv=4~guv:(3) 58

PAGE 59

Hence,~guvcomputedusingthesquarerootofthedensitydiersonlybyaconstantfactorfromthetrueFisherinformation.Usingthisalgebraicrelationship,theFisherinformationofthewaveletdensityestimatorcanbecalculatedbysubstitutingthewaveletexpansionofp p(x;)givenin( 3 ).Thisgives ~guv=u(x)v(x)dx=8>><>>:1ifu=v0ifu6=v;(3)wherewehaveleveragedtheorthogonalpropertyofthewaveletbasisfunctions.ThisisanidentitymatrixandtheFisherinformationofp(x;)canbewrittenasguv=4I.ThereisanotheralgebraicmanipulationthatallowsustocomputetheFisherinformationusingtheHessianoftheloglikelihood,specically guv=ErrTlogp(x;)=E[H];(3)whereristhegradientoperatorw.r.t.theparametersandHistheHessianmatrixofthemulti-parameternegativeloglikelihood.Recallingthatequation( 3 )isthenegativeloglikelihood,wecanimmediatelymaketheconnectionthat( 3 )'sasymptoticHessianshouldbeHL=guv=4I.Toverifythis,let HL=Hnll+Hh(3)whereHnllandHharetheHessianofthenegativeloglikelihood( 3 )andconstraintequation( 3 ),respectively.Equation( 3 )istheHessianoftheLagrangianwhichistypicalofconstrainedminimizationproblems.WeillustratethecomputationoftheasymptoticHnllbyprovidingresultsforaparticularcoecient;othercoecientsarecalculatedinasimilarmanner.Thesecondpartialderivativeof( 3 )is @2 @h;l@p;m[logp]=2h;l(x)p;m(x) p(x;=fj0;k;j;kg)(3) 59

PAGE 60

andtakingitsexpectedvalue,weget Eh@2 @h;l@p;m[logp]i=2h;l(x)p;m(x)dx=2hplm:(3)(Note:forreadabilitywehaveletpp(X;fj0;k;j;kg).)TheHessianof( 3 )isconstantandisconsequentlyequaltoitsexpectedvalue,i.e. @2 @h;l@p;mh=Eh@2 @h;l@p;mhi=2hplm:(3)Both( 3 )and( 3 )are2Imatrices.Hence,referringbackto( 3 ),inorderforHL=4Iwerequire=1attheoptimalsolutionpoint;theproofofwhichisobtainedthroughalgebraicmanipulationoftheLagrangian's,eq.( 3 ),rst-ordernecessaryconditions.Intuitively,whatwehaveshownisthatunderanorthonormalexpansionofthesquarerootofdensity,theFisherinformationmatrixessentiallyspeciesahypersphere[ 81 ]. 3.3.3EcientMinimizationusingaModiedNewton'sMethodInlightofthediscussionintheprevioussection,weproceedtodesignanecientoptimizationmethodtoiterativelysolveforthecoecients.ANewton'smethodsolutionto( 3 )wouldresultinthefollowingupdateequationsatiteration 264x+1+1375=264x375264HLA(A)T03751264lh375(3)wherex=(j0;k;j;k),A=rh(x),l=[rnll(x)+rh(x)],h=h(x),andHL=HL(x)andnll()def=logp(X;).WhentheHessianispositivedenitethroughoutthefeasiblesolutionspace,itispossibletodirectlysolveforxand[ 82 ].For( 3 ),itiscertainlytruethatxTHLx>0over.InordertoavoidthecomputationallytaxingHLupdateateachiteration,weadoptamodiedNewton'smethod[ 82 ].ModiedNewtontechniquesreplaceHLbyB,whereBisasuitableapproximationtoHL.Herewecan 60

PAGE 61

takeadvantageofthefactweknowHLattheoptimalsolutionpoint;hence,weletB=HL=4I.Inpractice,thismethodisimplementedbysolvingthesystem +1=C1[hFrnll(x)];d=B1[IAC1F]rnll(x)B1AC1h:(3)whereC=(A)TB1A,F=(A)TB1andthecoecientupdatesaregivenbyx+1=x+d.NoticethattheseequationscanbesimpliedevenfurtherbytakingadvantageofthesimplestructureofBtoavoidexplicitmatrixinversesandmakingitveryecientforimplementation,i.e.setB1=1 4I.ThismethoddependsonhavingaunitstepsizeandhasconvergencepropertiescomparabletothestandardNewton'smethod[ 82 ].Itwasalsoshownin[ 83 84 ]thatusingtheknownHessianattheoptimalsolutionpointshastheeectofdoublingtheconvergenceareathusmakingitrobusttovariousinitializations. 3.3.42DDensityEstimationExtensionstobivariate,waveletdensityestimationaremadepossiblebyusingthetensorproductmethodtoconstruct2Dwaveletbasisfunctionsfromtheir1Dcounterparts[ 74 ].Thenotationbecomesnoticeablymorecomplicatedandrequirescarefulattentionduringimplementation.Let(x1;x2)=x2R2andnowthep p(x)expansionisgivenby p p(x)=Xj0;kj0;kj0;k(x)+j1Xjj0;k3Xw=1wj;kwj;k(x)(3)where(k1;k2)=k2Z2isamulti-index.Thetensorproductsare j0;k(x)=2j0(2j0x1k1)(2j0x2k2)1j;k(x)=2j(2jx1k1)(2jx2k2)2j;k(x)=2j(2jx1k1)(2jx2k2)3j;k(x)=2j(2jx1k1)(2jx2k2):(3) 61

PAGE 62

Algorithm1WaveletdensityestimationusingmodiedNewton'smethod. 1. Initializex0=j0;k;j;k.WetypicallysetallvaluestobeequalsubjecttotheconstraintPj0;k2j0;k+Pjj0;k2j;k=1. 2. PerformmodiedNewtonupdatesin( 3 )togetcoecientincrementsd. 3. Updatecoecientsaccordingtox+1=x+d: 4. RepeatSteps2and3untilconvergencewhichgivesaminimum^xtoourobjective( 3 ). 5. Useestimatedsetofcoecients^x=n^j0;k;^j;kotoconstruct^p=(p p)2asin( 3 )for1Dor( 3 )for2D. Againourgoalistoestimatethesetofcoecientsj0;k;wj;k.Asintheunivariatecase,werepeatthenecessarystepsbyrstcreatingtheobjectivefunctionthatincorporatesthenegativeloglikelihoodandtheLagrangeparametertermtohandletheequalityconstraint.Thentheminimizationprocedurefollowsaccordingto 3.3.3 .Theresultingequationsareexactlythesameformwithstraightforwardadjustmentsduringimplementationof( 3 )toincorporatethe2Dnatureoftheindicesandwaveletbasis.ThealgorithmtoperformoneortwodimensionalwaveletdensityestimationusingourmodiedNewton'smethodispresentedinAlgorithm 1 3.4ExperimentalResultsAsourworkisageneraldensityestimationtechnique,itisapplicabletoawholehostofapplicationsthatrelyonestimatingdensitiesfromobservationaldata.Theexperimentalevaluationoftheproposedmethodswasconductedonbothsyntheticandrealdata.Wemeasureperformancebyvalidationagainsttrue,analyticaldensities(both1Dand2D)andillustrateproof-of-conceptscenariosforrealdataapplicationsthatrequiredensityestimationaspartoftheirsolution.Specically,thetwoapplicationsweshowcasehereareshapealignmentandmutual-informationbasedimageregistration.Thoughourmethodhassomeadvantagesovercontemporarywaveletdensityestimators,therearestillpracticalconsiderationsthatall,includingour,wavelet-basedsolutionsbumpagainst.Theseconsiderationsarepepperedthroughoutouranalysisoftheexperimentalresults. 3.4.1OneandTwoDimensionalDensityApproximationTheapproximationpowerofthepresentmethodwasvalidatedagainsttheclassofknowndensitiesaspresentedin[ 85 ]and[ 86 ],wheretheauthorsprovideconstructions 62

PAGE 63

ofseveraldensities(whichareallgeneratedasappropriatemixturesofGaussians)thatanalyticallyexerciserepresentativepropertiesofrealdensities,suchasspatialvariability,asymmetryandconcentratedpeaks.Mostotherwaveletdensityestimatorshavealsoshowcasedtheirresultsonasmallsubsetofthesedensities.Inordertoprovidecomprehensive,robustanalysisofourmethod,weselectedthefollowing13onedimensionaldensitiestoanalyzeapproximationcapabilities:Gaussian,skewedunimodal,stronglyskewedunimodal,kurtoticunimodal,outlier,bimodal,separatedbimodal,skewedbimodal,trimodal,claw,doubleclaw,asymmetricclaw,andasymmetricdoubleclaw.Thereaderisreferredto[ 85 ]foravisualdepictionofall13densities.Duringpreliminaryempiricalevaluation,wenoticedatrendthatbestresultswereobservedwhenusingasingle-level,scalingfunctionrepresentationofthedensity.ThiswasfurtherconrmedviaprivatecommunicationwithG.G.Walterandalsodiscussedin[ 87 ].Weperformedtheestimationoverarangeofscalevalues,i.e.j0inequation( 3 ),fromj0=1to5.(Note:weinitiallyusedacrossvalidationmethod,see[ 76 ],toautomaticallyselectj0butoptedtotestoverarangetobemorethorough.SeeChapter5formoredetailsregardinghowtochoosej0.).Wealsousedthreedierentfamiliesofwaveletbasis,withmultipleorderswithinafamily,toapproximateeachofthedensitiesDaubechiesoforder1-10,Symletsoforder4-10,andCoietsoforder1-5.Theanalysesacrossfamiliesprovidesomeguidanceastotheapproximationperformancecapabilitiesofdierentbases.Alldensitieswereestimatedusing2,000samplesdrawnfromtheknownanalyticdensities.Theapproximationerrorofeachtestwasmeasuredusingtheintegratedsquarederror(ISE)betweentheknownpandestimated^pdensities,i.e.R(p^p)2dx.Thiswascomputedbydiscretizingthe1Dsupportofthedensityatequallyspacedpointsandthensummingareameasures.Thebestwaveletbasis,orderandstartinglevelj0wereselectedbasedonthiserrormeasure.Weexecuted15iterationsofourmodiedNewton'smethodwithmanydensitiesconverginginfewer(8to10)iterations15iterationson2000samplestakesapproximately30secondswithaMatlabimplementation.We 63

PAGE 64

comparedthewaveletreconstructeddensitieswithkerneldensityestimators(KDE)[ 67 ].AllexperimentswereconductedusingaGaussiankernel,areasonablechoicesincethetruedensitiesaremixturesofGaussians,withthesmoothingparameter(bandwidthofkernel)selectedtwoways:(1)automaticallyasdescribedin[ 67 ,Ch.3]and(2)anearest-neighborvariablebandwidth,see[ 67 ,Ch.5].ThefreelyavailableKDEToolboxforMatlab[ 88 ]byA.Ihelerprovidesfast,robustimplementationofthesekernelmethods.Werefertotheautomaticallyselectedbandwidthasxedsinceitisthesameforallkernels.The1DresultsaresummarizedinTable 3-1 .Forthetwodimensionalevaluation,weselectedthefollowingvedicultdensities,i.e.onesthatexhibitedmorevariationsorcloselygroupedGaussians,from[ 86 ]:bimodalIV,trimodalIII,kurtotic,quadrimodalandskewed.TheexperimentalprocedurewassimilartothatoftheonedimensionaldensitiesandresultsaresummarizedinTable 3-2 .AgainourmodiedNewton'smethodwasabletoconvergewithfewerthan15iterations.Inall2Dtestcases,ourwaveletdensityestimatorwasabletooutperformboththexedandvariablebandwidthkerneldensityestimators.Overall,inboththe1Dand2Dcasesthewaveletbaseswereabletoaccuratelyrepresentthetruedensities.Ofthefamiliestested,therewasnoclear-cutwinnerastowhichbasiswasbetterthananother.Theperformanceofaparticularbasisdependedontheshapeofthetruedensity.SomeapplicationsmaypreferSymletsorCoietsastheyaremoresymmetricthanotherbases.Incomparisontothekernelestimators,ourmethodprovidedbetterresultsonthemoredicultdensitiesandperformedonlyslightlyworseonslowlyvaryingones.Also,therewereinstanceswherevariablebandwidthKDEprovidedalowerISEbutvisuallyexhibitedmorepeaksthanthetruedensity.Inthefuture,weplantodofurtheranalysistobetterevaluatethisbias-variancetradeowhenselectingthebestdensityapproximation.ExamplesofestimateddensitiesandsomeoftheirpropertiesareillustratedinFigures 3-1 and 3-2 64

PAGE 65

3.4.2WDEforRegistrationandShapeAlignmentInformation-theoreticapproacheshavebeenappliedtoavarietyofimageanalysisandmachinelearningproblems[ 55 89 90 ].Thesetechniquestypicallyrequireestimatingtheunderlyingdensityfromwhichthegivendataaregenerated.Weappliedournewdensityestimationproceduretotwoinformation-theoreticmethodsthatutilizetheShannonentropyofthedata.Webeginwiththewellestablishedimageregistrationmethodbasedonmutualinformation[ 72 ].Thismulti-modalregistrationmethoditerativelyoptimizesthemutualinformation(MI)betweenapairofimagesovertheassumedparameterspaceoftheirdieringtransformation.WenextimplementanadaptationofamorecontemporarymethodwhichminimizestheJensen-Shannondivergenceinordertoaligntwopoint-setrepresentationofshapes[ 49 ].Bothofthesemethodsrequireestimatingtwodimensionaldensities. 3.4.2.1RegistrationUsingMutualInformationFortheMIregistrationexperiments,weusedslicesfromtheBrainwebsimulatedMRIvolumesforanormalbrain[ 91 ].Thegoalwastorecoveraglobalanewarpbetweenanimagepair.(Note:Toexpediteexperiments,wedidnotincludetranslationsintheanewarp.)Wefollowtheanewarpdecompositionusedin[ 92 ],whichresultsinafourparametersearchspace(,,s,t).Inordertominimizeexperimentalvariability,wemanuallyimposedaknownanetransformationbetweenthesourceandtargetimage.Theoptimalparametersearchwasconductedusingacoarse-to-nesearchstrategyoverabounded,discretizedrangeoftheparameterspace.Calculatingthemutualinformationperformancecriterionbetweenimagepairsrequiresestimatingthejointdensitybetweenthem.Thisistypicallyaccomplishedusingasimple2DhistogramorParzenwindowestimator(i.e.thekernelestimatorsevaluatedin 3.4.1 ).(Analternativetothesekernelmethodsdiscussedin[ 93 ].)Wereplacethesemethodswithourwaveletbaseddensityestimator,leavingtherestofthealgorithmunchanged.Figure 3-3 showsthesourceandtargetimagesusedinthetrialsandtheresultsarelistedinTable 3-3 .Intheabsence 65

PAGE 66

ofnoise,wewereabletoperfectlyrecoverthetransformationparametersusingourmethodwhereasKDE(bothxedandvariablebandwidth)failedtoestimatetheoptimalparameters.Inthenoisetrial,wewereabletocorrectlyrecovertwo,sandt,outoffourparameterswiththeothertwo,and,valuesonlymissingthegroundtruthbyoneandtwodiscretizationsteps,respectively(seeTable 3-3 ).TheKDEperformedaboutthesameasourmethodinthesenoisetrials.AllexperimentswereconductedusingtheDaubechiesorder1(db1)family,withamulti-resolutionbasisstartingatlevelj0=3stoppingatj1=2.Thismeansthatbothscalingandwaveletfunctionswereusedinthedensityestimation,seeFigure 3-4 foranexample.Itisalsopossibletouseotherbasisfamilies.Becauseouroptimizationdoesnotuseastepsizeparameter,wedidencountersomecaseswherechoosingabadstartinglevelcausedconvergenceissues.Thiscanberemediedbyutilizinganystandardoptimizationmethodthatincorporatesalinesearchtocontrolthedescentdirection.Currentlyweareusingbothqualitativeanalysis(visualinspection)anda2Dversionofthecross-validationprocedure,asreferencedaboveintheapproximationexperiments,inordertoselectthestart(j0)andstop(j1)levels. 3.4.2.2ShapeAlignmentUnderHellingerDivergenceNextweappliedthisdensityestimationmethodtoshapeanalysis.Theapplicationsoflandmarkandpoint-setbasedshapeanalysisareoftencastinaprobabilisticframeworkwhichrequiresadensityestimationprocedure.Thecompact,localizednatureofthewaveletbasisallowsonetomodelarichclassofshapes,withintricatestructuresandarbitrarytopology.Wequalitativelyillustratethisbyestimatingthedensitycorrespondingtoadogsnappershshapeconsistingof3,040points,Figure 3-5 (A).ThedensityestimationwascarriedoutusingaCoiet4basiswithonlyscalingbasisfunctionsstartingatlevelj0=3.TheestimateddensityisshowninFigure 3-6 .Noticehowthewaveletbasiscapturesthedetailedstructuressuchasthensandcloselyhugsthespatialsupportregionoftheoriginalpoints.Weuseourestimationmethodinacorrespondence-freeregistrationframework,asdescribedin[ 48 49 ],torecoverananetransformation 66

PAGE 67

betweentwopointsets.In[ 49 ],theirgoalwastodetermineaprobabilisticatlasusingJensen-Shannondivergence.Here,wewishtoshowcasethewaveletdensityestimatorforprobabilisticshapematching,asin[ 48 ].However,ratherthanusingtheKullback-Leiblerdivergencemeasureasin[ 48 ],weelecttousetheHellingerdivergenceinstead[ 73 ]: DH(p1;p2)=R2(p p1p p2)2dx=22Pj0;k(1)j0;k(2)j0;k2Pj1jj0;k(1)j;k(2)j;k(3)where((1);(1))and((2);(2))arethewaveletparametersofp1andp2respectively.TheHellingerdivergenceisalsocloselyrelatedtothegeodesicdistanceonaspherewhereeachpointonthesphereisawaveletdensity.TheadvantageinusingtheHellingerdivergence(orthegeodesicdistance)overtheKullback-Leiblerdivergenceisthatthedivergenceisinclosedformanddoesnotneedtobeestimatedfromthedatausingalawoflargenumbers-basedapproachasin[ 48 49 ].Inordertocontrolexperimentalvariability,weusedabruteforcecoarse-to-nesearchovertheaneparameters.Thetargetshape'sdensity,p1in( 3 ),isestimatedonceatthebeginningusingourmethod.Ateachiterationoftheaneparameters,thesourcepointsetisdeformedbythecurrentaneparametersandanewp2isestimatedwiththewaveletdensityestimatorusingthesetransformedpoints.TheHellingerdivergenceerrorcriterionin( 3 )isminimizedwhenthetwodensitiesarebestalignedandthisinturngivestheoptimalparametersoftheanetransformation.FollowingastrategysimilartothosedescribedintheMIexperiments,wewereabletosuccessfullyrecovertheanetransformation.ThegroundtruthaneparameterswerethesameasintheMItests.Intheseexperiments,theKDEusingxedandvariablebandwidthsagainfailedtoestimatealloftheparameterscorrectly(seeTable 3-4 ).SomeoftheKDE'sinaccuraciescouldbeattributedtothefactthat( 3 )isavailableinclosedformunderourrepresentationbuthastobenumerically 67

PAGE 68

computedfortheKDE.Figure 3-5 (B)illustratesthesourceandtargetpointsetusedinthismatchingexperiment.Ourexperimentshavedetailedboththeapproximationpowerandpracticalutilityofourproposedwaveletdensityestimator.Theestimatorisabletoaccuratelyrepresentalargeclassofparametricandnon-parametricdensities,awell-knowntraitofwaveletbases.Ourmethodrobustlysatisestheintegrabilityandnon-negativityconstraintsdesiredfromdensityestimatorswiththeaddedlocalizationbenetsinherenttowaveletexpansions.Thisallowedustoseamlesslypluginourtechniqueintoseveralapplicationsthatcriticallydependonassessingdensitiesfromsampledata. 3.5DiscussionWehavepresentedanewtechniquefornon-negative,densityestimationusingwavelets.Thenon-negativityandunitintegrabilitypropertiesofbonadedensitiesarepreservedthroughdirectlyestimatingp pwhichallowsonetoobtainthedesireddensitythroughthesimpletransformationp=p p2.Insharpcontrasttopreviouswork,ourmethodcaststheestimationprocessinmaximumlikelihoodframework.Thisovercomessomeofthedrawbacksofmethodsthatrequiregoodpre-estimatorsforthedensitywearetryingtond.Themaximumlikelihoodsettingconsequentlyresultedinaconstrainedobjectivefunctionwhoseminimizationyieldedtherequiredbasisfunctioncoecientsforourwaveletexpansion.WewereabletodevelopaecientmodiedNewtonmethodtosolvetheconstrainedproblembyanalyzingtherelationshiptotheFisherinformationmatrixunderthewaveletbasisrepresentation.WeshowedthattheHessianmatrixatthesolutionpointofthemaximumlikelihoodobjectivefunctionhadahighlystructuredandsimpleform,allowingustoavoidmatrixinversestypicallyrequiredinNewton-typeoptimization.Vericationofourproposedmethodwasrstempiricallydemonstratedbytestingthismethod'scapabilitytoaccuratelyreproduceknowndensities.Successwasillustratedacrossarangeofdensitiesandwaveletfamiliesandvalidatedagainstkerneldensityestimators.Wealsoappliedtheestimationprocesstotwoimageanalysis 68

PAGE 69

problems:mutualinformationimageregistrationanddensityestimationforpoint-setshapealignment.Bothillustratedthesuccessfuloperationofourmethod.Asageneraldensityestimationprocedure,thismethodcouldbeappliedtonumerousapplications.Thecompact,localizednatureofwaveletsandthelargeclassoffunctionstheyarecapableofrepresentingmakethemanexcellentchoicefordensityestimation.InChapter4,weleveragetheusefulpropertythattheHellingerdivergenceandthegeodesicdistancebetweentwowaveletdensities(inthesamefamily)areavailableinclosedformanddevelopashapematchingandindexingframework. Table3-1.1Ddensityestimation.Optimalstartlevelj0wasselectedbytakinglowestISEforj02[1;5].(yBWisbandwidth.) WDE KDE Bestbasis j0 ISE FixedBWyISE Var.BWyISE Gaussian SYM10 -1 3.472E-04 3.189E-04 5.241E-03 SkewedUni. DB7 1 3.417E-04 1.970E-04 7.551E-03 Str.SkewedUni. SYM7 3 2.995E-03 6.947E-02 7.610E-03 KurtoticUni. COIF2 2 2.399E-03 2.869E-02 1.388E-02 Outlier SYM10 2 1.593E-03 3.911E-03 3.962E-02 Bimodal COIF5 0 5.973E-04 2.084E-04 4.223E-03 Sep.Bimodal SYM7 1 5.354E-04 6.237E-03 5.419E-03 SkewedBimodal DB10 1 8.559E-04 1.461E-03 3.885E-03 Trimodal COIF3 1 9.811E-04 1.439E-03 3.787E-03 Claw SYM10 2 1.511E-03 3.692E-02 7.014E-03 Dbl.Claw COIF1 2 2.092E-03 1.795E-03 5.283E-03 Asym.Claw DB3 3 2.383E-03 1.373E-02 6.490E-03 Asym.Dbl.Claw COIF1 2 2.250E-03 4.759E-03 4.940E-03 Table3-2.2Ddensityestimation.Optimalstartlevelj0wasselectedbytakinglowestISEforj02[1;3].(yBWisbandwidth.) WDE KDE Bestbasis j0 ISE FixedBWyISE Var.BWyISE BimodalIV SYM7 1 6.773E-03 1.752E-02 8.114E-03 TrimodalIII COIF2 1 6.439E-03 6.621E-03 1.037E-02 Kurtotic COIF4 0 6.739E-03 8.050E-03 7.470E-03 Quadrimodal COIF5 0 3.977E-04 1.516E-03 3.098E-03 Skewed SYM10 0 4.561E-03 8.166E-03 5.102E-03 69

PAGE 70

Table3-3.Mutualinformationregistrationresults.Theleft-mostcolumnareaneparameters(seetext).Withnonoise,2=0,theparameterswereexactlyrecoveredwhenusingWDEwhereasKDEestimateswereslightlyo.Withnoiseadded,therecoveredvalueswereclosetothetruthbuttherewasnotasignicantadvantageusingKDEversusourmethod. Truth WDE FixedBWKDE Var.BWKDE 2=0 2=0:05 2=0 2=0:05 2=0 2=0:05 10 10 9.8 9.8 10.2 10 10.4 -5 -5 -5.4 -5 -4.8 -4.6 -5 s 0.3 0.3 0.3 0.3 0.3 0.2 0.3 t -0.1 -0.1 -0.1 -0.1 -0.1 0 -0.1 Table3-4.Hellingerdivergenceshapealignment.TheWDErecoversallofthetransformationparametersexactly. Truth WDE FixedBWKDE Var.BWKDE 10 10 9.8 9.8 -5 -5 -5.2 -5.2 s 0.3 0.3 0.2 0.2 t -0.1 -0.1 -0.1 -0.1 70

PAGE 71

A B C D E F G H Figure3-1.1DDensityestimationcomparison.Trueanalyticdensityissolidline,WDEisdashedlineandKDEisdottedline.A)FixedBWKDEperformsslightlybetterthanourWDEonbimodaldensity.B)WDEcapturesthemiddlepeakoftrimodaldensity.ThexedBWKDEmissesit.C)FixedBWKDEunderestimatespeaksofseparatedbimodaldensity.D)VariableBWKDEhaslowerISEthanxedBWKDEin(C)butincorrectlygivesseveralpeaks.E)WDEcapturesthemainpeakareaofkurtoticdensity,xedBWKDEfails.F)VariableBWKDEalsofailsonkurtoticdensityestimatingseveralpeaks.G)TheWDEcapturesallpeaksofthisclawdensitywhilethevariableBWKDEovershootspeaks1,3and4.H)ThexedBWKDEhasalowerISEthanWDEondoubleclawdensitybutitmissesallthesharppeaks. 71

PAGE 72

ABC DEF Figure3-2.2DDensityestimationcomparison:WDEversusKDEcontours.SeeTable 3-2 forWDEestimationparameters.A)TrueTrimodalIII.B)WDEof(A).C)FixedBWKDEof(A).D)TrueQuadrimodal.E)WDEof(D).F)VariableBWofKDEof(D). AB CD Figure3-3.Registrationusingmutualinformation.A,B)Registrationimagepair.C)Anewarpappliedto(B)withoutnoise.D)Targetimage(C)withnoise. 72

PAGE 73

Figure3-4.Exampleofjointdensityestimationfromtwoimagesutilizedinregistrationexperiments;scalingandwaveletfunctionsfromHaarbasisusinglevelsj0=3toj1=2. AB Figure3-5.Fishpointsets.A)Dogsnapperrepresentedby3,040points.B)Overlayofsourceandtargetshape(lightershade)usedinHellingerdivergencebasedregistration. Figure3-6.Exampleof2DdensityestimatedfromshpointsetusingCoiet4,onlyscalingfunctionsatlevelj0=3. 73

PAGE 74

CHAPTER4SLIDINGWAVELETSFORINDEXINGANDRETRIEVAL 4.1IntroductionToday'sscientic(andnon-scientic)communitygeneratesinformationatafranticpace;thusplacingaparamountemphasisondevelopingexibleandrobustsystemsforminingthedata.Often,thedesiresaretoclassifytestdata,clustersimilargroupsordiscovertheclosestmatchtoanincomingquery.Thekeyenablersoftheseoperationsarethesimilaritymetricsusedforqueryingthedata[ 3 ].Inthischapterwefocusonsimilaritymetricsforshapemodelshavingapplicabilitytoavarietyofdisciplines,e.g.medicalimaging,remotesensingandrobotics.Ourframeworkintroducesanewshaperepresentationandthenusesthenaturalgeometryarisingfromthisrepresentationtoderiveageodesic-distance,similaritymetric.ThepresenteortismotivatedbythewaveletdensityestimationmethodofChapter3thatestimatesp p(x)andthenobtainsabonadedensityasp p(x)2.Thishasseveraladvantagesoverestimatingp(x)directlysuchasguaranteeingnon-negativityandimposingasimpleconstraintonthewaveletcoecients.Thisnewdensityestimatorusesawaveletexpansionofp p(x),i.e. p p(x)=Xj0;kj0;kj0;k(x)+1Xjj0;kj;kj;k(x);(4)wherej0;kandj;karecoecientsforthefather(x)andmother(x)basisfunction;thej-indexrepresentsthecurrentscalelevelandthek-indextheintegertranslationvalue.(Note:(x)and(x)arealsoreferredtoasthescalingandwaveletfunctionsrespectively.)Fornumericalimplementation,theinniteexpansionin( 4 )istruncatedtosomensetofscalelevelsandwemustalsoselectastartingscalelevelj0.Asdiscussed Contentforthischapterhasbeenreprintedwithpermissionfrom:A.Peter,A.RangarajanandJ.Ho,ShapeL'neRouge:SlidingWaveletsforIndexingandRetrieval,IEEEConferenceonComputerVisionandPatternRecognition(CVPR),(Accepted),June2008. 74

PAGE 75

inChapter3,thecoecientsin( 4 )areestimatedwithamaximumlikelihoodobjectivefunctionwhichisminimizedusingamodiedNewton'smethod.Expandingp pwithawaveletbasisservesasthespringboardtoourdevelopmentofanecientsimilaritymetricbetweenshapes.Wewillshowthatgivenpoint-setshapes,wecanusethisdensityestimationmethodtorepresentshapesasprobabilitydensitiesanaturalbyproductisthatthedensitiesvisuallyresembletheshapes.(Weconsideronlytwodimensionalshapesbutthetheoryandalgorithmicproceduresreadilyextendtohigherdimensions.)Allshapesinagivendatasetcanbesimilarlyrepresented.Thisrepresentationhasexcellentpropertieslike:(1)themultiscalewaveletcoecientsofthedensitiescanbethresholded[ 77 ]tocompressthestoragerequirements(2)severaldierentorthonormalbasescanbeusedtoestimatethedensities,thusenhancingtheirdescriptivecapabilitiesand(3)thecompactnatureofwaveletsprovidesbothspatialandfrequencylocalizationenablingthedensitiestocloselymimicshapefeatures.Basedonthisrepresentation,theintuitionforthesimilaritymetricfollowsfromconsideringthecoecientsoftheprobabilitydensityfj0;k;j;kgasthecoordinatesc=[j0;1;:::;j0;m;j;1;;:::;n;m]indexingthelocationofadensityonaunithypersphere;thenthedistancebetweentwodistributionsp1andp2indexedbytheircoordinatesc1andc2,respectively,isgivenby d(p1;p2)=cos1(cT1c2):(4)(Theunithyperspherecomesaboutfromtheconstraintsonthecoecientsasdiscussedin 4.2.2 .)Weexpandontheseintuitiveideastodevelopamatchingprocedurethatcastsdensitymatchinginalinearassignmentproblem.Thelinearassignmentisusedtohandlenon-rigiddierencesbetweenshapes.Itwarpsthedensitieswhilepreservingtheirdeningproperties,e.g.unitintegrabilityandnon-negativity.Sincethedensitiescloselyresembletheshapes,weareineectwarpingtheshapes.Itwillbeshownthatthisnon-rigidalignmentisnecessarytoobtainmoreaccuraterecognition.WhenoneusesaHaarbasis(boxfunction)forthedensityexpansion,thepermutationofthewavelet 75

PAGE 76

coecientsduetothelinearassignmentvisuallylookslikeslidingblocks.ThuswehaveinformallybrandedthisprocessasShapeL'neRougeaftertheFrenchmonikerforslidingblockpuzzles.Ourmethodhasseveralbenets,including: Shapesarenotlimitedbytopologicalconstraints(suchastheneedtorepresentshapesasclosedcurves),eliminatingextraeortoftenspentindevelopingparametrizationsorotherpreprocessing. Alloftheintensivecomputationshappenoine,e.g.thedensityestimation. Forquerying,thesimilaritymetriccomputationbetweensourceandtargetshapeisfast,satisfyingtherequirementsfordemandingindexingandretrievalapplications. Useofwaveletrepresentationsenablesexibilityincompressionandstorage. 4.1.1RelatedWorkExistingworkinshapemodelingandmatchingspanabroadspectrumofrepresentationsandtheircorrespondingmetrics.Thereareseveralrecentsurveys,e.g.[ 94 ],thatsuccinctlydescribeshaperepresentationssuchasunstructuredpoint-setsorcurves.Theyalsodetailthemyriadofsimilaritymeasuresthatprovideameansbywhichtocompareshapesunderacommonrepresentation.Theadvancesmadebyallthesemethodshavebeeninstrumentalinenablingrobustindexingandretrievalmechanisms.Becauseweincorporatealinearassignmentsolvertohandlenon-rigiddeformations,ourmethodissituatedincloseproximitytotechniquesthatusetransportationandassignmentproblemformulations[ 82 ]toobtaintheirdistancemeasures.OnesuchmeasureistheEarthMover'sDistance(EMD)[ 95 ],ametricbetweengeneralmassdistributionsofobjects.Giventwodistributionsxandy,thegoalbecomestondamatrixfi;jthatestablishesaowbetweenallfeaturesxiandyjinxandy.Feasibleowsmustsatisfyrowsum,columnsumandtotalsumconstraints.ObtainingtheowandsubsequentlytheEMDisgenerallybasedonthesolutiontothetransportationproblem[ 96 ].Hence,oneofthemaindierencesbetweenourapproachandEMDisthatwesolveamatchingproblemincontrasttothetransportationproblem.TheEMDalsorequiresonetodecideonthefeaturesaswellastheappropriateweightingofeachfeatureperobject.Forsomeapplicationsthesechoicesmayalreadybereadilyapparent,butformostthisrequiresan 76

PAGE 77

addedlevelofeortandinvestigation.Ourmethodsimplyworksonthepointsetsthatnaturallyariseeitherfromsamplingorpreprocessing.ThispresentmethodalsofallsinthesameparadigmastheshapeanalysisframeworkinChapter2,whichusesgeodesicdistancesonthemanifoldofGaussianmixturemodels(GMMs)toestablishashapesimilaritymetric.RecallthatpreviouslywerepresentedshapesasmixturemodelsandusedtheFisher-Raometricderiveddirectlyfromtherepresentationtoobtainintrinsicdistancesonthemanifoldofparametricmixtures.Likethismethod,thepresenttechniquealsoleveragesthegeometrythatresultsdirectlyfromtheshaperepresentation.However,whenusingGMMsitisnotfeasibletousetheresultingmetricforretrievalbecausethegeodesicsarenotinclosed-form.(GMMspresentalargecomputationalburdenofsolvingforgeodesicdistancesonarbitrary,high-dimensionalmanifolds.)Withthepresentmethod,wehaveawellunderstoodgeometrywithaneasytocomputemetric.Theremainderofthischapterisorganizedasfollows.Inthenextsectionweprovidedetaileddiscussionsofourmethodtherepresentationofshapesasdensityfunctionsexpandedinawaveletbasis,thegeometrythatarisesfromthisrepresentationandthederivationofthesimilaritymetric.Wethenfollowwithexperimentalvericationofourmethod, 4.3 .Theindexingandretrievalaccuraciesaretestedonashapedatabaseconsistingof1400shapesfromtheMPEG-7CoreExperimentCE-Shape-1[ 97 ].Ourmethodiscomparedwithanotherdensity-matchingtechniqueforretrieval:D2shapedistributions[ 98 ],forwhichwecomputefourdierentsimilaritymeasures.WealsocompareourresultswithpublishedrecognitionratesofotheralgorithmsontheMPEG-7data.Thelastsectionconcludesbysummarizingoureortandproposingdirectionsforfuturework. 4.2ShapeL'neRougeOursimilaritymetric,thegeodesicdistanceonaunithypersphere,isobtaineddirectlyfromourrepresentationofshapesasprobabilitydensitiesexpandedina 77

PAGE 78

orthonormalwaveletbasis.Thisshaperepresentationisdetailednext,followedbyadiscussiononhowthisleadstothehyperspheregeometryforthedistributions.Afterwards,weillustratetheneedfornon-rigidalignmentandhowitcanbeaccomplishedonthespaceofdistributionsthroughalinearassignmentformulation.Itwillturnoutthatthelinearassignmentprocesshastoberegularizedtoimprovematchingperformance.Tothisend,weformulateapenaltytermthatrestrictslargemovementsofwaveletbases. 4.2.1FromShapestoWaveletDensitiesTheideaofrepresentingshapesasdensitiesisusuallybroughttofruitionintwoways.Eitherthedensityisdirectlyestimatedfromtheshape'sdiscretesamples[ 49 ]orsomeotherfeatureisrstextractedfromtheshapeandthenthedensityisttothesefeatures[ 95 98 ];ourmethodfallsinlinewiththeformer.Toourknowledge,thisisthersttimeawaveletdensityestimatorhasbeenusedtodirectlyrepresentshapes.Previoususesofwaveletsinshapeanalysis[ 99 ]havebeenmainlyrestrictedtoextractingdescriptorsofcontourshapes.Manyoftheissuesofestimatingabonadedensitycanbeovercomebyrstestimatingp p(x)andthenobtainingthedesireddensityasp p2[ 70 71 ].Fortwodimensionaldensitiesthewaveletexpansionofthesquarerootofthedensityisgivenby p p(x)=Xj0;kj0;kj0;k(x)+j1Xjj0;k3Xw=1wj;kwj;k(x)(4)wherex2R2,j1issomestoppingscalelevelforthemultiscaledecompositionand(k1;k2)=k2Z2isamulti-indexthatrepresentsthespatiallocationofthebasis.(Thetranslationrangeofkcanbecomputedfromthespanofthedataandbasisfunctionsupportsize.)Thefatherandmotherbasisaretensorproductcombinationsoftheirone 78

PAGE 79

dimensionalcounterparts,i.e. j0;k(x)=2j0(2j0x1k1)(2j0x2k2)1j;k(x)=2j(2jx1k1)(2jx2k2)2j;k(x)=2j(2jx1k1)(2jx2k2)3j;k(x)=2j(2jx1k1)(2jx2k2):(4)Thegoalistoestimatethesetofcoecientsj0;k;wj;kandreconstructthedensityusing( 4 ).TheyareestimatedusingthemaximumlikelihoodmethodofChapter3..Duetotheincreasedindexingnotationfortwodimensionalwaveletexpansion,wewilltypicallyresorttoonedimensionalarguments,asin 4.1 ,withitbeingunderstoodthatallresultsdirectlytranslatetotwodimensions.Underawaveletexpansionofp p(x),theunitintegrabilityrequirementofallprobabilitydensitiestranslatestoaconstraintonthewaveletcoecients p p(x)2dx=Xj0;k2j0;k+j1Xjj0;k2j;k=1:(4)RecallthatweareusingonlyorthonormalbasessuchasHaar,CoietsorSymlets.Figure 4-1 illustratesestimateddensitiesforfourpointsetshapes,usingasinglelevelwaveletdecomposition(withonlyscalingfunctions).ThepointswereextractedfromtheMPEG-7binaryimagedataset.Noticehowthecompactnatureofthebasesdoesanexcellentjobinmodelingtheshapefeatures.Intheoverheadviews,itisreadilyapparenthowcloselythedensitiesresembletheshapes.Wefeelthisdirectvisualassociationofthedensityandtheshapeprovidesaniceadvantageovertryingtoextractfeaturesandthentthedensitytothefeatures.Also,noticethatshapesexhibitavarietyoftopologicalpropertieslikeinteriorstructuresanddisconnectedcomponents. 4.2.2TheGeometryofWaveletDensities:GeodesicDistancesontheHyper-sphereEquation( 4 )showedthatanaturalby-productofworkingwiththesquarerootofthedensityandthenexpandingitwithanorthonormalwaveletexpansionwasthatitimposedaconstraintonthebasiscoecients;namelythesumofsquaredcoecient 79

PAGE 80

valuesmustequalone.Thisimmediatelyleadstotheinterpretationthatthebasiscoecientswhichareuniquetoaparticulardensitysincewaveletsserveasatruebasisforthespaceofcontinuousdistributionsgivethecoordinatesforapositionontheunithypersphere.Theorderingofthecoecientsinthecoordinatevectorcanbetakeninanyarrangementbutitmustbeconsistentacrossalldensities.Thedimensionalityofthehypersphereisdeterminedbythecardinalityofthesetcontainingallthecoecients.Thehyperspheregeometryofthedensitiescanbemorerigorouslyjustiedwhenweanalyzethep p(x)representationunderthetheoreticalbasisofinformationgeometry[ 23 36 ].Inthiscontext,theFisherinformationmatrix(FIM)servesasthemetrictensoronthemanifoldofaparametricfamilyofdistributions.OneofthealgebraicformsoftheFIMisgivenby gu;v=4@p p(xj) @u@p p(xj) @vdx(4)where=f1;:::;mgdenotestheparametersofthedistributionanduandvindicatetherowandcolumnindex,i.e.forafamilywithmparameterstheFIMismm.Underanorthonormalexpansionofp p(xj),Eq.( 4 )reducestothecanonicalmetrictensorofaunithypersphereembeddedinanm+1Euclideanspace.Ratherthanusethemetrictensortointrinsicallycomputegeodesicsonthehypersphere(anundertakingwhichwouldrequireustoparametrizethemanifold),wecanaccomplishthesamecomputationbyrealizingthattheconstraintPm+1i=1(i)2=1alsoimpliestheunithyperspheregeometry.Hence,closed-formgeodesicsdistancescanbesimplycomputedusingtheusualanglemeasurebetweentwounitvectors.Suchisthecaseinourframeworkwherep p(xj)hasbeenexpandedinaorthonormalwaveletbasiswiththecoecientsoftheexpansionservingastheparametersofthedensity,i.e.=fj0;k;j;kg.Twoshapesrepresentedaswaveletdensitiesendupastwopointsonthehypersphere,seeFigure 4-2 .Sincethisisaunithyperspherewiththewaveletcoecientsforeachshapeplayingtheroleoftwounitvectors,theanglebetweentheseunitvectors[Eq.( 4 )]immediatelygivesthegeodesicdistancebetweentheshapes. 80

PAGE 81

Itisalsointerestingtonotethatwecanobtainthissameinnerproductinterpretationrequiredin( 4 )bytakingtheapproachofworkingwithasimilaritymeasuredirectlybetweenthedensities,insteadofanalyzingthegeometryimpliedbythecoecientconstraintsandthemetrictensor.Inparticular,usingtheHellingerdivergence[ 73 ]tocalculatethedistancebetweentwodensitiesp1andp2gives DH(p1;p2)=R2(p p1p p2)2dx=22hPj0;k(1)j0;k(2)j0;k+Pj1jj0;k(1)j;k(2)j;ki(4)where(1);(1)and(2);(2)arethewaveletparametersofp1andp2respectively.Noticethatwecanfactorouta2anddroptheconstantwithouteectingthequalitiesofthemeasure.Thisreduces( 4 )toaninnerproductbetweenthecoecientsofthedensities,henceessentiallygivingthesamemeasureastheonewederivedabovebyanalyzingthegeometryofthespaceofdistributions(cos1()isnotpresent).ThereareothernotionsofsimilaritymeasuresbetweendensitiessuchastheKullback-LeiblerdivergenceandEuclideandistancebutnoneofthemoperateonthesquarerootofthedensityandtheyalsodonotprovideaclosed-formexpressionforthedistance.Wereferthereaderto[ 95 ]forasummaryofotherdistancemeasuresbetweendensities. 4.2.3SlidingWaveletsIfouranalysisendedwiththeprevioussection,wewouldbeequippedwithaveryfastsimilaritymetric.Givenapairofpoint-setshapes,wewouldmerelyestimatethewaveletcoecientsofthesquare-rootdensityofeachshapeandthentaketheirinnerproducttogetameasureoftheirclosenesstoeachother.However,thisapproachissomewhatnaveinthatitdoesnotleveragethefullmathematicalformalismsthatrelateoneshapetoanother.FollowingtheKleinschoolofthought[ 100 ],similaritybetweenshapesisoftenconsideredafterquotientingoutsometransformationgroup,typicallythegroupofsimilaritytransformations[ 101 ].Removingthetransformationsenablesustoanalyzeeectsthatareintrinsictotheshapes.Non-rigidtransformationsarethemostgeneral, 81

PAGE 82

basicallyencompassinganycontinuoustransformation.Practicallyitisexpectedthatmostshapesfromthesamecategoryshoulddierbysmallernon-rigidwarpscomparedtoshapesfromotherarbitrarycategories;hencecorrectingforthispriortoevaluatingthesimilaritymetricshouldenhanceitsdiscriminability.Inourframework,wecouldincorporatenon-rigidalignmentinoneoftwoways:performnon-rigidalignmentofthepointsetspriortottingthewaveletdensityortthedensitytothedataandthenadjustfornon-rigiddeformationsbywarpingthedensities.Theformermethodusuallyinvolvesadoptingasplinebasedmodeltorepresentthenon-rigidtransformation[ 40 ]andcaninvolveiterativeoptimizationtosolveforthesplineparameters.Thoughthesemethodsareabletomodelalargeclassofnon-rigiddeformations,theydonotpossessthecomputationaleciencyneededforqueryingsystems.Ourmethodtakesthesecondoptionofwarpingthedensitieswhichweaccomplishbylocallytranslatingwaveletcoecients.Wenowgiveasimpleexampletoillustratehowwarpingthedensitiesbylocaltranslationscanincreaserecognition.Supposetwoshapeshavebeenanealignedandthereonlyremainsanon-rigidwarpbetweenthetwo.Wemodelthenon-rigiddeformation,intheinnitesimal,aslocaltranslations.Figure 4-3 showstheestimateddensitiesoftwohypotheticalshapes,see(a)and(b).Thecoecientsforthebasisfunctionsofeachshapeareindicatedbyaredbar.Thedensityfunctionshownin(a)onlydiersbyatranslationtodensity(b).Noticethatifweweretostackthecoecientsinavector(frombottomlefttotopright)foreachdensityandperformaninnerproductbetweenthem,theresultingvaluewouldbezero.Thisleadstohighgeodesicdistance,cos1(0)= 2.However,ifwesimplyslidethewaveletbasesofoneshapetoaligntolocationsontheother,ourinnerproductwouldthenyieldaveryhighcorrelationindicatingthetruesimilaritybetweentheshapes.Alsowemustbecarefulthatwhatevermechanismweusetotranslatethebasesdoesnotalterthevaluesoftheircoecientsandcompromisethepropertiesofabonadedensity,i.e.( 4 )mustholdtomaintainunitintegrability.Themoststraightforwardwaytoaccommodatetheseobjectivesisto 82

PAGE 83

reformulateoursimilaritymetricundertheactionofapermutationgroupontheorderingofthecoecients.Thesespecicrequirementscanbeaddressedwithinalinearassignmentconstruct[ 82 ];thusourdeformationmodelcanbeinterpretedasaslidinggrammarwhereinweonlyallowwaveletsateachleveljtoindependentlyslidetogetagoodmatch.Theindependentslidingassumptionateachlevelimpliesthattheprobabilitydensitymasscorrespondingtoeachwaveletisindependentoftherest.Consequently,thisallowsustoindependentlyslideeachwavelettogetabestmatchwhilemaintainingtheunitintegrabilityconstraint.Whilethisjustiestheindependenceassumption,deformationgrammarsmorecomplexthanslidingcouldbeconsidered,e.g.splittingcoecients.However,werestrictedourselvestoonlyslidingthewaveletsleavingmoreexoticrulesforfutureresearch.Eventhougheachwaveletisallowedtoslide,wecannotallowtheslidingwaveletstocollideandendupatthesamespatiallocation.ThisimposesapermutationconstraintontheslidingwaveletsandtheresultingdeformationpictureevokestheL'neRougepuzzle,seeFigure 4-4 .Thusournewobjectivetominimizebecomes D(p1;p2;)=2+2hPj0k(1)j0;k(2)j0;(k)+Pj1jj0;k(1)j;k(2)j;(k)i(4)where(k)isapermutationoperatorthattakesasinputthewaveletspatialindexkandreturnsanewindexk0atthesamelevel.(Sincethewaveletcoecientscanallbereversedtogetthesamedensity,there'sanoverallsignsymmetrywhichisaccountedforinthelinearassignmentalgorithmbyrunningittwiceoncewiththesetofcoecientsfj0;k;j;kgandasecondtimewithfj0;k;j;kg.)Thespaceofpossiblepermutationsislargeandhencethisobjectiveneedstoberegularizedtoyieldusefulresults.Otherwise,everysourceshape'scoecientscouldbere-orderedtobeintheshapeofthetarget;thisisadetrimenttorecognitionsinceanyshapecanessentiallymatchanother.Toovercomethiseect,wepenalizelargespatialmovementsbyincorporatingacostbasedontheEuclideandistancebetweenthecentersofbasisfunctions.Thisrestrictslargemovements 83

PAGE 84

ofthecoecientsforcingthemtobeonlylocallytranslated.Incorporatingthispenaltygivesournalobjectivefunction E()=D(p1;p2;)+hPj0;kkr(j0;k)r((j0;k))k2+Pj;kkr(j;k)r((j;k))k2i(4)wherer(j;k)isalocationoperatoressentiallygivingusthecenterofthewaveletbasisat(j;k)whichhastwoinputs,thelevelj(andthisincludesj0),thewaveletspatialindexkandreturnsaspatiallocationr2R2.Thebasicideahereisthatastheregularizationparameterisincreased,theobjectiveincreasinglyfavorsshorterwaveletslidingmovementsandhencesmallerdeformations.Theoptimalpermutationcanbeobtainedbysettingupthecostmatrix C=c1cT2+d(4)whereciisavectorizedrepresentationofallthedensitywaveletcoecientsforshapeiandthematrixdcontainspairwisedistancesbetweenthewaveletbasislocations.Figure 4-4 illustratestheeectofonthelinearassignmentandhencethesimilaritymetric. 4.3ExperimentsThepresentedtechniquewasevaluatedontheMPEG-7database[ 97 ].Theoriginaldatasetconsistsof70dierentcategorieswith20observationspercategoryforatotalof1400binaryimages.Eachimageconsistsofasingleshape.Oneofthemainstrengthsofourmethodisitsaccessibilityandeaseofuse.Therstpartinvolvessimplytakingthedatasamplesforeachobjectandusingthemtoestimatej0;k;wj;kforthewaveletexpansionofp p.Inthecontextofshapeindexingthisphaseiscompletelyo-line,i.e.waveletdensitiesfortheentiredatabasecanbeestimatedonceandbeforetheactualsimilaritycomputationtakesplace.Next,tocomparetwoshapes,werstusetheregularizedlinearassignment( 4 )tohandlenon-rigideectsandthenuseclosed-form 84

PAGE 85

distanceonunithyperspheretoobtainthesimilaritymeasurebetweenthem.Wecomparetheperformanceofourmethod,ShapeL'neRouge,toD2shapedistributions[ 98 ].FortheMPEG-7data,eachshapewasrepresentedwithasubsetofpoints.Therearenotopologyorequalpoint-setcardinalityrequirementsamongstshapes,allowingshapeswithricherfeaturestoberepresentedwithagreaternumberofpoints,seeFigure 4-1 forsomeexamples.Inthispreliminaryeort,wehavefocusedonhandlingnon-rigideects.Tothisend,shapeswithineachcategorywereanealignedtoacategoryreferenceshape.Weusedarecentlyintroducedanealignmentalgorithmthatenablesalignmentof2Dpoint-setdatawithoutiterativeoptimization[ 102 ].Oncetheshapeswerealigned,allofthemwerebroughtintoacommoneldofviewbyplacingthemina[10;10][10;10]coordinatesystem.Thiswasdonetocontrolthetranslationrangeoverwhichweestimatethedensities.NextweestimatedcoecientsforthewaveletdensityofeachshapeusingaHaarbasiswithj0=1.Noteitispossibletouseseveralotherfamilies,buttheHaarbasisisavailableinclosedformandreducesthetimerequiredtoestimatethedensities(onaverageabout2to3minutespershape).Itisworthmentioningthatregardlessofthenumberofpointsusedtorepresenteachshape,oncethedensitiesareestimatedallofthemwillhavethesamenumberofwaveletcoecients.(Recallthatthedensitiesareallestimatedinthesamesquarecoordinatesystem.)Perthesespecications,eachwaveletdensitywasrepresentedwith1;764coecients.Oncethedensitiesareestimatedforalltheshapes,pairwisematchingbetweendensitiesonlyinvolvesworkingwiththewaveletcoecientsofthedensities.Whenmatchingtwoshapes,thewaveletdensitycoecientsofeachareusedtocreatethecostmatrixinEq.( 4 ).Withthiscostmatrix,wecanthenusethelinearassignmentsolverpresentedin[ 103 ]toobtainthewavelet-coecientrearrangementsofthesourceshapewithrespecttothetarget.Allofourexperimentswereconductedwithmultiplevaluesof.Forashapepair,ittypicallytakeslessthan5secondstoperformthelinearassignment.Oncethecoecientsarere-orderedwecanuseEq.( 4 )toobtainthegeodesicdistance 85

PAGE 86

betweentheshapes.Infactweexperimentedwiththreepossiblesimilaritymeasuresthatcanbecomputedafterthelinearassignment:(1)thestandardarclengthgeodesicdistanceafterlinearassignment,(2)geodesicdistanceplusthetotaldistancepenaltyincurredforslidingand(3)justthetotalslidingpenalty.(Note:Thelasttwometricsarenotbeconfusedwiththefactthatthedistancepenaltyisalsousedtoregularizetheslidingprocesswhichisdierentfromtreatingthetotalamountofmovementasametric.)WecomparedourmethodtoD2shapedistributionsasthisisalsoadensity-basedshaperetrievalmetric.Foreachshape,aD2shapedistributionwascreatedbytaking10;000randompairwisedistancesbetweenpointsontheshape.In[ 98 ],theauthorsthenusethesedistancestoconstructa1Dhistogramforeachshape;thisservesasauniqueshapesignature.Insteadofusinghistograms,weestimatea1Dwaveletdensityforeachshape.Distancemetricsbetweenshapescanbeobtainedbyusingavarietyof1Ddensitydissimilaritymeasures.InadditiontotheHellingerdivergence,Eq.( 4 ),wecomputedthreeothermeasures: Bhattacharyya:D(p1;p2)=1p p1p2dx 2:D(p1;p2)=(p1p2)2 p1+p2 L2:D(p1;p2)=(p1p2)2dx1 2Figure 4-5 showssomeexampleD2shapedistributionsusingthe1Dwaveletdensityestimator;thesedistributionscorrespondtoshapesshowninFigure 4-1 .PerformanceontheMPEG-7ismostcommonlyevaluatedusingthebulls-eyecriterion[ 19 97 ].Eachshapeisusedasaqueryshapeandthetop40matchesareretrievedfromall1400shapes(thetestshapeisnotremoved).Forasinglequery,maximumpossiblecorrectretrievalsare20coincidingwiththenumberofshapesineachcategory.Hencethereareatotalof28;000possiblematcheswiththerecognitionratereectingthenumberofcorrectmatchesdividedbythistotal.Table 4-1 liststherecognitionratesusingseveraldensitysimilaritymeasuresforbothShapeL'neRougeandD2shapedistributions.ShapeL'neRougesignicantlyoutperformsD2shapedistributions. 86

PAGE 87

Thisgivescredencetotheideaofworkingwithfeaturerepresentationsthatmimicthetruevisualpropertiesofshapes,i.e.D2shapedistributionsrepresentobjectsusinga1Dsignaturederivedfromthe2DpointswhereasShapeL'neRougerepresentsshapesusing2Ddensitieswhicharevisuallysimilartotheshapes.ThethreedierentmetricscomputedforShapeL'neRougeillustratehowimpactsrecognitionperformance.Ajudiciouschoiceforcanbemadebyoptimizingoveratrainingset.Thedierentmetricsalsoshowthatthewaveletdensityrepresentationprovidesarichsetoffeaturesevidentbythatthefactthegeodesicdistance(withlinearassignment)outperformsthemetricsthatincludetheEuclideandistancepenalty.Hencetheslidingaloneisnotsucienttodiscriminatebetweenshapes.(Forhigh,thetotalslidingpenaltydominatesthesecondmetricgivingsimilarperformancetothethird.)Recently,methodsbasedonhierarchicalrepresentations[ 19 20 ]havealsoreportedrecognitionratesgreaterthan85%ontheMPEG-7dataset.However,thesemethodsworkonamoresimpliedversionoftheproblemthanwhatwehaveaddressed.Theyassumeshapesarerepresentedbytheirboundaryoutlinesandtypicallyuselessthan200pointsfortheshapes.Ahierarchicalrepresentationisusedtocapturebothglobalandlocalproperties.Thesemethodshavethedrawbackofextractingoriented,boundarycurveswhichcanbeatroublesomepreprocessingprocedure.Wealsolosethedescriptivepoweraordedbyallowingarbitraryshapetopologiesandunconstrainedpointsetcardinalities.Theclosestmethod,intermsofoperatingonunstructuredpointsetsandnotrestrictingshapetopology,is[ 15 ]whichhaspublishedrecognitionrateof76:51%ontheMPEG-7dataset.Ourresultsclearlyshowreasonablegainsoverthismethod.WearestillinthepreliminarystagesofexploitingthefullcapabilitiesoftheShapeL'neRougeframework,i.e.usingmultiscalerepresentationstogetmoredescriptiveattributes,experimentingwithdierentwaveletfamilies,etc.Sincewearealreadyabove85%,webelieveinthefuturetheseenhancementswillimproveourrecognitionratessignicantlywithoutsacricingourease-of-useandrichdescriptivepower. 87

PAGE 88

4.4DiscussionThedevelopmentofrobustandeectiveshapeindexingandretrievalmechanismslargelydependsontherepresentationmodelforthedataandalsothemetricsusedtodistinguishoneobservationfromanother.Wehavepresentedanovelshaperepresentationschemewhichgivesrisetoanaturalmetricthatcomesdirectlyfromtherepresentation.Givenanunstructuredpointsetmodelofashape,ourShapeL'neRougeframeworkestimatesp p,underawaveletexpansion,directlyfromthepointdataandrecoverstheprobabilitydensityasp p2.Asweillustrated,thesedensitieshaveadirectvisualsimilaritytotheoriginalshape.Theunitintegrabilitypropertyofalldensitiestranslatestoaconstraintonthewaveletcoecients,i.e.thesumsquaredcoecientsequalone,seeEq.( 4 ).Sincethedensitiesareuniquelyidentiedbytheirwaveletcoecients,theseareineectthecoordinatesbywhichprobabilitydensitiesareindexedonaunithypersphere.Andsincethedensitiesrepresenttheoriginalshapes,intuitivelytheshapesarealsoontheunithypersphere.Asaresultofthisrepresentation,weimmediatelygainanaturalsimilaritymeasurebetweenshapesbycomputingthearclengthbetweenprobabilitydensitiesontheunithypersphere.Shaperecognitioncanbeimprovedifweadjustfornon-rigiddierencesbetweenapairofshapesbeforecomputingasimilaritymeasure.Ratherthandothisintheoriginalshapespace,wehaveintroduceanovelwayofdeformingtheirwaveletdensityrepresentationthroughtheuseofpenalizedlinearassignment;allowingustolocallywarpthedensitywhilemaintainingitsdeningintegrabilityandpositivityproperties.Thepresentedframeworkhasseveraladvantagesoverothercontemporaryshapemodelingandmatchingschemes: Eachshapecanhaveanarbitrarynumberofpointswithouttopologicalrestrictions.Thisisinsharpcontrasttomethodsthatworkonlyonshapesilhouettesorarelimitedtoonlyafewsamplepoints.Hence,thecardinalityofashapepointsetisdictatedbytheamountofpointsneededtoaccuratelyrepresentashape'sfeaturesandnotbyalgorithmiclimitations. 88

PAGE 89

Limitedpreprocessingisrequiredsincewedirectlytaketheshapepointsandestimatethedensity. Themetricisinclosedformandwhenincorporatinglinearassignmentourmethodisstillcomputationallyecientenoughforqueryingapplications. Figure4-1.Examplewaveletdensitiesestimatedfrompoints-setsofMPEG-7shapes.Toprowarepointsets,cardinalityfromlefttoright:4,948;5,578;7,773;11,984.Secondrowisanadirviewoftheestimateddensitiesusingthefollowingwaveletfamilies(fromlefttoright):Haar(j0=2),Coiet-4(j0=1),Symlet-10(j0=0)andHaar(j0=2).Thirdrowistheperspectiveview.Noticehowthewaveletdensitiesaccuratelyrepresenttheshapes. Figure4-2.Hypersphereofdensities.UnitintegrabilityfordensitiesrequiresPj0;k2j0;k+Pj1jj0;k2j;k=1,alsotheFIMisreducedtothecanonicalmetricoftheunithyperspherewhenp pisexpandedinanorthonormalbasis.Thisplacestheshapesrepresentedbythedensitiesonunithyperspherewithcoordinatesgivenbythewaveletcoecients.Theabovegureshowstwodensities,seecoecientsuperscript,onthehyperspheretheirgeodesicdistanceistheanglebetweentheunitvectors. 89

PAGE 90

AB c1=h001 p 30001 p 30001 p 300000iTc2=h000001 p 30001 p 30001 p 300iT Figure4-3.Localnon-rigideectsandtheneedforlinearassignment.A)Densityp1oftherstshape,withonlyscalingcoecients,c1=h(1)j;kiT,shown.B)Thesecondshapewithdensityp2withcoecientsc2=h(2)j;kiT.Locallythepointsetsonlydieredbyatranslationwhichresultedinthedensitiesdieringbyatranslation.Withoutlinearassignmentthecoecientvectorsofthesewouldgiveainnerproductof0andconsequentlylargegeodesicdistanceonthehypersphere.Linearassignmentcancorrectlyrecoverthelocaltranslationandthenthegeodesicdistancewillbesmall,reectingthetruesimilaritybetweentheshapes. Figure4-4.Eectsofonlinearassignment.Toprowfarleftistargetshapeandfarrightisthesource.Secondrowshowsforsmallthesourceshapeisalmostperfectlytransformedtothetargetwhileforlargethesourceshaperetainsoriginalshape;valuesfromlefttoright:10,250,500,and1000.Thirdrowillustratesthewaveletcoecientsmovementinrowtwo(bestviewedincolor).ThedensitieswereestimatedusingtheHaarfamilywithj0=1. 90

PAGE 91

Figure4-5.ExampleD2shapedistributionsusingwaveletdensitiesestimators..ThesedistributionscorrespondtoshapesinFigure 4-1 fromlefttoright.AlldensitieswereestimatedusingaSymlet-7,j0=1. Table4-1.MPEG-7recognitionrate.OurmethodShapeL'neRougeoutperformsD2ShapeDistributions[ 98 ].Inourmethodthechoiceofeectstherecognitionrate.Seetextforexplanationofmetrics.(LAlinearassignment,EDPEuclideandistancepenalty). ShapeL'neRouge Metrics =500 =2250 Geodesicw/LA 81:7% 85:25% Geodesic+EDP 32:6% 12:1% EDP 32:5% 11:8% D2ShapeDistributions Metrics 2 59:3% Hellinger 58:6% Bhattacharyya 58:6% L2 56:6% 91

PAGE 92

CHAPTER5MDLFORWAVELETDENSITYESTIMATION Numquamponendaestpluralitassinenecessitate.(Pluralitiesarenevertobeputforwardwithoutnecessity.)WilliamofOckham(Ockham'srazor). 5.1IntroductionInChapter3,webrieymentionedthatselectingthelevelsofamultiscalewaveletdecompositioncanbeaddressedinaprincipledwaywithinthemodelselectionparadigm.WenowreturntothisissuebyinvestigatingtheapplicationoftheMinimumDescriptionLength(MDL)criterion[ 104 105 ]toselecttheoptimalmultiresolutionlevel(s)forwaveletdensityestimation(WDE).Thoughinitiallyderivedfrominformationtheoryprinciples,morerecently[ 21 106 ]ithasbeenshownthatMDLcanbederivedfromdierentialgeometryargumentsinthemodelspace.ThisinterpretationiswellsuitedforourframeworksincewehaveasimplegeometryassociatedwiththeWDEmethodofChapter3:alldensitieslieonaunithypersphere(see 4.2.2 ).Duetothiswell-understoodgeometry,itwillturnoutthattheMDLcomplexitytermisavailableinclosed-formatremendoustheoreticalandcomputationalbenetusuallyrelegatedtoverysimpliedmodels.Theremainderofthischapterisorganizedasfollows.Thenextsection, 5.2 ,providesahigh-leveldiscussionofthemodelselectionparadigmandexaminessomeofthepopularselectioncriteria,e.g.theAkaikeandBayesianinformationcriteria.WethenprovideabriefrecapofthegeometricderivationofMDLandinterpretationofthevolume-basedcomplexityterms.In 5.3 ,weexaminetheuseofMDLforselectingtheoptimalwaveletdecompositionlevelforestimatingthesquare-rootoftheprobabilitydensity.ExperimentstovalidatetheuseofMDLinthecontextofwaveletdensityestimationareprovidedin 5.4 ,wherefavorableresultsareobservedforavarietyof1Ddensityestimates(apreliminary2Dexampleisalsoillustrated).WeconcludewithadiscussiononthepracticalapplicationofMDLforourWDEframeworkanddirectionsforfutureresearch. 92

PAGE 93

5.2MotivationandRelatedWorkThemaximatthestartofthischapterprovidesthedrivingmantrabehindallmodelselectionapproaches:selectmodelsthatareaccurateandsimple.Howonegoesaboutmathematicallydeningaccuracyandsimplicityisamatterofphilosophicaldebateanddiscussionbutultimatelytheirdenitionsmustreectthefactthatwewanttondmodelsthatareabletocorrectlydescribetheobserveddatawhilehavingtherightamountofcomplexitytohandlefutureobservations.Butwhydevelopcriteriathatconsiderbothgoodness-of-t(accuracy)andcomplexity?Isn'tgoodness-of-tenough?Ofcoursegoodness-of-tisalwaysrequiredbutthebesttisundoubtedlygivenbythemostcomplexmodels.Considerforexamplethecaseof1-DpolynomialregressionwithNpointsofdata.AnypolynomialwithdegreeN1orgreatercanreproducetheobserveddatawithouterror.However,atusinghigherdegreepolynomialssuersfromovertting;hencethereisaneedtobalancethegoodness-of-twiththecomplexitylevelofacandidatemodelclass.ThisisthelawofparsimonyembodiedbyOckham'srazor.Severalmodelselectioncriteriahavebeenproposed[ 107 ],butarguablythefollowingarethemostcommonlyused:Akaikeinformationcriterion(AIC)[ 108 ],Bayesianinformationcriterion(BIC)[ 109 ]andMDL.AfourthBayesianmodelselection(BMS)[ 110 ]hasbeenproventobeasymptoticallyequivalenttoMDL[ 106 ].Thebasicpremisebehindtheresultingfunctionalformofthesecriteriaistoassignagoodness-of-tmeasure(viathelikelihoodoftheobserveddatasample)andacomplexitypenaltythatcandependonthenumberofparametersinthemodelaswellasthesamplesize.TheAICcriterionisgivenby AIC=2lnp(Ej^)+2k(5)andBIC BIC=2lnp(Ej^)+kln(N);(5)whereEistheevidence(currentobserveddatasamples),^themaximumlikelihoodestimate(MLE)oftheparameters,Nthenumberofsamples,andkthecardinalityofthe 93

PAGE 94

modelparameters.Forexample,k=2foralinearmodelwheretheparameterscorrespondto(m;b),i.e.theslopeandinterceptoftheline.InthecontextofWDE,kwillrepresentthenumberofcoecientsperthemultiresolutiondecompositionstructure.Foreachcriterion,thebestmodelistheminimizerofthesemeasures.BothAICandBICrewardpaucityofparametersasapenaltyispaidforlargevaluesofk.SinceBIC'ssecondtermalsoincorporatesthesamplesize,ittendstoprefersmallercomplexitymodels(versusAIC)forsamplesizesgreaterthaneight.Aftereightsamples,thesecondtermofBIC,kln(N),alwayshasalowervaluethanAIC'ssecondterm,2k.ThecomplexityofamodelunderAICandBICisonlymeasuredbythecardinalityoftheparameters.Thisisthebasicdeparturepointofthese(andothers)versusMDL:theyfailtotakeintoaccountthefunctionalform(howtheparametersinteractinthemodel)ofthemodels.Forexample(takenfrom[ 107 ]),AICandBICwouldassignthesamelevelofcomplexitytobothofthesetwo-parameterpsychophysicsmodels: Stevens'sModel:Y=1X2+Fechner'sModel:Y=1ln(X+2)+:(5)TheMDLcriterionisgivenby MDL=lnp(Ej^)+k 2lnN 2+lnq detgij()d(5)whichhasanextraterm(thethirdterm)thatpenalizesbasedonthe(logarithmofthe)volumeoccupiedbythemodel'smanifoldinthespaceofprobabilitydistributions(moreonthisinthenextsection).(Asdenedpreviously,inChapters2-4,gijistheexpectedFisherinformationoftheparametricdistribution,akaFisherinformationmetrictensor.).Forthepsychophysicsmodelsin( 5 ),itwasshown[ 107 ]thattheSteven'smodelhadagreatervolumethanFechner'sandonlyMDL(versusseveralcompetingcriteriaincludingAICandBIC)wasabletocorrectlyselectbetweenthesemodelsundergeneralizabilitytests.Chronologically,eq. 5 isthemorerecentversionofMDL[ 105 ],theoriginalMDL[ 104 ]wassimilartoAICandBICinthatitonlycontainedthersttwotermsin( 5 ), 94

PAGE 95

thuslackingapenaltybasedonthefunctionalform.Ourexperimentsin 5.4 willassesstheusefulnessofincorporatingtheadditionalvolumeterm.Inpracticallyallusefulmodels,theRiemannianvolumetermin( 5 )mustbecomputedbytruncatingtheparameterspaceandusingnumericaltechniquessuchasMonteCarlointegration.Itwillbeshownin 5.3 thatinthecontextofourWDEframework,thistermisknowninclosed-form.MDLwasoriginallydevelopedusingcodingtheoryargumentsthatarebasedonthenotionofndingtheshortestcodetodescribetheobserveddata[ 111 ].Themoreregularityinthedatatheshorterthecode.Shortercodelengthscanbeshowntobeinverselyproportionaltothelikelihoodofobservingthedata,i.e.higherprobabilitiesareassociatedwithshortercodelengthsandsmallerprobabilitieswithlargecodelengths.Hencetheuseoftheterminology'minimumdescriptionlength'tondthebestmodel.Thecriterionasgivenin( 5 )isanapproximationtothecodelengthforthemaximum-likelihoodcode[ 105 ].Inthenextsection,weillustratehowMDLcanbere-derivedusingdierentialgeometry.Itwillallowustotransitionfromdescribingthesecondandthirdtermsofeq.( 5 )aspenaltiesforthenumberofparametersandfunctionalform,respectively.Instead,wewillseethattogethertheydetermineavolumeratiodesignedtomeasuretheellipsoidalvolumearoundthemaximumlikelihoodestimaterelativetothetotalvolumeoccupiedbythemodelinthespaceofprobabilitydensities. 5.2.1MDLfromDierentialGeometryInthissectionwerecapthegeometricdevelopmentofMDLasrstpresentedbyBalasubramanian[ 106 ].Theauthorreferstothemodelselectioncriterionastherazor.ItisasymptoticallyequivalenttoMDL.ThederivationsbeginfromaBayesianapproachbyconsideringtheposteriorofaparametricmodelclassM p(MjE)=p(M)p()p(Ej)d p(E)(5)where2Rdaretheparametersofthemodelclass.Hence,p()isthepriordistributionontheparametersandp(Ej)isthelikelihood.Whencomparingtwocandidatemodel 95

PAGE 96

classesM1andM2,wecandropp(E)sinceitiscommonfactorandwecanalsoomittheprioronthemodels,p(Mi),byassumingtheyareequallylikely.(Toavoidaberrantcases,weassumethroughoutthattheparameterspacesofcandidatemodelsarecompact.)Theseassumptionsreduce( 5 )top(MjE)/p()p(Ej)d:Itwasshowin[ 21 106 ]thattheJerey'sprior[ 112 ] p()=p detgij() p detgij()d(5)istheappropriatepriortochoosewhenthedesireisto:treatallparametersequally(uniform),beinvarianttoreparametrizationsoftheparameterspaceandgeometricallycountonlydistinguishablevolumesontheparameterdomain.(Thenotionofdistinguishabilitywasrigorouslyderivedin[ 106 ].)FinallyweassumetheobserveddataE=fxigNi=1arei.i.d.,hencep(Ej)=QNi=1p(xij).Withtheaforementionedsubstitutions,therazorisgivenas R(M)=p detgij()expnNlnp(Ej) Nod p detgij()d:(5)Inordertousetherazorforpracticalevaluationofcandidatemodels,theintegralinthenumeratorofEq.( 5 )mustbeapproximatedaroundthemaximumlikelihoodestimateoftheparameters,^.(TheintegralapproximationtechniqueisreferredtoastheLaplaceapproximation[ 113 ].)Toasecondorderapproximation,thisyieldsthenalversionoftherazor (M)=lnR(M)=lnp(Ej^)+k 2lnN 2+lnq detgij()d+1 2lndet~gij() detgij()(5)where~gijistheempiricalFisherinformation[ 114 ]computedfromourobservedsamplevalues.Noticethattherstthreetermsof( 5 )correspondtotheMDLcriterionin( 5 ).ThelasttermconsiderstheratiooftheexpectedFishertotheempiricalFisher,whichhas 96

PAGE 97

thepropertythatasN!1,~gij!gij(empiricalFisherapproachesexpectedFisher)sothistermvanishes;givingusbacktheMDLeq.( 5 ).TobetterunderstandtheconnectionofMDLtotheRiemannianvolumesassociatedwithamodelclass,wecanrewrite( 5 )as (M)=lnp(Ej^)+lnVM V^:(5)ThenumeratorofthesecondtermisthetotalRiemannianvolume,VM=p detgij()d,oftheprobabilisticmanifold(i.e.totalvolumeofthemodelclass).ThedenominatorV^=2 Nk 2G();whereG()=detgij() det~gij()1 2,isatermthatmeasuresappreciablevolumeofdistinguishabledistributionsaroundthemaximumlikelihoodestimatethatcomesclosetothetruth(closeinthesensethatthemodelisabletopredicttheevidenceEwithhighprobability).Asobservedabove,thisdatadependenttermhasthepropertythatG()!1asN!1.HencetheellipsoidalvolumearoundtheMLEcanbeapproximatedbyV^2 Nk 2:Giventhisapproximation,wehave (M)=MDL=lnp(Ej^)+lnVM V^:(5)HenceitcanbeseenthatMDLpenalizesmodelsthathaveexcessivelysmalldistinguishablevolumesclosetothetruth(smallV^)orthosethatoccupyalargevolumeinthespaceofdistributions(largeVM).Thevolumesinthesecondtermofeq.( 5 )areanintrinsicpropertyofthemodelandtogetherareoftenreferredtoasthegeometriccomplexityofthemodel.MDLselectsthosemodelsthathavealowgeometrycomplexitybypickingthosemodelswithhighestmaximumlikelihoodpertherelativeratioofthedistinguishabledistributions[ 21 ]. 97

PAGE 98

5.3MDLandtheGeometryofSquare-RootWaveletDensitiesUptonowwehavediscussedthederivationandinterpretationsoftheMDLcriterionforanarbitraryparametermanifoldofaprobabilisticmodelclass.WenowturnourattentiontotheapplicationoftheMDLcriteriontoselectthedecompositionlevelsforourwaveletdensityestimationframeworkdescribedinChapter3.Recallthatweexpandthesquarerootofthedensityinawaveletbasis,i.e. p p(x)=Xj0;kj0;kj0;k(x)+j1Xjj0;kj;kj;k(x):(5)Hence,wewouldliketobeabletouseeq.( 5 )todecidehowtopickthebestj0andj1.Thenumberofparameters,kin( 5 ),foraparticularchoiceofj0andj1isgivenbythecardinalityofthecoecientsetoveralllevelsofthedecomposition,i.e.k=#fg=#fj0;l;j;lg.Asdiscussedin 4.2.2 ,thecoecientsarecoordinatesforthelocationofthedensityontheunithypersphereembeddedinak-dimensionalspace.Thuseachcandidatemodel,givenbychoiceofj0andj1,isaunithypersphereandcomputingtheRiemannianvolumeVMin( 5 )amountstocalculatingthesurfaceareaofaunithypersphere.Withthisunderstanding,wenowhaveasystematicproceduretoselectthebestj0andj1: 1. Foreachvalueofj0andj1estimatethewaveletdensityusingAlgorithm 1 in 3.3 ,thiswillgiveyouthelikelihoodtermneededfor( 5 ). 2. Thecardinalityofthecoecientsetresultingfortheselectionofj0andj1willprovidethevalueofkneededtocomputevolumesVMandV^(theremainingtermsofMDL). 3. Theoptimalfj0;j1gistheonethatminimizes( 5 ).Thoughsystematic,theaboveprocessfailstotakefulladvantageofthetheoreticalconsequencesassociatedwiththeuseofwavelets.Forexample,therearesignicantcomputationalsavingsbyleveragingthenestedsubspacestructureofwaveletbases.Anotherissueisthatwemustaddressananomalythatariseswhencomputingthevolumeofaunithypersphereasthedimensionsincrease:VM!0ask!1.Thefollowingsubsectionsexpandonthesetopics. 98

PAGE 99

5.3.1MDLisInvarianttoMultiresolutionAnalysisTherstobservationwemakeisthattheMDLcriterionisinvarianttomultiresolu-tiondecompositions(consistingofscalingandwaveletfunctions)incomparisontotheircorrespondingsinglelevelscalingcounterparts.Thisisasignicantresultthatenablesustoperformourmodelsearchoverj0insteadofj0andj1.Thisresultdirectlyfollowsfromthenestedsubspacepropertyofwaveletbasesandthedyadicrelationshipofthebasisfunctionsatdierentlevels.(See 3.2 foradiscussionontheseproperties.)InordertoestablishtheinvarianceofMDLtomultiresolutionanalysis(MRA)versusanappropriatesinglelevelexpansion,wehavetoestablishthatthegoodness-of-tandgeometriccomplexitytermsareidenticalforboth.Firstletusestablishequivalenceofthegoodness-of-tasmeasuredbytheloglikelihood.Considerawaveletdensityestimateusingonlyscalingfunctionsfromanarbitrarylevelj.TheseformabasisforVj.However,functionsexpandedusingscalingfunctionsfromleveljcanbeequivalentlyrepresentedusingbothscalingandwaveletbasesthatspanlevelj1,Vj1andWj1respectively.ThenVj1canberecursivelybrokendownagainandagain.Therecursivedecompositionrelationshipgivenby Vj=Vj1LWj1=Vj2LWj2LWj1=Vj0LLj1l=j0Wl:(5)Hence,densitiesestimatedusingonlyscalingfunctionshaveanequivalentrepresentationinamultiresolutionhierarchy.Sincetheestimateddensities(eitherfromonlyleveljorMRAfromj0toj1)areequivalent,theircorrespondingloglikelihoodswouldbethesame.Sotwomodels,onewithonlyscalingfunctionsandonewithanequivalentMRArepresentation,givethesamegoodness-of-tmeasurefortheMDLcriterion.Toshowthatgeometriccomplexitiesareidentical,wehavetoestablishthatanexpansionusingonlyscalingfunctionshasthesamenumberofcoecientsasits 99

PAGE 100

correspondingMRA.ThisisclearlytruebytheverynatureofthedyadicrelationshipsbetweenlevelsinaMRA:basisfunctionsatacoarserlevelj1havetwicethesupportofthoseatlevelj,hencehalfthenumberofcoecients.Thenumberofcoecientsataparticularlevelisassociatedwiththenumberoftranslationsneedtospanadenedspatialsupport.Theoretically,aninnitenumberoftranslationsareused,seeeq.( 3 ),butforanynitesamplesetthespanoftranslationsneededtocoverthedatawillalsobenite.Thecardinalityofthecoecientsetfromaleveljwithonlyscalingfunctionswouldequalthecardinalityofcoecientsfromcoarserlevelj1thathasbothscalingandwaveletbases,i.e. k=#fVjg=#fVj1g+#fWj1g=k 2+k 2=#fVj2g+#fWj2g+#fWj1g=k 4+k 4+k 2=#fVj0g+Pj1l=j0#fWlg;(5)wherewehaveslightlyabusedthenotation#fgtocountthenumberofcoecientsforachosenbasislevel'sfunctionspace.Sincethevalueofkessentiallydeterminesthegeometriccomplexity,itwillbeidenticalforsingleleveldecompositionatleveljoraMRAfromj0uptoj1.(ThenumberofsamplesNisalsoafactorintheV^termofgeometriccomplexity,butitwillbethesameforallmodelssocanbeignoredinthisanalysis.)Withboththegoodness-of-tandgeometriccomplexityshowntobethesameforMRAversussinglelevelscalingfunctionbases,itissucientfordensityestimationtouseonlyscalingfunctionsandsearchforthebestmodelbyiteratingovervariousstartinglevelsj0.SoisMRAforwaveletdensityestimationnotneeded?Itdepends.Ifyourgoalistosimplyobtainareconstructionofthedensity,thenitcanbearguedthatscalingfunctionsaloneareenough.Butifone'sgoalissparsityamongthecoecients(whichiswhatMRAisdesignedfor),thenadierentmechanismthatmeasuresthispropertymustbeincorporatedintothemodelselectionframework.Suchameasurewouldinclude 100

PAGE 101

waveletthresholding[ 77 ]aspartofthecriterionforselectingthemodel.Thisisanavenueoffutureresearch. 5.3.2Closed-FormComputationofVMInpractice,theapplicationoftheMDLeq.( 5 )almostalwaysrequiresnumericalintegrationtocomputeVM,theRiemannianvolumeofthestatisticalmanifold.ThisinvolvesderivationoftheFisherinformationmetric(FIM),appropriatetruncationoftheparameterspacetoperformtheintegrationandothernumericaladjustmentstoinsurethattheFIMdoesnotbecomesingular.Forveryhighdimensionalparameterspaces,onehastoemployMonteCarlointegrationmethods.OnlyforverysimplemodelsisVMinananalyticform,sometimeseventheFIMisnotinclosed-formandmayrequireanadditionalnumericalintegrationstep.Onesignicantadvantageofourwaveletdensityestimationframeworkisthatallofourmodelshaveaunithyperspheregeometry.Hence,VMisknowninclosed-form.Itismerelythesurfacearea(VS)ofaunithypersphereofdimensionk1wherek=#fg=#fj0;l;j;lg.(Choosingthej0decompositionleveldeterminesthecoecientset,thecardinalityofwhichisk.)Onewouldintuitivelyexpectthevolumeofamanifoldtoincreaseasthenumberofdimensionsincrease.However,theunithypersphereexhibitsanoddpropertyinthatitdecreasesinvolume(andsurfacearea)asthedimensionsincrease[ 66 ].ThesurfaceareaofaunithypersphereSisgivenbyVS=8>><>>:kk 2 (k 2)!;keven2kk1 2(k1 2)! (k1)!;kodd:AsshowninFigure 5-1 ,themaximumsurfaceareaisreachedatdimensionsevenandthenthesurfacearearapidlydecreasestozero.RecallthatthegeometriccomplexityassessesacostbasedontheratioofthemanifoldvolumetotheellipsoidalvolumearoundtheMLE,i.e.thepenaltyislnVS V^.IftheVSshrinkstozerosofastthatitissmallerthanV^thenourpenaltytermisnotvalidsinceitwouldbecomenegative.HavingV^>VStellsus 101

PAGE 102

thatthemodelismisspecied[ 115 ].GeometricallywecanvisualizethisastheellipsoidalvolumearoundtheMLEprotrudingoutofthesmallermodelmanifold.Inpractice,onehastobecarefultoconsiderthetrade-obetweenthenumberofsamplesandthenumberofparameters.Avalidregionofwell-speciedmodelsiseasilyachievedwhenweconsiderV^=2 Nk 2.Oncewereachabovesevensamples,i.e.N7,theellipsoidalvolumestartstodeclineexponentiallyasthenumberofparameterskincreases.Sinceweneedthenumberofsamplestobegenerallygreaterthanthenumberofparameterstoavoidanill-poseddensityestimationproblem,wecaneasilysatisfyourrequirementofneedingV^
PAGE 103

thecolumnj1=2,wehaveanMRAexpansionoftheform p p(x)=Xj0=1;kj0;kj0;k(x)+2Xj=1;kj;kj;k(x):(5)Per( 5 ),thisfunctionspacespannedbytheMRAwithj0=1andj1=2isexactlyequivalenttotheonespannedbyj0=3alone.Hence,thedensitiesestimatedusingeithertheMRAorscalingfunctionsareidentical.Thisisveriedbythedensityshownunderthe'NoMRA'columnforj0=3:Giventhisunderstanding,weperformallfurthermodelevaluationusingonlyscalingfunctions.TheMDLcriterionofeq.( 5 )(denotedMDL-3inresults)wasappliedtotheselectionofthebestj0levelforourdensityestimationframeworkofChapter3.Forcomparativeanalysis,wecomputedseveralotherinformationtheoreticmodelselectioncriteria:rsttwotermsMDL(MDL-2),AICandBIC.Inaddition,sincethetruedensitiesareknown,wecalculatedthreestandarddiscrepancymeasures:mean-squarederror(MSE),Hellingerdivergence(HELL)andL1loss.Thebeststartinglevelj0wasselectedastheminimumofthesemeasuresforj02[1;6].Alargervalueofj0indicatesamorecomplexmodelsinceitcorrespondstoanerresolutionlevelinthewaveletdecomposition.TheoptimalvaluesfortworepresentativebasisfromtheDaubechies,SymletandCoietfamiliesareshowninTables 5-1 5-2 and 5-3 .MDL-3andMDL-2generallyagreedonbestlevelsacrossdensitiesandfamilies.ThereareafewcasesinwhichMDL-3(withtheadditionalvolumeterm)selectedmorecomplexmodelsthanMDL-2.Ineachofthesecases,theselectionofthehighercomplexitymodelwasjustiedbytheneedtoaccuratelycapturetheabruptvariationsofthetruedata-generatingdensities.InFigure 5-4 ,weseeexamplesoftwosuchcaseswheretheMDL-3selectedvalueofj0providesabettersuitedmodel.ThustheinclusionofthefullgeometriccomplexitylnVS V^canaidintheselectionofmoreaccuratemodels.Ingeneral,ourMDLcriterionalsoagreeswiththeAICandBIC.Asexpected,AICtendstopickslightlymorecomplexmodelsthanMDLandBIC.ThisisbecauseAIC 103

PAGE 104

doesnotincurapenaltydependentonthesamplesize.Thisslightoverestimationcanbeabenetwhenconsideringcomplexdensitiesbutitcanalsoovercompensate,seeFigure 5-5 .AICselectsamorecomplexmodelthannecessaryforthebimodaldensity[see(A)and(B)].Itstartstofavortrendsinthedata,degradingitsgeneralizationcapability.However,foracomplexdensityliketheasymmetricdoubleclaw,theAICselectionisabettermodel.BICtendstosomewhatunderestimatethemodels,selectinglesscomplexmodelsthannecessarytoaccuratelyrepresentthedensities[seeFigure 5-6 (A)and(B)].Inreal-worldapplications,theMSE,HELLandL1arenotusefulmodelselectioncriteriasincethetrueunderlyingdensitiesarenotaccessibleorunknown.Theyalsolackthetrade-obetweengoodness-of-tandcomplexity,onlyusingtheformerastheperformancemeasure.Hence,biasederrormeasuresliketheMSE,tendtopickmorecomplexmodels,Figure 5-6 (C)and(D).Sinceweknowthetruedensities,theglobalagreementbetweentheseerrormeasuresandtheinformation-theoreticmodelselectioncriteriashowcasesthepowerofthesemethodswithoutknowledgeoftruedensitiestheyareabletoselectmodelsthatbestdescribethedatawhilebalancingthecomplexityofthemodel.Forapplicationsto2Ddata,Figure 5-7 illustratesthej0levelschosenbyMDL-3,MDL-2,AICandBICforthreeoftheMPEG-7shapesusedinChapter4.TherecognitionexperimentsinChapter4wereconductedusingtheHaarfamilywithalldensitiesestimatedusingj0=1.ThisvaluewasinitiallychosenbasedonvisualanalysisbutnowweseethatitpreciselycorrespondstoanoptimallevelchosenbytheMDL-3andAICmodelselectioncriteria. 5.5DiscussionIntherealmofmodelselection,thereareaplethoraofcriteria(e.g.AIC,BIC,BMS,etc.)thataredesignedtopickthebestmodelfromasetofcompetingoneswhiletakingintoaccountabalanceofgoodness-of-tandcomplexity.Mostoftheaforementionedmethodsdenecomplexitysimplybythenumberofparameters:paucityofparameters 104

PAGE 105

ispreferred.MDL,ontheotherhand,takesintoaccountthenotionofmodelvolumesandtheiroccupationinthespaceofalldistributions.Itpenalizesbasedonthenumberofdistinguishabledistributionswithinthemodelthatareclosetothetruthrelativetothetotalvolumeofthemodel.Theseareintrinsicpropertiesofthemodels,remaininginvarianttoreparameterizations,andprovidingamorenaturalmeasureofcomplexitythanjustthenumberofparameters.GeometriccomplexityasrealizedbytheMDLcriteriontakesintoaccountboththenumberofparametersandfunctionalformofthemodels.TheuseofMDLforestimatingtheMRAstructureforthewaveletexpansionofthesquare-rootofthedensity,ledtoseveralinterestingresults.WeillustratedthattheMDLcriterionisgenerallyinvarianttoMRA.Thisallowedthesearchforthebestdensitiesforagivensampletobeconductedbydensityestimationusingjustscalingfunctionsoverdierentlevels.Experimentalresultsvalidatedtheselectionofsuitablemodelsacrossavarietyofdensitiesandwaveletfamilies.Thegeometryassociatedwiththesquare-root,waveletframeworkallowedustocomputetheRiemannianvolumetermoftheMDLcriterioninclosed-form.Thewaveletdensitieswererestrictedtoliveonaunithypersphere.Forachoiceofdecompositionlevelj0,wegetacountofcoecientsinthewaveletexpansionofthedensitydeterminedbythetranslationsneededtospanthesampledata.Dierentchoicesofj0yieldingdierentmodelsareidentiedbythenumberofparameters(coecients)whichcorrespondtothenumberofdimensionsofthehypersphere.Closed-formcomputationofVMisararityforsucharichandexibledensityestimationmodel.Validatingtheuseofthevolumetermalsoillustratedcaseswhereitwasusefulinpickingamoresuitedmodel. 105

PAGE 106

Table5-1.Optimalj0undervariousmodelselectioncriteriausingtheDaubechiesfamily. DB2(Optimalstartlevelj0) DB4(Optimalstartlevelj0) Density MDL-3 MDL-2 AIC BIC MSE HELL L1 MDL-3 MDL-2 AIC BIC MSE HELL L1 Gaussian 0 0 1 0 1 0 1 0 0 0 0 0 0 0 SkewedUni. 1 1 1 1 2 1 2 0 0 1 0 1 1 1 Str.SkewedUni. 2 2 2 2 4 3 2 3 2 3 2 3 3 3 KurtoticUni. 3 2 3 2 4 3 3 2 2 3 2 3 2 3 Outlier 3 3 3 3 4 3 4 3 3 3 3 4 3 3 Bimodal 1 1 1 1 1 2 1 1 1 1 0 1 1 1 Sep.Bimodal 1 1 2 1 2 2 2 1 1 1 1 1 1 1 SkewedBimodal 1 1 1 1 2 2 2 1 1 1 1 2 1 2 Trimodal 1 1 1 1 2 2 2 1 1 1 0 1 1 1 Claw 3 2 3 2 3 2 3 3 2 3 2 3 3 3 Dbl.Claw 1 1 1 1 1 2 1 1 1 1 0 1 1 1 Asym.Claw 2 2 3 2 3 3 3 2 2 3 1 3 2 3 Asym.Dbl.Claw 1 1 1 0 2 2 1 1 1 2 1 2 2 1 106

PAGE 107

Table5-2.Optimalj0undervariousmodelselectioncriteriausingtheSymletfamily. SYM4(Optimalstartlevelj0) SYM5(Optimalstartlevelj0) Density MDL-3 MDL-2 AIC BIC MSE HELL L1 MDL-3 MDL-2 AIC BIC MSE HELL L1 Gaussian 0 0 0 0 0 0 0 0 0 0 0 1 0 1 SkewedUni. 0 0 1 0 1 0 1 1 0 1 0 1 0 1 Str.SkewedUni. 2 2 3 2 3 3 3 2 2 2 2 4 2 2 KurtoticUni. 2 2 2 2 3 2 3 3 2 3 2 3 3 3 Outlier 2 2 3 2 4 3 4 3 3 3 3 4 3 3 Bimodal 0 0 2 0 1 0 1 1 0 1 0 1 1 1 Sep.Bimodal 1 1 1 1 1 4 1 1 1 1 1 1 1 1 SkewedBimodal 1 1 1 1 1 1 1 1 1 1 0 1 1 1 Trimodal 1 1 1 1 1 2 1 1 0 1 0 1 1 1 Claw 2 2 2 2 3 2 3 3 2 3 2 3 3 3 Dbl.Claw 0 0 2 0 1 0 1 1 0 1 0 1 1 1 Asym.Claw 2 1 2 1 3 3 3 2 2 3 2 3 3 3 Asym.Dbl.Claw 1 0 2 0 2 0 2 0 0 1 0 2 1 2 107

PAGE 108

Table5-3.Optimalj0undervariousmodelselectioncriteriausingtheCoietfamily. COIF1(Optimalstartlevelj0) COIF2(Optimalstartlevelj0) Density MDL-3 MDL-2 AIC BIC MSE HELL L1 MDL-3 MDL-2 AIC BIC MSE HELL L1 Gaussian 0 0 1 0 1 1 1 -1 -1 0 -1 0 0 0 SkewedUni. 1 1 1 1 2 1 1 0 0 1 0 1 0 1 Str.SkewedUni. 2 2 3 2 4 3 3 2 2 2 2 4 2 3 KurtoticUni. 2 2 2 1 4 2 2 2 2 2 2 2 2 2 Outlier 2 2 3 2 5 3 4 2 2 2 2 4 2 4 Bimodal 1 0 1 0 2 1 1 0 0 0 0 1 0 1 Sep.Bimodal 1 1 2 1 2 1 2 1 1 1 1 1 1 1 SkewedBimodal 1 1 1 1 2 2 2 1 1 1 1 1 1 1 Trimodal 1 1 1 1 1 1 1 1 1 1 1 1 2 1 Claw 2 2 2 2 2 2 2 2 2 2 2 2 2 2 Dbl.Claw 1 0 1 0 2 1 1 0 0 0 0 1 0 1 Asym.Claw 2 1 2 1 3 2 3 2 1 2 1 3 2 3 Asym.Dbl.Claw 1 1 1 0 2 1 2 0 0 2 0 2 2 2 108

PAGE 109

Figure5-1.Surfaceareaofunithypersphere.Maximumsurfaceareaisatdimensionseven. Figure5-2.Riemannianvolumecomparisons,ln(VS)versusln(V^).Thedashedlinerepresentstheln(V^)andthesolidlineln(VS).MisspeciedmodelsoccurwhenV^>VS.ForsucientlyhighnumberofsamplesweseethatV^
PAGE 110

j0nj1NoMRA-10123456 -1 0 1 2 3 4 5 6 Figure5-3.Multiresolutionanalysis(MRA)fordensityestimation.Illustratesequivalenceofscaling-function-onlydensityestimateswithMRAversionsusingasymmetricclawdensity.Alldensitiesinthe'NoMRA'columnwereestimatedusingonlyscalingfunctionsatthej0levelindicatedonthefarleftcolumn.OthercolumnsuseMRA,startingatj0uptoj1levelonthetoprow. 110

PAGE 111

A B C D Figure5-4.ModelselectionusingMDL-3versusMDL-3.MDL-3isabletoselectmorecomplexmodelsthanMDL-2.A)j0=2byMDL-2.B)j0=3byMDL-3.C)j0=1byMDL-2.D)j0=2byMDL-3.WaveletfamilyDB4usedfor(A)and(B).WaveletfamilyCOIF2usedfor(C)and(D). 111

PAGE 112

A B C D Figure5-5.ModelselectionusingMDL-3versusAIC.AICgenerallyselectsmorecomplexmodelsthanMDL-3.Thiscanbehelpfulforcomplexdensitieslikein(C)and(D),butcanalsooverestimatesmoothoneslikein(A)and(B).A)j0=0byMDL-3.B)j0=2byAIC.C)j0=1byMDL-3.D)j0=2byAIC.WaveletfamilySYM4usedfor(A)and(B).WaveletfamilyDB4usedfor(C)and(D). 112

PAGE 113

A B C D Figure5-6.ModelselectionusingMDL-3versusBICandMSE.BICtendstofavorlesscomplexmodelsthanMDL-3,see(A)and(B).TheMSEgenerallyovertssinceitisonlyagoodness-of-tmeasure.A)j0=1byMDL-3.B)j0=0byBIC.C)j0=2byMDL-3.D)j0=4byMSE.WaveletfamilyDB4usedfor(A)and(B).WaveletfamilyCOIF2usedfor(C)and(D). 113

PAGE 114

Shapej0=3j0=2j0=1j0=0j0=1j0=2 Figure5-7.Modelselectionfor2DdensitiesusingMDL-3,MDL-2,AICandBIC.Rows1to3representtheMPEG-7shapesCattle-05(8,671points),Device6-01(8,947points),Device6-08(11,301points)respectively.Forallthreeshapes,MDL-3andAICselectedj0=1,whileMDL-2andBICselectedj0=0. 114

PAGE 115

CHAPTER6CONCLUSIONS 6.1ContributionsThisresearchhasillustratedtheutilityandeectivenessofusinginformationgeometryfortheapplicationofshapeanalysis.Thepresenttrendwithintheshapeanalysiscommunityistoselectshaperepresentationsthatrequireaprescribedglobaltopology,e.g.closed-curves.Themethodsdevelopedinthisworkareinsharpcontrasttotheseasourrepresentationsaresimplyunstructuredpointsets.Thishasseveraladvantagesincludingtheeliminationofpreprocessingstagesoftenneededtoextractorientedcurves.Giventheseshapepointsets,wedirectlyestimatedaprobabilitydensity.(For2Dshapes,thedensitieshaveadirectvisualcorrelationwiththeshape.)Oncetheshapesarerepresentedasdensityfunctions,thetenetsofinformationgeometryenabledustoperformintrinsicanalysisonthespaceofdistributionsusinggeodesicstoestablishsimilaritymeasuresandanalyzethenon-rigidtransformationbetweenapairofshapes.InChapter2,wedemonstratedthisframeworkusingGaussianmixturemodels.Itwasshownthatonecanrecovertheintermediatenon-rigidtransformationsbetweenshapesbyndinggeodesicsonthespaceofmixtures.Inalmostallpastandpresentwork,non-rigidtransformationshavebeenrecoveredusingsplinemodels.Wealsoderivedanewinformationmetrictensor,usingthenotionofageneralizedentropy.Forthersttime,thisresultedinaclosed-formmetrictensorforGaussianmixtures.Thoughtheseadvancesweresignicant,theirpracticalitywassomewhatdiminishedasthegeodesicswerenotinclosedformandoptimizingforthemonhighdimensionalmanifoldswasacomputationalburden.Toovercomethis,wechosetochangetheunderlyingrepresentationofthedensities.WemovedfromGaussianmixturestoanorthogonalseriesexpansionusingwavelets.AsdiscussedinChapter3(andChapters4and5),thisrepresentationplacesalldensitiesonaunithyperspherewithdimensionalityequaltothecardinalityofthewaveletcoecients 115

PAGE 116

inthedensityexpansion.Inordertoassureabonadedensity(mainlynon-negativityandunitintegrability),weexpandedp pinawaveletbasisandobtainedthedensityasitssquare.Chapter3presentedaconstrainedmaximumlikelihoodframeworktoecientlyestimatethecoecientsofthewaveletexpansion.Thismethodtookadvantageoftheconvexityofthelikelihoodobjectiveandknowledgeoftheobjective'sasymptoticHessianattheoptimalpoint.Weexaminedtheapproximationcapabilitiesofthewaveletexpansionandempiricallyestablisheditssuperioritytotheubiquitouskerneldensityestimator.Applicationstoshapealignmentandmutualinformationbasedimageregistrationwerealsovalidated.Chapter4usedthisnewdensityestimationframeworktodevelopafastshapesimilaritymeasurethatconcurrentlyadjustedfornon-rigiddierencesbetweenshapes.Werstrepresentedpoint-setshapesbytheirestimatedwaveletdensity.Thisimmediatelyplacedalldensities(andineectallshapes)ontheunithypersphere.(Thegeodesicontheunithypersphereisknowninclosedformasitissimplythearclengthbetweentwosurfacepoints.)Weshowedthatiftherewerenon-rigiddierencesbetweenapairofshapes,wecouldmodeltheseasinnitesimaltranslationsandconsequentlyadjustforthembyslidingthewaveletcoecientspriortocomputingthegeodesicdistance.Thisslidingwasaccomplishedbyusinglinearassignmenttosolveforapenalizedpermutationorderingbetweenthewaveletcoecientsofeachdensity.Sincethewaveletcoecientscorrespondtoaspatiallocation,permutingthecoecientshastheeectofspatiallyslidingthemtoanewlocation.TheslidingwaveletalgorithmwasvalidatedontheMPEG-7datasetwithresultsbeingcompetitivetothestateoftheart.Finally,Chapter5addressedacrucialmodelselectionissueattheheartofourdensityestimationframework:Howdoesoneselectthewaveletdecompositionlevelswhenestimatingthedensity?Weadoptedtheminimumdescriptionlength(MDL)asourmodelselectioncriterionandanalyzeditsgeometricderivationonthespaceofprobabilitymodels.Usuallythecomputationofthecompletecriterionrequiresnumericalintegration 116

PAGE 117

tondtheRiemannianvolumeoftheprobabilisticmanifold.Inourframework,thevolumewasknowninclosedformsinceourgeometryisthatofaunithypersphere.Thisrarelyoccursforpowerfulgeneralestimationproceduressuchasours.ItwasalsoshownthatMDLisinvarianttowaveletmultiresolutionanalysis(MRA),allowingustosearchforthebestdecompositionlevelusingonlyscalingfunctions. 6.2FutureWorkThereareseverallinesofinquiryandinvestigationthatcanbepursuedfromthispresentwork.ForthemixtureframeworkofChapter2,animmediatenextstepwouldbetomovebeyondlandmarksanduseGaussianmixturemodelsonunlabeledpoint-setstherebyestimatingthefreeparameter2directlyfromthedata.Insteadofjustanisotropicvariance,onemaywanttoincorporatethefullcovariancematrix,enablingthemixturedensityrepresentationtohavericherdescriptivepowerforpoint-setshapes.Inaddition,theframeworkcanbeextendedto3Dshapematchinganddieomorphicwarpingoftheextrinsicspace.Amoresubstantivestepwouldbetoeliminatetheneedtohaveequalparametersintheshapemixturedensitiesandmatchtwopoint-setshapeswithdieringcardinalities.Thiscanbeaccomplishedinaprincipalwaybydevelopingametricthatreliesonthedensitiesdirectly(ratherthantheirparameters).Forthewaveletdensityestimationprocedure,improvementscouldberealizedthroughtheincorporationofstate-of-the-artconvexoptimizationtechniquestosolvethepresentobjective.Fortheapplicationofshapematchingthroughslidingwavelets,furtherstudiesareneededtounderstandtheimpactofusingdierentwaveletfamilies.Weanticipatetheseotherfamilieswillprovideadditionalattributesforeachshapewhichwillfurtherincreaseshapediscriminabilityandsubsequentlyimproverecognitionrates.Onecanalsoinvestigatenewpenaltytermsforthelinearassignmentobjectivefunctionandbettermechanismsforchoosing.Amoreesotericandchallengingtaskwouldbetodevelopgrammarsotherthansliding. 117

PAGE 118

TheuseoftheMDLmodelselectioncriteriaforselectingthebestwaveletlevelcanserveaslaunchpadforothercriteriabettersuitedtothemultiresolutioncapabilitiesofthewavelets.Inparticular,itwouldbeinterestingtoincorporatethenotionofsparsityintothedensityestimationframeworkandconsequentlymodelselection.ThiswoulddirectlyaddresstheinvarianceofMDLtoMRA,enablingamoreecientdensityrepresentationwithonlyfewcoecientscontainingpertinentknowledgeaboutthedensity. 118

PAGE 119

References [1] W.BrownandC.Swonger,Aprospectusforautomatictargetrecognition,IEEETransactionsonAerospaceandElectronicSystems,vol.25,no.3,pp.401,May1989. [2] A.Smeulders,M.Worring,S.Santini,A.Gupta,andR.Jain,Content-basedimageretrievalattheendoftheearlyyears,TransactionsonPatternAnalysisandMachineIntelligence,vol.22,no.12,pp.1349,2000. [3] J.W.H.TangelderandR.C.Veltkamp,Asurveyofcontentbased3Dshaperetrievalmethods,inShapeModelingApplications,2004,pp.145. [4] A.M.MartinezandJ.R.Serra,Anewapproachtoobject-relatedimageretrieval,JournalofVisualLanguagesandComputing,vol.11,no.3,pp.345,2000. [5] R.Luck,R.Tjon-Fo-Sang,L.Mango,J.Recht,E.Lin,andJ.Knapp,PAPNET:Anautomatedcytologyscreenerusingimageprocessingandneuralnetworks,inProceedingsoftheSPIETheInternationalSocietyforOpticalEngineering,1992,pp.161. [6] Y.C.Wu,K.Doi,andM.L.Giger,Detectionoflungnodulesindigitalchestradiographsusingarticialneuralnetworks:Apilotstudy,JournalofDigitalImaging,vol.8,pp.88,1995. [7] M.V.BolandandR.F.Murphy,Aneuralnetworkclassiercapableofrecognizingthepatternsofallmajorsubcellularstructuresinuorescencemicroscopeimagesofhelacells,Bioinformatics,vol.1,pp.213,2001. [8] A.M.Waxman,J.J.LeMoigne,L.S.Davis,andB.Srinivasan,Visualnavigationofroadways,inProc.IEEEInt'lConf.RoboticsandAutomation,1985,pp.862. [9] C.Thorpe,M.H.Herbert,T.Kanade,andS.A.Shafer,VisionandnavigationfortheCarnegie-MellonNavlab,IEEETransactionsonPatternAnalysisandMachineIntelligence,vol.10,no.3,pp.362,1988. [10] V.Graefe,VisionforautonomousmobilerobotsinProc.IEEEWorkshopAd-vancedMotionControl,1992,pp.57. [11] D.KimandR.Nevatia,Recognitionandlocalizationofgenericobjectsforindoornavigationusingfunctionality,ImageandVisionComputing,vol.16,no.11,pp.729,1998. 119

PAGE 120

[12] G.N.DeSouzaandA.C.Kak,Visionformobilerobotnavigation:Asurvey,IEEETransactionsonPatternAnalysisandMachineIntelligence,vol.24,no.2,pp.237,2002. [13] J.Ponce,R.Bajcsy,D.Metaxas,T.O.Binford,D.A.Forsyth,M.Hebert,K.Ikeuchi,A.C.Kak,L.Shapiro,S.Sclaro,A.Pentland,andG.C.Stockman,Objectrepresentationforobjectrecognition,inIEEEComputerSocietyConferenceonComputerVisionandPatternRecognition,1994,pp.147. [14] H.ChuiandA.Rangarajan,Anewpointmatchingalgorithmfornon-rigidregistration,ComputerVisionandImageUnderstanding,vol.89,pp.114,2003. [15] S.Belongie,J.Malik,andJ.Puzicha,Shapematchingandobjectrecognitionusingshapecontexts,IEEETransactionsonPatternAnalysisMachineIntelligence,vol.24,no.4,pp.509,2002. [16] K.Siddiqi,A.Shokoufandeh,S.J.Dickinson,andS.W.Zucker,Shockgraphsandshapematching,inICCV,1998,pp.222.[Online].Available: citeseer.ist.psu.edu/siddiqi98shock.html [17] H.Tagare,D.O'Shea,andA.Rangarajan,Ageometriccriterionforshapebasednon-rigidcorrespondence,inFifthIntl.Conf.ComputerVision(ICCV),1995,pp.434. [18] P.ThompsonandA.W.Toga,Asurface-basedtechniqueforwarpingthree-dimensionalimagesofthebrain,IEEETransactionsMedicalImaging,vol.5,no.4,pp.402,August1996. [19] G.McNeillandS.Vijayakumar,HierarchicalProcrustesmatchingforshaperetrieval,inIEEEConferenceonComputerVisionandPatternRecognition(CVPR),2006,pp.885. [20] P.F.FelzenszwalbandJ.D.Schwartz,Hierarchicalmatchingofdeformableshapes,inIEEEConferenceonComputerVisionandPatternRecognition(CVPR),2007,pp.1. [21] I.J.Myung,V.Balasubramanian,andM.A.Pitt,Countingprobabilitydistributions:Dierentialgeometryandmodelselection,ProceedingsoftheNationalAcademyofSciences,vol.97,pp.11170175,2000. 120

PAGE 121

[22] N.N.Cencov,Statisticaldecisionrulesandoptimalinference.AmericanMathematicalSociety,1982. [23] S.-I.AmariandH.Nagaoka,MethodsofInformationGeometry.AmericanMathematicalSociety,2001. [24] C.R.Rao,Informationandaccuracyattainableinestimationofstatisticalparameters,BulletinoftheCalcuttaMathematicalSociety,vol.37,pp.81,1945. [25] W.M.Boothby,AnIntroductiontoDierentiableManifoldsandRiemannianGeometry.SanDiego:AcademicPress,2002. [26] J.BurbeaandR.Rao,Entropydierentialmetric,distanceanddivergencemeasuresinprobabilityspaces:Auniedapproach,JournalofMultivariateAnalysis,vol.12,pp.575,1982. [27] M.L.Menendez,D.Morales,L.Pardo,andM.Salicru,(h,)-Entropydierentialmetric,ApplicationsofMathematics,vol.42,pp.81,1997. [28] A.PeterandA.Rangarajan,Anewclosed-forminformationmetricforshapeanalysis,inMedicalImageComputingandComputerAssistedIntervention(MICCAI),2006,pp.249. [29] Y.Liu,S.Luo,A.Li,andH.Yu,Informationgeometryonpruningofneuralnetwork,inInternationalConferenceonMachineLearningandCybernetics,vol.6,2004,pp.3479. [30] S.Amari,Naturalgradientworksecientlyinlearning,NeuralComputation,vol.10,no.2,pp.251,1998. [31] G.Lebanon,Metriclearningfortextdocuments,IEEETransactionsonPatternAnalysisandMachineIntelligence,vol.28,no.4,pp.497,2006. [32] S.J.Maybank,DetectionofimagestructuresusingtheFisherinformationandtheRaometric,TransactionsonPatternAnalysisandMachineIntelligence,vol.26,no.12,pp.1579,2004. [33] ,TheFisher-Raometricforprojectivetransformationsoftheline,InternationalJournalofComputerVision,vol.63,no.3,pp.191,2005. [34] W.Mio,D.Badlyans,andX.Liu,AcomputationalapproachtoFisherinformationgeometrywithapplicationstoimageanalysis,inEnergyMinimizationMethodsin 121

PAGE 122

ComputerVision,2005. [35] C.Lenglet,M.Rousson,R.Deriche,andO.Faugeras,Statisticsonthemanifoldofmultivariatenormaldistributions:TheoryandapplicationtodiusiontensorMRIprocessing,JournalofMathematicalImagingandVision,vol.25,no.3,pp.423,2006. [36] A.Srivastava,I.Jermyn,andS.Joshi,Riemanniananalysisofprobabilitydensityfunctionswithapplicationsinvision,inComputerVisionandPatternRecognition(CVPR),2007,pp.1. [37] F.L.Bookstein,Morphometrictoolsforlandmarkdata:Geometryandbiology.CambridgeUniversityPress,1991. [38] A.PeterandA.Rangarajan,ShapematchingusingtheFisher-RaoRiemannianmetric:Unifyingshaperepresentationanddeformation,InternationalSymposiumonBiomedicalImaging(ISBI),pp.1164,2006. [39] C.Small,Thestatisticaltheoryofshape.NewYork,NY:Springer,1996. [40] F.L.Bookstein,Principalwarps:Thin-platesplinesandthedecompositionofdeformations,IEEETransactionsonPatternAnalysisMachineIntelligence,vol.11,no.6,pp.567,June1989. [41] K.Rohr,H.Stiehl,R.Sprengel,T.Buzug,J.Weese,andM.Kuhn,Landmark-basedelasticregistrationusingapproximatingthin-platesplines,IEEETransactionsonMedicalImaging,vol.20,no.6,pp.526,June2001. [42] R.Davies,C.Twining,T.Cootes,andC.Taylor,Aninformationtheoreticapproachtostatisticalshapemodelling,inEuropeanConferenceonComputerVision(ECCV),ser.LectureNotesinComputerScience,LNCS2351.Springer,2002,pp.III:3. [43] V.CamionandL.Younes,Geodesicinterpolatingsplines,inEnergyMinimizationMethodsforComputerVisionandPatternRecognition.NewYork:Springer,2001,pp.513. [44] S.JoshiandM.Miller,Landmarkmatchingvialargedeformationdieomorphisms,IEEETransactionsImageProcessing,vol.9,pp.1357,2000. [45] H.Guo,A.Rangarajan,andS.Joshi,Dieomorphicpointmatching,inTheHandbookofMathematicalModelsinComputerVision,N.Paragios,Y.Chen,andO.Faugeras,Eds.SpringerVerlag,2005,pp.205. 122

PAGE 123

[46] A.Srivastava,S.Joshi,W.Mio,andX.Liu,Statisticalshapeanalysis:Clustering,learningandtesting,IEEETransactionsonPatternAnalysisandMachineIntelligence,vol.27,no.4,pp.590,2003. [47] M.E.HavrdaandF.Charvt,Quanticationmethodofclassicationprocesses:Conceptofstructural-entropy,Kybernetica,vol.3,pp.30,1967. [48] Y.W.K.WoodsandM.McClain,Information-theoreticmatchingoftwopointsets,IEEETransactionsImageProcessing,vol.11,no.8,pp.868,August2002. [49] F.Wang,B.C.Vemuri,A.Rangarajan,I.M.Schmalfuss,andS.J.Eisenschenk,Simultaneousnonrigidregistrationofmultiplepointsetsandatlasconstruction,inEuropeanConferenceonComputerVision(ECCV),vol.LNCS3953,2006,pp.551. [50] B.JianandB.C.Vemuri,Arobustalgorithmforpointsetregistrationusingmixtureofgaussians,inIEEEInternationalConferenceonComputerVision(ICCV),2005,pp.1246. [51] J.Lin,DivergencemeasuresbasedontheShannonentropy,IEEETransactionsonInformationTheory,vol.37,no.1,pp.145,1991. [52] N.Paragios,M.Rousson,andV.Ramesh,Non-rigidregistrationusingdistancefunctions,ComputerVisionandImageUnderstanding,vol.89,pp.142,2003. [53] J.Glaunes,A.Trouve,andL.Younes,Dieomorphicmatchingofdistributions:anewapproachforunlabelledpoint-setsandsub-manifoldsmatching,inComputerVisionandPatternRecognition(CVPR),2004,pp.712. [54] G.Wahba,Splinemodelsforobservationaldata.Philadelphia,PA:SIAM,1990. [55] T.CootesandC.Taylor,Amixturemodelforrepresentingshapevariation,inProceedingsofBritishMachineVisionConference,1997,pp.110. [56] G.J.McLachlanandK.E.Basford,MixtureModels:InferenceandApplicationstoClustering.NewYork:MarcelDekker,1988. [57] M.A.T.FigueiredoandA.K.Jain,Unsupervisedlearningofnitemixturemodels,IEEETransactionsonPatternAnalysisandMachineIntelligence,vol.24,no.3,pp.381,2002. 123

PAGE 124

[58] R.CourantandD.Hilbert,MethodsofMathematicalPhysics.Wiley-Interscience,1989(newed.). [59] W.MioandX.Liu,LandmarkrepresentationofshapesandFisher-Raogeometry,inIEEEInternationalConferenceonImageProcessing,2006,pp.2113. [60] B.K.P.Horn,RobotVision.MITPress,1986. [61] S.Costa,S.Santos,andJ.Strapasson,Fisherinformationmatrixandhyperbolicgeometry,IEEEInformationTheoryWorkshop,pp.28,2005. [62] D.G.Kendall,Shape-manifolds,Procrusteanmetricsandcomplexprojectivespaces,BulletinoftheLondonMathematicalSociety,vol.16,pp.81,1984. [63] D.P.Huttenlocher,G.A.Klanderman,andW.J.Rucklidge,ComparingimagesusingtheHausdordistance,IEEETransactionsonPatternAnalysisMachineIntelligence,vol.15,no.9,pp.850,Sep.1993. [64] M.Fornefett,K.Rohr,andH.S.Stiehl,Radialbasisfunctionswithcompactsupportforelasticregistrationofmedicalimages,ImageandVisionComputing,vol.19,no.1,pp.87,January2001. [65] A.Banerjee,I.Dhillon,J.Ghosh,andS.Sra,ClusteringontheunithypersphereusingvonMises-Fisherdistributions,TheJournalofMachineLearningResearch,vol.6,pp.1345,2005. [66] D.W.Scott,MultivariateDensityEstimation:Theory,Practice,andVisualization.Wiley-Interscience,2001. [67] B.W.Silverman,DensityEstimationforStatisticsandDataAnalysis.ChapmanandHall/CRC,1986. [68] P.Doukhan,FormesdeTeplitzassociesuneanalysemultiechlle,ComptesRendusdel'AcadmiedesSciences,vol.306,pp.663,1988. [69] M.Vannucci,Nonparametricdensityestimationusingwavelets,ISDS,DukeUniversity,Tech.Rep.DP95-26,September1995,availableathttp://www.isds.duke.edu. [70] A.PinheiroandB.Vidakovic,Estimatingthesquarerootofadensityviacompactlysupportedwavelets,ComputationalStatisticsandDataAnalysis,vol.25,no.4,pp.399,1997. 124

PAGE 125

[71] S.PenevandL.Dechevsky,Onnon-negativewavelet-baseddensityestimators,JournalofNonparametricStatistics,vol.7,pp.365,1997. [72] P.ViolaandW.M.WellsIII,Alignmentbymaximizationofmutualinformation,InternationalJournalofComputerVision,vol.24,no.2,pp.137,1997. [73] R.Beran,MinimumHellingerdistanceestimatesforparametricmodels,TheAnnalsofStatistics,vol.5,no.3,pp.445,1977. [74] I.Daubechies,TenLecturesonWavelets,ser.CBMS-NSFReg.Conf.SeriesinAppliedMath.SIAM,1992. [75] G.StrangandT.Nguyen,WaveletsandFilterBanks.Wellesley-CambridgePress,1997. [76] B.Vidakovic,StatisticalModelingbyWavelets.NewYork:JohnWileyandSons,1999. [77] D.Donoho,I.Johnstone,G.Kerkyacharian,andD.Picard,Densityestimationbywaveletthresholding,TheAnnalsofStatistics,vol.24(2),pp.508,1996. [78] W.Hardle,G.Kerkyacharian,D.Pickard,andA.Tsybakov,Wavelets,Approximation,andStatisticalApplications,ser.LectureNotesinStatistics129.NewYork:Springer-Verlag,1998. [79] D.C.Montgomery,DesignandAnalysisofExperiments.Wiley,2004. [80] G.G.Walter,WaveletsandOtherOrthogonalSystemswithApplications.BocaRaton:CRCPressInc.,1994. [81] G.Lebanon,Riemanniangeometryandstatisticalmachinelearning,Ph.D.dissertation,CarnegieMellonUniversity,Pittsburg,PA,2005. [82] D.Luenberger,LinearandNonlinearProgramming.Reading,MA:AddisonWesley,1984. [83] H.BurkhardtandN.Diehl,Simultaneousestimationofrotationandtranslationinimagesequences,ProceedingsoftheEuropeanSignalProcessingConference,pp.821,1986. 125

PAGE 126

[84] B.Vemuri,S.Huang,S.Sahni,C.M.Leonard,C.Mohr,R.Gilmore,andJ.Fitzsimmons,Anecientmotionestimatorwithapplicationtomedicalimageregistration,MedicalImageAnalysis,vol.2,no.1,pp.79,March1998. [85] S.J.MarronandM.P.Wand,Exactmeanintegratedsquarederror,TheAnnalsofStatistics,vol.20,no.2,pp.712,1992. [86] M.P.WandandM.C.Jones,Comparisonofsmoothingparameterizationsinbivariatekerneldensityestimation,JournaloftheAmericanStatisticalAssociation,vol.88,no.422,pp.520,1993. [87] G.G.WalterandJ.K.Ghorai,Advantagesanddisadvantagesofdensityestimationwithwavelets,inProceedingsofthe24rdSymposiumontheInterface,1992,pp.234. [88] A.Ihler,Kerneldensityestimationtoolboxformatlab,2003.[Online].Available: http://www.ics.uci.edu/~ihler/code/kde.html [89] A.PeterandA.Rangarajan,ShapematchingusingtheFisher-RaoRiemannianmetric:Unifyingshaperepresentationanddeformation,IEEEInternationalSymposiumonBiomedicalImaging(ISBI),pp.1164,2006. [90] J.Principe,D.Xu,andJ.Fisher,Informationtheoreticlearning,inUnsupervisedAdaptiveFiltering,S.Haykin,Ed.Wiley,2000,pp.265. [91] D.L.Collins,A.P.Zijdenbos,V.Kollokian,J.G.Sled,N.J.Kabani,C.J.Holmes,andA.C.Evans,Designandconstructionofarealisticdigitalbrainphantom,IEEETransactionsonMedicalImaging,vol.17,no.3,pp.463,June1998. [92] J.ZhangandA.Rangarajan,Aneimageregistrationusinganewinformationmetric,inIEEEComputerVisionandPatternRecognition(CVPR),vol.1,June2004,pp.848. [93] A.B.A.RajwadeandA.Rangarajan,ProbabilityDensityEstimationusingIsocontoursandIsosurfaces:ApplicationstoInformationTheoreticImageRegistration,IEEETransactionsonPatternAnalysisandMachineIntelligence,2008. [94] P.Shilane,P.Min,M.Kazhdan,andT.Funkhouser,ThePrincetonshapebenchmark,inProceedingsoftheShapeModelingInternational2004SMI'04,vol.00.IEEEComputerSociety,2004,pp.167. 126

PAGE 127

[95] Y.Rubner,C.Tomasi,andL.J.Guibas,Theearthmover'sdistanceasametricforimageretrieval,InternationalJournalofComputerVision,vol.40,no.2,pp.99,2000. [96] F.L.Hitchcock,Thedistributionofaproductfromseveralsourcestonumerouslocalities.J.Math.Phys.,vol.20,pp.224,1941. [97] L.J.Latecki,R.Lakmper,andU.Eckhardt,Shapedescriptorsfornon-rigidshapeswithasingleclosedcontour,inIEEEConferenceonComputerVisionandPatternRecognition(CVPR),2000,pp.424. [98] R.Osada,T.Funkhouser,B.Chazelle,andD.Dobkin,Shapedistributions,ACMTransactionsonGraphics,no.4,pp.807,2004. [99] G.C.-H.ChuangandC.-C.J.Kuo,Waveletdescriptorofplanarcurves:theoryandapplications,IEEETransactionsonImageProcessing,vol.5,no.1,pp.5670,January1996. [100] F.Klein,Acomparativereviewofrecentresearchesingeometry,Bull.NewYorkMathSoc.,vol.2,pp.215,1892,trans.M.W.Haskell. [101] I.L.DrydenandK.V.Mardia,StatisticalShapeAnalysis.Wiley,1998. [102] J.Ho,M.Yang,A.Rangarajan,andB.Vemuri,Anewaneregistrationalgorithmformatching2Dpointsets,inProceedingsoftheEighthIEEEWorkshoponApplicationsofComputerVision,2007,p.25. [103] R.JonkerandA.Volgenant,Ashortestaugmentingpathalgorithmfordenseandsparselinearassignmentproblems,Computing,vol.38,pp.325,1987. [104] J.Rissanen,Modelingbyshortestdatadescription,Automatica,vol.14,pp.465,1978. [105] ,Fisherinformationandstochasticcomplexity,IEEETransactionsonInformationTheory,vol.42,pp.40,1996. [106] V.Balasubramanian,Statisticalinference,Occam'srazor,andstatisticalmechanicsonthespaceofprobabilitydistributions,NeuralComputation,vol.9,no.2,pp.349,February1997. [107] I.M.M.PittandS.Zhang,Towardamethodofselectingamongcomputationalmodlesofcognition,PsychologicalReview,vol.109,no.3,pp.472,2002. 127

PAGE 128

[108] H.Akaike,Informationtheoryandanextensionofthemaximumlikelihoodprinciple,inProc.2ndInternationalSymposiumonInformationTheory,B.N.PetrovandF.Csaki,Eds.,1973,pp.267. [109] G.Schwarz,Estimatingthedimensionofamodel,TheAnnalsofStatistics,vol.6,no.2,pp.461,1978. [110] R.E.KassandA.E.Raftery,Bayesfactors,JournaloftheAmericanStatisticalAssociation,vol.90,pp.773,1995. [111] P.Grnwald,Atutorialintroductiontotheminimumdescriptionlengthprinciple,inAdvancesinMinimumDescriptionLength:TheoryandApplications,P.Grnwald,I.Myung,andM.Pitt,Eds.MITPress,2005. [112] H.Jereys,TheoryofProbability,3rded.NewYork:OxfordUniversityPress,1961. [113] C.M.BenderandS.A.Orszag,AdvancedMathematicalMethodsforScientistsandEngineersAsymptoticMethodsandPerturbationTheory.Springer,1999. [114] J.BernardoandA.Smith,BayesianTheory.NewYork,NY:JohnWileyandSons,1994. [115] D.J.Navarro,Anoteontheapplieduseofmdlapproximations,NeuralComputation,vol.16,pp.1763,September2004. 128

PAGE 129

BIOGRAPHICALSKETCH AdrianM.Peter(December30,1975)wasborninTrincomalee,SriLanka.HecametotheUnitedStatesofAmericain1984,becomingacitizenin1994.HereceivedtheB.S.degree(highhonors)incomputerengineeringandtheM.S.degreeinelectricalengineeringfromtheUniversityofFloridain1999and2003,respectively.AfterhisB.S.degree,hewaswithIntelCorporationwherehelastheldthepositionofatechnologyinitiativesprogrammanagerresponsibleforshortrangewirelessproducts.AfterhisM.S.degree,hewaswithHarrisCorporationdevelopingimageanalysisalgorithmsforavarietyofremotesensingplatformsandwasaco-inventoroneightpendingpatents.Hisresearchinterestsareinmachinelearningandcomputervisionwithafocusonapplyinginformationgeometrytoshapeanalysis.UponcompletinghisPh.D.,heisreturningtoHarrisCorporation.AdrianismarriedtoJancyandtogethertheyhaveoneson,Elijah. 129