<%BANNER%>

Relighting, Pose Change and Recognition of Faces

Permanent Link: http://ufdc.ufl.edu/UFE0041052/00001

Material Information

Title: Relighting, Pose Change and Recognition of Faces
Physical Description: 1 online resource (114 p.)
Language: english
Creator: Kumar, Ritwik
Publisher: University of Florida
Place of Publication: Gainesville, Fla.
Publication Date: 2009

Subjects

Subjects / Keywords: abrdf, face, pose, recognition, reflectance, relighting
Computer and Information Science and Engineering -- Dissertations, Academic -- UF
Genre: Computer Engineering thesis, Ph.D.
bibliography   ( marcgt )
theses   ( marcgt )
government publication (state, provincial, terriorial, dependent)   ( marcgt )
born-digital   ( sobekcm )
Electronic Thesis or Dissertation

Notes

Abstract: Relighting, pose change and recognition of faces from images are intimately connected fundamental problems in the field of Computer Vision and Graphics. These problems are particularly interesting and difficult when examined in presence of constraints like limited number of input images, cast shadows and specularities. Though numerous solutions have been proposed in the past, none effectively addresses these problems when considered in the aforementioned constrained setting. In this dissertation, we present a set of techniques, which accomplish relighting, pose change and recognition of facial images in presence of specularities and shadows, using as few as one input image. We start by presenting a method for relighting and pose change for facial images using nine or more input images. We accomplish this by representing the Apparent Bidirectional Reflectance Distribution Function (ABRDF) fields of human faces using Tensor Splines. We then present a method for improving the quality of relighted images by enhancing the ABRDFs with face specific subspace representations. Next, we present a novel technique for estimating facial ABRDF fields for the difficult case of single input image. Finally, we focus on the face recognition problem and present a novel face image classification scheme as well as a framework for enhancing face recognition using relighting methods outlined above. All of the above mentioned techniques are supported by extensive experiments on the Yale A, the CMU PIE, the Extended Yale B and the MERL Dome face image databases. We show that our relighting, pose change and recognition systems outperform various state-of-the-art methods in terms of image quality and recognition rates.
General Note: In the series University of Florida Digital Collections.
General Note: Includes vita.
Bibliography: Includes bibliographical references.
Source of Description: Description based on online resource; title from PDF title page.
Source of Description: This bibliographic record is available under the Creative Commons CC0 public domain dedication. The University of Florida Libraries, as creator of this bibliographic record, has waived all rights to it worldwide under copyright law, including all related and neighboring rights, to the extent allowed by law.
Statement of Responsibility: by Ritwik Kumar.
Thesis: Thesis (Ph.D.)--University of Florida, 2009.
Local: Adviser: Vemuri, Baba C.
Local: Co-adviser: Banerjee, Arunava.
Electronic Access: RESTRICTED TO UF STUDENTS, STAFF, FACULTY, AND ON-CAMPUS USE UNTIL 2010-06-30

Record Information

Source Institution: UFRGP
Rights Management: Applicable rights reserved.
Classification: lcc - LD1780 2009
System ID: UFE0041052:00001

Permanent Link: http://ufdc.ufl.edu/UFE0041052/00001

Material Information

Title: Relighting, Pose Change and Recognition of Faces
Physical Description: 1 online resource (114 p.)
Language: english
Creator: Kumar, Ritwik
Publisher: University of Florida
Place of Publication: Gainesville, Fla.
Publication Date: 2009

Subjects

Subjects / Keywords: abrdf, face, pose, recognition, reflectance, relighting
Computer and Information Science and Engineering -- Dissertations, Academic -- UF
Genre: Computer Engineering thesis, Ph.D.
bibliography   ( marcgt )
theses   ( marcgt )
government publication (state, provincial, terriorial, dependent)   ( marcgt )
born-digital   ( sobekcm )
Electronic Thesis or Dissertation

Notes

Abstract: Relighting, pose change and recognition of faces from images are intimately connected fundamental problems in the field of Computer Vision and Graphics. These problems are particularly interesting and difficult when examined in presence of constraints like limited number of input images, cast shadows and specularities. Though numerous solutions have been proposed in the past, none effectively addresses these problems when considered in the aforementioned constrained setting. In this dissertation, we present a set of techniques, which accomplish relighting, pose change and recognition of facial images in presence of specularities and shadows, using as few as one input image. We start by presenting a method for relighting and pose change for facial images using nine or more input images. We accomplish this by representing the Apparent Bidirectional Reflectance Distribution Function (ABRDF) fields of human faces using Tensor Splines. We then present a method for improving the quality of relighted images by enhancing the ABRDFs with face specific subspace representations. Next, we present a novel technique for estimating facial ABRDF fields for the difficult case of single input image. Finally, we focus on the face recognition problem and present a novel face image classification scheme as well as a framework for enhancing face recognition using relighting methods outlined above. All of the above mentioned techniques are supported by extensive experiments on the Yale A, the CMU PIE, the Extended Yale B and the MERL Dome face image databases. We show that our relighting, pose change and recognition systems outperform various state-of-the-art methods in terms of image quality and recognition rates.
General Note: In the series University of Florida Digital Collections.
General Note: Includes vita.
Bibliography: Includes bibliographical references.
Source of Description: Description based on online resource; title from PDF title page.
Source of Description: This bibliographic record is available under the Creative Commons CC0 public domain dedication. The University of Florida Libraries, as creator of this bibliographic record, has waived all rights to it worldwide under copyright law, including all related and neighboring rights, to the extent allowed by law.
Statement of Responsibility: by Ritwik Kumar.
Thesis: Thesis (Ph.D.)--University of Florida, 2009.
Local: Adviser: Vemuri, Baba C.
Local: Co-adviser: Banerjee, Arunava.
Electronic Access: RESTRICTED TO UF STUDENTS, STAFF, FACULTY, AND ON-CAMPUS USE UNTIL 2010-06-30

Record Information

Source Institution: UFRGP
Rights Management: Applicable rights reserved.
Classification: lcc - LD1780 2009
System ID: UFE0041052:00001


This item has the following downloads:


Full Text

PAGE 1

1

PAGE 2

2

PAGE 3

3

PAGE 4

IamextremelygratefultoDr.BabaC.VemuriandDr.ArunavaBanerjeefortheirguidanceandsupportduringmygraduatestudies.Theyhavebeenaconstantsourceofinspirationandencouragementforme,themostimportantingredientsnecessaryforresearch.IamalsothankfultoDr.JeeryHo,Dr.AnandRangarajanandDr.TrevorParkforbeingonmysupervisorycommitteeandprovidingextremelyusefulinsightsintotheworkpresentedinthisdissertation.IwouldliketothanktheDepartmentofComputerandInformationScienceandEngineering(CISE)andtheUniversityofFlorida(UF)forgivingmetheopportunitytopursuemygraduatestudiesinaveryconstructiveenvironment.IamespeciallythankfultomyUFAlumniFellowship,theDept.ofCISEandtheNIHgrantsNS46812(toDr.BabaC.Vemuri),EB007082(toDr.BabaC.Vemuri)andEB004752(toDr.PaulR.CarneyandDr.ThomasH.Mareci)forfundingmydoctoralstudiesandtravelstovariousconferences.Duringmygraduatestudies,IenjoyedmyjobasateachingassistantandforthatIamgratefultoDr.ManuelBermudezforbeingaterricboss.IamgratefultoDr.TanveerSyeda-MahmoodforgivingmetheopportunitytoworkwithherwonderfulresearchgroupattheIBMAlmadenResearchCenter.IappreciatetheopportunitygiventomebyDr.MichaelJonesandDr.TimMarkstoworkattheMitsubishiElectricResearchLaboratories(MERL).IamespeciallygratefultoMERLforprovidingdataandsoftwaretogeneratesomeoftheresultsincludedinthisdissertation.AttheCenterforVisionGraphicsandMedicalImaging(CVGMI),Iwasveryfortunatetogettospendtimeincompanyofmanywonderfulpeople.IdeeplyappreciateAngelosBarmpoutisforhisfriendshipandguidanceduringthecourseofmygraduatestudies.FeiWanghasbeenawonderfulfriendthroughoutmygraduatestudiesandIamgratefulforallthementoringandsupporthehasprovided.ImustthankAjitRajwade,BingJianandSanthoshKodipakaforbeingextremelyhelpfulandpatient,asIendlesslybotheredthemwithmyquestions.Ialsoappreciatethecamaraderieofmylab-mates 4

PAGE 5

5

PAGE 6

page ACKNOWLEDGMENTS ................................. 4 LISTOFTABLES ..................................... 8 LISTOFFIGURES .................................... 9 ABSTRACT ........................................ 11 CHAPTER 1PROBLEMDESCRIPTION ............................. 13 1.1Introduction ................................... 13 1.2FacialRelighting ................................ 14 1.3PoseChange ................................... 14 1.4FaceRecognition ................................ 14 1.5TheInterconnections .............................. 15 1.6Organization .................................. 15 2FACIALRELIGHTINGANDPOSECHANGEWITHMULTIPLEIMAGES 17 2.1Introduction ................................... 17 2.2RelatedWork .................................. 19 2.3Overview .................................... 27 2.4TensorSplines .................................. 28 2.4.1SphericalFunctionsModeledasTensors ................ 29 2.4.2TensorSplines .............................. 30 2.4.3FacialABRDFApproximationUsingTensorSplines ......... 31 2.5MixtureofSingle-lobedFunctions ....................... 36 2.6RecoveringShapefromtheABRDFField .................. 37 2.6.1RotationEstimation .......................... 38 2.6.2SurfaceNormalComputation ...................... 40 2.6.3ShapeRecovery ............................. 41 2.6.4NovelPoseRelighting .......................... 42 2.7ExperimentalResults .............................. 44 2.7.1RelightingFaces ............................. 45 2.7.2EstimatingShape ............................ 48 2.7.3FaceRecognition ............................ 50 2.8Conclusions ................................... 52 3EIGENBUBBLES:THEENHANCEDABRDFREPRESENTATION ...... 55 3.1Introduction ................................... 55 3.2Eigenbubbles .................................. 55 3.3Experiments&Discussion ........................... 57 6

PAGE 7

................................ 57 3.3.2FaceRecognition ............................ 63 3.3.3ABRDFFieldCompression ....................... 64 3.4Conclusions ................................... 65 4FACERELIGHTINGANDPOSECHANGEWITHSINGLEIMAGE ..... 66 4.1Introduction ................................... 66 4.2RelatedWork .................................. 67 4.3Overview .................................... 68 4.4TheReferenceABRDFFieldModel ...................... 69 4.5ModelFitting .................................. 71 4.6ExperimentalResults .............................. 73 4.6.1Relighting ................................ 74 4.6.2PoseChange ............................... 75 4.7Conclusions ................................... 75 5FACERECOGNITION ............................... 81 5.1Introduction ................................... 81 5.2VolterraKernelApproximations ........................ 83 5.3KernelComputationasGeneralizedEigenvalueProblem .......... 85 5.3.1FirstOrderApproximation ....................... 87 5.3.2SecondOrderApproximation ...................... 88 5.4TrainingandTestingAlgorithms ....................... 89 5.5Experiments ................................... 90 5.6DiscussionandConclusion ........................... 93 6ENHANCINGFACERECOGNITIONWITHRELIGHTING .......... 95 6.1Introduction ................................... 95 6.2GallerySetAugmentation ........................... 95 6.3ExperimentalResults .............................. 96 6.3.1NearestNeighborClassier ....................... 96 6.3.2MERLClassier ............................. 96 6.3.3LocalBinaryPatternClassier ..................... 97 6.3.4NaiveRelighting ............................. 97 6.3.5PracticalRelighting ........................... 97 6.3.6Results .................................. 98 6.4Conclusion .................................... 100 7CONCLUSION .................................... 104 REFERENCES ....................................... 106 BIOGRAPHICALSKETCH ................................ 114 7

PAGE 8

Table page 2-1Requirements,assumptionsandcapabilitiesofexistingmethods ......... 20 2-2FacerecognitionerrorrateswithTensorSplines .................. 52 3-1FacerecognitionerrorrateswithEigenbubbles ................... 64 3-2ABRDFeldcompression .............................. 65 5-1State-of-the-artmethodsforfacerecognition .................... 90 5-2FacerecognitionresultsontheYaleAdataset ................... 91 5-3FacerecognitionresultsontheCMUPIEdataset ................. 91 5-4FacerecognitionresultsontheExtendedYaleBdataset ............. 92 6-1Eectofthegalleryaugmentationsizeontherecognitionrates .......... 98 6-2FacerecognitionratesfortheCMUPIEdataset .................. 99 6-3FacerecognitionratesfortheMERLDomedataset ................ 99 8

PAGE 9

Figure page 2-1Lambertianmodelvs.CartesianTensorsmodel .................. 19 2-2SymmetricandantisymmetricABRDFapproximations .............. 24 2-3ABRDFalignment .................................. 25 2-4ABRDFsonaface .................................. 29 2-5Novelsynthesizedimages ............................... 32 2-6Novelimagesincomplexlighting .......................... 33 2-7Shadowsandspecularitiescomparison ....................... 39 2-8Impactoftheinputimages ............................. 40 2-9Perpixelintensityerrorcomparison ......................... 42 2-10Shapesrecoveredusing9images ........................... 43 2-11Posevariation ..................................... 47 2-12ShapecomparisonwiththeRobustPhotometricStereo .............. 49 2-13Simultaneousposeandilluminationvariation ................... 51 3-1GlobalEigenbubbles ................................. 57 3-2LocalEigenbubbles .................................. 58 3-3ABRDFsonaface:Eigenbubbles .......................... 58 3-4Globalvs.LocalEigenbubbles ............................ 59 3-5ImagesrelightedwiththeEigenbubbles ....................... 61 3-6QuantitativecomparisonwithEigenbubbles .................... 62 3-7Shadowsandspecularitiesinrelightedimages ................... 62 3-8ExtrapolatedlightingconditionswithEigenbubbles ................ 63 4-1OverviewoftheABRDFeldtting ........................ 71 4-2Comparisonwiththegroundtruthimages ..................... 74 4-3RelightedimagesfromtheCMUPIEdataset ................... 76 4-4RelightedimagesfromtheCMUPIEdataset ................... 77 9

PAGE 10

.................. 78 4-6RelightedimagesfromtheMERLDomedataset .................. 79 4-7Posechangedimagesusingsingleinputimage ................... 80 5-1StructureofA1iandA2imatrices ........................... 82 5-2Trainingalgorithm .................................. 83 5-3Testingalgorithm ................................... 86 6-1ROCCurvefortheCMUPIEdatasetwiththeMERLclassier. ......... 100 6-2ROCCurvefortheMERLDomedatasetwiththeMERLclassier. ....... 101 6-3ROCCurvefortheCMUPIEdatasetwiththeNearestNeighborclassier. ... 101 6-4ROCCurvefortheMERLDomedatasetwiththeNearestNeighborclassier. 102 6-5ROCCurvefortheCMUPIEdatasetwiththeLBPclassier. .......... 102 6-6ROCCurvefortheMERLDomedatasetwiththeLBPclassier. ........ 103 10

PAGE 11

Relighting,posechangeandrecognitionoffacesfromimagesareintimatelyconnectedfundamentalproblemsintheeldofComputerVisionandGraphics.Theseproblemsareparticularlyinterestinganddicultwhenexaminedinpresenceofconstraintslikelimitednumberofinputimages,castshadowsandspecularities.Thoughnumeroussolutionshavebeenproposedinthepast,noneeectivelyaddressestheseproblemswhenconsideredintheaforementionedconstrainedsetting.Inthisdissertation,wepresentasetoftechniques,whichaccomplishrelighting,posechangeandrecognitionoffacialimagesinpresenceofspecularitiesandshadows,usingasfewasoneinputimage. Westartbypresentingamethodforrelightingandposechangeforfacialimagesusingnineormoreinputimages.WeaccomplishthisbyrepresentingtheApparentBidirectionalReectanceDistributionFunction(ABRDF)eldsofhumanfacesusingTensorSplines.WethenpresentamethodforimprovingthequalityofrelightedimagesbyenhancingtheABRDFswithfacespecicsubspacerepresentations.Next,wepresentanoveltechniqueforestimatingfacialABRDFeldsforthedicultcaseofsingleinputimage.Finally,wefocusonthefacerecognitionproblemandpresentanovelfaceimageclassicationschemeaswellasaframeworkforenhancingfacerecognitionusingrelightingmethodsoutlinedabove. AlloftheabovementionedtechniquesaresupportedbyextensiveexperimentsontheYaleA,theCMUPIE,theExtendedYaleBandtheMERLDomefaceimagedatabases. 11

PAGE 12

12

PAGE 13

Theworkpresentedinthisdissertationaddressesthethreefundamentalproblemsinfacialimageanalysis-facialrelighting,posechangeandrecognition.Thesehavebeenselectedbecausetheseinterconnectedproblemscoverasubstantialsubsetofthefacialimagemanipulationandunderstandingbasedapplicationsdescribedabove.Theexactnatureofeachoftheseproblemsdependsonthechosenassumptions.Inabroadsense,wewillpresentsolutionstotheproblemofrelightingandposechangeforboththemultipleimageinputandthesingleimageinputcases.Wewillalsoexaminetheproblemofthemultipleimagefacerecognitionbypresentinganovelfaceimageclassicationmethodandexplorethesingleimagefacerecognitionproblemingeneralasitrelatestofaceimagerelighting.Notethatthesingleandmultipleimageshererefertothegalleryorthetrainingset. 13

PAGE 14

14

PAGE 15

Thecomplexityoftherecognitionproblemisgovernedbyboththeprobeandthegalleryimages.Theproblemofrecognizingthefaceissimpleriftheprobeisobtainedunderpredeterminedposeandlightingconditionsanditbecomesharderasthecontrolovertheprobeimageisrelaxedandphoto-eectslikeshadowsandspecularitiesareallowed.Theproblemisagainsimplerifthegalleryimagesareobtainedunderpredeterminedconditions.Thenumberofimagesinthegalleryalsoimpactsthedicultyoftheproblem. Theproblemsofrelightingandposechangearealsointimatelyconnected.Thechangeintheobservedimageintensityvaluesastheenvironmentlightingchangesisknowntobein-partgovernedbythelocalshapeoftheobject([ 1 ],[ 2 ]).Theglobalshapecanalsoaectphoto-eectslikeshadowsinanimage. 2 ,wherewepresentadetaileddescriptionoftheseproblems,literaturesurvey,ourproposedsolutionsandtheexperimentalresults.InChapter 3 wediscussamethodforenhancingthequalityofrelightedimagesusingEigenbubbles.Themoredicultcaseofrelightingandposechangeproblem,whichworkswithsingleinputimageisdiscussedinChapter 4 .Thenweshiftfocusonthemultiple 15

PAGE 16

5 .InChapter 6 wedetailournovelframeworkforsingleimagefacerecognitionthatdrawsupontherelightingschemedescribedinChapter 4 toenhancerecognitionrates.Finally,inChapter 7 weconcludewithasummaryofthecontributionsmadeinthisdissertation.Partoftheworkpresentedinthisdissertationhasbeenpublishedin[ 3 ],[ 4 ]and[ 5 ]. 16

PAGE 17

Ourunderstandingoftheprocessofimageformationandtheinteractionoflightandthefacialsurfacehascomealongwaysincewestarted[ 6 ],withmanyimpressivestridesalongtheway(e.g.[ 7 ],[ 8 ],[ 9 ]),butwearestillsomedistancefromanidealsolution.Inourview,anidealsolutiontotheproblemofmodelingandrenderingappearancesandshapesofhumanfacesshouldbeabletogenerateextremelyphoto-realisticrenderingsofaperson'sface,givenjustone2Dimageoftheface,inanydesiredilluminationconditionandpose,ataclickofabutton(realtime).Furthermore,suchasystemshouldnotrequireanymanualinterventionandshouldnotbefazedbythepresenceofcommonphoto-eectslikeshadowsandspecularitiesintheinput.Lastly,suchanidealsystemshouldnotrequireexpensivedatacollectiontoolsandprocesses,e.g.3Dscanners,andshouldnotassumeavailabilityofmeta-informationabouttheimagingenvironment(e.g.lightingdirections,lightingwavelengthetc.). Thesegeneralrequirementshavebeensingledoutbecausethestate-of-the-artislargelycomprisedofsystemswhichrelaxoneormoreoftheseconditionswhilesatisfying 17

PAGE 18

7 ]),availabilityof3Dfacemodel(e.g.[ 10 ]),manualinitialization(e.g.[ 11 ]),absenceofcastshadowsininputimages(e.g.[ 12 ]),availabilityoflargeamountofdataobtainedfromcustombuiltrigs(e.g.[ 8 ])etc.Theseassumptionsarenotedas\simplifying"because{humanfacesareknowntobeneitherexactlyLambertiannorconvex(andthuscanhavecastshadows),ttinga3Dmodelrequirestimeconsuminglarge-scaleoptimizationwithmanualselectionoffeaturesforinitialization,specializeddataacquisitioncanbecostlyandinmostrealapplicationsonlyafewimagesofafaceareavailable. Themethodweproposeinthischaptermovesthestate-of-the-artclosertotheidealsolutionbysatisfyingmoreoftheabovementionedattributessimultaneously.Ourtechniquecanproducephoto-realisticrenderingsofhumanfacesacrossarbitraryilluminationandposeusingasfewas9images(xedpose,knownilluminationdirection)withaspatiallyvaryingnon-Lambertianreectancemodel.Unlikemosttechniques,ourmethoddoesnotrequireinputimagestobefreeofcastshadowsorspecularitiesandcanreproducetheseinthenovelrenderings.Itdoesnotrequireanymanualinitializationandisapurelyimagebasedtechnique(noexpensive3Dscansareneeded).Furthermore,itiscapableofworkingwithimagesobtainedfromstandardbenchmarkdatasetsanddoesnotrequirespecializeddataacquisition. OurtechniqueisbasedontheTensorSplinesframeworkwhichcanbeusedtoapproximateanyn-dimensionaleldofsphericalfunctions(originallyproposedin[ 13 ]).Inthecaseoffaces,we,forthersttime,useTensorSplinestoapproximatetheeldofApparentBidirectionalReectanceDistributionfunction(ABRDF)foraxedviewingdirection.UnliketheBRDF,theABRDF(alsoknownasthereectancefunction)ateachpixelcapturesthevariationinintensityasafunctionofilluminationandviewingdirectionandisthussensitivetothecontextofthepixel.OncetheABRDFeldhasbeencaptured,imagesofthefaceunderthesameposebutwitharbitraryilluminationcanbegenerated 18

PAGE 19

bysimplytakingweightedcombinationsoftheABRDFeldsamples.Next,weestimatethesurfacenormalateachpixelbyrobustlycombiningtheshapeinformationfromitsneighboringpixels.Towardsthis,weputforwardaniterativealgorithmwhichworksbyregisteringneighborhoodABRDFsusinganextremelyecientlineartechnique.Withasfewas1or2iteration,wecanrecoverthesurfacenormaleldsofmostfaceswhicharethennumericallyintegratedtoobtainthefacesurfaces.Novelposewithnovelilluminationconditionscanthusberenderedwhileseamlesslyaccountingforattachedaswellascastshadows. 2-1 19

PAGE 20

Requirements,assumptionsandcapabilitiesofexistingmethods AssumedSur-faceBRDFModel No.ofImagesasInput RelightedImagesPre-sented ShapeorPoseResultsPre-sented CastShadowsinInput PurelyImagebased(No3DScans) OtherAssumptions,RequirementsandLimitations 1999MVIEW[ 14 ] Lambertian 4 8 4 15 ] Non-Lamb. 1 4 8 8 8 ] Non-Lamb. 4 4 4 9 ] Lambertian 8 8 8 16 ] Non-Lamb. 4 8 8 12 ] Lambertian 4 8 4 17 ] Lambertian 1 8 8 4 18 ] Lambertian 1 4 8 4 19 ] Non-Lamb. 4 4 4 20 ] Non-Lamb. 4 8 4 21 ] Lambertian 1 4 8 8 7 ] Lambertian 1 8 8 8 22 ] Lambertian 4 8 4 23 ] Non-Lamb. 12 4 8 4 24 ] Non-Lamb. 4 8 4 25 ] Lambertian 1 8 4 8 11 ] Lambertian 1 4 8 8 26 ] Non-Lamb. 8 8 8 27 ] Lambertian 4 4 4 28 ] Non-Lamb. 4 4 4 29 ] Lambertian 1 4 8 8 30 ] Lambertian 1 4 8 4 31 ] Lambertian 15 4 8 4 32 ] Non-Lamb. 4 8 4 4 4 4

PAGE 21

33 ].Whenanambientlightingcomponentisincluded,thissubspaceexpandstobecome4-dimensional[ 34 ]andwhenattachedshadowsaretakenintoaccount,thesubspacegrowstobecomeaninnitedimensional{illuminationcone[ 35 ]. SphericalharmonicanalysisoftheLambertiankernelhasshownthateventhoughtheilluminationconeisinnitedimensional,itcanbeapproximatedquitewellbyalowerdimensionalsubspaces([ 36 ],[ 9 ],[ 7 ]).Inparticular,thesemethodscanproduceimpressiveresultswith9basisimages,thoughtheyrequirethe3Dshapesandthealbedoeldsasinput.Thesebasisimagescanalsobedirectlyacquiredusingthe\universalvirtual"lightingconditions[ 37 ].Morerecently,thisideahasbeenextendedto3Dsurfacesin[ 11 ]buildingonthepriorseminalworkpresentedin[ 15 ]calledMorphableModels.MorphableModelscanrecover3Dshapeofafacebyttinganaverage3Dfacialmodeltoagiven2Dimage,accountingfornecessaryshapeandtextureadjustments.MorphableModelsareknowntoproduceexcellentresultsforacrossposefacerecognitionbutcannothandlecastshadowsorspecularitiesrobustly.Moreimportantly,theyrequiremanualdelineationoffacialfeaturestoinitializeacomplicatednon-linearoptimizationwhichcantakealongtimetoconvergeandcansuerfromlocalminima.Usingtheideaofalowdimensionalsubspaceexploredabove,[ 22 ]representedtheentirelight-eldusingalowdimensionaleigenlight-eld. 21

PAGE 22

38 ].MethodsthatarepurelyimagebasedandworkwiththeLambertianassumptiongenerallyapplyphotometricstereoorshapefromshadingtorecoverfacialshapefromthegivenimages.Forinstance,resultsforsimultaneousshaperecoveryusingphotometricstereoandreectancemodelingwerepresentedin[ 14 ]and[ 12 ].Bothofthesemethodsworkwithmultipleimagesandexpectnocastshadowsandverylittleattachedshadowsintheimages.Herethecastshadowsintherelightedimagesarerenderedusingraytracing,whichcanbecomputationallyexpensive.ExamplesofmethodsthatrecovershapefromshadingworkingundertheLambertianassumptioncanbefoundin[ 18 ]and[ 39 ].Asthesemethodsworkwithjustoneimage,besidesrequiringtheabsenceofcastshadowsintheinput,theymakeadditionalassumptionslikefacialsymmetry(asin[ 18 ])etc.Animportantpointtonotehereisthattheuncalibratedphotometricstereoortheshapefromshadingmethods,thatworkwiththeLambertianassumptionandorthographicallyprojectedimages,alsosuerfromtheBas-ReliefAmbiguity([ 40 ]).Resolvingthisrequiresadditionalassumptionslikethesymmetryofface,thenoseandtheforeheadbeingatthesameheight,knownlightingdirections,etc.,andmanualassistance. Recently,shaperecoveryusingthegeneralizedphotometricstereowaspresentedin[ 31 ]whichrelaxessomeoftheassumptionsmadebythetraditionalphotometricstereotechniques.Thismethodcanrecovershapefromimagestakenundergeneralunknownlighting.OnaccountoftheLambertianassumption,castshadowsarenotentertainedintheinputimagesandtheshapeoftheobjectisassumedtobeconvex.Notethattheaccuraterecoveryofshapeusingthismethodrequires15to60imagesasinput.AnothermethodforLambertianshaperecoverywithmultipleilluminants,butwithoutignoringshadows,waspresentedin[ 27 ]wherethegraphcutsmethodwasusedtoidentifylightsourcevisibilityandinformationfromshadowmapswereusedtorecovertheshape. 22

PAGE 23

17 ],wherethesocalled\QuotientImages",generatedusingratioofalbedovalues,wereusedtogenerateimagesundernovelilluminationconditions.Morerecently,useofinvariantswasinvokedin[ 21 ],wheretheradianceenvironmentmapwasdeducedusingtheratioimagetechnique([ 41 ],[ 17 ]).Notethattheshaperecoveryin[ 21 ],liketheMorphableModels,requiresmanualinitialization.Forgoingtheratiotechnique,directuseofalbedoasanilluminationinvariantsignatureoffaceimageswasexploredin[ 25 ],whereusingauniversal3Dfacemodel,illuminationnormalizedimagesoffacesweregenerated.Thismethodworkedwithlowresolutionimagesanddidnotrenderhighqualityrelightedimages.Morerecentlyanimprovementwaspresentedin[ 29 ]wherethealbedoestimationwasmademorerobustusingtheerrorstatisticsofsurfacenormalsandtheknownilluminationdirection.Thismethodrequiresaregisteredaverage3Dmodelofthefaceanddoesnotallowcastshadowsintheinputbutunlike[ 25 ],alsoprovidesafacialshapeestimate.Improvingupontheideaoftheidealclassassumption([ 17 ]),anothergeneralizedphotometricstereotechniquewaspresentedin[ 30 ].UsingabootstrapsetoffacialimagesandexploitingthesubspacespannedbyasetofbasisobjectswithLambertiansurfaces,imageswithnovelposeandilluminationweregenerated.Herethefaceswereassumedtobesymmetricandtheinputwasassumedtobefreeofshadows. Next,welookattechniquesthatdonotmaketheLambertianassumption.Seminalworkinthisclassoftechniqueswaspresentedin[ 8 ],whereusingacustombuiltrig,densesamplingoftheilluminationspaceforfaceswasobtained.Inthiswork,thefacialshapewasobtainedusingstructuredlightingandnoassumptionaboutthesurfaceBRDFwasmade.Thiscompletelydatadriventechniquewasabletoproduceextremelyphoto-realisticimagesofthefaceinnovelilluminationsandposes.Thespecularcomponentwascapturedusingpolarizedlightingandmodiedappropriatelyforpose 23

PAGE 24

BApproximationof(A)usingsymmetricfunctions CMaxoffunctionin(B)andzero DApproximationof(A)usinganti-symmetricfunctions EMaxoffunctionin(D)andzero FABRDFapproximatedwithsymmetricfunctionsleadstounnaturallighting GABRDFapproximatedwithanti-symmetricfunc-tionleadstomorenaturallighting SymmetricandantisymmetricABRDFapproximations variation.Thismethoddemonstratedthatifalargenumberofimages(>2000)foreachsubjectcanbeobtainedundervariouslightingcongurations,therelightingandtheposegenerationproblemscanbesolved,butthecostofsuchasystemcanbeextremelyhigh. Useofbiquadraticpolynomialstomodeltexturewasexploredin[ 16 ].Thismethodrequiredacustombuiltrigandmorethan50specularityfreeimagestorecoverthemodelparameters.Theshapeoftheobjectwasnotrecoveredinthismethod.Useofalargenumber(300)ofimagestorecovertheshapewithoutmakinganyassumptionaboutthenatureoftheBRDFwasrevisitedin[ 19 ].Thismethodrequiredtheinputimagesto 24

PAGE 25

doublycovertheilluminationdirectionswhichcalledforspecializeddataacquisition.Noattempttocapturethereectancepropertiesoftheobjectwasmadeinthiswork. Oneofthersttechniquesthatworkedwithstandardfacedatabasesanddidnotrequirecustomdatawaspresentedin[ 20 ]wherethemoregeneralTorrance-Sparrow([ 2 ])modelfortheBRDFwasused.Thismethodpresentedtherelightingandtheposevariationresultswith12imagesasinputbutdidnotallowcastshadows.Further,thismethodrequiredeachpixeltobelitbyatleast3lightsourcesinordertoworkproperly. Importantcontributionintheeldofexamplebasedshaperecoverywasmadeby[ 42 ]whereobjectsofinterestwereimagedalongwithareferenceobjectofknowngeometry(e.g.sphere).Multiple(8)castshadowsfreeimageswereusedasinput.ThisworkwasexpandedtoallowspatiallyvaryingBRDFbyusingmultiplereferencesin[ 24 ].Aninterestingextensionofthisworkwaspresentedin[ 23 ]wheretheshapeofanobjectwas 25

PAGE 26

38 ]presentedatechniqueforposeandilluminationvariationusingsingleimage,thatusedtheMorphableModelstorecoverthe3Dshapeandthehigher-orderSVDtorecovertheilluminationsubspace.ThismethoddoesnotexplicitlymaketheLambertianassumptionbutitrequiresmanualinitializationforthe3Dmodeltting. Whenthe3Dshapesoftheobjectsareassumedavailable,[ 26 ]presentedatechniquewhich,attimesusingjustoneimage,canrecovertheirspatiallyvaryingnon-parametricBRDFelds.Forthecaseofthehumanface,thisworkpresentedresultswith4imageswherethespecularcomponentwasseparatelycapturedusingpolarizedlighting.Theimageswereacquiredfromknownilluminationdirectionsandnocastshadowswereallowed. Recently,[ 28 ]presentedanewmethodforphotometricreconstructionofshapeassumingspatiallyvaryingbutisotropicBRDFs.Given32ormoreimageswithknownillumination,thismethodrecoversisocontoursofthesurfacedepthmapfromwhichtheshapecanberecoveredbyimposingadditionalconstraints.Anextensionofthisworkwaspresentedin[ 32 ]wheretheneedforadditionalconstraintstorecovertheshapefromthedepthmapisocontourwasalleviatedbyassumingthesurfacetobecomposedofafewfundamentalmaterialsandthattheBRDFateachpointcanbeapproximatedbyabivariatefunction.Resultspresentedinthisworkrequired102ormoreimages.AnotherinterestingframeworkforphotometricstereousingtheMarkovRandomFieldapproachwaspresentedin[ 43 ]. Lastly,wenotethatinthecaseswhenextremelyhighqualityrenderingsarerequiredandcost-timeconstraintsarerelaxed,customhardwareisemployed.Forinstance,highlyaccuratemeasurementsofmaterialBRDFswerecarriedoutusingagonioreectometer 26

PAGE 27

44 ]andvariouscustomizedhardwarecomponentsandsoftwarewereusedtorenderfaceimagesinthemovie\TheMatrixReloaded"[ 45 ].Inordertomeasureaccurateskinreectancewhileaccountingforsub-surfacescattering,custombuiltdeviceswereagainemployedin[ 46 ]torenderhighqualityfacialimages. ItcannoticedthatmostoftheimagebasedtechniquesthatdonotmakethesimplifyingLambertianassumptionendupusingalargeamountofcustomacquireddataorassumingsomeotherparametricformfortheBRDF(besidestheotherassumptions).Inthischapterweexplorethepossibilityofacquiringthenon-Lambertianreectanceandshapewithjustnineimagesinapurelydatadrivenfashion. EmbeddedintheABRDFsateachpixelalsoliesthesurfacenormalatthispoint.ToextractthenormalfromtheABRDFeldriddledwithcastshadowsandspecularities,weinvokethehomogeneityoftheABRDFsinlocalneighborhoods,andinfersurfacenormalatapixelusingtheinformationfromitsimmediateneighbors.Moreconcretely,ateachpixelwealigntheABRDFwithitsneighbors'ABRDFusingalinearizedalgorithmforrotationrecoveryandtakeaweightedgeodesicmeanofthenormalssuggestedby 27

PAGE 28

Equippedwiththismechanismtocapturebothreectancepropertiesandshapesofthehumanfaces,wecangenerateimagesofanyfaceinnovelposesandilluminationconditions. 23 ]workedwith12imagesobtainedfromknownlightingdirectionsinxedpose.Asthenumberofinputimagesincreases,theperformanceofourmethodimproves.Wedonotrequiretheinputimagestobefreeoftheattachedorthecastshadows.WealsodonotrestricttheBRDFtobeLambertian([ 12 ])orisotropic([ 28 ],[ 26 ]).Thoughglobalphoto-eectslikesubsurfacescatteringandinterreectionarenotexplicitlymodeled,TensorSplinescancapturethemtosomeextent.

PAGE 29

RecoveredABRDFsforahumanface.ComplexshapesoftheABRDFsinvariousregionsofthefacecanbereadilynoted. 47 ](atensorinR3),isexpressedas whereTklmarethereal-valuedtensorcoecientsandk,l&marenon-negativeintegers.ThisisaCartesiantensorwithallthenargumentssettobev.TheexpressivepowerofsuchCartesiantensorsincreaseswiththeirorder.Geometricallythistranslatestopresenceofmore\lobes"onahigherorderCartesiantensor. NotethattheLambertianmodelisintricatelyconnectedtoaspecialcaseofthisCartesiantensorformulation.Ifv=(v1v2v3)Tisthelightsourcedirection,n=(n1n2 1Inthenotationusedinthispaper,thisorderisnotsameasthenumberofindicesusedtorepresentthetensor. 29

PAGE 30

(2{2) withT100=n1,T010=n2andT001=n3.AcomparisonwithEq. 2{1 revealsthattheLambertiankernelisexactlythepositivehalfofthe1storderCartesiantensor. The1st,2nd,3rdand5thorderCartesiantensorshave3,6,10and21uniquecoecientsrespectively.Forevenorders,theCartesiantensorsaresymmetric,T(v)=T(v),whileforoddorderstheyareanti-symmetric,T(v)=T(v).Wemustpointoutthatthesedenitionsofsymmetryandanti-symmetryaredierentfromthestandarddenitionsbasedonswitchingofthearguments'order.Inthischapter,wewouldusethedenitionsweprovidedabove. Finally,thoughthehigherordertensorscanbemoreexpressive,theycanbeperceivedtobemoresensitivetonoiseduetotheirabilitytomodelhighfrequencydetails.Incontrast,thelowerordertensorsareincapableofmodelinghighfrequencyinformationbutarguablyaremorerobusttonoise.Sinceitisimpossibletodiscriminatebetweenhighfrequencydetailandnoiseinthedata,itisreasonabletosaythatthehigherordertensorspossesshighernoisesensitivity.Thus,likeinanyotherapproximationtask,wemuststrikeabalancebetweenthehighfrequencydatadelityandthenoisesensitivity. 48 ])acrossthelatticeofthesphericalfunctions. 30

PAGE 31

and ti+kti+1;(2{4) TheNi;k+1(t)arepolynomialsofdegreek,associatedwithn+k+2monotonicallyincreasingnumberscalled\knots"(tk;tk+1;:::;tn+1). TheTensorSplineforap-dimensionallatticeofsphericalfunctions,withkthdegreesplineandnthorderCartesiantensorisdenedas wheret=(t1:::tp)istheindexintothesphericalfunctionlattice,v=(v1v2v3)Tisaunitvector,Disthep-dimensionalsplinecontrolpointlatticeandTi1:::ip(v)isgivenbyEq. 2{1 .InTensorSplinestheusualB-SplinecontrolpointshavebeenreplacedbythecontroltensorsTi1:::ip(v).TheformulationpresentedinEq. 2{5 isquitegeneralasitcanbeusedtoestimateasphericalfunctionelddenedoveranarbitrarydimensionallattice,withanydesireddegreeofB-Splinesmoothing. 31

PAGE 32

ImagessynthesizedusingTensorSplinesundernovelilluminationdirections.Theilluminationdirectionismentionedoneachimageas(azimuth,elevation).Thenineimagesusedasinputwereilluminatedfrom(-20,60),(0,45),(20,60),(-50,0),(0,0),(50,0),(-50,-40),(0,-35)and(50,40)directions. homogeneityalsoholdstoareasonabledegree.Inordertoensurethatthesmoothnessismanifestedonlyinalocalizedfashion,wehavechosentousebi-cubicB-SplinesintheABRDF-specializedversionoftheTensorSplines. TheabilityoftheCartesiantensorstobettermodeldatawithcomplexdistributionscanbenotedinFig. 2-1 ([ 5 ]),whereintherstrow,weshowthatforthecaseofsyntheticcirculardata(shownbythegreenarrows),theCartesiantensorscanmoreaccuratelyapproximatethedatathantheLambertiancosinebumps.InthesecondrowweshowrealfacialABRDFsapproximatedbytheTensorSplinesandtheLambertianmodelfromashadowproneregionoftheface.ItcanbereadilynotedthattheTensorSplinescapturethevariabilityinintensityvalues,asafunctionofilluminationdirection,moreaccuratelythantheLambertianreectancemodel. WemustpointoutthatastheorderoftheCartesiantensorsincreases,sodoestheamountofdatasamplesrequiredtoestimatetheunknowncoecients.Whenthereareonlyafewimagesavailable,inordertosatisfyourdesiretousethehigherordertensors, 32

PAGE 33

Imagesrelightedwithcomplexlighting.TherstimageofeachsubjectislitbyapointsourcewhilethenexttwoarelitbyEucalyptusGroveandSt.Peter'sBasilicalightprobesrespectively.Lightprobesareprovidedbelowthefacialimages. wemustchoosebetweenitsodd(anti-symmetric)oreven(symmetric)components.NotethatsincemostofthetimeweareinterestedintheABRDFs'behavioronthefrontalhemisphere,bothsymmetricandanti-symmetricversionsprovidethesamerepresentationpower.Theirbehavioronlybecomespertinentwhentheilluminationdirectionisexactlyperpendiculartotheposedirection,andthisiswheretheuseofanti-symmetricversionsisadvantageous. Thishasbeenexplainedviaa2DexampleinFig. 2-2 ([ 5 ]).Fig. 2-2A showsasemicircularfunctionwherethebluecircleinthegureisconsideredtobethezerovalue.Fig. 2-2B and 2-2D showthesamefunctionapproximatedbyanantipodallysymmetricfunctionandanantipodallyanti-symmetricfunctionrespectively.Incanbenotedthatforboththecasestheapproximationisquiteaccurateexceptneartheangles0and180.Whentheoriginalfunction(Fig. 2-2A )issuchthatithaspositivevalueatoneoftheseantipodalpointsandnearzerovalueattheother,asymmetricfunctionforcesthevalueatbothofthesecrucialanglestobepositivewhiletheanti-symmetricfunctionforcesonetobepositiveandtheothertobenegative.Now,ifweassumethatonlythepositivevaluesofthefunctionarepreservedwegettheresultsaspresentedinFig. 2-2C and 2-2E 33

PAGE 34

2-2A .Thisisbecauseifapixelhashighintensityvaluewhenlitfrom0,mostofthetimeitwouldhavealowintensityvaluewhenlitfrom180(duetoattachedandcastshadows),andviceversa.Thus,ifasymmetricfunctionisusedforapproximatingsuchanABRDF,itwouldcausenon-negativevaluesatboth0and180andwouldleadtovisuallysignicantartifacts(unnaturallighting)intheimages(Fig. 2-2F ).Ontheotherhand,inpractice,useofananti-symmetricfunctiondoesnotcausevisuallysignicantartifacts(Fig. 2-2G ).Tosummarize,eventhoughboth,anti-symmetricandsymmetricfunctions,introduceartifactsnear0and180directions,theartifactscreatedbyananti-symmetricapproximationarevisualinsignicantandhencewehavechosentoworkwiththeanti-symmetriccomponentsoftheCartesiantensors. TwodimensionalTensorSplineswithbi-cubicB-Splinesandoddordertensorscanbewrittenas wherevectorsi;j;D;tandvhavethesamemeaningasbeforeandthetensorhasanoddorder. TheproblemathandisthatgivenasetofQfaceimages(Iq,q=1:::Q)ofasubjectinaxedposealongwiththeassociatedlightingdirectionsvq=(vq1vq2vq3),wewanttoestimatetheABRDFeldofthefaceusingabi-cubicTensorSpline.WeproposetoaccomplishthisbyminimizingthefollowingenergyfunctionwhichminimizestheL2distancebetweenthemodelandthegivendata, 34

PAGE 35

2{7 isdonewithrespecttotheunknowntensorcoecientsTi;j;k;l;mthatcorrespondtothecontroltensorsTi;j(vn). IftheimagesizeisMM,thereareM2unknownABRDFtensorswhichareinterpolatedfromthecontroltensors(Eq. 2{6 ).Weuseauniformgrid,DD,ofcontroltensors,whichtranslatesto3D2,10D2and21D2unknowncontroltensorcoecientsfor1st,3rdand5thordertensorsrespectively.AvalueforDischosenaccordingtothedesiredsmoothness.Forthecaseswhenthenumberofunknownspercontroltensorisonemorethanthenumberofdataconstraints,weuseanadditionalconstraintwhichdiscouragessolutionswithlargenorms.ThisisenforcedbyaddingthetermPijPklmT2ijklmtotheerrorfunctioninEq. 2{7 ,whereistheregularizationconstant. WerecovertheunknownsinEq. 2{7 usingthegradientdescentmethodwiththecontroltensorcoecienteldinitializedusingall-onesunitvectors.Thistechniquecanbeecientlyimplementedbecausewehaveobtainedtheclosedformforthederivativeoftheobjectivefunctionwithrespecttotheunknowncoecientsas Oncethecoecientshavebeenrecovered,imagesunderanovelilluminationdirection,v,canbesynthesizedbyevaluatingtheABRDFeldinthedirectionv,whereeachABRDFisgivenbyEq. 2{5 .PossiblenegativevaluesobtainedinEq. 2{5 aresettozero(asintheLambertianmodel).Furthermore,itshouldbenotedthatthegeneratedimagescanbereadilyup-sampledbyevaluatingEq. 2{5 onamoredensesamplinglatticesincetheTensorSplineisacontinuousfunction. 35

PAGE 36

wherevisthelightingdirectionandthevectorsiareuniformlydistributedontheunitsphere. Ofthevariouschoicesforsingledlobedsphericalfunctionsthatcanbeusedasthekernelfunctionk(;v),wepickedk(;v)=ev1duetotworeasons{ithasasinglepeakandk(;v)=0forallvsuchthatv=0(sinceiftheviewingandtheilluminationdirectionsareperpendicularweexpectzerointensity).NotethatthesetwopropertiesarealsosatisedbytheLambertiankernel. ThetaskofestimatingABRDFsusingthismixturemodelrequiresustorecovertheunknownweightssuchthattheweightedcombinationleadstoasphericalfunctionwhichcloselyapproximatestheABRDFs.GivenasetofNfacialimageswiththesamexedposeandtheassociatedlightingdirectionsvn,wecansetupaNMmatrixAn;mbyevaluatingev1foreveryvnandi.Misthenumberofipickedinthemodel.Theunknownweights(Eq. 2{9 )foreachpixelcanthenbeestimatedbysolvingtheover-determinedsystemAW=B,whereBisaN-dimensionalvectoroftheintensitiesataxedpixelintheNgivenimages,andWisthevectoroftheunknownweights.SincetheABRDFisanonnegativefunction,wesolvethissystemwiththepositivityconstraintusingthenon-negativeleastsquareminimizationalgorithmdevelopedin[ 49 ]. 36

PAGE 37

2 ]),Phong([ 1 ])etc.,whichexplicitlyassumearoleforthesurfacenormalintheirformulae,TensorSplinesmakenosuchassumption.ThisallowsspatiallyvaryingandaccurateapproximationoftheABRDFs,butalsomakestherecoveryofthesurfacenormalsnon-trivial. TorecoverthesurfacenormalsfromtheTensorSplinesmodelweinvokethelocalhomogeneityoftheABRDFeld.Thisassumptionisphysicallysoundbecausethereectancepropertiesofahumanfacedoesnotchangedrasticallyinsmallneighborhoods(33pixels)andmathematicallyrobustasTensorSplinesmodelensuresthatthecoecientsvarysmoothlyacrosstheABRDFlattice.WeassumethattheABRDFsattwoneighboringpixelshavethesameshapeanddieronlybyarotation,Randthus,ifthesurfacenormalatoneofthesepixelsisknown,thesurfacenormalattheotherpixelcanbederivedbyrotatingitbyR. Foragiveninternalpixel(x;y)intheimage,thereareeightimmediateneighbors.Ifthesurfacenormalat(x;y)isinferredasdescribedabove,itwouldreceiveeightsuggestionsforpossiblesurfacenormals(assumingthatthesurfacenormalsfortheneighborsareknown).Insteadofpickingoneofthesuggestionsasitssurfacenormal,wetakeaweightedgeodesicaverageofthesuggestednormals.Theweightsaresettobeinverselyproportionaltotheregistrationerrorobtainedduringtherotation-alignmentoftheABRDFpairs.Therearetwomainadvantagesofcomputingthesurfacenormalisthis 37

PAGE 38

ThisprocessissummarizedinFig. 2-3 ([ 5 ]),wherethecentralABRDF,(A),isshowntobealignedwithahigheraccuracytoitsleftneighbor,ARBDF(B),thantoitsrightneighbor,ABRDF(C).Forbothcase,beforeandafteralignmentcongurationsareshownfrom2dierentpointsofviews.Asmentionedbefore,themisalignmenterrorisusedtoweighthenormalsuggestionfromaneighborandhencethesuggestionfromtheleftABRDF,(B),wouldeventuallybeweightedmorethanthesuggestionfromtheABRDFontheright,(C). Oncetherotationmatricesforallthepixelsintheimagehavebeencomputed,weinitializeallthenormalswiththedirectionsinwhichtheABRDFshavetheirmaxima.Thisinitializationisfollowedbyweightedgeodesicmeancomputations,whichprovideuswitharobustestimateofthesurfacenormals.Theprocessofmeancomputationiscarriedoutiterativelybutempiricallyitwasnoticedthatgoodresultscanbeobtainedinallcaseswith1or2iterations.Notethatusingthemaximadirectlyasanormalestimateprovidesinaccurateresults.Weattributethistothefactthatunlikesomereectancemodels(e.g.Lambertian),TensorSplinesmodeldoesnotenforcethatthemaximalresponseoftheABRDFliesalongthesurfacenormaldirection. 38

PAGE 39

The1stimageisgeneratedusingtheTensorSplinesmodel,the2ndimageisthegroundtruthandthe3rdisgeneratedusingtheLambertianmodel.ThecastshadowsandthespecularitiesaremuchmorerealisticallyrenderusingtheTensorSplinesmodelthantheLambertianmodel. accordingtothisscheme,twoABRDFsrepresentedbytheirCartesiantensorcoecientsw1andw2,canbealignedbyminimizingthefollowingobjectivefunction suchthat wheretheunitvectorsvareobtainedbysomeuniformsamplingofthesphere,BisthevectorofCartesiantensorbasisdenedinEq. 2{1 andRisthesoughtrotationmatrix.Thismethodfortherotationmatrixrecoverywouldrequirenonlinearoptimizationtoberun8L2timesforanimageofsizeLLpixels.Evenforanaveragesizedimagethisprocesscanbequiteintractableandhence,weproposethefollowingmoreecientalgorithmfortherotationmatrixrecovery. LetT1(v)andT2(v)bethetwoABRDFs(Eq. 2{1 )thatneedtobealignedviaarotation.Thisimpliesthatweseekavsuchthat SincetheABRDFsarefromneighboringpixels,weassumethattherequiredvwouldbesmallandthususingtherstorderTaylor'sexpansion,weget 39

PAGE 40

B C D Distributionoferrorsasthecongurationofinputchanges.X-axisrepresentstheazimuthandY-axisrepresentstheelevationangles.Hottercolorsshowlargererrors.Thewhitedotsrepresenttheexactdirectionsofilluminationinimagesusedasinput. Asweexpect whereListhelineartransformationcontainingtherotationmatrix,weget whichleadstothelinearsystem wheretheithrowofAcontainsvectorizedentriesofrT2(vi)viT,xcontainsthevectorizedentriesofL,theithentryofBisT1(vi)T2(vi)+rT2(vi)Tviandviaretheunitvectorsobtainedfromuniformsamplingofasphere.TheembeddedrotationmatrixRcanberecoveredusingtheQRdecompositionfromL. 40

PAGE 41

whered()isthegeodesicdistancedenedonthespaceofunitnormals,thearclength.WeseekageodesicmeanbecausethedomainofunitnormalsistheunitsphereandnottheEuclideanspace.ThismeanisalsoknownastheweightedKarchermeanandcanbecomputedusingthefollowingiterativescheme{ whereexp,theexponentialmap,isgivenas andexp1(np),thelogmap,isdenedas where andistheiterationstepsize.Formoredetailsoncomputingmeansonmanifoldssee[ 50 ]andreferencestherein. 51 ])torecoverthesurface.Ifz(x;y)denesthesurface,thenormalatalocation(x;y)isgivenby(zxzy1)Twherezxandzydenotethepartialderivativesofthesurfacewithrespecttoxandy.If(nxnynz)Tdenotesthesurfacenormalatlocation(x;y),wehave 41

PAGE 42

Perpixelintensityerrorcomparison.Thebluecolorshowstheerrorsonthe1stand2ndsubsetscombined,whichcontainthelightingdirections,,smallerthan25,thegreencolorshowstheerrorsonthe3rdsubsetwith25<<50andtheredcolorshowstheerrorsonthe4thsubsetwith50<<70. thefollowingrelations Usingtheforwarddierenceapproximationofthepartialderivatives,weobtainthefollowingtwoequations whichprovidealinearrelationbetweenthesurfacevaluesatthegridpointsandtheknownsurfacenormals.Thesurfacecanthusberecoveredbysolvinganover-determinedsystemoflinearequations.Attheboundarypoints,theaboveformulationisnotvalidandthesurfaceisrecoveredbysolvingthefollowingequation,obtainedbyeliminatingnzabove 42

PAGE 43

BQuiverplotofthenormaleld CColorencodednormals(R-x,G-z,B-y) DRecoveredshapeinnovelpose E9inputimages FQuiverplotofthenormaleld GColorencodednormals(R-x,G-z,B-y) HRecoveredshapeinnovelpose Shapesrecoveredusing9images

PAGE 44

Wewouldliketopointoutthatthespecularitiesareviewdependentandaccuratelyspeaking,cannotbedirectlytransferredfromoneposetoanother.MostoftheexistingLambertianmethodsignorethiseectbutthefewwhichdealwiththisproblem,handleitbyeitherexplicitlyobtainingthespecularcomponentbyusingpolarizedlighting(e.g.[ 26 ],[ 8 ]),whichrequiredspecializeddataacquisition,orbyassumingaparametricformforthespecularcomponentoflighting(e.g.[ 20 ]). OurCartesiantensorrepresentationfortheABRDFsdoesnotdiscriminateagainstspecularitiesandestimatestheABRDFasbestaspossiblefromtheavailableintensityvalues.Thus,itshouldbepossibletorecoverandmanipulatethespecularcomponentseparately,butatthisstage,wehavemadetheassumptionthatspecularitiesdonotchangedrasticallyacrossfacialposes.Thevalidityofthisassumptionissupportedbytheresultspresentedinthenextsection. 7 ],[ 9 ]),wehavetakenonthechallengeofalsoworkingwithjust9image.Notethatwith9samplesoftheABRDFeldandthesolutionnormminimizationconstraint,atmost10unknowncoecientscanberecoveredperpixelandhenceourcentralresultsusebi-cubic3rdorderTensorSplines. 44

PAGE 45

37 ](38subjects,in9posesand64illuminationconditions)andtheCMUPIE[ 52 ](68subjectsin13posesand43illuminationconditions)benchmarkdatabases.NotethattheCMUPIEhas21usablepointsourceilluminatedimageswhileintheExtendedYaleBall64illuminationsarepointsource. 2-4 ([ 5 ])weshowtheABRDFeldofasubjectfromtheExtendedYaleBdatabaseestimatedusing9images.ThreedierentregionsofthefacehavebeenshownindetailwherecomplicatedshapesoftheABRDFcanbenoticed.TheregionsAandBhavemorecomplicatedshapesbecausetheseABRDFshavetoaccommodateshadows.Thesphericalfunctionsintheimagehavebeencolorcodedbasedontheirmaximalvaluedirections.Themappingofthedirectionstocolorsisprovidedinthelowerrightcorner. Nextwepresentresultsforrelightingoffacesinnovelilluminationdirections.InFig. 2-5 ([ 5 ])fourdierentsubjectslitinvariousnovelpointsourceilluminationsaredepicted.Forthersttworows,theilluminationdirectionvariesacrosstheazimuthanglewhileinthenexttworows,thevariationisintheelevationangle.Itcanbenoticedthatourmethodcanaccuratelyinterpolateaswellasextrapolatefromtheimagesprovidedasinput.Further,diculteectslikethecastshadowsandthespecularitieshavebeenphoto-realisticallyrenderedwithoutusinganyadditionalraytracing. Startingwith9images,ourtechniqueestimatestheentireABRDFeldandthusimageslitinfairlycomplexlightingconditionscanalsoberendered.InFig. 2-6 ([ 5 ])wepresentsuchresultsfortwosubjectsfromtheCMUPIEdatabase.Beloweachimageisitslightingcondition.TherstimageofeachsubjectisoneofthenineinputimagesusedtoestimatetheirABRDFelds.Thenexttwoimagesforeachofthesubjectsarelitbythelightprobes([ 53 ])namedEucalyptusGroveandSt.Peter'sBasilicarespectively.Forthese 45

PAGE 46

InFig. 2-7 ([ 5 ])weprovideaqualitativecomparisonbetweenourmethodandtheLambertianmodel.TherstfaceimageinFig. 2-7 ([ 5 ])isrenderedusingtheTensorSplinesmodel,thesecondisthegroundtruthandthethirdimageisrenderedusingtheLambertianmodel.ItcanbereadilynotedthattheimageobtainedusingourmethodisclosertothegroundtruththantheonerenderedusingtheLambertianmodel.Thearrowsshowthelocationsofimportantdierences{thecastshadowsandthespecularities. Thenexttwoexperimentstrytoquantitativelycapturetheperformanceofourmethod.First,weexploretheimpactoftheinputdataontheestimationwhenthetensororderisxed(3rdorderinthiscase).ForthisweusetheExtendedYaleBdatasetasitprovidesthegroundtruthfor64directions.Tosetabaseline,weestimatedtheABRDFeldfor10subjectsusingallthe64imagesasinput,renderedimagesinthesame64directionandcomputedthetotalerrorwithrespecttothegroundtruthimages.Next,theerrorswerecomputedsimilarlyfor3othercaseswhereonly9imageswereusedasinputtoourmethod,butindierentcongurations.Twoofthecaseshadimageswithilluminationdirectionsuniformlydistributedinfrontofthefacewhileonehadimageswiththedirectionsbiasedtowardsoneside. Tovisualizethedistributionsofobtainederrors,wecolorcodedthemwithhottercolorsdenotinglargererrorsandplottedthemasacontinuousimagesinFig. 2-8 ([ 5 ]).TheXaxesoftheseimagesshowtheazimuthanglesvaryingfrom130to130(fromtheleftmostwhitedottotherightmost)andtheYaxesshowtheelevationanglesvaryingform40to90(fromthetopmostwhitedottothebottommost).Thewhitedotsintheseimagesshowtheexactdirectionofilluminationintheimagesusedasinput.Itcanbereadilynotedthatwhenallthe64imagesareusedasinput,Fig. 2-8A ,theerroristhe 46

PAGE 47

Detailedposevariationwithtexture-lessinupperrightanddepth-mapinlowerright.Accuraterenderingseveninextremeposescanbenoticed. least.Forthe9imagecases,Fig. 2-8B andFig. 2-8C ,wheretheilluminationdirectionsoftheinputimagesaresomewhatuniformlydistributed,theerrorismorethanthatinFig. 2-8A ,butnotedlylessthanthecasewhenthedistributionisskewedinonedirection,Fig. 2-8D .Hence,asexpected,ourmethodperformsbetterwhentheinputimageshavethelightingdirectionsthatareuniformlysampledfromthesphere.Moreover,theerrorsinallthecasesareconcentratedtowardstheextremeilluminationanglesandforthenearfrontalilluminationconditions,theperformanceisnotparticularlyaectedbytheinputimagedistribution. NextwepresentaquantitativecomparisonofourmethodwiththeLambertianmodelandthevalidationmodelpresentedinSection 2.5 .Anaturalquestionthatarisesiswhyshouldanorder3CartesiantensorbesuitableforestimatingthefacialABRDFs?Toanswerthisquestion,wecomputedtheaverageintensityerrorperpixeloverall38subjectsinthe64illuminationdirectionsoftheExtendedYaleBdatasetusingtheLambertianmodel,3rdorderTensorSplines,5thorderTensorSplinesandthemixtureofsinglelobedfunctions(Eq. 2{9 ).All64illuminationdirectionswereusedforthemixture 47

PAGE 48

2-8B )wereused.Wesettheivaluesrequiredforthemixturemodelusingadensesampling(642directions)oftheunitsphereobtainedbythe4th-ordertessellationoftheicosahedron.WehavepresentedresultsinFig. 2-9 ([ 5 ])shatteredalongthestandardsubsets(subset4inred,3ingreenand1+2inblue)oftheExtendedYaleBdatabase.Asexpected,theerrorforthesubsetwithextremelighting(subset4)ismorethantheothersets,forallmethods.Moreimportantly,evenwithaconsiderablylargeamountofinputdataandaveryexibleestimationmodel,theerrorsobtainedfromthemixturemodelarequitesimilartothoseobtainedfromthe3rdorderTensorSplinesmodel.Thisindicatesthatthougha3rdorderTensorSplinesmodelcanonlyaccommodatethreelobes,formostfacialABRDFsthissuces.The3rdorderTensorSplinesmodeloutperformstheLambertianmodelandeventhe5thorderTensorSplinesmodel,whichsuggestspossibleover-ttinginthe5thordermodel. 2.6 wecansimultaneouslyvarytheilluminationandtheposeofaface.Fig. 2-10 ([ 5 ])summarizestheresultsproducedbyourshaperecoveryalgorithmforonesubjecteachfromtheExtendedYaleBandtheCMUPIEdatabases.Therstcolumnshowsthe9inputimages,thesecondcolumnshowsthequiverplotoftheestimatednormaleld(zoomintoseedetails),thethirdcolumnpresentsthesurfacenormalinformationinacolorcodedform(xcomponentsofthenormaleldaremappedtotheredchannel,ycomponentstothebluechannelandzcomponentstothegreenchannel)andthefourthcolumnshowstherecoveredshapeinanovelpose.Forthecaseofcolorimages,shapeestimationwascarriedoutusingonlytheluminancecomponent.Inboththecases,occlusionofappropriateregionsofthefaceduetoposechangecanbenotedfromtheimagesinthefourthcolumn. 48

PAGE 49

ShapecomparisonwiththeRobustPhotometricStereoontheExtendedYaleBdataset InFig. 2-11 ([ 5 ])wepresentmoredetailedresultsforposevariationwithxedillumination.The3rowsofimagesshowasubjectfromtheExtendedYaleBindierentposesrangingfromtherightproletotheleftprole,aswegofromlefttoright,andviewpointvaryingfrombelowthefacetoabovetheface,aswegofromtoptobottom.NotethattheABRDFeldforthissubjectwasrecoveredusingjust9imagesundertheilluminationcongurationshowninFig. 2-8B .Therecoveredshapeforthesamesubject,renderedwithconstantalbedoandspecularities,isalsopresentedattherightendofthegure.Thisallowsnerdetailsoftheshapetobeshownwithoutanytexturetobiastheobserver.Finally,tothelowerrightofthegureistheheightmapforthesamesubject.ItcanbenotedthatourshaperecoveryalgorithmcanproducegoodresultswithoutmakingthesimplifyingLambertianassumption. WepresentfaceshapesestimatedbyourmethodandtheRobustPhotometricStereo[ 54 ](9inputimagesforbothmethods)inFig. 2-12 ([ 5 ]).Resultsforfour 49

PAGE 50

Finally,wepresentresultswhenboththeposeandtheilluminationconditionsaresimultaneouslyvaried.InFig. 2-13 ([ 5 ])onesubjecteachfromtheCMUPIEandtheExtendedYaleBdatabasesareshowninvariousposesandilluminationconditions.TheABRDFeldsforboththecaseswererecoveredusing9images,andtheshapeforthecolorimageswererecoveredusingtheluminancechannel.Withthechangeofpose,wehaveretainedtheABRDFeldlearntusingthefrontalpose,butitcannotedthattheresultsarephoto-realisticevenwhenthespecularitiesarenotexplicitlymodiedandtransferred.

PAGE 51

Simultaneousposeandilluminationvariation 5 andChapter 6 Inrecenttimes,illuminationinvariantfacerecognitionhasattractedparticularinterestduetoadvancesinourunderstandingofthereectancemodeling.Herewepresentacomparativestudyofilluminationinvariantfacerecognition.WhenusingtheTensorSplinesmethod,weassumethatforeachsubject9galleryimageswithknownilluminationdirectionsareavailable.FromtheseimageswecomputetheABRDFeldsandgenerateimageswithnovelilluminationforadensesamplingofdirections.Thisstepexpandsourcollectionof9galleryimagestoanydesiredsize.Theprobeimageisthenmatchedtoall 51

PAGE 52

Facerecognitionerrorsrates.Nisthegallerysetsize. Method N Subset1&2 Subset3 Subset4 Total Correlation[ 55 ] 4 0.0 23.3 73.6 29.1 Eigenfaces[ 56 ] 6 0.0 25.8 75.7 30.4 Linearsubspace[ 57 ] 7 0.0 0.0 15.0 4.7 Cones-attached[ 12 ] 7 0.0 0.0 8.6 2.7 Cones-cast[ 12 ] 7 0.0 0.0 0.0 0.0 9PL[ 37 ] 9 0.0 0.0 2.8 0.8 3DSH[ 11 ] 1 0.0 0.0 2.8 0.8 Harmonic(SFS)[ 58 ] 1 0.0 0.0 12.8 4.0 0.0 0.0 1.6 0.5 WehaveusedtheExtendedYaleBdataforthisexperimentprimarilybecausemostoftheexistingmethodshavepresentedfacerecognitionresultsonthesamedatabase.Asmentionedbefore,thisdatabaseisdividedinto4subsetswiththelightinggettingmoreandmoreextremeaswegofromsubset1to4andthus,thedicultyinclassifyingtheimagesfromthesesubsetsalsoincreases.TheobtainedrecognitionerrorratesarereportedinTable 3-1 .Wehavealsopresentedresultsreportedbyexistingmethodsandtheirrespectivereferencesarelistednexttothenameofthemethod.Resultsfortherstseventechniquesaretakenfrom[ 11 ]andtherestaretakenfromtherespectivereferences.Alongwiththeerrorrates,wehavealsolistedthenumberofimagesrequiredbyeachmethodinthegalleryset.ForourmethodweusedthenineimagesinthecongurationshowninFig. 2-8B ([ 5 ]).Itcanbenotedthatevenwiththenaivenearestneighborclassicationstrategyourmethodproducesnearperfectresults. 52

PAGE 53

15 ])suerfromtheneedofmanualinterventionandcumbersomecomputationalrequirements.WewouldexploretheuseofTensorSplinesABRDFeldsaspriorinformationtomeaningfullypredicttheABRDFeldsusingsingleinputimagesinChapter 4 .Useofashapepriorcanalsopotentiallyaidinshaperecovery. Whilerelightingimagesinnovelposes,wemaketheassumptionthattheABRDFeldmaintainsthesamespecularinformationacrossposes.Thoughpracticallyuseful,thisisnotfullyvalid.Wehavedealtwiththespecularitiesinadatadrivenfashionbutpossibleattemptcanbemadetoexplicitlymodelthespecularities,whichwewouldliketoexploreinfuture.Itshouldbenotedthatthoughtheproblemofdetectingspecularitiesisrelativelywellstudied,theproblemofrealisticallypredictingspecularitiesinnovelposeswithoutusingspecializedimagingtricks(likespeciallters)remainschallenging.Possibleimprovementcanalsobemadeinourmodelbyincorporatingnon-uniformsmoothnessasopposedtothecurrentsetup. Besidestherelightingandtheposechangeapplicationsdescribedinthechapter,ourtechniquecanalsobeusedforimageup-samplingandcompression.TheformerispossiblebecausetheTensorSplinesrepresentationcreatesacontinuouseldoftheABRDFcoecientsacrosstheimage,whichcanbesampledatasub-pixelresolution.ThelaterexploitsthecapabilityoftheABRDFstorepresentimagesofafaceunderinnitelymanylightingdirectionsusingjustafewcoecientsperpixel. Inconclusion,theTensorSplinesframeworkfortheanalysisandmodelingoftheilluminationandtheposevariationoffacialimagesprovidesausefulalternativetotheLambertianassumption.Italsoseemsthatthecollectiveanalysisoftheshapeandthe 53

PAGE 54

54

PAGE 55

59 ].ThebasiselementsofsuchasubspacearethemselvessphericalfunctionsandwecallthemEigenbubbles. TheonlychangethatwemaketotheABRDFrepresentationfromChapter 2 istoreplacetheCartesiantensorsintheTensorSplinesrepresentationofthesphericalfunctionswiththeSphericalHarmonicsbasis.Wedenotetherealsphericalharmonicbasisfunctionsasml(order:l,degree:m),withl=0;1;2;:::andlml: ml(;)=s 4(lm)! (l+m)!Pljmj(;)m(;):(3{1) wherePljmjaretheassociatedLegendrefunctionsandm(;)isdenedas m(;)=8>>>><>>>>:p 55

PAGE 56

Now,givenabagofABRDFsfig(whereiaretheSphericalHarmoniccoecientsoftheABRDFs),wedenethemeanABRDFas ^=Xii=N:(3{3) Thedatacovariancematrixcanthusbedenedas NotethateventhoughthenumberofABRDFscanbelarge,thecovariancematrixhasdimensions1010forthethirdorderABRDFestimation.NextwedecomposethesquarematrixCintoitseigenvectors(V)andeigenvalues(U)as Arrangedintheincreasingorderofcorrespondingeigenvalues,theeigenvectors-inourcaseEigenbubbles-denealowvariancesubspacefortheABRDFrepresentation. AtthispointwedenetwokindsofEigenbubbles-LocalandGlobal.GlobalEigenbubblesaredenedtobetheonesobtainedbyputtingtheABRDFsfromalldierentlocationsanddierentindividualsintotheinitialbagofABRDFs.Ontheotherhand,LocalEigenbubblesarethoseobtainedseparatelyforeachpixel,byconsideringonlytheABRDFslyingatthesame(corresponding)pixellocations,fromallthegivenfaces.Forthisweonlyrequirearoughalignmentofthefaces(e.g.alignmentofeyes)sinceweassumethattheABRDFsinthesameneighborhoodsaresimilar.Thesetwo 56

PAGE 57

GlobalEigenbubbleslearntfromtheExtendedYaleBfacedatabase.Eigenvaluesincreasefromlefttorightinrowoneandtheninrowtwo. denitionswouldallowustoanalyzetheimpactoftheuseofEigenbubblefortheABRDFrepresentationinthenextsection. 3.3.1Relighting 3-1 therst10GlobalEigenbubbleslearntfromtheExtendedYaleBfacedatabaseareshown.ItcanbenotedthattheEigenbubblescorrespondingtolowereigenvaluesconsistofasmallernumberofbluntlobeswhiletheEigenbubblescorrespondingtohighereigenvaluesshowalargernumberofshaperpeaks.ThisindicatesthattheEigenbubblescorrespondingtohighereigenvaluesencodehigherfrequencydetailsofthefacialABRDFelds.Asmentionedbefore,thesignicanceoftheseEigenbubblesliesinthefactthatanABRDFfunctionatanylocationonanyfacecanberepresentedwithhighaccuracyasalinearcombinationofthesesphericalfunctions.Next,wepresenttherst10LocalEigenbubblesinFig. 3-2 ,attwodierentlocationsonahumanface.ItcanbenotedthatshapesofthesetwosetsofsphericalfunctionsarequietdierentformeachotherastheyarelearntusingtheABRDFfunctionsfromtwodierentregionsoftheface.Foreheadshowsmuchmoreuniformvariationinintensityvaluesascomparedtotheshadowinfestednasalregion. NextwelookattheactualABRDFfunctionswhicharerepresentedusingtheEigenbubbleframework.InFig. 3-3 theestimatedABRDFeldhasbeenshownsuperimposedonaface.LargerimagesofindividualABRDFsrepresentedusing 57

PAGE 58

LocalEigenbubbleslearntfromtwodierentlocationsonaface.Eigenvaluesincreasefromlefttorightinrowoneandtheninrowtwo. Figure3-3. EstimatedABRDFfunctionsatvariouslocationsonahumanface. Eigenbubblesshowthathighdelityrepresentationofintensityvariationatpixelswhichaccountsforthecastshadowsandthespecularitiesrequiremorecomplicatedsphericalfunctionsthanahalf-cosinebumpusedintheLambertianmodel. InordertoanalyzeandunderstandtherepresentationalpoweroftheGlobalandtheLocalEigenbubbles,inthenextsetofexperimentswequalitativelyandquantitativelyexaminethequalityofthesynthesizedrelightedimagesusingtheproposedtechniques. 58

PAGE 59

Ontheleft(5columns)synthesizedimagesusingtheGlobalEigenbubblesareshown.Fromtoptobottom,eachrowuses1,2,3,5and10EigenbubblesrespectivelyforitsABRDFeldrepresentation.ThesamesetupisrepeatedontherightusingtheLocalEigenbubbles.Theazimuthandtheelevationanglesoftheilluminationsourcearegivenatthebottomrightcornerofeachimage.AllABRDFeldswereestimatedusing9inputimagesilluminatedfrom(-20,60),(0,45),(20,60),(-50,0),(0,0),(50,0),(-50,-40),(0,-35)and(50,40)directions. FirstwepresentsynthesizedimagesofasubjectfromtheExtendedYaleBdatasetwhileusingvaryingnumberofsubspacedimensionsforitsABRDFeldrepresentationinFig. 3-4 .Therst5columnsontheleftshowimagessynthesizedusingtheGlobalEigenbubbleswhilethelastvecolumnsshowimagessynthesizedusingtheLocalEigenbubbles.FromtoptobottomeachrowshowsimagesgeneratedfromtheABRDFeldrepresentedusing1,2,3,5and10eigenbubblesrespectively.AlltheABRDFeldswereestimatedusing9inputimages.Theseimageshavebeensynthesizedusingnovelilluminationdirectionswhoseazimuthalandelevationanglesarementionedinthebottomrightcornerofeachimage.Fromtheseimages,foremost,itcanbenotedthatasthesubspacedimensionisincreased,sodoesthevisualqualityoftheimages.Secondly,qualityoftheimagesproducedusingtheLocalEigenbubbles,especiallyforlowdimensionalsubspaces,isbetterthanwhentheGlobalEigenbubblesareused.Thirdly,forbothtypes 59

PAGE 60

TodemonstratetheversatilityoftheGlobalEigenbubblesforaccuratelyrepresentingthefacialABRDFacrossdierentfacetypesanddatabases,wepresentnovelimagesofvariousfacesrenderedusingourtechniqueinFig. 3-5 .Whenusedforeachchannelindependently,theABRDFframeworkproposedherecanbeeasilyextendedtocolorimages,andthisisdemonstratedintherstfourrowswheresubjectsfromtheCMUPIEdatasetarerenderundernovelilluminationconditions.GlobalEigenbubbleswerelearntforeachchannelseparately.ThebottomtworowsshowimagesoftwosubjectsfromtheExtendedYaleBdatabase.TheGlobaleigenbubblesusedherewerethesameasusedinFig. 3-4 .Usingfacesbelongingtodierentraces,wehavedemonstratedthecapabilityoftheGlobalEigenbubblestorepresenttheABRDFsofsurfaceswhichcanhavesomewhatdierentsurfaceproperties.Notethattheshadowshavebeencrisplygeneratedandthespecularitieshavebeenmeaningfullyrendered. Nextwequantitativelyexaminetheimagesgeneratedusingourmethod.ForthisexperimentwehavechosentheExtendedYaleBdatasetasitprovidesimagesofeachsubjecttakenin64pointilluminationconditionswhichcanbeusedastheground-truthforevaluatingourmethod.BesidestheproposedenhancedABRDFrepresentationmethodwehavealsoincludedresultsfromtheTensorSplinesmethodandtheLambertianmodel,examinedin[ 60 ],inordertocompareourresultswiththesealternatemethods.Using9imagespersubject,wesynthesizedimagesinall64illuminationdirectionsusingjusttheSplineModulatedSphericalHarmonics,theLocalandGlobalEigenbubbles,theTensorSplinesmodelandtheLambertianmodelandthencomputedthepixel-wiseintensityerrorsforeachusingtheground-truthimages.WehavepresentedtheseresultsinFig. 3-6 .ItcanbenotedthattheLambertianmodel,withitslimitedrepresentativepowerinthepresenceofthecastshadowsandthespecularitiesperformstheworst.Asexpected,the 60

PAGE 61

TherstfourrowsshowcolorimagesofsubjectsfromtheCMUPIEdatabase,whilethelasttworowsshowimagesofsubjectsfromtheExtendedYaleBdataset.The(azimuth,elevation)anglesoftheilluminationsourcearementionedinthebottomrightcornerofeachimage.InputimagesfortheCMUPIEsubjectswerelitfrom(32,2),(0,-9),(32,-9),(0,3),(-32,-8),(-32,3),(38,6),(0,10),(-32,5)directionswhilefortheExtendedYaleBsubjectsweusedsamedirectionsasinFig. 3-4 61

PAGE 62

Perpixelintensityerrorsobtainedusingtheproposedandthestate-of-the-artmethods.TheY-axisrepresentstheerrorwhiletheX-axisrepresentsthedimensionofthesubspaceusedintheEigenbubblerepresentation.NotethattheerrorratesdonotvaryalongtheX-axisfortheSplineModulatedSphericalHarmonics,theTensorSplinesmodelandtheLambertianmodel. Figure3-7. Qualitativecomparisonbetweentheproposedtechnique,theTensorSplinesmodelandtheLambertianmodel[ 60 ]. LocalEigenbubbleoutperformstheGlobalEigenbubbles,butbothofthesetechniques,outperformtheTensorSplinesmodel.Theimprovementintheimagequalityasthenumberofsubspacedimensionsisincreased,asapparentintheimagespresentedinFig. 3-4 ,canbeclearlynotedinFig. 3-6 Theobservedsuperiorityoftheproposedtechniqueoverthestate-of-the-artmethodsintermsofrelightingwithfaithfulshadowsandspecularitiesreproductioncanbevisuallynotedinFig. 3-7 .Herewehaveshownarepresentativeimagerenderedusingvarious 62

PAGE 63

Novelimagesgeneratedunderextremeilluminationdirections.Thecentralgureshowstheilluminationdirectionsoftheinputimagesasbluecirclesandtheilluminationdirectionsofthenovelimagesasredsquares. methodsalongwiththegroundtruthimage.ItcanbenotedthattheLambertianmodellargelyfailstomodelthespecularities(greenarrow)andthecastshadows(redarrows).TheTensorSplinesmodeldoesabetterjobthantheLambertianmodelbuttheshadowsandspecularitiesaresmudgedonaccountofitsassumedacross-eld-smoothness.Ourmethod,Eigenbubbles,ontheotherhand,hasproducedresultswhichhaveaccuratelydepictedthecastshadowsandthespecularities.HerewehaveusedtheLocalEigenbubbleswhichdonotsuerfromthesmoothingartifactspresentintheTensorSplinesmodel. 63

PAGE 64

N Set1&2 Set3 Set4 Total 55 ] 4 0.0 23.3 73.6 29.1 56 ] 6 0.0 25.8 75.7 30.4 57 ] 7 0.0 0.0 15.0 4.7 12 ] 7 0.0 0.0 8.6 2.7 12 ] 7 0.0 0.0 0.0 0.0 37 ] 9 0.0 0.0 2.8 0.8 11 ] 1 0.0 0.0 2.8 0.8 58 ] 1 0.0 0.0 12.8 4.0 0.0 0.0 1.6 0.5 0.0 0.0 0.9 0.29 Facerecognitionerrorsrates.Nisthenumberofinputimages. subject,9imagesareusedasthegallerysetwhiletherestareusedasprobes.UsingtheGlobalEigenbubbles,wegeneratealotmoreimagesofeachsubjectbyuniformlysamplingtheilluminationdirectionsphere.ThenalclassicationisperformedusingtheNearestNeighborclassier.Theresultsobtained(shatteredalongthesubsets)arepresentedintheTable. 3-1 .Theilluminationconditionsgetharsherfromsubset1to4andsodoestherecognitiontask.Itcanbenotedthatourmethodproducesextremelysmallerrorratescomparabletothestate-of-the-art.Wemustemphasizethatingeneral,facerecognitionsystemsusemoresophisticatedclassiersthanthesimpleNearestNeighborclassierusedbyusandthusinsteadofafacerecognitionsystem,ourmethodshouldbeseenasadatabaseaugmentationmethodwhichcanaidanyotherfaceclassicationmethodbyaddingmeaningfulgalleryimagestothetrainingset. 3-6 )thattheGlobalEigenbubblebasedrepresentationoftheABRDFcangeneratehighqualityimagesevenwhenonly5dimensionalsubspaceisusedtorepresenttheABRDFs.Intermsofspacerequirement,thistranslatestostoringonly5coecientsperpixelandthe5GlobalEigenbubbles,inordertogenerateimagesunderanyarbitraryilluminationcondition.Ifweconsideronlythe64pointsourcelitimagespresentintheExtendedYaleBdatabase,ittakesabout2MBofmemory 64

PAGE 65

Raw Eigenbubbles ReductionRatio Error/Pixel 1 2016KB 127.6KB 6.25% 18.3 2 2016KB 253.6KB 12.50% 15.0 3 2016KB 379.6KB 18.75% 13.8 4 2016KB 505.6KB 25.00% 13.9 5 2016KB 631.6KB 31.24% 13.8 6 2016KB 757.6KB 37.51% 14.2 7 2016KB 883.6KB 43.76% 13.3 8 2016KB 1009.6KB 50.01% 13.2 9 2016KB 1135.6KB 56.26% 13.1 10 2016KB 1261.6KB 62.51% 12.9 ABRDFeldcompression.Disthesubspacedimensionandtheerrorsaretheintensityerrorsperpixel. tostoretheraw192168imagesforeachsubject,whileusingEigenbubblesbasedrepresentationwith5coecients,itwouldtakelessthanonethirdofthatmemorytosummarizealltheABRDFs.Thisadvantageincreasesifweconsidermorethan64imagessincetheEigenbubblebasedABRDFrepresentationcangenerateasmanyimagesasdesired.ItmustbenotedthatourtechniquecannotbedirectlycomparedwiththeimagecompressionmethodsastheABRDFeldscangenerateasmanyimagesasdesired,whiletheimagecompressiontechniquescannot.DetailedresultsforvarioussubspacedimensionsarepresentedinTable 3-2 65

PAGE 66

2 asolutiontothefacialrelightingandposechangeproblemwithmultipleinputimageswaspresented,andinthischapterwewillexaminearelatedbutrelativelyharderproblemofrelightingandposechangeoffacesusingsingleimageasinput.Mostoftheapplicationsassociatedwithmultipleimagerelightingandposechange,likethepost-productionvideosceneeditingorfacerecognition,naturallyextendtothecasewithsingleinputimagealso. IntermsoftheABRDFeldestimation,thefundamentaldierencebetweenthecaseswithmultipleandsingleinputimageliesintheamountofface-specicbackgroundknowledgerequired.Inthecaseofmultipleinputimages,sincetherearemultiplesamplesoftheABRDFsavailable,interpolationandextrapolationusingtheappropriatemathematicalfunctionscanprovidegoodapproximationstotheunderlyingABRDFeld.Butinthecaseofsingleinputimage,onlyonesampleoftheABRDFateachimagepixelisprovided,andhenceitbecomesimperativetousebackgroundknowledgeaboutthefacialABRDFsinordertohallucinatethecompleteABRDF. Theabovementionedbackgroundknowledgecanbederivedfromanumberofsources.OnepossibleoptionistousethesamplevaluesattheneighboringpixelsinordertobetterestimatetheABRDFatagivenpixel.AnotherapproachcanexploittheknownABRDFeldofsomereferencefacetogeneratetheABRDFeldofthegiveninputimage.ThisideacanbefurtherextendedtousemultiplesuchreferenceABRDFeldsinordertogeneratetheABRDFeldofthegiveninput.InthisworkweuseacombinationofthesepossibletechniquestoestimatetheABRDFeldofagivenface. Therestofthischapterisorganizedasfollows:inSection2webeingwithasurveyofafewrelightingandposechangemethodswhicharespecicallymeanttoworkwithsingleinputimage.ThesemayhavesomeoverlapwiththetechniquesmentionedinChapter 2 66

PAGE 67

Thesubsetoftechniquesthataremostcloselyrelatedtoourtechniqueconsistsofthemethodsthatinvolvettingacanonicalreectance-shapefacemodeltothegiveninputimage.Usingthesingleinputimage,thettingprocessisexpectedtowarpormodifythecanonicalfacemodelinsuchawaythattheendresultisthereectance-shapemodelforthegivenface,whichcanthenbeusedforrelightingandposechange.MorphableModels[ 10 ]andSphericalHarmonicsMorphableModels[ 11 ]aretheprimeexamplesofsuchtechniques.Bothofthesemethodsinvolvebuildingshapeandtexturemodelsusingabootstrapsetofimagesand3Dshapes,whicharethentontothegiveninputimageusingsomenon-linearoptimizationtechniques.ThelateroftheseusestheSphericalHarmonicsbasedlightingmodelinsteadofthePhong'slightingmodelusedintheformer.Theupsideofusingthesemethodsisthattheycanobtainthereectanceeldaswellastheshapeinoneshot.Onthedownside,thesemethodsrequiremanualinitializationoftheoptimizationprocess,whichitselfcanbequitecumbersomeandsusceptibletolocalminima. 67

PAGE 68

17 ])andthePracticalRelighting([ 61 ]).Thesemethodsuseabootstrapsetofimagesand/orshape,butassumethemtobeinaroughalignmentwiththeinputimage.Thisassumptionalleviatestheneedforcomputationallyexpensivenon-rigidalignmentatthecostofsomelossintheresults'quality.Givenabootstrapsetof3imagesofatleastoneotherface,theQuotientImagemethodcomputestheilluminationneutral"QuotientImage"fromagiveninputimage.Thisilluminationneutralimagecanthenbecombinedwithvariouslightingdirectionsandintensitiestoproducevariousrelightedimages.ThoughtheQuotientImagemethodworkswiththeLambertianassumption,surfacenormalsarenotrequiredtobeexplicitlycomputed.Ontheotherhand,thePracticalRelightingtechniqueassumesameanfaceshapeandameanfacetexturetobeapplicabletoallthefaces.UsingthebootstrapshapeandtexturemodelswiththeLambertianmodel,ititerativelyrecoversthealbedoandthelightingintheinputimage.Anotherinterestingpieceofworkwaspresentedin[ 52 ],whichmovesbeyondtheuseoftheLambertianmodelandusesstatisticalmodelsinconjunctionwiththeshape-from-shadingmethodtorecovertheshapeandthereectanceofthefaceinthegiveninputimage. 11 ],[ 10 ]),likethemanualinitializationandthecumbersomeoptimizationproceduresareavoided.Ourmethodiscomposedoftwoparts-buildingareferencereectancemodelandttingthismodeltothegiveninputimage(s).Itisfullyautomatedwiththeonlyrequirementthatatleastoneoftheinputimagesbesomewhatfrontallylit.Italso 68

PAGE 69

Ourreferencereectancemodelisdierentthanthereferencemodelsusedbyexistingtechniquesinthatthoughitimbibesthe3Dshapeinformation,itisfullydenedbya2Deldofsphericalfunctions.Thisallowsthettingofthemodeltobecarriedoutwithoutany3Dto2Dprojections.Inthefollowingsectionswerstdescribeourreferencemodelandthenthettingprocedureindetail. Wehavechosentousedenitionsofthetextureandtheilluminationmodelswhichareindependentofanyparticularreectancemodel.Inadatadrivenfashion,wedenethetextureatapixeltobethemeanvalueoftheABRDF,whilethequotientfunctionobtainedbydividingtheABRDFwiththetextureisdenedtobetheilluminationfunction(illuminationmodelisaeldofsuchilluminationfunctions).Inotherwords,textureistakentobetheaverageofallpossibleintensityvaluesobtainedatapixelby 69

PAGE 70

Withthesedenitions,theABRDFeldofanygivenfacecanbeeasilyfactoredintoascalareldcalledtextureandanilluminationmodel.Thus,ourreferenceABRDFmodeliscomposedofareferencetexturemodelandareferenceilluminationmodel.Nowinordertobuildthereferencetextureandilluminationmodels,weusethetechniquedevelopedinChapter 2 .GivenNsubjectswithmimageseach,underknownpointsourcelighting,theirABRDFeldscanbebuiltusingtheTensorSplinesmodel(Eigenbubblescanalsobeused).Theaverageofallthemimages,whichuniformlysamplesthelightingdirections,istakenasanestimateforthetexture,whilethequotientfunctioneldobtainedbydividingeachABRDFwiththetexturevalueistakenasanestimatefortheilluminationmodel.Thetextureimagesarethenusedtonon-rigidlyalignalltheNfacestoareferenceface.Theobtaineddeformationeldisalsousedtoaligntheilluminationmodelsacrossdierentsubjecttothesamereferenceface. Usingthe3rdorderTensorSplinesrepresentation,weobtain10coecientsperpixelfortheilluminationmodelandascalarvalueperpixelforthetexturemodel.Foreachsubject,westringallthe10coecientsatallthepixelsintoasinglevector,Lj,andsimilarly,alsoobtainthetexturevectorTi.Further,weassumethatboththetextureandtheilluminationmodelforanygivenfacecomefromthemulti-variateGaussiandistributions.Givenabootstrapsetofalignedfacialtextures(Ti)andilluminationmodels(Lj),thecovariancematricesforthetextureandtheilluminationmodeldistributionscanbedenedasCT=Pi(TiT)(TiT)=NandCL=Pj(LjL)(LjL)=Nrespectively.TandLaretheaveragetextureandtheaverageilluminationmodels,respectively. WeobtainanorthonormalbasisforthecovariancematricesCTandCLbythePrincipalComponentAnalysisastheireigenvectors^tiand^ljrespectively.Notethattheeigenvectorshereareorderedaccordingtodecreasingeigenvalues.Hence,foranyABRDF 70

PAGE 71

Anoverviewofourtechnique. eld,A,thetexture,Tcanbeexpressedas whiletheilluminationmodel,Lcanbewrittenas Thenumberoftermsinboththemodelscanbechoseninaccordancewiththecomputationalandthequalityrequirements.ThesetwoquantitiescannowbecomposedtogettheABRDFelds.Wecallthesetf^ti;T;^lj;LgasthereferenceABRDFeldofthereferencereectancemodel. 71

PAGE 72

whereTxandTyarethexandycomponentsofthenon-rigiddeformationappliedtotheinputimages,kandkaretheilluminationdirectionsofthekinputimages,iaretheilluminationcoecientsandjarethetexturecoecients.ThefunctionDisanabstractionoftheprocesswhichtakesthevectorofcoecientsandcomputesthesphericalilluminationfunctionateachpixelusingtheTensorSplinesbasis.Thesefunctionsarethensampledat(x;y)locationin(;)directionandscaledbytheestimatedtexturevalueatthelocation(x;y).ThesetupforthettingproceduredescribedaboveisgraphicallydepictedinFig. 4-1 Inadditiontothesum-of-dierencesobjectivefunctiondenedabove,weconstrainthesearchspacefortheilluminationmodelfurtherbyaddingthefollowingTikhonovregularizertoEq. 4{3 whereistheregularizationparameter.Thisconstrainteectivelykeepstheestimatedilluminationmodelfromgoingtoofarofromthemeanilluminationmodelandresultsinartifactfreerelightedimages.Thevalueoftheparameterissetbytheuserbasedonthedesiredrelightedimagequality. Webreakdowntheprocessofrecoveringtheunknownintofoursteps.Intherststep,theinputimagesarealignedwiththereferenceABRDFmodel.Weusethe2DMorphableModels[ 62 ]methodtocomputethenon-rigiddeformationparameters.Theinputstothissteparetwoimages-thereferencefaceimageusedtoaligntheABRDFeldsandtheinputimagewithsomewhatfrontalillumination.TheoutputofthissteparethedeformationparametersTxandTywhichcanbeusedtowarptheinputimage(s)tothereferencemodel. 72

PAGE 73

usingagradientdescentbasedtechnique.Theunknownilluminationandtextureparametersareinitializedwithonesandtheilluminationdirectionsareinitializedwithzeros.InpracticewehavefoundthatusingMATLAB'sfminuncfunctionwith200iterationsprovidesgoodestimateoftheunknownparameters. Oncetheunknownshavebeenrecovered,wehaveanestimatefortheABRDFeldoftheinputfacebutitisstillalignedwiththereferencefaces.Tohandlethis,asthethirdstepinourmodelttingprocess,webackwardwarptheestimatedABRDFeldusingthedeformationparameterscomputedearlier.Butsincetheprocessdescribedaboveinvolvestworegistrationsteps,theresultantABRDFeldprovidesimagesthatappeargrainy. Inordertoremovetheseinterpolationartifacts,wehaveincorporatedanalstepinthettingprocesscalledquotientmapping.Towardsthis,wegenerateanimagefromthecomputedABRDFeldwiththesamelightingdirectionasthenear-frontalinputimage.Notethatthelightingdirectionforthisimagewascomputedaspartoftheoptimizationproceduredescribedabove.Nextwecomputethequotientmapbydividingthenearfrontalimagewithitssynthesizedestimate.ThisquotientmapisthenusedtoscaletheestimatedABRDFeldandsuppressestheartifactsintroducedbyinterpolationandextrapolationduringthenon-rigidalignmentsoftheABRDFeld. 73

PAGE 74

ComparisonofrelightedimageswithgroundtruthimagesfromtheCMUPIEandMERLDOMEdatabases. usedimagesfromtheCMUPIEandtheMERLDomedatabasesasinputimagesofourrelightingmethod.Thoughourmethodcanbeusedwithmultipleinputimages,herewehavepresentedresultsprimarilyonsingleinputimagecasesince,asdescribedbefore,itisthemoredicultandusefulspecialcaseoftherelightingandtheposechangeproblems. 4-2 .ThetoptworowsshowresultsforaninputfromtheCMUPIEdatasetwhilethebottomtworowsshowresultsforaninputfromtheMERLDomedataset.Forvariousgroundtruthlightingdirection,relightedimagesarepresented.Notethattheclosestrelightedimagesweremanuallyselected.Wehavealsoincludedacoupleofimagesunderextremeilluminations,forwhichtherewerenogroundtruthimagesavailable.ThisshowstheimportanceofagoodABRDFreferencemodel.SincetheExtendedYaleBimageshadmoreextremelightingexamplesthantheCMUPIEortheMERLDomedatasets,ourmethodwasabletopredictappearancesofthesefacesundermoreextremeilluminationthanthoseincludedintheCMUPIEortheMERLDomedatasets. 74

PAGE 75

4-3 andFig. 4-4 )andMERLDome(Fig. 4-5 andFig. 4-6 )datasets,wepresentthewholerangeofimagesgeneratedbyourmethodastheelevationandazimuthangleforthepointlightsourcevariesfrom60to+60.Thegradualappearanceofcastandattachedshadowscanbenotedastheilluminationdirectionmovesawayfromthefrontaldirection. 2 ,wecanusetheshaperecoverymethoddescribedtheretoestimateitsshapetoo.InFig. 4-7 weshowthenovelposesrenderedofthreedierentfacesfromtheCMUPIEdatasetusingsingleimageasinput. 6 75

PAGE 76

RelightedimagesfromCMUPIEdataset.Lefttorightandtoptobottomtheilluminationanglevariesfrom60to+60

PAGE 77

RelightedimagesfromCMUPIEdataset.Lefttorightandtoptobottomtheilluminationanglevariesfrom60to+60

PAGE 78

RelightedimagesfromMERLDOMEdataset.Lefttorightandtoptobottomtheilluminationanglevariesfrom60to+60

PAGE 79

RelightedimagesfromMERLDOMEdataset.Lefttorightandtoptobottomtheilluminationanglevariesfrom60to+60

PAGE 80

PosechangedimagesofthreesubjectsfromtheCMUPIEdataset.Thesingleimageusedastheinputisalsoshownintherstcolumn. 80

PAGE 81

Ifwetrytoorganizetheepitomeofliteraturepresentinthiseld,looselyadichotomyofapproachesemerges.Therstclassofthesetriestocapturethephysicalprocessesofimageformationundervarioussceneparametervariationslikeillumination(HarmonicImageExemplar[ 63 ],GenericABRDF[ 3 ],IlluminationCone[ 64 ],UniversalLighting[ 65 ]),pose(Shape+SphericalHarmonicsBasis[ 66 ],MorphableModels[ 67 ]),expression(Isometry-invariantSimilarity[ 68 ],Geometry-Texture[ 69 ])etc.Incontrast,thesecondclassofapproachesinvokesmathematicalandstatisticaltoolstocapturethestructureoftheoft-invisiblerelationsamongthenumbersthatmakeupthefaceimages.Thesetechniquesexploretheintrinsicdatageometryassumingimagestobeeithervectors(e.g.Eigenfaces[ 70 ],Fisherfaces[ 71 ],Laplacianfaces[ 72 ],orthogonalLaplacianfaces(OLAP)[ 73 ],NeighborhoodPreservingEmbedding[ 74 ],MarginalFisherAnalysis[ 75 ],LaplacianEigenmaps[ 76 ],LocallyLinearEmbedding[ 77 ],LocalityPreservingProjections[ 78 ],KernelLocalityPreservingProjectionswithSideInformation(KLPPSI)[ 79 ],MLASSO[ 80 ],KernelRidgeRegression(KRR)[ 81 ]),orhigherdimensionaltensors(e.g.TensorSubspaceAnalysis[ 82 ],2-DimensionalLinearDiscriminantAnalysis[ 83 ],TensorMarginalFisherAnalysis[ 75 ],Multi-LinearDiscriminantAnalysis[ 84 ],Tensorfaces[ 85 ],OrthogonalRankOneTensorProjection(ORO)[ 86 ],TensorAverageNeighborhoodMarginMaximization(TANMM)[ 87 ],CorrelationTensorAnalysis(CTA)[ 88 ],SpectralRegression[ 89 ],RegularizedDiscriminantAnalysis[ 89 ],SmoothLDA[ 90 ]). 81

PAGE 82

StructureofA1iandA2iforanimageofsize55andkernelofsize33.Intherstrow,9neighborhoodsoftheimageIiarehighlighted.Forrstorderapproximation,eachofoftheseneighborhoodsbecomearowinA1i.Forsecondordercase,wetakeallthesecondordercombinationsofpixelvaluesineachneighborhoodandusethemastherst81(b4)elementsofarowinA2i.Therest9(b2)elementsaresimplythepixelvalues.Rowsarenumberedtoshowwhichneighborhoodtheycorrespondto. Amajoradvantageofthetechniquesintherstclasscomesfromtheirbeinggenerativeinnature.Thispropertyallowsthesemethodstoaccomplishtaskslikefacerelighting(e.g.[ 3 ],[ 63 ],[ 91 ])ornovelposegenerationorcomplete3Dimagereconstruction(e.g.[ 66 ],[ 67 ])inadditiontorecognition.Atthesametime,methodsintherstclasstendtodemandmoresideinformationfromthedataascomparedtothesecondclassofmethods(e.g.[ 3 ]requiresilluminationdirectionforthetrainingset,[ 91 ]requiresfacialfeaturepointsforinitializationetc).Thesecondclassofmethodsareinasensemoreversatileastheycanbeseamlesslyappliedtoavarietyofdierentimagesetswithoutanysignicantrequirementofsideinformation. Themethodthatweproposeinthischapterlooselyfallsintothesecondcategoryoftechniques.Weseekamappingoffaceimagepatchessuchthatintherangespace,discriminationamongdierentclassesiseasier.WechooseVolterrakernelstoaccomplishthisbecauseitallowsustosystematicallybuildprogressivelybetterapproximationstosuchamapping.Furthermore,Volterrakernelscanbelearntinadatadrivenfashionwhichrelievesusfrombeingpredisposedtowardsanyxedkernelform(e.g.Gaussian, 82

PAGE 83

Trainingimagesfromeachclassarestackedupanddividedintoequalsizedpatches.CorrespondingpatchesfromeachclassarethenusedtolearnhigherorderconvolutionalVolterrakernelbyminimizingintraclassdistanceoverinterclassdistance.WeendupwithoneVolerrakernelpergroupofspatiallycorrespondingpatches.Thesizeandtheorderofthekernelisheldconstantforagiventrainingprocess.Notethatthecolorimagesareonlyusedforillustration,sofarourimplementationworkswithgrayscaleimages. RadialBasisFunctionetc).ThefaceimagesintherangespacearecalledVolterrafacesinthischapter. Volterraseriestheorygeneralizesthisconceptandstatesthatanynon-lineartranslationinvariantfunctional@:H!H,whichmapsthefunctionx(t)tofunctiony(t),canbedescribedbyasequenceoffunctionshn(t)as 83

PAGE 84

where Herehn(1;:::;n)arecalledtheVolterraKernelsofthefunctional.Itmustbenotedthattheaboveequationcanbeseamlesslygeneralizedto2dimensionalfunctions,I(u;v),whichforinstance,canbeanimage.Itcanbenotedthateq.( 5{1 )isjustaspecialcaseofthemoregeneraleq.( 5{3 )ifonlytherstordertermsaretakenintoaccount. Sinceweareinterestedincomputingusingthistheory,wewouldbeusingthefollowingdiscreteformofeq.( 5{3 ). Theinniteseriesformineq.( 5{4 )doesnotlenditselfwellforpracticalimplementations.Further,foragivenapplication,onlytherstfewtermsmaybeabletogivethedesiredapproximationofthefunctional.ThusweneedatruncatedformofVolterraseries,whichisdenotedinthischapteras wherepdenotesthemaximalorderofthetermstakeninaccountfortheapproximation.NotethatinthistruncatedVolterraseriesrepresentation,h(m)isaplaceholderforallthedierentorderofkernels. Ingeneral,givenasetofinputfunctionsI,weareinterestedinndingafunctional@,suchthat@(I)hassomedesiredproperty.Thisdesiredpropertycanbecapturedbydeningagoodnessfunctionalontherangespaceof@.IncaseswhentheexplicitequationrelatingtheinputsetIto@(I)isknown,varioustechniqueslikeharmonicinputmethod,directexpansionetc.([ 92 ])canbeusedtocomputekernelsoftheunknownfunctional.In 84

PAGE 85

Inthisframework,theproblemofpatternclassicationcanbeposedasfollows.GivenasetofinputdataI=fgigwherei=1:::N,asetofclassesC=fckgwherek=1:::Kandamappingwhichassociateseachgitoaclassck,ndafunctionalsuchthatintherangespace,data@(I)iseasilyclassiable.Herethegoodnessfunctionalcouldbeameasureofseparabilityofclassesintherangespace.OncetheVolterrakernelshavebeendetermined,anewdatapointcanbeclassiedusingthelearntfunctional.@(I)canbeapproximatedtoappropriateaccuracybasedoncomputationaleciencyandclassicationaccuracyconstraints. 85

PAGE 86

Testinginvolvesdividingthetestimageaccordingtotheschemeusedwhiletraining.TheneachpatchismappedtotherangespacebythecorrespondingVolterrakernel.AfterthemappingeachpatchisclassiedusingaNearestNeighborclassierintherangespace.Afterallpatcheshavebeenindividuallyclassied,eachpatchfromthetestimagecastsavotetowardstheparentimageclassication.Theclasswiththemaximumvoteswins. wherethenumeratormeasurestheaggregateintraclassdistanceforallclassesandthedenominatormeasurestheaggregatedistanceofclassckfromallotherclassesinC.Equation( 5{6 )canbefurtherexpandedas whereK,likeh(t)ineq.( 5{5 ),isaplaceholderforalldierentorderconvolutionkernels. Atthisjuncturewemakethelinearnatureofconvolutionexplicitbyconvertingtheconvolutionoperationtomultiplication.Thisconversiontoexplicitlineartransformationalformcanbedoneinmanyways,butastheconvolutionkernelistheunknowninoursetup,wewishtokeepitasavectorandthuswetransformtheimageIiintoanewrepresentationApisuchthat 86

PAGE 87

where TheexactformofApidependsontheorderoftheconvolutionsp.InSection5wehavepresentedresultsforuptillthesecondorderapproximationsandthusthestructureofApiisexplainedforonlyuptosecondorder,butitshouldbenotedthattherecognitionframeworkusingvolterrakernelsthatweproposeisverygeneralandthestructureofApiforanyordercanbeanalogouslyderived. 5-1 ([ 4 ]).Borderpixelscanbeignoredortakenintoaccountduringconvolutionbypaddingtheimagewithzeroswithoutaectingtheperformancesignicantly. Substitutingtheabovedenedrepresentationforconvolutionineq.( 5{7 ),weobtain Thiscanbewrittenas where and 87

PAGE 88

5{10 )leadstosolvingthegeneralizedeigenvalueproblemandthustheminimumofO(I)isgivenbytheminimumeigenvalueofSB1SWanditisattainedwhen (5{13) Thersttermineq.( 5{13 )correspondstoaweightedsumoftherstorderterms,x(mq1),whilethesecondordertermcorrespondstoweightedsumofthesecondorderterms,x(mq1)x(mq2).ForanimageIiofsizemnpixelsandkernelsofsizebb,thetransformedmatrixA2iforthesecondorderapproximationineq.( 5{8 )hasdimensionsmn(b4+b2)andthekernelvectorthatmultipliesit, 5-1 ([ 4 ]).Itmustnotedthattheproblemisstilllinearinthevariablesbeingsolvedforandinfactbyuseofthisformulationwehaveensuredthatregardlessoftheorderofapproximation,theproblemislinearinthecoecientsofthevolterraconvolutionkernels. WiththisdenitionofA2iweproceedasfortherstorderapproximationtoobtainequations( 5{9 )and( 5{10 )withthedierencebeingthatthematricesSBandSWnowhavedimensions(b4+b2)(b4+b2).Herewemustpointoutanimportantmodication 88

PAGE 89

92 ])andthissymmetryalsomanifestsitselfintothestructureofA2i.ByallowingonlyuniqueentriesinA2iwecanreducethedimensionsofA2itomnb4+3b2 Training(Fig. 5-2 [ 4 ])intheproposedframeworkinvolveslearningavolterrakernelfromthecorrespondingpatchesofthetrainingimages.Testing(Fig. 5-3 [ 4 ])inourframeworkinvolvestwostages.Therststageclassieseachpatchusingthemappingframeworkdescribedintheprevioussectionandanearestneighborclassier.Inthesecondstage,eachpatchcastsavotetowardstheparentimageclassicationandtheclasswithmaximumvoteswins.Inthecaseofatie,weclassifytheimagebyarun-oamongthetiedclasses.Asthenumberofcaseswithtiesaresmall,asimplerstrategyofusingacointosstobreakthetiealsoshowedsimilarresults. 80 ],parameter2[0;1]in[ 90 ],dimensionalityDin[ 93 ],reduceddimensionalityinTSA[ 82 ],2D-LDA[ 83 ],energythresholdin[ 94 ]etc.),Volterra 89

PAGE 90

State-of-the-artmethodswithwhichwecompareourtechniquealongwiththetrainingsetsizesusedintheirexperiments. Method YaleA CMUPIE Ext.YaleB 79 ] 5102030 5102030 80 ] 234 234 93 ] 2-8 88 ] 5101520 5102030 88 ] 5102030 81 ] 5102030 5102030 86 ] 5 30 20 90 ] 2345 87 ] 234 51020 89 ] 3040 10203040 73 ] 2345 5102030 73 ] 2345 5102030 90 ],[ 81 ])andacceptedmethods,crossvalidation,forparameterselection. Foremostistheselectionofthepatchsize,andforthis,startingwiththewholefaceimagewedeneaquad-treeofsub-images.Weprogressivelygodownthetreestoppingatthelevelbeyondwhichthereisnoimprovementintherecognitionratesincrossvalidation.Empiricallywefoundthatapatchsizeof88pixelsprovidesthebestresultsinallcases.Next,weallowpatchestobeoverlappingornon-overlapping.TheVolterrakernelsizecanbeofsize33or55pixels(anythingbiggerthanthisseverelyover-tsapatchofsize88).Lastly,theorderofthekernelcanbequadraticorlinear. Herewehavepresentedresultsforbothquadraticandlinearkernels.Restoftheparametersweresetusinga20-foldleave-one-outcrossvalidationonthetrainingset.Itcanbenotedfromtheresultspresentedinthenextsectionthatthebestparametercongurationisfairlyconsistentnotjustwithinaparticulardatabasebutalsoacrossdatabases. 90

PAGE 91

TrainSetSize 2 3 4 5 S-LDA[ 90 ] 42.4 27.7 22.2 18.3 S-LDA[ 90 ](updated) 37.5 25.5 19.3 14.7 UVF[ 93 ] 27.11 17.38 11.71 8.16 TANMM[ 87 ] 44.69 29.57 18.44 OLAP[ 73 ] 44.3 29.9 22.7 17.9 Eigenfaces[ 73 ] 56.5 51.1 47.8 45.2 Fisherfaces[ 73 ] 54.3 35.5 27.3 22.5 Laplacianfaces[ 73 ] 43.5 31.5 25.4 21.7 Volterrafaces(Linear) 12.33 9.47 6.11 13.36 10.19 TrainSetSize 6 7 8 9 S-LDA[ 90 ](updated) 12.3 10.3 8.7 UVF[ 93 ] 6.27 5.07 3.82 Volterrafaces(Linear) 3.96 2.61 1.43 10.04 9.66 9.49 8.74 TrainSetSize 5 10 20 30 KLPPSI[ 79 ] 27.88 12.32 5.48 3.62 KRR[ 81 ] 26.4 13.1 5.97 4.02 ORO[ 86 ] 6.4 TANMM[ 87 ] 26.98 17.22 5.68 SR[ 89 ] 6.1 OLAP[ 73 ] 21.4 11.4 6.51 4.83 Eigenfaces[ 73 ] 69.9 55.7 38.1 27.9 Fisherfaces[ 73 ] 31.5 22.4 15.4 7.77 Laplacianfaces[ 73 ] 30.8 21.1 14.1 7.13 Volterrafaces(Linear) 10.24 4.94 2.85 25.29 11.94 TrainSetSize 2 3 4 40 SR[ 89 ] 5.2 MLASSO[ 80 ] 54.0 43.0 34.0 Volterrafaces(Linear) 36.30 23.98 2.37 39.66 32.67 3.04

PAGE 92

TrainSetSize 5 10 20 30 ORO[ 86 ] 9.0 SR[ 89 ] 12.0 4.7 2.0 RDA[ 89 ] 11.6 4.2 1.8 KLPPSI[ 79 ] 24.74 9.93 3.15 1.39 KRR[ 81 ] 23.9 11.04 3.67 1.43 CTA[ 88 ] 16.99 7.60 4.96 2.94 Eigenfaces[ 88 ] 54.73 36.06 31.22 27.71 Fisherfaces[ 88 ] 37.56 18.91 16.87 14.94 Laplacianfaces[ 88 ] 34.08 18.03 30.26 20.20 Volterrafaces(Linear) 2.67 0.90 0.42 3.98 1.27 0.58 2 3 4 40 MLASSO[ 80 ] 58.0 54.0 50.0 SR[ 89 ] 1.0 RDA[ 89 ] 0.9 Volterrafaces(Linear) 18.23 9.33 0.34 20.47 14.42 0.43 73 ])experimentalsetup.Alloftheseareembeddingmethodsmentionedintheprologuewiththeexceptionof[ 93 ]whichbuildsontheconceptofUniversalVisualFeatures(UVF).Inourstudy,wepresentresultsonYaleA,CMUPIEandExtendedYaleBbenchmarkfacedatabasespartlybecausetheyarefewofthemostpopulardatabaseswhichmakesacomparativestudyeasy.Inadditiontotherecentmethodswealsoprovidecomparisonwiththetraditionalbaselinemethods-Eigenfaces,FisherfaceandLaplacianfaces.Table. 5-1 ([ 4 ])liststhesemethodsalongwiththenumberoftrainingimagesusedbythemontheabovementioneddatabasesintheirexperiments.Wehavepresentedourresultsforthewholerangeoftrainingsetsizessothatcomparisonwithmaximumnumberoftechniquescanbemade. 92

PAGE 93

95 ])weusedall170images(exceptafewcorruptedimages)from5nearfrontalposes(C05,C07,C09,C27,C29)foreachofthe68subjects.ForExtendedYaleBdatabase([ 65 ])weused64(exceptforafewcorruptedimages)imageseach(frontalpose)of38individualspresentinthedatabase.NotethatthemethodsinTable. 5-1 ([ 4 ])usedthesamesubsetofimages.Weobtainedthedatafromthewebsiteoftheauthorsof[ 90 ]2.Theseimagesaremanuallyaligned(twoeyeswerealignedatthesameposition)andcroppedtoextractfaces,with256grayvaluelevelsperpixel. Theresults(averagerecognitionerrorrates)ontheYaleA,CMUPIEandExtendedYaleBdatabasesarepresentedinTable. 5-2 ([ 4 ]),Table. 5-3 ([ 4 ])andTable. 5-4 ([ 4 ])respectively.RowstitledTrainSetSizeindicatethenumberoftrainingimagesusedandtherowsbelowthemlisttheratesreportedbyvariousstate-of-the-artandourmethod(Volterrafaces).Eachexperimentwasrepeated10timesfor10randomchoicesoftrainingset.Allimagesotherthanthetrainingsetwereusedfortesting.SpecicexperimentalsetupusedforVolterrafacesismentionedbeloweachtable.Wehavereportedresultswithbothlinearandquadraticmasksforthesakeofcompleteness.Bestresultsforaparticulartrainingsetsizearehighlightedinbold. 93

PAGE 94

WehaveintroducedtheuseofVolterrakernelapproximationsforimagerecognitionfunctionalsinthischapter.Thekernellearningisdrivenbytrainingdata,basedonagoodnessfunctionaldenedintherangespaceoftherecognitionfunctional.Itisshownthatforthegoodnessfunctionalthattriestominimizeintraclassdistanceswhilemaximizinginterclassdistances,kernelcomputationreducestothegeneralizedeigenvalueproblemwhichtranslatestoveryecientcomputationofkernelsforanyorderapproximationofthefunctional.Eectivenessofthistechniqueforfacerecognitionisdemonstratedbyexperimentsonthreebenchmarkdatabasesandtheresultsarecomparedtotraditionalaswellasthestateofthearttechniquesindiscriminantanalysisforfaces.FromtheresultspresentedinthischapteritcanbeconcludedthatVolterrakernelapproximationsshowgreatpromiseforapplicationsinimagerecognitiontasks. 94

PAGE 95

Theliteratureisrepletewithvariousproposals(e.g.[ 17 ],[ 61 ],[ 52 ],[ 96 ])onhowtomarryrelightingtechniqueswithfacerecognition.Wecanbroadlyputtheexistingtechniquesthathaveexaminedimagerelightinginconjunctionwithrecognitionintotwoclasses.Therstincludestechniqueswhichuserelightingasawaytoobtainanilluminationinvariantrepresentationsofthegalleryandprobeimages(e.g.[ 97 ],[ 98 ])whilethesecondclassincludesthetechniqueswhichseektoimproverecognitionbyaugmentingthegallerysetwithnovelrelightedimages(e.g.[ 52 ]). 4 togeneratemultiplerelightedversionofthesameimage.Thesenewimagesareaddedtothegallerysetandanygivenprobeimageisnowmatchedagainstalloftheimages. Onewouldexpectthataprobeimagewhichhasverydierentlightingthantheinputimagewouldnotbeagoodmatchforit,butitislikelythatoneofthenewlyaddedimagemightbeaclosermatchforit.Eveninthecasewheretheprobeimagehasbeenlitfrommultiplelightsources,alinearcombinationofthenewgalleryimageswouldbeacloser 95

PAGE 96

Herewedescribetherecognizersweusedinourexperiments. 99 ].ThisclassierusesalargesetofpairsofimageslabeledassameordierenttolearnasetofgoodrecognitionfeaturesandclassierparametersusingBoosting.Oncetheclassierparametersandthediscriminatingfeatureshavebeenlearnt,givenanypairofimages,itassignsasimilarityscoretothepair.Thissimilarityscorecanthenbeusedinconjunctionwithaclassicationthresholdtocategorizeimagesasdierentsubjects. 96

PAGE 97

100 ],whichhavebeenverysuccessfullyappliedtofacerecognition.Atcontrasttotheabovementionedclassiers,thismethodusesahistogramoffeaturesandthusisnotsensitivetotheabsolutelocationsofthefeatures. Tocompareourresultswithotherrelightingmethods,wealsoimplementfollowingtworelightingmethods: 61 ])estimatesthelightingdirectionintheinputimage.Thelightingestimateandtheshapemodelarethenusedtoestimateafacespecicalbedoeld.Withthealbedoandtheshapeavailable,imageofthefaceunderanyilluminationcanberenderedusingtheLambertianModel.Notethatthismethodassumesthatallthefacehavethesameshapeandrequiresonlyaroughalignmentamongthem. 97

PAGE 98

Eectofgalleryaugmentationsize:FaceRecognitionratesfortheCMUPIEdatasetwiththeMERLclassier. AugmentationSize RelightingMethod 256 128 64 32 16 8 4 1 NaiveRelighting 92.5 92.2 91.8 93.1 92.8 90.4 90.9 90.1 PracticalRelighting 92.7 92.2 92.3 91.6 90.7 90.2 91.3 90.1 ProposedTechnique 98.4 98.4 98.1 97.4 97.9 96.6 94.8 90.1 Intherstsetofresultsweshowhowtherecognitionrateschangeasthenumberofimagesusedtoaugmentthegalleryincreases.HerewehaveusedtheMERLclassierwithourrelightingontheCMUPIEdataset.Thegalleryandtheprobesetsarecomposedoffacialimagesinthefrontalpose.TheseresultsarepresentedinTable. 6-1 .Itcanbenotedthatacrossdierentrelightingschemes,therecognitionratesshowimprovementasthenumberofaugmentedimagesisincreased.Forthisexperimentwesynthesized256imagesofeachgallerysubjectbyuniformlysamplingtheilluminationdirectionspacefrom+60to60inboththeelevationandtheazimuthangles.Otheraugmentationswereobtainedbyuniformlyspacingtheimagesinthepreviouslymentionedrange. Fortherestoftheexperimentswewouldxthenumberofaugmentedimagesto64.Thisisbecauseitprovidesareasonabletrade-obetweenthememory-timerequirementsandtherecognitionrates.WepresentrecognitionratesfortheCMUPIEdatabasein 98

PAGE 99

FacerecognitionratesfortheCMUPIEdataset NearestNeighbor(L1) MERLClassier LocalBinaryPatterns RelightingMethod Augmented SingleImage Augmented SingleImage Augmented SingleImage NaiveRelighting 70.4 58.9 91.8 90.1 93.9 93.4 PracticalRelighting 80.7 58.9 92.3 90.1 96.6 93.4 ProposedTechnique 93.5 58.9 98.1 90.1 99.2 93.4 FacerecognitionratesfortheMERLDomedataset NearestNeighbor(L1) MERLClassier LocalBinaryPatterns RelightingMethod Augmented SingleImage Augmented SingleImage Augmented SingleImage NaiveRelighting 54.3 42.7 78.2 71.2 66.3 65.6 PracticalRelighting 77.5 42.7 80.8 71.2 85.4 65.6 ProposedTechnique 79.8 42.7 82.9 71.2 88.2 65.6 6-2 andfortheMERLDOMEdatabaseinTable 6-3 .Itcanbenotedthatacrossthedierentdatabasesandacrossthedierentclassicationschemes,ourrelightingmethodprovideshigherrecognitionratesthanthecompetingmethods. Rank-onerecognitionrates,aspresentedinTables 6-2 and 6-3 ,thoughuseful,donotprovideinsightintotheimpactofgalleryaugmentationontheFalseAcceptanceRates.Weexplorethisaspectofthegalleysetaugmentationusingthereceiveroperatingcharacteristic(ROC)curveswhichplottheFalseRejectionRatesagainsttheFalseAcceptanceRatesastherecognitionthresholdisvaried.Foreachprobeimage,adistanceiscomputedfromeverygalleryclass.Thoughthisdistanceiscomputeddierentlyfordierentrecognitionstrategies,itisascalarvalueinallthecases.TherecognitionthresholdusedtogeneratetheROCcurvesdecideswhentoclassifyaprobeimageasbelongingtoaclass.Ifthedistancebetweenaprobeimageandaclassislessthanthethresholdvalue,theprobeimageisclassiedasbelongingtothatclass,otherwisenot.Notethattheclassesherecorrespondtothesubjects. InFig. 6-1 andFig. 6-2 wepresenttheReceiverOperatingCharacteristic(ROC)curvesfortheMERLclassier,ontheCMUPIEandtheMERLDomedatasetsrespectively.Curvesforvariousrelightingmethodsandthesingleimagecaseareincludedintheplots.Therearethreeimportantguresofmeritforsuchcurves{rstistheEqualErrorRate(EER),whichisthepointonthecurvewheretheFalseAcceptanceRate 99

PAGE 100

ROCCurvefortheCMUPIEdatasetwiththeMERLclassier. equalstheFalseRejectionRate,thesecondistheareaundertheROCcurveandthirdistheFalseRejectionRateat0.1%FalseAcceptanceRate.Itcanbenotedthatforallofthesemeritcriterions,ourrelightingoutperformstheexistingmethods.SimilartrendscanbenotedinthecurvesfortheNearestNeighborclassierinFig. 6-3 andFig. 6-4 .ROCsfortheLocalBinaryPatternclassierontheCMUPIEandtheMERLDomedatabasesarepresentedinFig. 6-5 andFig. 6-6 respectively. 100

PAGE 101

ROCCurvefortheMERLDomedatasetwiththeMERLclassier. Figure6-3. ROCCurvefortheCMUPIEdatasetwiththeNearestNeighborclassier. 101

PAGE 102

ROCCurvefortheMERLDomedatasetwiththeNearestNeighborclassier. Figure6-5. ROCCurvefortheCMUPIEdatasetwiththeLBPclassier. 102

PAGE 103

ROCCurvefortheMERLDomedatasetwiththeLBPclassier. 103

PAGE 104

Inthisdissertationwehaveexaminedthethreefundamentalproblemsinfacialimageanalysis-relighting,posechangeandrecognition.Forallthethreeproblemswehavepresentednovelsolutionsforboththecasesofmultipleimagesandsingleimageasinput. Duetotheirtightlyinterdependentnature,wehavetreatedtheproblemsofrelightingandposechangetogetherandpresentedaTensorSplinesbasedframeworkformultipleimagerelightingandposechange.WehaveshownthatourframeworkissuperiortothepopularLambertianmodelintermsoftheshadowsandthespecularitiesreproductionintherelightedimages.Furthermore,wehaveshownthatthefaceshapeestimatedusingourmethodhasadvantagesoverpopularRobustPhotometricStereotechniquewhentheinputimagesarecastshadowridden.WehavealsopresentedamethodforenhancingtherelightedimagequalityusingEigenbubbles. Forthemoredicultproblemofsingleimagerelightingandposechange,wedrawuponabootstrapsetofABRDFelds.WedenethereferenceABRDFeldmodelintermsoftheilluminationeldmodelandthetexturemodel.Givenasingleinputimage,usinganon-linearoptimizationtechnique,wendthosecoecientsoftheilluminationandthetexturemodel,alongwiththeilluminationdirectionandthenon-rigiddeformation,whichexplaintheinputimagethebest.OncewehaveobtainedtheABRDFeld,wefollowthesameschemeasdevisedforthemultipleinputimagescasetogenerateimagesinnovelposes. Inordertoaddresstheproblemoffacerecognitionwithmultipleinputimages,wepresentedanovelclassicationschemeusingtheVolterrakernelswhichwecallVolterrafaces.Wehaveshownthatthismethodoutperformsvariousstate-of-the-artmethodsintermsofrecognitionaccuracyonthreepopularbenchmarkfacedatabases.Wehaveaddressedtheproblemofsingleimagerecognitioninabroadersenseandhaveproposedagallerysetaugmentationframeworkwhichcanbeusedwithmosto-the-shelf 104

PAGE 105

Thetechniquespresentedinthisdissertationimproveuponthestate-of-the-artinvariouswaysbuttherearenumerousavenuesthatarestillopenforexploration.Fewoftheimportantthreadsforpossiblefutureexplorationincluderemovingtheneedforhavingsomewhatfrontallylitinputimageforsingleimagerelightingandposechange,removingtheneedforpointlightsourcelitimagesasinputforbothmultipleandsingleinputimagecasesandreducingthenumberofsystemparametersintheVolterrafacesclassicationscheme.Thecomputationaleciencyaspectsofallthealgorithmspresentedinthispapercanalsobeexploredinfuturesincetheyplayanimportantroleinreal-lifeapplications. 105

PAGE 106

[1] B.T.Phong,\Illuminationforcomputergeneratedpictures,"CommunicationsoftheACM,vol.18,no.6,pp.311{317,1975. [2] K.E.TorranceandE.M.Sparrow,\Theoryforo-specularreectionfromroughenedsurfaces,"J.OpticalSocietyofAmerica,vol.57,pp.1105{1112,1967. [3] A.Barmpoutis,R.Kumar,B.C.Vemuri,andA.Banerjee,\Beyondthelambertianassumption:Agenerativemodelforapparentbrdfeldsoffacesusinganti-symmetrictensorsplines,"Proc.IEEECSConf.ComputerVisionandPatternRecognition,2008. [4] R.Kumar,A.Banerjee,andB.C.Vemuri,\Volterrafaces:Discriminantanalysisusingvolterrakernels,"Proc.IEEECSConf.ComputerVisionandPatternRecogni-tion,2009. [5] R.Kumar,A.Barmpoutis,A.Banerjee,andB.C.Vemuri,\Non-lambertianreectancemodelingandshaperecoveryforfacesusinganti-symmetrictensorsplines,"TechnicalReportREP-2009-467,Dept.ofCISE,Univ.ofFlorida,2009. [6] H.ChanandW.W.Bledsoe,\Aman-machinefacialrecognitionsystem-somepreliminaryresults,"PanoramicResearchInc,Tech.Rep.,1965. [7] R.BasriandD.W.Jacobs,\Lambertianreectanceandlinearsubspaces,"IEEETrans.PatternAnalysisandMachineIntelligence,vol.25,no.2,pp.218{233,2003. [8] P.Debevec,T.Hawkins,C.Tchou,H.Duiker,W.Sarokin,andM.Sagar,\Acquiringthereectanceeldofahumanface,"ACMTrans.onGraphics(Proc.SIGGRAPH),vol.1,pp.145{156,2000. [9] R.RamamoorthiandP.Hanrahan,\Asignal-processingframeworkforinverserendering,"ACMTrans.onGraphics(Proc.SIGGRAPH),vol.1,pp.117{128,2001. [10] V.BlanzandT.Vetter,\Facerecognitionbasedonttinga3dmorphablemodel,"IEEETrans.PatternAnalysisandMachineIntelligence,vol.25,no.9,pp.1063{1074,Sept.2003. [11] L.ZhangandD.Samaras,\Facerecognitionfromasingletrainingimageunderarbitraryunknownlightingusingsphericalharmonics,"IEEETrans.PatternAnalysisandMachineIntelligence,vol.28,no.3,pp.351{363,March2006. [12] A.S.Georghiades,P.N.Belhumeur,andD.J.Kriegman,\Fromfewtomany:Illuminationconemodelsforfacerecognitionundervariablelightingandpose,"IEEETrans.PatternAnalysisandMachineIntelligence,vol.23,no.6,pp.643{660,June2001. 106

PAGE 107

A.Barmpoutis,B.C.Vemuri,andJ.R.Forder,\Robusttensorsplinesforapproximationofdiusiontensormridata,"Proc.IEEECSWorkshoponMathe-maticalMethodsinBiomedicalImageAnalysis,pp.86{86,2006. [14] A.S.Georghiades,P.N.Belhumeur,andD.J.Kriegman,\Illumination-basedimagesynthesis:Creatingnovelimagesofhumanfacesunderdieringpose&lighting,"IEEEWorkshoponMulti-ViewModelingandAnalysisofVisualScenes,1999. [15] V.BlanzandT.Vetter,\Amorphablemodelforthesynthesisof3dfaces,"ACMTrans.onGraphics(Proc.SIGGRAPH),pp.187{194,1999. [16] T.Malzbender,D.Gelb,andH.Wolters,\Polynomialtexturemaps,"ACMTrans.onGraphics(Proc.SIGGRAPH),pp.519{528,2001. [17] A.ShashuaandT.Riklin-Raviv,\Thequotientimage:Class-basedre-renderingandrecognitionwithvaryingilluminations,"IEEETrans.PatternAnalysisandMachineIntelligence,vol.23,no.2,pp.129{139,Feb.2001. [18] W.Y.ZhaoandR.Chellappa,\Symmetricshape-from-shadingusingself-ratioimage,"Int'lJ.ComputerVision,vol.45,no.1,pp.55{75,2001. [19] S.Magda,D.J.Kriegman,T.Zickler,andP.N.Belhumeur,\Beyondlambert:Reconstructingsurfaceswitharbitrarybrdfs,"Proc.IEEEInt'lConf.ComputerVision,vol.2,pp.391{398,7{14July2001. [20] A.S.Georghiades,\Recovering3-dshapeandreectancefromasmallnumberofphotographs,"Proc.EurographicsWorkshoponRendering,pp.230{240,2003. [21] Z.Wen,Z.Liu,andT.S.Huang,\Facerelightingwithradianceenvironmentmaps,"Proc.IEEECSConf.ComputerVisionandPatternRecognition,vol.2,pp.II{158{65,18{20June2003. [22] R.Gross,I.Matthews,andS.Baker,\Appearance-basedfacerecognitionandlight-elds,"IEEETrans.PatternAnalysisandMachineIntelligence,vol.26,no.4,pp.449{465,April2004. [23] D.B.Goldman,B.Curless,A.Hertzmann,andS.M.Seitz,\Shapeandspatially-varyingbrdfsfromphotometricstereo,"Proc.IEEEInt'lConf.Com-puterVision,vol.1,pp.341{348,17{21Oct.2005. [24] A.HertzmannandS.M.Seitz,\Example-basedphotometricstereo:Shapereconstructionwithgeneral,varyingbrdfs,"IEEETrans.PatternAnalysisandMachineIntelligence,vol.27,no.8,pp.1254{1264,Aug.2005. [25] K.C.LeeandB.Moghaddam,\Apracticalfacerelightingmethodfordirectionallightingnormalization,"Int'lWorkshoponAnalysisandModellingofFacesandGestures,2005. 107

PAGE 108

T.Zickler,R.Ramamoorthi,S.Enrique,andP.N.Belhumeur,\Reectancesharing:Predictingappearancefromasparsesetofimagesofaknownshape,"IEEETrans.PatternAnalysisandMachineIntelligence,vol.28,no.8,pp.1287{1302,Aug.2006. [27] M.Chandraker,S.Agarwal,andD.Kriegman,\Shadowcuts:Photometricstereowithshadows,"Proc.IEEECSConf.ComputerVisionandPatternRecognition,2007. [28] N.AlldrinandD.Kriegman,\Shapefromvaryingilluminationandviewpoint,"Proc.IEEEInt'lConf.ComputerVision,2007. [29] S.BiswasandG.Aggarwal,\Robustestimationofalbedoforillumination-invariantmatching&shaperecovery,"Proc.IEEEInt'lConf.ComputerVision,2007. [30] S.K.Zhou,G.Aggarwal,R.Chellappa,andD.W.Jacobs,\Appearancecharacterizationoflinearlambertianobjects,generalizedphotometricstereo,andillumination-invariantfacerecognition,"IEEETrans.PatternAnalysisandMachineIntelligence,vol.29,no.2,pp.230{245,Feb.2007. [31] R.Basri,D.Jacobs,I.Kemelmacher,andR.Basri,\Photometricstereowithgeneral,unknownlighting,"Int'lJ.ComputerVision,vol.72,no.3,pp.239{257,2007. [32] N.Alldrin,T.Zickler,andD.Kriegman,\Photometricstereowithnon-parametricandspatially-varyingreectance,"Proc.IEEECSConf.ComputerVisionandPatternRecognition,pp.1{8,23{28June2008. [33] A.Shashua,\Onphotometricissuesin3dvisualrecognitionfromasingle2dimage,"Int'lJ.ComputerVision,vol.21,pp.99{122,1997. [34] A.L.Yuille,D.Snow,R.Epstein,andP.N.Belhumeur,\Determininggenerativemodelsofobjectsundervaryingillumination:Shapeandalbedofrommultipleimagesusingsvdandintegrability,"Int'lJ.ComputerVision,vol.35,no.3,pp.203{222,1999. [35] P.BelhumeurandD.Kriegman,\Whatisthesetofimagesofanobjectunderallpossibleilluminationconditions?"Int'lJ.ComputerVision,vol.28,pp.245{260,1998. [36] R.RamamoorthiandP.Hanrahan,\Therelationshipbetweenradianceandirradiance:Determiningtheilluminationfromimagesofaconvexlambertianobject,"J.OpticalSocietyofAmericaA,vol.18,no.10,pp.2448{2459,2001. [37] K.-C.Lee,J.Ho,andD.J.Kriegman,\Acquiringlinearsubspacesforfacerecognitionundervariablelighting,"IEEETrans.PatternAnalysisandMachineIntelligence,vol.27,no.5,pp.684{698,May2005. 108

PAGE 109

J.Lee,B.Moghaddam,H.Pster,andR.Machiraju,\Abilinearilluminationmodelforrobustfacerecognition,"Proc.IEEEInt'lConf.ComputerVision,vol.2,pp.1177{1184,17{21Oct.2005. [39] J.P.O'Shea,M.S.Banks,andM.Agrawala,\Theassumedlightdirectionforperceivingshapefromshading,"Proc.SymposiumonAppliedPerceptioninGraphicsandVisualization,pp.135{142,2008. [40] P.N.Belhumeur,D.J.Kriegman,andA.L.Yuille,\Thebas-reliefambiguity,"Int'lJ.ComputerVision,vol.35,no.1,pp.33{44,1999. [41] Z.Liu,Y.Shan,andZ.Zhang,\Expressiveexpressionmappingwithratioimages,"ACMTrans.onGraphics(Proc.SIGGRAPH),pp.271{276,2001. [42] A.HertzmannandS.M.Seitz,\Shapeandmaterialsbyexample:Aphotometricstereoapproach,"Proc.IEEECSConf.ComputerVisionandPatternRecognition,vol.1,pp.I{533{I{540,18{20June2003. [43] T.-P.Wu,K.-L.Tang,C.-K.Tang,andT.-T.Wong,\Densephotometricstereo:Amarkovrandomeldapproach,"IEEETrans.PatternAnalysisandMachineIntelligence,vol.28,no.11,pp.1830{1846,Nov.2006. [44] S.C.Foo,\Agonioreectometerformeasuringthebidirectionalreectanceofmaterialforuseinilluminationcomputation,"Master'sthesis,CornellUniversity,1997. [45] G.BorshukovandJ.P.Lewis,\Realistichumanfacerenderingfor"thematrixreloaded","ACMSIGGRAPH2003Sketches&Applications,pp.1{1,2003. [46] T.Weyrich,W.Matusik,H.Pster,B.Bickel,C.Donner,C.Tu,J.McAndless,J.Lee,A.Ngan,H.W.Jensen,andM.Gross,\Analysisofhumanfacesusingameasurement-basedskinreectancemodel,"ACMTrans.onGraphics(Proc.SIGGRAPH),pp.1013{1024,2006. [47] H.Jereys,CartesianTensors.Cambridge:TheUniversityPress,1931. [48] C.deBoor,\Oncalculatingwithb-splines,"J.ApproximationTheory,vol.6,p.5062,1972. [49] C.LawsonandR.J.Hanson,SolvingLeastSquaresProblems.Prentice-Hall,EnglewoodClis,1974. [50] A.Barmpoutis,B.C.Vemuri,T.M.Shepherd,andJ.R.Forder,\Tensorsplinesforinterpolationandapproximationofdt-mriwithapplicationstosegmentationofisolatedrathippocampi,"IEEETrans.MedicalImaging,vol.26,no.11,pp.1537{1546,Nov.2007. [51] B.K.P.Horn,RobotVision.McGrawHill:NewYork,1986. 109

PAGE 110

T.SimandT.Kanade,\Combiningmodelsandexemplarsforfacerecognition:Anilluminatingexample,"Proc.IEEECSConf.ComputerVisionandPatternRecognitionWorkshoponModelsversusExemplarsinComputerVision,2001. [53] P.Debevec,\Renderingsyntheticobjectsintorealscenes:Bridgingtraditionalandimage-basedgraphicswithglobalilluminationandhighdynamicrangephotography,"ACMTrans.onGraphics(Proc.SIGGRAPH),pp.189{198,1998. [54] R.GrossandV.Brajovic,\Animagepreprocessingalgorithmforilluminationinvariantfacerecognition,"Int'lConf.Audio-andVideo-BasedBiometricPersonAuthentication,pp.10{18,2003. [55] R.BrunelliandT.Poggio,\Facerecognition:Featuresversustemplates,"IEEETrans.PatternAnalysisandMachineIntelligence,vol.15,no.10,pp.1042{1052,Oct.1993. [56] P.W.Hallinan,\Alow-dimensionalrepresentationofhumanfacesforarbitrarylightingconditions,"Proc.IEEECSConf.ComputerVisionandPatternRecogni-tion,pp.995{999,21{23June1994. [57] A.U.BaturandM.H.HayesIII,\Linearsubspacesforilluminationrobustfacerecognition,"Proc.IEEECSConf.ComputerVisionandPatternRecognition,vol.2,pp.296{301,2001. [58] W.A.SmithandE.R.Hancock,\Facialshape-from-shadingandrecognitionusingprincipalgeodesicanalysisandrobuststatistics,"Int'lJ.ComputerVision,vol.76,no.1,pp.71{91,2008. [59] C.DingandX.He,\K-meansclusteringviaprincipalcomponentanalysis,"Int'lConf.MachineLearning,2004. [60] R.BasriandD.Jacobs,\Lambertianreectanceandlinearsubspaces,"IEEETrans.PatternAnalysisandMachineIntelligence,vol.25,no.2,pp.218{233,2003. [61] K.chihLeeandB.Moghaddam,\Apracticalfacerelightingmethodfordirectionallightingnormalization,"IEEEInt'lWorkshoponAnalysisandModelingofFacesandGestures,2005. [62] M.JonesandT.Poggio,\Multidimensionalmorphablemodels:Aframeworkforrepresentingandmatchingobjectclasses,"Proc.IEEEInt'lConf.ComputerVision,pp.683{688,1998. [63] L.ZhangandD.Samaras,\Facerecognitionundervariablelightingusingharmonicimageexemplars,"Proc.IEEECSConf.ComputerVisionandPatternRecognition,pp.I{19{I{25,2003. [64] A.S.Georghiades,P.Belhumeur,andD.J.Kriegman,\Fromfewtomany:Illuminationconemodelsforfacerecognitionundervariablelightingandpose," 110

PAGE 111

[65] K.Lee,J.Ho,andD.J.Kriegman,\Acquiringlinearsubspacesforfacerecognitionundervariablelighting,"IEEETrans.PatternAnalysisandMachineIntelligence,vol.27,no.5,pp.684{698,2005. [66] S.Wang,L.Zhang,andD.Samaras,\Facereconstructionacrossdierentposesandarbitraryilluminationconditions,"BiometricAuthenticationWorkshop,pp.91{101,2005. [67] V.BlanzandT.Vetter,\Facerecognitionbasedonttinga3dmorphablemodel,"IEEETrans.PatternAnalysisandMachineIntelligence,vol.25,no.9,pp.1063{1074,2003. [68] A.M.Bronstein,M.M.Bronstein,andR.Kimmel,\Robustexpression-invariantfacerecognitionfrompartiallymissingdata,"EuropeanConf.ComputerVision,pp.396{408,2006. [69] X.Li,G.Mori,andH.Zhang,\Expression-invariantfacerecognitionwithexpressionclassication,"CanadianConf.ComputerandRobotVision,pp.77{83,2006. [70] A.Pentland,B.Moghaddam,andT.Starner,\View-basedandmodulareigenfacesforfacerecognition,"Proc.IEEECSConf.ComputerVisionandPatternRecogni-tion,1994. [71] P.N.Belhumeur,J.Hespanha,andD.J.Kriegman,\Eigenfacesvs.sherfaces:Recognitionusingclassspeciclinearprojection,"IEEETrans.PatternAnalysisandMachineIntelligence,vol.19,no.7,pp.711{720,1997. [72] X.He,S.Yan,Y.Hu,P.Niyogi,andH.Zhang,\Facerecognitionusinglaplacianfaces,"IEEETrans.PatternAnalysisandMachineIntelligence,vol.27,no.3,2005. [73] D.Cai,X.He,J.Han,andH.J.Zhang,\Orthogonallaplacianfacesforfacerecognition,"IEEETrans.ImageProcessing,vol.15,no.11,2006. [74] X.He,D.Cai,S.Yan,andH.-J.Zhang,\Neighborhoodpreservingembedding,"Proc.IEEEInt'lConf.ComputerVision,pp.1208{1213,2005. [75] S.Yan,D.Xu,B.Zhang,H.-J.Zhang,Q.Yang,andS.Lin,\Graphembeddingandextension:Ageneralframeworkfordimensionalityreduction,"IEEETrans.PatternAnalysisandMachineIntelligence,vol.29,no.1,pp.830{837,2007. [76] M.BelkinandP.Niyogi,\Laplacianeigenmapsandspectraltechniquesforembeddingandclustering,"AdvancesinNeuralInformationProcessingSystems,pp.585{591,2002. 111

PAGE 112

S.RoweisandL.Saul,\Nonlineardimensionalityreductionbylocallylinearembedding,"Science,pp.2323{2326,2002. [78] X.He,D.Cai,andP.Niyogi,\Localitypreservingprojections,"AdvancesinNeuralInformationProcessingSystems,2003. [79] S.An,W.Liu,andS.Venkatesh,\Exploitingsideinformationinlocalitypreservingprojection,"Proc.IEEECSConf.ComputerVisionandPatternRecognition,2008. [80] D.-S.PhamandS.Venkatesh,\Robustlearningofdiscriminativeprojectionformulticategoryclassicationonthestiefelmanifold,"Proc.IEEECSConf.ComputerVisionandPatternRecognition,2008. [81] S.An,W.Liu,andS.Venkatesh,\Facerecognitionusingkernelridgeregression,"Proc.IEEECSConf.ComputerVisionandPatternRecognition,2007. [82] X.He,D.Cai,andP.Niyogi,\Tensorsubspaceanalysis,"AdvancesinNeuralInformationProcessingSystems,2005. [83] J.Ye,R.Janardan,andQ.Li,\Two-dimensionallineardiscriminantanalysis,"AdvancesinNeuralInformationProcessingSystems,2004. [84] S.Yan,D.Xu,Q.Yang,L.Zhang,X.Tang,andH.J.Zhang,\Multilineardiscriminantanalysisforfacerecognition,"IEEETrans.ImageProcessing,vol.16,no.1,2007. [85] M.VasilescuandD.Terzopoulos,\Multilinearsubspaceanalysisofimageensembles,"Proc.IEEECSConf.ComputerVisionandPatternRecognition,pp.212{220,2003. [86] G.Hua,P.Viola,andS.Drucker,\Facerecognitionusingdiscriminativelytrainedorthogonalrankonetensorprojections,"Proc.IEEECSConf.ComputerVisionandPatternRecognition,2007. [87] F.WangandC.Zhang,\Featureextractionbymaximizingtheaverageneighborhoodmargin,"Proc.IEEECSConf.ComputerVisionandPatternRecogni-tion,2007. [88] Y.FuandT.S.Huang,\Imageclassicationusingcorrelationtensoranalysis,"IEEETrans.ImageProcessing,vol.17,no.2,2008. [89] D.Cai,X.He,andJ.Han,\Spectralregressionforecientregularizedsubspacelearning,"Proc.IEEEInt'lConf.ComputerVision,2007. [90] D.Cai,X.He,Y.Hu,J.Han,andT.Huang,\Learningaspatiallysmoothsubspaceforfacerecognition,"Proc.IEEECSConf.ComputerVisionandPatternRecogni-tion,2007. 112

PAGE 113

L.Zhang,S.Wang,andD.Samaras,\Facesynthesisandrecognitionfromasingleimageunderarbitraryunknownlightingusingasphericalharmonicbasismorphablemodel,"Proc.IEEECSConf.ComputerVisionandPatternRecognition,pp.209{216,2005. [92] J.A.Cherry,\Introductiontovolterramethods,"DistortionAnalysisofWeaklyNonlinearFiltersUsingVolterraSeries,1994. [93] H.ShanandG.W.Cottrell,\Lookingaroundthebackyardhelpstorecognizefacesanddigits,"Proc.IEEECSConf.ComputerVisionandPatternRecognition,2008. [94] S.Rana,W.Liu,M.Lazarescu,andS.Venkatesh,\Recognisingfacesinunseenmodes:Atensorbasedapproach,"Proc.IEEECSConf.ComputerVisionandPatternRecognition,2008. [95] T.Sim,S.Baker,andM.Bsat,\Thecmupose,illumination,andexpression(pie)database,"Proc.IEEEInt'lConf.AutomaticFaceandGestureRecognition,2002. [96] C.-P.ChenandC.-S.Chen,\Lightingnormalizationwithgenericintrinsicilluminationsubspaceforfacerecognition,"Proc.IEEEInt'lConf.ComputerVision,2005. [97] D.-H.Liu,K.-M.Lama,andL.-S.Shenb,\Illuminationinvariantfacerecognition,"PatternRecognition,vol.38,no.10,pp.1705{1716,2005. [98] J.E.N.ColemanandR.Jain,\Obtaining3-dimensionalshapeoftexturedandspecularsurfacesusingfour-sourcephotometry,"ComputerGraphicsandImageProcessing,2003. [99] M.JonesandP.Viola,\Facerecognitionusingboostedlocalfeatures,"TechnicalReportTR2003-25,MERL,2003. [100] T.Ahonen,A.Hadid,andM.Pietikainen,\Facerecognitionwithlocalbinarypatterns,"Proc.EuropeanConf.ComputerVision,2004. 113

PAGE 114

RitwikKumarreceivedhisBachelorofTechnologydegreeinInformationandCommunicationTechnologyfromDhirubhaiAmbaniInstituteofInformationandCommunicationTechnology(DAIICT),Gandhinagar,Indiain2005.Since2005hehasbeenaPh.D.studentattheCenterforVision,GraphicsandMedicalImagingattheDepartmentofComputerandInformationScienceandEngineeringattheUniversityofFlorida,Gainesville,FL,USA.Hisresearchinterestsincludemachinelearning,colorvideoanalysis,facerecognitionandmedicalimageanalysis.HeisarecipientofDAIICTPresident'sGoldMedal(2005)andtheUniversityofFloridaAlumniFellowship(2005-2009).HereceivedthebeststudentpaperawardattheTwelfthInternationalConferenceonAdvancedComputingandCommunications(ADCOM)2004. 114