<%BANNER%>

Advanced Techniques for Digital Image Compression and Analysis

Permanent Link: http://ufdc.ufl.edu/UFE0042025/00001

Material Information

Title: Advanced Techniques for Digital Image Compression and Analysis
Physical Description: 1 online resource (163 p.)
Language: english
Creator: Han, Bing
Publisher: University of Florida
Place of Publication: Gainesville, Fla.
Publication Date: 2010

Subjects

Subjects / Keywords: 3d, compressive, image, motion, object, video
Electrical and Computer Engineering -- Dissertations, Academic -- UF
Genre: Electrical and Computer Engineering thesis, Ph.D.
bibliography   ( marcgt )
theses   ( marcgt )
government publication (state, provincial, terriorial, dependent)   ( marcgt )
born-digital   ( sobekcm )
Electronic Thesis or Dissertation

Notes

Abstract: Digital images and videos are widely used in many areas, such as digital TV broadcasting, space imagery and aerial photography, magnetic resonance imaging, traffic monitoring and video surveillance. In this dissertation, we study two important areas, namely, image compression and video analysis. In the first part of this dissertation, we study compressive sensing (CS) and its application to image/video representation and compression. CS theory states that it is possible to recover certain signals and images from far fewer samples or measurements than those required by traditional approaches. We use a CS technique to represent visual data and propose a new image representation scheme in visual sensor networks. Different from the previous works on compressive imaging, which treat the input image as a whole signal, we decompose the visual data into two components before sampling: a dense component and a sparse component. We represent the dense component by the traditional approach and represent the sparse component by compressive sensing. The advantage of our scheme is that we use the correlation of the two components to recover the signal, which helps to reduce the number of measurements and computation time required for reconstruction with the same accuracy. We propose and implement a projection onto convex sets based optimization algorithm to recover the signal. We also propose a new image/video compression system, which combines CS with traditional block based image/video compression schemes, such as JPEG and H.264. In the second part of this dissertation, we study video analysis. There are a lot of image processing areas that employ video analysis. In this dissertation, we attack three problems in video analysis, i.e., image registration, motion analysis, and object tracking. Firstly, we propose a new strategy of image registration by leveraging the depth information via 3D reconstruction. One novel idea is to recover the depth in the image region with high-rise objects to build accurate transform function. The traditional image registration algorithms suffer from the parallax problem due to their underlying assumption that the scene can be regarded approximately planar. Our method overcomes this weakness and achieves more accurate registration performance. Secondly, we propose a new method for motion segmentation based scene interpretation. The segmentation of optical motion field is based on the minimal coding length criterion. The experimental results show that our proposed scheme could greatly improve the performance of motion field segmentation. Finally, to overcome the limitations of the traditional KLT feature tracker, we propose a novel object tracking algorithm. For each object to be tracked, we use a set of KLT features to represent and a weighting function to balance the contribution of different features, according to their position, quality and consistency. The algorithm could adequately track multiple objects of arbitrary shapes in an image sequence with partial occlusion.
General Note: In the series University of Florida Digital Collections.
General Note: Includes vita.
Bibliography: Includes bibliographical references.
Source of Description: Description based on online resource; title from PDF title page.
Source of Description: This bibliographic record is available under the Creative Commons CC0 public domain dedication. The University of Florida Libraries, as creator of this bibliographic record, has waived all rights to it worldwide under copyright law, including all related and neighboring rights, to the extent allowed by law.
Statement of Responsibility: by Bing Han.
Thesis: Thesis (Ph.D.)--University of Florida, 2010.
Local: Adviser: Wu, Dapeng.

Record Information

Source Institution: UFRGP
Rights Management: Applicable rights reserved.
Classification: lcc - LD1780 2010
System ID: UFE0042025:00001

Permanent Link: http://ufdc.ufl.edu/UFE0042025/00001

Material Information

Title: Advanced Techniques for Digital Image Compression and Analysis
Physical Description: 1 online resource (163 p.)
Language: english
Creator: Han, Bing
Publisher: University of Florida
Place of Publication: Gainesville, Fla.
Publication Date: 2010

Subjects

Subjects / Keywords: 3d, compressive, image, motion, object, video
Electrical and Computer Engineering -- Dissertations, Academic -- UF
Genre: Electrical and Computer Engineering thesis, Ph.D.
bibliography   ( marcgt )
theses   ( marcgt )
government publication (state, provincial, terriorial, dependent)   ( marcgt )
born-digital   ( sobekcm )
Electronic Thesis or Dissertation

Notes

Abstract: Digital images and videos are widely used in many areas, such as digital TV broadcasting, space imagery and aerial photography, magnetic resonance imaging, traffic monitoring and video surveillance. In this dissertation, we study two important areas, namely, image compression and video analysis. In the first part of this dissertation, we study compressive sensing (CS) and its application to image/video representation and compression. CS theory states that it is possible to recover certain signals and images from far fewer samples or measurements than those required by traditional approaches. We use a CS technique to represent visual data and propose a new image representation scheme in visual sensor networks. Different from the previous works on compressive imaging, which treat the input image as a whole signal, we decompose the visual data into two components before sampling: a dense component and a sparse component. We represent the dense component by the traditional approach and represent the sparse component by compressive sensing. The advantage of our scheme is that we use the correlation of the two components to recover the signal, which helps to reduce the number of measurements and computation time required for reconstruction with the same accuracy. We propose and implement a projection onto convex sets based optimization algorithm to recover the signal. We also propose a new image/video compression system, which combines CS with traditional block based image/video compression schemes, such as JPEG and H.264. In the second part of this dissertation, we study video analysis. There are a lot of image processing areas that employ video analysis. In this dissertation, we attack three problems in video analysis, i.e., image registration, motion analysis, and object tracking. Firstly, we propose a new strategy of image registration by leveraging the depth information via 3D reconstruction. One novel idea is to recover the depth in the image region with high-rise objects to build accurate transform function. The traditional image registration algorithms suffer from the parallax problem due to their underlying assumption that the scene can be regarded approximately planar. Our method overcomes this weakness and achieves more accurate registration performance. Secondly, we propose a new method for motion segmentation based scene interpretation. The segmentation of optical motion field is based on the minimal coding length criterion. The experimental results show that our proposed scheme could greatly improve the performance of motion field segmentation. Finally, to overcome the limitations of the traditional KLT feature tracker, we propose a novel object tracking algorithm. For each object to be tracked, we use a set of KLT features to represent and a weighting function to balance the contribution of different features, according to their position, quality and consistency. The algorithm could adequately track multiple objects of arbitrary shapes in an image sequence with partial occlusion.
General Note: In the series University of Florida Digital Collections.
General Note: Includes vita.
Bibliography: Includes bibliographical references.
Source of Description: Description based on online resource; title from PDF title page.
Source of Description: This bibliographic record is available under the Creative Commons CC0 public domain dedication. The University of Florida Libraries, as creator of this bibliographic record, has waived all rights to it worldwide under copyright law, including all related and neighboring rights, to the extent allowed by law.
Statement of Responsibility: by Bing Han.
Thesis: Thesis (Ph.D.)--University of Florida, 2010.
Local: Adviser: Wu, Dapeng.

Record Information

Source Institution: UFRGP
Rights Management: Applicable rights reserved.
Classification: lcc - LD1780 2010
System ID: UFE0042025:00001


This item has the following downloads:


Full Text

PAGE 2

2

PAGE 3

toShanshan,mybelovedwifeforherpatience,understandingandsupport; toDadandMomforinstillingtheimportanceofcuriosityandhardwork; toJing,myeldersisterforherencouragement. 3

PAGE 4

AtUniversityofFlorida,especiallyinmyresearchgroup,Ihavehadtheprivilegetoworkwithmanytalentedindividuals,whohavemadecontributionstomyresearchexperience.Firstofall,IwanttothankmyadvisorProfessorDapengWuforallthehopehehasputonme,beforeIthoughtIcoulddoanyresearchatall.ProfessorDapengWuisanexcellentrolemodelformewhoreallyknowshowtobalancescienticresearch,teaching,andfamily.Withouthisinspirationalguidance,hisencouragements,hisenthusiasm,andhisunselshhelp,IcouldnevernishmydoctoralworkinUniversityofFlorida.Hehasalwaysencouragedmetoliveintensively,andtaughtmehowtoappreciatethegoodscienticworkandlife.IalsowouldliketothankProfessorScottBanks,ProfessorTaoLiandProfessorYijunSunforservingonmydissertationcommittee.Theyhaveprovidedmanyvaluablesuggestionsonmyresearchanddissertation.IamthankfultomyfellowstudentsandthevisitingscholarsinMultimediaCommunicationsandNetworkingLab.Dr.JieyanFanandDr.XiaochenLihavehelpedmealotinmyearlydaysatUF.ThankstoDr.JunXu,WenxingYe,ZhifengChenandTaoranLuforvaluablediscussionsandhelpinmyresearch.IwouldliketothankDr.XihuaDong,YiranLi,LeiYang,ZongruiDing,QianChen,YakunHu,JiangpingWang,YuejiaHe,HuanghuangLiandZhengYuan.Finally,IwanttoexpressmyappreciationtomywifeShanshanRen,myparents,YuqingHanandJinrongLiu,andmysisterJingHan,fortheirlove,understanding,patience,endlesssupport,andneverfailingfaithinme. 4

PAGE 5

page ACKNOWLEDGMENTS ................................. 4 LISTOFTABLES ..................................... 9 LISTOFFIGURES .................................... 10 ABSTRACT ........................................ 13 CHAPTER 1INTRODUCTION .................................. 15 1.1Motivation .................................... 17 1.1.1CompressiveSensinginImageCompression .............. 17 1.1.2VideoAnalysis .............................. 18 1.2OutlineoftheDissertation ........................... 19 2COMPRESSIVESENSINGFORIMAGEPROCESSINGAPPLICATIONS:ANOVERVIEW ................................... 22 2.1Introduction ................................... 22 2.2CompressiveSensingTheory .......................... 23 2.3RecoveryAlgorithms .............................. 26 2.3.1Algorithmwithl1Constraint ...................... 26 2.3.2AlgorithmswithlpConstraint ..................... 28 2.4CSMeasurementEnsembles .......................... 29 2.5CSforImageProcessing ............................ 29 3VideoAnalysis:ANOVERVIEW .......................... 32 3.1MotionAnalysis ................................. 32 3.2ObjectTracking ................................. 34 3.2.1PointTrackingAlgorithm ........................ 34 3.2.2KernelTrackingAlgorithm ....................... 35 3.2.3SilhouetteTrackingAlgorithm ..................... 36 3.3ImageRegistration ............................... 37 4COMPRESSIVESENSINGBASEDIMAGEREPRESENTATION ....... 40 4.1Introduction ................................... 40 4.2OverviewofCompressiveSensing ....................... 42 4.3ProposedImageRepresentationScheme ................... 43 4.3.1ReconstructionErrorBounds ...................... 44 4.3.2ImageDecomposition .......................... 45 4.3.3CorrelationbetweenSparseandDenseComponents ......... 47 4.4PracticalSignalReconstruction ........................ 48 5

PAGE 6

..................... 48 4.4.2ImageReconstructionAlgorithm .................... 50 4.5ExperimentalResults .............................. 51 4.6Conclusion .................................... 53 5COMPRESSIVESENSINGINBLOCKBASEDIMAGE/VIDEOCODING .. 60 5.1Introduction ................................... 60 5.2CompressiveSensingTheory .......................... 61 5.3NewImage/VideoCompressionScheme .................... 63 5.4CSRecoveryinDecompression ........................ 63 5.4.1TVMinimizationOptimization .................... 64 5.4.2BoundedResidueConstraint ...................... 65 5.5ExperimentalResult .............................. 66 5.6Conclusion .................................... 67 63DGEOMETRICSEGMENTATIONANDFITTING .............. 71 6.1Introduction ................................... 71 6.2The3DGeometricFittingProblem ...................... 73 6.2.1DeterministicAnnealing ........................ 75 6.2.2Non-linearPartitioning ......................... 76 6.3Non-linearDeterministicAnnealing ...................... 77 6.4ExperimentalResults .............................. 80 6.4.1NDAonSyntheticDatawithoutNoise ................ 80 6.4.2NDAonSyntheticDatawithNoise .................. 81 6.4.3NDAonRealWorldData ....................... 81 6.5Conclusion .................................... 82 73DDENSERECONSTRUCTIONFROM2DVIDEOSEQUENCE ....... 89 7.1Introduction ................................... 89 7.2BackgroundandProblemFormation ..................... 90 7.2.13DReconstruction ........................... 91 7.2.2GeometricFitting ............................ 93 7.33DVideoReconstruction ............................ 94 7.3.1Overviewof3DReconstructionSystem ................ 94 7.3.2FeatureSelection ............................ 94 7.3.3FeatureCorrespondence ........................ 95 7.3.4EstimationofCameraMotionParameters ............... 96 7.3.5DepthEstimation ............................ 97 7.3.6GeometricSegmentation ........................ 97 7.3.7DepthRecovery ............................. 98 7.4GeometricSegmentationbasedDenseReconstruction ............ 99 7.4.1Non-linearDeterministicAnnealing .................. 100 7.5ExperimentalResults .............................. 102 7.5.13DVideoDenseReconstruction .................... 102 6

PAGE 7

.................................... 103 8IMAGEREGISTRATION .............................. 108 8.1Introduction ................................... 108 8.2ANewArchitecturefor2DImageRegistrationwithDepthInformation .. 110 8.33DReconstructionfrom2DVideoSequence ................. 111 8.3.1FeatureSelection ............................ 111 8.3.2FeatureCorrespondence ........................ 112 8.3.3EstimationofCameraMotionParameters ............... 113 8.3.4DepthEstimation ............................ 114 8.4ImageRegistrationwithDepthInformation ................. 114 8.4.1GeometricalSegmentation ....................... 114 8.4.2DepthEstimation ............................ 118 8.5ExperimentalResults .............................. 119 8.6Conclusion .................................... 120 9MOTIONSEGMENTATION ............................ 126 9.1Introduction ................................... 126 9.2OpticalFlowField ............................... 127 9.3MotionFieldSegmentation ........................... 129 9.3.1MinimalDescriptionLengthCriterion ................. 129 9.3.2CodingLengthbasedOpticalFieldSegmentation .......... 130 9.3.3MinimizingtheCodingLength ..................... 131 9.4GlobalMotionEstimation ........................... 131 9.5ExperimentResults ............................... 132 9.5.1MotionFieldSegmentation ....................... 132 9.5.2GlobalMotionEstimation ....................... 132 9.6Conclusion .................................... 133 10OBJECTTRACKING ................................ 138 10.1Introduction ................................... 138 10.2SystemOverview ................................ 138 10.3ObjectDetection ................................ 140 10.4ObjectTracking ................................. 142 10.4.1KLTFeatureSelectionandTracking .................. 142 10.4.2TrajectoryEstimationandFeatureUpdate .............. 143 10.4.3OcclusionHandling ........................... 144 10.5ExperimentalResults .............................. 145 10.6Conclusion .................................... 145 11CONCLUSION .................................... 149 11.1SummaryofthisDissertation ......................... 149 11.1.1CompressiveSensinginImageCompression .............. 149 7

PAGE 8

.............................. 150 11.2FutureWork ................................... 151 REFERENCES ....................................... 152 BIOGRAPHICALSKETCH ................................ 163 8

PAGE 9

Table page 4-1Comparisonofreconstructionresultswiththesamenumberofmeasurements .. 54 5-1Experimentalresultsofthetestblocksfrom`Cameraman' ............ 67 5-2ExperimentalresultsofBCS ............................. 67 6-1Theaveragesquaredapproximationerror ...................... 82 6-2Thecorrectidenticationrate ............................ 83 6-3Theaveragesquaredapproximationerror ...................... 84 9

PAGE 10

Figure page 1-1Sourceencoderanddecodermodel ......................... 16 4-1Onedimensionalsignalrecoverybycompressivesensing .............. 54 4-2Comparisonoferrorbounds ............................. 55 4-3Wavelettransformof`Lena' ............................. 55 4-4Powerlawmodelofsparsesignalcoecients .................... 56 4-5Imageinterpolationscheme ............................. 56 4-6Imagerepresentationscheme ............................. 57 4-7Predictionofthesparsecomponentof`Lena' .................... 57 4-8Comparisonofpredictionandoriginalsignal .................... 58 4-9Comparisonoferrorreductionwithsamenumberofiterations .......... 58 4-10Recoveredimage`Lena'from20000measurements ................ 59 4-11Recoveredimage`Boats'from20000measurements ................ 59 5-1Theowchartofthenewcompressionscheme ................... 68 5-2Thetestimage`Cameraman' ............................ 69 5-3Thetestimageblocksin`Cameraman' ....................... 69 5-4Thetestimageblocks ................................ 69 5-5Testresultof`Boats' ................................. 70 5-6Testresultof`Cameraman' ............................. 70 6-1Thesyntheticdataset ................................ 84 6-2TherstgrouppartitionedbyK-means ....................... 85 6-3Theinputdatapointsontherstframe ...................... 86 6-4ThegeometricalsegmentationresultbytheNDAalgorithm ........... 87 6-5ThegeometricalsegmentationresultbythePIalgorithm ............. 88 7-1Thepipelinefor3Dvideoreconstructionsystem .................. 104 7-2Theschemefor3Dvideoreconstructionsystem .................. 105 10

PAGE 11

..................... 105 7-4Thefeaturepointsontherstframe ........................ 106 7-5Theestimatedsparsedepthmapandcamerapose ................. 107 7-6Theestimateddense3Dconguration ....................... 107 8-1Thepipelinefor2Dimageregistrationsystem ................... 121 8-2Thenewimageregistrationsystemscheme ..................... 122 8-3Ouralgorithmtestresult ............................... 122 8-4ThetestresultofDavisandKeck'salgorithm ................... 123 8-5Thedierenceimageofouralgorithm. ....................... 123 8-6ThedierenceimageofDavisandKeck'salgorithm ................ 124 8-7The37thframeinthe`oldhousing'videosequence ................. 124 8-8Ouralgorithmtestresult ............................... 125 8-9OuralgorithmtestresultcomparingtoDavisandKeck'salgorithm ....... 125 9-1Thesecondframeofimagesequence ........................ 133 9-2Thefourthframeofimagesequence ......................... 134 9-3Theopticaloweldoftheinputimagesequence ................. 134 9-4Therstframeofimagesequence`Coastguard' .................. 134 9-5Theopticaloweldofimagesequence`Coastguard' ............... 135 9-6The`ship'segmentin`Coastguard' ......................... 135 9-7The`boat'segmentin`Coastguard' ......................... 135 9-8The`land'segmentin`Coastguard' ......................... 136 9-9The`river'segmentin`Coastguard' ......................... 136 10-1Flowchartofourmultipleobjecttrackingsystem ................. 146 10-2Segmentationofthe20thframefromthe`Coastguard'imagesequence ..... 146 10-3Segmentationresultofthe20thframeaftercorrection ............... 147 10-4Therstframeinthe`Coastguard'imagesequence ................ 147 10-5Thelastframeofthe`Coastguard'imagesequence ................ 148 11

PAGE 12

.............. 148 12

PAGE 13

Digitalimagesandvideosarewidelyusedinmanyareas,suchasdigitalTVbroadcasting,spaceimageryandaerialphotography,magneticresonanceimaging,tracmonitoringandvideosurveillance.Inthisdissertation,westudytwoimportantareas,namely,imagecompressionandvideoanalysis. Intherstpartofthisdissertation,westudycompressivesensing(CS)anditsapplicationtoimage/videorepresentationandcompression.CStheorystatesthatitispossibletorecovercertainsignalsandimagesfromfarfewersamplesormeasurementsthanthoserequiredbytraditionalapproaches.WeuseaCStechniquetorepresentvisualdataandproposeanewimagerepresentationschemeinvisualsensornetworks.Dierentfromthepreviousworksoncompressiveimaging,whichtreattheinputimageasawholesignal,wedecomposethevisualdataintotwocomponentsbeforesampling:adensecomponentandasparsecomponent.Werepresentthedensecomponentbythetraditionalapproachandrepresentthesparsecomponentbycompressivesensing.Theadvantageofourschemeisthatweusethecorrelationofthetwocomponentstorecoverthesignal,whichhelpstoreducethenumberofmeasurementsandcomputationtimerequiredforreconstructionwiththesameaccuracy.Weproposeandimplementaprojectionontoconvexsetsbasedoptimizationalgorithmtorecoverthesignal.Wealsoproposeanewimage/videocompressionsystem,whichcombinesCSwithtraditionalblockbasedimage/videocompressionschemes,suchasJPEGandH.264. 13

PAGE 14

14

PAGE 15

Withthefastdevelopmentofdigitalcamerasanddisplaydevices,digitalimagesandvideosarebecomingmoreandmorepopularinoureverydaylives.Adigitalimageiscomposedofanitenumberofpixels.Digitalimagesandvideosarewidelyusedinmanyareas,suchasdigitalTVbroadcasting,spaceimageryandaerialphotography,magneticresonanceimaging,tracmonitoringandvideosurveillance.Sincetheopticaldevicescancoveralmosttheentireelectromagneticspectrum,thecapturedimagesneedtobeprocessedforhumanvision.Generally,digitalimageprocessingreferstoprocessingdigitalimagesviadigitalcomputers.Sinceweareusingcomputerstoprocessandanalyzethedigitalimages,thereisnoclearboundarybetweendigitalimageprocessing,computervisionandimageanalysis.Computervisionstudieshowtousecomputerstoemulatehumanvisionandimageanalysis,orsceneanalysisistousecomputerstounderstandtheimages.Althoughtherearenoclear-cutboundariesbetweentheseresearchelds,theprocessesareclassiedintothreelevels,lowlevel,midlevelandhighlevel.Thelowlevelprocessesincludesnoisereduction,imageenhancement,andimagecompression.Theinputsandoutputsofalowlevelprocessarebothimages.Themidlevelprocessesinvolvesfeatureextraction,edgedetection,andimagesegmentation.Theinputsofamidlevelprocessarestillimages,buttheoutputsaretexturefeaturesextractedfromtheinputimages.Thehighlevelprocessesinvolvesobjectrecognition,objecttrackingandmotionanalysis. Inthisdissertation,wefocusontwoimportantareas,i.e.,imagecompressionandvideoanalysis. Everyday,anenormousvolumeofdigitalimagesandvideosisgenerated,stored,processedandtransmitted.Imagecompressionisnecessaryforhandlingthelargespatialresolutionsoftoday'simagingsensors.Imagecompressioniswidelyusedinavarietyofapplications,suchasvideoteleconferencing,remotesensing,remotedesktop,andvideo 15

PAGE 16

Sourceencoderanddecodermodel. streaming.Thebasicideaofimagecompressionistoremovetheredundantdatafromtherawdigitalimages.Therearetwotypeofimageredundancies,namely,interpixelredundancyandpsychovisualredundancy.Interpixelredundancyrelatestotheinterpixelcorrelationswithinanimage,whichisalsoknownasspatialredundancyandgeometricredundancy.Psychovisualredundancyisassociatedwiththelimitationofhumanvisualprocessing.Becauseahumanbeingisnotabletoperceiveallvisualinformationinanimage,theinformation,whichisnotessentialfornormalhumanvisualprocessing,iseliminable.AconventionalimagecompressionsystemisshowninFigure 1-1 Generally,therearethreestagesintheencodingprocess.Intherststage,atransformfunctionblocktransformstheinputdataf(x,y)intoatransformdomaintoreduceinterpixelredundancies.Thetransformoperationisusuallyreversibleandtheimageistransformedintoanarrayofcoecients.Inthesecondstage,anquantizerblockperformsquantizationontheinputcoecients.Thequantizationisusefultoreducethepsychovisualredundanciesoftheinputimage.Usuallythisoperationisirreversible.Thelaststageissourceencodingprocessanditisreversible.Accordingly,thedecodingprocesscontainsthreestages,symboldecoder,inversequantizer,andinversetransform. Inthepasttwodecades,quiteafewcompressionmethodsareproposed,suchasJPEGandJPEG2000forimagecompression,andH.263andH.264forvideocompression.Asisknowntoall,naturalimagesarepiecewisesmoothandhighlycompressibleontransformdomain.Therefore,allofthepreviouscompressionmethodsarebasedon 16

PAGE 17

Withthelargevolumeofdigitalimagesandvideos,itishardforhumanstoanalyzethedatamanually.Takevideosurveillanceasanexample.Videosurveillancewasinitiallydevelopedforsecurityreasons.Supposeweneedtondasuspectfromavideosequencewiththelengthofoneday,itmaytakeahumanseveralhourstolocatetheframescontainingthesuspect.Withcomputervisiontechnology,wecoulduseacomputertohelplocatingthesuspectwiththesamevideosequenceinonlyafewminutes.Therearealotofapplicationareasthatemploycomputervision,suchaspatternrecognition,motionanalysis,objecttrackingandimageregistration.Inthisdissertation,wewillattackthreeproblemsinthisarea,motionanalysis,objecttracking,andimageregistration. 1.1.1CompressiveSensinginImageCompression 17

PAGE 18

Withthemotioninformationofthevideosequence,objecttrackingistoestimateandanalyzethetrajectoryofanobjectintheimageplaneasitmovesthroughavideosequence.Objecttrackinghasbeenwidelyusedinvideosurveillance,videoindexing,tracmonitoringandmotionbasedrecognition.Thechallengeinobjecttrackingistoassociatetargetlocationsinconsecutiveframes,especiallyinmultipletargetstrackingproblem.Whentherearemultipleobjectsmovingatthesametime,objectocclusionhappensandthetargetmaynotbevisibleonafewframes,whichmaycausesevereproblem. Imageregistrationisanotherfundamentalproblemincomputervision.Becauseofthecameramotion,thesetofimagesofthesamescenemaybeindierentcoordinatesystemswhenacquiredatdierenttimesorfromdierentperspectives.Imageregistrationistotransformthesequenceofimagesbackintoonecoordinatesystem.Itisacrucialstepinallimageanalysistasksinwhichnalinformationisgainedfromcombinationofvariousdatasource,suchas,changedetection,imagemosaicing,andintegratinginformationintogeographicinformationsystems. 18

PAGE 19

InChapters2and3,weoverviewthetechniquesinthetwoareasrespectively. InChapter4,weusecompressivesensingtorepresentvisualdataandproposeanewimagerepresentationschemeinvisualsensornetworks.Dierentfromthepreviousworksoncompressiveimaging,whichtreattheinputimageasawholesignal,wedecomposethevisualdataintotwocomponentsbeforesampling:adensecomponentandasparsecomponent.Wesamplethedensecomponentbythetraditionalapproachandthesparsecomponentbycompressivesensing.Theadvantageofourworkisthatweusethecorrelationofthetwocomponentstorecoverthesignal,whichhelpstoreducethenumberofmeasurementsandcomputationtimerequiredforreconstructionwiththesameaccuracy.Weproposeandimplementaprojectionontoconvexsetsbasedoptimizationalgorithmtorecoverthesignal. InChapter5,weproposeanewimage/videocompressionsystem,whichcombinescompressivesensingintotraditionalblockbasedimage/videocompressionschemes,suchasJPEGandH.264.Asknowntoall,mostofthecodingerrorintraditionallossycompressionmethodsiscausedbyscalarquantization.CSrecoverytendstosolveaoptimizationproblemtoreconstructtheoriginalsignal,whichcanhelpmitigatingthequantizationerrorindecodingprocess.WealsoproposeaboundedresidueconstrainttobeusedinCSreconstructiontofurtherimprovereconstructionaccuracy. InChapter6,weproposeakerneldeterministicannealingapproachforgeometricttingin3Dspace.Duetothefactthatthe3Ddataislocalizedtoafewrelativelydenseclusters,wedesignakernelfunctiontomapthedatapointfromgeometricalspacetosurfacemodelspaceandapplydeterministicannealingtopartitionthefeaturespaceintoseparateregions.Foreachregion,wecaneasilyndalinearplanemodeltotthedata. 19

PAGE 20

InChapter8,weproposeanewarchitectureforimageregistrationbyleveragingthedepthinformationvia3Dreconstruction.Onenovelideaistorecoverthedepthintheimageregionwithhigh-riseobjectstobuildaccuratetransformfunction.Thetraditionalimageregistrationalgorithmssuerfromparallaxproblemduetotheirunderlyingassumptionthatthescenecanberegardedapproximatelyplanar.However,theassumptionisnotvalidanymoreinthecaseoflargedepthvariationintheimageswithhigh-riseobjects.Ourmethodovercomesthisweaknessandachievesmoreaccurateregistrationperformance.Ouralgorithmisattractivetomanypracticalapplications. InChapter9,weproposeanovelapproachtoestimatemodelparametersofallmotionsbasedonsegmentationofbothintensitymapandopticaloweld.Thenoveltyofourworkisthatweintroduceminimumcodinglengthasacriterioningroupmerging.Theexperimentalresultsshowthatourproposedschemecouldgreatlyimprovetheperformanceofmotioneldsegmentation.Anothernoveltyisthatweproposeaheuristicapproachtolocateglobalmotionbasedonthemotionsegments. InChapter10,weproposeanewtrackingalgorithmbasedonbothtemplatetrackingandsilhouettetracking.Thealgorithmattemptstoadequatelytrackmultipleobjectsofarbitraryshapesinanimagesequence.Inordertoaccuratelyestimatethetrajectory,werstgenerateabinaryobjectmaskandthenonlytrackthefeaturesinsidethemask.ToovercomethelimitationsofthetraditionalKLTfeaturetracker,weproposeanoveltrajectoryestimationalgorithmbasedonaweightingfunctionoftrackedfeaturemotionvectors. 20

PAGE 21

21

PAGE 22

1 ]providessomekeymathematicalinsightsunderlyingthisnewtheory. Duetoalargenumberofnewworkspublishedinthisarea,thischapterwillnotbeanexhaustivesurveyofliteratureoncompressivesensingbutconcentrateonsignalrecoveryalgorithmsandframeworksofapplyingcompressivesensingtoimage/videoprocessing.Therestofthischapterisorganizedasfollows.Section 2.2 givesabriefreviewofcompressivesensingtheory.Section 2.3 providesacomparisonofcurrentsignalreconstructionalgorithmsundercompressivesensingprinciple.Section 2.4 comparestheperformanceofCSreconstructionunderseveraldierentsamplingensembles.Section 2.5 discusseshowtoemployCSinimageprocessing. 22

PAGE 23

2 3 ]andDonoho[ 4 ]demonstratesthatinformationcontainedinafewsignicantcoecientscanbecapturedbyasmallnumberofrandomlinearprojections.Theoriginalsignalcanthenbereconstructedfromtherandomprojectionsusinganappropriatedecodingscheme. WeadoptlanguageandnotationfromCandesetal.'spaper[ 5 ];Lettheobjectofinterestbeadiscretetimesignalf2Rn-thiscouldrepresentansampledvaluesofadiscretetimesignalorimage.Thereasonwefocusondiscretetimesignalinsteadofcontinuoustimesignalistwofold:rstly,thenaturalimagesandvideosaregenerallytakenasdiscretesignalsandsecondly,itisconceptuallysimplerandtheavailablediscreteCStheoryismoredeveloped.Weareinterestedinthesituationinwhichthenumberofmeasurementsismuchsmallerthanthedimensionofthesignalf.Itisespeciallyusefulwhenthemeasurementisextremelyexpensive. InCStheory,thesignalfisobtainedbyanumberoflinearfunctionals. Themeasurementprocessissimplytocorrelatethesignaltobeacquiredwithwaveformk.Thenthequestioniswhetherwecanreconstructthesignalfrommmeasurements,wherem<
PAGE 24

Mathematicallyspeaking,supposewecanexpandsignalf2Rninanorthonormalbasis=[12...n]: wherexisthecoecientsequenceoffindomain.Sparsityrequiresthatthereislittleperceptuallosswhendiscardingmostofthesmallvaluesinx.DenotefK=xKasthesignalobtainedbykeepingonlyKlargestcoecientsandsetallotherszero,thenfKisexactlyK-sparse.Becauseisanorthonormalbasis,wehavekffKkl2=kxxKkl2.IfxKisagoodapproximationofx,thentheerrorkffKkl2issmall. Thedenitionofcoherencebetweentwoorthobasesandis Thecoherencemeasuresthelargestcorrelationbetweenanytwoelementsoftwoorthobasesand.CSrequiresmeasurementbasisandsparsebasistobealowcoherencepair.Usually,CSuserandommatricesasmeasurementmatricessinceitisproventhatrandommatricesarelargelyincoherentwithanyxedbasis. Withthesupportofsparsityandincoherence,CSreconstructthesignalfwithncoecientsfromonlymmeasurementsbysolvinganoptimizationproblem. 24

PAGE 25

Furthermore,CSisabletodealwithnearlysparsesignalwithnoise.Supposewehavethefollowingproblemwhichisslightlydierentfromequation( 2{1 ): whereAisthemnmeasurementmatrixandzisthenoiseterm.Thequestionnowiscanweusethesameorsimilarwaytoreconstructxfrom\downsampled"measurementy? 6 ].RIPguaranteesthatwithapropermatrixA,allsubsetsofKcolumnstakenfromAareinfactalmostorthogonal.WithRIP,wecanusethefollowingmethodtoreconstructx: whereisthevarianceofnoise. Givenalltheinformationabove,therearestillseveralimportantquestionstobeanswered. 25

PAGE 26

7 ],gureprominentlyinthedesignoftractableCSdecoders,highcomplexityO(N3)makesthemimpracticalformanyapplications.Weoftenencountersparsesignalswithlargedimensions.Forexample,currentdigitalcamerasacquireimageswiththenumberofpixelsNoftheorderof106ormore.Forsuchapplications,theneedforfasterdecodingalgorithmsiscritical.Inthissection,wewillreviewcurrentreconstructionalgorithmsandcomparetheperformanceofdierentalgorithms.Weclassifythealgorithmsintotwocategories:algorithmswithl1constraint,andalgorithmswithlpconstraint,wherep<1. 8 { 10 ].OMPrequiresMKln(N)measurementstosucceedwithhighprobabilityandthedecodingcomplexityisO(NK2).Unfortunately,O(NK2)iscubicinNandK,thereforeOMPisimpracticalforlargeKandN. Donohoetal.[ 11 ]recentlyproposedtheStage-wiseOrthogonalMatchingPursuit(StOMP).StOMPisanenhancedversionofOMPwheremultiplecoecientsareresolvedateachstageofthegreedyalgorithm,asopposedtoonlyoneinthecaseofOMP.Moreover,StOMPtakesaxednumberofstageswhileOMPtakesmorestagestorecoverlargercoecientsofx.TheauthorsshowthatStOMPwithfastoperatorsof(suchaspermutedFFTs)canrecoverthesignalinNlogNcomplexity.ThereforeStOMPrunsmuchfasterthanOMPandcanbeusedforsolvinglarge-scaleproblems. 26

PAGE 27

12 13 ]isproposedifsparsesignalshavedistinct\connectedtree"structureinwaveletdomain.TMPalgorithmsignicantlyreducesthesearchspacecomparedtotraditionalmatchingpursuitgreedyalgorithms,resultinginasubstantialdecreaseincomputationalcomplexityforrecoveringpiecewisesmoothsignals.AnotheradvantageofTMPisthatitperformsimplicitregularizationtocombatnoiseinreconstruction.TMPalsoappliestomoregeneralcaseof\incoherent"measurementvectors. Sudocodesisanewschemeforlosslesscompressivesamplingandreconstructionofsparsesignals.Sarvothametal.[ 14 ]proposedanon-adaptivereconstructionalgorithmforsparsecomprisingonlythevalues0and1;hencethecomputationofmeasurementinvolvesonlysumsofsubsetsoftheelementsofthesignal.Anaccompanyingsudodecodingstrategyecientlyrecoversthesignalgiventhemeasurements.SudocodesrequireM=O(Klog(N))measurementsforexactreconstructionwithworst-casecomputationalcomplexityO(Klog(K)log(N)).Sudocodescouldbeusedaserasurecodesforreal-valueddataandhavepotentialapplicationsinpeer-to-peernetworksanddistributeddatastoragesystems.Itcouldalsobeeasilyextendedtosignalsthataresparseinarbitrarybasis. InBioucas-DiasandFigueiredo'spaper[ 15 ],theyintroducedatwo-stepiterativeshrinkagethreshold(TwIST)algorithm,exhibitingmuchfasterconvergenceratethanIST.TheyshowedthatTwISTconvergestoaminimizerofobjectivefunctionwithagivenrangeofvaluesofitsparametersforavastclassofnon-quadraticconvexregularizers(lpnorms,someBesovnorms,andtotalvariation).Fornon-invertibleobservationoperators,theyintroduceamonotonicversionofTwIST(MTwIST);althoughtheconvergenceproofdoesnotapplytothisscenario,theygiveexperimentalevidencethatMTwISTexhibitssimilarspeedgainsoverIST.TheeectivenessofTwISTisexperimentallyconrmedonproblemsofimagedeconvolutionandrestorationwithmissingsamples. 27

PAGE 28

Chantrand'spaper[ 16 ]showsthatexactreconstructionispossiblewithsubstantiallyfewermeasurementsbyreplacingl1normwithlpnorm,whenp<1.Hegivesatheoreminthisdirection,andmanynumericalexamples,bothinonecomplexdimension,andlargerscaleexamplesintworealdimensions. InCandesetal.'spaper[ 17 ],theyproposedanovelmethodforsparsesignalrecoverythatoutperformsl1minimizationalgorithmsinmanysituationsinthesensethatsubstantiallyfewermeasurementsareneededforexactrecovery.Thealgorithmsolvesasequenceofweightedl1minimizationproblemswheretheweightsusedfornextiterationarecomputedfromthevalueofcurrentsolution.Theypresentaseriesofexperimentsdemonstratingtheremarkableperformanceandbroadapplicabilityofthisalgorithminareasofsparsesignalrecovery,statisticalestimation,errorcorrectionandimageprocessing. InCharchandandYin'paper[ 18 ]theyfurtherconsideredusingiterativelyre-weightedalgorithmstocomputelocalminimaofnonconvexproblems.Inparticular,aregularizationstrategyisfoundtogreatlyimprovetheabilityofre-weightedleast-squaresalgorithmtorecoversparsesignals,withexactrecoveryobservedforsignalsthataremuchlesssparsethanrequiredbyunregularizedversion.Improvementsarealsoobservedforreweightedl1approachofCandesetal.'swork[ 17 ]. InDaviesandBlumensath'spaper[ 19 ],theyderivedtwoalgorithmsthatoperatedirectlyonl0regularizedcostfunctionandM-sparseconstrainedoptimizationproblem,respectively.Theyderivednoveltheoreticalresultsforthemethodsandproposedtousethemintwocontexts.Firstly,themethodscouldbeusedtoimprovetheresultscalculatedbyothermethodssuchasMPmethod.Secondly,theyshowedthatthemethodscouldbe 28

PAGE 29

20 ],theycomparefourrandomensemblesusedinCS.Herewegiveabriefreview. Thesechoicesareinspiredbythefollowingearlierworks.Donoho[ 4 ]provedthatrandomsignsmatricesanduniformSphericalensemblesaresuitabletobeusedwhenp=1.Candesetal.[ 2 ]recentlyhavegeneratedagreatdealofexcitementbyshowingseveralinterestingpropertiesofrandompartialFouriermatricesandmakingclaimsabouttheirpossibleuseinCS.Donohocomparedthequasiboundwithactualerrorsindierentmatrixensemblesjustdened.Itpromptsseveralobservations.Firstofall,thesimulationresultsusingdierentensemblesareallqualitativelyinagreementwiththetheoreticalformoferrorbehavior.Moreover,itisobservedthatdierentensemblesshowsimilarbehavior.Thissuggeststhatallsuchensemblesareequallygoodinpractice. 29

PAGE 30

21 ]rstshowedpracticalsignicanceofCStheory.Intheirwork,theyapplyCSrecoveryalgorithmonaseriesofsignalsandnaturalimageswhichcanbewellapproximatedinwaveletbases.Theframeworkistotallydierentfromcurrentimagecompressionmethods.Thepreviousmethods,suchasJPEG2000tendtorepresentanimagewithitsMlargestwaveletcoecients.CSmeasurementsaremadecompletelyrandomandtheyhavenothingtodowiththestructureoftheimage.Fromtheirexperiment,itispossibletorecoveranimagefromabout3Mto5MprojectionsontogenericallychosenvectorswiththesameaccuracyastheidealM-termwaveletapproximation. Gan[ 22 ]furtherdevelopedthisideaandproposedablock-basedsamplingmethodforfastCSofnaturalimages.Theoriginalimageisdividedintosmallblocksandeachblockissampledindependentlyusingthesamemeasurementoperator.ThepossibilityofexploitingblockCSismotivatedbythegreatsuccessofblockDCTcodingsystemswhicharewidelyusedinJPEGandMPEGstandards.Fornaturalimages,thepreliminaryresultsshowthatblockCSsystemsoercomparableperformancestoexistingCSschemeswithmuchsmallerimplementationcost. Zhangetal.[ 23 ]proposedanewimage/videocodingframeworkwhichcombinesCStheoryintotraditionalimage/videocompressionapproaches.TheyassumeCSsampling/recoveryalgorithmismoresuitableforimageblockswithsparsegradientswhileconventionalDCTbasedmethodismoresuitableforcomplicatedimageblocks.Therefore,intheirframework,CSsampling/recoveryalgorithmisintegratedintoJPEGandH.264/AVCcodingmethodsasanewcodingmodeandrate-distortionoptimization(RDO)isemployedformodeselectionbetweenthenewcodingmodeandconventionalcodingmodes.Each88imageblockisencodedanddecodedineitherDCTcodingmodeorCScodingmode.Theexperimentalresultsshowthatwiththenewcodingmode,theaveragebitratereductionisapproximately4%. 30

PAGE 31

24 ]proposedamodel-guidedadaptiverecoveryofcompressivesensing,sothesignalcanberecoveredfaithfully.Inthenewframework,apiecewisestationaryautoregressivemodelisintegratedintorecoveryprocessforCS-codedimages.Comparingtototalvariation(TV)basedcompressivesensingcodingalgorithm,thereconstructionqualityisincreasedby2to7dB. 31

PAGE 32

25 ].Thetraditionalapproachtoanalyzeasingleframeisthroughimagesegmentation[ 26 ].However,incasewehaveasequenceofimagesandthereareseveralobjectsmovinginthescene,orobjectsatdierentdepthswithaglobalcameramotion,motiondiscontinuitywilloccur.Inthiscase,analysisofdierentobjectmotionscanprovidemoreessentialinformationforunderstandingofthescene[ 27 ].Therefore,motionsegmentationisneededtosegmenttheimageframeintoregionsbelongingtodierentmovingobjects. Motionsegmentationcouldbedirectlyappliedtomanyareas.Suchasvideocompression,videodatabasequerying,andsceneanalysis.MPEG-4standard[ 28 ]describesacontentbasedmanipulationofobjectsinimagesequences.Tocreateanobjectbasedscenerepresentation,itisnecessarytosegmentdierentobjectsintheframe.Videoquerying[ 29 ]isanotherneweldwhichaimstoautomaticallyclassifyvideosequencesbasedontheircontent.Acommonvideoquerytaskrequiresretrievingalltheimagesinadatabasethathaveasimilarcontenttothequeryexampleimage.Motionsegmentationenablesanindexingschemethatusestrajectories,shapesandowvectorsofindependentlymovingobjectstoqueryimagesequencesinadatabase.Therefore,thesystemwillgiveamoreaccurateresult.Thedevelopmentofunmannedaerialvehicle[ 30 ]makesreconnaissancemucheasierwhichrequiressceneanalysistechnologytoidentifysuspiciousmilitaryvehiclesinavideosequence.Objectshavingdierentmovingvelocitiesordirectionsneedtobeidentiedandsegmentedforfurtheranalysis.Arealtimevideomonitorsystemwouldhelpheadquartertolearntheenemymovementatthersttime. 32

PAGE 33

31 ].Theyusespatial-temporalderivativesoftheevolvingimagebrightnessfunctiontogiveasingleequationwhichpartiallydeterminestheopticalow.Anassumptionmadeisthatbrightnessofanypartoftheimagechangesveryslowly,sothatthetotalderivativeofbrightnessiszero.Thereasonisobvious.Becauseofapertureproblem,itwouldbehardtodeterminethemotionvectorofthecompleteeldwithoutthisassumption. Inordertodealwithnoisewhichexistsinmostrealworldimagedata,adenoisingprocessisneeded.However,simpledenoisingprocesswilldestroytheboundariesofobjectsofinterest.Therefore,inMumfordandShah'spaper[ 32 ],itissuggestedthattheproblemsofdenoisingandmotionestimationarecloselyinterlacedandshouldbesolvedsimultaneously.InCremers'paper[ 33 ],hepresentedavariationalapproachcalledmotioncompetitionwhichjointlysolvestheproblemsofmotionestimationandsegmentationfortwoconsecutiveframesfromasequenceinasimilarwayastheMumford-Shahapproach. Analyzingopticoweldisoneapproachtomotionsegmentation.WangandAdelson[ 34 ]describedasystemforrepresentingmovingimageswithsetsofoverlappinglayers.Theyusedk-meansclustermethodstodecomposeimagesequencesintolayersbasedonmotion.Thelayersareinorderofdierentdepths.Avelocitymapisusedtodenehowthelayerswarpedovertime.BorshukovandBozdagi[ 35 ]presentedamultistageanemotionsegmentationmethodthatfurthermodiedWangandAdelson'salgorithm.Theyreplacedtheadaptivek-meansclusteringstepbyamergingstepandintroduceamultiplestagespixellabelingmethod.Inthisway,thesegmentationperformanceisimprovedanddemonstratedonrealvideodata. 33

PAGE 34

36 ],theyproposedanapproachtosolvingtheperceptualgroupingproblembasedonextractionoftheglobalimpressionofanimage.Theresultsareencouragingbutcomputationalexpensive. Theobjectiveofatrackingalgorithmistolabelthetargetconsistentlyindierentframesofavideosequence.ObjecttrackingisanimportantprobleminComputerVision.Itisalsoverydicultbecauseofarbitraryobjectshapes,illuminationchanges,objectocclusion,complexobjectmotions,andcameramotion.Eachproblemneedstobesolvedinordertopreventfailureofthetrackingalgorithm.Inanobjecttrackingalgorithmtherearegenerallythreesteps:objectdetection,trackingobjectsfromframetoframe,andtrajectoryestimation.Theprimarydierenceofdierenttrackingalgorithmsisthewaytheyaddressthethreesteps.Moreover,dierentalgorithmsmayimposevariousassumptionsandconstraints,suchassmoothobjectmotion,rigidobject,andprioriknowledgeofobjectappearance.Inthissection,webrieyreviewthepreviousobjecttrackingalgorithmsinthreecategories. 34

PAGE 35

SethiandJain[ 37 ]proposedagreedyapproachbasedonproximityandrigidityconstraints.Theiralgorithmiterativelyminimizesthecostfunctionofcorrespondenceintwoconsecutiveframes.However,thismethodisnotabletohandleocclusions.ThetrackingalgorithmproposedbyRangarajanandShah[ 38 ]takesagreedyapproachwithproximalanduniformityconstrains.Thisalgorithmisabletoobtaininitialfeaturecorrespondencebycomputingopticalowofthersttwoframes,whichmakesitcapabletohandleocclusionproblems.Veenmanetal.[ 39 ]extendedtheworkofSethiandJain,andRangarajanandShahbyintroducingacommonmotionconstraintforpointcorrespondence.TheconstraintofVeenman'salgorithmistheassumptionthatallpointsonthesameobjecthavesimilarmotiondirections,whichisnotsuitablefortrackingisolatedobjects.Duetothefactthatvideosensorsintroducenoises,statisticalcorrespondencemethodsareproposedtosolvetheobjecttrackingprobleminnoisyimagesbytakingthemeasurementofuncertaintiesintoaccountwhenestimatingtheobjectstate.Kalmanlterhasbeenextensivelyusedinobjecttracking[ 40 41 ].Itassumesthatstatevariablesarenormallydistributed.Particleltersarealsousedtoaddressotherdistributions[ 42 ].SincebothKalmanltersandparticleltersarenotsuitableformultipleobjecttracking,JointProbabilisticDataAssociationFilter(JPDAF)[ 43 44 ]andMultipleHypothesisTracker(MHT)[ 45 46 ]areproposedandwidelyusedformultipledataassociation. 35

PAGE 36

47 ]proposedanecientalgorithmfortemplatematching.Theynotonlyusetemplatematching,butalsousecolorhistogramsandmixturemodelstomodeltheobjectsimilarities.AnotheralgorithmthatuseskerneltrackingisproposedbyComaniciuandMeer[ 48 ]whichusesamean-shifttrackertotrackobjectsbyusingaweightedhistogramcomputedfromacircularregiontorepresenttheobject.ComaniciuandMeerproposedanotherapproachtotrackregionsofinterestbyusingprimitiveinformationtocomputetranslation.In1994,ShiandTomasi[ 49 ]proposedthefamousKLTtrackerwhichiterativelycomputesthetranslationofanimagepatchcenteredatapointofinterest.KLTtrackerissimpleandecient.However,featureswillbeeliminatedifthesumofsquareddierences(SSD)issubstantial,becauseSSDindicatesthesimilaritybetweentheselectedobjects.Therefore,KLTisnotsuitableforobjecttrackinginalongimagesequencebecauseitwillincreasethepossibilityoferror.Thegreatestadvantageofkerneltrackingisreal-timeapplicability,andthegoalofsuchtrackersistoestimatethemotionofobject,whichisusuallyinformsoftranslation,aneorprojective.Thelimitationofkernaltrackingisthatprimitivegeometricshapesforobjectrepresentationmaycontainpartsofthebackground.Insuchcases,themotioncomputedbymaximizingmodelsimilaritywillnotbeaccurateduetotheeectofpartialbackgroundinthemodel. 36

PAGE 37

50 ].Thealgorithmmodelsobjectswithedgeinformationfrominnerregionsofanobject'ssilhouette.In2004,Kangetal.[ 51 ]proposedanalgorithmthatusescolorhistogramandedgestomodelobjects.Besides,SatoandAggarwal[ 52 ]proposedanobjecttrackingalgorithmbasedonsilhouettematchingwhichusesHoughtransformtocomputeobjecttrajectory.Recently,contourevolutionisalsousedtotrackobjectsinconsecutiveframes.Bertalmoetal.proposedanalgorithm[ 53 ]thatcomputesmotionvectorsontheedgeofsilhouettesiterativelyforeachcontourpositionusinglevelsetrepresentation.Similarly,Mansouri[ 54 ]proposedacontourtrackingalgorithmbasedonopticalowconstraintwhichcomputesmotionvectorsforallpointsinsidethesilhouette.Theadvantageofsilhouettetrackingistheabilitytotrackobjectsofvariousshapes.Howevercomputationalcomplexityofthesealgorithmsishigh. 55 ],includingmanyclassicmethodswhicharestillinuseuptonow.Duetotherapiddevelopmentofimageacquisitiondevices,moreimageregistrationtechniquesemergesafterwardsandarecoveredinanothersurvey[ 56 ]publishedin2003.Dierentapplicationsduetodistinctimageacquisitionrequiredierentimageregistrationtechniques.Ingeneral,mannersoftheimageacquisitioncanbedividedintothreemaincategories: 37

PAGE 38

Duetothediversityofimagestoberegisteredandvarioustypesofdegradations,itisimpossibletodesignauniversalmethodapplicabletoallregistrationtasks.Everymethodshouldtakeintoaccountnotonlytheassumedtypeofgeometricdeformationbetweenimagesbutalsoradiometricdeformationsandnoisecorruption,requiredregistrationaccuracyandapplication-dependentdatacharacteristics.Nevertheless,mostoftheregistrationmethodsconsistthefollowingfoursteps:featuredetection,featurematching,transformmodelestimation,imageresamplingandtransformation. Awidelyusedfeaturedetectionmethodiscornerdetection.KitchenandRosenfeld[ 57 ]proposedtoexploitthesecond-orderpartialderivativesoftheimagefunctionforcornerdetection.DreschlerandNagel[ 58 ]searchedforthelocalextremeoftheGaussiancurvature.However,cornerdetectorsbasedonthesecond-orderderivativesoftheimagefunctionaresensitivetonoise.ThusForstner[ 59 ]developedamorerobust,althoughtimeconsuming,cornerdetector,whichisbasedontherst-orderderivativesonly.ThereputableHarrisdetector[ 60 ]isinfactitsinverse. Featurematchingincludesarea-basedmatchingandfeature-basedmatching.Classicalarea-basedmethodiscross-correlation(CC)exploitformatchingdirectlyimageintensities.Forfeature-basedmatching,Goshtasbyproposedaregistrationmethodbasedonthegraphmatchingalgorithm[ 61 ].Clusteringtechnique,presentedbyStockmanetal.[ 62 ],triestomatchpointsconnectedbyabstractedgesorlinesegments. Afterfeaturecorrespondenceestablished,themappingfunctionisconstructed.Itwilltransformthesensedimageandoverlayitoverthereferenceimage.Theprevailingimageregistrationmethods,suchasDavisandKeck'registrationalgorithm[ 63 ],assumeallfeaturepointsarecoplanarandtheybuildahomographytransformmatrixfor 38

PAGE 39

Finallyinterpolationmethodssuchasnearestneighborfunction,bilinearandbicubicfunctionsareappliedtooutputtheregisteredimages. 39

PAGE 40

64 ].Inrecentyears,anewtheoryCompressiveSensing(CS)alsoreferredasCompressedSensingorCompressiveSampling,hasbeenproposedasamoreecientsamplingschemeforasparsesignal. ThetheoreticalframeworkofCSisdevelopedbyCandesetal.[ 2 ]andDonoho[ 4 ].TheCSprincipleclaimsthatasparsesignalcanberecoveredfromasmallnumberofrandomlinearmeasurements.Itmeansthatitispossibletoreconstructthesignalx,whichissparseindomain,byasmallnumberofmeasurements,y=x,wherethemeasurementensembleobeystherestrictedisometryhypothesis[ 65 ].Therecoveryprocedureistominimizethel1normofthesignalxindomain,whichisshowntobealinearprogrammingproblemandcouldalsobecastasaconvexoptimizationproblem. ComparedwiththetraditionalNyquist-Shannonsamplingtheory,theCStheoryprovidesagreatreductioninsamplingrate,powerconsumptionandcomputationalcomplexitytoacquireandrepresentasparsesignal.R.Baraniuketal.[ 66 ]haveproposedhardwaretosupportthenewtheoryofCompressiveImaging(CI).ItshowsthatCIisabletoobtainanimagewithasingledetectionelementwhilemeasuringtheimage/videofewertimesthanthenumberofpixels,whichcansignicantlyreducethecomputationrequiredforvideoacquisition/encoding. CShasbeenconnectedwithmanyothereldssuchasinformationtheory[ 6 67 68 ],highdimensiongeometry[ 69 { 72 ],statisticalsignalprocessing[ 73 74 ],anddatastreamingalgorithms[ 75 76 ].Besidestheconnectionstotheexistingtheories,CShasalsobeenused 40

PAGE 41

22 66 77 ],medicalimaging[ 78 ],distributedcompressedsensing[ 79 80 ]andanalogtoinformationconversion[ 81 { 84 ]. MostoftherecentpapersstudytwoproblemsofCS.OneistondtheoptimalsamplingensemblesandstudythemethodsforfastimplementationoftheCSensembles[ 1 20 85 86 ].TheotheroneistodevelopfastandpracticalreconstructionalgorithmstorecoverthesignalandsuppressthenoiseintroducedbyCS[ 21 65 87 88 ]. Donohoetal.[ 20 ]reportseveralfamiliesofrandommeasurementensembleswhichbehaveequivalently,includingrandomspherical,randomsigns,partialFourierandpartialHadamardintheirpaper.Thefollowingworksonmeasurementensemblesstudiestheoptimalmeasurementensemble,whichenablesrecoveringmoreentriesofthesignalwithasfewmeasurementsaspossible.InBaraniuketal.'spaper[ 85 ],theyprovedtheexistenceofoptimalCSmeasurementensemblesandtheyhavecertainuniversalitywithrespecttothesparsityinducingbasis.Elad[ 86 ]furtherproposedanaveragemeasurementofthemutual-coherenceoftheeectivedictionaryanddemonstratedthatitleadstobetterCSreconstructionperformance. Amongthereconstructionalgorithms,BasisPursuit(BP)[ 2 4 ]istherstonetosolvethisproblem.OrthogonalMatchingPursuit(OMP)[ 8 ]isproposedforfastreconstruction.Donohoetal.showthattheHomotopymethodrunsmuchmorerapidlythangeneralpurposedlinearprogrammingsolverswhensucientsparsityispresented[ 10 ].Inordertosuppressthenoiseandincreasethecomputationspeed,Figueiredoetal.[ 89 ]proposedagradientprojectionalgorithmforthebound-constrainedquadraticprogrammingformulationofCSproblem. Inthischapter,weuseCStorepresentvisualdataandproposeanewimagerepresentationschemeforvisualsensornetworks.ComparedtoJPEG2000,CSismoresuitableforapplicationsinsensornetworksbecausethesensorsareresourceconstrained.Dierentfromthepreviousworkoncompressiveimaging,whichtreatstheinputimageasawhole,wedecomposethevisualdataintotwocomponents:adensecomponentand 41

PAGE 42

Thischapterisorganizedasfollows.Section 4.2 givesabriefoverviewofCS.Section 4.3 discusseshowtoapplyCStopracticalsignals.Section 4.4 proposesaschemeforimagerepresentationusingCS.Theproposedreconstructionalgorithmisdiscussedindetails.TheexperimentalresultsarepresentedinSection 4.5 andSection 4.6 concludesthischapter. CSgivesananswertotheabovequestionthatitispossibletorecovertheK-sparsesignalxbytakingMrandommeasurementswhichismuchlessthanN.InordertotakeCSmeasurements,werstdenoteasanMbyNmatrixwithM<
PAGE 43

Dierentfromthetraditionalsampling,CSmeasurementmeasurestheinformationofthewholesignalatonetime.Therefore,eachmeasurementcontainsalittleinformationfromallelementsoftheoriginalsignal.Inthisway,CSisabletoreducethesamplingrate,powerconsumption,andcomputationalcomplexityofthevisualsensors.TheCStheorystatesthatthesignalcouldbeexactlyrecoveredifthenumberofmeasurementsMsatisestheconditionMConstKlogN[ 4 ],whereConstisanover-measuringfactorthatismorethan1. Sinceyisalowerdimensionvectorcomparedtox,itisimpossibletorecoverxdirectlybyapplyingtheinversetransformoftoy.Thesignalisreconstructedbysolvingthefollowingoptimizationproblem. Thereconstructedsignal^xisthesignalamongallsignalsgeneratingthesamemeasureddata,thathastransformcoecientswiththeminimall1norm.Thereconstructioncanbecastasalinearprogrammingproblem. Figure 4-1 givesanexampletoexplainthesignalrecoverybyCSreconstruction.Figure 4-1 (a)showstheoriginalsignalthathas250sampleswithonly25nonzeros,whichisverysparse.Figure 4-1 (b)istheCSmeasurementsbyaGaussianensemble,wherethereareonly90measurements.Figure 4-1 (c)istherecoveryresultfromtheCSmeasurementsbyaPOCSbasedalgorithm.Itisclearthatthesignaliswellrecovered. 43

PAGE 44

Inthissection,wewilldiscusstherstproblemfromthreeaspects:CSreconstructionerrorboundinnoiseenvironment,imagedecompositionandthecorrelationbetweenthedenseandsparsecomponents.Thesecondproblemwillbediscussedinthenextsection. Letusstillusetheexampleofsignalx,wexanorthonormalbasisandthesignalcouldberepresentedbyf(x)=x.Asiswellknown,thecompressibilityofthesignalxrelatestothedecayrateofthecoecientsoff.Ifthecoecientsoffobeyapowerlaw,wehave1nN,jfn(x)jRn1=p,whereRandpareconstantanddependonthesignal,fn(x)isthen-thlargestcoecientsinf(x).ThedierencebetweenxandapproximatesignalxKisobtainedbykeepingthelargestKcoecientsanditobeysthefollowingequation, whereC1isaconstant. InTsaigandDonoho'spaper[ 20 ],theygivetheerrorboundoftheCSapproximationxCSwhichisreconstructedfromthenon-adaptivemeasurements.GivenMmeasurements 44

PAGE 45

whereC2isconstant. Ifthesignalisexactlysparse,thetwoerrorboundsshouldbeatthesamelevel;otherwisetherewillbeabiggapbetweenthem.WecomparetheerrorboundofCSforsignalswithdierentdecayratesanddrawtheerrorbounds.C1,C2,R,MandNareconstantintheexperimentandonlypisavariable,whichcontrolsthespeedofdecay:thesmallerpis,thefasteritdecays.TheexperimentalresultisdepictedinFigure 4-2 ,whereKisthenumberofnon-zerocoecientsforreconstruction.Foreachgroup,thedashlinerepresentstheerrorboundofthetraditionalbestKmethodandthesolidlinerepresentstheresultofCS.Onecanobservethat,whenp=7=16,theerrorboundofCSisveryclosetothatofthetraditionalbestKmethod;however,whenp=9=16,thereisabiggapbetweenthesetwoerrorbounds.Obviously,theperformanceofCSreconstructionreliesonthedecayrateofthesignal.Thefasteritdecays,thebetteritrecovers. 90 ].However,itdoesnotsolveourproblem.Letusrsttakeonedimensionalsignalasanexample.SupposewehaveavectortwithlengthNandapre-chosenbasis,wecanrepresenttas 45

PAGE 46

LettD=PTi=11i1iandtS=PNTi=12i2idenotedenseandsparsecomponentsrespectively.However,fornaturalimages,itishardtondsuchasparsecomponent.Inthispaper,weusethepowerlawmodeltodenethesparsityofthesignal.TaketDforexample,wereorderitscoecients1iandcomputetheparameterpinthemodel, whereRandpareconstantandonlydependontD.Withthismodel,weuseparameterptorepresentthesparsityofthesignal.Thesmallerpis,thefasterthecoecientsdecayandthesparserthesignalis. Theresearchtondafunctionwhichleadstobetterdecompositionofimageisahotareainrecentyears[ 91 ].Previousworkhasprovedthatwavelettransformiswelldesignedforsparserepresentationofnaturalimages.ExpandtheimageIinthewaveletbasis whereWj0,kandWj,karewaveletsatdierentscales.Forsimplicity,inthispaper,wetakethelowestbandofwavelettransformID=Pk1j0,kWj0,kasdensecomponentandtheotherbandsIS=Pj2j=j1Pk2j,kWj,kassparsecomponent,wherej0isthecoarsestscale,j1isthenextscaleandj2isthenestscale. Applyingthepowerlawmodelto1and2respectively,wecanndthatthereisabigdierencebetweenp1andp2,whichmeansthesparsityofthetwocomponentsdierstremendously.Figure 4-3 showstheresultofwavelettransformofLenabyathree-leveldecomposition.Figure 4-4 depictsthedecaycurvesofdenseandsparsecomponents,respectively.Theleftcurverepresentscoecients1ofsignalIDandshowsthat1decaysslowly.Onthecontrary,thecurveontherightsiderepresenting2indicates 46

PAGE 47

where(i,j)isthepixeltobeinterpolated,Bi,jisthewindowcenteredatpixel(i,j)andi,jisarandomperturbationindependentofpixel(i,j)andtheimagesignal. InZhangetal.'spaper[ 92 ],theyformulatetheinterpolationproblemasanoptimizationproblem: whereIuistheimagetobeinterpolatedandtheIvistheoriginalimage.ui2Iuandvi2IvarethepixelsoftheimageIuandIvrespectively.Bisthewindowsize.Thesuperscripts(4)and(8)indicate4-connectneighboringand8-connectneighboringrespectively.Figure 4-5 depictsthesamplerelationshipsinequation( 4{10 ). Zhangetal.[ 92 ]alsogivealinearleast-squaresolutiontothisproblem,whichestimateneighboringpixelssimultaneouslyinwindowB. where^aand^bareestimatedfromIu. 47

PAGE 48

4.3.2 ,wehavediscussedthedecompositionofanaturalimagesignalIintoadensecomponentIDandasparsecomponentIS.Withtheaboveadaptiveinterpolationalgorithm,wecantakeIDasasub-sampleoftheoriginalimageIandinterpolateit.Fromtheabovediscussion,wecangetapredictionofIbysolvingequation( 4{10 ).Thisprediction^Irecoversmostofthehighfrequencyinformationandcouldagainbeexpandedbywaveletbasisasinequation( 4{8 ). Thenewsparsecomponent^IS=Pj2j=j1Pk^2j,kWj,kcanbeusedasagoodpredictionforIS=Pj2j=j1Pk2j,kWj,k. 4.4.1ImageRepresentationScheme 21 22 ],whichtaketheinputimageasanon-separatesignal,inourscheme,theinputimageisdecomposedintotwocomponents,denseandsparsecomponents.Thenthetwocomponentsaresampledusingdierentmethodsrespectively.Forthedensecomponent,weusethetraditionalapproach.Inotherwords,wesamplethedensecomponentpixelbypixel.Whileforthesparsecomponent,weapplyCSbyrandomsampling.TheproposedschemeisdepictedinFigure 4-6 TheinputimageIisrstdecomposedintoadensecomponentIDandasparsecomponentISthroughatransformT,whereTcouldbewavelet,curvelet,oranyothertransforms.Inourscheme,weusediscretewavelettransformWtodecomposetheimage.WeexpandtheimageIasinequation( 4{8 ). 48

PAGE 49

wherej0isthepresetcoarselevelofthewavelettransform.Normally,j0issettobe1. InordertotakemeasurementsofthesparsecomponentsIS,weuseaGaussianrandomensemble.Asweknow,thedimensionoftheinputimageisveryhigh,sodirectlyapplyingthe2DGaussianrandomensembletothesignalISisnotpractical.Inordertoapplytherandomensemblemoreeciently,weneedtoregroupthesignalandtakeablockbasedsamplingstrategy.WerstdivideISintoseveralgroupsbyscalesandthenreorderitintoanumberofvectorsofthesamedimension.Inthisway,wecantakerandommeasurementstothevectorwithamoderatesizeinsteadofthetremendoussize. wherenisthenumberofgroupsandxiisthei-thgroupofIS. Inordertorecovertheoriginalsignal,wehavetoprocessthesignalseparately.Sincethedensecomponentismeasuredpixelbypixel,~IDisexactlythewaveletcoecientsofID.Therefore,wecoulddirectlyapplyinversetransformW1to~IDgogetID.InordertorecoverIS,weneedtosolvetheoptimizationprobleminequation( 4{2 ).InCandesandRomberg'spaper[ 21 ],theyproposeaprojectionontoconvexsets(POCS)algorithmtoreconstructtheoriginalsignalfromtherandommeasurements.WefollowtheirapproachandimprovethealgorithmbyusingpredictionofISasthestartingpointoftheiterations.Theprediction^Ioftheinputimagecouldbeobtainedbyadaptiveinterpolationofthedensecomponentusingequation( 4{10 ).Thenwecouldapplywavelettransformto^Iandget^ISasthepredictionofIS.WewilldiscussthedetailsoftheCSrecoveryalgorithminthenextsub-section. 49

PAGE 50

21 ],theyproposeadierentrecoveryprocedure,whichrequiresasmallamountofprioriinformationofthesignaltoberecoveredbutcostslesscomputationineachiteration.Intheiralgorithm,theyassumethel1normoftherecoveredsignalisknown.WefurtherdevelopthisalgorithmandproposeanewreconstructionmethodbasedonPOCSandpredictionfromadaptiveinterpolation. Sincexinequation( 4{2 )istheuniquesolutionbytheCompressedSensingprinciple,weareabletoclaimthatthel1-ballB=~x:k~xkl1kxkl1andthehyperplaneH=~x:~x=ymeetatexactlyonepoint:BTH=x.BecausebothBandHareconvex,xcanberecoveredbyanalternateprojectionsontoconvexsets(PoCS)algorithm[ 93 ]. Aswehavediscussedinlastsection,wedecomposetheinputimageintodenseandsparsecomponentsandweutilizethedensecomponent~IDtopredictthesparsecomponentISbyadaptiveinterpolation.Thepredictionhelpsintwoaspects:rst,itcouldbeusedastheinitializationoftheiteration.Asknowntoall,theinitializationisveryimportanttoaniterativealgorithmandtheinitialvalueneedtobeinacertainspacefornalconvergenceatlocaloptimal.Secondly,thepredictioncanbeusedasareferencewhichenablesthealgorithmconvergingfasterandmoreaccurately. Fromthestartingpointof~xi,thealgorithmiteratebyalternatingprojectionsontoH,thenontoB.ItisguaranteedtoconvergetoapointinBTH[ 93 ]. Tondtheclosestvector~xHiinHforanarbitrary~xi,weapplytheequation Thesteplengthfor~xHicombinestwoparts,oneiscomputedfromdirectprojectionandtheotherpartisfromthedierencebetween~xiandthepredictionsignalxpi.isauser 50

PAGE 51

Inordertoprojectthevector~xHiontothel1-ballB,weapplyasoftthresholdingoperation. Inordertodeterminethethresholdsuchthatk~xBikl1kxikl1,wesortthecoecientsbyabsolutevalueandperformalinearsearch. 21 22 ].Inthethirdpart,wefurthercomparetheperformanceofouralgorithmtoPOCSbyerrorreductionwiththesameiterations.Atlast,weshowsomerecoveredimages. Figure 4-7 depictsthecomparisonbetweenthepredictionofthesparsecomponentandthecorrespondingcomponent.Inordertoverifythesimilaritybetweentheoriginalsparsecomponentandthepredictionfromthedensecomponent,werstdecomposetheinputsignal,wheretheinputdataisanimagepatchfrom`Lena',intotwocomponents.Theleftpictureshowsthesparsecomponentinwavelettransformdomain.Thenweinterpolatethedensecomponenttogetapredictionoftheoriginalimagepatch.Thepredictionisfurtherdecomposedintosparseanddensecomponents.Thepredictedsparsecomponentisshownintherightpicture. Inordertocomparethemmoreclearly,wescantheimagebyrowsfromlowfrequencytohighfrequency.Thentheimageisscannedintoaone-dimensionalvectoranddepictedinFigure 4-8 .Theuppersubplotshowstheoriginalsignalandthelowersubplotshows 51

PAGE 52

4-8 ,itisclearthatthepredictionbyinterpolatingthedensecomponentisveryclosetotheoriginalsignal.ItmeansthatwecanuselessiterationforconvergenceinCSreconstructionanditisalsopossibletousethepredictionasaweightedconstraintinthereconstructionalgorithm.Theimagesweuseinourexperimentare`Lena',`Boat',`Cameraman'and`Peppers'asusedin[ 21 ]and[ 22 ].AswedescribedinSection 4.4 ,foreachimageI,wecomputetherecoveryimage~Ibasedonequation( 4{15 )andequation( 4{16 ).Theexperimenttestsfordierentsparsityofimagesanddierentsizesofmeasurementensemble.TherecoveryerrorismeasuredbyPSNRindbandtabulatedinTable 4-1 Whynotusebit-ratebutrathernumberofmeasurements?ThereasonisthatthetraditionalquantizationschemeisnotsuitableforCSmeasurements.HowtoquantizetheCSmeasurementsitselfisacrucialproblemtobesolved[ 20 ].Soitisoutofthescopeofthiswork.Therefore,insteadofusingbit-rate,weusenumberofmeasurementsfollowingCandes'swork[ 21 ]. Theresultsarecomparedtothepreviousworks[ 21 ]and[ 22 ].Itisclearthatfor`Lena',`Boats'and`Peppers',ouralgorithmoutperformsthereconstructionalgorithmsin[ 21 ]and[ 22 ].ItmeansweneedfewermeasurementstoachievethesamePSNRorachievebetterPSNRwiththesamenumberofmeasurements.OurnumberofmeasurementsinTable 4-1 includesthemeasurementsofbothIDandIS.Therearetworeasonsfortheimprovement:rst,weremovethedensecomponentfromtheinputdatabydecompositionanditincreasesthesamplingeciency;secondly,byintroducingthepredictionoftherecovereddata,weareabletoconstraintheunknownsignalinasmallspaceneartheoriginalsignalandrequirefewermeasurementsforconvergence.Fortheimage`Cameraman',ouralgorithmlosesalittlebitto[ 21 ]inhighrateend.However,comparingto[ 22 ],ourrecoveryresultismoreaccurate. Inthischapter,wewillnotcompareourresultstoJPEG2000.Thereasonisthatthepurposeofthisworkistoexploreanimagerepresentationschemewhich 52

PAGE 53

94 ]. Inthethirdexperiment,weuse`Peppers'asanexampletotesttheconvergencetime.WecompareerrorreductionbythetotalerrorandPSNRseparatelyinFigure 4-9 .Thesolidlinerepresentsouralgorithmandthedashlineplottheresultsfrom[ 21 ].InFigure 4-9 ,itshowsthatweneed28iterationscomparingto43withthealgorithmin[ 21 ]toreducetheerrorto5106and46iterationscomparingto78toreducetheerrorto4.5106.ThepredictiongreatlyreducesthenumberofiterationsforthesamePSNR,whichmeansweuselesstimeforsignalreconstructionandthereforeagainsavesenergyneeded. Atlast,wegivesomeexamplesofrecoveredimagesinFigure 4-10 andFigure 4-11 .Therecovered`Lena'and`Boats'arebothobtainedfrom20000measurementsintotal. 53

PAGE 54

ExperimentalresultofrecoveringaonedimensionalsparsesignalbyCompressedSensing.Figure 4-1 (a)showstheoriginalsignal.Figure 4-1 (b)istheCSmeasurements.Figure 4-1 (c)istherecoveryresult. Table4-1. Comparisonofreconstructionresultswiththesamenumberofmeasurements. Measurements10000150002000025000 `Lena'[ 21 ]26.528.730.432.1[ 22 ]26.528.630.632.2OurAlgorithm30.031.833.034.2 `Boat'[ 21 ]26.729.831.833.7[ 22 ]27.029.932.534.8OurAlgorithm29.131.033.034.4 `Cameraman'[ 21 ]26.228.730.933.0[ 22 ]24.026.127.929.4OurAlgorithm26.328.529.730.7 `Peppers'[ 21 ]21.625.327.529.4[ 22 ]27.230.332.734.7OurAlgorithm27.430.732.734.6 54

PAGE 55

ComparisonoferrorboundsbetweentraditionalmethodandCompressedSensing.K=100:200,M=4K,N=1024. Wavelettransformofimage`Lena'.Theleftoneistheoriginalimageandtherightisthetransformedimageinwaveletdomain. 55

PAGE 56

Powerlawmodelsofcoecients1ofsignalIDatcoarsescale(left)and2ofsignalISatnescale(right). Figure4-5. 56

PAGE 57

BlockdiagramofourproposedimagerepresentationschemebasedonCompressedSensing. Figure4-7. Predictionofthesparsecomponentin`Lena'.Theleftimageisthesparsecomponentoftheoriginalimageandtherightimageispredictedbyinterpolationofthedensecomponent. 57

PAGE 58

Comparisonofpredictionfromdensecomponentandoriginalsparsecomponent.Botharescannedintoaone-dimensionalvector. Figure4-9. Comparisonoferrorreductionwithsamenumberofiterations.TheleftgureshowsthetotalerrorreductionandtherightgureismeasuredbyPSNRindb.Theresultistestedonimage`Peppers'. 58

PAGE 59

Recoveredimage`Lena'from20000measurements.Theleftimageistheoriginalimageandtherightimageistherecovered. Figure4-11. Recoveredimage`Boats'from20000measurements.Theleftimageistheoriginalimageandtherightimageistherecovered. 59

PAGE 60

2 ]andDonoho[ 4 ],statesthattheinformationthatcontainedinthefewsignicantcoecientsofasparsesignalcanbecapturedbyasmallnumberofrandomlinearprojections.Theoriginalsignalthencanberecoveredbysolvingaoptimizationproblem.Duetothegreatreductionofsamplingrate,powerconsumption,andcomputationalcomplexitytoacquiringsparsesignals,CShasbeenintroducedtomanyareas,suchasinformationtheory[ 6 ],medicalimaging[ 95 ],andimage/videocompression[ 21 { 23 ]. Sincemostnaturalimagesandvideosarehighlycompressibleinthesensethatonlyafewcoecientsarelargewhenexpressedinproperbasis,suchasDCTorWavelet,itispossibletouseCStoreduceencodingcomplexityandimprovecodingperformance.InCandesandRomberg'spaper[ 21 ],theyrstproposeapracticalrecoveryalgorithmandshowthatitispossibletorecoveranimagefromabout3M-5MprojectionsontogenericallychosenvectorswiththesameaccuracyastheidealM-termwaveletapproximation.InGan'spaper[ 22 ],heproposesablock-basedsamplingmethodforfastCSofnaturalimages.TheexperimentalresultsshowthatblockCSsystemoerscomparableperformancestoexistingCSschemeswithmuchlowerimplementationcostfornaturalimages.InZhangetal.'spaper[ 23 ],theyuseCSreconstructioninJPEGcodingframeworkasanewcodingmodeandclaimthatthereisaveragerate-distortion(RD)gainbyapplyingthenewCScodingmode. Inthischapter,weproposeanewimage/videocompressionsystem,whichcombinesCSintotraditionalblockbasedimage/videocompressionschemes,suchasJPEGand 60

PAGE 61

Thischapterisorganizedasfollows.Section 5.2 givesabriefoverviewofCS.Section 5.3 presentstheframeworkofourproposedimage/videocompressionmethod.Section 5.4 discusseshowtoapplyCSrecoveryalgorithmtomitigatequantizationerror.TheexperimentalresultsaregiveninSection 5.5 andSection 5.6 concludesthispaper. 21 ].Supposewehaveanite-lengthsignalx2RN.AtypicalscenarioofCSistotakeMrandommeasurements,whereM<
PAGE 62

CStheorystatesthatxcouldberecoveredexactlybysolvingthefollowingoptimizationproblem. Thereconstructedsignalxisthesignalamongallsignalsgeneratingthesamemeasureddata,whichhastransformcoecientsinwiththeminimall0norm.However,solving( 5{2 )isNP-hardandnumericallyunstable.Therefore,thel1normminimizationisproposedtoapproximatethesignalx, Therecoveryproblem( 5{3 )isaconvexproblemandwhenxisreal,itcanberecastasalinearprogram.Whilelinearprogrammingtechniques,suchasbasispursuit[ 7 ],gureprominentlyinthedesignoftractableCSdecoders,highcomplexityO(N3)makesthemimpracticalformanyapplications.Weoftenencountersparsesignalswithlargedimensions;forexample,currentdigitalcamerasacquireimageswiththenumberofpixelsNoftheorderof106ormore.Forsuchapplications,theneedforfasterdecodingalgorithmsiscritical.Attheexpenseofslightlymoremeasurements,iterativegreedyalgorithmshavebeendevelopedtorecoverxfromy.Examplesincludethematchingpursuit,iterativeorthogonalmatchingpursuit(OMP)[ 8 { 10 ].Candesetal.[ 21 ]proposeapracticalrecoveryalgorithmwhichrequiresaprioriinformationaboutx,butreducesthecostofeachiteration. Ingeneral,naturalimagesandvideosarehighlycompressibleanditispossiblethatwecanuseCStohelpreducingsamplingrateandpowerconsumption. 62

PAGE 63

21 { 23 ].Inthissection,weproposeanewimage/videocompressionsystem,whichtakesadvantageofCSrecoveryintraditionalcodingschemes. WetakeJPEGasanexample.TheJPEGencodingschemeofanimageblockincludesthreesteps:DCTtransform,scalarquantizationandentropycoding.Thecorrespondingdecodingschemeincludesentropydecoding,de-quantizationandinverseDCTtransform.Mostofthecodingerroriscausedbyscalarquantization.StandardJPEGdecoderreconstructsquantizedDCTcoecientstothecenterofthequantizationbin.Thisfailstoexploitthenon-uniformdistributionofACcoecients.Previousde-quantizationmethodsusingLaplaciandistributioninsteadofuniformdistributioncanachieve0.25dBoverstandardJPEGdecoder[ 96 ].CSrecoverybyminimizingtotalvariation(TV)canhelptondthedistributionofACcoecientsratherthanuniformorLaplaciandistribution. Inourframework,weuseCSrecoveryalgorithmasanewcodingmodeinJPEGcodingframework.Ratedistortionoptimization(RDO)isemployedforadaptivemodeselection(MS)betweenthenewmodeandtheconventionalcodingmode.ThesystemschemechartisshowninFig. 5-1 Intheencoderside,weusetruncationinsteadofrandomprojectioninCSsamplingwhichissimilartothemethodusedinZhangetal.'spaper[ 23 ].ThetruncationmeansthatwewillonlykeeptherstKACcoecientsandsettheresttozero.Inthisway,weavoidthepossiblelossthatcausedbyrandommeasurements[ 97 ].Indecoderside,inordertomitigatethescalarquantizationerror,weproposetouseboundedresidueconstraintinsteadofquadraticconstraint[ 21 ]torecovertheimageblock. 63

PAGE 64

98 ].TherecoveryalgorithmwithTVminimizationinCShasbeendiscussedinCandesandRomberg'spaper[ 21 ]. Thenaturalimageitselfisnotsparse,butthegradientmapoftheimagecanbetakenasa2Dsparsesignal.LetIi,jdenotethepixelintheithrowandjthcolumnofannnimageI,anddenetheoperators ThetotalvariationofIisthendenedasthesumofthemagnitudesofthisdiscretegradientinequation( 5{6 )ateverypoint: InZhangetal'spaper[ 23 ],theyalsouseTVminimizationoptimizationtorecovertheimageblockinJPEG.However,intheirscheme,1DDCTisappliedinsteadof2DDCT.Inourexperiment,itshowsthatthereisnoreasonweshoulduse1DDCTintherecoveryprocess.2DDCTfurtherexploresparsityofimageblocks. 64

PAGE 65

wherexo2RNisthevectorbeforequantization,e2RNisthequantizationerror.A2RKNisthetruncationmatrixandy2RKisthemeasurements.InCandesandRomberg'spaper[ 21 ],aquadraticconstraintisusedtosolvethisproblem, Donoho[ 4 ]showsthatthesolutiontoequation( 5{9 )recoversanunknownsparseobjectwithanerroratmostproportionaltothenoiselevel. wherex]isthesolutiongivenbyequation( 5{9 ),CisaconstantanddependsonA.ItisshownthatthetypicalvalueofCisaround10. Inourwork,insteadofndingaglobaloptimalsolution,wearemoreinterestinginafeasiblepointxwhichislocallyoptimal.Tomakeitclear,whatwedoistochangetheinequalityconstraintintoaboundedregion.ThereasonwewanttouseaboundedresidueconstraintisthatweusescalarquantizationinthestandardJPEG.Belowisrewriteoftheproblemwiththeboundedresidualconstraint. whereqisthequantizationstepsize. Comparingtothequadraticconstraint,whichtriestoconstrainthel2normofthequantizationerrorwithapresetthreshold",theboundedresidualconstrainttriestorestrictthevalueofeachpixeloftheimageintoaboundedregion.Sinceweknowthe 65

PAGE 66

23 ]inmeansquareerror(MSE);secondly,wetestouralgorithmonaseriesoftestimagesandcompareourresultswithstandardJPEG. TherstpartofourexperimentistocomparetheperformanceofourmethodwithZhangetal'salgorithm.ThetwomethodsbothapplyRDOforMS,thekeydierenceisthatouralgorithmisbasedon2DDCTandweuseanewconstrainforCSrecovery.SincetheblocksthatchooseCSmodearegiveninZhangetal'spaper[ 23 ],wecansimplyusetheseblocksastestdatatodothecomparison. InFigure 5-4 ,wechoosevetypicalblocksfromtheblocksetgiveninFigure 5-2 asthetestblocksusedinourexperiment.Inourexperiment,wecomparethetotalrecoveryerrorinsteadofPSNRbyapplyingstandardDCT/IDCT,Zhangetal'salgorithm[ 23 ]andourapproach.WechoosethesameQPvaluesandtruncationrate[ 23 ]andbelowinTable1arethetestresults.InTable 5-1 ,theerrorisgivenindB.FromTable 5-1 ,itisclearthatunderthegiventruncationrateandquantizationstepsize,theresultsinpaper[ 23 ]canhardlygetgaininPSNRcomparingtotheconventionalcodingscheme,JPEG.Itispossiblethatmostofthegaingiveninthework[ 23 ]comesfromthesavingofmeasurements.FromTable 5-1 ,wecanndthatCSreconstructionbasedonourapproachcanreducethereconstructionerrorgreatly. Thesecondexperimentistoapplyourcompressionmethodtoseveraltestimages.TheexperimentalresultsaregiveninTable 5-2 InFigure 5-5 ,wegivethetestresultofimage`Boat'.Theexperimentalresultsshowthataveragegainisabout0.5dB.Actually,ifweusepartof`cameraman',wecangetapproximately1dBgain.TheresultisshownisFigure 5-6 .TheproblemisCSencoder/decoderdoesnotworkforallkindsofblocks.Itworksbetterwhenthereisa 66

PAGE 67

Experimentalresultsofthetestblocksfrom`Cameraman'. K(TruncationLength)20263240 Q(Quantizationstepsize)4525126 23 ]39.738439.08339.738442.828OurAlgorithm39.738439.08339.738442.9292 23 ]25.768429.251831.273442.828OurAlgorithm27.456539.08339.738437.9906 23 ]24.794825.6432.571637.471OurAlgorithm27.920230.352736.918939.2378 23 ]25.337628.47631.796137.5773OurAlgorithm27.428130.57834.097536.4984 23 ]27.558232.295631.559643.4742OurAlgorithm31.99334.052139.650844.4607 Table5-2. ExperimentalresultsofBCS. Bitratereduction(%)PSNRgain(dB) Cameraman-10.650.83Boats-2.330.20Lena-0.760.12Peppers-2.950.26 singleedgeintheimageblock.Therefore,thegainwillbeaveragedwhenthedimensionoftheimageislarge. 67

PAGE 68

Theowchartofthenewcompressionscheme. 68

PAGE 69

Theimage`Cameraman'usedinZhangetal'spaper[ 23 ] Figure5-3. TheimageblockswhichuseCSmode[ 23 ]. Figure5-4. Thetestblocksusedintheexperiment. 69

PAGE 70

Testresultof`Boats'. Figure5-6. Testresultof`Cameraman'. 70

PAGE 71

Givena3DpointdatasetX=fxig,xi2R3,i=1,2,...,n,thegeometricalttingproblemisusuallystatedastheoptimizationofacostthatmeasureshowthegeometricalsurfacefunctionS=fx:g(x)=0gtsthedatasetX.Themostcommonlyusedobjectivefunctionistheleastsquarescost, where Thettingfunctiongislearnedbyminimizingthedesigncost,D,measuredovertheinputdataset,X.Itiswell-knownthatformostchoicesofD,thecostmeasuredduringdesignmonotonicallydecreasesasthesizeofthelearnedttingfunctiongisincreased.Withalargesetoffunctions,itiseasytocreateasurfacewhichpassesthrougheachinputdatapointbutissuspiciouslycomplicated.TheprincipleofOccam'srazorstatesthatthesimplestmodelthataccuratelyrepresentsthedataismostdesirable.Soweprefertouseafewbasisfunctionswhichyieldasmoother,simplersurfacewhichcouldwellapproximatestheoriginaldata. Generally,therearetwoapproachestosolvetheoverttingproblem.Oneapproachistoaddpenaltytermstothedataset,likesmoothnessorregularizationconstraints.Anotherapproachistorstbuildalargemodelandthenremovesomeparametersbyretainingonlythevitalmodelstructure.Althoughbothapproachescangenerate 71

PAGE 72

Inthischapter,weproposeadierentapproachtosolvethegeometricalttingproblem.Insteadofestimateacomplicatedfunctiontotallthedatapoints,wepartitionthedatasetintoseveralsubsetsuchthatthedatapointsineachsubsetcouldbeapproximatedbyasimplermodel.Thespacepartitioninghelpstoreducethesizeofthesurfacemodelwhilekeepingthedesigncostsmallenough. OneofthemostpopularclusteringalgorithmisLloyd'salgorithm,whichstartsbypartitioningtheinputdataintokinitialsets.Itcalculatesthecentroidofeachsetviasomemetric.Usually,Lloyd'salgorithmisusedinaEuclideanspaceandcentroidiscalculatedbyaveragingdimensionsinEuclideanspace.Ititerativelyassociateseachpointwiththeclosestcentroidandrecalculatesthecentroidsofthenewclusters.Althoughwidelyusedinrealworldapplications,therearetwoseriouslimitationsofLloyd'salgorithm.Therstlimitationisthatthepartitioningresultdependsontheinitializationoftheclustercenters,whichmayleadtopoorlocalminima.ThesecondlimitationisthatLloyd'salgorithmcanonlypartitionlinearseparableclusters. Inordertoavoidinitializationdependence,asimplebutusefulsolutionistousemultiplerestartswithdierentinitializationstoachieveabetterlocalminima.Globalk-means[ 99 ]isproposedtobuildtheclustersdeterministically,whichusetheoriginalk-meansalgorithmasalocalsearchstep.Ateachstep,globalk-meansaddonemoreclusterbasedonpreviouspartitioningresult.Deterministicannealing[ 100 ]isanotheroptimizationtechniquetondaglobalminimumofacostfunction.Deterministicannealingexplorealargercostsurfacebyintroducingaconstraintofrandomness.Ateachiteration,therandomnessisconstrainedandalocaloptimizationisperformed.Finally, 72

PAGE 73

Kernelmethod[ 101 ]isusedtosolvethesecondproblembymappingthedatapointsfrominputspacetoahigherdimensionalfeaturespacethroughanon-lineartransformation.Thentheoptimizationisappliedinthefeaturespace.Thelinearseparationinthefeaturespaceturnsouttobeanon-linearseparationintheoriginalinputspace. Inthischapter,weproposeanon-lineardeterministicannealingapproachforspacepartitioningin3DEuclideanspace.Weusedeterministicannealingtodividetheinputspaceintoseveralregionswithdierentsizesandshapes.Withthepartition,wecaneasilyndalinearlocalsurfacetotthedatainsideeachregion.Deterministicannealingmethodoerstwogreatfeatures:1)theabilitytoavoidmanypoorlocaloptima;2)theabilitytominimizethecostfunctionevenitsgradientsvanishalmosteverywhere.Duetothefactthatthedataislocalizedtoafewrelativelydenseclusters,wedesignakernelfunctiontomapthedatapointfromthegeometricspacetosurfacefeaturespaceandapplydeterministicannealinginthefeaturespaceinsteadofthegeometricspace.Wecomparetheproposednon-lineardeterministicannealing(NDA)algorithmwiththewidelyusedLloyd'salgorithmonbotharticialdataandrealworlddata.TheexperimentalresultsshowthatNDAalgorithmoutperformsLloyd'salgorithminbothmeansquaredapproximationerroranderrorprobability. Inthefollowingsectionweformallydenethe3Dgeometricttingproblemandbrieydescribedeterministicannealingandkernelmethodforspacepartitioning.InSection 6.3 wepresenttheproposedkerneldeterministicannealingalgorithmalongwithananalysisofitscomputationalcomplexity.TheexperimentalresultisshowninSection 6.4 .FinallySection 6.5 concludesthischapter. 73

PAGE 74

GivenasetofdataXofscattered3Dpoints,wewouldliketondthegeometricsurfacethatbesttstothescattereddata.Thettingproblemisusuallystatedastheoptimizationofacostthatmeasureshowwellthettingfunctiong(xi)tsthedata.Themostcommonlyusedobjectivefunctionistheleastsquarescost.Findingagoodtisachallengingproblemandmaybemoreofanartthanascience.Ifweusealargesetoffunctionsasthebasis,wemaycreateasurfacewhichpassesthrougheachdatapointbutissuspiciouslycomplicated.Usingfewbasisfunctionsmayyieldasmoother,simplersurfacewhichonlyapproximatestheoriginaldata.Duetotheoverttingproblem,weproposeannewapproachtooptimizetheobjectivefunctionviaspacepartitioning.Werstpartitionthedatasetintoseveralsubsetssuchthatthedatapointsxineachsubsetcouldbeapproximatedbyalinearsurfacemodel.Inotherwords,wewouldliketouseasetofplainmodelstoapproximatethedateset.Theobjectiveofspacepartitioningistominimizethegeometricttingerror. where,xi=[xi,yi,zi]Tisthei-thpointdata,k=[ak,bk,ck]Tisthek-thlinearsurfacemodel,anddi,kisisthettingerrorbetweenxiandplanemodelgk=0whichisdenedas 74

PAGE 75

100 ]toclusteringhasdemonstratedsubstantialperformanceimprovementovertraditionalsupervisedandunsupervisedlearningalgorithms.DAmimicstheannealingprocessinstaticTheadvantageofdeterministicannealingisitsabilitytoavoidmanypoorlocaloptima.Thereasonisthatdeterministicannealingminimizesthedesignedcostfunctionsubjecttoaconstraintontherandomnessofthesolution.Theconstraint,Shannonentropy,isgraduallyloweredandeventuallydeterministicannealingoptimizeontheoriginalcostfunction.Deterministicannealingmimicsthesimulatedannealing[ 102 ]instatisticalphysicsbytheuseofexpectation.Deterministicannealingderivesaneectiveenergyfunctionthroughexpectationandisdeterministicallyoptimizedatsuccessivelyreducedtemperatures.Thedeterministicannealingapproachhasbeenadoptedinavarietyofresearchelds,suchasgraph-theoreticoptimizationandcomputervision.A.Raoetal.[ 103 ]extendedtheworkforpiecewiseregressionmodeling.Inthissubsection,wewillbrieyreviewtheirwork. Givenadataset(x,y),theregressionproblemistooptimizethecostthatmeasureshowwelltheregressionfunctionf(x)approximatestheoutputy,wherex2Rm,y2Rn,andg:Rm!Rn.Inthebasicspacepartitioningapproach,theinputspaceispartitionedintoKregionsandthecostfunctionbecomes whered(,)isthedistortionmeasurefunction.Insteadofseekingtheoptimalhardpartitiondirectly,randomnessisintroducedforrandomizedassignmentforinputsamples. InA.Raoetal.'swork,theyusethenearestprototype(NP)structureasconstraintandgiventhesetofprototypesfsj:j=1,2,3,...,Kgintheinputspace,aVoronoi 75

PAGE 76

Althoughtheultimategoalistondthehardpartition,some\randomness"isdesiredduringtheassignment.Shannonentropyisintroducedasaconstraintoftherandomness. Eventually,thisconstrainedoptimizationproblemcouldberewrittenastheminimizationofthecorrespondingLagrangian where,isanonnegativeLagrangemultiplierwhichcontrolstherandomnessofthespacepartition. Takethemostpopulark-meansalgorithm[ 104 ]asanexample,kernelk-meansmapsdatapointsfromtheinputspacetoahigherdimensionalfeaturespacethroughanonlineartransformationandthenapplystandardk-meansinthefeaturespace.Theclusteringresultinlinearseparatorsinfeaturespacecorrespondstononlinearseparatorsininputspace.Thuskernelk-meansavoidthelimitationofstandardk-meansthattheclustersmustbelinearlyseparable. 76

PAGE 77

Tosolvethespacepartitioningproblem,wedonotuseprototypetocalculatethedierence.Thereasonisthattheprototypeinspacepartitioningisgenerallynotsucienttorepresentaplanein3Dspace.Instead,weestimatethelinearplanemodelandcalculatethettingerrorastheEuclideandistancebetweenthedataandtheplane.Thetraditionallocaloptimizationalgorithmwilllikelystuckatalocaloptima.Inordertoavoidlocaloptima,weuselocalgeometricstructurefromneighboringdatapointsandembeddedthedatavectorstoahigherdimensionasfollows. Theinputdataisgivenasa3Dpoint,xi=[xi,yi,zi]T.Withtheassumptionthatnearestdatapointsareonthesameplane,wecouldestimatethelocalplanemodel,Li=[ai,bi,ci]TofdatapointxianditsKnearestneighborpoints. Thenwerevisethedistortionfunctionasfollows, 77

PAGE 78

whereD1=di,jcalculatethettingerrorbetweenthedatapointandtheestimatedplane,andD2calculatethedierencebetweenthelocalestimatedplanemodelandtheclusterscaleestimatedplanemodel.D2isdenedasfollows. Afterthemapping,weapplydeterministicannealingalgorithmtopartitionthedataintoseveralclustersasfollows. wheregj=[aj,bj,cj]isthegeometricalsurfacemodelparametertobeestimated,DisthesumofsquareofgeometricalttingerrorandHistheentropyconstraint.WedeneDandHasfollows: Toperformoptimizationweneedtofurtheranalyzeitsterms.Wecanrewriteequation( 6{18 )byapplyingthechainruleofentropyas 78

PAGE 79

TheminimizationofFwithrespecttoassociationprobabilitiesp(gjjxi)givesrisetotheGibbsdistribution wherethenormalizationis ThecorrespondingminimumofFisobtainedbypluggingequation( 6{21 )backintoequation( 6{16 ) TominimizetheLagrangianwithrespecttotheclustermodelgj,itsgradientsaresettozeroyieldingthecondition Non-lineardeterministicannealingmethod(NDA)introducestheentropyconstrainttoexplorealargeportionofthecostsurfaceusingrandomness,whilestillperformingoptimizationusinglocalinformation,whichissimilartofuzzyc-meansalgorithm.Eventually,theamountofimposedrandomnessisloweredsothatuponterminationNDAoptimizesovertheoriginalcostfunctionandyieldsasolutiontotheoriginalproblem. However,thereisnocloseformsolutionforNDA,thereforeweuseagradientdescentalgorithmtosolvethisproblem.Inthispaper,WecompareNDAbasedgeometricalsegmentationalgorithmtotheprojectionbasediterativealgorithm(PI)andadaptive 79

PAGE 80

1 .Forcomparisonpurpose,IalsogivePIalgorithminAlgorithm 2 6-1 .Krepresentsthenumberofplanesinatestdataset.Foreachplane,100randompointsaregenerated.Thedateset1contains300dataintotalfrom3nonparallelplanes.Thedataset2contains400datafrom4planes.Thedataset3contains500datafrom5planesandthedataset4contains600datafrom6planes.TheaveragesquaredapproximationerrorofNDAisignorablecomparingtotheerrorsofPIandNPI.Fromtheexperimentalresult,wecansaythatNDAalgorithmoutperformsbothPIandAPIalgorithmsintheaveragesquaredapproximationerror.ThereasonNDAalgorithmoutperformsPIandAPIalgorithmsisthatNDAisabletoseparatethespacenon-linearlyandavoidmanypoorlocaloptima. Wealsomeasuretheperformanceofthesegmentationalgorithmsinpercentageofcorrectidenticationofplanes.Wetestthesamedatasetasusedinthepreviousexperimentandcomputethecorrectidenticationpercentageaveragingoveralltests.BelowistheexperimentalresultinTable 6-2 .WeobservedthatcorrectidenticationratesofNDAandAPIaremuchhigherthanthecorrectidenticationrateofPIalgorithm.ThereasonAPIalgorithmoutperformsPIalgorithmisthatAPIalgorithmdoesnotdepends 80

PAGE 81

6-3 .Krepresentsthenumberofplanesinatestdataset.ItshowsthatNDAalgorithmoutperformsbothPIandAPIalgorithm.TheaveragesquaredapproximationttingerrorofNDAalgorithmislessthan50%comparetothettingerrorofPIalgorithm.However,theperformancegainislesscomparedtotherstexperiment.Thereasonisthatthenon-linearmappinginNDAdependsontheestimationofthelocalgeometricstructures.Whiletheestimationofthelocalgeometricstructuresisverysensitivetotheaddednoises.Eventhoughtheperformancegainisless,wecanstillsaythattheNDAalgorithmoutperformsbothPIandAPIalgorithmsintheaveragesquaredapproximationerrorfromtheexperimentalresult.Wealsoshowtheexperimentalresultin3DviewinFigure 6-1 andFigure 6-2 .Figure 6-1 showsthesegmentationresultsoftestdataset1withthreeplanesbytheNDAalgorithm.Figure 6-2 showsthesegmentationresultsofthesametestdatasetbythePIalgorithm. 6-3 showsthe 81

PAGE 82

Theaveragesquaredapproximationerror. KPIAPINDA 33.771013.001091.17101244.011019.811082.21101252.431012.861093.06101262.941018.8011093.001012 6-4 showsthegeometricsegmentationresultbyNDAalgorithmandFigure 6-5 showsthegeometricalsegmentationresultbyPIalgorithm.ItisprettyclearthatNDAalgorithmpartitionstheinputdatasetintothreeclustersandeachclusterrepresentsawallintheimage.PIalgorithmfailstondthegeometricmodelofthewallsandthedatapointsaremixed.TheexperimentalresultonrealworlddatashowsthatNDAalgorithmcanwellsegmentthedatasetsbasedontheirgeometricrelationship. 82

PAGE 83

2,1 2],8i. Thecorrectidenticationrate. KPIAPINDA 383%96%99%479%93%99%582%94%97%678%97%98%

PAGE 84

Table6-3. Theaveragesquaredapproximationerror. KPIAPINDA 36.611018.961012.4110148.181015.981013.1910156.981014.421013.9610161.169.441016.71101 Thesyntheticdataset. 84

PAGE 85

TherstgrouppartitionedbyK-means. 85

PAGE 86

Theinputdatapointsonthe1stframeof`oldhousing'imagesequence. 86

PAGE 87

ThegeometricalsegmentationresultbytheNDAalgorithmof`oldhousing'dataset. 87

PAGE 88

ThegeometricalsegmentationresultbythePIalgorithmof`oldhousing'dataset. 88

PAGE 89

105 { 108 ].Currently,mostofthesystemsandapplicationsin3Dreconstructionareusedforvisualinspectionandarchitecturemodeling.However,thereismoredemandfor3Dentertainment,forexample,3Dmovies.Thechangeofdemandresultsinanattentionforsmoothvisualqualityofthereconstructedscene.Inthiscase,visualqualityofthevirtualscenebecomesthedominantfactor.Whiletheforemostgoalinpreviousapproachesistheaccuracyofthepositionofeachpointin3Dgeometry. Inthelasttwodecades,tremendousprogresshasbeenmadeonself-calibrationand3Dsurfacemodeling[ 109 { 112 ].Mostofthemethodsuse2Dvideosequencesor2Dimagesasinputandtrytoretrievethedepthinformationofthescenecapturedbytheinputvideosequence.Theestimateddepthinformationhelpstoreconstructthefull3Dviewofthescene.Theexistingtechniquesareabletowellcalculatethecameramotionandcomputeasparsedepthmapfromtheoriginalimagesequence[ 105 113 { 116 ].However,fullyreconstructionofa3Dscenerequiresthedepthinformationofmuchmoreimagepixelswhichrequiresthealignmentofalmostallpixelsoftheinputimages.Thisproblemisknownasdensematchingproblem[ 117 { 119 ]. Atraditionalsolutiontothedensematchingproblemiscalledepi-linesearching.Epi-linesearchmethodusesthegeometricconstraintstodegradea2Dsearchingtoa1Drangesearching[ 120 { 122 ].Althoughthesearchisconstraintto1Dwhichseemseasiertosearch,theblankwallproblem,whichisnotsolvedin2Dfeaturecorrespondence,stillexistinepi-linesearch.Theblankwallproblemisthatgivenatexturelessblankwall,itisveryhardtondanaccuratepixeltopixelcorrespondenceacrosstheinputimages. 89

PAGE 90

107 123 ].Insteadofusingpixel-basedsearchingandmatching,volumetricreconstructiontakesthesceneasatessellationof3Dcubes,calledvoxels.Eachvoxelmaybeeitheremptyoroccupiedbythescenestructure.Variousmethodshasbeenproposedtobuildthevolumetricmodelwhichisusedtogeneratethemostconsistentprojectionswiththeoriginalimages.Volumetricreconstructioncouldwellrecoverthesceneofthemovingforeground,however,itishardtorevealthestaticbackgroundstructureusingvolumetricmethods. Inthischapter,weproposeanovel3Ddensereconstructionmethodbasedongeometricsegmentationandsurfacetting.Weusetheexistingtechniquesforfeaturecorrespondence,projectivereconstructionandself-calibrationtogetthesparsepointsreconstruction.Toaddressthedensematchingproblem,weusegeometricsegmentationtosegmentthe3Dspaceintoseveralseparateregions,andforeachregion,weestimatethedense3Ddepthmapbysurfacetting.Weproposeanon-lineardeterministicannealingalgorithminordertopartitionthe3Dspacegeometrically.Withtheassumptionthateachsubspacecouldbemodeledbyalinearplane,wecanretrievethedepthinformationforeachpixelusingsurfacetting.Thenewapproachisabletogenerateamuchsmoother3Ddensereconstructioncomparingtothetraditionalmethods. Thischapterisorganizedasfollows.Section 7.2 presentthebackgroundandproblemformulation.Wepresentthesystemschemefor3DreconstructioninSection 7.3 .ThenwesolvethegeometricsegmentationandsurfacettingprobleminSection 7.4 .TheexperimentalresultsareshowninSection 7.5 .Finally,Section 7.6 concludesthispaper. 90

PAGE 91

122 ].ThepipelineisgiveninFigure 7-1 Therststepin3Dreconstructionfromavideosequenceistogroupthewholevideosequenceintoseveralscenesbykeyframes.Foreachscene,motiondetectionisneededtondmovingregionsfromthestaticbackground.Inthelaterpart,movingforegroundandstaticbackgroundwillbetreatedseparatelyandthencombinedtogethertoreconstructthesceneasawhole. Thesecondstepissparsereconstruction.Sparsereconstructionincludesseveralcomponent,featurecorrespondence,projectionreconstructionandEuclideanreconstruction.ThecameramotionisestimatedandTheEuclideanstructureofthestaticbackgroundsceneisrecovered.Forthemovingregions,weintroducethevirtualcameraconceptandapplythesamereconstructionalgorithmtorecoverthe3Dstructure.Duringthelasttwodecades,tremendousprogresshasbeenmadetocameraself-calibrationandstructurecomputation.Sparsereconstructionstartsfromfeaturecorrespondencewhichisthemostcrucialpartoftheprocess.ThegoalofImagecorrespondence,alsocalledfeaturecorrespondence,istoaligndierentimages,fromavideosequenceortakenseparately,byndingcorrespondingpointsthatdescribethesamepointin3Dgeometry[ 124 125 ].Asknowntoall,notallpointsaresuitableformatchingortrackingthroughdierentimages,soonlyafewpointsareselectedasfeaturepointsformatching[ 49 ].Sosparsereconstructiononlyrelyonanumberofdistinctpointswhichisdierentfromthefollowingdensereconstructionwhichrequirethecorrespondenceofallpoints,ifpossible.Furthermore,featurepointsmaybemismatched,knownasoutliers[ 126 ],whichmayrestricttheaccuracyofthereconstructionresult.Givencorrectlymatchedfeaturepointsfromtwoinputimages,projectionreconstructionistondtherelativeposebetween 91

PAGE 92

Thesparsereconstructiongivesasparsestructureofthedesiredscene;however,itcouldnotgiveasatisedvisualpresentation.Thus,westillneedtocomputethedepthofalotmorepoints,whichisknownasdensereconstructionorsurfacereconstruction.Thetraditionalapproachesfordensereconstructioncouldbeclassiedastwoapproaches,namelystereoscopicreconstructionandvolumetricreconstruction.Inthischapter,weproposeanovelapproachtoobtainthestaticbackgroundstructure.Unlikethepreviousapproach,weapplygeometricalsegmentationandsurfacettinginsteadofdensesearchingandmatching.Hereweassumethatthestaticbackgroundcouldbedecomposedofseveraluniformregionsorregularsurfaces.Wecanthensegmentthewholesurfaceintoseveralregionsbasedontheirgeometricproperties.Foreachregion,weobtainamathematicalexpressionbysurfacetting.Withtheassumptionthateachregionhassucientnumberofsparsefeaturepoints,combinedwiththesparsedepthmap,wecouldthencomputethedepthinformationbyttingeachpixelwithintheestimatedsurface.Combiningthedepthmapofdierentregions,wecouldnallyobtainthedepthmapofthewholescene.Themeritofthisapproachisthatitwellhandlesuniformregions 92

PAGE 93

7.2.2 andwegivethesolutiontotheproblemindetailsinSection 7.4 Givena3DpointdatasetX=fxig,xi2R3,i=1,2,...,n,thegeometricalttingproblemisusuallystatedastheoptimizationofacostthatmeasureshowthegeometricalsurfacefunctionS=fx:g(x)=0gtsthedatasetX.Themostcommonlyusedobjectivefunctionistheleastsquarescost, Thettingfunctiongislearnedbyminimizingthedesigncost,D,measuredovertheinputdataset,X.Itiswell-knownthatformostchoicesofD,thecostmeasuredduringdesignmonotonicallydecreasesasthesizeofthelearnedttingfunctiongisincreased.Withalargesetoffunctions,itiseasytocreateasurfacewhichpassesthrougheachinputdatapointbutissuspiciouslycomplicated.TheprincipleofOccam'srazorstatesthatthesimplestmodelthataccuratelyrepresentsthedataismostdesirable.Soweprefertouseafewbasisfunctionswhichyieldasmoother,simplersurfacewhichcouldwellapproximatestheoriginaldata.Generally,therearetwoapproachestosolvetheoverttingproblem.Oneapproachistoaddpenaltytermstothedataset,likesmoothnessorregularizationconstraints.Anotherapproachistorstbuildalargemodelandthenremovesomeparametersbyretainingonlythevitalmodelstructure.Althoughbothapproachescangenerateparsimoniousmodels,thedescentbasedlearningmethodsallsuerfromaseriouslimitation.Thenon-globaloptimaofthecostsurfacemayeasilyresultinpoorlocal 93

PAGE 94

111 ]onwhichourexperimentsarebased.Whendevelopingastereovisionalgorithmforregistration,therequirementsforaccuracyvaryfromthoseofstandardstereoalgorithmsusedfor3Dreconstruction.Forexample,amulti-pixeldisparityerrorinanareaoflowtexture,suchasawhitewall,willresultinsignicantlylessintensityerrorintheregisteredimagethanthesamedisparityerrorinahighlytexturedarea.Inparticular,edgesandstraightlinesinthesceneneedtoberenderedcorrectly. 7.4 .Finallythedensedepthmapisreconstructedbygeometrictting.ThesystemschemeisgiveninFigure 7-2 111 ]usepointfeatureinreconstructionwhichismeasuredbyHarris'criterion, 94

PAGE 95

whereW(x)isarectangularwindowcenteredatxandIxandIyarethegradientsalongthexandydirectionswhichcanbeobtainedbyconvolvingtheimageIwiththederivativesofapairofGaussianlters.Thesizeofthewindowcanbedecidedbytheuser,forexample77.IfC(x)exceedsacertainthreshold,thenthepointxisselectedasacandidatepointfeature. Weusethesumofsquareddierences(SSD)asthemeasurementofthesimilarityoftwopointfeatures.Thenthecorrespondenceproblembecomeslookingforthedisplacementdthatsatisesthefollowingoptimizationproblem: mindXx2W(x)[I2(x+d)I1(x)]2(7{5) wheredisthedisplacementofapointfeatureofcoordinatesxbetweentwoconsecutiveframesI1andI2.LucasandKanadealsogivethecloseformsolutionofequation( 7{5 ). where 7{3 ),andIt.=I2I1. 95

PAGE 96

111 ].Forthedetailoftheproofofthisalgorithm,pleaserefertothereference. Thereconstructionalgorithmisbasedonaperspectiveprojectionmodelwithapinholecamera.Supposewehaveagenericpointp2E3withcoordinatesX=[X,Y,Z,1]Trelativetoaworldcoordinateframe.Giventwoframesofonescenewhichisrelatedbyamotiong=(R,T),thetwoimageprojectionpointx1andx2arerelatedasfollows: wherex0=[x,y,1]Tismeasuredinpixels,1and2arethedepthscaleofx1andx2,1=[K,0]and2=[KR,KT]arethecameraprojectionmatricesandKisthecameracalibrationmatrix.Inordertoestimate1,2,1and2,weneedtointroducetheepipolarconstraint.Fromequation( 7{8 ),wehave Thefundamentalmatrixisdenedas: Withtheabovemodel,wecouldestimatethefundamentalmatrixFviatheEight-pointalgorithm.Thenwecoulddecomposethefundamentalmatrixtorecovertheprojectionmatrices1and2andthe3Dstructure.Weonlygivethesolutionherebycanonicaldecomposition: 96

PAGE 97

wheremeansequalityuptoascalefactorand WiththeassumptionthatKisconstant,wecouldestimatetheunknownsKandwithagradientdecentoptimizationalgorithm.Inordertoobtainauniquesolution,wealsoassumethatthesceneisgenericandthecameramotionisrichenough. Inlastchapter,wehaveproposedanewgeometricttingmethodbasedongeometricsegmentation.Werstsegmentthesurfaceofthe3Dsceneintoseveralregionsbasedonthegeometricrelationship.Foreachsmallhomogeneoussurface,weareabletomodelitbyaplane.Withthedepthinformationofthefeaturepointsthatwealreadygetfromthesparsereconstruction,wecouldcomputethedepthinformationforeachpixelintheentireregion.Sincethedepthinformationweobtainedisbasedonaplanemodel,theimagerenderedfromthe3Dmodelismuchsmootherthanthetraditionalapproaches.Inordertosimplifytheproblemofsurfacetting,werstsegmenttheinputimagebasedonitsgeometricstructure.Itisdierentfromthetraditionalobjectbasedimagesegmentation.Thesegmentationprocessiscriticalbecausepropersegmentationcouldsimplifythe 97

PAGE 98

Duetothefactthatthe3Ddataislocalizedtoafewrelativelydenseclusters,wedesignanon-linearfunctiontomapthedatapointfromgeometricalspacetosurfacemodelspaceandapplydeterministicannealinginthefeaturespacetopartitionthefeaturespaceintoseveralregionswithdierentsizesandshapes.Foreachregion,wecaneasilyndalinearplanemodeltotthedata.Non-lineardeterministicannealingmethodoersthreeimportantfeatures:1)theabilitytoavoidmanypoorlocaloptima;2)theabilitytominimizethecostfunctionevenitsgradientsvanishalmosteverywhere;3)theabilitytoachievenon-linearseparation.However,thereisnocloseformsolutionfornon-lineardeterministicannealingproblem,thereforeweuseagradientdescentalgorithmtosolvethisproblem.ThedetailsofthisalgorithmisdiscussedinSection 7.4 whereA=[Xie1],i=1,...,mandp=[a,b,c]Tistheplaneparameter. Givenanarbitrarypointxi=[xi,yi]Tmeasuredinpixelsintherstcluster,wecouldestimateit'sdepthscaleibysolvingthefollowingequation. 98

PAGE 99

7{15 ,onlyiisunknownandwiththeconstraintonXiewithequation 7{14 ,wecouldeasilygetthevalueofi. Then,with1=[I,0],wecouldhaveXip=[i1xi,i1yi,i1,1].fromequation 7{8 ,wecangettherelationbetweentwoimageprojectionpointxi1andxi2asfollows: wherecxi20=[i2xi2,i2yi2,i2].Wecouldthengetthepositionofthecorrespondingpointxi2=[xi2,yi2]inthesecondimage. Inlastchapter,wehaveproposedanon-lineardeterministicannealingapproachforspacepartitioningin3DEuclideanspace.Weusedeterministicannealingtodividetheinputspaceintoseveralregionswithdierentsizesandshapes.Withthepartition,wecaneasilyndalinearlocalsurfacetotthedatawithineachregion.Deterministicannealingmethodoerstwogreatfeatures:1)theabilitytoavoidmanypoorlocaloptima;2)theabilitytominimizethecostfunctionevenitsgradientsvanishalmosteverywhere.Duetothefactthatthedataislocalizedtoafewrelativelydenseclusters,wedesignanon-linear 99

PAGE 100

6 .Forreader'sconvenience,webrieyrepeatthealgorithminthissubsection. Tosolvethespacepartitioningproblem,wedonotuseprototypetocalculatethedierence.Thereasonisthattheprototypeinspacepartitioningisgenerallynotsucienttorepresentaplanein3Dspace.Instead,weestimatethelinearplanemodelandcalculatethettingerrorastheEuclideandistancebetweenthedataandtheplane.Thetraditionallocaloptimizationalgorithmwilllikelystuckatalocaloptima.Inordertoavoidlocaloptima,weuselocalgeometricstructurefromneighboringdatapointsandembeddedthedatavectorstoahigherdimensionasfollows. Theinputdataisgivenasa3Dpoint,xi=[xi,yi,zi]T.Withtheassumptionthatnearestdatapointsareonthesameplane,wecouldestimatethelocalplanemodel,Li=[ai,bi,ci]TofdatapointxianditsKnearestneighborpoints. 100

PAGE 101

Thenwerevisethedistortionfunctionasfollows, whereD1=di,jcalculatethettingerrorbetweenthedatapointandtheestimatedplane,andD2calculatethedierencebetweenthelocalestimatedplanemodelandtheclusterscaleestimatedplanemodel.D2isdenedasfollows: Afterthemapping,weapplydeterministicannealingalgorithmtopartitionthedataintoseveralclustersasfollows. wheregj=[aj,bj,cj]isthegeometricalsurfacemodelparametertobeestimated,DisthesumofsquareofgeometricalttingerrorandHistheentropyconstraint.WedeneDandHasfollows: 101

PAGE 102

7{25 )byapplyingthechainruleofentropyas NoticethatthersttermH(X)istheentropyofthesourceandisthereforeconstantwithrespecttotheclustergjandassociationprobabilitiesp(gjjxi).Thuswecanjustfocusontheconditionalentropy TheminimizationofFwithrespecttoassociationprobabilitiesp(gjjxi)givesrisetotheGibbsdistribution wherethenormalizationis ThecorrespondingminimumofFisobtainedbypluggingequation( 7{28 )backintoequation( 7{23 ) TominimizetheLagrangianwithrespecttotheclustermodelgj,itsgradientsaresettozeroyieldingthecondition 7.5.13DVideoDenseReconstruction 102

PAGE 103

7-3 showstherstframeandthe88thframeofthetestimagesequence`oldhousing'.Werstextractpointfeaturesonalltheinputimages.Thenweapplyfeaturecorrespondencealgorithmtorelateallthefeatures.Figure 7-4 showtheselectedfeaturepointsontherstframe.Wethenestimatethecameraposeandintrinsicparameters.Withthecameraparameters,weareabletorecoverthesparseEuclidianstructureofthefeaturepoints.Figure 7-5 showstheestimateddepthmapoftheselectedfeaturepointsandthecamerapose.Aftersparsereconstruction,weseparatethe3DspaceintoseveralregionsusingNDAalgorithm.Foreachregion,weusethesurfacettingalgorithmpresentedinSection 7.3 toestimatethedepthinformationofeachpixel.Combiningthedepthmapofallregions,wecanrecoverthe3Ddensedepthmapofthewholeframe.Figure 7-6 showstheestimateddensedepthmapofthewholeframe.Sinceweusesurfacettinginsteadofsearchingfordensedepthestimation,wedonotneedtoworryaboutmatchingerrorsandoutliers.Theestimateddensedepthmapisverysmoothandwellrepresentthegeometricstructureofthe3Dscene. 103

PAGE 104

Thepipelinefor3Dvideoreconstructionsystem. 104

PAGE 105

Theschemefor3Dvideoreconstructionsystem. BThe88thframeinthe`oldhousing'videosequence Originalframesusedforimageregistration. 105

PAGE 106

Thefeaturepointsontherstframeof`oldhousing'imagesequence. 106

PAGE 107

Theestimatedsparsedepthmapandcameraposefortheselectedfeaturepointsofthe1stand88thframes. Figure7-6. Theestimateddense3Dconguration. 107

PAGE 108

55 ],includingmanyclassicmethodsstillinuse.Duetotherapiddevelopmentofimageacquisitiondevices,moreimageregistrationtechniquesemergedafterwardsandwerecoveredinanothersurveypublishedin2003[ 56 ]. Dierentapplicationsduetodistinctimageacquisitionrequiredierentimageregistrationtechniques.Ingeneral,mannersoftheimageacquisitioncanbedividedintothreemaingroups: Theprevailingimageregistrationmethods,suchasDavisandKeck'salgorithm[ 63 127 ],assumeallthefeaturepointsarecoplanarandbuildahomographytransformmatrixtodoregistration.Theadvantageisthattheyhavelowcomputationalcostandcanhandleplanarscenesconveniently;however,withtheassumptionthatthescenesareapproximatelyplanar,theyareinappropriateintheregistrationapplicationswhentheimageshavelargedepthvariationduetothehigh-riseobjects,knownastheparallax 108

PAGE 109

Inthischapter,weproposeadepthbasedimageregistrationalgorithmbyleveragingthedepthinformation.Ourmethodcanmitigatetheparallaxproblemcausedbyhigh-risescenesintheimagesbybuildingaccuratetransformfunctionbetweencorrespondingfeaturepointsinmultipleimages.Givenanimagesequence,werstselectanumberoffeaturepointsandthenmatchthefeaturesinallimages.Thenweestimatethedepthofeachfeaturepointfromfeaturecorrespondences.Withthedepthinformation,wecanprojecttheimagein3Dinsteadofusingahomographytransform.Furthermore,afastandrobustimageregistrationalgorithmcanbeachievedbycombiningthetraditionalimageregistrationalgorithmsanddepthbasedimageregistrationmethod.Theideaisthatwerstcomputethe3Dstructureofasparsefeaturepointssetandthendividethescenegeometricallyintoseveralapproximatelyplanarregions.Foreachregion,wecanperformadepthbasedimageregistration.Accordingly,robustimageregistrationisachieved. Theremainderofthischapterisorganizedasfollows.Wepresentthesystemschemefor2DimageregistrationinSection 8.2 .Section 8.3 reviewsthe3Dreconstructionalgorithmweusedinournewmethod.InSection 8.4 ,wedescribehowtouse3Ddepthinformationfor2Dimageregistrationandproposeanon-lineardeterministicannealingalgorithmforspacepartitioning.Section 8.5 presentstheexperimentalresultsandwecompareouralgorithmwithDavisandKeck'salgorithmonthesametestvideosequence.WeconcludethispaperinSection 8.6 109

PAGE 110

8-1 Awidelyusedfeaturedetectionmethodiscornerdetection.KitchenandRosenfeld[ 57 ]proposedtoexploitthesecond-orderpartialderivativesoftheimagefunctionforcornerdetection.DreschlerandNagel[ 58 ]searchedforthelocalextremaoftheGaussiancurvature.However,cornerdetectorsbasedonthesecond-orderderivativesoftheimagefunctionaresensitivetonoise.ThusForstner[ 59 ]developedamorerobust,althoughtimeconsuming,cornerdetector,whichisbasedontherst-orderderivativesonly.ThereputableHarrisdetector[ 60 ]alsousesrst-orderderivativesforcornerdetection.Featurematchingincludesarea-basedmatchingandfeature-basedmatching.Classicalarea-basedmethodiscross-correlation(CC),whichexploitsformatchingimageintensitiesdirectly.Forfeature-basedmatching,Goshtasby[ 61 ]describedtheregistrationbasedonthegraphmatchingalgorithm.Clusteringtechnique,presentedbyStockmanetal.[ 62 ],triestomatchpointsconnectedbyabstractedgesorlinesegments.Afterthefeaturecorrespondencehasbeenestablishedthemappingfunctionisconstructed.Themappingfunctionshouldtransformthesensedimagetooverlayitoverthereferenceimage.Andnallyinterpolationmethodssuchasnearestneighborfunction,bilinear,andbicubicfunctionsareappliedtotheoutputoftheregisteredimages. 110

PAGE 111

8-2 .Inthenewsystemscheme,werstapply3Dreconstructiontotheinputimagesandrecoverthe3Dgeometricstructureofthesceneintheimages.The3Dmodelismoreaccuratecomparedtothe2Dmotionmodelsestimatedinthepreviousworks.Thenwesegmentthe3DEuclideanspacegeometricallyintoseveralseparateregions.Eachregioncouldbemodeledbyalinearplane.Withthesegmentation,wecanestimatethe3Ddepthforeverypixelineachregionandrecoverthedensestructureofthescene.The3Ddensestructureenablesthepixelbypixelmappingoftheinputimages.Wedescribethe3DreconstructionalgorithminSection 8.3 .InSection 8.4 ,wepresentthegeometricsegmentationanddepthbasedmappingin3D,andalsoproposeanon-lineardeterministicannealingalgorithmforspacepartitioning. 111 ].Whendevelopingastereovisionalgorithmforregistration,therequirementsforaccuracyvaryfromthoseofstandardstereoalgorithmsusedfor3Dreconstruction.Forexample,amulti-pixeldisparityerrorinanareaoflowtexture,suchasawhitewall,willresultinsignicantlylessintensityerrorintheregisteredimagethanthesamedisparityerrorinahighlytexturedarea.Inparticular,edgesandstraightlinesinthesceneneedtoberenderedcorrectly. The3Dreconstructionalgorithmisimplementedusingthefollowingsteps.First,geometricfeaturesaredetectedautomaticallyineachindividualimages.Secondly,featurecorrespondenceisestablishedacrossalltheimages.Thenthecameramotionisretrievedandthecameraiscalibrated.FinallytheEuclideanstructureofthesceneisrecovered. 111 ]usepointfeatureinreconstruction 111

PAGE 112

wherex=[x,y]Tisacandidatefeature,C(x)isthequalityofthefeature,kisapre-chosenconstantparameterandGisa22matrixthatdependsonx,givenby whereW(x)isarectangularwindowcenteredatxandIxandIyarethegradientsalongthexandydirectionswhichcanbeobtainedbyconvolvingtheimageIwiththederivativesofapairofGaussianlters.Thesizeofthewindowcanbedecidedbythedesigner,forexample77.IfC(x)exceedsacertainthreshold,thenthepointxisselectedasacandidatepointfeature. Weusethesumofsquareddierences(SSD)[ 124 ]asthemeasurementofthesimilarityoftwopointfeatures.Thenthecorrespondenceproblembecomeslookingforthedisplacementdthatsatisesthefollowingoptimizationproblem: mind.=Xx2W(x)[I2(x+d)I1(x)]2(8{3) wheredisthedisplacementofapointfeatureofcoordinatesxbetweentwoconsecutiveframesI1andI2.LucasandKanadealsogivethecloseformsolutionofequation( 8{3 ): 112

PAGE 113

8{1 ),andIt.=I2I1. 111 ].Forthedetailoftheproofofthisalgorithm,pleaserefertothereference. Thereconstructionalgorithmisbasedonaperspectiveprojectionmodelwithapinholecamera.Supposewehaveagenericpointp2E3withcoordinatesX=[X,Y,Z,1]Trelativetoaworldcoordinateframe.Giventwoframesofonescenewhichisrelatedbyamotiong=(R,T),thetwoimageprojectionpointx1andx2arerelatedasfollows: wherex0=[x,y,1]Tismeasuredinpixels,1and2arethedepthscaleofx1andx2,1=[K,0]and2=[KR,KT]arethecameraprojectionmatricesandKisthecameracalibrationmatrix.Inordertoestimate1,2,1and2,weneedtointroducetheepipolarconstraint.Fromequation( 8{6 ),wehave Thefundamentalmatrixisdenedas: Withtheabovemodel,wecouldestimatethefundamentalmatrixFviatheEight-pointalgorithm[ 111 ].Thenwecoulddecomposethefundamentalmatrixtorecovertheprojectionmatrices1and2andthe3Dstructure.Weonlygivethesolutionhere 113

PAGE 114

wheremeansequalityuptoascalefactorand WiththeassumptionthatKisconstant,wecouldestimatetheunknowns,Kand,withagradientdecentoptimizationalgorithm.Inordertoobtainauniquesolution,wealsoassumethatthesceneisgenericandthecameramotionisrichenough. 63 127 ],trytoregisterthetwoimagesbycomputingthehomographymatrixHbetweencorrespondingfeaturepoints.Thelimitofthisalgorithmisthattheyassumeallthepointsinthephysicalworldarecoplanarorapproximatelycoplanar.Theassumptionisnottruewithhigh-risescenes.Inordertomitigatethisproblem,weproposeanovelalgorithmwhichrstsegmentstheimagegeometricallyandthenperformtheregistrationtoeachregionwithdepthestimation. 114

PAGE 115

8.3 .Withtheassumptionthateachsegmentregionofthesceneisapproximatelycoplanarinthephysicalworld,wecouldeasilyestimatetheplanemodelandprojectthe3Dplaneontotheimageframes.Comparedwiththetraditionalassumptionthatthewholesceneiscoplanarinthephysicalworld,ourassumptionisvalidinmostcircumstances. Therearealotofalgorithmsfordataclustering.Themostfamoushard-clusteringalgorithmisk-means[ 128 ].Thek-meansalgorithmassignseachdatapointtotheclusterwhosecentroidisnearest.Here,weusethedistancetoa3Dplaneinthephysicalworldasthemeasurement.Foreachcluster,wecouldchoosetheplanethathasthesmallestsumofdistanceofallthedatapointsinthecluster.However,thedescentbasedlearningmethodssuerfromaseriouslimitation.Thenon-globaloptimaofthecostsurfacemayeasilyresultinginpoorlocalminimatotheabovemethods.Techniquesaddingpenaltytermstothecostfunctionfurtherincreasesthecomplexityofthecostsurfaceandworsenthelocalminimumproblem. Inthissection,wepresentanon-lineardeterministicannealingapproachtosolvethe3Dgeometricalttingproblem.ThealgorithmisrstintroducedinChapter 6 Theinputdataisgivenasa3Dpoint,xi=[xi,yi,zi]T.Withtheassumptionthatnearestdatapointsareonthesameplane,wecouldestimatethelocalplanemodel,Li=[ai,bi,ci]TofdatapointxianditsKnearestneighborpoints. 115

PAGE 116

Thenwerevisethedistortionfunctionasfollows, whereD1=di,jcalculatethettingerrorbetweenthedatapointandtheestimatedplane,andD2calculatethedierencebetweenthelocalestimatedplanemodelandtheclusterscaleestimatedplanemodel.D2isdenedasfollows: Afterthemapping,weapplydeterministicannealingalgorithmtopartitionthedataintoseveralclustersasfollows. wheregj=[aj,bj,cj]isthegeometricalsurfacemodelparametertobeestimated,DisthesumofsquareofgeometricalttingerrorandHistheentropyconstraint.WedeneD

PAGE 117

Toperformoptimizationweneedtofurtheranalyzeitsterms.Wecanrewriteequation( 8{20 )byapplyingthechainruleofentropyas NoticethatthersttermH(X)istheentropyofthesourceandisthereforeconstantwithrespecttotheclustergjandassociationprobabilitiesp(gjjxi).Thuswecanjustfocusontheconditionalentropy TheminimizationofFwithrespecttoassociationprobabilitiesp(gjjxi)givesrisetotheGibbsdistribution wherethenormalizationis ThecorrespondingminimumofFisobtainedbypluggingequation( 8{23 )backintoequation( 8{18 ) TominimizetheLagrangianwithrespecttotheclustermodelgj,itsgradientsaresettozeroyieldingthecondition 117

PAGE 118

whereA=[Xie1],i=1,...,mandp=[a,b,c]Tistheplaneparameter. Givenanarbitrarypointxi=[xi,yi]Tmeasuredinpixelsintherstcluster,wecouldestimateitsdepthscaleibysolvingthefollowingequation: wherex0i=[xi,yi,1]T,H11and1areestimatedinSection 8.3 .Inequation( 8{28 ),onlyiisunknownandwiththeconstraintonXiewithequation( 8{27 ),wecouldeasilygetthevalueofi. Then,with1=[I,0],wehaveXip=[i1xi,i1yi,i1,1].Fromequation( 8{6 ),wegettherelationbetweentwoimageprojectionpointsxi1andxi2asfollows: wherecxi20=[i2xi2,i2yi2,i2].Wecouldthengetthepositionofthecorrespondingpointxi2=[xi2,yi2]inthesecondimage. 118

PAGE 119

8.3 Inourexperiment,weregardtherstimage'slocalcoordinatesystemasworldcoordinatesystemsotherstimagecanbeviewedasareferenceimage.Thentherestoftheimagesareregisteredtothereferenceimage.WealsoappliedthealgorithmproposedbyDavisandKeck[ 63 ]toregistertheinputimagesforcomparisonpurpose. Figure 8-3 istheregistrationresultusingouralgorithmandFigure 8-4 istheoutputofthealgorithmproposedbyDavisandKeck[ 63 ].Figure 8-5 showsthedierenceimagebetweentheregisteredimageandtherstimageusingouralgorithmandFigure 8-6 showsthedierenceimagefromthealgorithmofDavisandKeck.Wecanseethatourresultcanmitigatetheparallaxproblemsincetheroofandwallcornersareregisteredcorrectly;onthecontrary,theregisteredimagebythealgorithmofDavisandKeckhasalotofartifactscausedbytheparallaxproblem.WealsoshowsomeregistrationresultsusingouralgorithminFigure 8-7 throughFigure 8-8 InordertofurthercompareouralgorithmtothealgorithmproposedbyDavisandKeck,wecomputetherootofmeansquarederrors(RMSE)oftheregistrationresultsfrombothalgorithms.Figure 8-9 showsthattheregistrationerrorofouralgorithmislessthan50%thanthatofthealgorithmproposedbyDavisandKeck. Theresultshowsthatourimageregistrationalgorithmcanmitigatetheparallaxproblembecausemostofthesceneisregisteredwithoutvibration,asopposedtotheregistrationresultsunderthealgorithmofDavisandKeckinwhichthehigh-risesceneinthesensedimagessignicantlymovedafterregistrationtothereferenceimages.ThereasonisthatthealgorithmofDavisandKeckassumesallthepointsintheimagesarecoplanar.Whilethisassumptionissatisedwhenthedistancebetweenthecameraand 119

PAGE 120

Finally,wewouldliketopointoutthatthealgorithmproposedbyDavisandKeck[ 63 ]assumesaplanarregistration.Theirschemewasdesignedforusewithhigh-altitudeaerialimagerywhereplanartransformationsarefairlygoodapproximations.Furthermore,theirschemeusesRANSACtoremovepoormatchingpointsduringthecomputation.Thiscanhelptodealwithsomedepthdiscontinuitiesthatmaybepresentinthehigh-altitudeaerialimages.Inourexperiments,thetestimagescontainsalient3Dscenes;theseimagesareoutofthedomainforthealgorithmofDavisandKeck.ThisisthereasonwhythealgorithmofDavisandKeckdoesnotperformwell. Ourfutureworksinclude: 129 ][ 130 ]givenavideosequence.Thereliabilityofthedepthestimatesiscrucialtodepth-basedregistrationalgorithm;therefore,thehighlyrobust3Dreconstructiontechniqueisrequiredtoimplementouralgorithm.Uptonow,mostrecentdepthrecoveryalgorithmsreportedintheliteratureclaimtorecoverconsistentdepthfromsomechallengingvideosequences[ 129 ][ 130 ].Wecanapplyormodifythisstate-of-the-artdepthmaprecoverymethodtodevelopdepth-basedimageregistrationalgorithm. 120

PAGE 121

Thepipelinefor2Dimageregistrationsystem. 121

PAGE 122

Thenewimageregistrationsystemscheme. Figure8-3. Ouralgorithmtestresult,inwhichthe88thframeisregisteredtotherstframe. 122

PAGE 123

ThetestresultofDavisandKeck'salgorithm,inwhichthe88thframeisregisteredtotherstframe. Figure8-5. Thedierenceimagebetweentheregistered88thimageandtherstimage(usingouralgorithm). 123

PAGE 124

Thedierenceimagebetweentheregistered88thimageandtherstimage(usingDavisandKeck'salgorithm). Figure8-7. The37thframeinthe`oldhousing'videosequence. 124

PAGE 125

Ouralgorithmtestresult,inwhichthe37thframeisregisteredtothe1stframe. Figure8-9. OuralgorithmtestresultcomparingtothatunderthealgorithmofDavisandKeck,inwhichallthe88framesareregisteredtothe1stframe. 125

PAGE 126

25 ].Forasingleframe,imagesegmentationisthetraditionalwaytoanalysisthescene[ 26 ].However,incasewehaveasequenceofimages,whentherearedierentmovingobjectsinthescene,orobjectsatdierentdepthswithaglobalcameramotion,motiondiscontinuitywilloccur.Inthiscase,motionofdierentobjectscanprovidemoreessentialinformationtounderstandthescene[ 27 ].Therefore,motionsegmentationisneededtodividetheframeofanimagesequenceintoregionsbelongingtodierentmotions. Motionsegmentationcouldbedirectlyappliedtomanyareas.Suchasvideocompression,videodatabasequerying,andsceneanalysis.MPEG-4standard[ 28 ]describesacontentbasedmanipulationofobjectsinimagesequences.Tocreateanobjectbasedscenerepresentation,itisnecessarytosegmentdierentobjectsinaframe.Sincebackgroundtypicallychangeslessthantheobjectsmotion,whichindicatesdierentcompressionrates,thissegmentationisbasedmoreonmotioninformationofthescene.Videoquerying[ 29 ]isanotherneweldwhichaimstoautomaticallyclassifyvideosequencesbasedontheircontent.Acommonvideoquerytaskrequiresretrievingalltheimagesinadatabasethathaveasimilarcontenttothequeryexampleimage.Motionsegmentationenablesanindexingschemethatusesthetrajectories,shapesandowvectorsoftheindependentlymovingobjectstoquerythesequencesinadatabase.Therefore,thesystemwillgiveamoreaccurateresult.Thedevelopmentofunmannedaerialvehicle[ 30 ]makesreconnaissancemucheasierwhichrequiressceneanalysistechnologytoidentifysuspiciousmilitaryvehiclesinavideosequence.Objectshavingdierentmovingvelocitiesordirectionsneedtobeidentiedandsegmentedforfurther 126

PAGE 127

Inthischapter,weproposeanovelapproachbasedonbothpurelyperpixelopticaloweld.Thenoveltyofourworkisthatweintroducecodinglengthasacriterioningroupmerging.TheoriginalalgorithmisrstproposedinHong'sworkwhichisusedtostaticimagecompressionandsegmentation.Basedontheexperimentsimulationsandresults,weprovethatusingcodinglengthcouldgreatlyimprovetheperformanceofmotioneldsegmentation.Anothernoveltyofthismethodisthatweproposeaheuristicapproachtolocateglobalmotionbasedonthemotionsegments. Theremainderofthischapterisorganizedasfollows.Insection 9.2 ,weintroduceopticaloweldanddiscussthelimitationofthismotionestimationalgorithm.Section 9.3 describesourcodinglengthbasedmotionsegmentationapproach.Section 9.4 describesourapproachonglobalmotionlocation.Section 9.5 showstheexperimentalresultsandanperformanceevaluationoftheproposedalgorithmandwiththepreviousapproaches.Finally,wedescribefutureworkinsection 9.6 Thetraditionalapproachforcomputingopticalowcanbeclassiedintothreecategories,namely,featurebased,correlationbasedandgradientbased.Amongalltheapproaches,gradientbasedalgorithmsreceiveaspecialinterestforitsmathematicalsimplicityandrelativelycomputationaleciency.Inthispaper,weuseagradientbasedopticalowestimationalgorithm,whichisrstproposedbyLucasandKanade[ 124 ],toestimatethemotioneldofanimagesequence. 127

PAGE 128

Withtheassumptionthatthemovementofanobjectissmallenough,theimageconstraintatI(x,y,t)canbeexpandedwithTaylorseries:I(x+x,y+y,t+t)=I(x,y,t)+@I whereH.O.T.standsforhighordertermswhichareignoredhere. Fromtheaboveequation,wecanachieve: whichresultsin whereVx,Vyarevelocitiesalongxandydirections,andIx,Iy,Itarethespacialandtemporalderivativesofthepixelatposition(x,y,t). Thisistheequationwhichisknownastheapertureproblemofopticalowalgorithms.Inordertosolvethisproblem,anadditionalconstraintisneeded.InLucasandKanade'ssolution[ 124 ],theyuseanon-iterativemethodwhichassumesalocallyconstantow.Withthisassumption,weareabletosolveanoverconstraintsystemofequations,whichgivestheresult: whereAistheconstantowinasmallwindowofsizemm. 128

PAGE 129

Figure 9-1 and 9-2 showtwoframesoftheinputimagesequence.Whilegure 9-3 showsthecalculatedopticaloweld.Forabettervisualeect,weonlydrawmotionvectorsforeach5by5blocks.Theactualmotioneldiscalculatedperpixel. 9.3.1MinimalDescriptionLengthCriterion Atraditionaldenitionofsegmentationistochooseaclassofmodelswhicheachsubsetissupposedtot.Thetypicalapproachistodecomposethemixtureofallmodelsintoindividualonessimultaneouslyorsubsequently.Variousapproacheshavebeenproposedtoresolvethisproblem,suchasK-meansclusteringalgorithmandEMalgorithm,etc. Intheproblemofmotioneldsegmentation,thenumberofsegmentsisunknown.Therefore,determiningthenumberofmodelsforthedatasetisnecessaryanditisverydicult.Inordertosolvethisproblem,weproposeanewapproachwhichisbasedonminimumdescriptionlength(MDC)criterion. Supposewehaveadataset2RMN,whichisasetofrandomsamplesfromamixtureofmodels.Theoptimalsegmentationofthedataisthepartitioning=1[2...[Nthattheoverallcodinglengthofthedataisminimalamongallpossible 129

PAGE 130

Ingeneral,thelengthfunctionischosenaccordingtotheoptimalShannoncoding.However,becausesegmentsofthemotionelddataismultivariate,alossydatacompressionviewpointmayhelptosegmentthenoisydata. Thenwehavetheexpectedtotalnumberofbitsrequiredtoencodethedataaccordingtotheabovesegmentation:Ls(,)=NXn=1L(n)+jnj(log2(jnj=M))=NXn=1tr(n+K) 2log2det(I+K Thesuperscript`s'indicatesthecodinglengthaftersegmentation,andidenotesthediagonalmatrixthatencodestheprobabilityofMvectorsingroupi. 130

PAGE 131

Analyzingtheoverallcodinglengthequationhelpsustondtheoptimalsolutioninabottomupmannerbymergingregionsofsegments.InHongetal.'spaper[ 131 ],adetailedproofisgivenandwewillnotrewriteithere. asafunctionin. Sincethenumberofgroupsisunknown,wehavetominimizeRs(,)overN2Z+.Frompreviousworks,weknowthatanygradientbaseddescentalgorithmreliesontheinitializationofdatasetinordertoconvergetoglobalminimum.Becausemotionelddoesnotnecessarilysatisfythisrequirement,itisquitediculttominimizethecodinglengthfunctiondirectly.Instead,weuseasteepest-descentalgorithmtominimizethelengthfunctionLs(,). ThealgorithmisgiveninAlgorithm1.IneachstepwechoosetwosubsetsofvectorsS1,S2suchthatbymergingthetwosubsets,decrementinthecodinglengthisthelargest.Whenthedimensionofthespaceisrelativelylow,greedyalgorithmsusuallyperformwell.However,whenthedimensionofthesubspacebecomeshigh,greedyalgorithmsdonotalwaysconvergetotheoptimalsolution. 131

PAGE 132

Generally,globalmotionregioncontainsthecornerregionsofthescene.Althoughitisnotalwaystrue,westillcouldutilizethispropertycombinedwiththevariancestatisticsofthemotioneldsegmentstoestimatetheglobalmotion.Foreachoutputsegmentation,wewillndoutthemotionregionwhichcontainsmorecornersofthescene.Iftherearemorethanonesegmentscontainingcornersofthescene,thesegmentwiththeminimalvarianceofinsidemotionvectorswillbeconsideredastheglobalmotion. Forthechosenglobalmotion,wewillcalculatetheaveragemotiondirectionandvelocitybasedonthemotionvectorsinsidetheregiontorepresenttheglobalmotion.ThealgorithmisgiveninAlgorithm 5 9.5.1MotionFieldSegmentation Figure 9-4 istherstframeoftheinputimagesequence.Figure 9-5 showsthecalculatedmotioneldoftheimagesequence`coastguard'.Figure 9-6 9-7 9-8 and 9-9 aretheoutputofourproposedalgorithm.Eachregionhasdierentmotionandthemotioneldiswellpartitioned. 9-6 andFigure 9-7 132

PAGE 133

Thesecondframeoftheinputimagesequence. couldnotbeglobalmotionsincetheydonotcontaincornerregionsofthescene.OnlyFigure 9-8 andFigure 9-9 containfourcorners.Therefore,weonlyconsiderthemasgroundmotion.Aftercalculatingthevarianceofmotionvectorsseparately,wecanconcludethattheregionofFigure 9-8 representstheglobalmotionduetoasmallmotionvariance. Becausethesegmentationalgorithmisformotionvectoreld,theperformanceofouralgorithmislimitedbytheaccuracyofopticaloweldcalculatedfrominputimagesequence.Inordertoimprovetheperformance,thetextureinformationcouldbeutilizedtosegmentthemotioneldmoreaccurately.Furthermore,wecanstudymoreinherenttemporalpropertiesoftheglobalmotiontohelpsceneinterpretation. 133

PAGE 134

Thefourthframeoftheinputimagesequence. Figure9-3. Theopticaloweldoftheinputimagesequence. Figure9-4. Therstframeofimagesequence`Coastguard'. 134

PAGE 135

Theopticaloweldofimagesequence`Coastguard'. Figure9-6. The`ship'segmentin`Coastguard'. Figure9-7. The`boat'segmentin`Coastguard'. 135

PAGE 136

The`land'segmentin`Coastguard'. Figure9-9. The`river'segmentin`Coastguard'. 136

PAGE 137

137

PAGE 138

Inthischapter,weproposeanewtrackingalgorithmbasedonbothtemplatetrackingandsilhouettetracking.Thealgorithmattemptstoadequatelytrackmultipleobjectsofarbitraryshapeinanimagesequencethatexperiencescameramotion.Inordertosuccessfullyestimatethemotiontrajectory,werstsegmenttheimagetogenerateabinaryobjectmask,andthentrackthefeaturesinsidethemask.ToovercomethelimitationofthetraditionalKLTtracker,weproposeanoveltrajectoryestimationmethodbasedonaweightingfunctionoftrackedfeaturemotionvectors. Thischapterisorganizedasfollows.Section 10.1 reviewspriorartsinobjecttracking.Insection 10.2 ,weintroduceourtrackingsystem.Section 10.3 presentstheobjectdetectorandSection 10.4 discussesthetrajectoryestimationprocess.TheexperimentalresultsareshowninSection 10.5 andSection 10.6 drawstheconclusion. 10-1 showsthesystemscheme. 138

PAGE 139

Aftertargetdetection,wemodelthemovingobjectwithspecialimagepropertieswithinthemaskregion.Anypropertycouldbeusedtorepresenttheobject,suchasedges,silhouette,colors,andprimitiveinformation.Inouralgorithm,weusethetraditionalKLTfeaturedetectortoselectinitialfeatures.Asweknow,KLTfeaturesareinvarianttoanetransformation,whichisabletoapproximatetheglobalmotioncausedbycameramovement. AlthoughweuseKLTdetectortoselectfeatures,weproposeacompletelydierentfeaturetrackingandupdatingalgorithmcomparedtothetraditionalKLTtracker.ThetraditionalKLTtrackerselectsfeaturesinthewholeimageandtracksallfeaturesinthesameway,i.e.,nofeatureismoreimportantthanothers.Inouralgorithm,eachfeatureistreateddierentlyaccordingtoitstrackingperformance.ThishelpstoachievebettertrackingperformancecomparedtothetraditionalKLTtracker.Inouralgorithm,thetrackedfeaturesareevaluatedaftertrackedateachframe,andthe\bad"featureswillberemovedandnewfeatureswillbereselectedfromtherestofobjectmaskarea.Wealsoproposeaweightingfunctionfortrajectoryestimation,whichconsidersboththequalityofthefeatureandtheconsistenceofthetrackingresult.Inthisway,themotionofthefeaturescouldbetterrepresentsthemotionoftheobject. 139

PAGE 140

Ingeneral,backgroundsubtractionisabletocompensateforlightingchangesandbackgroundclutter,anditiscomputationallyecient.However,moststate-of-the-artbackgroundmodelingmethodsaredesignedforimagesequencesfromxedcameras.Imagesegmentationmethodsareabletopartitiontheimageintoperceptuallysimilarregions,butthecriteriaforagoodpartitionandeciencyaretwoproblemsthatimagesegmentationalgorithmsneedtoaddress.Thedrawbackofsupervisedlearningmethodsisthattheyusuallyrequirealargecollectionofsamplesfromeachobjectclassandthesamplesmustbemanuallylabeled. Theobjectiveofourobjectdetectionalgorithmistondthelocationsofmultiplemovingtargetsintherstfewframesfromanon-stationarycamera.Sinceimagesegmentationonlyutilizesspatialcorrelationofasingleimage,itishardtodetecttheobjectregionduetothedierenttypesofobjecttobetracked.Weproposeanimagesegmentationalgorithmwhichconsidersbothspatialandtemporalinformationfromtherstfewframes.Ouralgorithmobtainsthetemporalinformationbycomputingopticalowfromtherstfewimages.Thisallowsthealgorithmtomodelthemotionuniformityinsidearegionofinterest.Theoutputofthealgorithmincludesseveralbinarymasks.Eachbinarymaskrepresentatargettobetracked.Thesegmentationprocesscanbe 140

PAGE 141

10{1 ). Thenalsegmentationresultdependsontwoterms,S(x,I1)andT(x,I1,I2).S(x,I1)isabinarymaskcomputedfromprimitivecorrelationintherstframeandT(x,I1,I2)representstheuniformityofthemotioneldbyopticalowfromthersttwoframes.isaparametertoadjusttheimportancebetweentemporalandspatialinformation.Whenincreases,thealgorithmgivesmoreandmoreweightonthetemporalinformationandwhendecreases,thealgorithmcaresmoreaboutthespatialinformation. Computingbinarymasksusingspatialcorrelationissimpleandcomputationalecient.Therearetwostepstoobtainthemask:edgedetectionandmorphologicalconnection.WeuseCannyedgedetectorforedgedetection,andweassumethattheobjecttobetrackedisdominantintheimageplane.Therefore,wecanremovethetrivialedgesbysettingthreshold.Wenoticethatthedetectededgesarediscontinuousandcannotbedirectlyusedtorepresentthetarget.Therefore,weneedtoconnecttheedgesandndaclosingboundaryofthetarget.Toservethispurpose,weusemathematicalmorphologywhichisatheoreticalmodelbasedonlatticetheoryandtopology.Morphologicalimageprocessingisgenerallybuiltonshiftinvariantoperators,whichisbasedonMinkowskiaddition.Therearefourbasicoperatorsinmorphologicalimageprocessing:opening,closing,erosion,anddilation.Inouralgorithm,weusedilationandclosingoperators.Weapplydilationtothedetectededgesinordertostrengthentheedgesaswellasconnecttheadjacentedges.Asubsequentclosingoperationistoremovethesmallholesinsidetheobjectmask. Afteredgedetectionandmorphologicaloperations,thedominantobjectismarkedwithabinarymask.However,someareasinthebackgroundwithrichtexturemayalsobemarkedasatarget,duetothefactthatrichtexturecontainsmanyedges.Tosolvethisproblem,weneedtocomputeT(x,I1,I2)tohelpremovingthefalsedetectedobjects.The 141

PAGE 142

124 ],whichcomputesopticalowusingpartialderivativeswithrespecttospatialandtemporalcoordinates.Theimageconstraintequationisgivenas: RemovingthehigherordertermsbyTaylorexpansion,theequationcanbewrittenas: Thecomputedopticaloweldwillbesegmentedusingsimilarmorphologicaloperationsandgeneratethenalobjectmask.ThesegmentationresultisgiveninFigure 10-2 andFigure 10-3 49 ].WithagivenimageI,KLTtrackerevaluatesthevariationofeachpixelinasmallneighborhood. 142

PAGE 143

10{5 ). ThefeaturepointsselectedbyKLTtrackerareinvarianttobothrotationandtranslation.Oncetheobjectmaskisdetected,wecoulduseKLTtrackertoselectandtrackfeaturepointsovermultipleframes. Ourproposedweightingfunctionisgiveninequation( 10{6 ). Therearethreetermsintheweightingfunction.WpisaGaussianweightconsideringthepositionofthepointofinterestintheobjectmask,expressedinequation( 10{7 ). 143

PAGE 144

10{5 .AccordingtothecriteriaofKLTfeaturedetector,thehigherthefeaturequality,themorereliablethefeaturewillbetracked. ThelasttermintheweightingfunctionisWc,whichstandsfortheconsistencyofthefeaturetrackedovermultipleframes.TheoriginalKLTfeaturetrackerisnotabletotrackthroughalongimagesequence,becausesomefeaturepointswillgetlostifthetrackercannotndacorrespondingoneaftertrackingeachframe.Inordertoovercomethelimitation,weproposeafeatureupdatingmechanismtondnewfeaturesonceanyfeaturegetslostandthenumberoffeaturestorepresentanobjectisconstant.Theconsistencyoffeatureisrepresentedbythenumberofframesthefeaturesurvives,whichmeansthatthelongerthefeaturestaysinthefeatureupdatingmechanismprocess,themorestableitis.Weuseexponentialfunctiontocalculatetheweightforfeatureconsistency. Intheaboveequation,nisthenumberofframesthefeaturehassurvived.Theconsistencyisespeciallyimportantwhenthereisobjectocclusion,whichwillbediscussedinthenextsection. 144

PAGE 145

Figure 10-4 andFigure 10-5 showtherstandlastframesinthetest`coastguard.cif'imagesequence.Figure 10-6 showsthetrackingresultoftwoobjectinthesequence.Therearetwoobjectstobetrackedinthissequence:alargershipandasmallboat.Ontherstframe,theboatisinthecenteroftheframeandtheshipcomeintotheframefromtheleft.Thecamerafollowstheboat,sotheboatstaysinthecenterandtheshiptravelsfromlefttoright.Afterseveralframes,thetwoobjectsmeetandtheshipisoccludedbytheboat.Inourexperiment,wearestillabletotrackbothobjectsaslongastheshipisnotfullyoccluded. 145

PAGE 146

Flowchartofourmultipleobjecttrackingsystem. Figure10-2. Segmentationofthe20thframefromthe`Coastguard'imagesequence. 146

PAGE 147

Segmentationresultofthe20thframeaftercorrection. Figure10-4. Therstframeinthe`Coastguard.cif'imagesequence. 147

PAGE 148

Thelastframeinthetestimagesequence. Figure10-6. Thetrackingresultofthetest`Coastguard'imagesequence.Each`+'indicatesthepositionoftheshipinoneframe. 148

PAGE 149

149

PAGE 150

Weproposeanon-lineardeterministicannealing(NDA)approachforgeometricttingin3Dspace.Duetothefactthatthe3Ddataislocalizedtoafewrelativelydenseclusters,wedesignakernelfunctiontomapthedatapointfromgeometricalspacetosurfacemodelspaceandapplydeterministicannealingtopartitionthefeaturespaceintoseparateregions.WefurtherusetheDNAmethodfor3Ddensereconstruction.Weusetheexistingtechniquesforfeaturecorrespondence,projectivereconstructionandself-calibrationtogetthesparsepointsreconstruction.Thenwesegmentthe3Dspaceintoseveralregionsbasedonthegeometricrelationship.Foreachregion,giventheintrinsicparametersfromself-calibration,wecanretrievethedepthinformationforeachpixelusingsurfacetting.Finally,weproposeanewstrategyofimageregistrationbyleveragingthedepthinformationvia3Ddensereconstruction.Thetraditionalimageregistrationalgorithmscannotsolveparallaxproblemduetotheirunderlyingassumptionthatthescenecanberegardedapproximatelyplanar.Ourmethodovercomesthisweaknessandachievesmorerobustregistrationresults.Ouralgorithmisattractivetotremendouspracticalapplications. Wealsoproposeanovelapproachtoestimatemodelparametersofallmotionsbasedonsegmentationofbothintensitymapandopticaloweld.Thenoveltyofourworkisthatweintroducecodinglengthasacriterioningroupmerging.Basedontheexperimentsimulationsandresults,weprovethatusingcodinglengthcouldgreatlyimprovetheperformanceofmotioneldsegmentation.Anothernoveltyisthatweproposeaheuristicapproachtolocateglobalmotionbasedonthemotionsegments. Anothercontributionofthisdissertationisthatweproposeanewtrackingalgorithmbasedonbothtemplatetrackingandsilhouettetracking.Thealgorithmattemptstoadequatelytrackmultipleobjectsofarbitraryshapesinanimagesequence.Inordertoaccuratelyestimatethetrajectory,werstgenerateabinaryobjectmaskandthenonly 150

PAGE 151

Ourfutureworksinclude: 151

PAGE 152

[1] E.Candes,\Compressivesampling,"inProceedingsoftheInternationalCongressofMathematicians,vol.1,2006,pp.1433{1452. [2] E.Candes,J.Romberg,andT.Tao,\Robustuncertaintyprinciples:Exactsignalreconstructionfromhighlyincompletefrequencyinformation,"IEEETransactionsonInformationTheory,vol.52,no.2,pp.489{509,2006. [3] E.CandesandJ.Romberg,\Quantitativerobustuncertaintyprinciplesandoptimallysparsedecompositions,"FoundationsofComputationalMathematics,vol.6,no.2,pp.227{254,2006. [4] D.Donoho,\Compressedsensing,"IEEETransactionsonInformationTheory,vol.52,no.4,pp.1289{1306,2006. [5] E.CandesandM.Wakin,\Peoplehearingwithoutlistening:Anintroductiontocompressivesampling,"IEEESignalProcessingMagazine,vol.25,no.2,pp.21{30,2008. [6] E.CandesandT.Tao,\Decodingbylinearprogramming,"IEEETransactionsonInformationTheory,vol.51,no.12,p.4203,2005. [7] E.vandenBergandM.Friedlander,\Probingtheparetofrontierforbasispursuitsolutions,"SIAMJournalonScienticComputing,vol.31,no.2,pp.890{912,2008. [8] J.TroppandA.Gilbert,\Signalrecoveryfromrandommeasurementsviaorthogonalmatchingpursuit,"IEEETransactionsonInformationTheory,vol.53,no.12,p.4655,2007. [9] D.NeedellandJ.Tropp,\CoSaMP:Iterativesignalrecoveryfromincompleteandinaccuratesamples,"AppliedandComputationalHarmonicAnalysis,vol.26,no.3,pp.301{321,2009. [10] D.DonohoandY.Tsaig,\Fastsolutionofl1-normminimizationproblemswhenthesolutionmaybesparse,"IEEETransactionsonInformationTheory,vol.54,no.11,pp.4789{4812,2008. [11] D.Donoho,Y.Tsaig,I.Drori,andJ.Starck,\Sparsesolutionofunderdeterminedlinearequationsbystagewiseorthogonalmatchingpursuit,"submittedforpublica-tion,2006. [12] M.Duarte,M.Wakin,andR.Baraniuk,\Fastreconstructionofpiecewisesmoothsignalsfromrandomprojections,"inProceedingsofSignalProcessingwithAdapta-tiveSparseStructuredRepresentations,vol.1,Rennes,France,2005,pp.1064{1070. [13] C.LaandM.Do,\Signalreconstructionusingsparsetreerepresentation,"inProceedingsofWaveletsXIatSPIEOpticsandPhotonics,vol.5914,SanDiego,California,2005,pp.273{283. 152

PAGE 153

S.Sarvotham,D.Baron,andR.Baraniuk,\Sudocodes{fastmeasurementandreconstructionofsparsesignals,"inProceedingsofIEEEInternationalSymposiumonInformationTheory,2006,pp.2804{2808. [15] J.Bioucas-DiasandM.Figueiredo,\AnewTwIST:Two-stepiterativeshrinkage/thresholdingalgorithmsforimagerestoration,"IEEETransactionsonImageprocessing,vol.16,no.12,p.2992,2007. [16] R.Chartrand,\Exactreconstructionofsparsesignalsvianonconvexminimization,"IEEESignalProcessingLetters,vol.14,no.10,p.707,2007. [17] E.Candes,M.Wakin,andS.Boyd,\Enhancingsparsitybyreweightedl1minimization,"JournalofFourierAnalysisandApplications,vol.14,no.5,pp.877{905,2008. [18] R.ChartrandandW.Yin,\Iterativelyreweightedalgorithmsforcompressivesensing,"inProceedingsofInternationalConferenceonAcoustics,Speech,andSignalProcessing,2008,pp.3869{3872. [19] T.BlumensathandM.Davies,\Iterativethresholdingforsparseapproximations,"JournalofFourierAnalysisandApplications,vol.14,no.5,pp.629{654,2008. [20] Y.TsaigandD.Donoho,\Extensionsofcompressedsensing,"SignalProcessing,vol.86,no.3,pp.549{571,2006. [21] E.CandesandJ.Romberg,\Practicalsignalrecoveryfromrandomprojections,"IEEETransactionsonSignalProcessing,vol.5674,p.76,2005. [22] L.Gan,\Blockcompressedsensingofnaturalimages,"inProceedingsofInterna-tionalConferenceonDigitalSignalProcessing,2007,pp.403{406. [23] Y.Zhang,S.Mei,Q.Chen,andZ.Chen,\Amultipledescriptionimage/videocodingmethodbycompressedsensingtheory,"inProceedingsofIEEEInternationalSymposiumonCircuitsandSystems,May2008,pp.1830{1833. [24] X.Wu,X.Zhang,andJ.Wang,\Model-guidedadaptiverecoveryofcompressivesensing,"inProceedingsofDataCompressionConference,2009,pp.123{132. [25] P.BouthemyandE.Francois,\Motionsegmentationandqualitativedynamicsceneanalysisfromanimagesequence,"InternationalJournalofComputerVision,vol.10,no.2,pp.157{182,1993. [26] N.PalandS.Pal,\Areviewonimagesegmentationtechniques,"PatternRecogni-tion,vol.26,no.9,pp.1277{1294,1993. [27] D.MurrayandB.Buxton,\Scenesegmentationfromvisualmotionusingglobaloptimization,"IEEETransactionsonPatternAnalysisandMachineIntelligence,pp.220{228,1987. 153

PAGE 154

D.LeGall,\MPEG:Avideocompressionstandardformultimediaapplications,"CommunicationsoftheACM,vol.34,pp.46{58,1991. [29] W.Aref,M.Hammad,A.Catlin,I.Ilyas,T.Ghanem,A.Elmagarmid,andM.Marzouk,\VideoqueryprocessingintheVDBMStestbedforvideodatabaseresearch,"inProceedingsofthe1stACMinternationalWorkshoponMultimediaDatabases,NewYork,NY,USA,2003,pp.25{32. [30] R.PlessandD.Jurgens,\Roadextractionfrommotioncuesinaerialvideo,"inProceedingsofthe12thannualACMInternationalWorkshoponGeographicInformationSystems,NewYork,NY,USA,2004,pp.31{38. [31] B.HornandB.Schunck,\Determiningopticalow,"ArticialIntelligence,vol.17,pp.185{203,1981. [32] D.MumfordandJ.Shah,\Optimalapproximationsbypiecewisesmoothfunctionsandassociatedvariationalproblems,"CommunicationsonPureandAppliedMathematics,vol.42,no.5,pp.577{685,1989. [33] D.CremersandC.Schnorr,\Motioncompetition:Variationalintegrationofmotionsegmentationandshaperegularization,"PatternRecognition,pp.472{480,2002. [34] E.H.AdelsonandJ.Y.A.Wang,\Representingmovingimageswithlayers,"IEEETransactionsonImageProcessing,vol.3,pp.625{638,1993. [35] G.D.Borshukov,G.Bozdagi,Y.Altunbasak,andA.M.Tekalp,\Motionsegmentationbymulti-stageaneclassication,"IEEETransactionsonImageProcessing,vol.6,pp.1591{1594,1997. [36] J.ShiandJ.Malik,\Normalizedcutsandimagesegmentation,"IEEETransactionsonPatternAnalysisandMachineIntelligence,vol.22,pp.888{905,1997. [37] I.K.SethiandR.Jain,\Findingtrajectoriesoffeaturepointsinamonocularimagesequence,"IEEETransactionsonPatternAnalysisandMachineIntelligence,vol.9,no.1,pp.56{73,1987. [38] K.RangarajanandM.Shah,\Establishingmotioncorrespondence,"inProceedingsofIEEEComputerSocietyConferenceonComputerVisionandPatternRecognition,Jun1991,pp.103{108. [39] C.J.Veenman,M.J.T.Reinders,andE.Backer,\Resolvingmotioncorrespondencefordenselymovingpoints,"IEEETransactionsonPatternAnalysisandMachineIntelligence,vol.23,no.1,pp.54{72,2001. [40] T.J.BroidaandR.Chellappa,\Estimationofobjectmotionparametersfromnoisyimages,"IEEETransactionsonPatternAnalysisandMachineIntelligence,vol.8,no.1,pp.90{99,1986. 154

PAGE 155

D.BeymerandK.Konolige,\Real-timetrackingofmultiplepeopleusingcontinuousdetection,"inProceedingsofInternationalConferenceonComputerVisionFrame-rateWorkshop,1999. [42] G.Kitagawa,\Non-gaussianstate-spacemodelingofnonstationarytimeseries,"JournaloftheAmericanStatisticalAssociation,vol.82,no.400,pp.1032{1041,1987. [43] Y.-L.ChangandJ.Aggarwal,\3dstructurereconstructionfromanegomotionsequenceusingstatisticalestimationanddetectiontheory,"inProceedingsoftheIEEEWorkshoponVisualMotion,Oct1991,pp.268{273. [44] C.RasmussenandG.D.Hager,\Probabilisticdataassociationmethodsfortrackingcomplexvisualobjects,"IEEETransactionsonPatternAnalysisandMachineIntelligence,vol.23,no.6,pp.560{576,2001. [45] D.Reid,\Analgorithmfortrackingmultipletargets,"IEEETransactionsonAutomaticControl,vol.24,no.6,pp.843{854,Dec1979. [46] C.Hue,J.-P.LeCadre,andP.Perez,\Sequentialmontecarlomethodsformultipletargettrackinganddatafusion,"IEEETransactionsonSignalProcessing,vol.50,no.2,pp.309{325,Feb2002. [47] H.Schweitzer,J.W.Bell,andF.Wu,\Veryfasttemplatematching,"inProceedingsofthe7thEuropeanConferenceonComputerVision,London,UK,2002,pp.358{372. [48] D.ComaniciuandP.Meer,\Meanshiftanalysisandapplications,"inThePro-ceedingsoftheSeventhIEEEInternationalConferenceonComputerVision,vol.2,1999,pp.1197{1203. [49] J.ShiandC.Tomasi,\Goodfeaturestotrack,"inIEEEConferenceonComputerVisionandPatternRecognition,1994,pp.593{600. [50] I.Haritaoglu,D.Harwood,andL.S.David,\W4:Real-timesurveillanceofpeopleandtheiractivities,"IEEETransactionsonPatternAnalysisandMachineIntelli-gence,vol.22,no.8,pp.809{830,2000. [51] J.Kang,I.Cohen,andG.Medioni,\Continuoustrackingwithinandacrosscamerastreams,"inIEEEComputerSocietyConferenceonComputerVisionandPatternRecognition,vol.1,June2003,pp.267{272. [52] K.SatoandJ.K.Aggarwal,\Temporalspatio-velocitytransformanditsapplicationtotrackingandinteraction,"ComputerVisionandImageUnderstanding,vol.96,no.2,pp.100{128,2004. [53] M.Bertalmo,G.Sapiro,andG.Randall,\Morphingactivecontours,"IEEETransactionsonPatternAnalysisandMachineIntelligence,vol.22,no.7,pp.733{737,2000. 155

PAGE 156

A.-R.Mansouri,\Regiontrackingvialevelsetpdeswithoutmotioncomputation,"IEEETransactionsonPatternAnalysisandMachineIntelligence,vol.24,no.7,pp.947{961,2002. [55] L.Brown,\Asurveyofimageregistrationtechniques,"ACMComputingSurveys(CSUR),vol.24,no.4,pp.325{376,1992. [56] B.ZitovaandJ.Flusser,\Imageregistrationmethods:Asurvey,"ImageandVisionComputing,vol.21,no.11,pp.977{1000,2003. [57] L.KitchenandA.Rosenfeld,\Gray-levelcornerdetection,"PatternRecognitionLetters,vol.1,no.2,pp.95{102,1982. [58] L.DreschlerandH.Nagel,\Volumetricmodeland3Dtrajectoryofamovingcarderivedfrommonoculartvframesequencesofastreetscene,"ComputerGraphicsandImageProcessing,vol.20,no.3,pp.199{228,1982. [59] W.ForstnerandE.Gulch,\Afastoperatorfordetectionandpreciselocationofdistinctpoints,cornersandcentresofcircularfeatures,"inProceedingsofIntercom-missionConferenceonFastProcessingofPhotogrammetricData(ISPRS),1987,pp.281{305. [60] J.Noble,\Findingcorners,"ImageandVisionComputing,vol.6,no.2,pp.121{128,1988. [61] A.GoshtasbyandG.Stockman,\Pointpatternmatchingusingconvexhulledges,"IEEETransactionsonSystems,Man,andCybernetics,vol.15,no.5,pp.631{636,1985. [62] G.Stockman,S.Kopstein,andS.Benett,\Matchingimagestomodelsforregistrationandobjectdetectionviaclustering,"IEEETransactionsonPatternAnalysisandMachineIntelligence,vol.4,pp.229{241,1982. [63] J.DavisandM.Keck,\OSUregistrationalgorithm,"InternalReport,OhioStateUniversity,USA. [64] A.Jerri,\TheShannonsamplingtheorem-Itsvariousextensionsandapplications:Atutorialreview,"inProceedingsoftheIEEE,vol.65,1977,pp.1565{1596. [65] E.CandesandT.Tao,\Near-optimalsignalrecoveryfromrandomprojections:Universalencodingstrategies?"IEEETransactionsonInformationTheory,vol.52,no.12,pp.5406{5425,2006. [66] M.Wakin,J.Laska,M.Duarte,D.Baron,S.Sarvotham,D.Takhar,K.Kelly,andR.Baraniuk,\Anarchitectureforcompressiveimaging,"inProceedingsofIEEEInternationalConferenceonImageProcessing,2006,pp.1273{1276. 156

PAGE 157

S.Sarvotham,D.Baron,andR.Baraniuk,\Measurementsvs.bits:Compressedsensingmeetsinformationtheory,"inProceedingsof44thAllertonConferenceonCommunication,ControlandComputing,2006. [68] M.Wainwright,\Information-theoreticboundsonsparsityrecoveryinthehigh-dimensionalandnoisysetting,"inProceedingsofIEEEInternationalSym-posiumonInformationTheory,2007,pp.961{965. [69] D.Donoho,\High-dimensionalcentrallysymmetricpolytopeswithneighborlinessproportionaltodimension,"DiscreteandComputationalGeometry,vol.35,no.4,pp.617{652,2006. [70] D.DonohoandJ.Tanner,\Neighborlinessofrandomlyprojectedsimplicesinhighdimensions,"ProceedingsoftheNationalAcademyofSciencesoftheUnitedStatesofAmerica,vol.102,no.27,p.9452,2005. [71] ||,\Countingfacesofrandomly-projectedpolytopeswhentheprojectionradicallylowersdimension,"AmericanMathematicalSociety,vol.22,no.1,pp.1{53,2009. [72] R.BaraniukandM.Wakin,\Randomprojectionsofsmoothmanifolds,"Founda-tionsofComputationalMathematics,vol.9,no.1,pp.51{77,2009. [73] J.Haupt,R.Castro,R.Nowak,G.Fudge,andA.Yeh,\Compressivesamplingforsignalclassication,"inProceedingsof40thAsilomarConferenceonSignals,SystemsandComputers,2006,pp.1430{1434. [74] M.Duarte,M.Davenport,M.Wakin,andR.Baraniuk,\Sparsesignaldetectionfromincoherentprojections,"inProceedingsofIEEEInternationalConferenceonAcoustics,SpeechandSignalProcessing,vol.6,2006,p.1. [75] A.Gilbert,M.Strauss,J.Tropp,andR.Vershynin,\Algorithmiclineardimensionreductioninthel1normforsparsevectors,"Arxivpreprintcs/0608079,2006. [76] ||,\Onesketchforall:Fastalgorithmsforcompressedsensing,"inProceedingsofthe39thannualACMsymposiumonTheoryofcomputing,2007,pp.237{246. [77] D.Takhar,J.Laska,M.Wakin,M.Duarte,D.Baron,S.Sarvotham,K.Kelly,andR.Baraniuk,\Anewcompressiveimagingcameraarchitectureusingoptical-domaincompression,"inProceedingsofSPIE,theInternationalSocietyforOpticalEngi-neering,vol.6065,2006,pp.43{52. [78] M.Lustig,D.Donoho,andJ.Pauly,\SparseMRI:TheapplicationofcompressedsensingforrapidMRimaging,"MagneticResonanceinMedicine,vol.58,no.6,pp.1182{1195,2007. [79] M.Rabbat,J.Haupt,A.Singh,andR.Nowak,\Decentralizedcompressionandpredistributionviarandomizedgossiping,"inProceedingsofthe5thInternationalConferenceonInformationProcessinginSensorNetworks,2006,pp.51{59. 157

PAGE 158

W.Wang,M.Garofalakis,andK.Ramchandran,\Distributedsparserandomprojectionsforrenableapproximation,"inProceedingsofthe6thInternationalConferenceonInformationProcessinginSensorNetworks,2007,pp.331{339. [81] S.Kirolos,J.Laska,M.Wakin,M.Duarte,D.Baron,T.Ragheb,Y.Massoud,andR.Baraniuk,\Analog-to-informationconversionviarandomdemodulation,"inProceedingsofIEEEDallas/CASWorkshoponDesign,Applications,IntegrationandSoftware,2006,pp.71{74. [82] J.Laska,S.Kirolos,Y.Massoud,R.Baraniuk,A.Gilbert,M.Iwen,andM.Strauss,\Randomsamplingforanalog-to-informationconversionofwidebandsignals,"inProceedingsofIEEEDallas/CASWorkshoponDesign,Applications,IntegrationandSoftware,2006,pp.119{122. [83] J.Laska,S.Kirolos,M.Duarte,T.Ragheb,R.Baraniuk,andY.Massoud,\Theoryandimplementationofananalog-to-informationconverterusingrandomdemodulation,"inProceedingsofIEEEInternationalSymposiumonCircuitsandSystems,2007,pp.1959{1962. [84] J.Ragheb,S.Kirolos,J.Laska,A.Gilbert,M.Strauss,R.Baraniuk,andY.Massoud,\Implementationmodelsforanalog-to-informationconversionviarandomsampling,"inProceedingsofthe50thIEEEInternationalMidwestSympo-siumonCircuitsandSystems,2007,pp.119{122. [85] R.Baraniuk,M.Davenport,R.DeVore,andM.Wakin,\TheJohnson-Lindenstrausslemmameetscompressedsensing,"ConstructiveApproximation,2007. [86] M.Elad,\Optimizedprojectionsforcompressedsensing,"IEEETransactionsonSignalProcessing,vol.55,no.12,p.5695,2007. [87] A.Cohen,W.Dahmen,andR.DeVore,\Compressedsensingandbestk-termapproximation,"AmericanMathematicalSociety,vol.22,no.1,pp.211{231,2009. [88] H.Rauhut,K.Schnass,andP.Vandergheynst,\Compressedsensingandredundantdictionaries,"IEEETransactionsOnInformationTheory,vol.54,no.5,pp.2210{2219,2008. [89] M.Figueiredo,R.Nowak,andS.Wright,\Gradientprojectionforsparsereconstruction:Applicationtocompressedsensingandotherinverseproblems,"IEEEJournalonSelectedTopicsinSignalProcessing,vol.1,no.4,pp.586{597,2007. [90] J.Starck,M.Elad,andD.Donoho,\Imagedecompositionviathecombinationofsparserepresentationsandavariationalapproach,"IEEETransactionsOnImageProcessing,vol.14,no.10,pp.1570{1582,2005. 158

PAGE 159

M.EladandA.Bruckstein,\Ageneralizeduncertaintyprincipleandsparserepresentationinpairsofbases,"IEEETransactionsonInformationTheory,vol.48,no.9,pp.2558{2567,2002. [92] X.Zhang,X.Wu,andF.Wu,\Imagecodingonquincunxlatticewithadaptiveliftingandinterpolation,"inProceedingsofthe2007DataCompressionConference,2007,pp.193{202. [93] L.Bregman,\Therelaxationmethodofndingthecommonpointofconvexsetsanditsapplicationtothesolutionofproblemsinconvexprogramming,"USSRComputationalMathematicsandMathematicalPhysics,vol.7,no.3,pp.200{217,1967. [94] W.Bajwa,J.Haupt,A.Sayeed,andR.Nowak,\Compressivewirelesssensing,"inProceedingsofthe5thInternationalConferenceonInformationProcessinginSensorNetworks,2006,p.142. [95] C.Qiu,W.Lu,andN.Vaswani,\Real-timedynamicmrimagereconstructionusingkalmanlteredcompressedsensing,"inIEEEInternationalConferenceonAcoustics,Speech,andSignalProcessing,LosAlamitos,CA,USA,2009,pp.393{396. [96] Z.FanandR.deQueiroz,\Identicationofbitmapcompressionhistory:JPEGdetectionandquantizerestimation,"IEEETransactionsonImageProcessing,vol.12,no.2,pp.230{235,2003. [97] V.Goyal,A.Fletcher,andS.Rangan,\Compressivesamplingandlossycompression,"IEEESignalProcessingMagazine,vol.25,no.2,pp.48{56,2008. [98] L.Rudin,S.Osher,andE.Fatemi,\Nonlineartotalvariationbasednoiseremovalalgorithms,"PhysicaD:NonlinearPhenomena,vol.60,no.1-4,pp.259{268,1992. [99] A.Likas,N.Vlassis,etal.,\Theglobalk-meansclusteringalgorithm,"PatternRecognition,vol.36,no.2,pp.451{461,2003. [100] K.Rose,\Deterministicannealingforclustering,compression,classication,regression,andrelatedoptimizationproblems,"ProceedingsoftheIEEE,vol.86,no.11,pp.2210{2239,1998. [101] B.Kulis,S.Basu,I.Dhillon,andR.Mooney,\Semi-supervisedgraphclustering:akernelapproach,"MachineLearning,vol.74,no.1,pp.1{22,2009. [102] S.Kirkpatrick,\Optimizationbysimulatedannealing:Quantitativestudies,"JournalofStatisticalPhysics,vol.34,no.5,pp.975{986,1984. [103] A.Rao,D.Miller,K.Rose,andA.Gersho,\Adeterministicannealingapproachforparsimoniousdesignofpiecewiseregressionmodels,"IEEETransactionsonPatternAnalysisandMachineIntelligence,vol.21,no.2,pp.159{173,1999. 159

PAGE 160

F.CamastraandA.Verri,\Anovelkernelmethodforclustering,"IEEETransac-tionsonPatternAnalysisandMachineIntelligence,pp.801{804,2005. [105] Z.Zhang,\Aexiblenewtechniqueforcameracalibration,"IEEETransactionsonPatternAnalysisandMachineIntelligence,vol.22,no.11,pp.1330{1334,2000. [106] C.Strecha,T.Tuytelaars,andL.VanGool,\Densematchingofmultiplewide-baselineviews,"inInternationalConferenceonComputerVision,vol.2,2003,pp.1194{1201. [107] M.LhuillierandL.Quan,\Aquasi-denseapproachtosurfacereconstructionfromuncalibratedimages,"IEEETransactionsonPatternAnalysisandMachineIntelligence,pp.418{433,2005. [108] H.Jin,S.Soatto,andA.Yezzi,\Multi-viewstereoreconstructionofdenseshapeandcomplexappearance,"InternationalJournalofComputerVision,vol.63,no.3,p.189,2005. [109] R.HartleyandA.Zisserman,Multipleviewgeometryincomputervision.CambridgeUniversityPress,2003. [110] E.TruccoandA.Verri,Introductorytechniquesfor3-Dcomputervision.PrenticeHall,1998. [111] Y.Ma,S.Soatto,andJ.Kosecka,Aninvitationto3-Dvision:fromimagestogeometricmodels.SpringerVerlag,2004. [112] H.Li,B.Adams,L.Guibas,andM.Pauly,\Robustsingle-viewgeometryandmotionreconstruction,"inACMSIGGRAPHAsia,2009,pp.1{10. [113] P.Beardsley,P.Torr,andA.Zisserman,\3Dmodelacquisitionfromextendedimagesequences,"inProceedingsofthe4thEuropeanConferenceonComputerVision,1996,pp.683{695. [114] A.FitzgibbonandA.Zisserman,\Automaticcamerarecoveryforclosedoropenimagesequences,"inProceedingsofthe5thEuropeanConferenceonComputerVision,1998,pp.311{326. [115] M.Pollefeys,R.Koch,andV.Gool,\Self-calibrationandmetricreconstructioninspiteofvaryingandunknowninternalcameraparameters,"InternationalJournalofComputerVision,pp.7{25,1998. [116] F.DevernayandO.Faugeras,\Automaticcalibrationandremovalofdistortionfromscenesofstructuredenvironments,"InvestigativeandTrialImageProcessing,vol.2567,pp.62{72,1995. [117] J.YagnikandK.Ramakrishnan,\Amodelbasedfactorizationapproachfordense3Drecoveryfrommonocularvideo,"inSeventhIEEEInternationalSymposiumonMultimedia,2005,p.4. 160

PAGE 161

V.Popescu,E.Sacks,andG.Bahmutov,\Interactivepoint-basedmodelingfromdensecolorandsparsedepth,"inEurographicsSymposiumonPoint-BasedGraphics,2004. [119] H.Chang,J.Moura,Y.Wu,K.Sato,andC.Ho,\Reconstructionof3DdensecardiacmotionfromtaggedMRsequences,"inIEEEInternationalSymposiumonBiomedicalImaging:NanotoMacro,2004,pp.880{883. [120] O.FaugerasandR.Keriven,\Completedensestereovisionusinglevelsetmethods,"inProceedingsofthe5thEuropeanConferenceonComputerVision,1998,p.393. [121] R.Koch,M.Pollefeys,andL.Gool,\Multiviewpointstereofromuncalibratedvideosequences,"inProceedingsofthe5thEuropeanConferenceonComputerVision,1998,p.71. [122] M.Pollefeys,R.Koch,M.Vergauwen,andL.VanGool,\Automatedreconstructionof3Dscenesfromsequencesofimages,"ISPRSJournalOfPhotogrammetryAndRemoteSensing,vol.55,no.4,pp.251{267,2000. [123] M.LhuillierandL.Quan,\Surfacereconstructionbyintegrating3Dand2Ddataofmultipleviews,"inProceedingsoftheNinthIEEEInternationalConferenceonComputerVision,2003,pp.1313{1320. [124] B.LucasandT.Kanade,\Aniterativeimageregistrationtechniquewithanapplicationtostereovision,"inInternationalJointConferenceonArticialIntelli-gence,vol.3,1981,p.3. [125] J.Barron,D.Fleet,andS.Beauchemin,\Performanceofopticalowtechniques,"InternationalJournalofComputerVision,vol.12,no.1,pp.43{77,1994. [126] M.FischlerandR.Bolles,\Randomsampleconsensus:Aparadigmformodelttingwithapplicationstoimageanalysisandautomatedcartography,"CommunicationsoftheACM,vol.24,no.6,pp.381{395,1981. [127] O.Mendoza,G.Arnold,andP.Stiller,\Furtherexplorationoftheobject-imagemetricwithimageregistrationinmind,"inProceedingsoftheSPIE,SymposiumonMultisensor,MultisourceInformationFusion:Architectures,Algorithms,andApplications,vol.6974,April2008,pp.5{12. [128] S.P.Lloyd,\LeastsquaresquantizationinPCM,"IEEETransactionsonInforma-tionTheory,vol.28,pp.129{137,1982. [129] G.Zhang,J.Jia,T.Wong,andH.Bao,\Recoveringconsistentvideodepthmapsviabundleoptimization,"inIEEEConferenceonComputerVisionandPatternRecognition,2008,pp.1{8. [130] G.Zhang,J.Jia,T.-T.Wong,andH.Bao,\Consistentdepthmapsrecoveryfromavideosequence,"IEEETransactionsonPatternAnalysisandMachineIntelligence,vol.31,no.6,pp.974{988,June2009. 161

PAGE 162

W.Hong,J.Wright,K.Huang,andY.Ma,\Multiscalehybridlinearmodelsforlossyimagerepresentation,"IEEETransactionsonImageProcessing,vol.15,no.12,p.3655,2006. 162

PAGE 163

BingHanwasborninAnyang,Henan,China.HegothisB.S.degreeinelectricalengineeringatPekingUniversity,Beijing,China,in2005.HereceivedthePh.D.degreeinElectricalandComputerEngineeringfromUniversityofFlorida,Gainesville,FLinAugust2010.Hisresearchinterestsincludeimageandvideocompression,compressivesensing,computervisionandvideoanalysis. 163