Citation
Advanced Techniques for Digital Image Compression and Analysis

Material Information

Title:
Advanced Techniques for Digital Image Compression and Analysis
Creator:
Han, Bing
Place of Publication:
[Gainesville, Fla.]
Publisher:
University of Florida
Publication Date:
Language:
english
Physical Description:
1 online resource (163 p.)

Thesis/Dissertation Information

Degree:
Doctorate ( Ph.D.)
Degree Grantor:
University of Florida
Degree Disciplines:
Electrical and Computer Engineering
Committee Chair:
Wu, Dapeng
Committee Members:
Sun, Yijun
Li, Tao
Banks, Scott A.
Graduation Date:
8/7/2010

Subjects

Subjects / Keywords:
Algorithms ( jstor )
Annealing ( jstor )
Computer vision ( jstor )
Experimental results ( jstor )
Geometric planes ( jstor )
Image compression ( jstor )
Image processing ( jstor )
Images ( jstor )
Pixels ( jstor )
Signals ( jstor )
3d, compressive, image, motion, object, video
Electrical and Computer Engineering -- Dissertations, Academic -- UF
Genre:
Electronic Thesis or Dissertation
bibliography ( marcgt )
theses ( marcgt )
government publication (state, provincial, terriorial, dependent) ( marcgt )
Electrical and Computer Engineering thesis, Ph.D.

Notes

Abstract:
Digital images and videos are widely used in many areas, such as digital TV broadcasting, space imagery and aerial photography, magnetic resonance imaging, traffic monitoring and video surveillance. In this dissertation, we study two important areas, namely, image compression and video analysis. In the first part of this dissertation, we study compressive sensing (CS) and its application to image/video representation and compression. CS theory states that it is possible to recover certain signals and images from far fewer samples or measurements than those required by traditional approaches. We use a CS technique to represent visual data and propose a new image representation scheme in visual sensor networks. Different from the previous works on compressive imaging, which treat the input image as a whole signal, we decompose the visual data into two components before sampling: a dense component and a sparse component. We represent the dense component by the traditional approach and represent the sparse component by compressive sensing. The advantage of our scheme is that we use the correlation of the two components to recover the signal, which helps to reduce the number of measurements and computation time required for reconstruction with the same accuracy. We propose and implement a projection onto convex sets based optimization algorithm to recover the signal. We also propose a new image/video compression system, which combines CS with traditional block based image/video compression schemes, such as JPEG and H.264. In the second part of this dissertation, we study video analysis. There are a lot of image processing areas that employ video analysis. In this dissertation, we attack three problems in video analysis, i.e., image registration, motion analysis, and object tracking. Firstly, we propose a new strategy of image registration by leveraging the depth information via 3D reconstruction. One novel idea is to recover the depth in the image region with high-rise objects to build accurate transform function. The traditional image registration algorithms suffer from the parallax problem due to their underlying assumption that the scene can be regarded approximately planar. Our method overcomes this weakness and achieves more accurate registration performance. Secondly, we propose a new method for motion segmentation based scene interpretation. The segmentation of optical motion field is based on the minimal coding length criterion. The experimental results show that our proposed scheme could greatly improve the performance of motion field segmentation. Finally, to overcome the limitations of the traditional KLT feature tracker, we propose a novel object tracking algorithm. For each object to be tracked, we use a set of KLT features to represent and a weighting function to balance the contribution of different features, according to their position, quality and consistency. The algorithm could adequately track multiple objects of arbitrary shapes in an image sequence with partial occlusion. ( en )
General Note:
In the series University of Florida Digital Collections.
General Note:
Includes vita.
Bibliography:
Includes bibliographical references.
Source of Description:
Description based on online resource; title from PDF title page.
Source of Description:
This bibliographic record is available under the Creative Commons CC0 public domain dedication. The University of Florida Libraries, as creator of this bibliographic record, has waived all rights to it worldwide under copyright law, including all related and neighboring rights, to the extent allowed by law.
Thesis:
Thesis (Ph.D.)--University of Florida, 2010.
Local:
Adviser: Wu, Dapeng.
Statement of Responsibility:
by Bing Han.

Record Information

Source Institution:
UFRGP
Rights Management:
Applicable rights reserved.
Embargo Date:
10/8/2010
Resource Identifier:
004979779 ( ALEPH )
705932754 ( OCLC )
Classification:
LD1780 2010 ( lcc )

Downloads

This item has the following downloads:


Full Text

PAGE 2

2

PAGE 3

toShanshan,mybelovedwifeforherpatience,understandingandsupport; toDadandMomforinstillingtheimportanceofcuriosityandhardwork; toJing,myeldersisterforherencouragement. 3

PAGE 4

AtUniversityofFlorida,especiallyinmyresearchgroup,Ihavehadtheprivilegetoworkwithmanytalentedindividuals,whohavemadecontributionstomyresearchexperience.Firstofall,IwanttothankmyadvisorProfessorDapengWuforallthehopehehasputonme,beforeIthoughtIcoulddoanyresearchatall.ProfessorDapengWuisanexcellentrolemodelformewhoreallyknowshowtobalancescienticresearch,teaching,andfamily.Withouthisinspirationalguidance,hisencouragements,hisenthusiasm,andhisunselshhelp,IcouldnevernishmydoctoralworkinUniversityofFlorida.Hehasalwaysencouragedmetoliveintensively,andtaughtmehowtoappreciatethegoodscienticworkandlife.IalsowouldliketothankProfessorScottBanks,ProfessorTaoLiandProfessorYijunSunforservingonmydissertationcommittee.Theyhaveprovidedmanyvaluablesuggestionsonmyresearchanddissertation.IamthankfultomyfellowstudentsandthevisitingscholarsinMultimediaCommunicationsandNetworkingLab.Dr.JieyanFanandDr.XiaochenLihavehelpedmealotinmyearlydaysatUF.ThankstoDr.JunXu,WenxingYe,ZhifengChenandTaoranLuforvaluablediscussionsandhelpinmyresearch.IwouldliketothankDr.XihuaDong,YiranLi,LeiYang,ZongruiDing,QianChen,YakunHu,JiangpingWang,YuejiaHe,HuanghuangLiandZhengYuan.Finally,IwanttoexpressmyappreciationtomywifeShanshanRen,myparents,YuqingHanandJinrongLiu,andmysisterJingHan,fortheirlove,understanding,patience,endlesssupport,andneverfailingfaithinme. 4

PAGE 5

page ACKNOWLEDGMENTS ................................. 4 LISTOFTABLES ..................................... 9 LISTOFFIGURES .................................... 10 ABSTRACT ........................................ 13 CHAPTER 1INTRODUCTION .................................. 15 1.1Motivation .................................... 17 1.1.1CompressiveSensinginImageCompression .............. 17 1.1.2VideoAnalysis .............................. 18 1.2OutlineoftheDissertation ........................... 19 2COMPRESSIVESENSINGFORIMAGEPROCESSINGAPPLICATIONS:ANOVERVIEW ................................... 22 2.1Introduction ................................... 22 2.2CompressiveSensingTheory .......................... 23 2.3RecoveryAlgorithms .............................. 26 2.3.1Algorithmwithl1Constraint ...................... 26 2.3.2AlgorithmswithlpConstraint ..................... 28 2.4CSMeasurementEnsembles .......................... 29 2.5CSforImageProcessing ............................ 29 3VideoAnalysis:ANOVERVIEW .......................... 32 3.1MotionAnalysis ................................. 32 3.2ObjectTracking ................................. 34 3.2.1PointTrackingAlgorithm ........................ 34 3.2.2KernelTrackingAlgorithm ....................... 35 3.2.3SilhouetteTrackingAlgorithm ..................... 36 3.3ImageRegistration ............................... 37 4COMPRESSIVESENSINGBASEDIMAGEREPRESENTATION ....... 40 4.1Introduction ................................... 40 4.2OverviewofCompressiveSensing ....................... 42 4.3ProposedImageRepresentationScheme ................... 43 4.3.1ReconstructionErrorBounds ...................... 44 4.3.2ImageDecomposition .......................... 45 4.3.3CorrelationbetweenSparseandDenseComponents ......... 47 4.4PracticalSignalReconstruction ........................ 48 5

PAGE 6

..................... 48 4.4.2ImageReconstructionAlgorithm .................... 50 4.5ExperimentalResults .............................. 51 4.6Conclusion .................................... 53 5COMPRESSIVESENSINGINBLOCKBASEDIMAGE/VIDEOCODING .. 60 5.1Introduction ................................... 60 5.2CompressiveSensingTheory .......................... 61 5.3NewImage/VideoCompressionScheme .................... 63 5.4CSRecoveryinDecompression ........................ 63 5.4.1TVMinimizationOptimization .................... 64 5.4.2BoundedResidueConstraint ...................... 65 5.5ExperimentalResult .............................. 66 5.6Conclusion .................................... 67 63DGEOMETRICSEGMENTATIONANDFITTING .............. 71 6.1Introduction ................................... 71 6.2The3DGeometricFittingProblem ...................... 73 6.2.1DeterministicAnnealing ........................ 75 6.2.2Non-linearPartitioning ......................... 76 6.3Non-linearDeterministicAnnealing ...................... 77 6.4ExperimentalResults .............................. 80 6.4.1NDAonSyntheticDatawithoutNoise ................ 80 6.4.2NDAonSyntheticDatawithNoise .................. 81 6.4.3NDAonRealWorldData ....................... 81 6.5Conclusion .................................... 82 73DDENSERECONSTRUCTIONFROM2DVIDEOSEQUENCE ....... 89 7.1Introduction ................................... 89 7.2BackgroundandProblemFormation ..................... 90 7.2.13DReconstruction ........................... 91 7.2.2GeometricFitting ............................ 93 7.33DVideoReconstruction ............................ 94 7.3.1Overviewof3DReconstructionSystem ................ 94 7.3.2FeatureSelection ............................ 94 7.3.3FeatureCorrespondence ........................ 95 7.3.4EstimationofCameraMotionParameters ............... 96 7.3.5DepthEstimation ............................ 97 7.3.6GeometricSegmentation ........................ 97 7.3.7DepthRecovery ............................. 98 7.4GeometricSegmentationbasedDenseReconstruction ............ 99 7.4.1Non-linearDeterministicAnnealing .................. 100 7.5ExperimentalResults .............................. 102 7.5.13DVideoDenseReconstruction .................... 102 6

PAGE 7

.................................... 103 8IMAGEREGISTRATION .............................. 108 8.1Introduction ................................... 108 8.2ANewArchitecturefor2DImageRegistrationwithDepthInformation .. 110 8.33DReconstructionfrom2DVideoSequence ................. 111 8.3.1FeatureSelection ............................ 111 8.3.2FeatureCorrespondence ........................ 112 8.3.3EstimationofCameraMotionParameters ............... 113 8.3.4DepthEstimation ............................ 114 8.4ImageRegistrationwithDepthInformation ................. 114 8.4.1GeometricalSegmentation ....................... 114 8.4.2DepthEstimation ............................ 118 8.5ExperimentalResults .............................. 119 8.6Conclusion .................................... 120 9MOTIONSEGMENTATION ............................ 126 9.1Introduction ................................... 126 9.2OpticalFlowField ............................... 127 9.3MotionFieldSegmentation ........................... 129 9.3.1MinimalDescriptionLengthCriterion ................. 129 9.3.2CodingLengthbasedOpticalFieldSegmentation .......... 130 9.3.3MinimizingtheCodingLength ..................... 131 9.4GlobalMotionEstimation ........................... 131 9.5ExperimentResults ............................... 132 9.5.1MotionFieldSegmentation ....................... 132 9.5.2GlobalMotionEstimation ....................... 132 9.6Conclusion .................................... 133 10OBJECTTRACKING ................................ 138 10.1Introduction ................................... 138 10.2SystemOverview ................................ 138 10.3ObjectDetection ................................ 140 10.4ObjectTracking ................................. 142 10.4.1KLTFeatureSelectionandTracking .................. 142 10.4.2TrajectoryEstimationandFeatureUpdate .............. 143 10.4.3OcclusionHandling ........................... 144 10.5ExperimentalResults .............................. 145 10.6Conclusion .................................... 145 11CONCLUSION .................................... 149 11.1SummaryofthisDissertation ......................... 149 11.1.1CompressiveSensinginImageCompression .............. 149 7

PAGE 8

.............................. 150 11.2FutureWork ................................... 151 REFERENCES ....................................... 152 BIOGRAPHICALSKETCH ................................ 163 8

PAGE 9

Table page 4-1Comparisonofreconstructionresultswiththesamenumberofmeasurements .. 54 5-1Experimentalresultsofthetestblocksfrom`Cameraman' ............ 67 5-2ExperimentalresultsofBCS ............................. 67 6-1Theaveragesquaredapproximationerror ...................... 82 6-2Thecorrectidenticationrate ............................ 83 6-3Theaveragesquaredapproximationerror ...................... 84 9

PAGE 10

Figure page 1-1Sourceencoderanddecodermodel ......................... 16 4-1Onedimensionalsignalrecoverybycompressivesensing .............. 54 4-2Comparisonoferrorbounds ............................. 55 4-3Wavelettransformof`Lena' ............................. 55 4-4Powerlawmodelofsparsesignalcoecients .................... 56 4-5Imageinterpolationscheme ............................. 56 4-6Imagerepresentationscheme ............................. 57 4-7Predictionofthesparsecomponentof`Lena' .................... 57 4-8Comparisonofpredictionandoriginalsignal .................... 58 4-9Comparisonoferrorreductionwithsamenumberofiterations .......... 58 4-10Recoveredimage`Lena'from20000measurements ................ 59 4-11Recoveredimage`Boats'from20000measurements ................ 59 5-1Theowchartofthenewcompressionscheme ................... 68 5-2Thetestimage`Cameraman' ............................ 69 5-3Thetestimageblocksin`Cameraman' ....................... 69 5-4Thetestimageblocks ................................ 69 5-5Testresultof`Boats' ................................. 70 5-6Testresultof`Cameraman' ............................. 70 6-1Thesyntheticdataset ................................ 84 6-2TherstgrouppartitionedbyK-means ....................... 85 6-3Theinputdatapointsontherstframe ...................... 86 6-4ThegeometricalsegmentationresultbytheNDAalgorithm ........... 87 6-5ThegeometricalsegmentationresultbythePIalgorithm ............. 88 7-1Thepipelinefor3Dvideoreconstructionsystem .................. 104 7-2Theschemefor3Dvideoreconstructionsystem .................. 105 10

PAGE 11

..................... 105 7-4Thefeaturepointsontherstframe ........................ 106 7-5Theestimatedsparsedepthmapandcamerapose ................. 107 7-6Theestimateddense3Dconguration ....................... 107 8-1Thepipelinefor2Dimageregistrationsystem ................... 121 8-2Thenewimageregistrationsystemscheme ..................... 122 8-3Ouralgorithmtestresult ............................... 122 8-4ThetestresultofDavisandKeck'salgorithm ................... 123 8-5Thedierenceimageofouralgorithm. ....................... 123 8-6ThedierenceimageofDavisandKeck'salgorithm ................ 124 8-7The37thframeinthe`oldhousing'videosequence ................. 124 8-8Ouralgorithmtestresult ............................... 125 8-9OuralgorithmtestresultcomparingtoDavisandKeck'salgorithm ....... 125 9-1Thesecondframeofimagesequence ........................ 133 9-2Thefourthframeofimagesequence ......................... 134 9-3Theopticaloweldoftheinputimagesequence ................. 134 9-4Therstframeofimagesequence`Coastguard' .................. 134 9-5Theopticaloweldofimagesequence`Coastguard' ............... 135 9-6The`ship'segmentin`Coastguard' ......................... 135 9-7The`boat'segmentin`Coastguard' ......................... 135 9-8The`land'segmentin`Coastguard' ......................... 136 9-9The`river'segmentin`Coastguard' ......................... 136 10-1Flowchartofourmultipleobjecttrackingsystem ................. 146 10-2Segmentationofthe20thframefromthe`Coastguard'imagesequence ..... 146 10-3Segmentationresultofthe20thframeaftercorrection ............... 147 10-4Therstframeinthe`Coastguard'imagesequence ................ 147 10-5Thelastframeofthe`Coastguard'imagesequence ................ 148 11

PAGE 12

.............. 148 12

PAGE 13

Digitalimagesandvideosarewidelyusedinmanyareas,suchasdigitalTVbroadcasting,spaceimageryandaerialphotography,magneticresonanceimaging,tracmonitoringandvideosurveillance.Inthisdissertation,westudytwoimportantareas,namely,imagecompressionandvideoanalysis. Intherstpartofthisdissertation,westudycompressivesensing(CS)anditsapplicationtoimage/videorepresentationandcompression.CStheorystatesthatitispossibletorecovercertainsignalsandimagesfromfarfewersamplesormeasurementsthanthoserequiredbytraditionalapproaches.WeuseaCStechniquetorepresentvisualdataandproposeanewimagerepresentationschemeinvisualsensornetworks.Dierentfromthepreviousworksoncompressiveimaging,whichtreattheinputimageasawholesignal,wedecomposethevisualdataintotwocomponentsbeforesampling:adensecomponentandasparsecomponent.Werepresentthedensecomponentbythetraditionalapproachandrepresentthesparsecomponentbycompressivesensing.Theadvantageofourschemeisthatweusethecorrelationofthetwocomponentstorecoverthesignal,whichhelpstoreducethenumberofmeasurementsandcomputationtimerequiredforreconstructionwiththesameaccuracy.Weproposeandimplementaprojectionontoconvexsetsbasedoptimizationalgorithmtorecoverthesignal.Wealsoproposeanewimage/videocompressionsystem,whichcombinesCSwithtraditionalblockbasedimage/videocompressionschemes,suchasJPEGandH.264. 13

PAGE 14

14

PAGE 15

Withthefastdevelopmentofdigitalcamerasanddisplaydevices,digitalimagesandvideosarebecomingmoreandmorepopularinoureverydaylives.Adigitalimageiscomposedofanitenumberofpixels.Digitalimagesandvideosarewidelyusedinmanyareas,suchasdigitalTVbroadcasting,spaceimageryandaerialphotography,magneticresonanceimaging,tracmonitoringandvideosurveillance.Sincetheopticaldevicescancoveralmosttheentireelectromagneticspectrum,thecapturedimagesneedtobeprocessedforhumanvision.Generally,digitalimageprocessingreferstoprocessingdigitalimagesviadigitalcomputers.Sinceweareusingcomputerstoprocessandanalyzethedigitalimages,thereisnoclearboundarybetweendigitalimageprocessing,computervisionandimageanalysis.Computervisionstudieshowtousecomputerstoemulatehumanvisionandimageanalysis,orsceneanalysisistousecomputerstounderstandtheimages.Althoughtherearenoclear-cutboundariesbetweentheseresearchelds,theprocessesareclassiedintothreelevels,lowlevel,midlevelandhighlevel.Thelowlevelprocessesincludesnoisereduction,imageenhancement,andimagecompression.Theinputsandoutputsofalowlevelprocessarebothimages.Themidlevelprocessesinvolvesfeatureextraction,edgedetection,andimagesegmentation.Theinputsofamidlevelprocessarestillimages,buttheoutputsaretexturefeaturesextractedfromtheinputimages.Thehighlevelprocessesinvolvesobjectrecognition,objecttrackingandmotionanalysis. Inthisdissertation,wefocusontwoimportantareas,i.e.,imagecompressionandvideoanalysis. Everyday,anenormousvolumeofdigitalimagesandvideosisgenerated,stored,processedandtransmitted.Imagecompressionisnecessaryforhandlingthelargespatialresolutionsoftoday'simagingsensors.Imagecompressioniswidelyusedinavarietyofapplications,suchasvideoteleconferencing,remotesensing,remotedesktop,andvideo 15

PAGE 16

Sourceencoderanddecodermodel. streaming.Thebasicideaofimagecompressionistoremovetheredundantdatafromtherawdigitalimages.Therearetwotypeofimageredundancies,namely,interpixelredundancyandpsychovisualredundancy.Interpixelredundancyrelatestotheinterpixelcorrelationswithinanimage,whichisalsoknownasspatialredundancyandgeometricredundancy.Psychovisualredundancyisassociatedwiththelimitationofhumanvisualprocessing.Becauseahumanbeingisnotabletoperceiveallvisualinformationinanimage,theinformation,whichisnotessentialfornormalhumanvisualprocessing,iseliminable.AconventionalimagecompressionsystemisshowninFigure 1-1 Generally,therearethreestagesintheencodingprocess.Intherststage,atransformfunctionblocktransformstheinputdataf(x,y)intoatransformdomaintoreduceinterpixelredundancies.Thetransformoperationisusuallyreversibleandtheimageistransformedintoanarrayofcoecients.Inthesecondstage,anquantizerblockperformsquantizationontheinputcoecients.Thequantizationisusefultoreducethepsychovisualredundanciesoftheinputimage.Usuallythisoperationisirreversible.Thelaststageissourceencodingprocessanditisreversible.Accordingly,thedecodingprocesscontainsthreestages,symboldecoder,inversequantizer,andinversetransform. Inthepasttwodecades,quiteafewcompressionmethodsareproposed,suchasJPEGandJPEG2000forimagecompression,andH.263andH.264forvideocompression.Asisknowntoall,naturalimagesarepiecewisesmoothandhighlycompressibleontransformdomain.Therefore,allofthepreviouscompressionmethodsarebasedon 16

PAGE 17

Withthelargevolumeofdigitalimagesandvideos,itishardforhumanstoanalyzethedatamanually.Takevideosurveillanceasanexample.Videosurveillancewasinitiallydevelopedforsecurityreasons.Supposeweneedtondasuspectfromavideosequencewiththelengthofoneday,itmaytakeahumanseveralhourstolocatetheframescontainingthesuspect.Withcomputervisiontechnology,wecoulduseacomputertohelplocatingthesuspectwiththesamevideosequenceinonlyafewminutes.Therearealotofapplicationareasthatemploycomputervision,suchaspatternrecognition,motionanalysis,objecttrackingandimageregistration.Inthisdissertation,wewillattackthreeproblemsinthisarea,motionanalysis,objecttracking,andimageregistration. 1.1.1CompressiveSensinginImageCompression 17

PAGE 18

Withthemotioninformationofthevideosequence,objecttrackingistoestimateandanalyzethetrajectoryofanobjectintheimageplaneasitmovesthroughavideosequence.Objecttrackinghasbeenwidelyusedinvideosurveillance,videoindexing,tracmonitoringandmotionbasedrecognition.Thechallengeinobjecttrackingistoassociatetargetlocationsinconsecutiveframes,especiallyinmultipletargetstrackingproblem.Whentherearemultipleobjectsmovingatthesametime,objectocclusionhappensandthetargetmaynotbevisibleonafewframes,whichmaycausesevereproblem. Imageregistrationisanotherfundamentalproblemincomputervision.Becauseofthecameramotion,thesetofimagesofthesamescenemaybeindierentcoordinatesystemswhenacquiredatdierenttimesorfromdierentperspectives.Imageregistrationistotransformthesequenceofimagesbackintoonecoordinatesystem.Itisacrucialstepinallimageanalysistasksinwhichnalinformationisgainedfromcombinationofvariousdatasource,suchas,changedetection,imagemosaicing,andintegratinginformationintogeographicinformationsystems. 18

PAGE 19

InChapters2and3,weoverviewthetechniquesinthetwoareasrespectively. InChapter4,weusecompressivesensingtorepresentvisualdataandproposeanewimagerepresentationschemeinvisualsensornetworks.Dierentfromthepreviousworksoncompressiveimaging,whichtreattheinputimageasawholesignal,wedecomposethevisualdataintotwocomponentsbeforesampling:adensecomponentandasparsecomponent.Wesamplethedensecomponentbythetraditionalapproachandthesparsecomponentbycompressivesensing.Theadvantageofourworkisthatweusethecorrelationofthetwocomponentstorecoverthesignal,whichhelpstoreducethenumberofmeasurementsandcomputationtimerequiredforreconstructionwiththesameaccuracy.Weproposeandimplementaprojectionontoconvexsetsbasedoptimizationalgorithmtorecoverthesignal. InChapter5,weproposeanewimage/videocompressionsystem,whichcombinescompressivesensingintotraditionalblockbasedimage/videocompressionschemes,suchasJPEGandH.264.Asknowntoall,mostofthecodingerrorintraditionallossycompressionmethodsiscausedbyscalarquantization.CSrecoverytendstosolveaoptimizationproblemtoreconstructtheoriginalsignal,whichcanhelpmitigatingthequantizationerrorindecodingprocess.WealsoproposeaboundedresidueconstrainttobeusedinCSreconstructiontofurtherimprovereconstructionaccuracy. InChapter6,weproposeakerneldeterministicannealingapproachforgeometricttingin3Dspace.Duetothefactthatthe3Ddataislocalizedtoafewrelativelydenseclusters,wedesignakernelfunctiontomapthedatapointfromgeometricalspacetosurfacemodelspaceandapplydeterministicannealingtopartitionthefeaturespaceintoseparateregions.Foreachregion,wecaneasilyndalinearplanemodeltotthedata. 19

PAGE 20

InChapter8,weproposeanewarchitectureforimageregistrationbyleveragingthedepthinformationvia3Dreconstruction.Onenovelideaistorecoverthedepthintheimageregionwithhigh-riseobjectstobuildaccuratetransformfunction.Thetraditionalimageregistrationalgorithmssuerfromparallaxproblemduetotheirunderlyingassumptionthatthescenecanberegardedapproximatelyplanar.However,theassumptionisnotvalidanymoreinthecaseoflargedepthvariationintheimageswithhigh-riseobjects.Ourmethodovercomesthisweaknessandachievesmoreaccurateregistrationperformance.Ouralgorithmisattractivetomanypracticalapplications. InChapter9,weproposeanovelapproachtoestimatemodelparametersofallmotionsbasedonsegmentationofbothintensitymapandopticaloweld.Thenoveltyofourworkisthatweintroduceminimumcodinglengthasacriterioningroupmerging.Theexperimentalresultsshowthatourproposedschemecouldgreatlyimprovetheperformanceofmotioneldsegmentation.Anothernoveltyisthatweproposeaheuristicapproachtolocateglobalmotionbasedonthemotionsegments. InChapter10,weproposeanewtrackingalgorithmbasedonbothtemplatetrackingandsilhouettetracking.Thealgorithmattemptstoadequatelytrackmultipleobjectsofarbitraryshapesinanimagesequence.Inordertoaccuratelyestimatethetrajectory,werstgenerateabinaryobjectmaskandthenonlytrackthefeaturesinsidethemask.ToovercomethelimitationsofthetraditionalKLTfeaturetracker,weproposeanoveltrajectoryestimationalgorithmbasedonaweightingfunctionoftrackedfeaturemotionvectors. 20

PAGE 21

21

PAGE 22

1 ]providessomekeymathematicalinsightsunderlyingthisnewtheory. Duetoalargenumberofnewworkspublishedinthisarea,thischapterwillnotbeanexhaustivesurveyofliteratureoncompressivesensingbutconcentrateonsignalrecoveryalgorithmsandframeworksofapplyingcompressivesensingtoimage/videoprocessing.Therestofthischapterisorganizedasfollows.Section 2.2 givesabriefreviewofcompressivesensingtheory.Section 2.3 providesacomparisonofcurrentsignalreconstructionalgorithmsundercompressivesensingprinciple.Section 2.4 comparestheperformanceofCSreconstructionunderseveraldierentsamplingensembles.Section 2.5 discusseshowtoemployCSinimageprocessing. 22

PAGE 23

2 3 ]andDonoho[ 4 ]demonstratesthatinformationcontainedinafewsignicantcoecientscanbecapturedbyasmallnumberofrandomlinearprojections.Theoriginalsignalcanthenbereconstructedfromtherandomprojectionsusinganappropriatedecodingscheme. WeadoptlanguageandnotationfromCandesetal.'spaper[ 5 ];Lettheobjectofinterestbeadiscretetimesignalf2Rn-thiscouldrepresentansampledvaluesofadiscretetimesignalorimage.Thereasonwefocusondiscretetimesignalinsteadofcontinuoustimesignalistwofold:rstly,thenaturalimagesandvideosaregenerallytakenasdiscretesignalsandsecondly,itisconceptuallysimplerandtheavailablediscreteCStheoryismoredeveloped.Weareinterestedinthesituationinwhichthenumberofmeasurementsismuchsmallerthanthedimensionofthesignalf.Itisespeciallyusefulwhenthemeasurementisextremelyexpensive. InCStheory,thesignalfisobtainedbyanumberoflinearfunctionals. Themeasurementprocessissimplytocorrelatethesignaltobeacquiredwithwaveformk.Thenthequestioniswhetherwecanreconstructthesignalfrommmeasurements,wherem<
PAGE 24

Mathematicallyspeaking,supposewecanexpandsignalf2Rninanorthonormalbasis=[12...n]: wherexisthecoecientsequenceoffindomain.Sparsityrequiresthatthereislittleperceptuallosswhendiscardingmostofthesmallvaluesinx.DenotefK=xKasthesignalobtainedbykeepingonlyKlargestcoecientsandsetallotherszero,thenfKisexactlyK-sparse.Becauseisanorthonormalbasis,wehavekffKkl2=kxxKkl2.IfxKisagoodapproximationofx,thentheerrorkffKkl2issmall. Thedenitionofcoherencebetweentwoorthobasesandis Thecoherencemeasuresthelargestcorrelationbetweenanytwoelementsoftwoorthobasesand.CSrequiresmeasurementbasisandsparsebasistobealowcoherencepair.Usually,CSuserandommatricesasmeasurementmatricessinceitisproventhatrandommatricesarelargelyincoherentwithanyxedbasis. Withthesupportofsparsityandincoherence,CSreconstructthesignalfwithncoecientsfromonlymmeasurementsbysolvinganoptimizationproblem. 24

PAGE 25

Furthermore,CSisabletodealwithnearlysparsesignalwithnoise.Supposewehavethefollowingproblemwhichisslightlydierentfromequation( 2{1 ): whereAisthemnmeasurementmatrixandzisthenoiseterm.Thequestionnowiscanweusethesameorsimilarwaytoreconstructxfrom\downsampled"measurementy? 6 ].RIPguaranteesthatwithapropermatrixA,allsubsetsofKcolumnstakenfromAareinfactalmostorthogonal.WithRIP,wecanusethefollowingmethodtoreconstructx: whereisthevarianceofnoise. Givenalltheinformationabove,therearestillseveralimportantquestionstobeanswered. 25

PAGE 26

7 ],gureprominentlyinthedesignoftractableCSdecoders,highcomplexityO(N3)makesthemimpracticalformanyapplications.Weoftenencountersparsesignalswithlargedimensions.Forexample,currentdigitalcamerasacquireimageswiththenumberofpixelsNoftheorderof106ormore.Forsuchapplications,theneedforfasterdecodingalgorithmsiscritical.Inthissection,wewillreviewcurrentreconstructionalgorithmsandcomparetheperformanceofdierentalgorithms.Weclassifythealgorithmsintotwocategories:algorithmswithl1constraint,andalgorithmswithlpconstraint,wherep<1. 8 { 10 ].OMPrequiresMKln(N)measurementstosucceedwithhighprobabilityandthedecodingcomplexityisO(NK2).Unfortunately,O(NK2)iscubicinNandK,thereforeOMPisimpracticalforlargeKandN. Donohoetal.[ 11 ]recentlyproposedtheStage-wiseOrthogonalMatchingPursuit(StOMP).StOMPisanenhancedversionofOMPwheremultiplecoecientsareresolvedateachstageofthegreedyalgorithm,asopposedtoonlyoneinthecaseofOMP.Moreover,StOMPtakesaxednumberofstageswhileOMPtakesmorestagestorecoverlargercoecientsofx.TheauthorsshowthatStOMPwithfastoperatorsof(suchaspermutedFFTs)canrecoverthesignalinNlogNcomplexity.ThereforeStOMPrunsmuchfasterthanOMPandcanbeusedforsolvinglarge-scaleproblems. 26

PAGE 27

12 13 ]isproposedifsparsesignalshavedistinct\connectedtree"structureinwaveletdomain.TMPalgorithmsignicantlyreducesthesearchspacecomparedtotraditionalmatchingpursuitgreedyalgorithms,resultinginasubstantialdecreaseincomputationalcomplexityforrecoveringpiecewisesmoothsignals.AnotheradvantageofTMPisthatitperformsimplicitregularizationtocombatnoiseinreconstruction.TMPalsoappliestomoregeneralcaseof\incoherent"measurementvectors. Sudocodesisanewschemeforlosslesscompressivesamplingandreconstructionofsparsesignals.Sarvothametal.[ 14 ]proposedanon-adaptivereconstructionalgorithmforsparsecomprisingonlythevalues0and1;hencethecomputationofmeasurementinvolvesonlysumsofsubsetsoftheelementsofthesignal.Anaccompanyingsudodecodingstrategyecientlyrecoversthesignalgiventhemeasurements.SudocodesrequireM=O(Klog(N))measurementsforexactreconstructionwithworst-casecomputationalcomplexityO(Klog(K)log(N)).Sudocodescouldbeusedaserasurecodesforreal-valueddataandhavepotentialapplicationsinpeer-to-peernetworksanddistributeddatastoragesystems.Itcouldalsobeeasilyextendedtosignalsthataresparseinarbitrarybasis. InBioucas-DiasandFigueiredo'spaper[ 15 ],theyintroducedatwo-stepiterativeshrinkagethreshold(TwIST)algorithm,exhibitingmuchfasterconvergenceratethanIST.TheyshowedthatTwISTconvergestoaminimizerofobjectivefunctionwithagivenrangeofvaluesofitsparametersforavastclassofnon-quadraticconvexregularizers(lpnorms,someBesovnorms,andtotalvariation).Fornon-invertibleobservationoperators,theyintroduceamonotonicversionofTwIST(MTwIST);althoughtheconvergenceproofdoesnotapplytothisscenario,theygiveexperimentalevidencethatMTwISTexhibitssimilarspeedgainsoverIST.TheeectivenessofTwISTisexperimentallyconrmedonproblemsofimagedeconvolutionandrestorationwithmissingsamples. 27

PAGE 28

Chantrand'spaper[ 16 ]showsthatexactreconstructionispossiblewithsubstantiallyfewermeasurementsbyreplacingl1normwithlpnorm,whenp<1.Hegivesatheoreminthisdirection,andmanynumericalexamples,bothinonecomplexdimension,andlargerscaleexamplesintworealdimensions. InCandesetal.'spaper[ 17 ],theyproposedanovelmethodforsparsesignalrecoverythatoutperformsl1minimizationalgorithmsinmanysituationsinthesensethatsubstantiallyfewermeasurementsareneededforexactrecovery.Thealgorithmsolvesasequenceofweightedl1minimizationproblemswheretheweightsusedfornextiterationarecomputedfromthevalueofcurrentsolution.Theypresentaseriesofexperimentsdemonstratingtheremarkableperformanceandbroadapplicabilityofthisalgorithminareasofsparsesignalrecovery,statisticalestimation,errorcorrectionandimageprocessing. InCharchandandYin'paper[ 18 ]theyfurtherconsideredusingiterativelyre-weightedalgorithmstocomputelocalminimaofnonconvexproblems.Inparticular,aregularizationstrategyisfoundtogreatlyimprovetheabilityofre-weightedleast-squaresalgorithmtorecoversparsesignals,withexactrecoveryobservedforsignalsthataremuchlesssparsethanrequiredbyunregularizedversion.Improvementsarealsoobservedforreweightedl1approachofCandesetal.'swork[ 17 ]. InDaviesandBlumensath'spaper[ 19 ],theyderivedtwoalgorithmsthatoperatedirectlyonl0regularizedcostfunctionandM-sparseconstrainedoptimizationproblem,respectively.Theyderivednoveltheoreticalresultsforthemethodsandproposedtousethemintwocontexts.Firstly,themethodscouldbeusedtoimprovetheresultscalculatedbyothermethodssuchasMPmethod.Secondly,theyshowedthatthemethodscouldbe 28

PAGE 29

20 ],theycomparefourrandomensemblesusedinCS.Herewegiveabriefreview. Thesechoicesareinspiredbythefollowingearlierworks.Donoho[ 4 ]provedthatrandomsignsmatricesanduniformSphericalensemblesaresuitabletobeusedwhenp=1.Candesetal.[ 2 ]recentlyhavegeneratedagreatdealofexcitementbyshowingseveralinterestingpropertiesofrandompartialFouriermatricesandmakingclaimsabouttheirpossibleuseinCS.Donohocomparedthequasiboundwithactualerrorsindierentmatrixensemblesjustdened.Itpromptsseveralobservations.Firstofall,thesimulationresultsusingdierentensemblesareallqualitativelyinagreementwiththetheoreticalformoferrorbehavior.Moreover,itisobservedthatdierentensemblesshowsimilarbehavior.Thissuggeststhatallsuchensemblesareequallygoodinpractice. 29

PAGE 30

21 ]rstshowedpracticalsignicanceofCStheory.Intheirwork,theyapplyCSrecoveryalgorithmonaseriesofsignalsandnaturalimageswhichcanbewellapproximatedinwaveletbases.Theframeworkistotallydierentfromcurrentimagecompressionmethods.Thepreviousmethods,suchasJPEG2000tendtorepresentanimagewithitsMlargestwaveletcoecients.CSmeasurementsaremadecompletelyrandomandtheyhavenothingtodowiththestructureoftheimage.Fromtheirexperiment,itispossibletorecoveranimagefromabout3Mto5MprojectionsontogenericallychosenvectorswiththesameaccuracyastheidealM-termwaveletapproximation. Gan[ 22 ]furtherdevelopedthisideaandproposedablock-basedsamplingmethodforfastCSofnaturalimages.Theoriginalimageisdividedintosmallblocksandeachblockissampledindependentlyusingthesamemeasurementoperator.ThepossibilityofexploitingblockCSismotivatedbythegreatsuccessofblockDCTcodingsystemswhicharewidelyusedinJPEGandMPEGstandards.Fornaturalimages,thepreliminaryresultsshowthatblockCSsystemsoercomparableperformancestoexistingCSschemeswithmuchsmallerimplementationcost. Zhangetal.[ 23 ]proposedanewimage/videocodingframeworkwhichcombinesCStheoryintotraditionalimage/videocompressionapproaches.TheyassumeCSsampling/recoveryalgorithmismoresuitableforimageblockswithsparsegradientswhileconventionalDCTbasedmethodismoresuitableforcomplicatedimageblocks.Therefore,intheirframework,CSsampling/recoveryalgorithmisintegratedintoJPEGandH.264/AVCcodingmethodsasanewcodingmodeandrate-distortionoptimization(RDO)isemployedformodeselectionbetweenthenewcodingmodeandconventionalcodingmodes.Each88imageblockisencodedanddecodedineitherDCTcodingmodeorCScodingmode.Theexperimentalresultsshowthatwiththenewcodingmode,theaveragebitratereductionisapproximately4%. 30

PAGE 31

24 ]proposedamodel-guidedadaptiverecoveryofcompressivesensing,sothesignalcanberecoveredfaithfully.Inthenewframework,apiecewisestationaryautoregressivemodelisintegratedintorecoveryprocessforCS-codedimages.Comparingtototalvariation(TV)basedcompressivesensingcodingalgorithm,thereconstructionqualityisincreasedby2to7dB. 31

PAGE 32

25 ].Thetraditionalapproachtoanalyzeasingleframeisthroughimagesegmentation[ 26 ].However,incasewehaveasequenceofimagesandthereareseveralobjectsmovinginthescene,orobjectsatdierentdepthswithaglobalcameramotion,motiondiscontinuitywilloccur.Inthiscase,analysisofdierentobjectmotionscanprovidemoreessentialinformationforunderstandingofthescene[ 27 ].Therefore,motionsegmentationisneededtosegmenttheimageframeintoregionsbelongingtodierentmovingobjects. Motionsegmentationcouldbedirectlyappliedtomanyareas.Suchasvideocompression,videodatabasequerying,andsceneanalysis.MPEG-4standard[ 28 ]describesacontentbasedmanipulationofobjectsinimagesequences.Tocreateanobjectbasedscenerepresentation,itisnecessarytosegmentdierentobjectsintheframe.Videoquerying[ 29 ]isanotherneweldwhichaimstoautomaticallyclassifyvideosequencesbasedontheircontent.Acommonvideoquerytaskrequiresretrievingalltheimagesinadatabasethathaveasimilarcontenttothequeryexampleimage.Motionsegmentationenablesanindexingschemethatusestrajectories,shapesandowvectorsofindependentlymovingobjectstoqueryimagesequencesinadatabase.Therefore,thesystemwillgiveamoreaccurateresult.Thedevelopmentofunmannedaerialvehicle[ 30 ]makesreconnaissancemucheasierwhichrequiressceneanalysistechnologytoidentifysuspiciousmilitaryvehiclesinavideosequence.Objectshavingdierentmovingvelocitiesordirectionsneedtobeidentiedandsegmentedforfurtheranalysis.Arealtimevideomonitorsystemwouldhelpheadquartertolearntheenemymovementatthersttime. 32

PAGE 33

31 ].Theyusespatial-temporalderivativesoftheevolvingimagebrightnessfunctiontogiveasingleequationwhichpartiallydeterminestheopticalow.Anassumptionmadeisthatbrightnessofanypartoftheimagechangesveryslowly,sothatthetotalderivativeofbrightnessiszero.Thereasonisobvious.Becauseofapertureproblem,itwouldbehardtodeterminethemotionvectorofthecompleteeldwithoutthisassumption. Inordertodealwithnoisewhichexistsinmostrealworldimagedata,adenoisingprocessisneeded.However,simpledenoisingprocesswilldestroytheboundariesofobjectsofinterest.Therefore,inMumfordandShah'spaper[ 32 ],itissuggestedthattheproblemsofdenoisingandmotionestimationarecloselyinterlacedandshouldbesolvedsimultaneously.InCremers'paper[ 33 ],hepresentedavariationalapproachcalledmotioncompetitionwhichjointlysolvestheproblemsofmotionestimationandsegmentationfortwoconsecutiveframesfromasequenceinasimilarwayastheMumford-Shahapproach. Analyzingopticoweldisoneapproachtomotionsegmentation.WangandAdelson[ 34 ]describedasystemforrepresentingmovingimageswithsetsofoverlappinglayers.Theyusedk-meansclustermethodstodecomposeimagesequencesintolayersbasedonmotion.Thelayersareinorderofdierentdepths.Avelocitymapisusedtodenehowthelayerswarpedovertime.BorshukovandBozdagi[ 35 ]presentedamultistageanemotionsegmentationmethodthatfurthermodiedWangandAdelson'salgorithm.Theyreplacedtheadaptivek-meansclusteringstepbyamergingstepandintroduceamultiplestagespixellabelingmethod.Inthisway,thesegmentationperformanceisimprovedanddemonstratedonrealvideodata. 33

PAGE 34

36 ],theyproposedanapproachtosolvingtheperceptualgroupingproblembasedonextractionoftheglobalimpressionofanimage.Theresultsareencouragingbutcomputationalexpensive. Theobjectiveofatrackingalgorithmistolabelthetargetconsistentlyindierentframesofavideosequence.ObjecttrackingisanimportantprobleminComputerVision.Itisalsoverydicultbecauseofarbitraryobjectshapes,illuminationchanges,objectocclusion,complexobjectmotions,andcameramotion.Eachproblemneedstobesolvedinordertopreventfailureofthetrackingalgorithm.Inanobjecttrackingalgorithmtherearegenerallythreesteps:objectdetection,trackingobjectsfromframetoframe,andtrajectoryestimation.Theprimarydierenceofdierenttrackingalgorithmsisthewaytheyaddressthethreesteps.Moreover,dierentalgorithmsmayimposevariousassumptionsandconstraints,suchassmoothobjectmotion,rigidobject,andprioriknowledgeofobjectappearance.Inthissection,webrieyreviewthepreviousobjecttrackingalgorithmsinthreecategories. 34

PAGE 35

SethiandJain[ 37 ]proposedagreedyapproachbasedonproximityandrigidityconstraints.Theiralgorithmiterativelyminimizesthecostfunctionofcorrespondenceintwoconsecutiveframes.However,thismethodisnotabletohandleocclusions.ThetrackingalgorithmproposedbyRangarajanandShah[ 38 ]takesagreedyapproachwithproximalanduniformityconstrains.Thisalgorithmisabletoobtaininitialfeaturecorrespondencebycomputingopticalowofthersttwoframes,whichmakesitcapabletohandleocclusionproblems.Veenmanetal.[ 39 ]extendedtheworkofSethiandJain,andRangarajanandShahbyintroducingacommonmotionconstraintforpointcorrespondence.TheconstraintofVeenman'salgorithmistheassumptionthatallpointsonthesameobjecthavesimilarmotiondirections,whichisnotsuitablefortrackingisolatedobjects.Duetothefactthatvideosensorsintroducenoises,statisticalcorrespondencemethodsareproposedtosolvetheobjecttrackingprobleminnoisyimagesbytakingthemeasurementofuncertaintiesintoaccountwhenestimatingtheobjectstate.Kalmanlterhasbeenextensivelyusedinobjecttracking[ 40 41 ].Itassumesthatstatevariablesarenormallydistributed.Particleltersarealsousedtoaddressotherdistributions[ 42 ].SincebothKalmanltersandparticleltersarenotsuitableformultipleobjecttracking,JointProbabilisticDataAssociationFilter(JPDAF)[ 43 44 ]andMultipleHypothesisTracker(MHT)[ 45 46 ]areproposedandwidelyusedformultipledataassociation. 35

PAGE 36

47 ]proposedanecientalgorithmfortemplatematching.Theynotonlyusetemplatematching,butalsousecolorhistogramsandmixturemodelstomodeltheobjectsimilarities.AnotheralgorithmthatuseskerneltrackingisproposedbyComaniciuandMeer[ 48 ]whichusesamean-shifttrackertotrackobjectsbyusingaweightedhistogramcomputedfromacircularregiontorepresenttheobject.ComaniciuandMeerproposedanotherapproachtotrackregionsofinterestbyusingprimitiveinformationtocomputetranslation.In1994,ShiandTomasi[ 49 ]proposedthefamousKLTtrackerwhichiterativelycomputesthetranslationofanimagepatchcenteredatapointofinterest.KLTtrackerissimpleandecient.However,featureswillbeeliminatedifthesumofsquareddierences(SSD)issubstantial,becauseSSDindicatesthesimilaritybetweentheselectedobjects.Therefore,KLTisnotsuitableforobjecttrackinginalongimagesequencebecauseitwillincreasethepossibilityoferror.Thegreatestadvantageofkerneltrackingisreal-timeapplicability,andthegoalofsuchtrackersistoestimatethemotionofobject,whichisusuallyinformsoftranslation,aneorprojective.Thelimitationofkernaltrackingisthatprimitivegeometricshapesforobjectrepresentationmaycontainpartsofthebackground.Insuchcases,themotioncomputedbymaximizingmodelsimilaritywillnotbeaccurateduetotheeectofpartialbackgroundinthemodel. 36

PAGE 37

50 ].Thealgorithmmodelsobjectswithedgeinformationfrominnerregionsofanobject'ssilhouette.In2004,Kangetal.[ 51 ]proposedanalgorithmthatusescolorhistogramandedgestomodelobjects.Besides,SatoandAggarwal[ 52 ]proposedanobjecttrackingalgorithmbasedonsilhouettematchingwhichusesHoughtransformtocomputeobjecttrajectory.Recently,contourevolutionisalsousedtotrackobjectsinconsecutiveframes.Bertalmoetal.proposedanalgorithm[ 53 ]thatcomputesmotionvectorsontheedgeofsilhouettesiterativelyforeachcontourpositionusinglevelsetrepresentation.Similarly,Mansouri[ 54 ]proposedacontourtrackingalgorithmbasedonopticalowconstraintwhichcomputesmotionvectorsforallpointsinsidethesilhouette.Theadvantageofsilhouettetrackingistheabilitytotrackobjectsofvariousshapes.Howevercomputationalcomplexityofthesealgorithmsishigh. 55 ],includingmanyclassicmethodswhicharestillinuseuptonow.Duetotherapiddevelopmentofimageacquisitiondevices,moreimageregistrationtechniquesemergesafterwardsandarecoveredinanothersurvey[ 56 ]publishedin2003.Dierentapplicationsduetodistinctimageacquisitionrequiredierentimageregistrationtechniques.Ingeneral,mannersoftheimageacquisitioncanbedividedintothreemaincategories: 37

PAGE 38

Duetothediversityofimagestoberegisteredandvarioustypesofdegradations,itisimpossibletodesignauniversalmethodapplicabletoallregistrationtasks.Everymethodshouldtakeintoaccountnotonlytheassumedtypeofgeometricdeformationbetweenimagesbutalsoradiometricdeformationsandnoisecorruption,requiredregistrationaccuracyandapplication-dependentdatacharacteristics.Nevertheless,mostoftheregistrationmethodsconsistthefollowingfoursteps:featuredetection,featurematching,transformmodelestimation,imageresamplingandtransformation. Awidelyusedfeaturedetectionmethodiscornerdetection.KitchenandRosenfeld[ 57 ]proposedtoexploitthesecond-orderpartialderivativesoftheimagefunctionforcornerdetection.DreschlerandNagel[ 58 ]searchedforthelocalextremeoftheGaussiancurvature.However,cornerdetectorsbasedonthesecond-orderderivativesoftheimagefunctionaresensitivetonoise.ThusForstner[ 59 ]developedamorerobust,althoughtimeconsuming,cornerdetector,whichisbasedontherst-orderderivativesonly.ThereputableHarrisdetector[ 60 ]isinfactitsinverse. Featurematchingincludesarea-basedmatchingandfeature-basedmatching.Classicalarea-basedmethodiscross-correlation(CC)exploitformatchingdirectlyimageintensities.Forfeature-basedmatching,Goshtasbyproposedaregistrationmethodbasedonthegraphmatchingalgorithm[ 61 ].Clusteringtechnique,presentedbyStockmanetal.[ 62 ],triestomatchpointsconnectedbyabstractedgesorlinesegments. Afterfeaturecorrespondenceestablished,themappingfunctionisconstructed.Itwilltransformthesensedimageandoverlayitoverthereferenceimage.Theprevailingimageregistrationmethods,suchasDavisandKeck'registrationalgorithm[ 63 ],assumeallfeaturepointsarecoplanarandtheybuildahomographytransformmatrixfor 38

PAGE 39

Finallyinterpolationmethodssuchasnearestneighborfunction,bilinearandbicubicfunctionsareappliedtooutputtheregisteredimages. 39

PAGE 40

64 ].Inrecentyears,anewtheoryCompressiveSensing(CS)alsoreferredasCompressedSensingorCompressiveSampling,hasbeenproposedasamoreecientsamplingschemeforasparsesignal. ThetheoreticalframeworkofCSisdevelopedbyCandesetal.[ 2 ]andDonoho[ 4 ].TheCSprincipleclaimsthatasparsesignalcanberecoveredfromasmallnumberofrandomlinearmeasurements.Itmeansthatitispossibletoreconstructthesignalx,whichissparseindomain,byasmallnumberofmeasurements,y=x,wherethemeasurementensembleobeystherestrictedisometryhypothesis[ 65 ].Therecoveryprocedureistominimizethel1normofthesignalxindomain,whichisshowntobealinearprogrammingproblemandcouldalsobecastasaconvexoptimizationproblem. ComparedwiththetraditionalNyquist-Shannonsamplingtheory,theCStheoryprovidesagreatreductioninsamplingrate,powerconsumptionandcomputationalcomplexitytoacquireandrepresentasparsesignal.R.Baraniuketal.[ 66 ]haveproposedhardwaretosupportthenewtheoryofCompressiveImaging(CI).ItshowsthatCIisabletoobtainanimagewithasingledetectionelementwhilemeasuringtheimage/videofewertimesthanthenumberofpixels,whichcansignicantlyreducethecomputationrequiredforvideoacquisition/encoding. CShasbeenconnectedwithmanyothereldssuchasinformationtheory[ 6 67 68 ],highdimensiongeometry[ 69 { 72 ],statisticalsignalprocessing[ 73 74 ],anddatastreamingalgorithms[ 75 76 ].Besidestheconnectionstotheexistingtheories,CShasalsobeenused 40

PAGE 41

22 66 77 ],medicalimaging[ 78 ],distributedcompressedsensing[ 79 80 ]andanalogtoinformationconversion[ 81 { 84 ]. MostoftherecentpapersstudytwoproblemsofCS.OneistondtheoptimalsamplingensemblesandstudythemethodsforfastimplementationoftheCSensembles[ 1 20 85 86 ].TheotheroneistodevelopfastandpracticalreconstructionalgorithmstorecoverthesignalandsuppressthenoiseintroducedbyCS[ 21 65 87 88 ]. Donohoetal.[ 20 ]reportseveralfamiliesofrandommeasurementensembleswhichbehaveequivalently,includingrandomspherical,randomsigns,partialFourierandpartialHadamardintheirpaper.Thefollowingworksonmeasurementensemblesstudiestheoptimalmeasurementensemble,whichenablesrecoveringmoreentriesofthesignalwithasfewmeasurementsaspossible.InBaraniuketal.'spaper[ 85 ],theyprovedtheexistenceofoptimalCSmeasurementensemblesandtheyhavecertainuniversalitywithrespecttothesparsityinducingbasis.Elad[ 86 ]furtherproposedanaveragemeasurementofthemutual-coherenceoftheeectivedictionaryanddemonstratedthatitleadstobetterCSreconstructionperformance. Amongthereconstructionalgorithms,BasisPursuit(BP)[ 2 4 ]istherstonetosolvethisproblem.OrthogonalMatchingPursuit(OMP)[ 8 ]isproposedforfastreconstruction.Donohoetal.showthattheHomotopymethodrunsmuchmorerapidlythangeneralpurposedlinearprogrammingsolverswhensucientsparsityispresented[ 10 ].Inordertosuppressthenoiseandincreasethecomputationspeed,Figueiredoetal.[ 89 ]proposedagradientprojectionalgorithmforthebound-constrainedquadraticprogrammingformulationofCSproblem. Inthischapter,weuseCStorepresentvisualdataandproposeanewimagerepresentationschemeforvisualsensornetworks.ComparedtoJPEG2000,CSismoresuitableforapplicationsinsensornetworksbecausethesensorsareresourceconstrained.Dierentfromthepreviousworkoncompressiveimaging,whichtreatstheinputimageasawhole,wedecomposethevisualdataintotwocomponents:adensecomponentand 41

PAGE 42

Thischapterisorganizedasfollows.Section 4.2 givesabriefoverviewofCS.Section 4.3 discusseshowtoapplyCStopracticalsignals.Section 4.4 proposesaschemeforimagerepresentationusingCS.Theproposedreconstructionalgorithmisdiscussedindetails.TheexperimentalresultsarepresentedinSection 4.5 andSection 4.6 concludesthischapter. CSgivesananswertotheabovequestionthatitispossibletorecovertheK-sparsesignalxbytakingMrandommeasurementswhichismuchlessthanN.InordertotakeCSmeasurements,werstdenoteasanMbyNmatrixwithM<
PAGE 43

Dierentfromthetraditionalsampling,CSmeasurementmeasurestheinformationofthewholesignalatonetime.Therefore,eachmeasurementcontainsalittleinformationfromallelementsoftheoriginalsignal.Inthisway,CSisabletoreducethesamplingrate,powerconsumption,andcomputationalcomplexityofthevisualsensors.TheCStheorystatesthatthesignalcouldbeexactlyrecoveredifthenumberofmeasurementsMsatisestheconditionMConstKlogN[ 4 ],whereConstisanover-measuringfactorthatismorethan1. Sinceyisalowerdimensionvectorcomparedtox,itisimpossibletorecoverxdirectlybyapplyingtheinversetransformoftoy.Thesignalisreconstructedbysolvingthefollowingoptimizationproblem. Thereconstructedsignal^xisthesignalamongallsignalsgeneratingthesamemeasureddata,thathastransformcoecientswiththeminimall1norm.Thereconstructioncanbecastasalinearprogrammingproblem. Figure 4-1 givesanexampletoexplainthesignalrecoverybyCSreconstruction.Figure 4-1 (a)showstheoriginalsignalthathas250sampleswithonly25nonzeros,whichisverysparse.Figure 4-1 (b)istheCSmeasurementsbyaGaussianensemble,wherethereareonly90measurements.Figure 4-1 (c)istherecoveryresultfromtheCSmeasurementsbyaPOCSbasedalgorithm.Itisclearthatthesignaliswellrecovered. 43

PAGE 44

Inthissection,wewilldiscusstherstproblemfromthreeaspects:CSreconstructionerrorboundinnoiseenvironment,imagedecompositionandthecorrelationbetweenthedenseandsparsecomponents.Thesecondproblemwillbediscussedinthenextsection. Letusstillusetheexampleofsignalx,wexanorthonormalbasisandthesignalcouldberepresentedbyf(x)=x.Asiswellknown,thecompressibilityofthesignalxrelatestothedecayrateofthecoecientsoff.Ifthecoecientsoffobeyapowerlaw,wehave1nN,jfn(x)jRn1=p,whereRandpareconstantanddependonthesignal,fn(x)isthen-thlargestcoecientsinf(x).ThedierencebetweenxandapproximatesignalxKisobtainedbykeepingthelargestKcoecientsanditobeysthefollowingequation, whereC1isaconstant. InTsaigandDonoho'spaper[ 20 ],theygivetheerrorboundoftheCSapproximationxCSwhichisreconstructedfromthenon-adaptivemeasurements.GivenMmeasurements 44

PAGE 45

whereC2isconstant. Ifthesignalisexactlysparse,thetwoerrorboundsshouldbeatthesamelevel;otherwisetherewillbeabiggapbetweenthem.WecomparetheerrorboundofCSforsignalswithdierentdecayratesanddrawtheerrorbounds.C1,C2,R,MandNareconstantintheexperimentandonlypisavariable,whichcontrolsthespeedofdecay:thesmallerpis,thefasteritdecays.TheexperimentalresultisdepictedinFigure 4-2 ,whereKisthenumberofnon-zerocoecientsforreconstruction.Foreachgroup,thedashlinerepresentstheerrorboundofthetraditionalbestKmethodandthesolidlinerepresentstheresultofCS.Onecanobservethat,whenp=7=16,theerrorboundofCSisveryclosetothatofthetraditionalbestKmethod;however,whenp=9=16,thereisabiggapbetweenthesetwoerrorbounds.Obviously,theperformanceofCSreconstructionreliesonthedecayrateofthesignal.Thefasteritdecays,thebetteritrecovers. 90 ].However,itdoesnotsolveourproblem.Letusrsttakeonedimensionalsignalasanexample.SupposewehaveavectortwithlengthNandapre-chosenbasis,wecanrepresenttas 45

PAGE 46

LettD=PTi=11i1iandtS=PNTi=12i2idenotedenseandsparsecomponentsrespectively.However,fornaturalimages,itishardtondsuchasparsecomponent.Inthispaper,weusethepowerlawmodeltodenethesparsityofthesignal.TaketDforexample,wereorderitscoecients1iandcomputetheparameterpinthemodel, whereRandpareconstantandonlydependontD.Withthismodel,weuseparameterptorepresentthesparsityofthesignal.Thesmallerpis,thefasterthecoecientsdecayandthesparserthesignalis. Theresearchtondafunctionwhichleadstobetterdecompositionofimageisahotareainrecentyears[ 91 ].Previousworkhasprovedthatwavelettransformiswelldesignedforsparserepresentationofnaturalimages.ExpandtheimageIinthewaveletbasis whereWj0,kandWj,karewaveletsatdierentscales.Forsimplicity,inthispaper,wetakethelowestbandofwavelettransformID=Pk1j0,kWj0,kasdensecomponentandtheotherbandsIS=Pj2j=j1Pk2j,kWj,kassparsecomponent,wherej0isthecoarsestscale,j1isthenextscaleandj2isthenestscale. Applyingthepowerlawmodelto1and2respectively,wecanndthatthereisabigdierencebetweenp1andp2,whichmeansthesparsityofthetwocomponentsdierstremendously.Figure 4-3 showstheresultofwavelettransformofLenabyathree-leveldecomposition.Figure 4-4 depictsthedecaycurvesofdenseandsparsecomponents,respectively.Theleftcurverepresentscoecients1ofsignalIDandshowsthat1decaysslowly.Onthecontrary,thecurveontherightsiderepresenting2indicates 46

PAGE 47

where(i,j)isthepixeltobeinterpolated,Bi,jisthewindowcenteredatpixel(i,j)andi,jisarandomperturbationindependentofpixel(i,j)andtheimagesignal. InZhangetal.'spaper[ 92 ],theyformulatetheinterpolationproblemasanoptimizationproblem: whereIuistheimagetobeinterpolatedandtheIvistheoriginalimage.ui2Iuandvi2IvarethepixelsoftheimageIuandIvrespectively.Bisthewindowsize.Thesuperscripts(4)and(8)indicate4-connectneighboringand8-connectneighboringrespectively.Figure 4-5 depictsthesamplerelationshipsinequation( 4{10 ). Zhangetal.[ 92 ]alsogivealinearleast-squaresolutiontothisproblem,whichestimateneighboringpixelssimultaneouslyinwindowB. where^aand^bareestimatedfromIu. 47

PAGE 48

4.3.2 ,wehavediscussedthedecompositionofanaturalimagesignalIintoadensecomponentIDandasparsecomponentIS.Withtheaboveadaptiveinterpolationalgorithm,wecantakeIDasasub-sampleoftheoriginalimageIandinterpolateit.Fromtheabovediscussion,wecangetapredictionofIbysolvingequation( 4{10 ).Thisprediction^Irecoversmostofthehighfrequencyinformationandcouldagainbeexpandedbywaveletbasisasinequation( 4{8 ). Thenewsparsecomponent^IS=Pj2j=j1Pk^2j,kWj,kcanbeusedasagoodpredictionforIS=Pj2j=j1Pk2j,kWj,k. 4.4.1ImageRepresentationScheme 21 22 ],whichtaketheinputimageasanon-separatesignal,inourscheme,theinputimageisdecomposedintotwocomponents,denseandsparsecomponents.Thenthetwocomponentsaresampledusingdierentmethodsrespectively.Forthedensecomponent,weusethetraditionalapproach.Inotherwords,wesamplethedensecomponentpixelbypixel.Whileforthesparsecomponent,weapplyCSbyrandomsampling.TheproposedschemeisdepictedinFigure 4-6 TheinputimageIisrstdecomposedintoadensecomponentIDandasparsecomponentISthroughatransformT,whereTcouldbewavelet,curvelet,oranyothertransforms.Inourscheme,weusediscretewavelettransformWtodecomposetheimage.WeexpandtheimageIasinequation( 4{8 ). 48

PAGE 49

wherej0isthepresetcoarselevelofthewavelettransform.Normally,j0issettobe1. InordertotakemeasurementsofthesparsecomponentsIS,weuseaGaussianrandomensemble.Asweknow,thedimensionoftheinputimageisveryhigh,sodirectlyapplyingthe2DGaussianrandomensembletothesignalISisnotpractical.Inordertoapplytherandomensemblemoreeciently,weneedtoregroupthesignalandtakeablockbasedsamplingstrategy.WerstdivideISintoseveralgroupsbyscalesandthenreorderitintoanumberofvectorsofthesamedimension.Inthisway,wecantakerandommeasurementstothevectorwithamoderatesizeinsteadofthetremendoussize. wherenisthenumberofgroupsandxiisthei-thgroupofIS. Inordertorecovertheoriginalsignal,wehavetoprocessthesignalseparately.Sincethedensecomponentismeasuredpixelbypixel,~IDisexactlythewaveletcoecientsofID.Therefore,wecoulddirectlyapplyinversetransformW1to~IDgogetID.InordertorecoverIS,weneedtosolvetheoptimizationprobleminequation( 4{2 ).InCandesandRomberg'spaper[ 21 ],theyproposeaprojectionontoconvexsets(POCS)algorithmtoreconstructtheoriginalsignalfromtherandommeasurements.WefollowtheirapproachandimprovethealgorithmbyusingpredictionofISasthestartingpointoftheiterations.Theprediction^Ioftheinputimagecouldbeobtainedbyadaptiveinterpolationofthedensecomponentusingequation( 4{10 ).Thenwecouldapplywavelettransformto^Iandget^ISasthepredictionofIS.WewilldiscussthedetailsoftheCSrecoveryalgorithminthenextsub-section. 49

PAGE 50

21 ],theyproposeadierentrecoveryprocedure,whichrequiresasmallamountofprioriinformationofthesignaltoberecoveredbutcostslesscomputationineachiteration.Intheiralgorithm,theyassumethel1normoftherecoveredsignalisknown.WefurtherdevelopthisalgorithmandproposeanewreconstructionmethodbasedonPOCSandpredictionfromadaptiveinterpolation. Sincexinequation( 4{2 )istheuniquesolutionbytheCompressedSensingprinciple,weareabletoclaimthatthel1-ballB=~x:k~xkl1kxkl1andthehyperplaneH=~x:~x=ymeetatexactlyonepoint:BTH=x.BecausebothBandHareconvex,xcanberecoveredbyanalternateprojectionsontoconvexsets(PoCS)algorithm[ 93 ]. Aswehavediscussedinlastsection,wedecomposetheinputimageintodenseandsparsecomponentsandweutilizethedensecomponent~IDtopredictthesparsecomponentISbyadaptiveinterpolation.Thepredictionhelpsintwoaspects:rst,itcouldbeusedastheinitializationoftheiteration.Asknowntoall,theinitializationisveryimportanttoaniterativealgorithmandtheinitialvalueneedtobeinacertainspacefornalconvergenceatlocaloptimal.Secondly,thepredictioncanbeusedasareferencewhichenablesthealgorithmconvergingfasterandmoreaccurately. Fromthestartingpointof~xi,thealgorithmiteratebyalternatingprojectionsontoH,thenontoB.ItisguaranteedtoconvergetoapointinBTH[ 93 ]. Tondtheclosestvector~xHiinHforanarbitrary~xi,weapplytheequation Thesteplengthfor~xHicombinestwoparts,oneiscomputedfromdirectprojectionandtheotherpartisfromthedierencebetween~xiandthepredictionsignalxpi.isauser 50

PAGE 51

Inordertoprojectthevector~xHiontothel1-ballB,weapplyasoftthresholdingoperation. Inordertodeterminethethresholdsuchthatk~xBikl1kxikl1,wesortthecoecientsbyabsolutevalueandperformalinearsearch. 21 22 ].Inthethirdpart,wefurthercomparetheperformanceofouralgorithmtoPOCSbyerrorreductionwiththesameiterations.Atlast,weshowsomerecoveredimages. Figure 4-7 depictsthecomparisonbetweenthepredictionofthesparsecomponentandthecorrespondingcomponent.Inordertoverifythesimilaritybetweentheoriginalsparsecomponentandthepredictionfromthedensecomponent,werstdecomposetheinputsignal,wheretheinputdataisanimagepatchfrom`Lena',intotwocomponents.Theleftpictureshowsthesparsecomponentinwavelettransformdomain.Thenweinterpolatethedensecomponenttogetapredictionoftheoriginalimagepatch.Thepredictionisfurtherdecomposedintosparseanddensecomponents.Thepredictedsparsecomponentisshownintherightpicture. Inordertocomparethemmoreclearly,wescantheimagebyrowsfromlowfrequencytohighfrequency.Thentheimageisscannedintoaone-dimensionalvectoranddepictedinFigure 4-8 .Theuppersubplotshowstheoriginalsignalandthelowersubplotshows 51

PAGE 52

4-8 ,itisclearthatthepredictionbyinterpolatingthedensecomponentisveryclosetotheoriginalsignal.ItmeansthatwecanuselessiterationforconvergenceinCSreconstructionanditisalsopossibletousethepredictionasaweightedconstraintinthereconstructionalgorithm.Theimagesweuseinourexperimentare`Lena',`Boat',`Cameraman'and`Peppers'asusedin[ 21 ]and[ 22 ].AswedescribedinSection 4.4 ,foreachimageI,wecomputetherecoveryimage~Ibasedonequation( 4{15 )andequation( 4{16 ).Theexperimenttestsfordierentsparsityofimagesanddierentsizesofmeasurementensemble.TherecoveryerrorismeasuredbyPSNRindbandtabulatedinTable 4-1 Whynotusebit-ratebutrathernumberofmeasurements?ThereasonisthatthetraditionalquantizationschemeisnotsuitableforCSmeasurements.HowtoquantizetheCSmeasurementsitselfisacrucialproblemtobesolved[ 20 ].Soitisoutofthescopeofthiswork.Therefore,insteadofusingbit-rate,weusenumberofmeasurementsfollowingCandes'swork[ 21 ]. Theresultsarecomparedtothepreviousworks[ 21 ]and[ 22 ].Itisclearthatfor`Lena',`Boats'and`Peppers',ouralgorithmoutperformsthereconstructionalgorithmsin[ 21 ]and[ 22 ].ItmeansweneedfewermeasurementstoachievethesamePSNRorachievebetterPSNRwiththesamenumberofmeasurements.OurnumberofmeasurementsinTable 4-1 includesthemeasurementsofbothIDandIS.Therearetworeasonsfortheimprovement:rst,weremovethedensecomponentfromtheinputdatabydecompositionanditincreasesthesamplingeciency;secondly,byintroducingthepredictionoftherecovereddata,weareabletoconstraintheunknownsignalinasmallspaceneartheoriginalsignalandrequirefewermeasurementsforconvergence.Fortheimage`Cameraman',ouralgorithmlosesalittlebitto[ 21 ]inhighrateend.However,comparingto[ 22 ],ourrecoveryresultismoreaccurate. Inthischapter,wewillnotcompareourresultstoJPEG2000.Thereasonisthatthepurposeofthisworkistoexploreanimagerepresentationschemewhich 52

PAGE 53

94 ]. Inthethirdexperiment,weuse`Peppers'asanexampletotesttheconvergencetime.WecompareerrorreductionbythetotalerrorandPSNRseparatelyinFigure 4-9 .Thesolidlinerepresentsouralgorithmandthedashlineplottheresultsfrom[ 21 ].InFigure 4-9 ,itshowsthatweneed28iterationscomparingto43withthealgorithmin[ 21 ]toreducetheerrorto5106and46iterationscomparingto78toreducetheerrorto4.5106.ThepredictiongreatlyreducesthenumberofiterationsforthesamePSNR,whichmeansweuselesstimeforsignalreconstructionandthereforeagainsavesenergyneeded. Atlast,wegivesomeexamplesofrecoveredimagesinFigure 4-10 andFigure 4-11 .Therecovered`Lena'and`Boats'arebothobtainedfrom20000measurementsintotal. 53

PAGE 54

ExperimentalresultofrecoveringaonedimensionalsparsesignalbyCompressedSensing.Figure 4-1 (a)showstheoriginalsignal.Figure 4-1 (b)istheCSmeasurements.Figure 4-1 (c)istherecoveryresult. Table4-1. Comparisonofreconstructionresultswiththesamenumberofmeasurements. Measurements10000150002000025000 `Lena'[ 21 ]26.528.730.432.1[ 22 ]26.528.630.632.2OurAlgorithm30.031.833.034.2 `Boat'[ 21 ]26.729.831.833.7[ 22 ]27.029.932.534.8OurAlgorithm29.131.033.034.4 `Cameraman'[ 21 ]26.228.730.933.0[ 22 ]24.026.127.929.4OurAlgorithm26.328.529.730.7 `Peppers'[ 21 ]21.625.327.529.4[ 22 ]27.230.332.734.7OurAlgorithm27.430.732.734.6 54

PAGE 55

ComparisonoferrorboundsbetweentraditionalmethodandCompressedSensing.K=100:200,M=4K,N=1024. Wavelettransformofimage`Lena'.Theleftoneistheoriginalimageandtherightisthetransformedimageinwaveletdomain. 55

PAGE 56

Powerlawmodelsofcoecients1ofsignalIDatcoarsescale(left)and2ofsignalISatnescale(right). Figure4-5. 56

PAGE 57

BlockdiagramofourproposedimagerepresentationschemebasedonCompressedSensing. Figure4-7. Predictionofthesparsecomponentin`Lena'.Theleftimageisthesparsecomponentoftheoriginalimageandtherightimageispredictedbyinterpolationofthedensecomponent. 57

PAGE 58

Comparisonofpredictionfromdensecomponentandoriginalsparsecomponent.Botharescannedintoaone-dimensionalvector. Figure4-9. Comparisonoferrorreductionwithsamenumberofiterations.TheleftgureshowsthetotalerrorreductionandtherightgureismeasuredbyPSNRindb.Theresultistestedonimage`Peppers'. 58

PAGE 59

Recoveredimage`Lena'from20000measurements.Theleftimageistheoriginalimageandtherightimageistherecovered. Figure4-11. Recoveredimage`Boats'from20000measurements.Theleftimageistheoriginalimageandtherightimageistherecovered. 59

PAGE 60

2 ]andDonoho[ 4 ],statesthattheinformationthatcontainedinthefewsignicantcoecientsofasparsesignalcanbecapturedbyasmallnumberofrandomlinearprojections.Theoriginalsignalthencanberecoveredbysolvingaoptimizationproblem.Duetothegreatreductionofsamplingrate,powerconsumption,andcomputationalcomplexitytoacquiringsparsesignals,CShasbeenintroducedtomanyareas,suchasinformationtheory[ 6 ],medicalimaging[ 95 ],andimage/videocompression[ 21 { 23 ]. Sincemostnaturalimagesandvideosarehighlycompressibleinthesensethatonlyafewcoecientsarelargewhenexpressedinproperbasis,suchasDCTorWavelet,itispossibletouseCStoreduceencodingcomplexityandimprovecodingperformance.InCandesandRomberg'spaper[ 21 ],theyrstproposeapracticalrecoveryalgorithmandshowthatitispossibletorecoveranimagefromabout3M-5MprojectionsontogenericallychosenvectorswiththesameaccuracyastheidealM-termwaveletapproximation.InGan'spaper[ 22 ],heproposesablock-basedsamplingmethodforfastCSofnaturalimages.TheexperimentalresultsshowthatblockCSsystemoerscomparableperformancestoexistingCSschemeswithmuchlowerimplementationcostfornaturalimages.InZhangetal.'spaper[ 23 ],theyuseCSreconstructioninJPEGcodingframeworkasanewcodingmodeandclaimthatthereisaveragerate-distortion(RD)gainbyapplyingthenewCScodingmode. Inthischapter,weproposeanewimage/videocompressionsystem,whichcombinesCSintotraditionalblockbasedimage/videocompressionschemes,suchasJPEGand 60

PAGE 61

Thischapterisorganizedasfollows.Section 5.2 givesabriefoverviewofCS.Section 5.3 presentstheframeworkofourproposedimage/videocompressionmethod.Section 5.4 discusseshowtoapplyCSrecoveryalgorithmtomitigatequantizationerror.TheexperimentalresultsaregiveninSection 5.5 andSection 5.6 concludesthispaper. 21 ].Supposewehaveanite-lengthsignalx2RN.AtypicalscenarioofCSistotakeMrandommeasurements,whereM<
PAGE 62

CStheorystatesthatxcouldberecoveredexactlybysolvingthefollowingoptimizationproblem. Thereconstructedsignalxisthesignalamongallsignalsgeneratingthesamemeasureddata,whichhastransformcoecientsinwiththeminimall0norm.However,solving( 5{2 )isNP-hardandnumericallyunstable.Therefore,thel1normminimizationisproposedtoapproximatethesignalx, Therecoveryproblem( 5{3 )isaconvexproblemandwhenxisreal,itcanberecastasalinearprogram.Whilelinearprogrammingtechniques,suchasbasispursuit[ 7 ],gureprominentlyinthedesignoftractableCSdecoders,highcomplexityO(N3)makesthemimpracticalformanyapplications.Weoftenencountersparsesignalswithlargedimensions;forexample,currentdigitalcamerasacquireimageswiththenumberofpixelsNoftheorderof106ormore.Forsuchapplications,theneedforfasterdecodingalgorithmsiscritical.Attheexpenseofslightlymoremeasurements,iterativegreedyalgorithmshavebeendevelopedtorecoverxfromy.Examplesincludethematchingpursuit,iterativeorthogonalmatchingpursuit(OMP)[ 8 { 10 ].Candesetal.[ 21 ]proposeapracticalrecoveryalgorithmwhichrequiresaprioriinformationaboutx,butreducesthecostofeachiteration. Ingeneral,naturalimagesandvideosarehighlycompressibleanditispossiblethatwecanuseCStohelpreducingsamplingrateandpowerconsumption. 62

PAGE 63

21 { 23 ].Inthissection,weproposeanewimage/videocompressionsystem,whichtakesadvantageofCSrecoveryintraditionalcodingschemes. WetakeJPEGasanexample.TheJPEGencodingschemeofanimageblockincludesthreesteps:DCTtransform,scalarquantizationandentropycoding.Thecorrespondingdecodingschemeincludesentropydecoding,de-quantizationandinverseDCTtransform.Mostofthecodingerroriscausedbyscalarquantization.StandardJPEGdecoderreconstructsquantizedDCTcoecientstothecenterofthequantizationbin.Thisfailstoexploitthenon-uniformdistributionofACcoecients.Previousde-quantizationmethodsusingLaplaciandistributioninsteadofuniformdistributioncanachieve0.25dBoverstandardJPEGdecoder[ 96 ].CSrecoverybyminimizingtotalvariation(TV)canhelptondthedistributionofACcoecientsratherthanuniformorLaplaciandistribution. Inourframework,weuseCSrecoveryalgorithmasanewcodingmodeinJPEGcodingframework.Ratedistortionoptimization(RDO)isemployedforadaptivemodeselection(MS)betweenthenewmodeandtheconventionalcodingmode.ThesystemschemechartisshowninFig. 5-1 Intheencoderside,weusetruncationinsteadofrandomprojectioninCSsamplingwhichissimilartothemethodusedinZhangetal.'spaper[ 23 ].ThetruncationmeansthatwewillonlykeeptherstKACcoecientsandsettheresttozero.Inthisway,weavoidthepossiblelossthatcausedbyrandommeasurements[ 97 ].Indecoderside,inordertomitigatethescalarquantizationerror,weproposetouseboundedresidueconstraintinsteadofquadraticconstraint[ 21 ]torecovertheimageblock. 63

PAGE 64

98 ].TherecoveryalgorithmwithTVminimizationinCShasbeendiscussedinCandesandRomberg'spaper[ 21 ]. Thenaturalimageitselfisnotsparse,butthegradientmapoftheimagecanbetakenasa2Dsparsesignal.LetIi,jdenotethepixelintheithrowandjthcolumnofannnimageI,anddenetheoperators ThetotalvariationofIisthendenedasthesumofthemagnitudesofthisdiscretegradientinequation( 5{6 )ateverypoint: InZhangetal'spaper[ 23 ],theyalsouseTVminimizationoptimizationtorecovertheimageblockinJPEG.However,intheirscheme,1DDCTisappliedinsteadof2DDCT.Inourexperiment,itshowsthatthereisnoreasonweshoulduse1DDCTintherecoveryprocess.2DDCTfurtherexploresparsityofimageblocks. 64

PAGE 65

wherexo2RNisthevectorbeforequantization,e2RNisthequantizationerror.A2RKNisthetruncationmatrixandy2RKisthemeasurements.InCandesandRomberg'spaper[ 21 ],aquadraticconstraintisusedtosolvethisproblem, Donoho[ 4 ]showsthatthesolutiontoequation( 5{9 )recoversanunknownsparseobjectwithanerroratmostproportionaltothenoiselevel. wherex]isthesolutiongivenbyequation( 5{9 ),CisaconstantanddependsonA.ItisshownthatthetypicalvalueofCisaround10. Inourwork,insteadofndingaglobaloptimalsolution,wearemoreinterestinginafeasiblepointxwhichislocallyoptimal.Tomakeitclear,whatwedoistochangetheinequalityconstraintintoaboundedregion.ThereasonwewanttouseaboundedresidueconstraintisthatweusescalarquantizationinthestandardJPEG.Belowisrewriteoftheproblemwiththeboundedresidualconstraint. whereqisthequantizationstepsize. Comparingtothequadraticconstraint,whichtriestoconstrainthel2normofthequantizationerrorwithapresetthreshold",theboundedresidualconstrainttriestorestrictthevalueofeachpixeloftheimageintoaboundedregion.Sinceweknowthe 65

PAGE 66

23 ]inmeansquareerror(MSE);secondly,wetestouralgorithmonaseriesoftestimagesandcompareourresultswithstandardJPEG. TherstpartofourexperimentistocomparetheperformanceofourmethodwithZhangetal'salgorithm.ThetwomethodsbothapplyRDOforMS,thekeydierenceisthatouralgorithmisbasedon2DDCTandweuseanewconstrainforCSrecovery.SincetheblocksthatchooseCSmodearegiveninZhangetal'spaper[ 23 ],wecansimplyusetheseblocksastestdatatodothecomparison. InFigure 5-4 ,wechoosevetypicalblocksfromtheblocksetgiveninFigure 5-2 asthetestblocksusedinourexperiment.Inourexperiment,wecomparethetotalrecoveryerrorinsteadofPSNRbyapplyingstandardDCT/IDCT,Zhangetal'salgorithm[ 23 ]andourapproach.WechoosethesameQPvaluesandtruncationrate[ 23 ]andbelowinTable1arethetestresults.InTable 5-1 ,theerrorisgivenindB.FromTable 5-1 ,itisclearthatunderthegiventruncationrateandquantizationstepsize,theresultsinpaper[ 23 ]canhardlygetgaininPSNRcomparingtotheconventionalcodingscheme,JPEG.Itispossiblethatmostofthegaingiveninthework[ 23 ]comesfromthesavingofmeasurements.FromTable 5-1 ,wecanndthatCSreconstructionbasedonourapproachcanreducethereconstructionerrorgreatly. Thesecondexperimentistoapplyourcompressionmethodtoseveraltestimages.TheexperimentalresultsaregiveninTable 5-2 InFigure 5-5 ,wegivethetestresultofimage`Boat'.Theexperimentalresultsshowthataveragegainisabout0.5dB.Actually,ifweusepartof`cameraman',wecangetapproximately1dBgain.TheresultisshownisFigure 5-6 .TheproblemisCSencoder/decoderdoesnotworkforallkindsofblocks.Itworksbetterwhenthereisa 66

PAGE 67

Experimentalresultsofthetestblocksfrom`Cameraman'. K(TruncationLength)20263240 Q(Quantizationstepsize)4525126 23 ]39.738439.08339.738442.828OurAlgorithm39.738439.08339.738442.9292 23 ]25.768429.251831.273442.828OurAlgorithm27.456539.08339.738437.9906 23 ]24.794825.6432.571637.471OurAlgorithm27.920230.352736.918939.2378 23 ]25.337628.47631.796137.5773OurAlgorithm27.428130.57834.097536.4984 23 ]27.558232.295631.559643.4742OurAlgorithm31.99334.052139.650844.4607 Table5-2. ExperimentalresultsofBCS. Bitratereduction(%)PSNRgain(dB) Cameraman-10.650.83Boats-2.330.20Lena-0.760.12Peppers-2.950.26 singleedgeintheimageblock.Therefore,thegainwillbeaveragedwhenthedimensionoftheimageislarge. 67

PAGE 68

Theowchartofthenewcompressionscheme. 68

PAGE 69

Theimage`Cameraman'usedinZhangetal'spaper[ 23 ] Figure5-3. TheimageblockswhichuseCSmode[ 23 ]. Figure5-4. Thetestblocksusedintheexperiment. 69

PAGE 70

Testresultof`Boats'. Figure5-6. Testresultof`Cameraman'. 70

PAGE 71

Givena3DpointdatasetX=fxig,xi2R3,i=1,2,...,n,thegeometricalttingproblemisusuallystatedastheoptimizationofacostthatmeasureshowthegeometricalsurfacefunctionS=fx:g(x)=0gtsthedatasetX.Themostcommonlyusedobjectivefunctionistheleastsquarescost, where Thettingfunctiongislearnedbyminimizingthedesigncost,D,measuredovertheinputdataset,X.Itiswell-knownthatformostchoicesofD,thecostmeasuredduringdesignmonotonicallydecreasesasthesizeofthelearnedttingfunctiongisincreased.Withalargesetoffunctions,itiseasytocreateasurfacewhichpassesthrougheachinputdatapointbutissuspiciouslycomplicated.TheprincipleofOccam'srazorstatesthatthesimplestmodelthataccuratelyrepresentsthedataismostdesirable.Soweprefertouseafewbasisfunctionswhichyieldasmoother,simplersurfacewhichcouldwellapproximatestheoriginaldata. Generally,therearetwoapproachestosolvetheoverttingproblem.Oneapproachistoaddpenaltytermstothedataset,likesmoothnessorregularizationconstraints.Anotherapproachistorstbuildalargemodelandthenremovesomeparametersbyretainingonlythevitalmodelstructure.Althoughbothapproachescangenerate 71

PAGE 72

Inthischapter,weproposeadierentapproachtosolvethegeometricalttingproblem.Insteadofestimateacomplicatedfunctiontotallthedatapoints,wepartitionthedatasetintoseveralsubsetsuchthatthedatapointsineachsubsetcouldbeapproximatedbyasimplermodel.Thespacepartitioninghelpstoreducethesizeofthesurfacemodelwhilekeepingthedesigncostsmallenough. OneofthemostpopularclusteringalgorithmisLloyd'salgorithm,whichstartsbypartitioningtheinputdataintokinitialsets.Itcalculatesthecentroidofeachsetviasomemetric.Usually,Lloyd'salgorithmisusedinaEuclideanspaceandcentroidiscalculatedbyaveragingdimensionsinEuclideanspace.Ititerativelyassociateseachpointwiththeclosestcentroidandrecalculatesthecentroidsofthenewclusters.Althoughwidelyusedinrealworldapplications,therearetwoseriouslimitationsofLloyd'salgorithm.Therstlimitationisthatthepartitioningresultdependsontheinitializationoftheclustercenters,whichmayleadtopoorlocalminima.ThesecondlimitationisthatLloyd'salgorithmcanonlypartitionlinearseparableclusters. Inordertoavoidinitializationdependence,asimplebutusefulsolutionistousemultiplerestartswithdierentinitializationstoachieveabetterlocalminima.Globalk-means[ 99 ]isproposedtobuildtheclustersdeterministically,whichusetheoriginalk-meansalgorithmasalocalsearchstep.Ateachstep,globalk-meansaddonemoreclusterbasedonpreviouspartitioningresult.Deterministicannealing[ 100 ]isanotheroptimizationtechniquetondaglobalminimumofacostfunction.Deterministicannealingexplorealargercostsurfacebyintroducingaconstraintofrandomness.Ateachiteration,therandomnessisconstrainedandalocaloptimizationisperformed.Finally, 72

PAGE 73

Kernelmethod[ 101 ]isusedtosolvethesecondproblembymappingthedatapointsfrominputspacetoahigherdimensionalfeaturespacethroughanon-lineartransformation.Thentheoptimizationisappliedinthefeaturespace.Thelinearseparationinthefeaturespaceturnsouttobeanon-linearseparationintheoriginalinputspace. Inthischapter,weproposeanon-lineardeterministicannealingapproachforspacepartitioningin3DEuclideanspace.Weusedeterministicannealingtodividetheinputspaceintoseveralregionswithdierentsizesandshapes.Withthepartition,wecaneasilyndalinearlocalsurfacetotthedatainsideeachregion.Deterministicannealingmethodoerstwogreatfeatures:1)theabilitytoavoidmanypoorlocaloptima;2)theabilitytominimizethecostfunctionevenitsgradientsvanishalmosteverywhere.Duetothefactthatthedataislocalizedtoafewrelativelydenseclusters,wedesignakernelfunctiontomapthedatapointfromthegeometricspacetosurfacefeaturespaceandapplydeterministicannealinginthefeaturespaceinsteadofthegeometricspace.Wecomparetheproposednon-lineardeterministicannealing(NDA)algorithmwiththewidelyusedLloyd'salgorithmonbotharticialdataandrealworlddata.TheexperimentalresultsshowthatNDAalgorithmoutperformsLloyd'salgorithminbothmeansquaredapproximationerroranderrorprobability. Inthefollowingsectionweformallydenethe3Dgeometricttingproblemandbrieydescribedeterministicannealingandkernelmethodforspacepartitioning.InSection 6.3 wepresenttheproposedkerneldeterministicannealingalgorithmalongwithananalysisofitscomputationalcomplexity.TheexperimentalresultisshowninSection 6.4 .FinallySection 6.5 concludesthischapter. 73

PAGE 74

GivenasetofdataXofscattered3Dpoints,wewouldliketondthegeometricsurfacethatbesttstothescattereddata.Thettingproblemisusuallystatedastheoptimizationofacostthatmeasureshowwellthettingfunctiong(xi)tsthedata.Themostcommonlyusedobjectivefunctionistheleastsquarescost.Findingagoodtisachallengingproblemandmaybemoreofanartthanascience.Ifweusealargesetoffunctionsasthebasis,wemaycreateasurfacewhichpassesthrougheachdatapointbutissuspiciouslycomplicated.Usingfewbasisfunctionsmayyieldasmoother,simplersurfacewhichonlyapproximatestheoriginaldata.Duetotheoverttingproblem,weproposeannewapproachtooptimizetheobjectivefunctionviaspacepartitioning.Werstpartitionthedatasetintoseveralsubsetssuchthatthedatapointsxineachsubsetcouldbeapproximatedbyalinearsurfacemodel.Inotherwords,wewouldliketouseasetofplainmodelstoapproximatethedateset.Theobjectiveofspacepartitioningistominimizethegeometricttingerror. where,xi=[xi,yi,zi]Tisthei-thpointdata,k=[ak,bk,ck]Tisthek-thlinearsurfacemodel,anddi,kisisthettingerrorbetweenxiandplanemodelgk=0whichisdenedas 74

PAGE 75

100 ]toclusteringhasdemonstratedsubstantialperformanceimprovementovertraditionalsupervisedandunsupervisedlearningalgorithms.DAmimicstheannealingprocessinstaticTheadvantageofdeterministicannealingisitsabilitytoavoidmanypoorlocaloptima.Thereasonisthatdeterministicannealingminimizesthedesignedcostfunctionsubjecttoaconstraintontherandomnessofthesolution.Theconstraint,Shannonentropy,isgraduallyloweredandeventuallydeterministicannealingoptimizeontheoriginalcostfunction.Deterministicannealingmimicsthesimulatedannealing[ 102 ]instatisticalphysicsbytheuseofexpectation.Deterministicannealingderivesaneectiveenergyfunctionthroughexpectationandisdeterministicallyoptimizedatsuccessivelyreducedtemperatures.Thedeterministicannealingapproachhasbeenadoptedinavarietyofresearchelds,suchasgraph-theoreticoptimizationandcomputervision.A.Raoetal.[ 103 ]extendedtheworkforpiecewiseregressionmodeling.Inthissubsection,wewillbrieyreviewtheirwork. Givenadataset(x,y),theregressionproblemistooptimizethecostthatmeasureshowwelltheregressionfunctionf(x)approximatestheoutputy,wherex2Rm,y2Rn,andg:Rm!Rn.Inthebasicspacepartitioningapproach,theinputspaceispartitionedintoKregionsandthecostfunctionbecomes whered(,)isthedistortionmeasurefunction.Insteadofseekingtheoptimalhardpartitiondirectly,randomnessisintroducedforrandomizedassignmentforinputsamples. InA.Raoetal.'swork,theyusethenearestprototype(NP)structureasconstraintandgiventhesetofprototypesfsj:j=1,2,3,...,Kgintheinputspace,aVoronoi 75

PAGE 76

Althoughtheultimategoalistondthehardpartition,some\randomness"isdesiredduringtheassignment.Shannonentropyisintroducedasaconstraintoftherandomness. Eventually,thisconstrainedoptimizationproblemcouldberewrittenastheminimizationofthecorrespondingLagrangian where,isanonnegativeLagrangemultiplierwhichcontrolstherandomnessofthespacepartition. Takethemostpopulark-meansalgorithm[ 104 ]asanexample,kernelk-meansmapsdatapointsfromtheinputspacetoahigherdimensionalfeaturespacethroughanonlineartransformationandthenapplystandardk-meansinthefeaturespace.Theclusteringresultinlinearseparatorsinfeaturespacecorrespondstononlinearseparatorsininputspace.Thuskernelk-meansavoidthelimitationofstandardk-meansthattheclustersmustbelinearlyseparable. 76

PAGE 77

Tosolvethespacepartitioningproblem,wedonotuseprototypetocalculatethedierence.Thereasonisthattheprototypeinspacepartitioningisgenerallynotsucienttorepresentaplanein3Dspace.Instead,weestimatethelinearplanemodelandcalculatethettingerrorastheEuclideandistancebetweenthedataandtheplane.Thetraditionallocaloptimizationalgorithmwilllikelystuckatalocaloptima.Inordertoavoidlocaloptima,weuselocalgeometricstructurefromneighboringdatapointsandembeddedthedatavectorstoahigherdimensionasfollows. Theinputdataisgivenasa3Dpoint,xi=[xi,yi,zi]T.Withtheassumptionthatnearestdatapointsareonthesameplane,wecouldestimatethelocalplanemodel,Li=[ai,bi,ci]TofdatapointxianditsKnearestneighborpoints. Thenwerevisethedistortionfunctionasfollows, 77

PAGE 78

whereD1=di,jcalculatethettingerrorbetweenthedatapointandtheestimatedplane,andD2calculatethedierencebetweenthelocalestimatedplanemodelandtheclusterscaleestimatedplanemodel.D2isdenedasfollows. Afterthemapping,weapplydeterministicannealingalgorithmtopartitionthedataintoseveralclustersasfollows. wheregj=[aj,bj,cj]isthegeometricalsurfacemodelparametertobeestimated,DisthesumofsquareofgeometricalttingerrorandHistheentropyconstraint.WedeneDandHasfollows: Toperformoptimizationweneedtofurtheranalyzeitsterms.Wecanrewriteequation( 6{18 )byapplyingthechainruleofentropyas 78

PAGE 79

TheminimizationofFwithrespecttoassociationprobabilitiesp(gjjxi)givesrisetotheGibbsdistribution wherethenormalizationis ThecorrespondingminimumofFisobtainedbypluggingequation( 6{21 )backintoequation( 6{16 ) TominimizetheLagrangianwithrespecttotheclustermodelgj,itsgradientsaresettozeroyieldingthecondition Non-lineardeterministicannealingmethod(NDA)introducestheentropyconstrainttoexplorealargeportionofthecostsurfaceusingrandomness,whilestillperformingoptimizationusinglocalinformation,whichissimilartofuzzyc-meansalgorithm.Eventually,theamountofimposedrandomnessisloweredsothatuponterminationNDAoptimizesovertheoriginalcostfunctionandyieldsasolutiontotheoriginalproblem. However,thereisnocloseformsolutionforNDA,thereforeweuseagradientdescentalgorithmtosolvethisproblem.Inthispaper,WecompareNDAbasedgeometricalsegmentationalgorithmtotheprojectionbasediterativealgorithm(PI)andadaptive 79

PAGE 80

1 .Forcomparisonpurpose,IalsogivePIalgorithminAlgorithm 2 6-1 .Krepresentsthenumberofplanesinatestdataset.Foreachplane,100randompointsaregenerated.Thedateset1contains300dataintotalfrom3nonparallelplanes.Thedataset2contains400datafrom4planes.Thedataset3contains500datafrom5planesandthedataset4contains600datafrom6planes.TheaveragesquaredapproximationerrorofNDAisignorablecomparingtotheerrorsofPIandNPI.Fromtheexperimentalresult,wecansaythatNDAalgorithmoutperformsbothPIandAPIalgorithmsintheaveragesquaredapproximationerror.ThereasonNDAalgorithmoutperformsPIandAPIalgorithmsisthatNDAisabletoseparatethespacenon-linearlyandavoidmanypoorlocaloptima. Wealsomeasuretheperformanceofthesegmentationalgorithmsinpercentageofcorrectidenticationofplanes.Wetestthesamedatasetasusedinthepreviousexperimentandcomputethecorrectidenticationpercentageaveragingoveralltests.BelowistheexperimentalresultinTable 6-2 .WeobservedthatcorrectidenticationratesofNDAandAPIaremuchhigherthanthecorrectidenticationrateofPIalgorithm.ThereasonAPIalgorithmoutperformsPIalgorithmisthatAPIalgorithmdoesnotdepends 80

PAGE 81

6-3 .Krepresentsthenumberofplanesinatestdataset.ItshowsthatNDAalgorithmoutperformsbothPIandAPIalgorithm.TheaveragesquaredapproximationttingerrorofNDAalgorithmislessthan50%comparetothettingerrorofPIalgorithm.However,theperformancegainislesscomparedtotherstexperiment.Thereasonisthatthenon-linearmappinginNDAdependsontheestimationofthelocalgeometricstructures.Whiletheestimationofthelocalgeometricstructuresisverysensitivetotheaddednoises.Eventhoughtheperformancegainisless,wecanstillsaythattheNDAalgorithmoutperformsbothPIandAPIalgorithmsintheaveragesquaredapproximationerrorfromtheexperimentalresult.Wealsoshowtheexperimentalresultin3DviewinFigure 6-1 andFigure 6-2 .Figure 6-1 showsthesegmentationresultsoftestdataset1withthreeplanesbytheNDAalgorithm.Figure 6-2 showsthesegmentationresultsofthesametestdatasetbythePIalgorithm. 6-3 showsthe 81

PAGE 82

Theaveragesquaredapproximationerror. KPIAPINDA 33.771013.001091.17101244.011019.811082.21101252.431012.861093.06101262.941018.8011093.001012 6-4 showsthegeometricsegmentationresultbyNDAalgorithmandFigure 6-5 showsthegeometricalsegmentationresultbyPIalgorithm.ItisprettyclearthatNDAalgorithmpartitionstheinputdatasetintothreeclustersandeachclusterrepresentsawallintheimage.PIalgorithmfailstondthegeometricmodelofthewallsandthedatapointsaremixed.TheexperimentalresultonrealworlddatashowsthatNDAalgorithmcanwellsegmentthedatasetsbasedontheirgeometricrelationship. 82

PAGE 83

2,1 2],8i. Thecorrectidenticationrate. KPIAPINDA 383%96%99%479%93%99%582%94%97%678%97%98%

PAGE 84

Table6-3. Theaveragesquaredapproximationerror. KPIAPINDA 36.611018.961012.4110148.181015.981013.1910156.981014.421013.9610161.169.441016.71101 Thesyntheticdataset. 84

PAGE 85

TherstgrouppartitionedbyK-means. 85

PAGE 86

Theinputdatapointsonthe1stframeof`oldhousing'imagesequence. 86

PAGE 87

ThegeometricalsegmentationresultbytheNDAalgorithmof`oldhousing'dataset. 87

PAGE 88

ThegeometricalsegmentationresultbythePIalgorithmof`oldhousing'dataset. 88

PAGE 89

105 { 108 ].Currently,mostofthesystemsandapplicationsin3Dreconstructionareusedforvisualinspectionandarchitecturemodeling.However,thereismoredemandfor3Dentertainment,forexample,3Dmovies.Thechangeofdemandresultsinanattentionforsmoothvisualqualityofthereconstructedscene.Inthiscase,visualqualityofthevirtualscenebecomesthedominantfactor.Whiletheforemostgoalinpreviousapproachesistheaccuracyofthepositionofeachpointin3Dgeometry. Inthelasttwodecades,tremendousprogresshasbeenmadeonself-calibrationand3Dsurfacemodeling[ 109 { 112 ].Mostofthemethodsuse2Dvideosequencesor2Dimagesasinputandtrytoretrievethedepthinformationofthescenecapturedbytheinputvideosequence.Theestimateddepthinformationhelpstoreconstructthefull3Dviewofthescene.Theexistingtechniquesareabletowellcalculatethecameramotionandcomputeasparsedepthmapfromtheoriginalimagesequence[ 105 113 { 116 ].However,fullyreconstructionofa3Dscenerequiresthedepthinformationofmuchmoreimagepixelswhichrequiresthealignmentofalmostallpixelsoftheinputimages.Thisproblemisknownasdensematchingproblem[ 117 { 119 ]. Atraditionalsolutiontothedensematchingproblemiscalledepi-linesearching.Epi-linesearchmethodusesthegeometricconstraintstodegradea2Dsearchingtoa1Drangesearching[ 120 { 122 ].Althoughthesearchisconstraintto1Dwhichseemseasiertosearch,theblankwallproblem,whichisnotsolvedin2Dfeaturecorrespondence,stillexistinepi-linesearch.Theblankwallproblemisthatgivenatexturelessblankwall,itisveryhardtondanaccuratepixeltopixelcorrespondenceacrosstheinputimages. 89

PAGE 90

107 123 ].Insteadofusingpixel-basedsearchingandmatching,volumetricreconstructiontakesthesceneasatessellationof3Dcubes,calledvoxels.Eachvoxelmaybeeitheremptyoroccupiedbythescenestructure.Variousmethodshasbeenproposedtobuildthevolumetricmodelwhichisusedtogeneratethemostconsistentprojectionswiththeoriginalimages.Volumetricreconstructioncouldwellrecoverthesceneofthemovingforeground,however,itishardtorevealthestaticbackgroundstructureusingvolumetricmethods. Inthischapter,weproposeanovel3Ddensereconstructionmethodbasedongeometricsegmentationandsurfacetting.Weusetheexistingtechniquesforfeaturecorrespondence,projectivereconstructionandself-calibrationtogetthesparsepointsreconstruction.Toaddressthedensematchingproblem,weusegeometricsegmentationtosegmentthe3Dspaceintoseveralseparateregions,andforeachregion,weestimatethedense3Ddepthmapbysurfacetting.Weproposeanon-lineardeterministicannealingalgorithminordertopartitionthe3Dspacegeometrically.Withtheassumptionthateachsubspacecouldbemodeledbyalinearplane,wecanretrievethedepthinformationforeachpixelusingsurfacetting.Thenewapproachisabletogenerateamuchsmoother3Ddensereconstructioncomparingtothetraditionalmethods. Thischapterisorganizedasfollows.Section 7.2 presentthebackgroundandproblemformulation.Wepresentthesystemschemefor3DreconstructioninSection 7.3 .ThenwesolvethegeometricsegmentationandsurfacettingprobleminSection 7.4 .TheexperimentalresultsareshowninSection 7.5 .Finally,Section 7.6 concludesthispaper. 90

PAGE 91

122 ].ThepipelineisgiveninFigure 7-1 Therststepin3Dreconstructionfromavideosequenceistogroupthewholevideosequenceintoseveralscenesbykeyframes.Foreachscene,motiondetectionisneededtondmovingregionsfromthestaticbackground.Inthelaterpart,movingforegroundandstaticbackgroundwillbetreatedseparatelyandthencombinedtogethertoreconstructthesceneasawhole. Thesecondstepissparsereconstruction.Sparsereconstructionincludesseveralcomponent,featurecorrespondence,projectionreconstructionandEuclideanreconstruction.ThecameramotionisestimatedandTheEuclideanstructureofthestaticbackgroundsceneisrecovered.Forthemovingregions,weintroducethevirtualcameraconceptandapplythesamereconstructionalgorithmtorecoverthe3Dstructure.Duringthelasttwodecades,tremendousprogresshasbeenmadetocameraself-calibrationandstructurecomputation.Sparsereconstructionstartsfromfeaturecorrespondencewhichisthemostcrucialpartoftheprocess.ThegoalofImagecorrespondence,alsocalledfeaturecorrespondence,istoaligndierentimages,fromavideosequenceortakenseparately,byndingcorrespondingpointsthatdescribethesamepointin3Dgeometry[ 124 125 ].Asknowntoall,notallpointsaresuitableformatchingortrackingthroughdierentimages,soonlyafewpointsareselectedasfeaturepointsformatching[ 49 ].Sosparsereconstructiononlyrelyonanumberofdistinctpointswhichisdierentfromthefollowingdensereconstructionwhichrequirethecorrespondenceofallpoints,ifpossible.Furthermore,featurepointsmaybemismatched,knownasoutliers[ 126 ],whichmayrestricttheaccuracyofthereconstructionresult.Givencorrectlymatchedfeaturepointsfromtwoinputimages,projectionreconstructionistondtherelativeposebetween 91

PAGE 92

Thesparsereconstructiongivesasparsestructureofthedesiredscene;however,itcouldnotgiveasatisedvisualpresentation.Thus,westillneedtocomputethedepthofalotmorepoints,whichisknownasdensereconstructionorsurfacereconstruction.Thetraditionalapproachesfordensereconstructioncouldbeclassiedastwoapproaches,namelystereoscopicreconstructionandvolumetricreconstruction.Inthischapter,weproposeanovelapproachtoobtainthestaticbackgroundstructure.Unlikethepreviousapproach,weapplygeometricalsegmentationandsurfacettinginsteadofdensesearchingandmatching.Hereweassumethatthestaticbackgroundcouldbedecomposedofseveraluniformregionsorregularsurfaces.Wecanthensegmentthewholesurfaceintoseveralregionsbasedontheirgeometricproperties.Foreachregion,weobtainamathematicalexpressionbysurfacetting.Withtheassumptionthateachregionhassucientnumberofsparsefeaturepoints,combinedwiththesparsedepthmap,wecouldthencomputethedepthinformationbyttingeachpixelwithintheestimatedsurface.Combiningthedepthmapofdierentregions,wecouldnallyobtainthedepthmapofthewholescene.Themeritofthisapproachisthatitwellhandlesuniformregions 92

PAGE 93

7.2.2 andwegivethesolutiontotheproblemindetailsinSection 7.4 Givena3DpointdatasetX=fxig,xi2R3,i=1,2,...,n,thegeometricalttingproblemisusuallystatedastheoptimizationofacostthatmeasureshowthegeometricalsurfacefunctionS=fx:g(x)=0gtsthedatasetX.Themostcommonlyusedobjectivefunctionistheleastsquarescost, Thettingfunctiongislearnedbyminimizingthedesigncost,D,measuredovertheinputdataset,X.Itiswell-knownthatformostchoicesofD,thecostmeasuredduringdesignmonotonicallydecreasesasthesizeofthelearnedttingfunctiongisincreased.Withalargesetoffunctions,itiseasytocreateasurfacewhichpassesthrougheachinputdatapointbutissuspiciouslycomplicated.TheprincipleofOccam'srazorstatesthatthesimplestmodelthataccuratelyrepresentsthedataismostdesirable.Soweprefertouseafewbasisfunctionswhichyieldasmoother,simplersurfacewhichcouldwellapproximatestheoriginaldata.Generally,therearetwoapproachestosolvetheoverttingproblem.Oneapproachistoaddpenaltytermstothedataset,likesmoothnessorregularizationconstraints.Anotherapproachistorstbuildalargemodelandthenremovesomeparametersbyretainingonlythevitalmodelstructure.Althoughbothapproachescangenerateparsimoniousmodels,thedescentbasedlearningmethodsallsuerfromaseriouslimitation.Thenon-globaloptimaofthecostsurfacemayeasilyresultinpoorlocal 93

PAGE 94

111 ]onwhichourexperimentsarebased.Whendevelopingastereovisionalgorithmforregistration,therequirementsforaccuracyvaryfromthoseofstandardstereoalgorithmsusedfor3Dreconstruction.Forexample,amulti-pixeldisparityerrorinanareaoflowtexture,suchasawhitewall,willresultinsignicantlylessintensityerrorintheregisteredimagethanthesamedisparityerrorinahighlytexturedarea.Inparticular,edgesandstraightlinesinthesceneneedtoberenderedcorrectly. 7.4 .Finallythedensedepthmapisreconstructedbygeometrictting.ThesystemschemeisgiveninFigure 7-2 111 ]usepointfeatureinreconstructionwhichismeasuredbyHarris'criterion, 94

PAGE 95

whereW(x)isarectangularwindowcenteredatxandIxandIyarethegradientsalongthexandydirectionswhichcanbeobtainedbyconvolvingtheimageIwiththederivativesofapairofGaussianlters.Thesizeofthewindowcanbedecidedbytheuser,forexample77.IfC(x)exceedsacertainthreshold,thenthepointxisselectedasacandidatepointfeature. Weusethesumofsquareddierences(SSD)asthemeasurementofthesimilarityoftwopointfeatures.Thenthecorrespondenceproblembecomeslookingforthedisplacementdthatsatisesthefollowingoptimizationproblem: mindXx2W(x)[I2(x+d)I1(x)]2(7{5) wheredisthedisplacementofapointfeatureofcoordinatesxbetweentwoconsecutiveframesI1andI2.LucasandKanadealsogivethecloseformsolutionofequation( 7{5 ). where 7{3 ),andIt.=I2I1. 95

PAGE 96

111 ].Forthedetailoftheproofofthisalgorithm,pleaserefertothereference. Thereconstructionalgorithmisbasedonaperspectiveprojectionmodelwithapinholecamera.Supposewehaveagenericpointp2E3withcoordinatesX=[X,Y,Z,1]Trelativetoaworldcoordinateframe.Giventwoframesofonescenewhichisrelatedbyamotiong=(R,T),thetwoimageprojectionpointx1andx2arerelatedasfollows: wherex0=[x,y,1]Tismeasuredinpixels,1and2arethedepthscaleofx1andx2,1=[K,0]and2=[KR,KT]arethecameraprojectionmatricesandKisthecameracalibrationmatrix.Inordertoestimate1,2,1and2,weneedtointroducetheepipolarconstraint.Fromequation( 7{8 ),wehave Thefundamentalmatrixisdenedas: Withtheabovemodel,wecouldestimatethefundamentalmatrixFviatheEight-pointalgorithm.Thenwecoulddecomposethefundamentalmatrixtorecovertheprojectionmatrices1and2andthe3Dstructure.Weonlygivethesolutionherebycanonicaldecomposition: 96

PAGE 97

wheremeansequalityuptoascalefactorand WiththeassumptionthatKisconstant,wecouldestimatetheunknownsKandwithagradientdecentoptimizationalgorithm.Inordertoobtainauniquesolution,wealsoassumethatthesceneisgenericandthecameramotionisrichenough. Inlastchapter,wehaveproposedanewgeometricttingmethodbasedongeometricsegmentation.Werstsegmentthesurfaceofthe3Dsceneintoseveralregionsbasedonthegeometricrelationship.Foreachsmallhomogeneoussurface,weareabletomodelitbyaplane.Withthedepthinformationofthefeaturepointsthatwealreadygetfromthesparsereconstruction,wecouldcomputethedepthinformationforeachpixelintheentireregion.Sincethedepthinformationweobtainedisbasedonaplanemodel,theimagerenderedfromthe3Dmodelismuchsmootherthanthetraditionalapproaches.Inordertosimplifytheproblemofsurfacetting,werstsegmenttheinputimagebasedonitsgeometricstructure.Itisdierentfromthetraditionalobjectbasedimagesegmentation.Thesegmentationprocessiscriticalbecausepropersegmentationcouldsimplifythe 97

PAGE 98

Duetothefactthatthe3Ddataislocalizedtoafewrelativelydenseclusters,wedesignanon-linearfunctiontomapthedatapointfromgeometricalspacetosurfacemodelspaceandapplydeterministicannealinginthefeaturespacetopartitionthefeaturespaceintoseveralregionswithdierentsizesandshapes.Foreachregion,wecaneasilyndalinearplanemodeltotthedata.Non-lineardeterministicannealingmethodoersthreeimportantfeatures:1)theabilitytoavoidmanypoorlocaloptima;2)theabilitytominimizethecostfunctionevenitsgradientsvanishalmosteverywhere;3)theabilitytoachievenon-linearseparation.However,thereisnocloseformsolutionfornon-lineardeterministicannealingproblem,thereforeweuseagradientdescentalgorithmtosolvethisproblem.ThedetailsofthisalgorithmisdiscussedinSection 7.4 whereA=[Xie1],i=1,...,mandp=[a,b,c]Tistheplaneparameter. Givenanarbitrarypointxi=[xi,yi]Tmeasuredinpixelsintherstcluster,wecouldestimateit'sdepthscaleibysolvingthefollowingequation. 98

PAGE 99

7{15 ,onlyiisunknownandwiththeconstraintonXiewithequation 7{14 ,wecouldeasilygetthevalueofi. Then,with1=[I,0],wecouldhaveXip=[i1xi,i1yi,i1,1].fromequation 7{8 ,wecangettherelationbetweentwoimageprojectionpointxi1andxi2asfollows: wherecxi20=[i2xi2,i2yi2,i2].Wecouldthengetthepositionofthecorrespondingpointxi2=[xi2,yi2]inthesecondimage. Inlastchapter,wehaveproposedanon-lineardeterministicannealingapproachforspacepartitioningin3DEuclideanspace.Weusedeterministicannealingtodividetheinputspaceintoseveralregionswithdierentsizesandshapes.Withthepartition,wecaneasilyndalinearlocalsurfacetotthedatawithineachregion.Deterministicannealingmethodoerstwogreatfeatures:1)theabilitytoavoidmanypoorlocaloptima;2)theabilitytominimizethecostfunctionevenitsgradientsvanishalmosteverywhere.Duetothefactthatthedataislocalizedtoafewrelativelydenseclusters,wedesignanon-linear 99

PAGE 100

6 .Forreader'sconvenience,webrieyrepeatthealgorithminthissubsection. Tosolvethespacepartitioningproblem,wedonotuseprototypetocalculatethedierence.Thereasonisthattheprototypeinspacepartitioningisgenerallynotsucienttorepresentaplanein3Dspace.Instead,weestimatethelinearplanemodelandcalculatethettingerrorastheEuclideandistancebetweenthedataandtheplane.Thetraditionallocaloptimizationalgorithmwilllikelystuckatalocaloptima.Inordertoavoidlocaloptima,weuselocalgeometricstructurefromneighboringdatapointsandembeddedthedatavectorstoahigherdimensionasfollows. Theinputdataisgivenasa3Dpoint,xi=[xi,yi,zi]T.Withtheassumptionthatnearestdatapointsareonthesameplane,wecouldestimatethelocalplanemodel,Li=[ai,bi,ci]TofdatapointxianditsKnearestneighborpoints. 100

PAGE 101

Thenwerevisethedistortionfunctionasfollows, whereD1=di,jcalculatethettingerrorbetweenthedatapointandtheestimatedplane,andD2calculatethedierencebetweenthelocalestimatedplanemodelandtheclusterscaleestimatedplanemodel.D2isdenedasfollows: Afterthemapping,weapplydeterministicannealingalgorithmtopartitionthedataintoseveralclustersasfollows. wheregj=[aj,bj,cj]isthegeometricalsurfacemodelparametertobeestimated,DisthesumofsquareofgeometricalttingerrorandHistheentropyconstraint.WedeneDandHasfollows: 101

PAGE 102

7{25 )byapplyingthechainruleofentropyas NoticethatthersttermH(X)istheentropyofthesourceandisthereforeconstantwithrespecttotheclustergjandassociationprobabilitiesp(gjjxi).Thuswecanjustfocusontheconditionalentropy TheminimizationofFwithrespecttoassociationprobabilitiesp(gjjxi)givesrisetotheGibbsdistribution wherethenormalizationis ThecorrespondingminimumofFisobtainedbypluggingequation( 7{28 )backintoequation( 7{23 ) TominimizetheLagrangianwithrespecttotheclustermodelgj,itsgradientsaresettozeroyieldingthecondition 7.5.13DVideoDenseReconstruction 102

PAGE 103

7-3 showstherstframeandthe88thframeofthetestimagesequence`oldhousing'.Werstextractpointfeaturesonalltheinputimages.Thenweapplyfeaturecorrespondencealgorithmtorelateallthefeatures.Figure 7-4 showtheselectedfeaturepointsontherstframe.Wethenestimatethecameraposeandintrinsicparameters.Withthecameraparameters,weareabletorecoverthesparseEuclidianstructureofthefeaturepoints.Figure 7-5 showstheestimateddepthmapoftheselectedfeaturepointsandthecamerapose.Aftersparsereconstruction,weseparatethe3DspaceintoseveralregionsusingNDAalgorithm.Foreachregion,weusethesurfacettingalgorithmpresentedinSection 7.3 toestimatethedepthinformationofeachpixel.Combiningthedepthmapofallregions,wecanrecoverthe3Ddensedepthmapofthewholeframe.Figure 7-6 showstheestimateddensedepthmapofthewholeframe.Sinceweusesurfacettinginsteadofsearchingfordensedepthestimation,wedonotneedtoworryaboutmatchingerrorsandoutliers.Theestimateddensedepthmapisverysmoothandwellrepresentthegeometricstructureofthe3Dscene. 103

PAGE 104

Thepipelinefor3Dvideoreconstructionsystem. 104

PAGE 105

Theschemefor3Dvideoreconstructionsystem. BThe88thframeinthe`oldhousing'videosequence Originalframesusedforimageregistration. 105

PAGE 106

Thefeaturepointsontherstframeof`oldhousing'imagesequence. 106

PAGE 107

Theestimatedsparsedepthmapandcameraposefortheselectedfeaturepointsofthe1stand88thframes. Figure7-6. Theestimateddense3Dconguration. 107

PAGE 108

55 ],includingmanyclassicmethodsstillinuse.Duetotherapiddevelopmentofimageacquisitiondevices,moreimageregistrationtechniquesemergedafterwardsandwerecoveredinanothersurveypublishedin2003[ 56 ]. Dierentapplicationsduetodistinctimageacquisitionrequiredierentimageregistrationtechniques.Ingeneral,mannersoftheimageacquisitioncanbedividedintothreemaingroups: Theprevailingimageregistrationmethods,suchasDavisandKeck'salgorithm[ 63 127 ],assumeallthefeaturepointsarecoplanarandbuildahomographytransformmatrixtodoregistration.Theadvantageisthattheyhavelowcomputationalcostandcanhandleplanarscenesconveniently;however,withtheassumptionthatthescenesareapproximatelyplanar,theyareinappropriateintheregistrationapplicationswhentheimageshavelargedepthvariationduetothehigh-riseobjects,knownastheparallax 108

PAGE 109

Inthischapter,weproposeadepthbasedimageregistrationalgorithmbyleveragingthedepthinformation.Ourmethodcanmitigatetheparallaxproblemcausedbyhigh-risescenesintheimagesbybuildingaccuratetransformfunctionbetweencorrespondingfeaturepointsinmultipleimages.Givenanimagesequence,werstselectanumberoffeaturepointsandthenmatchthefeaturesinallimages.Thenweestimatethedepthofeachfeaturepointfromfeaturecorrespondences.Withthedepthinformation,wecanprojecttheimagein3Dinsteadofusingahomographytransform.Furthermore,afastandrobustimageregistrationalgorithmcanbeachievedbycombiningthetraditionalimageregistrationalgorithmsanddepthbasedimageregistrationmethod.Theideaisthatwerstcomputethe3Dstructureofasparsefeaturepointssetandthendividethescenegeometricallyintoseveralapproximatelyplanarregions.Foreachregion,wecanperformadepthbasedimageregistration.Accordingly,robustimageregistrationisachieved. Theremainderofthischapterisorganizedasfollows.Wepresentthesystemschemefor2DimageregistrationinSection 8.2 .Section 8.3 reviewsthe3Dreconstructionalgorithmweusedinournewmethod.InSection 8.4 ,wedescribehowtouse3Ddepthinformationfor2Dimageregistrationandproposeanon-lineardeterministicannealingalgorithmforspacepartitioning.Section 8.5 presentstheexperimentalresultsandwecompareouralgorithmwithDavisandKeck'salgorithmonthesametestvideosequence.WeconcludethispaperinSection 8.6 109

PAGE 110

8-1 Awidelyusedfeaturedetectionmethodiscornerdetection.KitchenandRosenfeld[ 57 ]proposedtoexploitthesecond-orderpartialderivativesoftheimagefunctionforcornerdetection.DreschlerandNagel[ 58 ]searchedforthelocalextremaoftheGaussiancurvature.However,cornerdetectorsbasedonthesecond-orderderivativesoftheimagefunctionaresensitivetonoise.ThusForstner[ 59 ]developedamorerobust,althoughtimeconsuming,cornerdetector,whichisbasedontherst-orderderivativesonly.ThereputableHarrisdetector[ 60 ]alsousesrst-orderderivativesforcornerdetection.Featurematchingincludesarea-basedmatchingandfeature-basedmatching.Classicalarea-basedmethodiscross-correlation(CC),whichexploitsformatchingimageintensitiesdirectly.Forfeature-basedmatching,Goshtasby[ 61 ]describedtheregistrationbasedonthegraphmatchingalgorithm.Clusteringtechnique,presentedbyStockmanetal.[ 62 ],triestomatchpointsconnectedbyabstractedgesorlinesegments.Afterthefeaturecorrespondencehasbeenestablishedthemappingfunctionisconstructed.Themappingfunctionshouldtransformthesensedimagetooverlayitoverthereferenceimage.Andnallyinterpolationmethodssuchasnearestneighborfunction,bilinear,andbicubicfunctionsareappliedtotheoutputoftheregisteredimages. 110

PAGE 111

8-2 .Inthenewsystemscheme,werstapply3Dreconstructiontotheinputimagesandrecoverthe3Dgeometricstructureofthesceneintheimages.The3Dmodelismoreaccuratecomparedtothe2Dmotionmodelsestimatedinthepreviousworks.Thenwesegmentthe3DEuclideanspacegeometricallyintoseveralseparateregions.Eachregioncouldbemodeledbyalinearplane.Withthesegmentation,wecanestimatethe3Ddepthforeverypixelineachregionandrecoverthedensestructureofthescene.The3Ddensestructureenablesthepixelbypixelmappingoftheinputimages.Wedescribethe3DreconstructionalgorithminSection 8.3 .InSection 8.4 ,wepresentthegeometricsegmentationanddepthbasedmappingin3D,andalsoproposeanon-lineardeterministicannealingalgorithmforspacepartitioning. 111 ].Whendevelopingastereovisionalgorithmforregistration,therequirementsforaccuracyvaryfromthoseofstandardstereoalgorithmsusedfor3Dreconstruction.Forexample,amulti-pixeldisparityerrorinanareaoflowtexture,suchasawhitewall,willresultinsignicantlylessintensityerrorintheregisteredimagethanthesamedisparityerrorinahighlytexturedarea.Inparticular,edgesandstraightlinesinthesceneneedtoberenderedcorrectly. The3Dreconstructionalgorithmisimplementedusingthefollowingsteps.First,geometricfeaturesaredetectedautomaticallyineachindividualimages.Secondly,featurecorrespondenceisestablishedacrossalltheimages.Thenthecameramotionisretrievedandthecameraiscalibrated.FinallytheEuclideanstructureofthesceneisrecovered. 111 ]usepointfeatureinreconstruction 111

PAGE 112

wherex=[x,y]Tisacandidatefeature,C(x)isthequalityofthefeature,kisapre-chosenconstantparameterandGisa22matrixthatdependsonx,givenby whereW(x)isarectangularwindowcenteredatxandIxandIyarethegradientsalongthexandydirectionswhichcanbeobtainedbyconvolvingtheimageIwiththederivativesofapairofGaussianlters.Thesizeofthewindowcanbedecidedbythedesigner,forexample77.IfC(x)exceedsacertainthreshold,thenthepointxisselectedasacandidatepointfeature. Weusethesumofsquareddierences(SSD)[ 124 ]asthemeasurementofthesimilarityoftwopointfeatures.Thenthecorrespondenceproblembecomeslookingforthedisplacementdthatsatisesthefollowingoptimizationproblem: mind.=Xx2W(x)[I2(x+d)I1(x)]2(8{3) wheredisthedisplacementofapointfeatureofcoordinatesxbetweentwoconsecutiveframesI1andI2.LucasandKanadealsogivethecloseformsolutionofequation( 8{3 ): 112

PAGE 113

8{1 ),andIt.=I2I1. 111 ].Forthedetailoftheproofofthisalgorithm,pleaserefertothereference. Thereconstructionalgorithmisbasedonaperspectiveprojectionmodelwithapinholecamera.Supposewehaveagenericpointp2E3withcoordinatesX=[X,Y,Z,1]Trelativetoaworldcoordinateframe.Giventwoframesofonescenewhichisrelatedbyamotiong=(R,T),thetwoimageprojectionpointx1andx2arerelatedasfollows: wherex0=[x,y,1]Tismeasuredinpixels,1and2arethedepthscaleofx1andx2,1=[K,0]and2=[KR,KT]arethecameraprojectionmatricesandKisthecameracalibrationmatrix.Inordertoestimate1,2,1and2,weneedtointroducetheepipolarconstraint.Fromequation( 8{6 ),wehave Thefundamentalmatrixisdenedas: Withtheabovemodel,wecouldestimatethefundamentalmatrixFviatheEight-pointalgorithm[ 111 ].Thenwecoulddecomposethefundamentalmatrixtorecovertheprojectionmatrices1and2andthe3Dstructure.Weonlygivethesolutionhere 113

PAGE 114

wheremeansequalityuptoascalefactorand WiththeassumptionthatKisconstant,wecouldestimatetheunknowns,Kand,withagradientdecentoptimizationalgorithm.Inordertoobtainauniquesolution,wealsoassumethatthesceneisgenericandthecameramotionisrichenough. 63 127 ],trytoregisterthetwoimagesbycomputingthehomographymatrixHbetweencorrespondingfeaturepoints.Thelimitofthisalgorithmisthattheyassumeallthepointsinthephysicalworldarecoplanarorapproximatelycoplanar.Theassumptionisnottruewithhigh-risescenes.Inordertomitigatethisproblem,weproposeanovelalgorithmwhichrstsegmentstheimagegeometricallyandthenperformtheregistrationtoeachregionwithdepthestimation. 114

PAGE 115

8.3 .Withtheassumptionthateachsegmentregionofthesceneisapproximatelycoplanarinthephysicalworld,wecouldeasilyestimatetheplanemodelandprojectthe3Dplaneontotheimageframes.Comparedwiththetraditionalassumptionthatthewholesceneiscoplanarinthephysicalworld,ourassumptionisvalidinmostcircumstances. Therearealotofalgorithmsfordataclustering.Themostfamoushard-clusteringalgorithmisk-means[ 128 ].Thek-meansalgorithmassignseachdatapointtotheclusterwhosecentroidisnearest.Here,weusethedistancetoa3Dplaneinthephysicalworldasthemeasurement.Foreachcluster,wecouldchoosetheplanethathasthesmallestsumofdistanceofallthedatapointsinthecluster.However,thedescentbasedlearningmethodssuerfromaseriouslimitation.Thenon-globaloptimaofthecostsurfacemayeasilyresultinginpoorlocalminimatotheabovemethods.Techniquesaddingpenaltytermstothecostfunctionfurtherincreasesthecomplexityofthecostsurfaceandworsenthelocalminimumproblem. Inthissection,wepresentanon-lineardeterministicannealingapproachtosolvethe3Dgeometricalttingproblem.ThealgorithmisrstintroducedinChapter 6 Theinputdataisgivenasa3Dpoint,xi=[xi,yi,zi]T.Withtheassumptionthatnearestdatapointsareonthesameplane,wecouldestimatethelocalplanemodel,Li=[ai,bi,ci]TofdatapointxianditsKnearestneighborpoints. 115

PAGE 116

Thenwerevisethedistortionfunctionasfollows, whereD1=di,jcalculatethettingerrorbetweenthedatapointandtheestimatedplane,andD2calculatethedierencebetweenthelocalestimatedplanemodelandtheclusterscaleestimatedplanemodel.D2isdenedasfollows: Afterthemapping,weapplydeterministicannealingalgorithmtopartitionthedataintoseveralclustersasfollows. wheregj=[aj,bj,cj]isthegeometricalsurfacemodelparametertobeestimated,DisthesumofsquareofgeometricalttingerrorandHistheentropyconstraint.WedeneD

PAGE 117

Toperformoptimizationweneedtofurtheranalyzeitsterms.Wecanrewriteequation( 8{20 )byapplyingthechainruleofentropyas NoticethatthersttermH(X)istheentropyofthesourceandisthereforeconstantwithrespecttotheclustergjandassociationprobabilitiesp(gjjxi).Thuswecanjustfocusontheconditionalentropy TheminimizationofFwithrespecttoassociationprobabilitiesp(gjjxi)givesrisetotheGibbsdistribution wherethenormalizationis ThecorrespondingminimumofFisobtainedbypluggingequation( 8{23 )backintoequation( 8{18 ) TominimizetheLagrangianwithrespecttotheclustermodelgj,itsgradientsaresettozeroyieldingthecondition 117

PAGE 118

whereA=[Xie1],i=1,...,mandp=[a,b,c]Tistheplaneparameter. Givenanarbitrarypointxi=[xi,yi]Tmeasuredinpixelsintherstcluster,wecouldestimateitsdepthscaleibysolvingthefollowingequation: wherex0i=[xi,yi,1]T,H11and1areestimatedinSection 8.3 .Inequation( 8{28 ),onlyiisunknownandwiththeconstraintonXiewithequation( 8{27 ),wecouldeasilygetthevalueofi. Then,with1=[I,0],wehaveXip=[i1xi,i1yi,i1,1].Fromequation( 8{6 ),wegettherelationbetweentwoimageprojectionpointsxi1andxi2asfollows: wherecxi20=[i2xi2,i2yi2,i2].Wecouldthengetthepositionofthecorrespondingpointxi2=[xi2,yi2]inthesecondimage. 118

PAGE 119

8.3 Inourexperiment,weregardtherstimage'slocalcoordinatesystemasworldcoordinatesystemsotherstimagecanbeviewedasareferenceimage.Thentherestoftheimagesareregisteredtothereferenceimage.WealsoappliedthealgorithmproposedbyDavisandKeck[ 63 ]toregistertheinputimagesforcomparisonpurpose. Figure 8-3 istheregistrationresultusingouralgorithmandFigure 8-4 istheoutputofthealgorithmproposedbyDavisandKeck[ 63 ].Figure 8-5 showsthedierenceimagebetweentheregisteredimageandtherstimageusingouralgorithmandFigure 8-6 showsthedierenceimagefromthealgorithmofDavisandKeck.Wecanseethatourresultcanmitigatetheparallaxproblemsincetheroofandwallcornersareregisteredcorrectly;onthecontrary,theregisteredimagebythealgorithmofDavisandKeckhasalotofartifactscausedbytheparallaxproblem.WealsoshowsomeregistrationresultsusingouralgorithminFigure 8-7 throughFigure 8-8 InordertofurthercompareouralgorithmtothealgorithmproposedbyDavisandKeck,wecomputetherootofmeansquarederrors(RMSE)oftheregistrationresultsfrombothalgorithms.Figure 8-9 showsthattheregistrationerrorofouralgorithmislessthan50%thanthatofthealgorithmproposedbyDavisandKeck. Theresultshowsthatourimageregistrationalgorithmcanmitigatetheparallaxproblembecausemostofthesceneisregisteredwithoutvibration,asopposedtotheregistrationresultsunderthealgorithmofDavisandKeckinwhichthehigh-risesceneinthesensedimagessignicantlymovedafterregistrationtothereferenceimages.ThereasonisthatthealgorithmofDavisandKeckassumesallthepointsintheimagesarecoplanar.Whilethisassumptionissatisedwhenthedistancebetweenthecameraand 119

PAGE 120

Finally,wewouldliketopointoutthatthealgorithmproposedbyDavisandKeck[ 63 ]assumesaplanarregistration.Theirschemewasdesignedforusewithhigh-altitudeaerialimagerywhereplanartransformationsarefairlygoodapproximations.Furthermore,theirschemeusesRANSACtoremovepoormatchingpointsduringthecomputation.Thiscanhelptodealwithsomedepthdiscontinuitiesthatmaybepresentinthehigh-altitudeaerialimages.Inourexperiments,thetestimagescontainsalient3Dscenes;theseimagesareoutofthedomainforthealgorithmofDavisandKeck.ThisisthereasonwhythealgorithmofDavisandKeckdoesnotperformwell. Ourfutureworksinclude: 129 ][ 130 ]givenavideosequence.Thereliabilityofthedepthestimatesiscrucialtodepth-basedregistrationalgorithm;therefore,thehighlyrobust3Dreconstructiontechniqueisrequiredtoimplementouralgorithm.Uptonow,mostrecentdepthrecoveryalgorithmsreportedintheliteratureclaimtorecoverconsistentdepthfromsomechallengingvideosequences[ 129 ][ 130 ].Wecanapplyormodifythisstate-of-the-artdepthmaprecoverymethodtodevelopdepth-basedimageregistrationalgorithm. 120

PAGE 121

Thepipelinefor2Dimageregistrationsystem. 121

PAGE 122

Thenewimageregistrationsystemscheme. Figure8-3. Ouralgorithmtestresult,inwhichthe88thframeisregisteredtotherstframe. 122

PAGE 123

ThetestresultofDavisandKeck'salgorithm,inwhichthe88thframeisregisteredtotherstframe. Figure8-5. Thedierenceimagebetweentheregistered88thimageandtherstimage(usingouralgorithm). 123

PAGE 124

Thedierenceimagebetweentheregistered88thimageandtherstimage(usingDavisandKeck'salgorithm). Figure8-7. The37thframeinthe`oldhousing'videosequence. 124

PAGE 125

Ouralgorithmtestresult,inwhichthe37thframeisregisteredtothe1stframe. Figure8-9. OuralgorithmtestresultcomparingtothatunderthealgorithmofDavisandKeck,inwhichallthe88framesareregisteredtothe1stframe. 125

PAGE 126

25 ].Forasingleframe,imagesegmentationisthetraditionalwaytoanalysisthescene[ 26 ].However,incasewehaveasequenceofimages,whentherearedierentmovingobjectsinthescene,orobjectsatdierentdepthswithaglobalcameramotion,motiondiscontinuitywilloccur.Inthiscase,motionofdierentobjectscanprovidemoreessentialinformationtounderstandthescene[ 27 ].Therefore,motionsegmentationisneededtodividetheframeofanimagesequenceintoregionsbelongingtodierentmotions. Motionsegmentationcouldbedirectlyappliedtomanyareas.Suchasvideocompression,videodatabasequerying,andsceneanalysis.MPEG-4standard[ 28 ]describesacontentbasedmanipulationofobjectsinimagesequences.Tocreateanobjectbasedscenerepresentation,itisnecessarytosegmentdierentobjectsinaframe.Sincebackgroundtypicallychangeslessthantheobjectsmotion,whichindicatesdierentcompressionrates,thissegmentationisbasedmoreonmotioninformationofthescene.Videoquerying[ 29 ]isanotherneweldwhichaimstoautomaticallyclassifyvideosequencesbasedontheircontent.Acommonvideoquerytaskrequiresretrievingalltheimagesinadatabasethathaveasimilarcontenttothequeryexampleimage.Motionsegmentationenablesanindexingschemethatusesthetrajectories,shapesandowvectorsoftheindependentlymovingobjectstoquerythesequencesinadatabase.Therefore,thesystemwillgiveamoreaccurateresult.Thedevelopmentofunmannedaerialvehicle[ 30 ]makesreconnaissancemucheasierwhichrequiressceneanalysistechnologytoidentifysuspiciousmilitaryvehiclesinavideosequence.Objectshavingdierentmovingvelocitiesordirectionsneedtobeidentiedandsegmentedforfurther 126

PAGE 127

Inthischapter,weproposeanovelapproachbasedonbothpurelyperpixelopticaloweld.Thenoveltyofourworkisthatweintroducecodinglengthasacriterioningroupmerging.TheoriginalalgorithmisrstproposedinHong'sworkwhichisusedtostaticimagecompressionandsegmentation.Basedontheexperimentsimulationsandresults,weprovethatusingcodinglengthcouldgreatlyimprovetheperformanceofmotioneldsegmentation.Anothernoveltyofthismethodisthatweproposeaheuristicapproachtolocateglobalmotionbasedonthemotionsegments. Theremainderofthischapterisorganizedasfollows.Insection 9.2 ,weintroduceopticaloweldanddiscussthelimitationofthismotionestimationalgorithm.Section 9.3 describesourcodinglengthbasedmotionsegmentationapproach.Section 9.4 describesourapproachonglobalmotionlocation.Section 9.5 showstheexperimentalresultsandanperformanceevaluationoftheproposedalgorithmandwiththepreviousapproaches.Finally,wedescribefutureworkinsection 9.6 Thetraditionalapproachforcomputingopticalowcanbeclassiedintothreecategories,namely,featurebased,correlationbasedandgradientbased.Amongalltheapproaches,gradientbasedalgorithmsreceiveaspecialinterestforitsmathematicalsimplicityandrelativelycomputationaleciency.Inthispaper,weuseagradientbasedopticalowestimationalgorithm,whichisrstproposedbyLucasandKanade[ 124 ],toestimatethemotioneldofanimagesequence. 127

PAGE 128

Withtheassumptionthatthemovementofanobjectissmallenough,theimageconstraintatI(x,y,t)canbeexpandedwithTaylorseries:I(x+x,y+y,t+t)=I(x,y,t)+@I whereH.O.T.standsforhighordertermswhichareignoredhere. Fromtheaboveequation,wecanachieve: whichresultsin whereVx,Vyarevelocitiesalongxandydirections,andIx,Iy,Itarethespacialandtemporalderivativesofthepixelatposition(x,y,t). Thisistheequationwhichisknownastheapertureproblemofopticalowalgorithms.Inordertosolvethisproblem,anadditionalconstraintisneeded.InLucasandKanade'ssolution[ 124 ],theyuseanon-iterativemethodwhichassumesalocallyconstantow.Withthisassumption,weareabletosolveanoverconstraintsystemofequations,whichgivestheresult: whereAistheconstantowinasmallwindowofsizemm. 128

PAGE 129

Figure 9-1 and 9-2 showtwoframesoftheinputimagesequence.Whilegure 9-3 showsthecalculatedopticaloweld.Forabettervisualeect,weonlydrawmotionvectorsforeach5by5blocks.Theactualmotioneldiscalculatedperpixel. 9.3.1MinimalDescriptionLengthCriterion Atraditionaldenitionofsegmentationistochooseaclassofmodelswhicheachsubsetissupposedtot.Thetypicalapproachistodecomposethemixtureofallmodelsintoindividualonessimultaneouslyorsubsequently.Variousapproacheshavebeenproposedtoresolvethisproblem,suchasK-meansclusteringalgorithmandEMalgorithm,etc. Intheproblemofmotioneldsegmentation,thenumberofsegmentsisunknown.Therefore,determiningthenumberofmodelsforthedatasetisnecessaryanditisverydicult.Inordertosolvethisproblem,weproposeanewapproachwhichisbasedonminimumdescriptionlength(MDC)criterion. Supposewehaveadataset2RMN,whichisasetofrandomsamplesfromamixtureofmodels.Theoptimalsegmentationofthedataisthepartitioning=1[2...[Nthattheoverallcodinglengthofthedataisminimalamongallpossible 129

PAGE 130

Ingeneral,thelengthfunctionischosenaccordingtotheoptimalShannoncoding.However,becausesegmentsofthemotionelddataismultivariate,alossydatacompressionviewpointmayhelptosegmentthenoisydata. Thenwehavetheexpectedtotalnumberofbitsrequiredtoencodethedataaccordingtotheabovesegmentation:Ls(,)=NXn=1L(n)+jnj(log2(jnj=M))=NXn=1tr(n+K) 2log2det(I+K Thesuperscript`s'indicatesthecodinglengthaftersegmentation,andidenotesthediagonalmatrixthatencodestheprobabilityofMvectorsingroupi. 130

PAGE 131

Analyzingtheoverallcodinglengthequationhelpsustondtheoptimalsolutioninabottomupmannerbymergingregionsofsegments.InHongetal.'spaper[ 131 ],adetailedproofisgivenandwewillnotrewriteithere. asafunctionin. Sincethenumberofgroupsisunknown,wehavetominimizeRs(,)overN2Z+.Frompreviousworks,weknowthatanygradientbaseddescentalgorithmreliesontheinitializationofdatasetinordertoconvergetoglobalminimum.Becausemotionelddoesnotnecessarilysatisfythisrequirement,itisquitediculttominimizethecodinglengthfunctiondirectly.Instead,weuseasteepest-descentalgorithmtominimizethelengthfunctionLs(,). ThealgorithmisgiveninAlgorithm1.IneachstepwechoosetwosubsetsofvectorsS1,S2suchthatbymergingthetwosubsets,decrementinthecodinglengthisthelargest.Whenthedimensionofthespaceisrelativelylow,greedyalgorithmsusuallyperformwell.However,whenthedimensionofthesubspacebecomeshigh,greedyalgorithmsdonotalwaysconvergetotheoptimalsolution. 131

PAGE 132

Generally,globalmotionregioncontainsthecornerregionsofthescene.Althoughitisnotalwaystrue,westillcouldutilizethispropertycombinedwiththevariancestatisticsofthemotioneldsegmentstoestimatetheglobalmotion.Foreachoutputsegmentation,wewillndoutthemotionregionwhichcontainsmorecornersofthescene.Iftherearemorethanonesegmentscontainingcornersofthescene,thesegmentwiththeminimalvarianceofinsidemotionvectorswillbeconsideredastheglobalmotion. Forthechosenglobalmotion,wewillcalculatetheaveragemotiondirectionandvelocitybasedonthemotionvectorsinsidetheregiontorepresenttheglobalmotion.ThealgorithmisgiveninAlgorithm 5 9.5.1MotionFieldSegmentation Figure 9-4 istherstframeoftheinputimagesequence.Figure 9-5 showsthecalculatedmotioneldoftheimagesequence`coastguard'.Figure 9-6 9-7 9-8 and 9-9 aretheoutputofourproposedalgorithm.Eachregionhasdierentmotionandthemotioneldiswellpartitioned. 9-6 andFigure 9-7 132

PAGE 133

Thesecondframeoftheinputimagesequence. couldnotbeglobalmotionsincetheydonotcontaincornerregionsofthescene.OnlyFigure 9-8 andFigure 9-9 containfourcorners.Therefore,weonlyconsiderthemasgroundmotion.Aftercalculatingthevarianceofmotionvectorsseparately,wecanconcludethattheregionofFigure 9-8 representstheglobalmotionduetoasmallmotionvariance. Becausethesegmentationalgorithmisformotionvectoreld,theperformanceofouralgorithmislimitedbytheaccuracyofopticaloweldcalculatedfrominputimagesequence.Inordertoimprovetheperformance,thetextureinformationcouldbeutilizedtosegmentthemotioneldmoreaccurately.Furthermore,wecanstudymoreinherenttemporalpropertiesoftheglobalmotiontohelpsceneinterpretation. 133

PAGE 134

Thefourthframeoftheinputimagesequence. Figure9-3. Theopticaloweldoftheinputimagesequence. Figure9-4. Therstframeofimagesequence`Coastguard'. 134

PAGE 135

Theopticaloweldofimagesequence`Coastguard'. Figure9-6. The`ship'segmentin`Coastguard'. Figure9-7. The`boat'segmentin`Coastguard'. 135

PAGE 136

The`land'segmentin`Coastguard'. Figure9-9. The`river'segmentin`Coastguard'. 136

PAGE 137

137

PAGE 138

Inthischapter,weproposeanewtrackingalgorithmbasedonbothtemplatetrackingandsilhouettetracking.Thealgorithmattemptstoadequatelytrackmultipleobjectsofarbitraryshapeinanimagesequencethatexperiencescameramotion.Inordertosuccessfullyestimatethemotiontrajectory,werstsegmenttheimagetogenerateabinaryobjectmask,andthentrackthefeaturesinsidethemask.ToovercomethelimitationofthetraditionalKLTtracker,weproposeanoveltrajectoryestimationmethodbasedonaweightingfunctionoftrackedfeaturemotionvectors. Thischapterisorganizedasfollows.Section 10.1 reviewspriorartsinobjecttracking.Insection 10.2 ,weintroduceourtrackingsystem.Section 10.3 presentstheobjectdetectorandSection 10.4 discussesthetrajectoryestimationprocess.TheexperimentalresultsareshowninSection 10.5 andSection 10.6 drawstheconclusion. 10-1 showsthesystemscheme. 138

PAGE 139

Aftertargetdetection,wemodelthemovingobjectwithspecialimagepropertieswithinthemaskregion.Anypropertycouldbeusedtorepresenttheobject,suchasedges,silhouette,colors,andprimitiveinformation.Inouralgorithm,weusethetraditionalKLTfeaturedetectortoselectinitialfeatures.Asweknow,KLTfeaturesareinvarianttoanetransformation,whichisabletoapproximatetheglobalmotioncausedbycameramovement. AlthoughweuseKLTdetectortoselectfeatures,weproposeacompletelydierentfeaturetrackingandupdatingalgorithmcomparedtothetraditionalKLTtracker.ThetraditionalKLTtrackerselectsfeaturesinthewholeimageandtracksallfeaturesinthesameway,i.e.,nofeatureismoreimportantthanothers.Inouralgorithm,eachfeatureistreateddierentlyaccordingtoitstrackingperformance.ThishelpstoachievebettertrackingperformancecomparedtothetraditionalKLTtracker.Inouralgorithm,thetrackedfeaturesareevaluatedaftertrackedateachframe,andthe\bad"featureswillberemovedandnewfeatureswillbereselectedfromtherestofobjectmaskarea.Wealsoproposeaweightingfunctionfortrajectoryestimation,whichconsidersboththequalityofthefeatureandtheconsistenceofthetrackingresult.Inthisway,themotionofthefeaturescouldbetterrepresentsthemotionoftheobject. 139

PAGE 140

Ingeneral,backgroundsubtractionisabletocompensateforlightingchangesandbackgroundclutter,anditiscomputationallyecient.However,moststate-of-the-artbackgroundmodelingmethodsaredesignedforimagesequencesfromxedcameras.Imagesegmentationmethodsareabletopartitiontheimageintoperceptuallysimilarregions,butthecriteriaforagoodpartitionandeciencyaretwoproblemsthatimagesegmentationalgorithmsneedtoaddress.Thedrawbackofsupervisedlearningmethodsisthattheyusuallyrequirealargecollectionofsamplesfromeachobjectclassandthesamplesmustbemanuallylabeled. Theobjectiveofourobjectdetectionalgorithmistondthelocationsofmultiplemovingtargetsintherstfewframesfromanon-stationarycamera.Sinceimagesegmentationonlyutilizesspatialcorrelationofasingleimage,itishardtodetecttheobjectregionduetothedierenttypesofobjecttobetracked.Weproposeanimagesegmentationalgorithmwhichconsidersbothspatialandtemporalinformationfromtherstfewframes.Ouralgorithmobtainsthetemporalinformationbycomputingopticalowfromtherstfewimages.Thisallowsthealgorithmtomodelthemotionuniformityinsidearegionofinterest.Theoutputofthealgorithmincludesseveralbinarymasks.Eachbinarymaskrepresentatargettobetracked.Thesegmentationprocesscanbe 140

PAGE 141

10{1 ). Thenalsegmentationresultdependsontwoterms,S(x,I1)andT(x,I1,I2).S(x,I1)isabinarymaskcomputedfromprimitivecorrelationintherstframeandT(x,I1,I2)representstheuniformityofthemotioneldbyopticalowfromthersttwoframes.isaparametertoadjusttheimportancebetweentemporalandspatialinformation.Whenincreases,thealgorithmgivesmoreandmoreweightonthetemporalinformationandwhendecreases,thealgorithmcaresmoreaboutthespatialinformation. Computingbinarymasksusingspatialcorrelationissimpleandcomputationalecient.Therearetwostepstoobtainthemask:edgedetectionandmorphologicalconnection.WeuseCannyedgedetectorforedgedetection,andweassumethattheobjecttobetrackedisdominantintheimageplane.Therefore,wecanremovethetrivialedgesbysettingthreshold.Wenoticethatthedetectededgesarediscontinuousandcannotbedirectlyusedtorepresentthetarget.Therefore,weneedtoconnecttheedgesandndaclosingboundaryofthetarget.Toservethispurpose,weusemathematicalmorphologywhichisatheoreticalmodelbasedonlatticetheoryandtopology.Morphologicalimageprocessingisgenerallybuiltonshiftinvariantoperators,whichisbasedonMinkowskiaddition.Therearefourbasicoperatorsinmorphologicalimageprocessing:opening,closing,erosion,anddilation.Inouralgorithm,weusedilationandclosingoperators.Weapplydilationtothedetectededgesinordertostrengthentheedgesaswellasconnecttheadjacentedges.Asubsequentclosingoperationistoremovethesmallholesinsidetheobjectmask. Afteredgedetectionandmorphologicaloperations,thedominantobjectismarkedwithabinarymask.However,someareasinthebackgroundwithrichtexturemayalsobemarkedasatarget,duetothefactthatrichtexturecontainsmanyedges.Tosolvethisproblem,weneedtocomputeT(x,I1,I2)tohelpremovingthefalsedetectedobjects.The 141

PAGE 142

124 ],whichcomputesopticalowusingpartialderivativeswithrespecttospatialandtemporalcoordinates.Theimageconstraintequationisgivenas: RemovingthehigherordertermsbyTaylorexpansion,theequationcanbewrittenas: Thecomputedopticaloweldwillbesegmentedusingsimilarmorphologicaloperationsandgeneratethenalobjectmask.ThesegmentationresultisgiveninFigure 10-2 andFigure 10-3 49 ].WithagivenimageI,KLTtrackerevaluatesthevariationofeachpixelinasmallneighborhood. 142

PAGE 143

10{5 ). ThefeaturepointsselectedbyKLTtrackerareinvarianttobothrotationandtranslation.Oncetheobjectmaskisdetected,wecoulduseKLTtrackertoselectandtrackfeaturepointsovermultipleframes. Ourproposedweightingfunctionisgiveninequation( 10{6 ). Therearethreetermsintheweightingfunction.WpisaGaussianweightconsideringthepositionofthepointofinterestintheobjectmask,expressedinequation( 10{7 ). 143

PAGE 144

10{5 .AccordingtothecriteriaofKLTfeaturedetector,thehigherthefeaturequality,themorereliablethefeaturewillbetracked. ThelasttermintheweightingfunctionisWc,whichstandsfortheconsistencyofthefeaturetrackedovermultipleframes.TheoriginalKLTfeaturetrackerisnotabletotrackthroughalongimagesequence,becausesomefeaturepointswillgetlostifthetrackercannotndacorrespondingoneaftertrackingeachframe.Inordertoovercomethelimitation,weproposeafeatureupdatingmechanismtondnewfeaturesonceanyfeaturegetslostandthenumberoffeaturestorepresentanobjectisconstant.Theconsistencyoffeatureisrepresentedbythenumberofframesthefeaturesurvives,whichmeansthatthelongerthefeaturestaysinthefeatureupdatingmechanismprocess,themorestableitis.Weuseexponentialfunctiontocalculatetheweightforfeatureconsistency. Intheaboveequation,nisthenumberofframesthefeaturehassurvived.Theconsistencyisespeciallyimportantwhenthereisobjectocclusion,whichwillbediscussedinthenextsection. 144

PAGE 145

Figure 10-4 andFigure 10-5 showtherstandlastframesinthetest`coastguard.cif'imagesequence.Figure 10-6 showsthetrackingresultoftwoobjectinthesequence.Therearetwoobjectstobetrackedinthissequence:alargershipandasmallboat.Ontherstframe,theboatisinthecenteroftheframeandtheshipcomeintotheframefromtheleft.Thecamerafollowstheboat,sotheboatstaysinthecenterandtheshiptravelsfromlefttoright.Afterseveralframes,thetwoobjectsmeetandtheshipisoccludedbytheboat.Inourexperiment,wearestillabletotrackbothobjectsaslongastheshipisnotfullyoccluded. 145

PAGE 146

Flowchartofourmultipleobjecttrackingsystem. Figure10-2. Segmentationofthe20thframefromthe`Coastguard'imagesequence. 146

PAGE 147

Segmentationresultofthe20thframeaftercorrection. Figure10-4. Therstframeinthe`Coastguard.cif'imagesequence. 147

PAGE 148

Thelastframeinthetestimagesequence. Figure10-6. Thetrackingresultofthetest`Coastguard'imagesequence.Each`+'indicatesthepositionoftheshipinoneframe. 148

PAGE 149

149

PAGE 150

Weproposeanon-lineardeterministicannealing(NDA)approachforgeometricttingin3Dspace.Duetothefactthatthe3Ddataislocalizedtoafewrelativelydenseclusters,wedesignakernelfunctiontomapthedatapointfromgeometricalspacetosurfacemodelspaceandapplydeterministicannealingtopartitionthefeaturespaceintoseparateregions.WefurtherusetheDNAmethodfor3Ddensereconstruction.Weusetheexistingtechniquesforfeaturecorrespondence,projectivereconstructionandself-calibrationtogetthesparsepointsreconstruction.Thenwesegmentthe3Dspaceintoseveralregionsbasedonthegeometricrelationship.Foreachregion,giventheintrinsicparametersfromself-calibration,wecanretrievethedepthinformationforeachpixelusingsurfacetting.Finally,weproposeanewstrategyofimageregistrationbyleveragingthedepthinformationvia3Ddensereconstruction.Thetraditionalimageregistrationalgorithmscannotsolveparallaxproblemduetotheirunderlyingassumptionthatthescenecanberegardedapproximatelyplanar.Ourmethodovercomesthisweaknessandachievesmorerobustregistrationresults.Ouralgorithmisattractivetotremendouspracticalapplications. Wealsoproposeanovelapproachtoestimatemodelparametersofallmotionsbasedonsegmentationofbothintensitymapandopticaloweld.Thenoveltyofourworkisthatweintroducecodinglengthasacriterioningroupmerging.Basedontheexperimentsimulationsandresults,weprovethatusingcodinglengthcouldgreatlyimprovetheperformanceofmotioneldsegmentation.Anothernoveltyisthatweproposeaheuristicapproachtolocateglobalmotionbasedonthemotionsegments. Anothercontributionofthisdissertationisthatweproposeanewtrackingalgorithmbasedonbothtemplatetrackingandsilhouettetracking.Thealgorithmattemptstoadequatelytrackmultipleobjectsofarbitraryshapesinanimagesequence.Inordertoaccuratelyestimatethetrajectory,werstgenerateabinaryobjectmaskandthenonly 150

PAGE 151

Ourfutureworksinclude: 151

PAGE 152

[1] E.Candes,\Compressivesampling,"inProceedingsoftheInternationalCongressofMathematicians,vol.1,2006,pp.1433{1452. [2] E.Candes,J.Romberg,andT.Tao,\Robustuncertaintyprinciples:Exactsignalreconstructionfromhighlyincompletefrequencyinformation,"IEEETransactionsonInformationTheory,vol.52,no.2,pp.489{509,2006. [3] E.CandesandJ.Romberg,\Quantitativerobustuncertaintyprinciplesandoptimallysparsedecompositions,"FoundationsofComputationalMathematics,vol.6,no.2,pp.227{254,2006. [4] D.Donoho,\Compressedsensing,"IEEETransactionsonInformationTheory,vol.52,no.4,pp.1289{1306,2006. [5] E.CandesandM.Wakin,\Peoplehearingwithoutlistening:Anintroductiontocompressivesampling,"IEEESignalProcessingMagazine,vol.25,no.2,pp.21{30,2008. [6] E.CandesandT.Tao,\Decodingbylinearprogramming,"IEEETransactionsonInformationTheory,vol.51,no.12,p.4203,2005. [7] E.vandenBergandM.Friedlander,\Probingtheparetofrontierforbasispursuitsolutions,"SIAMJournalonScienticComputing,vol.31,no.2,pp.890{912,2008. [8] J.TroppandA.Gilbert,\Signalrecoveryfromrandommeasurementsviaorthogonalmatchingpursuit,"IEEETransactionsonInformationTheory,vol.53,no.12,p.4655,2007. [9] D.NeedellandJ.Tropp,\CoSaMP:Iterativesignalrecoveryfromincompleteandinaccuratesamples,"AppliedandComputationalHarmonicAnalysis,vol.26,no.3,pp.301{321,2009. [10] D.DonohoandY.Tsaig,\Fastsolutionofl1-normminimizationproblemswhenthesolutionmaybesparse,"IEEETransactionsonInformationTheory,vol.54,no.11,pp.4789{4812,2008. [11] D.Donoho,Y.Tsaig,I.Drori,andJ.Starck,\Sparsesolutionofunderdeterminedlinearequationsbystagewiseorthogonalmatchingpursuit,"submittedforpublica-tion,2006. [12] M.Duarte,M.Wakin,andR.Baraniuk,\Fastreconstructionofpiecewisesmoothsignalsfromrandomprojections,"inProceedingsofSignalProcessingwithAdapta-tiveSparseStructuredRepresentations,vol.1,Rennes,France,2005,pp.1064{1070. [13] C.LaandM.Do,\Signalreconstructionusingsparsetreerepresentation,"inProceedingsofWaveletsXIatSPIEOpticsandPhotonics,vol.5914,SanDiego,California,2005,pp.273{283. 152

PAGE 153

S.Sarvotham,D.Baron,andR.Baraniuk,\Sudocodes{fastmeasurementandreconstructionofsparsesignals,"inProceedingsofIEEEInternationalSymposiumonInformationTheory,2006,pp.2804{2808. [15] J.Bioucas-DiasandM.Figueiredo,\AnewTwIST:Two-stepiterativeshrinkage/thresholdingalgorithmsforimagerestoration,"IEEETransactionsonImageprocessing,vol.16,no.12,p.2992,2007. [16] R.Chartrand,\Exactreconstructionofsparsesignalsvianonconvexminimization,"IEEESignalProcessingLetters,vol.14,no.10,p.707,2007. [17] E.Candes,M.Wakin,andS.Boyd,\Enhancingsparsitybyreweightedl1minimization,"JournalofFourierAnalysisandApplications,vol.14,no.5,pp.877{905,2008. [18] R.ChartrandandW.Yin,\Iterativelyreweightedalgorithmsforcompressivesensing,"inProceedingsofInternationalConferenceonAcoustics,Speech,andSignalProcessing,2008,pp.3869{3872. [19] T.BlumensathandM.Davies,\Iterativethresholdingforsparseapproximations,"JournalofFourierAnalysisandApplications,vol.14,no.5,pp.629{654,2008. [20] Y.TsaigandD.Donoho,\Extensionsofcompressedsensing,"SignalProcessing,vol.86,no.3,pp.549{571,2006. [21] E.CandesandJ.Romberg,\Practicalsignalrecoveryfromrandomprojections,"IEEETransactionsonSignalProcessing,vol.5674,p.76,2005. [22] L.Gan,\Blockcompressedsensingofnaturalimages,"inProceedingsofInterna-tionalConferenceonDigitalSignalProcessing,2007,pp.403{406. [23] Y.Zhang,S.Mei,Q.Chen,andZ.Chen,\Amultipledescriptionimage/videocodingmethodbycompressedsensingtheory,"inProceedingsofIEEEInternationalSymposiumonCircuitsandSystems,May2008,pp.1830{1833. [24] X.Wu,X.Zhang,andJ.Wang,\Model-guidedadaptiverecoveryofcompressivesensing,"inProceedingsofDataCompressionConference,2009,pp.123{132. [25] P.BouthemyandE.Francois,\Motionsegmentationandqualitativedynamicsceneanalysisfromanimagesequence,"InternationalJournalofComputerVision,vol.10,no.2,pp.157{182,1993. [26] N.PalandS.Pal,\Areviewonimagesegmentationtechniques,"PatternRecogni-tion,vol.26,no.9,pp.1277{1294,1993. [27] D.MurrayandB.Buxton,\Scenesegmentationfromvisualmotionusingglobaloptimization,"IEEETransactionsonPatternAnalysisandMachineIntelligence,pp.220{228,1987. 153

PAGE 154

D.LeGall,\MPEG:Avideocompressionstandardformultimediaapplications,"CommunicationsoftheACM,vol.34,pp.46{58,1991. [29] W.Aref,M.Hammad,A.Catlin,I.Ilyas,T.Ghanem,A.Elmagarmid,andM.Marzouk,\VideoqueryprocessingintheVDBMStestbedforvideodatabaseresearch,"inProceedingsofthe1stACMinternationalWorkshoponMultimediaDatabases,NewYork,NY,USA,2003,pp.25{32. [30] R.PlessandD.Jurgens,\Roadextractionfrommotioncuesinaerialvideo,"inProceedingsofthe12thannualACMInternationalWorkshoponGeographicInformationSystems,NewYork,NY,USA,2004,pp.31{38. [31] B.HornandB.Schunck,\Determiningopticalow,"ArticialIntelligence,vol.17,pp.185{203,1981. [32] D.MumfordandJ.Shah,\Optimalapproximationsbypiecewisesmoothfunctionsandassociatedvariationalproblems,"CommunicationsonPureandAppliedMathematics,vol.42,no.5,pp.577{685,1989. [33] D.CremersandC.Schnorr,\Motioncompetition:Variationalintegrationofmotionsegmentationandshaperegularization,"PatternRecognition,pp.472{480,2002. [34] E.H.AdelsonandJ.Y.A.Wang,\Representingmovingimageswithlayers,"IEEETransactionsonImageProcessing,vol.3,pp.625{638,1993. [35] G.D.Borshukov,G.Bozdagi,Y.Altunbasak,andA.M.Tekalp,\Motionsegmentationbymulti-stageaneclassication,"IEEETransactionsonImageProcessing,vol.6,pp.1591{1594,1997. [36] J.ShiandJ.Malik,\Normalizedcutsandimagesegmentation,"IEEETransactionsonPatternAnalysisandMachineIntelligence,vol.22,pp.888{905,1997. [37] I.K.SethiandR.Jain,\Findingtrajectoriesoffeaturepointsinamonocularimagesequence,"IEEETransactionsonPatternAnalysisandMachineIntelligence,vol.9,no.1,pp.56{73,1987. [38] K.RangarajanandM.Shah,\Establishingmotioncorrespondence,"inProceedingsofIEEEComputerSocietyConferenceonComputerVisionandPatternRecognition,Jun1991,pp.103{108. [39] C.J.Veenman,M.J.T.Reinders,andE.Backer,\Resolvingmotioncorrespondencefordenselymovingpoints,"IEEETransactionsonPatternAnalysisandMachineIntelligence,vol.23,no.1,pp.54{72,2001. [40] T.J.BroidaandR.Chellappa,\Estimationofobjectmotionparametersfromnoisyimages,"IEEETransactionsonPatternAnalysisandMachineIntelligence,vol.8,no.1,pp.90{99,1986. 154

PAGE 155

D.BeymerandK.Konolige,\Real-timetrackingofmultiplepeopleusingcontinuousdetection,"inProceedingsofInternationalConferenceonComputerVisionFrame-rateWorkshop,1999. [42] G.Kitagawa,\Non-gaussianstate-spacemodelingofnonstationarytimeseries,"JournaloftheAmericanStatisticalAssociation,vol.82,no.400,pp.1032{1041,1987. [43] Y.-L.ChangandJ.Aggarwal,\3dstructurereconstructionfromanegomotionsequenceusingstatisticalestimationanddetectiontheory,"inProceedingsoftheIEEEWorkshoponVisualMotion,Oct1991,pp.268{273. [44] C.RasmussenandG.D.Hager,\Probabilisticdataassociationmethodsfortrackingcomplexvisualobjects,"IEEETransactionsonPatternAnalysisandMachineIntelligence,vol.23,no.6,pp.560{576,2001. [45] D.Reid,\Analgorithmfortrackingmultipletargets,"IEEETransactionsonAutomaticControl,vol.24,no.6,pp.843{854,Dec1979. [46] C.Hue,J.-P.LeCadre,andP.Perez,\Sequentialmontecarlomethodsformultipletargettrackinganddatafusion,"IEEETransactionsonSignalProcessing,vol.50,no.2,pp.309{325,Feb2002. [47] H.Schweitzer,J.W.Bell,andF.Wu,\Veryfasttemplatematching,"inProceedingsofthe7thEuropeanConferenceonComputerVision,London,UK,2002,pp.358{372. [48] D.ComaniciuandP.Meer,\Meanshiftanalysisandapplications,"inThePro-ceedingsoftheSeventhIEEEInternationalConferenceonComputerVision,vol.2,1999,pp.1197{1203. [49] J.ShiandC.Tomasi,\Goodfeaturestotrack,"inIEEEConferenceonComputerVisionandPatternRecognition,1994,pp.593{600. [50] I.Haritaoglu,D.Harwood,andL.S.David,\W4:Real-timesurveillanceofpeopleandtheiractivities,"IEEETransactionsonPatternAnalysisandMachineIntelli-gence,vol.22,no.8,pp.809{830,2000. [51] J.Kang,I.Cohen,andG.Medioni,\Continuoustrackingwithinandacrosscamerastreams,"inIEEEComputerSocietyConferenceonComputerVisionandPatternRecognition,vol.1,June2003,pp.267{272. [52] K.SatoandJ.K.Aggarwal,\Temporalspatio-velocitytransformanditsapplicationtotrackingandinteraction,"ComputerVisionandImageUnderstanding,vol.96,no.2,pp.100{128,2004. [53] M.Bertalmo,G.Sapiro,andG.Randall,\Morphingactivecontours,"IEEETransactionsonPatternAnalysisandMachineIntelligence,vol.22,no.7,pp.733{737,2000. 155

PAGE 156

A.-R.Mansouri,\Regiontrackingvialevelsetpdeswithoutmotioncomputation,"IEEETransactionsonPatternAnalysisandMachineIntelligence,vol.24,no.7,pp.947{961,2002. [55] L.Brown,\Asurveyofimageregistrationtechniques,"ACMComputingSurveys(CSUR),vol.24,no.4,pp.325{376,1992. [56] B.ZitovaandJ.Flusser,\Imageregistrationmethods:Asurvey,"ImageandVisionComputing,vol.21,no.11,pp.977{1000,2003. [57] L.KitchenandA.Rosenfeld,\Gray-levelcornerdetection,"PatternRecognitionLetters,vol.1,no.2,pp.95{102,1982. [58] L.DreschlerandH.Nagel,\Volumetricmodeland3Dtrajectoryofamovingcarderivedfrommonoculartvframesequencesofastreetscene,"ComputerGraphicsandImageProcessing,vol.20,no.3,pp.199{228,1982. [59] W.ForstnerandE.Gulch,\Afastoperatorfordetectionandpreciselocationofdistinctpoints,cornersandcentresofcircularfeatures,"inProceedingsofIntercom-missionConferenceonFastProcessingofPhotogrammetricData(ISPRS),1987,pp.281{305. [60] J.Noble,\Findingcorners,"ImageandVisionComputing,vol.6,no.2,pp.121{128,1988. [61] A.GoshtasbyandG.Stockman,\Pointpatternmatchingusingconvexhulledges,"IEEETransactionsonSystems,Man,andCybernetics,vol.15,no.5,pp.631{636,1985. [62] G.Stockman,S.Kopstein,andS.Benett,\Matchingimagestomodelsforregistrationandobjectdetectionviaclustering,"IEEETransactionsonPatternAnalysisandMachineIntelligence,vol.4,pp.229{241,1982. [63] J.DavisandM.Keck,\OSUregistrationalgorithm,"InternalReport,OhioStateUniversity,USA. [64] A.Jerri,\TheShannonsamplingtheorem-Itsvariousextensionsandapplications:Atutorialreview,"inProceedingsoftheIEEE,vol.65,1977,pp.1565{1596. [65] E.CandesandT.Tao,\Near-optimalsignalrecoveryfromrandomprojections:Universalencodingstrategies?"IEEETransactionsonInformationTheory,vol.52,no.12,pp.5406{5425,2006. [66] M.Wakin,J.Laska,M.Duarte,D.Baron,S.Sarvotham,D.Takhar,K.Kelly,andR.Baraniuk,\Anarchitectureforcompressiveimaging,"inProceedingsofIEEEInternationalConferenceonImageProcessing,2006,pp.1273{1276. 156

PAGE 157

S.Sarvotham,D.Baron,andR.Baraniuk,\Measurementsvs.bits:Compressedsensingmeetsinformationtheory,"inProceedingsof44thAllertonConferenceonCommunication,ControlandComputing,2006. [68] M.Wainwright,\Information-theoreticboundsonsparsityrecoveryinthehigh-dimensionalandnoisysetting,"inProceedingsofIEEEInternationalSym-posiumonInformationTheory,2007,pp.961{965. [69] D.Donoho,\High-dimensionalcentrallysymmetricpolytopeswithneighborlinessproportionaltodimension,"DiscreteandComputationalGeometry,vol.35,no.4,pp.617{652,2006. [70] D.DonohoandJ.Tanner,\Neighborlinessofrandomlyprojectedsimplicesinhighdimensions,"ProceedingsoftheNationalAcademyofSciencesoftheUnitedStatesofAmerica,vol.102,no.27,p.9452,2005. [71] ||,\Countingfacesofrandomly-projectedpolytopeswhentheprojectionradicallylowersdimension,"AmericanMathematicalSociety,vol.22,no.1,pp.1{53,2009. [72] R.BaraniukandM.Wakin,\Randomprojectionsofsmoothmanifolds,"Founda-tionsofComputationalMathematics,vol.9,no.1,pp.51{77,2009. [73] J.Haupt,R.Castro,R.Nowak,G.Fudge,andA.Yeh,\Compressivesamplingforsignalclassication,"inProceedingsof40thAsilomarConferenceonSignals,SystemsandComputers,2006,pp.1430{1434. [74] M.Duarte,M.Davenport,M.Wakin,andR.Baraniuk,\Sparsesignaldetectionfromincoherentprojections,"inProceedingsofIEEEInternationalConferenceonAcoustics,SpeechandSignalProcessing,vol.6,2006,p.1. [75] A.Gilbert,M.Strauss,J.Tropp,andR.Vershynin,\Algorithmiclineardimensionreductioninthel1normforsparsevectors,"Arxivpreprintcs/0608079,2006. [76] ||,\Onesketchforall:Fastalgorithmsforcompressedsensing,"inProceedingsofthe39thannualACMsymposiumonTheoryofcomputing,2007,pp.237{246. [77] D.Takhar,J.Laska,M.Wakin,M.Duarte,D.Baron,S.Sarvotham,K.Kelly,andR.Baraniuk,\Anewcompressiveimagingcameraarchitectureusingoptical-domaincompression,"inProceedingsofSPIE,theInternationalSocietyforOpticalEngi-neering,vol.6065,2006,pp.43{52. [78] M.Lustig,D.Donoho,andJ.Pauly,\SparseMRI:TheapplicationofcompressedsensingforrapidMRimaging,"MagneticResonanceinMedicine,vol.58,no.6,pp.1182{1195,2007. [79] M.Rabbat,J.Haupt,A.Singh,andR.Nowak,\Decentralizedcompressionandpredistributionviarandomizedgossiping,"inProceedingsofthe5thInternationalConferenceonInformationProcessinginSensorNetworks,2006,pp.51{59. 157

PAGE 158

W.Wang,M.Garofalakis,andK.Ramchandran,\Distributedsparserandomprojectionsforrenableapproximation,"inProceedingsofthe6thInternationalConferenceonInformationProcessinginSensorNetworks,2007,pp.331{339. [81] S.Kirolos,J.Laska,M.Wakin,M.Duarte,D.Baron,T.Ragheb,Y.Massoud,andR.Baraniuk,\Analog-to-informationconversionviarandomdemodulation,"inProceedingsofIEEEDallas/CASWorkshoponDesign,Applications,IntegrationandSoftware,2006,pp.71{74. [82] J.Laska,S.Kirolos,Y.Massoud,R.Baraniuk,A.Gilbert,M.Iwen,andM.Strauss,\Randomsamplingforanalog-to-informationconversionofwidebandsignals,"inProceedingsofIEEEDallas/CASWorkshoponDesign,Applications,IntegrationandSoftware,2006,pp.119{122. [83] J.Laska,S.Kirolos,M.Duarte,T.Ragheb,R.Baraniuk,andY.Massoud,\Theoryandimplementationofananalog-to-informationconverterusingrandomdemodulation,"inProceedingsofIEEEInternationalSymposiumonCircuitsandSystems,2007,pp.1959{1962. [84] J.Ragheb,S.Kirolos,J.Laska,A.Gilbert,M.Strauss,R.Baraniuk,andY.Massoud,\Implementationmodelsforanalog-to-informationconversionviarandomsampling,"inProceedingsofthe50thIEEEInternationalMidwestSympo-siumonCircuitsandSystems,2007,pp.119{122. [85] R.Baraniuk,M.Davenport,R.DeVore,andM.Wakin,\TheJohnson-Lindenstrausslemmameetscompressedsensing,"ConstructiveApproximation,2007. [86] M.Elad,\Optimizedprojectionsforcompressedsensing,"IEEETransactionsonSignalProcessing,vol.55,no.12,p.5695,2007. [87] A.Cohen,W.Dahmen,andR.DeVore,\Compressedsensingandbestk-termapproximation,"AmericanMathematicalSociety,vol.22,no.1,pp.211{231,2009. [88] H.Rauhut,K.Schnass,andP.Vandergheynst,\Compressedsensingandredundantdictionaries,"IEEETransactionsOnInformationTheory,vol.54,no.5,pp.2210{2219,2008. [89] M.Figueiredo,R.Nowak,andS.Wright,\Gradientprojectionforsparsereconstruction:Applicationtocompressedsensingandotherinverseproblems,"IEEEJournalonSelectedTopicsinSignalProcessing,vol.1,no.4,pp.586{597,2007. [90] J.Starck,M.Elad,andD.Donoho,\Imagedecompositionviathecombinationofsparserepresentationsandavariationalapproach,"IEEETransactionsOnImageProcessing,vol.14,no.10,pp.1570{1582,2005. 158

PAGE 159

M.EladandA.Bruckstein,\Ageneralizeduncertaintyprincipleandsparserepresentationinpairsofbases,"IEEETransactionsonInformationTheory,vol.48,no.9,pp.2558{2567,2002. [92] X.Zhang,X.Wu,andF.Wu,\Imagecodingonquincunxlatticewithadaptiveliftingandinterpolation,"inProceedingsofthe2007DataCompressionConference,2007,pp.193{202. [93] L.Bregman,\Therelaxationmethodofndingthecommonpointofconvexsetsanditsapplicationtothesolutionofproblemsinconvexprogramming,"USSRComputationalMathematicsandMathematicalPhysics,vol.7,no.3,pp.200{217,1967. [94] W.Bajwa,J.Haupt,A.Sayeed,andR.Nowak,\Compressivewirelesssensing,"inProceedingsofthe5thInternationalConferenceonInformationProcessinginSensorNetworks,2006,p.142. [95] C.Qiu,W.Lu,andN.Vaswani,\Real-timedynamicmrimagereconstructionusingkalmanlteredcompressedsensing,"inIEEEInternationalConferenceonAcoustics,Speech,andSignalProcessing,LosAlamitos,CA,USA,2009,pp.393{396. [96] Z.FanandR.deQueiroz,\Identicationofbitmapcompressionhistory:JPEGdetectionandquantizerestimation,"IEEETransactionsonImageProcessing,vol.12,no.2,pp.230{235,2003. [97] V.Goyal,A.Fletcher,andS.Rangan,\Compressivesamplingandlossycompression,"IEEESignalProcessingMagazine,vol.25,no.2,pp.48{56,2008. [98] L.Rudin,S.Osher,andE.Fatemi,\Nonlineartotalvariationbasednoiseremovalalgorithms,"PhysicaD:NonlinearPhenomena,vol.60,no.1-4,pp.259{268,1992. [99] A.Likas,N.Vlassis,etal.,\Theglobalk-meansclusteringalgorithm,"PatternRecognition,vol.36,no.2,pp.451{461,2003. [100] K.Rose,\Deterministicannealingforclustering,compression,classication,regression,andrelatedoptimizationproblems,"ProceedingsoftheIEEE,vol.86,no.11,pp.2210{2239,1998. [101] B.Kulis,S.Basu,I.Dhillon,andR.Mooney,\Semi-supervisedgraphclustering:akernelapproach,"MachineLearning,vol.74,no.1,pp.1{22,2009. [102] S.Kirkpatrick,\Optimizationbysimulatedannealing:Quantitativestudies,"JournalofStatisticalPhysics,vol.34,no.5,pp.975{986,1984. [103] A.Rao,D.Miller,K.Rose,andA.Gersho,\Adeterministicannealingapproachforparsimoniousdesignofpiecewiseregressionmodels,"IEEETransactionsonPatternAnalysisandMachineIntelligence,vol.21,no.2,pp.159{173,1999. 159

PAGE 160

F.CamastraandA.Verri,\Anovelkernelmethodforclustering,"IEEETransac-tionsonPatternAnalysisandMachineIntelligence,pp.801{804,2005. [105] Z.Zhang,\Aexiblenewtechniqueforcameracalibration,"IEEETransactionsonPatternAnalysisandMachineIntelligence,vol.22,no.11,pp.1330{1334,2000. [106] C.Strecha,T.Tuytelaars,andL.VanGool,\Densematchingofmultiplewide-baselineviews,"inInternationalConferenceonComputerVision,vol.2,2003,pp.1194{1201. [107] M.LhuillierandL.Quan,\Aquasi-denseapproachtosurfacereconstructionfromuncalibratedimages,"IEEETransactionsonPatternAnalysisandMachineIntelligence,pp.418{433,2005. [108] H.Jin,S.Soatto,andA.Yezzi,\Multi-viewstereoreconstructionofdenseshapeandcomplexappearance,"InternationalJournalofComputerVision,vol.63,no.3,p.189,2005. [109] R.HartleyandA.Zisserman,Multipleviewgeometryincomputervision.CambridgeUniversityPress,2003. [110] E.TruccoandA.Verri,Introductorytechniquesfor3-Dcomputervision.PrenticeHall,1998. [111] Y.Ma,S.Soatto,andJ.Kosecka,Aninvitationto3-Dvision:fromimagestogeometricmodels.SpringerVerlag,2004. [112] H.Li,B.Adams,L.Guibas,andM.Pauly,\Robustsingle-viewgeometryandmotionreconstruction,"inACMSIGGRAPHAsia,2009,pp.1{10. [113] P.Beardsley,P.Torr,andA.Zisserman,\3Dmodelacquisitionfromextendedimagesequences,"inProceedingsofthe4thEuropeanConferenceonComputerVision,1996,pp.683{695. [114] A.FitzgibbonandA.Zisserman,\Automaticcamerarecoveryforclosedoropenimagesequences,"inProceedingsofthe5thEuropeanConferenceonComputerVision,1998,pp.311{326. [115] M.Pollefeys,R.Koch,andV.Gool,\Self-calibrationandmetricreconstructioninspiteofvaryingandunknowninternalcameraparameters,"InternationalJournalofComputerVision,pp.7{25,1998. [116] F.DevernayandO.Faugeras,\Automaticcalibrationandremovalofdistortionfromscenesofstructuredenvironments,"InvestigativeandTrialImageProcessing,vol.2567,pp.62{72,1995. [117] J.YagnikandK.Ramakrishnan,\Amodelbasedfactorizationapproachfordense3Drecoveryfrommonocularvideo,"inSeventhIEEEInternationalSymposiumonMultimedia,2005,p.4. 160

PAGE 161

V.Popescu,E.Sacks,andG.Bahmutov,\Interactivepoint-basedmodelingfromdensecolorandsparsedepth,"inEurographicsSymposiumonPoint-BasedGraphics,2004. [119] H.Chang,J.Moura,Y.Wu,K.Sato,andC.Ho,\Reconstructionof3DdensecardiacmotionfromtaggedMRsequences,"inIEEEInternationalSymposiumonBiomedicalImaging:NanotoMacro,2004,pp.880{883. [120] O.FaugerasandR.Keriven,\Completedensestereovisionusinglevelsetmethods,"inProceedingsofthe5thEuropeanConferenceonComputerVision,1998,p.393. [121] R.Koch,M.Pollefeys,andL.Gool,\Multiviewpointstereofromuncalibratedvideosequences,"inProceedingsofthe5thEuropeanConferenceonComputerVision,1998,p.71. [122] M.Pollefeys,R.Koch,M.Vergauwen,andL.VanGool,\Automatedreconstructionof3Dscenesfromsequencesofimages,"ISPRSJournalOfPhotogrammetryAndRemoteSensing,vol.55,no.4,pp.251{267,2000. [123] M.LhuillierandL.Quan,\Surfacereconstructionbyintegrating3Dand2Ddataofmultipleviews,"inProceedingsoftheNinthIEEEInternationalConferenceonComputerVision,2003,pp.1313{1320. [124] B.LucasandT.Kanade,\Aniterativeimageregistrationtechniquewithanapplicationtostereovision,"inInternationalJointConferenceonArticialIntelli-gence,vol.3,1981,p.3. [125] J.Barron,D.Fleet,andS.Beauchemin,\Performanceofopticalowtechniques,"InternationalJournalofComputerVision,vol.12,no.1,pp.43{77,1994. [126] M.FischlerandR.Bolles,\Randomsampleconsensus:Aparadigmformodelttingwithapplicationstoimageanalysisandautomatedcartography,"CommunicationsoftheACM,vol.24,no.6,pp.381{395,1981. [127] O.Mendoza,G.Arnold,andP.Stiller,\Furtherexplorationoftheobject-imagemetricwithimageregistrationinmind,"inProceedingsoftheSPIE,SymposiumonMultisensor,MultisourceInformationFusion:Architectures,Algorithms,andApplications,vol.6974,April2008,pp.5{12. [128] S.P.Lloyd,\LeastsquaresquantizationinPCM,"IEEETransactionsonInforma-tionTheory,vol.28,pp.129{137,1982. [129] G.Zhang,J.Jia,T.Wong,andH.Bao,\Recoveringconsistentvideodepthmapsviabundleoptimization,"inIEEEConferenceonComputerVisionandPatternRecognition,2008,pp.1{8. [130] G.Zhang,J.Jia,T.-T.Wong,andH.Bao,\Consistentdepthmapsrecoveryfromavideosequence,"IEEETransactionsonPatternAnalysisandMachineIntelligence,vol.31,no.6,pp.974{988,June2009. 161

PAGE 162

W.Hong,J.Wright,K.Huang,andY.Ma,\Multiscalehybridlinearmodelsforlossyimagerepresentation,"IEEETransactionsonImageProcessing,vol.15,no.12,p.3655,2006. 162

PAGE 163

BingHanwasborninAnyang,Henan,China.HegothisB.S.degreeinelectricalengineeringatPekingUniversity,Beijing,China,in2005.HereceivedthePh.D.degreeinElectricalandComputerEngineeringfromUniversityofFlorida,Gainesville,FLinAugust2010.Hisresearchinterestsincludeimageandvideocompression,compressivesensing,computervisionandvideoanalysis. 163