<%BANNER%>

Probabilistic Approaches to Image Registration and Denoising

Permanent Link: http://ufdc.ufl.edu/UFE0042349/00001

Material Information

Title: Probabilistic Approaches to Image Registration and Denoising
Physical Description: 1 online resource (240 p.)
Language: english
Creator: Rajwade, Ajit
Publisher: University of Florida
Place of Publication: Gainesville, Fla.
Publication Date: 2010

Subjects

Subjects / Keywords: bases, denoising, density, directional, estimation, filtering, hosvd, image, interpolant, interpolation, isocontours, isosurfaces, learning, orthonormal, probabilistic, probability, processing, random, restoration, singular, statistical, statistics, svd, tensor, transform, transformation, variable
Computer and Information Science and Engineering -- Dissertations, Academic -- UF
Genre: Computer Engineering thesis, Ph.D.
bibliography   ( marcgt )
theses   ( marcgt )
government publication (state, provincial, terriorial, dependent)   ( marcgt )
born-digital   ( sobekcm )
Electronic Thesis or Dissertation

Notes

Abstract: PROBABILISTIC APPROACHES TO IMAGE REGISTRATION AND DENOISING We present probabilistically driven approaches to two major applications in computer vision and image processing: image alignment (registration) and filtering of intensity values corrupted by noise. Some existing methods for these applications require the estimation of the probability density of the intensity values defined on the image domain. Most of the contemporary density estimation techniques employ different types of kernel functions for smoothing the estimated density values. These kernels are unrelated to the structure or geometry of the image. The present work chooses to depart from this conventional approach to one which seeks to approximate the image as a continuous or piecewise continuous function of the spatial coordinates, and subsequently expresses the probability density in terms of some key geometric properties of the image, such as its gradients and iso-intensity level sets. This framework, which regards an image as a signal as opposed to a bunch of samples, is then extended to the case of joint probability densities between two or more images and for different domains (2D and 3D). A biased density estimate that expressly favors the higher gradient regions of the image is also presented. These techniques for probability density estimation are used (1) for the task of affine registration of images drawn from different sensing modalities, and (2) to build neighborhood filters in the well-known mean shift framework, for the denoising of corrupted gray-scale and color images, chromaticity fields and gray-scale video. Using our new density estimators, we demonstrate improvement in the performance of these applications. A new approach for the estimation of the probability density of spherical data is also presented, taking into account the fact that the source of such data are commonly known or assumed to be Euclidean, particularly within the field of image analysis. We also develop two patch-based image denoising algorithms that revisit the old patch-based singular value decomposition (SVD) technique proposed in the seventies. Noise does not affect only the singular values of an image patch, but also severely affects its SVD bases leading to poor quality denoising if those bases are used. With this in mind, we provide motivation for manipulating the SVD bases of the image patches for improving denoising performance. To this end, we develop a probabilistic non-local framework which learns spatially adaptive orthonormal bases that are derived by exploiting the similarity between patches from different regions of an image. These bases act as a common SVD for the group of patches similar to any reference patch in the image. The reference image patches are then filtered by projection onto these learned bases, manipulation of the transform coefficients and inversion of the transform. We present or use principled criteria for the notion of similarity between patches under noise and manipulation of the coefficients, assuming a fixed known noise model. The several experimental results reported show that our method is simple and efficient, it yields excellent performance as measured by standard image quality metrics, and has principled parameter settings driven by statistical properties of the natural images and the assumed noise models. We term this technique the non-local SVD (NL-SVD) and extend it to produce a second, improved algorithm based upon the higher order singular value decomposition (HOSVD). The HOSVD-based technique filters similar patches jointly and produces denoising results that are better than most existing popular methods and very close to the state of the art technique in the field of image denoising.
General Note: In the series University of Florida Digital Collections.
General Note: Includes vita.
Bibliography: Includes bibliographical references.
Source of Description: Description based on online resource; title from PDF title page.
Source of Description: This bibliographic record is available under the Creative Commons CC0 public domain dedication. The University of Florida Libraries, as creator of this bibliographic record, has waived all rights to it worldwide under copyright law, including all related and neighboring rights, to the extent allowed by law.
Statement of Responsibility: by Ajit Rajwade.
Thesis: Thesis (Ph.D.)--University of Florida, 2010.
Local: Adviser: Rangarajan, Anand.
Local: Co-adviser: Banerjee, Arunava.
Electronic Access: RESTRICTED TO UF STUDENTS, STAFF, FACULTY, AND ON-CAMPUS USE UNTIL 2011-12-31

Record Information

Source Institution: UFRGP
Rights Management: Applicable rights reserved.
Classification: lcc - LD1780 2010
System ID: UFE0042349:00001

Permanent Link: http://ufdc.ufl.edu/UFE0042349/00001

Material Information

Title: Probabilistic Approaches to Image Registration and Denoising
Physical Description: 1 online resource (240 p.)
Language: english
Creator: Rajwade, Ajit
Publisher: University of Florida
Place of Publication: Gainesville, Fla.
Publication Date: 2010

Subjects

Subjects / Keywords: bases, denoising, density, directional, estimation, filtering, hosvd, image, interpolant, interpolation, isocontours, isosurfaces, learning, orthonormal, probabilistic, probability, processing, random, restoration, singular, statistical, statistics, svd, tensor, transform, transformation, variable
Computer and Information Science and Engineering -- Dissertations, Academic -- UF
Genre: Computer Engineering thesis, Ph.D.
bibliography   ( marcgt )
theses   ( marcgt )
government publication (state, provincial, terriorial, dependent)   ( marcgt )
born-digital   ( sobekcm )
Electronic Thesis or Dissertation

Notes

Abstract: PROBABILISTIC APPROACHES TO IMAGE REGISTRATION AND DENOISING We present probabilistically driven approaches to two major applications in computer vision and image processing: image alignment (registration) and filtering of intensity values corrupted by noise. Some existing methods for these applications require the estimation of the probability density of the intensity values defined on the image domain. Most of the contemporary density estimation techniques employ different types of kernel functions for smoothing the estimated density values. These kernels are unrelated to the structure or geometry of the image. The present work chooses to depart from this conventional approach to one which seeks to approximate the image as a continuous or piecewise continuous function of the spatial coordinates, and subsequently expresses the probability density in terms of some key geometric properties of the image, such as its gradients and iso-intensity level sets. This framework, which regards an image as a signal as opposed to a bunch of samples, is then extended to the case of joint probability densities between two or more images and for different domains (2D and 3D). A biased density estimate that expressly favors the higher gradient regions of the image is also presented. These techniques for probability density estimation are used (1) for the task of affine registration of images drawn from different sensing modalities, and (2) to build neighborhood filters in the well-known mean shift framework, for the denoising of corrupted gray-scale and color images, chromaticity fields and gray-scale video. Using our new density estimators, we demonstrate improvement in the performance of these applications. A new approach for the estimation of the probability density of spherical data is also presented, taking into account the fact that the source of such data are commonly known or assumed to be Euclidean, particularly within the field of image analysis. We also develop two patch-based image denoising algorithms that revisit the old patch-based singular value decomposition (SVD) technique proposed in the seventies. Noise does not affect only the singular values of an image patch, but also severely affects its SVD bases leading to poor quality denoising if those bases are used. With this in mind, we provide motivation for manipulating the SVD bases of the image patches for improving denoising performance. To this end, we develop a probabilistic non-local framework which learns spatially adaptive orthonormal bases that are derived by exploiting the similarity between patches from different regions of an image. These bases act as a common SVD for the group of patches similar to any reference patch in the image. The reference image patches are then filtered by projection onto these learned bases, manipulation of the transform coefficients and inversion of the transform. We present or use principled criteria for the notion of similarity between patches under noise and manipulation of the coefficients, assuming a fixed known noise model. The several experimental results reported show that our method is simple and efficient, it yields excellent performance as measured by standard image quality metrics, and has principled parameter settings driven by statistical properties of the natural images and the assumed noise models. We term this technique the non-local SVD (NL-SVD) and extend it to produce a second, improved algorithm based upon the higher order singular value decomposition (HOSVD). The HOSVD-based technique filters similar patches jointly and produces denoising results that are better than most existing popular methods and very close to the state of the art technique in the field of image denoising.
General Note: In the series University of Florida Digital Collections.
General Note: Includes vita.
Bibliography: Includes bibliographical references.
Source of Description: Description based on online resource; title from PDF title page.
Source of Description: This bibliographic record is available under the Creative Commons CC0 public domain dedication. The University of Florida Libraries, as creator of this bibliographic record, has waived all rights to it worldwide under copyright law, including all related and neighboring rights, to the extent allowed by law.
Statement of Responsibility: by Ajit Rajwade.
Thesis: Thesis (Ph.D.)--University of Florida, 2010.
Local: Adviser: Rangarajan, Anand.
Local: Co-adviser: Banerjee, Arunava.
Electronic Access: RESTRICTED TO UF STUDENTS, STAFF, FACULTY, AND ON-CAMPUS USE UNTIL 2011-12-31

Record Information

Source Institution: UFRGP
Rights Management: Applicable rights reserved.
Classification: lcc - LD1780 2010
System ID: UFE0042349:00001


This item has the following downloads:


Full Text

PAGE 1

PR OBABILISTICAPPROACHESTOIMAGEREGISTRATIONANDDENOISING By AJITRAJWADE ADISSERTATIONPRESENTEDTOTHEGRADUATESCHOOL OFTHEUNIVERSITYOFFLORIDAINPARTIALFULFILLMENT OFTHEREQUIREMENTSFORTHEDEGREEOF DOCTOROFPHILOSOPHY UNIVERSITYOFFLORIDA 2010

PAGE 2

c r 2010 AjitRajwade 2

PAGE 3

This thesisisbeingsubmittedwithafeelingofgratitudeformyparentsandbrother, whomIconsidertobemybestandclosestfriends. 3

PAGE 4

A CKNOWLEDGMENTS IwouldliketothankmyadvisorsDr.AnandRangarajanandDr.ArunavaBanerjee forsharingwithmetheirendlessenthusiasm,knowledge,expertiseandloveforthe subject.Ihavecometoadmirenotonlytheirintellectbutalsotheirunassumingand informalnature.Theytreattheirstudentslikefriends!AnandandArunavaaretwo individualswhoarefullofideas,andwhoarewillingtoselesslysharethoseideaswith everybody.Iamindebtedtothemforhavinggivenmethefreedomtopursuetowards myPh.D.aproblemthatIwaspassionateabout,namelyimagedenoising.Iamalso thankfultobothofthemforhavingplayedabigroleinencouragingstudent-student collaborationsonresearchproblemsofmutualinterest.Suchopen-mindednessand enthusiasmisrare! IwouldliketothankDr.JeffreyHo,Dr.BabaVemuriandDr.BrettPresnellfor servingonmycommittee.IdeeplyappreciateDr.Presnell'seffortsinreadingmythesis andsuggestingmeusefulchanges,andfordiscussionsonprobabilitydensityestimation techniques.AwordofsincereappreciationforseveralfacultymembersfromtheCISE department:Dr.AlperUngor,Dr.SanjayRanka,Dr.PeteDobbins,Dr.PaulGader andDr.TimDavis,withwhomIhaveworkedasteachingassistant;andforDr.Meera Sitharam,withwhomIparticipatedinourlocalchapterofSPICMACAY,anorganization forpromotionofIndianclassicalmusic. Gainesvillewouldhavebeenaboringplacewithoutmyroom-matesandlab-mates: VenkatakrishnanRamaswamy,SubhajitSengupta,KarthikGurumoorthy,Bhupinder Singh,AmitDhurandhar,GnanaSundarRajendiran,MilapjitSandhu,Ravneet SinghVohra,SayanBanerjee,AlokWhig,MeizhuLiu,TingChen,GuangChung, AngelosBarmpoutis,RitwikKumar,FeiWang,BingJian,SanthoshKodipaka,Esen Yuksel,WenxingYe,YuchenXie,DohyungSeo,SileHu,JasonChi,ShahedNejhum, ManuSethi,MohsenAli,AdrianPeter,NeilSmith,KarthikGopalkrishnan,Srikanth Subramaniam,andmanyothers.Theyallhelpedbuildalivelyenvironmentbothat 4

PAGE 5

home andinthelab.Iconsidermyselfluckytohavehadtworeallywonderfulfriends: VenkatakrishnanRamaswamy(hereatUF)andGurmanSinghGill(atMcGill),whohave beensuchgenuinewell-wishersallalong!IhavealsocometoadmireVenkat'sabilityto ask(innumerable:-))interestingquestionsonmattersbothtechnicalandnon-technical. Nowordscanbesufcienttothankmyparents,mybrotherandmygrandparents whoneverletmefeelthatIwasaloneonthislong,challengingandsometimes frustratingjourney.Thisthesiswouldhavebeenimpossiblewithouttheirsupport.I wishtoexpressmysincerestgratitudetotheSaswadkarandIyengarfamiliesback inPune,whohavebeenfriends,philosphersandguidesformyfamily,andwhohave helpedandsupportedusinjustsomany,manypricelessways! 5

PAGE 6

T ABLEOFCONTENTS page A CKNOWLEDGMENTS ..................................4 LISTOFTABLES ......................................10 LISTOFFIGURES .....................................12 ABSTRACT .........................................16 CHAPTER 1INTRODUCTION ...................................18 2PROBABILITYDENSITYWITHISOCONTOURSANDISOSURFACES ....21 2.1OverviewofExistingPDFEstimators .....................21 2.1.1TheHistogramEstimator ........................21 2.1.2TheFrequencyPolygon ........................22 2.1.3KernelDensityEstimators .......................22 2.1.4MixtureModels .............................24 2.1.5Wavelet-BasedDensityEstimators ..................25 2.2MarginalandJointDensityEstimation ....................26 2.2.1EstimatingtheMarginalDensitiesin2D ...............27 2.2.2RelatedWork ..............................29 2.2.3OtherMethodsforDerivation .....................29 2.2.4EstimatingtheJointDensity ......................30 2.2.5FromDensitiestoDistributions ....................33 2.2.6JointDensitybetweenMultipleImagesin2D ............35 2.2.7Extensionsto3D ............................36 2.2.8ImplementationDetailsforthe3Dcase ................38 2.2.9JointDensitiesbyCountingPointsandMeasuringLengths .....39 2.3ExperimentalResults:Area-BasedPDFsVersusHistogramswithSeveral Sub-PixelSamples ...............................42 3APPLICATIONTOIMAGEREGISTRATION ....................50 3.1EntropyEstimatorsinImageRegistration ..................50 3.2ImageEntropyandMutualInformation ....................53 3.3ExperimentalResults .............................55 3.3.1RegistrationofTwoimagesin2D ...................55 3.3.2RegistrationofMultipleImagesin2D .................58 3.3.3RegistrationofVolumeDatasets ...................58 3.4Discussion ...................................60 6

PAGE 7

4 APPLICATIONTOIMAGEFILTERING .......................70 4.1Introduction ...................................70 4.2Theory ......................................71 4.3ExtensionsofOurTheory ...........................75 4.3.1ColorImages ..............................75 4.3.2ChromaticityFields ...........................76 4.3.3Gray-scaleVideo ............................77 4.4LevelCurveBasedFilteringinaMeanShiftFramework ..........77 4.5ExperimentalResults .............................79 4.5.1Gray-scaleImages ...........................80 4.5.2TestingonaBenchmarkDatasetofGray-scaleImages .......80 4.5.3ExperimentswithColorImages ....................81 4.5.4ExperimentswithChromaticityVectorsandVideo ..........81 4.6Discussion ...................................82 5ARELATEDPROBLEM:DIRECTIONALSTATISTICSINEUCLIDEANSPACE 95 5.1Introduction ...................................95 5.2Theory ......................................96 5.2.1ChoiceofKernel ............................96 5.2.2UsingRandomVariableTransformation ...............97 5.2.3ApplicationtoKernelDensityEstimation ...............99 5.2.4MixtureModelsforDirectionalData ..................101 5.2.5PropertiesoftheProjectedNormalEstimator ............103 5.3EstimationoftheProbabilityDensityofHue .................104 5.4Discussion ...................................107 6IMAGEDENOISING:ALITERATUREREVIEW ..................110 6.1Introduction ...................................110 6.2PartialDifferentialEquations .........................111 6.3SpatiallyVaryingConvolutionandRegression ................113 6.4Transform-DomainDenoising .........................116 6.4.1ChoiceofBasis .............................117 6.4.2ChoiceofThresholdingSchemeandParameters ..........118 6.4.3MethodforAggregationofOverlappingEstimates ..........119 6.4.4ChoiceofPatchSize ..........................119 6.5Non-localTechniques .............................121 6.6UseofResidualsinImageDenoising .....................124 6.6.1ConstraintsonMomentsoftheResidual ...............124 6.6.2AddingBackPortionsoftheResidual .................125 6.6.3UseofHypothesisTests ........................125 6.6.4ResidualsinJointRestorationofMultipleImages ..........126 6.7DenoisingTechniquesusingMachineLearning ...............127 6.8CommonProblemswithContemporaryDenoisingTechniques .......129 7

PAGE 8

6.8.1 ValidationofDenoisingAlgorithms ..................129 6.8.2AutomatedFilterParameterSelection ................131 7BUILDINGUPONTHESINGULARVALUEDECOMPOSITIONFORIMAGE DENOISING .....................................132 7.1Introduction ...................................132 7.2MatrixSVD ...................................133 7.3SVDforImageDenoising ...........................133 7.4OracleDenoiserwiththeSVD .........................134 7.5SVD,DCTandMinimumMeanSquaredErrorEstimators .........136 7.5.1MMSEEstimatorswithDCT ......................136 7.5.2MMSEEstimatorswithSVD ......................138 7.5.3ResultswithMMSEEstimatorsUsingDCT ..............139 7.5.3.1Syntheticpatches ......................139 7.5.3.2Realimagesandalargepatchdatabase .........139 7.5.4ResultswithMMSEEstimatorsUsingSVD ..............140 7.5.4.1Syntheticpatches ......................140 7.5.4.2Realimagesandalargepatchdatabase .........141 7.6FilteringofSVDBases .............................142 7.7NonlocalSVDwithEnsemblesofSimilarPatches ..............143 7.7.1ChoiceofPatchSimilarityMeasure ..................147 7.7.2ChoiceofThresholdforTruncationofTransformCoefcients ....149 7.7.3OutlineofNL-SVDAlgorithm .....................150 7.7.4AveragingofHypotheses .......................150 7.7.5VisualizingtheLearnedBases ....................150 7.7.6RelationshipwithFourierBases ....................151 7.8ExperimentalResults .............................152 7.8.1DiscussionofResults .........................153 7.8.2ComparisonwithKSVD ........................153 7.8.3ComparisonwithBM3D ........................154 7.8.4ComparisonofNon-LocalandLocalConvolutionFilters ......156 7.8.5Comparisonwith3D-DCT .......................157 7.8.6ComparisonwithFixedBases .....................157 7.8.7VisualComparisonoftheDenoisedImages .............158 7.9SelectionofGlobalPatchSize ........................159 7.10DenoisingwithHigherOrderSingularValueDecomposition ........160 7.10.1TheoryoftheHOSVD .........................160 7.10.2ApplicationofHOSVDforDenoising .................161 7.10.3OutlineofHOSVDAlgorithm .....................162 7.11ExperimentalResultswithHOSVD ......................164 7.12ComparisonofTimeComplexity .......................164 8

PAGE 9

8 AUTOMATEDSELECTIONOFFILTERPARAMETERS .............200 8.1Introduction ...................................200 8.2LiteratureReviewonAutomatedFilterParameterSelection ........201 8.3Theory ......................................202 8.3.1IndependenceMeasures ........................202 8.3.2CharacterizingResidual`Noiseness' .................204 8.4ExperimentalResults .............................206 8.4.1ValidationMethod ............................207 8.4.2ResultsonNL-Means .........................208 8.4.3EffectofPatchSizeontheKSTest ..................209 8.4.4ResultsonTotalVariation .......................210 8.5DiscussionandAvenuesforFutureWork ..................210 9CONCLUSIONANDFUTUREWORK .......................220 9.1ListofContributions ..............................220 9.2FutureWork ...................................221 9.2.1TryingtoReachtheOracle .......................221 9.2.2BlindandNon-blindDenoising ....................221 9.2.3ChallengingDenoisingScenarios ...................222 APPENDIX ADERIVATIONOFMARGINALDENSITY ......................224 BTHEOREMONTHEPRODUCTOFACHAINOFSTOCHASTICMATRICES .226 REFERENCES .......................................227 BIOGRAPHICALSKETCH ................................240 9

PAGE 10

LIST OFTABLES T able page 2-1 Comparisonbetweendifferentmethodsofdensityestimationw.r.t.natureof domain,bias,speed,andgeometricnatureofdensitycontributions .......43 2-2TimingvaluesforcomputationofjointPDFsand L 1 normofdifferencebetween PDFcomputedbysamplingwiththatcomputedusingiso-contours;Number ofbinsis 128 128,sizeofimages 122 146 ...................45 3-1Averageandstd.dev.oferrorindegrees(absolutedifferencebetweentrue andestimatedangleofrotation)forMIusingParzenwindows ..........61 3-2Averagevalueandvarianceofparameters s and t predictedbyvariousmethods (32and64bins,noise =0.2);Groundtruth: =30, s = t = )Tj /T1_3 11.955 Tf (0.3 ......66 3-3Averagevalueandvarianceofparameters s and t predictedbyvariousmethods (32and64bins,noise =1 );Groundtruth: =30, s = t = )Tj /T1_3 11.955 Tf 9.29 0 Td (0.3 .......67 3-4Averageerror(absolutediff.)andvarianceinmeasuringangleofrotationusing MI,NMIcalculatedwithdifferentmethods,noise =0.05 ............67 3-5Averageerror(absolutediff.)andvarianceinmeasuringangleofrotationusing MI,NMIcalculatedwithdifferentmethods,noise =0.2 .............68 3-6Averageerror(absolutediff.)andvarianceinmeasuringangleofrotationusing MI,NMIcalculatedwithdifferentmethods,noise =1 ..............68 3-7Threeimagecase:anglesofrotationusingMMI,MNMIcalculatedwiththe iso-contourmethodandsimplehistograms,fornoisevariance =0.05,0.1,1 (Groundtruth 20 and 30 ) ..............................68 3-8Error(average,std.dev.)validatedover10trialswith LengthProb andhistograms for128bins; R referstotheintensityrangeoftheimage .............69 4-1MSEforlteredimagesusingourmethodandusingmeanshiftwithGaussian kernels ........................................84 4-2MSEforlteredimagesusingourmethod,usingmeanshiftwithGaussian kernelsandusingmeanshiftwithEpanechnikovkernels .............84 7-1Avg,maxandmedianerroronsyntheticpatchesfromFigure7-4withMAP andMMSEestimatorsforDCTbases .......................190 7-2Avg,maxandmedianerroronsyntheticpatchesfromFigure7-4withMAP andMMSEestimatorsforSVDbasisofthecleansyntheticpatch ........190 7-3PSNRvaluesfornoiselevel =5 onthebenchmarkdataset ..........191 7-4SSIMvaluesfornoiselevel =5 onthebenchmarkdataset ..........191 10

PAGE 11

7-5 PSNRvaluesfornoiselevel =10 onthebenchmarkdataset .........192 7-6SSIMvaluesfornoiselevel =10 onthebenchmarkdataset ..........192 7-7PSNRvaluesfornoiselevel =15 onthebenchmarkdataset .........193 7-8SSIMvaluesfornoiselevel =15 onthebenchmarkdataset ..........193 7-9PSNRvaluesfornoiselevel =20 onthebenchmarkdataset .........194 7-10SSIMvaluesfornoiselevel =20 onthebenchmarkdataset ..........194 7-11PSNRvalues:NL-SVDversusDCTfornoiselevel =20 onthebenchmark dataset ........................................195 7-12PSNRvaluesfornoiselevel =25 onthebenchmarkdataset .........195 7-13SSIMvaluesfornoiselevel =25 onthebenchmarkdataset ..........196 7-14PSNRvaluesfornoiselevel =30 onthebenchmarkdataset .........197 7-15SSIMvaluesfornoiselevel =30 onthebenchmarkdataset ..........197 7-16PSNRvaluesfornoiselevel =35 onthebenchmarkdataset .........198 7-17SSIMvaluesfornoiselevel =35 onthebenchmarkdataset ..........198 7-18Patch-sizeselectionfor =20 ...........................199 8-1(NL-Means)Gaussiannoise 2 n =0.0001 .....................212 8-2(NL-Means)Gaussiannoise 2 n =0.0005 .....................215 8-3(NL-Means)Gaussiannoise 2 n =0.001 ......................215 8-4(NL-Means)Gaussiannoise 2 n =0.005 ......................216 8-5(NL-Means)Gaussiannoise 2 n =0.01 .......................216 8-6(NL-Means)Gaussiannoise 2 n =0.05 .......................217 8-7(NL-Means)Uniformnoisewidth=0.001 .....................217 8-8(NL-Means)Uniformnoisewidth=0.01 ......................218 8-9(TV)Gaussiannoise 2 n =0.0005 ..........................218 8-10(TV)Gaussiannoise 2 n =0.005 ..........................219 11

PAGE 12

LIST OFFIGURES Figure page 2-1 p ( ) / area betweenlevelcurvesat and + (i.e.regionwithreddots) .42 2-2(A)Intersectionoflevelcurvesof I 1 and I 2 : p ( 1 2 ) / areaofdarkblack regions.(B)Parallelogramapproximation:PDFcontribution=area( ABCD ) .............................................43 2-3(A)Areaofparallelogramincreasesasanglebetweenlevelcurvesdecreases (lefttoright).Levelcurvesof I 1 and I 2 areshowninredandbluelinesrespectively (B)Jointprobabilitycontributioninthecaseofthreeimages ..........43 2-4Aretinogram[1]anditsrotatednegative .....................44 2-5Followinglefttorightandtoptobottom,jointdensitiesoftheretinogramimages computedbyhistograms(using16,32,64,128bins)andbyourarea-based method(using16,32,64and128bins) ......................44 2-6Marginaldensitiesoftheretinogramimagecomputedbyhistograms[from(A) to(D)]andourarea-basedmethod[from(E)to(H)]using16,32,64and128 bins(row-wiseorder) .................................45 2-7Probabilitycontributionandgeometryofisocontourpairs ............46 2-8Splittingavoxel(A)into12tetrahedra,twooneachofthesixfacesofthevoxel; and(B)into24tetrahedra,fouroneachofthesixfacesofthevoxel ......46 2-9Countinglevelcurveintersectionswithinagivenhalf-pixel ............47 2-10Biasedestimatesin3D:(A)Segmentofintersectionofplanariso-surfaces fromthetwoimages,(B)Pointofintersectionofplanariso-surfacesfromthe threeimages(eachinadifferentcolor) .......................47 2-11Jointprobabilityplotsusing:(A)histograms,128bins,(B)histograms,256 bins,(C) LengthProb ,128binsand(D) LengthProb ,256bins ..........48 2-12PlotsofthedifferencebetweenthejointPDF(oftheimagesinsubgure[A]) computedbythearea-basedmethodandbyhistogrammingwith N s sub-pixel samplesversus log N s using(B) L 1 norm,(C) L 2 norm,and(D)JSD ......49 3-1GraphsshowingtheaverageerroranderrorstandarddeviationwithMIasthe criterionfor16,32,64,128binswithanoise 2f 0.05,0.2and 1 g ........62 3-2MIwith32and128binsforanoiselevelof0.05,0.2and1 ...........63 3-3MRslicesofthebrain(A)MR-PDslice,(B)MR-T1slicerotatedby20degrees, (C)MR-T2slicerotatedby30degrees .......................64 12

PAGE 13

3-4 MIcomputedusing(A)histogrammingand(B) LengthProb (plottedversus Y and Z );MMIcomputedusing(C)histogrammingand(D) 3DPointProb (plottedversus 2 and 3 ) ..............................64 3-5MR-PDandMR-T1slicesbeforeandafterafneregistration ..........65 4-1Imagecontourmapsinaneighborhood ......................83 4-2True,degradedanddenoisedimages .......................85 4-3True,degradedanddenoisedimages .......................86 4-4True,degradedanddenoisedimages .......................87 4-5True,degradedanddenoisedngerprintimagesforthreenoiselevels .....88 4-6Performanceplotonthebenchmarkdataset ....................89 4-7True,degradedanddenoisedcolorimages ....................90 4-8True,degradedanddenoisedcolorimages ....................91 4-9True,degradedanddenoisedcolorimages ....................92 4-10True,degradedanddenoisedcolorimages ....................93 4-11True,degradedanddenoisedframesfromavideosequence ..........94 5-1Aprojectednormaldistribution( ~ 0 =(1,0), 0 =10)andavon-Misesdistribution (~ 0 =(1,0), 0 = j ~ 0 j 2 0 = 0.01 ) ............................109 5-2Plotofprojectednormalandvon-Misesdensities .................109 6-1Mandrillimage:(A)withnonoise,(B)withnoiseof =10,(C)withnoiseof =20;thenoiseishardlyvisibleinthetexturedfurregion(viewedbestwhen zoomedinthepdfle) ................................131 7-1GlobalSVDFilteringontheBarbaraimage ....................166 7-2Patch-basedSVDlteringontheBarbaraimage .................167 7-3OraclelterwithSVD ................................168 7-4Fifteensyntheticpatches ..............................168 7-5ThresholdfunctionsforDCTcoefcientsof(A)thesixthand(B)theseventh patchfromFigure7-4 ................................169 7-6DCTlteringwithMAPandMMSEmethods ....................170 7-7DCTlteringwithMAPandMMSEmethods ....................171 13

PAGE 14

7-8 Thresholdfunctionsforcoefcientsof(A)thesixthand(B)theseventhpatch fromFigure7-4whenprojectedontoSVDbasesofpatchesfromthedatabase 172 7-9SVDlteringwithMAPandMMSEmethods ....................173 7-10MotivationforRobustPCA .............................173 7-11Barbaraimage,(A)referencepatch,(B)patchessimilartothereferencepatch (similaritymeasuredonnoisyimagewhichisnotshownhere),(C)correlation matrices(toprow)andlearnedbases .......................174 7-12Mandrillimage,(A)referencepatch,(B)patchessimilartothereferencepatch (similaritymeasuredonnoisyimagewhichisnotshownhere),(C)correlation matrices(toprow)andlearnedbases .......................175 7-13DCTbases(8 8 ). ..................................176 7-14Barbaraimage:(A)cleanimage,(B)noisyversionwith =20 ,PSNR=22, (C)outputofNL-SVD,(D)outputofNL-Means,(E)outputofBM3D1,(F)output ofBM3D2,(G)outputofHOSVD ..........................177 7-15Residualwith(A)NL-SVD,(B)NL-Means,(C)BM3D1,(D)BM3D2,(E)HOSVD 178 7-16Boatimage:(A)cleanimage,(B)noisyversionwith =20,PSNR=22,(C) outputofNL-SVD,(D)outputofNL-Means,(E)outputofBM3D1,(F)output ofBM3D2,(G)outputofHOSVD ..........................179 7-17Residualwith(A)NL-SVD,(B)NL-Means,(C)BM3D1,(D)BM3D2,(E)HOSVD 180 7-18Streamimage:(A)cleanimage,(B)noisyversionwith =20,PSNR=22, (C)outputofNL-SVD,(D)outputofNL-Means,(E)outputofBM3D1,(F)output ofBM3D2,(G)outputofHOSVD ..........................181 7-19Residualwith(A)NL-SVD,(B)NL-Means,(C)BM3D1,(D)BM3D2,(E)HOSVD 182 7-20Fingerprintimage:(A)cleanimage,(B)noisyversionwith =20,PSNR= 22,(C)outputofNL-SVD,(D)outputofNL-Means,(E)outputofBM3D1,(F) outputofBM3D2,(G)outputofHOSVD ......................183 7-21Residualwith(A)NL-SVD,(B)NL-Means,(C)BM3D1,(D)BM3D2,(E)HOSVD 184 7-22For =20,denoisedBarbaraimagewithNL-SVD(A)[PSNR=30.96]and DCT(C)[PSNR=29.92].Forthesamenoiselevel,denoisedboatimagewith NL-SVD(B)[PSNR=30.24]andDCT(D)[PSNR=29.95]. ...........185 7-23(A)Checkerboardimage,(B)Noisyversionoftheimagewith =20 ,(C) DenoisedwithNL-SVD(PSNR=34)and(D)DCT(PSNR=27).Zoominfor betterview. ......................................186 14

PAGE 15

7-24 AbsolutedifferencebetweentrueBarbaraimageanddenoisedimageproduced by(A)NL-SVD,(B)BM3D1,(C)BM3D2.Allthreealgorithmswererunonimage withnoise =20. ..................................187 7-25AzoomedviewofBarbara'sfacefor(A)theoriginalimage,(B)NL-SVDand (C)BM3D2.NotetheshockartifactsonBarbara'sfaceproducedbyBM3D2. .187 7-26ReconstructedimageswhenBarbara(withnoise =20 )isdenoisedwith NL-SVDrunonpatchsizes(A) 4 4 ,(B) 6 6,(C) 8 8 ,(D) 10 10 ,(E) 12 12,(F) 14 14 and(G) 16 16. ........................188 7-27ResidualimageswhenBarbara(withnoise =20 )isdenoisedwithNL-SVD runonpatchsizes(A) 4 4,(B) 6 6,(C) 8 8,(D) 10 10 ,(E) 12 12,(F) 14 14 and(G) 16 16 ...............................189 8-1PlotsofCC,MI, P andMSEonanimagesubjectedtoupto16000iterations oftotalvariationdenoising ..............................212 8-2Imagesproducedbylterswhoseparameterswerechosenbydifferentnoiseness measures .......................................213 8-3Imagesproducedbylterswhoseparameterswerechosenbydifferentnoiseness measures .......................................214 15

PAGE 16

Abstr actofDissertationPresentedtotheGraduateSchool oftheUniversityofFloridainPartialFulllmentofthe RequirementsfortheDegreeofDoctorofPhilosophy PROBABILISTICAPPROACHESTOIMAGEREGISTRATIONANDDENOISING By AjitRajwade December2010 Chair:AnandRangarajan Cochair:ArunavaBanerjee Major:ComputerEngineering Wepresentprobabilisticallydrivenapproachestotwomajorapplicationsin computervisionandimageprocessing:imagealignment(registration)andltering ofintensityvaluescorruptedbynoise. Someexistingmethodsfortheseapplicationsrequiretheestimationofthe probabilitydensityoftheintensityvaluesdenedontheimagedomain.Mostof thecontemporarydensityestimationtechniquesemploydifferenttypesofkernel functionsforsmoothingtheestimateddensityvalues.Thesekernelsareunrelatedto thestructureorgeometryoftheimage.Thepresentworkchoosestodepartfromthis conventionalapproachtoonewhichseekstoapproximatetheimageasacontinuousor piecewisecontinuousfunctionofthespatialcoordinates,andsubsequentlyexpresses theprobabilitydensityintermsofsomekeygeometricpropertiesoftheimage,such asitsgradientsandiso-intensitylevelsets.Thisframework,whichregardsanimage asasignalasopposedtoabunchofsamples,isthenextendedtothecaseofjoint probabilitydensitiesbetweentwoormoreimagesandfordifferentdomains(2Dand 3D).Abiaseddensityestimatethatexpresslyfavorsthehighergradientregionsof theimageisalsopresented.Thesetechniquesforprobabilitydensityestimationare used(1)forthetaskofafneregistrationofimagesdrawnfromdifferentsensing modalities,and(2)tobuildneighborhoodltersinthewell-knownmeanshiftframework, forthedenoisingofcorruptedgray-scaleandcolorimages,chromaticityeldsand 16

PAGE 17

g ray-scalevideo.Usingournewdensityestimators,wedemonstrateimprovementinthe performanceoftheseapplications.Anewapproachfortheestimationoftheprobability densityofsphericaldataisalsopresented,takingintoaccountthefactthatthesourceof suchdataarecommonlyknownorassumedtobeEuclidean,particularlywithintheeld ofimageanalysis. Wealsodeveloptwopatch-basedimagedenoisingalgorithmsthatrevisittheold patch-basedsingularvaluedecomposition(SVD)techniqueproposedintheseventies. Noisedoesnotaffectonlythesingularvaluesofanimagepatch,butalsoseverely affectsitsSVDbasesleadingtopoorqualitydenoisingifthosebasesareused.With thisinmind,weprovidemotivationformanipulatingtheSVDbasesoftheimagepatches forimprovingdenoisingperformance.Tothisend,wedevelopaprobabilisticnon-local frameworkwhichlearnsspatiallyadaptiveorthonormalbasesthatarederivedby exploitingthesimilaritybetweenpatchesfromdifferentregionsofanimage.These basesactasacommonSVDforthegroupofpatchessimilartoanyreferencepatch intheimage.Thereferenceimagepatchesarethenlteredbyprojectionontothese learnedbases,manipulationofthetransformcoefcientsandinversionofthetransform. Wepresentoruseprincipledcriteriaforthenotionofsimilaritybetweenpatchesunder noiseandmanipulationofthecoefcients,assumingaxedknownnoisemodel.The severalexperimentalresultsreportedshowthatourmethodissimpleandefcient, ityieldsexcellentperformanceasmeasuredbystandardimagequalitymetrics,and hasprincipledparametersettingsdrivenbystatisticalpropertiesofthenaturalimages andtheassumednoisemodels.Wetermthistechniquethenon-localSVD(NL-SVD) andextendittoproduceasecond,improvedalgorithmbaseduponthehigherorder singularvaluedecomposition(HOSVD).TheHOSVD-basedtechniquelterssimilar patchesjointlyandproducesdenoisingresultsthatarebetterthanmostexistingpopular methodsandveryclosetothestateofthearttechniqueintheeldofimagedenoising. 17

PAGE 18

CHAPTER 1 INTRODUCTION Imageanalysisisaourishingeldthathasmadegreatprogressinthepastfew decades.Techniquesfromimageanalysishavebeenemployedineldsasdiverseas medicine,mechanicalengineering,remotesensing,biometricidentication,pathology andcellbiology,molecularchemistryandlithography.Anincompletelistofthekey problemsthatcurrentresearchersintheeldareworkingon,includes(1)image inpainting,(2)imagedenoisingandrestorationundervariousdegradationmodelssuch asdefocusblurormotionblur,fogorhaze,rainetc.,(3)alignmentofimagesofanobject sensedfromdifferentviewpointspotentiallyfromdifferentsensingmodalities(calledas rigidorafneimageregistration),andpossiblywithnontrivialdeformationsoftheobject itself,especiallyinapplicationsinvolvingmedicalimagingorfacerecognition(calledas non-rigidregistration),(4)tomography,(5)imagefusionormosaicing,(6)segmentation ofimagesintocoherentpartsorsegments,and(7)objectrecognitionunderdifferent viewsorlightingconditions. Manyofthesetechniquesheavilyemploystatisticalorprobabilisticapproaches. Afundamentalcomponentofallsuchapproachesistheestimationoftheprobability densityfunction(hereafterreferredtoasthePDF)oftheintensityvaluesoftheimage denedatdifferentpointsontheimagedomain.Thereexistseveraltechniquesfor PDFestimationintheliterature.Acommoncomponentofallofthesetechniquesis theestimationoffrequencycountsofthedifferentvaluesoftheintensityfollowedby smoothingorinterpolationbetweenthesevalues,usingkernelfunctions,yieldinga smoothedPDFestimate.Thesekernelsarenotrelatedtothegeometryoftheimage inanymanner.Thisthesistakestheoppositeapproachbasedonactuallytakinginto accountthefactthattheimageisageometricobject(ora`signal'asopposedtoa `bunchofsamples')andinterpolatestheavailablesamplestocreateacontinuousimage representation,whichisusedinitselfforPDFestimation.Theuseoftheinterpolant 18

PAGE 19

produces asmoothedestimatethatobviatestheneedforakernelandcriticalkernel parameterssuchasthebandwidth.MoreoverthismethodofbuildingaPDFevolvesa clearrelationshipbetweenprobabilisticquantities(suchasthePDFitself)andgeometric entities(suchasthegradientsandthelevelsets).ThisestimatorisdiscussedinChapter 2,followingaliteraturereviewofcontemporaryPDFestimators.InChapters 3 and 4 respectively,thenewPDFestimatorisemployedfortwoapplications-imageregistration underafnetransformations,anddenoisingofvarioustypesofimagesaffectedprimarily byindependentandidenticallydistributednoise.Theformerapplicationconsiders imagesacquiredpossiblyunderdifferentlightingconditionsordifferentmodalitiessuch asMR-T1,MR-T2,MR-PD(threedifferentmagneticresonanceimagingmodalities).The proposedPDFestimatorproducesresultsthataremorerobustthanothertechniques underneintensityquantizationandunderimagenoise.Thedenoisingtechnique inChapter 4 (aninterpolantdrivenlocalneighborhoodmethodinthemean-shift framework)istestedongray-scaleimages,colorimages,chromaticityeldsand gray-scalevideo.Forgray-scaleandcolorimages,theproposedPDFestimator producesbetterdenoisingresultsevenwhentheneighborhoodforaveragingand thesmoothingparametersaresmall.InChapter 5,thethesisalsodiscussesarelated problemintheeldofspherical(ordirectional)statisticswherethesamplesarepoints onaunitsphere.Thesedataareusuallyobtainedassomefunctioncomputedfromthe originaldatawhichareusuallyknownorassumedtolieinEuclideanspace.Examples includechromaticityvectorsofcolorimageswhichareunit-normalizedversionsofthe red-green-blue(RGB)valuesoutputbyacamera.Inthiswork,anestimatorispresented whichdoesnotimposeakerneldirectlyontheunitvectors,butwhichusesexisting estimatorsintheoriginalEuclideanspacefollowingrandomvariabletransformation. Chapter 6 presentsadetailedoverviewofcontemporaryimagedenoising techniques.Inchapter 7,weproposeaprobabilistictechniquethatstartsoffbyrevisiting theimagesingularvaluedecomposition(SVD).Weperformexperimentswithglobal 19

PAGE 20

and localimageSVDandproposedifferentwaystomanipulatetheSVDbasesof noisyimagepatches,orthecoefcientsofimagepatcheswhenprojectedontothese bases.Wediscusstheinefcacyofsomeofthesemanipulations,butdemonstrate thatreplacementoftheimagepatchSVDbyacommonbasisthatrepresentsan ensembleofpatcheswhichareallsimilartoareferencepatch,yieldsexcellentltering performance.Inthistechnique,whichwecallthenon-localSVD(NL-SVD),adifferent basisisproducedateverypixel.Wepresentanotionofpatchsimilarityundernoise, whichmakesuseofthepropertiesofthenoisemodel.Theactuallteringisperformed atthepatchlevelbyprojectingthepatchesontothebasistunedforthatpatch,followed bysubsequentmodicationoftheprojectioncoefcients,andinversionofthetransform. Ourtechniqueisthussimple,elegantandefcientandityieldsperformancecompetitive withthecurrentstateoftheart.Wealsopresentasecondandimprovedalgorithmthat employsthehigher-ordersingularvaluedecomposition(HOSVD),anextensionofthe SVDtohigherordermatrices. Whiletheresearchonimagelteringhasbeenextensive,thereisverylittle literatureonautomatedestimationoftheparametersofthelteringalgorithms(i.e. withoutreferencetothetrue,cleanimagewhichisunknowninpracticaldenoising scenarios).InChapter 8,wepresentanewstatisticallydrivencriterionforautomated lterparameterselectionundertheassumptionthatthenoiseisi.i.d.withaloose lowerboundonitsvariance.Thecriterionmeasuresthestatisticalsimilaritybetween non-overlappingpatchesoftheresidualimage(thedifferencebetweenthenoisyand thedenoisedimage).Thecriterionisempiricallyseentocorrelatewellwithknown full-referencequalitymeasures(i.e.thosethatmeasuretheerrorbetweenthedenoised imageandthetrueimage).WetestthecriterioninconjunctionwiththeNLMeans algorithm[ 2 ]andthetotalvariationPDEforselectingthesmoothingparameterinthese methods. 20

PAGE 21

CHAPTER 2 PROBABILITYDENSITYWITHISOCONTOURSANDISOSURFACES 2.1OverviewofExistingPDFEstimators ThemostcommonlyusedPDFestimatorsincludethehistogram,thefrequency polygon,theParzenwindow(orkernel)densityestimator,theGaussianmixturemodel, andthemuchmorerecentwavelet-baseddensityestimator.Inthefollowing,webriey reviewkeypropertiesofeach.Thereviewmaterial,presentedhereforthesakeof completeness,isabriefsummaryofwhatisfoundinstandardtextbooksonthetopic suchas[ 3]and[4 ]. 2.1.1TheHistogramEstimator Thehistogram-baseddensityestimator ^ p (x ) foradensity p ( x ) isdenedasfollows: ^ p ( x )= F (b j +1 ) )Tj /T1_3 11.955 Tf 11.95 0 Td (F (b j ) nh (2) where (b j b j +1 ] denes abin-boundary, h denotesthebin-width, F (b k ) denotesthe numberofsampleswhosevalueislessthanorequalto b k and n isthetotalnumberof samples.Thehistogramestimatoristhesimplestandthemostpopularoneowingto itssimplicity.Howeverithasanumberofproblems.Firstly,theestimatesitsproduces arealwaysnon-differentiable,eventhoughtheunderlyingdensitymaybedifferentiable. Theestimateishighlysensitivetothechoiceofbinboundariesandmoreimportantly tothechoiceofthebin-width h .Usingahighvalueof h producesahighlybiased(or over-smoothed)estimate,whereasaverysmallvalueof h leadstotheproblemofvery highvariabilityoftheestimateforsmallchangesinthesamplevalues.Thistradeoffis anotherinstanceoftheclassicbias-variancedilemmainmachinelearning.Thespecic expressionsforthebiasandvarianceofthisestimatoraregivenasfollows(dueto[ 4 ]): Bias(^ p (x ))= h )Tj /T1_2 11.955 Tf 11.96 0 Td (2 x +2 b j 2 p 0 (x ) + O (h 2 )for x 2 (b j b j +1 ] (2) Variance (^ p (x ))= f (x ) nh + O ( 1 n ). (2) 21

PAGE 22

The expressionsclearlyindicatethequadraticincreaseinbiaswithincreasein h ,and theincreaseinvarianceinverselyproportionalto h .Alsoclearisthefactthatthebias problemismorepronouncedfordensitieswithhigherderivativevalues. Thequalityofadensityestimatorisoftengivenbyitsmeansquarederror(MSE) whichisgivenasfollowsforthehistogram(dueto[ 4 ]): MSE[^ p (x )]= Variance (^ p ( x ))+ Bias 2 (^ p (x )) (2) = f (x ) nh + Kp 0 ( x ) 2 + O ( 1 n ) + O ( h 3 ). (2) UponintegratingtheMSEacross x ,wegetthemeanintegratedsquareerror(MSE), whichisgivenas(dueto[ 4]): MISE[^ p (x )]= 1 nh + O ( 1 n ) + O (h 3 )+ h 2 R p 0 (x ) 2 dx 12 (2) The bin-widthwhichminimizestheMISEisshowntobe O (n )Tj /T1_4 7.97 Tf 6.59 0 Td (1=3 ) andinversely proportionalto R p 0 (x ) 2 dx ,leadingtoanasymptoticMISEvaluewhichis O (n )Tj /T1_4 7.97 Tf 6.59 0 Td (2= 3 ) [4].Thisindicatesthattheoptimalrateofconvergenceofahistogram-baseddensity estimatoris O (n )Tj /T1_4 7.97 Tf 6.59 0 Td (2=3 ). 2.1.2TheFrequencyPolygon Histogramsarebydenitionpiecewiseconstantdensityestimators.Afrequency polygonissimplyapiecewiselinearextensiontothesimplehistogramandisobtained bystraightforwardlinearinterpolationinbetweentheestimateddensityvaluesdenedat themidpointsofadjacentbins.ThisinnocuouschangeproducesanMISEvaluewitha smallerbiasterm( O (h 2 ) asopposedtotheearlier O (h )).Theanalysisin[ 4]whichuses thebin-widthvaluethatoptimizestheMISE,indicatesanimprovedconvergencerateof O (n )Tj /T1_4 7.97 Tf 6.59 0 Td (4= 5 ) asopposedtotheearlier O ( n )Tj /T1_4 7.97 Tf (2 =3 ). 2.1.3KernelDensityEstimators Toalleviatethenon-differentiabilityofthehistogramandthefrequencypolygon, kerneldensityestimatorsbuildadifferentiablekernelcenteredateverysamplepoint. 22

PAGE 23

The estimatethusobtainedisgivenasfollows: ^ p (x )= 1 nh n X i =1 K ( x )Tj /T1_2 11.955 Tf 11.96 0 Td (x i h ) (2) where n is thenumberofsamplesand h isthebandwidth. K (.) iscalledasthekernel functionwhichisdenedtosatisfythefollowingconditions: Z K ( x )dx =1 (2) Z xK ( x )dx =0 (2) Z x 2 K (x )dx = 2 K > 0. (2) Thepropertiesofthekerneldensityestimatorareasfollows: Bias [^ p (x )]= h 2 2 K p 00 (x ) 2 + O (h 4 ) (2) V ariance [^ p (x )]= f (x )R ( K ) nh + O ( 1 n ) (2) MISE[ ^ p (x )]= O ( 1 nh ) + O (h 4 ). (2) TheoptimalMISE(correspondingtothevalueof h thatoptimizestheMISE)isshown in[ 4 ]tobe O ( n )Tj /T1_5 7.97 Tf (4 =5 ),indicatingasuperiorconvergenceoverhistograms,andhaving theaddedmeritofdifferentiabilityoverfrequencypolygons.Thecommonchoicesof thekernelfunctionincludetheGaussianandtheEpanechnikov.Thelatterisproved tobetheonewhichproducesthebestasymptoticMISE,thoughtheGaussianand manyotherknownkernelshavebeenprovedtobealmostasgood.Thisleadstothe conclusionthatatleastasymptotically,thechoiceofakernelisnotamajorissuein densityestimation.Thesmall-sample(i.e.non-asymptotic)analysisastowhichisthe bestkernelhasnotbeenpresentedhowever,atleasttotheauthor'sknowledge,and hencethekernelchoicewillhaveadistincteffectwhenalimitednumberofsamples areavailable.Moreover,saddledwiththeadvantagesmentionedearlier,aretwomore demerits.Therstoneisthatthechoiceofbandwidth h isagainquitecrucial,witha 23

PAGE 24

large h producing ahighbiasandasmall h producingahighvariance.Also,asper[ 3 ] (Section3.3.2),theidealwidthvalueforminimizingthemeanintegratedsquarederror betweenthetrueandestimateddensityisitselfdependentuponthesecondderivative ofthe(unknown)truedensity.Thisresultthereforedoesnotgiveanyindicationtoa practitioneraboutwhatthetruebandwidthshouldbe.Hence,thetypicalmethodto estimateabandwidthisa K -foldcross-validationbasedapproachwhichturnsouttobe bothcomputationallyexpensiveandquiteerror-prone.Secondly,inmanyapplications, thedomainisbounded.However,theestimatesproducedbythismethodyieldfalse valuesontheboundaryofsuchdomainsleadingtolargelocalizederrors(especiallyif kernelswithunboundedsupportareused). 2.1.4MixtureModels Themixturemodelapproachtodensityestimationisalsoalinearsuperpositionof kernels,wherethenumberofkernels M isnowtreatedasamodelingparameter[ 5]and isusuallymuchlessthanthetotalnumberofsamples n .Thealgebraicexpressionfor thesameisgivenasfollows: ^ p (x )= M X j =1 p (x jj )P (j ) (2) wherethecoefcients fP (j ) g arecalledthemixingparametersandaretheprior probabilitiesthatadatapointwasdrawnfromthe j th component,while p (x jj ) isthe conditionaldensitythatadatapoint x belongedtothe j th component.Theclass conditionaldensitiesareassumedtobeparametric(themostpopularmodelbeing theGaussian).Asaresult,themixturemodelisconsideredtobe`semi-parametric'in nature. Thepriorsareofcourseunknown,andneedtobeestimated,asalsotheparameters ofeachindividualclass.ThetypicalparametersforaGaussianclassarethemean j andthecovariancematrix j .Theunknownquantities P (j ), j and j areinferred throughanexpectationmaximizationframework(startingfromtheknowledgeof thesamplesthatareavailabletotheuser),whichisaniterativeprocedureproneto 24

PAGE 25

local minima.Thechoiceofthenumberofcomponents,i.e. M ,isalsoknowntobe quitecritical,withaverysmallvalueleadingtoinexpressivedensityestimates.Large valuesfor M reducetheefciencyofthemixturemodeloverthesimplekerneldensity estimator. 2.1.5Wavelet-BasedDensityEstimators Theseestimatorshavebeenintroducedrelativelyrecentlyandareinspired bytheoverwhelmingsuccessofwaveletsinfunctionapproximation.Anexcellent tutorialintroductiontowaveletdensityestimationexistsin[ 6 ]and[7 ],fromwhichthe followingmaterialissummarized.Traditionally,adensityestimate ^ p (x ) (foratrue underlyingdensity p ( x ))inthisparadigmisexpressedinthefollowingmanner,asa linearcombinationofmotherandfatherwaveletbases( (.) and (.) respectively): ^ p (x )= X L,k L,k L,k (x )+ 1 X j L,k f j k j ,k (x ) (2) where f L,k g and f j ,k arethecoefcientsofexpansionrespectively.Notethatthelevel L indicatesthecoarsestscale.Thebasisfunctionsataresolution j areexpressedinthe followingmanner: jk (x )=2 j =2 (2 j x )Tj /T1_1 11.955 Tf 11.96 0 Td (k ) (2) jk (x )=2 j =2 (2 j x )Tj /T1_1 11.955 Tf 11.96 0 Td (k ). (2) Theindices j (or j 0 )and k arethetranslationandscaleindicesrespectively.The coefcientsoftheentirewaveletexpansionaregivenbythefollowingformulae: L ,k = Z + 1 L, k (x )p (x )dx (2) f j ,k = Z + 1 j k (x )p (x )dx (2) andinpracticeareestimatedasfollows: ^ L ,k = 1 n n X i =1 L,k (x i ) (2) 25

PAGE 26

^ f j ,k = 1 n n X i =1 j ,k (x i ) (2) f orasampleset fx i g (1 i n ).Apractitionerusingthisparadigmneedstochoose asuitablewaveletkernel(Daubechies,symlets,coiets,Haaretc.)andevenmore criticallythemaximumlevelsoastotruncatetheaboveinniteexpansion.This maximumlevel(say L 1 )decideswhatisthenestresolutionoftheexpresseddensity ^ p (x ),andisamodelselectionissue.Anotherissueisthethresholdingofthewavelet coefcientsaftertheircomputationfromthegivensamples.Thisstrategyisadopted in[ 8 ].Thedrawbacksofthismethodarethattheestimatesubsequenttothresholding isnotguaranteedtobenon-negative,makingfurtherrenormalizationnecessary.An interestingmethodtocircumventthisnegativityissueistoexpressthesquarerootof thedensityastheaforementionedsummation,asopposedtothedensityitself.Inother words,wenowhave: p ^ p (x ) = X L,k L,k L, k (x )+ 1 X j L, k f j ,k j k (x ) (2) whichuponsquaringyieldsthedensityestimate ^ p (x ) whichisnowcertainlynon-negative. Animplicitconstraintonthecoefcients X k 2 L k + X j L, k f 2 j ,k =1 (2) isnowimposed,arisingfromthefactthat R p ( x )dx =1. 2.2MarginalandJointDensityEstimation Inthissection,weshowthederivationoftheprobabilitydensityfunction(PDF)for themarginalaswellasthejointdensityforapairof2Dimages.Wepointoutpractical issuesandcomputationalconsiderations,aswellasoutlinethedensityderivationsfor thecaseof3Dimages,aswellasmultipleimagesin2D.Thematerialpresentedhere 26

PAGE 27

is takenfromtheauthor'spreviouspublications[ 9],[ 10]and[11] 1 .Themajordifference betweentheapproachpresentedhereandthatofallthefourtechniquesdescribed inprevioussubsectionsliesinthis:theproposedapproachreallyregardsanimage (signal)asanimage(asignal)andnotabunchofsamplesthatcanbere-arranged withoutaffectingthedensityestimate.Thereforeessentialpropertiesonthesignal (image)canbedirectlyincorporatedintotheestimationprocedureitself. 2.2.1EstimatingtheMarginalDensitiesin2D Considerthe2Dgray-scaleimageintensitytobeacontinuous,scalar-valued functionofthespatialvariables,representedas w = I (x y ) .Letthetotalareaofthe imagebedenotedby A.Assumealocationrandomvariable Z =(X Y ) withauniform distributionovertheimageeldofview(FOV).Further,assumeanewrandomvariable W whichisatransformationoftherandomvariable Z andwiththetransformationgiven bythegray-scaleimageintensityfunction W = I ( X Y ).Thenthecumulativedistribution of W atacertainintensitylevel isequaltotheratioofthetotalareaofallregions whoseintensityislessthanorequalto tothetotalareaoftheimage Pr( W )= 1 A Z Z I ( x ,y ) dxdy (2) Now,theprobabilitydensityof W at isthederivativeofthecumulativedistribution in( 2 ).Thisisequaltothedifferenceintheareasenclosedwithintwolevelcurves thatareseparatedbyanintensitydifferenceof (orequivalently,theareaenclosed betweentwolevelcurvesofintensity and + ),perunitdifference,as 0 (see 1 P artsofthecontentofthisandsubsequentsectionsofthischapterhavebeen reprintedwithpermissionfrom:A.Rajwade,A.BanerjeeandA.Rangarajan,`Probability densityestimationusingisocontoursandisosurfaces:applicationstoinformation theoreticimageregistration',IEEETransactionsonPatternAnalysisandMachine Intelligence,vol.31,no.3,pp.475-491,2009. c r2009,IEEE 27

PAGE 28

Figure 2-1). Theformalexpressionforthisis p ( )= 1 A lim !0 R R I ( x ,y ) + dxdy )Tj /T1_7 11.955 Tf 11.96 9.63 Td [(RR I (x ,y ) dxdy (2) Hence ,wehave p ( )= 1 A d d Z Z I ( x ,y ) dxdy (2) Wecannowadoptachangeofvariablesfromthespatialcoordinates (x y ) to u (x y ) and I (x y ),where u and I arethedirectionsparallelandperpendiculartothelevel curveofintensity ,respectively.Observethat I pointsinthedirectionoftheimage gradient,orthedirectionofmaximumintensitychange.Notingthisfact,wenowobtain thefollowing: p ( )= 1 A Z I (x y )= f f f f f f f @ x @ I @ y @ I @ x @ u @ y @ u f f f f f f f du (2) Note thatinEq.(2), d and dI havecanceledeachotherout,astheyboth standforintensitychange.Afterperformingachangeofvariablesandsomealgebraic manipulations(seeAppendixAforthecompletederivation),wegetthefollowing expressionforthemarginaldensity p ( )= 1 A Z I ( x ,y )= du p I 2 x + I 2 y (2) F romtheaboveexpression,onecanmakesomeimportantobservations.Each pointonagivenlevelcurvecontributesacertainmeasuretothedensityatthatintensity whichisinverselyproportionaltothemagnitudeofthegradientatthatpoint.Inother words,inregionsofhighintensitygradient,theareabetweentwolevelcurvesatnearby intensitylevelswouldbesmall,ascomparedtothatinregionsoflowerimagegradient (seeFigure 2-1).Whenthegradientvalueatapointiszero(owingtotheexistenceofa peak,avalley,asaddlepointoraatregion),thecontributiontothedensityatthatpoint tendstoinnity.(Thepracticalrepercussionsofthissituationarediscussedlateronin thepaper.Lastly,thedensityatanintensitylevelcanbeestimatedbytraversingthe 28

PAGE 29

le velcurve(s)atthatintensityandintegratingthereciprocalofthegradientmagnitude. Onecanobtainanestimateofthedensityatseveralintensitylevels(atintensityspacing of h fromeachother)acrosstheentireintensityrangeoftheimage. 2.2.2RelatedWork Asimilardensityestimatorhasalsobeendevelopedbyanothergroupofresearchers [12 ],completelyindependentlyofthiswork.Theirdensityestimatorismotivated exclusivelybyrandomvariabletransformationsanddoesnotincorporatethenotion oflevelsets.Furthermore,apartfromdifferencesinthederivationoftheresults,there aredifferencesinimplementation.Moreovertheapplicationstheyhavetargetedare mainlyimagesegmentation,particularlyinthebiomedicaldomain[ 13 ].Similarnotions ofdensitiesobtainedfromrandomvariabletransformationshavebeenmentionedin[ 14] inthecontextofhistogrampreservingcontinuoustransformations,withapplicationsto studyingdifferentprojectionsof3Dmodels.However,intheiractualimplementation, onlydigitalsamplesareused,andthereisnonotionofanyjointstatistics.Thedensity estimatorpresentedinthisthesiswasspecicallydevelopedinthecontextofanimage registrationapplication(moreaboutthisinChapter3),andhasbeenextendedfor variousspecialcasessuchasimagesdenedin3D,twoormorethantwoimagesin 2D,andbiaseddensityestimatorsin2Daswellas3D(aswillbeenseeninsubsequent sectionsofthischapter). 2.2.3OtherMethodsforDerivation Thereexistatleasttwoothermethodsofderivingtheexpressionabove,whichare discussedbelow. 1. UsingDirac-deltafunctions: TheDirac-deltafunction(withitsdomainbeingthe realline)isdenedasfollows: (x )=+1( if x =0) (2) =0( if x 6=0) 29

PAGE 30

in suchawaythat Z + 1 (x )dx =1. (2) Thedeltafunctionhasanalogousdenitionsinhigherdimensions.Itisa well-knownpropertyofthedeltafunction(inanydimension)that Z + 1 f ( ~ x ) (I ( ~ x )) d ~ x = Z I )Tj /T1_2 5.978 Tf 5.75 0 Td (1 (0) f ( ~ x ) du jrI ( ~ x )j (2) Setting f ( ~ x ) to beunitythroughoutandconsideringthat I ( ~ x ) istheimagefunction, itiseasytoseethat p (I ( ~ x )= )= Z (I ( ~ x ) )Tj /T1_4 11.955 Tf 11.96 0 Td ( )dx = Z I )Tj /T1_2 5.978 Tf 5.76 0 Td (1 (0) du jrI ( ~ x ) j (2) 2. An intuitivegeometricapproach: Againconsiderthe2Dgray-scaleimage intensitytobeacontinuous,scalar-valuedfunctionofthespatialvariables, representedas z = I (x y ).Assuminglocationsareiid,thecumulativedistribution atacertainintensitylevel canbewrittenasfollows: Pr( z < )= 1 A Z Z z < dxdy (2) Now,theprobabilitydensityat isthederivativeofthecumulativedistribution. Thisisequaltothedifferenceintheareasenclosedwithintwolevelcurvesthat areseparatedbyanintensitydifferenceof (orequivalently,theareaenclosed betweentwolevelcurvesofintensity and + ),perunitdifference,as 0 (seeFigure( 2-1)).Ateverylocation (x y ) alongthelevelcurveat theperpendiculardistance(intermsofspatialcoordinates)tothelevelcurveat + isgivenas g (x ,y ) where g ( x y ) stands forthemagnitudeoftheintensity gradientat (x y ).Hencethetotalareaenclosedbetweenthetwolevelcurvescan becalculatedasthisdistanceintegratedallalongthecontourat .Denotingthe tangenttothelevelcurveas u ,andtakingthelimitas 0,weobtainthesame expression. 2.2.4EstimatingtheJointDensity Considertwoimagesrepresentedascontinuousscalarvaluedfunctions w 1 = I 1 (x y ) and w 2 = I 2 (x y ),whoseoverlapareais A.Asbefore,assumealocation randomvariable Z = f X Y g withauniformdistributionoverthe(overlap)eldofview. Further,assumetwonewrandomvariables W 1 and W 2 whicharetransformationsof therandomvariable Z andwiththetransformationsgivenbythegray-scaleimage intensityfunctions W 1 = I 1 (X Y ) and W 2 = I 2 (X Y ).Letthesetofallregionswhose 30

PAGE 31

intensity in I 1 islessthanorequalto 1 andwhoseintensityin I 2 islessthanorequal to 2 bedenotedby L .Thecumulativedistribution Pr( W 1 1 W 2 2 ) atintensity values ( 1 2 ) isequaltotheratioofthetotalareaof L tothetotaloverlaparea A.The probabilitydensity p ( 1 2 ) inthiscaseisthesecondpartialderivativeofthecumulative distributionw.r.t. 1 and 2 .Considerapairoflevelcurvesfrom I 1 havingintensity values 1 and 1 + 1 ,andanotherpairfrom I 2 havingintensity 2 and 2 + 2 .Let usdenotetheregionenclosedbetweenthelevelcurvesof I 1 at 1 and 1 + 1 as Q 1 andtheregionenclosedbetweenthelevelcurvesof I 2 at 2 and 2 + 2 as Q 2 .Then p ( 1 2 ) cangeometricallybeinterpretedastheareaof Q 1 \ Q 2 ,dividedby 1 2 inthelimitas 1 and 2 tendtozero.Theregions Q 1 Q 2 andalso Q 1 \ Q 2 (dark blackregion)areshowninFigure 2-2(left).Usingatechniqueverysimilartothatshown inEqs.( 2)-(2),weobtaintheexpressionforthejointcumulativedistributionas follows: Pr( W 1 1 W 2 2 )= 1 A Z Z L dxdy (2) Bydoingachangeofvariables,wearriveatthefollowingformula: Pr( W 1 1 W 2 2 )= 1 A Z Z L f f f f f f f @ x @ u 1 @ y @ u 1 @ x @ u 2 @ y @ u 2 f f f f f f f du 1 du 2 (2) Here u 1 and u 2 represent directionsalongthecorrespondinglevelcurvesofthetwo images I 1 and I 2 .Takingthesecondpartialderivativewithrespectto 1 and 2 ,weget theexpressionforthejointdensity: p ( 1 2 )= 1 A @ 2 @ 1 @ 2 ZZ L f f f f f f f @ x @ u 1 @ y @ u 1 @ x @ u 2 @ y @ u 2 f f f f f f f du 1 du 2 (2) It isimportanttonotehereagain,thatthejointdensityin(2)maynotexist becausethecumulativemaynotbedifferentiable.Geometrically,thisoccursif(a)both 31

PAGE 32

the imageshavelocallyconstantintensity,(b)ifonlyoneimagehaslocallyconstant intensity,or(c)ifthelevelsetsofthetwoimagesarelocallyparallel.Incase(a),we havearea-measuresandintheothertwocases,wehavecurve-measures.Thesecases aredescribedindetailinthefollowingsection,butforthemoment,weshallignorethese degeneracies. ToobtainacompleteexpressionforthePDFintermsofgradients,itwouldbe highlyintuitivetofollowpurelygeometricreasoning.Onecanobservethatthejoint probabilitydensity p ( 1 2 ) isthesumtotalofcontributionsateveryintersection betweenthelevelcurvesof I 1 at 1 andthoseof I 2 at 2 .Eachcontributionisthe areaofparallelogramABCD[seeFigure 2-2(right)]atthelevelcurveintersection,as theintensitydifferences 1 and 2 shrinktozero.(Weconsideraparallelogram here,becauseweareapproximatingthelevelcurveslocallyasstraightlines.)Letthe coordinatesofthepoint B be (~ x ,~ y ) andthemagnitudeofthegradientof I 1 and I 2 at thispointbe g 1 (~ x ,~ y ) and g 2 (~ x ,~ y ).Also,let (~ x ,~ y ) bethe acute anglebetweenthe gradientsofthetwoimagesat B .Observethattheintensitydifferencebetweenthe twolevelcurvesof I 1 is 1 .Then,usingthedenitionofgradient,theperpendicular distancebetweenthetwolevelcurvesof I 1 isgivenas 1 g 1 ( ~ x ,~ y ) .Lookingattriangle CDE (wherein CE isperpendiculartothelevelcurves)wecannowdeducethelengthof CD (orequivalentlythatof AB ).Similarly,wecanalsondthelength CB .Thetwo expressionsaregivenby: jAB j = 1 g 1 ( ~ x ,~ y )sin (~ x ,~ y ) jCB j = 2 g 2 ( ~ x ,~ y )sin (~ x ,~ y ) (2) Now,theareaoftheparallelogramisequalto jAB jj CB j sin (~ x ,~ y ) (2) = 1 2 g 1 ( ~ x ,~ y )g 2 (~ x ,~ y )sin (~ x ,~ y ) 32

PAGE 33

With this,wenallyobtainthefollowingexpressionforthejointdensity: p ( 1 2 )= 1 A X C 1 g 1 (x y ) g 2 ( x y ) sin (x y ) (2) wheretheset C representsthe(countable)locusofallpointswhere I 1 (x y )= 1 and I 2 ( x y )= 2 .ItiseasytoshowthroughalgebraicmanipulationsthatEqs.( 2 ) and( 2 )areequivalentformulationsofthejointprobabilitydensity p ( 1 2 ).These resultscouldalsohavebeenderivedpurelybymanipulationofJacobians(asdone whilederivingmarginaldensities),andthederivationforthemarginalscouldalsohave proceededfollowinggeometricintuitions. Theformuladerivedabovetalliesbeautifullywithintuitioninthefollowingways. Firstly,theareaoftheparallelogram ABCD (i.e.thejointdensitycontribution)inregions ofhighgradient[ineitherorbothimage(s)]issmallerascomparedtothatinthecaseof regionswithlowergradients.Secondly,theareaofparallelogram ABCD (i.e.thejoint densitycontribution)istheleastwhenthegradientsofthetwoimagesareorthogonal andmaximumwhentheyareparallelorcoincident[seeFigure 2-3(a)].Infact,the jointdensitytendstoinnityinthecasewhereeither(orboth)gradient(s)is(are) zero,orwhenthetwogradientsalign,sothat sin iszero.Therepercussionsofthis phenomenonarediscussedinthefollowingsection. 2.2.5FromDensitiestoDistributions Inthetwoprecedingsub-sections,weobservedthedivergenceofthemarginal densityinregionsofzerogradient,orofthejointdensityinregionswhereeither(or both)imagegradient(s)is(are)zero,orwhenthegradientslocallyalign.Thegradient goestozeroinregionsoftheimagethatareatintermsofintensity,andalsoatpeaks, valleysandsaddlepointsontheimagesurface.Wecanignorethelatterthreecases astheyareanitenumberofpointswithinacontinuum.Theprobabilitycontribution ataparticularintensityinaatregionisproportionaltotheareaofthatatregion. Some adhoc approachescouldinvolvesimplyweedingouttheatregionsaltogether, 33

PAGE 34

b utthatwouldrequirethechoiceofsensitivethresholds.Thekeythingistonotice thatintheseregions, thedensitydoesnotexistbuttheprobabilitydistributiondoes. So,wecanswitchentirelytoprobabilitydistributionseverywherebyintroducinga non-zerolowerboundonthevaluesof 1 and 2 .Effectively,thismeansthat wealwayslookatparallelogramsrepresentingtheintersectionbetweenpairsoflevel curvesfromthetwoimages,separatedby non-zero intensitydifference,denoted as,say, h .Sincetheseparallelogramshaveniteareas,wehavecircumventedthe situationofchoosingthresholdstopreventthevaluesfrombecomingunbounded, andtheprobabilityat 1 2 ,denotedas ^ p ( 1 2 ) isobtainedfromtheareasofsuch parallelograms.Wetermthisarea-basedmethodofdensityestimationas AreaProb. Lateroninthepaper,weshallshowthattheswitchtodistributionsisprincipledand doesnotreduceourtechniquetostandardhistogramminginanymannerwhatsoever. Thenotionofanimageasacontinuousentityisoneofthepillarsofourapproach. Weadoptalocallylinearformulationinthispaper,forthesakeofsimplicity,though thetechnicalcontributionsofthispaperareinnowaytiedtoanyspecicinterpolant. Foreachimagegridpoint,weestimatetheintensityvaluesatitsfourneighborswithin ahorizontalorverticaldistanceof 0.5 pixels.Wethendivideeachsquaredenedby theseneighborsintoapairof triangles.Theintensitieswithineachtrianglecanbe representedasaplanarpatch,whichisgivenbytheequation z 1 = A 1 x + B 1 y + C 1 in I 1 .Iso-intensitylinesatlevels 1 and 1 + h withinthistrianglearerepresentedbythe equations A 1 x + B 1 y + C 1 = 1 and A 1 x + B 1 y + C 1 = 1 + h (likewisefortheiso-intensity linesof I 2 atintensities 2 and 2 + h ,withinatriangleofcorrespondinglocation).The contributionfromthistriangletothejointprobabilityat ( 1 2 ),i.e. ^ p ( 1 2 ) isthe areaboundedbythetwopairsofparallellines,clippedagainstthebodyofthetriangle itself,asshowninFigure 2-7.Inthecasethatthecorrespondinggradientsfromthetwo imagesareparallel(orcoincident),theyencloseaninniteareabetweenthem,which whenclippedagainstthebodyofthetriangle,yieldsaclosedpolygonofnitearea,as 34

PAGE 35

sho wninFigure 2-7.Whenboththegradientsarezero(whichcanbeconsideredtobea specialcaseofgradientsbeingparallel),theprobabilitycontributionisequaltothearea oftheentiretriangle.Inthecasewherethegradientofonlyoneoftheimagesiszero, thecontributionisequaltotheareaenclosedbetweentheparalleliso-intensitylines ofthe other image,clippedagainstthebodyofthetriangle(seeFigure 2-7).Observe thatthoughwehavetotreatpathologicalregionsspecially(despitehavingswitchedto distributions),wenowdonotneedtoselectthresholds,nordoweneedtodealwitha mixtureofdensitiesanddistributions.Theothermajoradvantageisaddedrobustnessto noise,aswearenowworkingwithprobabilitiesinsteadoftheirderivatives,i.e.densities. Theissuethatnowarisesishowthevalueof h maybechosen.Itshouldbe notedthatalthoughthereisnooptimal h ,ourdensityestimatewouldconveymore andmoreinformationasthevalueof h isreduced(incompletecontrasttostandard histogramming).InFigure 2-5,wehaveshownplotsofourjointdensityestimateand comparedittostandardhistogramsfor P equalto16,32,64and128binsin each image(i.e. 32 2 64 2 etc.binsinthejoint),whichillustrateourpointclearly.Wefound thatthestandardhistogramshadafargreaternumberofemptybinsthanourdensity estimator,forthesamenumberofintensitylevels.Thecorrespondingmarginaldiscrete distributionsfortheoriginalretinogramimage[ 1]for16,32,64and128binsareshown inFigure 2-6. 2.2.6JointDensitybetweenMultipleImagesin2D Forthesimultaneousregistrationofmultiple(d > 2)images,theuseofasingle d -dimensionaljointprobabilityhasbeenadvocatedinpreviousliterature[15 ],[ 16 ].Our jointprobabilityderivationcanbeeasilyextendedtothecaseof d > 2 imagesbyusing similargeometricintuitiontoobtainthepolygonalareabetween d intersectingpairsof levelcurves[seeFigure 2-3(right)forthecaseof d =3 images].Noteherethatthe d -dimensionaljointdistributionliesessentiallyina2Dsubspace,aswearedealing with2Dimages.Ana veimplementationofsuchaschemehasacomplexityof O (NP d ) 35

PAGE 36

where P is thenumberofintensitylevelschosenforeachimageand N isthesizeof eachimage.Interestingly,however,thisexponentialcostcanbeside-steppedbyrst computingtheatmost ( d ( d )Tj /T1_4 7.97 Tf 6.59 0 Td (1) 2 )P 2 points ofintersectionbetweenpairsoflevelcurves fromall d imageswithoneanother,foreverypixel.Secondly,agraphcanbecreated, eachofwhosenodesisanintersectionpoint.Nodesarelinkedbyedgeslabeledwith theimagenumber(say k th image)iftheyliealongthesameiso-contourofthatimage.In mostcases,eachnodeofthegraphwillhaveadegreeoffour(andintheunlikelycase wherelevelcurvesfromallimagesareconcurrent,themaximaldegreeofanodewill be 2 d ).Now,thisisclearlyaplanargraph,andhence,byEuler'sformula,wehavethe numberof(convexpolygonal)faces ~ F = d (d )Tj /T1_4 7.97 Tf (1) 2 4P 2 )Tj /T1_3 7.97 Tf 13.37 5.48 Td (d (d )Tj /T1_4 7.97 Tf (1) 2 P 2 + 2= O (P 2 d 2 ),which isquadraticinthenumberofimages.Theareaofthepolygonalfacesarecontributions tothejointprobabilitydistribution.Inapracticalimplementation,thereisnorequirement toevencreatetheplanargraph.Instead,wecanimplementasimpleincremental face-splittingalgorithm([ 17 ],section8.3).Insuchanimplementation,wecreatealistof faces F whichisupdatedincrementally.Tostartwith, F consistsofjustthetriangular faceconstitutingthethreeverticesofachosenhalf-pixelintheimage.Next,weconsider asinglelevel-line l atatimeandsplitintotwoanyfacein F that l intersects.This procedureisrepeatedforalllevellines(separatedbyadiscreteintensityspacing)ofall the d images.Thenaloutputisalistingofallpolygonalfaces F createdbyincremental splittingwhichcanbecreatedinjust O ( ~ FPd ) time.Thestoragerequirementcanbe madepolynomialbyobservingthatfor d images,thenumberofuniqueintensitytuples willbeatmost ~ FN intheworstcase(asopposedto P d ).Henceallintensitytuplescan beefcientlystoredandindexedusingahashtable. 2.2.7Extensionsto3D Whenestimatingtheprobabilitydensityfrom3Dimages,thechoiceofanoptimal smoothingparameterisalesscriticalissue,asamuchlargernumberofsamples areavailable.However,atatheoreticallevelthisstillremainsaproblem,whichwould 36

PAGE 37

w orseninthemultipleimagecase.In3D,themarginalprobabilitycanbeinterpretedas thetotalvolumesandwichedbetweentwoiso-surfacesatneighboringintensitylevels. Theformulaforthemarginaldensity p ( ) ofa3Dimage w = I ( x y z ) isgivenas follows: p ( )= 1 V d d Z ZZ I (x ,y z ) dxdydz (2) Here V isthevolumeoftheimage I ( x y z ).Wecannowadoptachangeofvariables fromthespatialcoordinates x y and z to u 1 (x y z ) u 2 (x y z ) and I (x y z ),where I istheperpendiculartothelevelsurface(i.e.paralleltothegradient)and u 1 and u 2 are mutuallyperpendiculardirectionsparalleltothelevelsurface.Notingthisfact,wenow obtainthefollowing: p ( )= 1 V Z Z I (x ,y z )= f f f f f f f f f f @ x @ I @ y @ I @ z @ I @ x @ u 1 @ y @ u 1 @ z @ u 1 @ x @ u 2 @ y @ u 2 @ z @ u 2 f f f f f f f f f f du 1 du 2 (2) Upon aseriesofalgebraicmanipulationsjustasbefore,weareleftwiththefollowing expressionfor p ( ): p ( )= 1 V Z Z I (x ,y z )= du 1 du 2 q ( @ I @ x ) 2 + ( @ I @ y ) 2 + ( @ I @ z ) 2 (2) F orthejointdensitycase,considertwo3Dimagesrepresentedas w 1 = I 1 (x y z ) and w 2 = I 2 (x y z ),whoseoverlapvolume(theeldofview)is V .Thecumulative distribution Pr( W 1 1 W 2 2 ) atintensityvalues ( 1 2 ) isequaltotheratioof thetotalvolumeofallregionswhoseintensityintherstimageislessthanorequal to 1 andwhoseintensityinthesecondimageislessthanorequalto 2 ,tothetotal imagevolume.Theprobabilitydensity p ( 1 2 ) isagainthesecondpartialderivative ofthecumulativedistribution.Considertworegions R 1 and R 2 ,where R 1 istheregion trappedbetweenlevelsurfacesoftherstimageatintensities 1 and 1 + 1 ,and R 2 isdenedanalogouslyforthesecondimage.Thedensityisproportionaltothevolume 37

PAGE 38

of theintersectionof R 1 and R 2 dividedby 1 and 2 whenthelattertwotendtozero. Itcanbeshownthroughsomegeometricmanipulationsthattheareaofthebaseof theparallelepipedformedbytheiso-surfacesisgivenas 1 2 j ~ g 1 ~ g 2 j = 1 2 j g 1 g 2 sin( )j where ~ g 1 and ~ g 2 arethegradientsofthetwoimages,and istheanglebetweenthem.Let ~ h be avectorwhichpointsinthedirectionoftheheightoftheparallelepiped(paralleltothe basenormal,i.e. ~ g 1 ~ g 2 ),and d ~ h beaninnitesimalstepinthatdirection.Thenthe probabilitydensityisgivenasfollows: p ( 1 2 )= 1 V @ 2 @ 1 @ 2 ZZZ V s dxdydz = 1 V @ 2 @ 1 @ 2 ZZZ V s d ~ u 1 d ~ u 2 d ~ h j ~ g 1 ~ g 2 j = 1 V Z C d ~ h j ~ g 1 ~ g 2 j (2) In Eq.( 2), ~ u 1 and ~ u 2 aredirectionsparalleltotheiso-surfacesofthetwoimages,and ~ h istheircross-product(andparalleltothelineofintersectionoftheindividualplanes), while C isthe3Dspacecurvecontainingthepointswhere I 1 and I 2 havevalues 1 and 2 respectivelyand V s def = f(x y z ): I 1 (x y z ) 1 I 2 (x y z ) 2 g. 2.2.8ImplementationDetailsforthe3Dcase Thedensityformulationforthe3Dcasesuffersfromthesameproblemof divergencetoinnity,asinthe2Dcase.Similartechniquescanbeemployed,this timeusinglevel surfaces thatareseparatedbyniteintensitygaps.Totracethelevel surfaces,eachcube-shapedvoxelinthe3Dimagecanbedividedinto12tetrahedra. Theapexofeachtetrahedronislocatedatthecenterofthevoxelandthebaseis formedbydividingoneofthesixsquarefacesofthecubebyoneofthediagonalsof thatface[seeFigure 2-8(a)].Withineachtriangularfaceofeachsuchtetrahedron,the intensitycanbeassumedtobealinearfunctionoflocation.Notethattheintensities indifferentfacesofoneandthesametetrahedroncanthusbeexpressedby different functions,allofthemlinear.Hencetheiso-surfacesatdifferentintensitylevelswithina singletetrahedronarenon-intersectingbutnotnecessarilyparallel.Theselevelsurfaces atanyintensitywithinasingletetrahedronturnouttobeeithertrianglesorquadrilaterals 38

PAGE 39

in 3D.Thisinterpolationschemedoeshavesomebiasinthechoiceofthediagonals thatdividetheindividualsquarefaces.Aschemethatuses24tetrahedrawiththeapex atthecenterofthevoxel,andfourtetrahedrabasedoneverysingleface,hasnobias ofthiskind[seeFigure 2-8(b)].However,westillusedtheformer(andfaster)scheme asitissimpleranddoesnotnoticeablyaffecttheresults.Levelsurfacesareagain tracedatanitenumberofintensityvalues,separatedbyequalintensityintervals.The marginaldensitycontributionsareobtainedasthevolumesofconvexpolyhedratrapped inbetweenconsecutivelevelsurfacesclippedagainstthebodyofindividualtetrahedra. Thejointdistributioncontributionfromeachvoxelisobtainedbyndingthevolumeof theconvexpolyhedronresultingfromtheintersectionofcorrespondingconvexpolyhedra fromthetwoimages,clippedagainstthetetrahedrainsidethevoxel.Werefertothis schemeofndingjointdensitiesas VolumeProb 2.2.9JointDensitiesbyCountingPointsandMeasuringLengths Forthespeciccaseofregistrationoftwoimagesin2D,wepresentanother methodofdensityestimation.Thismethod,whichwaspresentedbyusearlierin[ 10], isabiasedestimatorthatdoesnotassumeauniformdistributiononlocation.Inthis technique,thetotalnumberofco-occurrencesofintensities 1 and 2 fromthetwo imagesrespectively,isobtainedbycountingthetotalnumberofintersectionsofthe correspondinglevelcurves.Eachhalf-pixelcanbeexaminedtoseewhetherlevel curvesofthetwoimagesatintensities 1 and 2 canintersectwithinthehalf-pixel. Thisprocessisrepeatedfordifferent(discrete)valuesfromthetwoimages( 1 and 2 ), separatedbyequalintervalsandselected apriori (seeFigure 2-9).Theco-occurrence countsarethennormalizedsoastoyieldajointprobabilitymassfunction(PMF).We denotethismethodas 2DPointProb.Themarginalsareobtainedbysummingupthe jointPMFalongtherespectivedirections.Thismethod,too,avoidsthehistogramming binningproblemasonehasthelibertytochooseasmanylevelcurvesasdesired. However,itisabiaseddensityestimatorbecausemorepointsarepickedfromregions 39

PAGE 40

with highimagegradient.Thisisbecausemorelevelcurves(atequi-spacedintensity levels)arepackedtogetherinsuchareas.Itcanalsoberegardedasaweightedversion ofthejointdensityestimatorpresentedintheprevioussub-section,witheachpoint weightedbythegradientmagnitudesofthetwoimagesatthatpointaswellasthesine oftheanglebetweenthem.ThusthejointPMFbythismethodisgivenas p ( 1 2 )= @ 2 @ 1 @ 2 1 K Z Z D g 1 ( x y )g 2 (x y )sin (x y )dxdy (2) where D denotestheregionswhere I 1 ( x y ) 1 I 2 ( x y ) 2 and K isanormalization constant.Thissimpliestothefollowing: p ( 1 2 )= 1 K X C 1. (2) Hence ,wehave p ( 1 2 )= jC j K where C isthe(countable)setofpointswhere I 1 (x y )= 1 and I 2 (x y )= 2 .Themarginal(biased)densityestimatescanberegarded aslengthsoftheindividualiso-contours.Withthisnotioninmind,themarginaldensity estimatesareseentohaveacloserelationwiththetotalvariationofanimage,which isgivenby TV = R I = jrI (x y )jdxdy [ 18].Weclearlyhave TV = R I = du ,bydoing thesamechangeofvariables(from x y to u I )asinEqs.(2)and( 2 ),thus givingusthelengthoftheiso-contoursatanygivenintensitylevel.In3D,weconsider thesegmentsofintersectionoftwoiso-surfacesandcalculatetheirlengths,which becomethePMFcontributions.Werefertothisas LengthProb [seeFigure 2-10(a)]. Both 2DPointProb and LengthProb,however,requireustoignorethoseregionsinwhich levelsetsdonotexistbecausetheintensityfunctionisat,orthoseregionswherelevel setsfromthetwoimagesareparallel.Thecaseofatregionsinoneorbothimagescan bexedtosomeextentbyslightblurringoftheimage.Thecaseofalignedgradients istrickier,especiallyifthetwoimagesareincompleteregistration.However,inthe multi-modalitycaseoriftheimagesarenoisy/blurred,perfectregistrationisarare occurrence,andhenceperfectalignmentoflevelsurfaceswillrarelyoccur. 40

PAGE 41

T osummarize,inboththesetechniques, locationistreatedasarandomvariablewithadistributionthatisnotuniform,butinsteadpeakedat(biasedtowards) locationswherespecicfeaturesoftheimageitself(suchasgradients)havelarge magnitudesorwheregradientvectorsfromthetwoimagesareclosertowardsbeing perpendicularthanparallel. Suchabiastowardshighgradientsisprincipled,asthese arethemoresalientregionsofthetwoimages.Empirically,wehaveobservedthat boththesedensityestimatorsworkquitewellonafneregistration,andthat LengthProb ismorethan10timesfasterthan VolumeProb .Thisisbecausethecomputation ofsegmentsofintersectionofplanariso-surfacesismuchfasterthancomputing polyhedronintersections.JointPMFplotsforhistogramsand LengthProb for128bins and256binsareshowninFigure 2-11. Thereexistsonemoremajordifferencebetween AreaProb and VolumeProb on onehand,and LengthProb or 2DPointProb ontheother.Theformertwocanbeeasily extendedtocomputejointdensitybetweenmultipleimages(neededforco-registration ofmultipleimagesusingmeasuressuchasmodiedmutualinformation(MMI)[ 15]).All thatisrequiredistheintersectionofmultipleconvexpolyhedrain3Dormultipleconvex polygonsin2D(seeSection 2.2.6).However, 2DPointProb isstrictlyapplicabletothe caseofthejointPMFbetweenexactlytwoimagesin2D,astheproblemofintersection ofthreeormorelevelcurvesat specic (discrete)intensitylevelsisover-constrained. In3D, LengthProb alsodealswithstrictlytwoimagesonly,butonecanextendthe LengthProb schemetoalsocomputethejointPMFbetweenexactlythreeimages.This canbedonebymakinguseofthefactthatthreeplanariso-surfacesintersectinapoint (exceptingdegeneratecases)[seeFigure 2-10(b)].ThejointPMFsbetweenthethree imagesarethencomputedbycountingpointintersections.Weshallnamethismethod as 3DPointProb.Thedifferencesbetweenalltheaforementionedmethods: AreaProb, 2DPointProb, VolumeProb LengthProb and 3DPointProb aresummarizedinTable 2-1 forquickreference.Itshouldbenotedthat 2DPointProb, LengthProb and 3DPointProb 41

PAGE 42

Figure 2-1. p ( ) / areabetweenlevelcurvesat and + (i.e.regionwithreddots) computePMFs,whereas AreaProb and VolumeProb computecumulativemeasures overniteintervals. 2.3ExperimentalResults:Area-BasedPDFsVersusHistogramswithSeveral Sub-PixelSamples TheaccuracyofthehistogramestimatewillnodoubtapproachthetruePDFas thenumberofsamples N s (drawnfromsub-pixellocations)tendstoinnity.However, wewishtopointoutthatourmethodimplicitlyandefcientlyconsiderseverypointas asample,therebyconstructingthePDFdirectly,i.e.theaccuracyofwhatwecalculate withthearea-basedmethodwillalwaysbeanupperboundontheaccuracyyieldedby any sample-basedapproach, undertheassumptionthatthetrueinterpolantisknownto us. Weshowhereananecdotalexampleforthesame,inwhichthenumberofhistogram samples N s isvariedfrom 5000 to 2 10 9 .The L 1 and L 2 normsofthedifference betweenthejointPDFoftwo90x109images(down-sampledMR-T1andMR-T2slices obtainedfromBrainweb[ 19 ])ascomputedbyourmethodandthatobtainedbythe histogrammethod,aswellastheJensen-Shannondivergence(JSD)betweenthetwo jointPDFs,areplottedintheguresbelowversus log N s (seeFigure 2-12).Thenumber ofbinsusedwas 128 128 (i.e. h =128).Visually,itwasobservedthatthejointdensity surfacesbegintoappearevermoresimilaras N s increases.Thetimingvaluesforthe jointPDFcomputationareshowninTable 2-2,clearlyshowingthegreaterefciencyof ourmethod. 42

PAGE 43

A B Figure 2-2.(A)Intersectionoflevelcurvesof I 1 and I 2 : p ( 1 2 ) / areaofdarkblack regions.(B)Parallelogramapproximation:PDFcontribution=area( ABCD ) A B Figure 2-3.(A)Areaofparallelogramincreasesasanglebetweenlevelcurves decreases(lefttoright).Levelcurvesof I 1 and I 2 areshowninredandblue linesrespectively(B)Jointprobabilitycontributioninthecaseofthree images Table2-1.Comparisonbetweendifferentmethodsofdensityestimationw.r.t.natureof domain,bias,speed,andgeometricnatureofdensitycontributions Method 2D/3DDensityContr.BiasNo.ofimages AreaProb 2DAreaNoAny VolumeProb3DVolumeNoAny LengthProb3DLengthYes2only 2DPointProb2DPointcountYes2only 3DPointProb3DPointcountYes3only 43

PAGE 44

A B Figure 2-4.Aretinogram[1]anditsrotatednegative A B C D E F G H Figure 2-5.Followinglefttorightandtoptobottom,jointdensitiesoftheretinogram imagescomputedbyhistograms(using16,32,64,128bins)andbyour area-basedmethod(using16,32,64and128bins) 44

PAGE 45

A B C D E F G H Figure 2-6.Marginaldensitiesoftheretinogramimagecomputedbyhistograms[from (A)to(D)]andourarea-basedmethod[from(E)to(H)]using16,32,64and 128bins(row-wiseorder) Table2-2.TimingvaluesforcomputationofjointPDFsand L 1 normofdifference betweenPDFcomputedbysamplingwiththatcomputedusingiso-contours; Numberofbinsis 128 128,sizeofimages 122 146 MethodTime (secs.)Diff.withiso-contourPDF Iso-contours 5.1 0 Hist. 10 6 samples1 0.0393 Hist. 10 7 samples11 0.01265 Hist. 10 8 samples106 0.0039 Hist. 5 10 8 samples 450 0.00176 Hist. 2 10 9 samples1927 8.58 10 )Tj /T1_2 7.97 Tf (4 45

PAGE 46

Figure 2-7.Left:Probabilitycontributionequaltoareaofparallelogrambetweenlevel curvesclippedagainstthetriangle,i.e.half-pixel.Middle:Caseofparallel gradients.Right:Casewhenthegradientofoneimageiszero(bluelevel lines)andthatoftheotherisnon-zero(redlevellines).Ineachcase, probabilitycontributionequalsareaofthedarkblackregion A B Figure 2-8.Splittingavoxel(A)into12tetrahedra,twooneachofthesixfacesofthe voxel;and(B)into24tetrahedra,fouroneachofthesixfacesofthevoxel 46

PAGE 47

Figure 2-9.Countinglevelcurveintersectionswithinagivenhalf-pixel A B Figure 2-10.Biasedestimatesin3D:(A)Segmentofintersectionofplanariso-surfaces fromthetwoimages,(B)Pointofintersectionofplanariso-surfacesfrom thethreeimages(eachinadifferentcolor) 47

PAGE 48

A B C D Figure 2-11.Jointprobabilityplotsusing:(A)histograms,128bins,(B)histograms,256 bins,(C) LengthProb,128binsand(D) LengthProb,256bins 48

PAGE 49

A B C D Figure 2-12.PlotsofthedifferencebetweenthejointPDF(oftheimagesinsubgure [A])computedbythearea-basedmethodandbyhistogrammingwith N s sub-pixelsamplesversus log N s using(B) L 1 norm,(C) L 2 norm,and(D) JSD 49

PAGE 50

CHAPTER 3 APPLICATIONTOIMAGEREGISTRATION 3.1EntropyEstimatorsinImageRegistration Informationtheoretictoolshaveforalongtimebeenestablishedasthe defacto techniqueforimageregistration,especiallyinthedomainsofmedicalimaging[ 20]and remotesensing[ 21 ]whichdealwithalargenumberofmodalities.Theground-breaking workforthiswasdonebyViolaandWells[ 22 ],andMaes etal. [23]intheirwidelycited papers 1 .Adetailedsurveyofsubsequentresearchoninformationtheoretictechniques inmedicalimageregistrationispresentedintheworksofPluim etal. [ 20]andMaes etal. [24 ].Arequiredcomponentofallinformationtheoretictechniquesinimage registrationisagoodestimatorofthejointentropiesoftheimagesbeingregistered. Mosttechniquesemployplug-inentropyestimators,whereinthejointandmarginal probabilitydensitiesoftheintensityvaluesintheimagesarerstestimatedandthese quantitiesarethenusedtoobtaintheentropy.Therealsoexistrecentmethodswhich deneanewformofentropyusingcumulativedistributionsinsteadofprobability densities(see[ 25],[ 26]and[ 27 ]).Furthermore,therealsoexisttechniqueswhich directlyestimatetheentropy,withoutestimatingtheprobabilitydensityordistributionas anintermediatestep[ 28].Below,wepresentabird'seyeviewofthesetechniquesand theirlimitations.Subsequently,weintroduceourmethodandbringoutitssalientmerits. Theplug-inentropyestimatorsrelyupontechniquesfordensityestimationasa keyrststep.Themostpopulardensityestimatorsarethesimpleimagehistogramand theParzenwindow.Thelatterhavebeenwidelyemployedasadifferentiabledensity estimatorforimageregistrationin[ 22].Theproblemsassociatedwiththeseestimators 1 P artsofthecontentsofthischapterhavebeenreprintedwithpermissionfrom: A.Rajwade,A.BanerjeeandA.Rangarajan,`Probabilitydensityestimationusing isocontoursandisosurfaces:applicationstoinformationtheoreticimageregistration ',IEEETransactionsonPatternAnalysisandMachineIntelligence,vol.31,no.3,pp. 475-491,2009. c r2009,IEEE 50

PAGE 51

ha vebeendiscussedinthepreviouschapter.ThekernelwidthparameterinParzen windowscanbeestimatedbytechniquessuchasmaximumlikelihood(seeSection 3.3.1of[ 29]).Suchmethods,however,requirecomplicatediterativeoptimizations, andalsoatrainingandvalidationset.Fromanimageregistrationstandpoint,thejoint densitybetweentheimagesundergoesachangeineachiteration,whichrequires re-estimationofthekernelwidthparameters.Thisstepisanexpensiveiterativeprocess withacomplexitythatisquadraticinthenumberofsamples.Methodssuchasthefast Gausstransform[ 30]reducethiscosttosomeextentbuttheyrequireapriorclustering step.Also,thefastGausstransformisonlyan approximation tothetrueParzendensity estimate,andhence,oneneedstoanalyzethebehavioroftheapproximationerrorover theiterationsifagradient-basedoptimizerisused.YetanotherdrawbackofParzen windowbaseddensityestimatorsisthewell-knowntaileffectinhigherdimensions, duetowhichalargenumberofsampleswillfallinthoseregionswheretheGaussian hasverylowvalue[ 3].Mixturemodelshavebeenusedforjointdensityestimationin registration[ 31],buttheyarequiteinefcientandrequirechoiceofthekernelfunction forthecomponents(usuallychosentobeGaussian)andthenumberofcomponents. Thisnumberagainwillchangeacrosstheiterationsoftheregistrationprocess,asthe imagesmovewithrespecttooneanother.Waveletbaseddensityestimatorshavealso beenrecentlyemployedinimageregistration[ 32]andinconjunctionwithMI[7].The problemswithawaveletbasedmethodfordensityestimationincludeachoiceofwavelet function,aswellastheselectionoftheoptimalnumberoflevels,whichagainrequires iterativeoptimization. Directentropyestimatorsavoidtheintermediatedensityestimationphase.While thereexistsaplethoraofpapersinthiseld(surveyedin[ 28]),themostpopularentropy estimatorusedinimageregistrationistheapproximationoftheRenyientropyasthe weightofaminimalspanningtree[ 33]ora K -nearestneighborgraph[ 34].Notethat theentropyusedhereistheRenyientropyasopposedtothemorepopularShannon 51

PAGE 52

entrop y.Drawbacksofthisapproachincludethecomputationalcostinconstructionof thedatastructureineachstepofregistration(thecomplexitywhereofisquadraticinthe numberofsamplesdrawn),thesomewhatarbitrarychoiceofthe parameterforthe Renyientropyandthelackofdifferentiabilityofthecostfunction.Someworkhasbeen donerecently,however,tointroducedifferentiabilityinthecostfunction[ 35 ].Amerit ofthesetechniquesistheeaseofestimationofentropiesofhigh-dimensionalfeature vectors,withthecostscalingupjustlinearlywiththedimensionalityofthefeaturespace. Recently,anewformoftheentropydenedoncumulativedistributions,and relatedcumulativeentropicmeasuressuchascrosscumulativeresidualentropy (CCRE)havebeenintroducedintheliteratureonimageregistration[ 25],[26 ],[ 27 ]. ThecumulativeentropyandtheCCREmeasurehaveperfectlycompatiblediscrete andcontinuousversions(quiteunliketheShannonentropy,thoughnotunlikethe Shannonmutualinformation),andareknowntobenoiseresistant(astheyaredened oncumulativedistributionsandnotdensities).Ourmethodofdensityestimationcanbe easilyextendedtocomputingcumulativedistributionsandCCRE. Allthetechniquesreviewedherearebasedondifferentprinciples,buthave onecrucialcommonpoint:theytreattheimageasasetofpixelsorsamples,which inherentlyignoresthefactthatthesesamplesoriginatefromanunderlyingcontinuous (orpiece-wisecontinuous)signal.Noneofthesetechniquestakeintoaccountthe orderingbetweenthegivenpixelsofanimage.Asaresult,allthesemethodscanbe termed sample-based .Furthermore,mostoftheaforementioneddensityestimators requireaparticularkernel,thechoiceofwhichis extrinsic totheimagebeinganalyzed andnotnecessarilylinkedeventothenoisemodel.Inthischapter,weemployour densityestimatordiscussedinthepreviouschapter.Ourapproachhereisbasedon theauthor'searlierworkpresentedin[ 9]and[ 11 ](theessenceofwhichistoregard themarginalprobabilitydensityastheareabetweentwoiso-contoursatinnitesimally closeintensityvalues)andin[ 10](usingbiaseddensityestimatorsforregistration). 52

PAGE 53

Other priorworkonimageregistrationusingsuchimagebasedtechniquesincludes [36 ]and[ 37 ].Theworkin[ 36],however,reportsresultsonlyontemplatematchingwith translations,whereasthemainfocusof[ 37 ]isonestimationofdensitiesinvanishingly smallcircularneighborhoods.Theformulaederivedareveryspecictotheshape oftheneighborhood.Theirpaper[ 37]showsthatlocalmutualinformationvaluesin smallneighborhoodsarerelatedtothevaluesoftheanglesbetweenthelocalgradient vectorsinthoseneighborhoods.Thefocusofthismethod,howeveristoolocalinnature, therebyignoringtherobustnessthatisanintegralpartofmoreglobaldensityestimates. Notethatourmethod,basedonndingareasbetweeniso-contours,issignicantly differentfromPartialVolumeInterpolation(PVI)[ 23 ],[ 38].PVIusesacontinuous imagerepresentationtobuildajointprobabilitytablebyassigningfractionalvotesto multipleintensitypairswhenadigitalimageiswarpedduringregistration.Thefractional votesareassignedtypicallyusingabilinearorbicubickernelfunctionincasesof non-alignmentwithpixelgridsafterimagewarping.Inessence,thedensityestimatein PVIstillrequireshistogrammingorParzenwindowing. Themainmeritoftheproposedgeometrictechniqueisthefactthatitside-steps theparameterselectionproblemthataffectsotherdensityestimatorsandalsodoesnot relyonanyformofsampling.Theaccuracyofourtechniqueswillalways upperbound allsample-basedmethodsiftheimageinterpolantisknown(seeSection 3.4).Infact, theestimateobtainedbyallsample-basedmethodswillconvergetothatyieldedby ourmethodonlyinthelimitwhenthenumberofsamplestendstoinnity.Empirically, wedemonstratetherobustnessofourtechniquetonoise,andsuperiorperformancein imageregistration.Weconcludewithadiscussionandclaricationofsomepropertiesof ourmethod. 3.2ImageEntropyandMutualInformation Weareultimatelyinterestedinusingtheestimatedvaluesofthejointdensity p ( 1 2 ) tocalculate(Shannon)jointentropyandMI.Amajorconcernisthat,inthelimit 53

PAGE 54

as thebin-width h 0,theShannonentropydoes not approachthecontinuousentropy, butbecomesunbounded[ 39].Therearetwowaystodealwiththis.Firstly,anormalized versionofthejointentropy(NJE)obtainedbydividingtheShannonjointentropy(JE)by log P (where P isthenumberofbins),couldbeemployedinsteadoftheShannonjoint entropy.As h 0 andtheShannonentropytendstoward +1,NJEwouldstillremain stable,owingtothedivisionby log P ,whichwouldalsotendtoward +1 (infact,NJEwill haveamaximalupperboundof log P 2 log P = 2,forauniformjointdistribution).Alternatively (andthisisthemoreprincipledstrategy),weobservethatunlikethecasewithShannon entropy,thecontinuousMIisindeedthelimitofthediscreteMIas h 0 (see[39 ]forthe proof).Now,as P increases,weeffectivelyobtainanincreasinglybetterapproximation tothecontinuousmutualinformation. Inthemultipleimagecase(d > 2 ),weavoidusingapair-wisesumofMIvalues betweendifferentimagepairs,becausesuchasumignoresthesimultaneousjoint overlapbetweenmultipleimages.Instead,wecanemploymeasuressuchasmodied mutualinformation(MMI)[ 15 ],whichisdenedastheKLdivergencebetweenthe d -way jointdistributionandtheproductofthemarginaldistributions,oritsnormalizedversion (MNMI)obtainedbydividingMMIbythejointentropy.TheexpressionsforMIbetween twoimagesandMMIforthreeimagesaregivenbelow: MI (I 1 I 2 )= H 1 (I 1 )+ H 2 (I 2 ) )Tj /T1_1 11.955 Tf 11.96 0 Td (H 12 (I 1 I 2 ) (3) whichcanbeexplicitlywrittenas MI ( I 1 I 2 )= X j 1 X j 2 p ( j 1 j 2 )log p (j 1 j 2 ) p (j 1 )p (j 2 ) (3) where thesummationindices j 1 and j 2 rangeoverthesetsofpossibilitiesof I 1 and I 2 respectively.Forthreeimages, MMI ( I 1 I 2 I 3 )= H 1 ( I 1 )+ H 2 (I 2 )+ H 3 (I 3 ) )Tj /T1_1 11.955 Tf 11.96 0 Td (H 123 ( I 1 I 2 I 3 ) (3) 54

PAGE 55

which hastheexplicitform MMI (I 1 I 2 I 3 )= X j 1 X j 2 X j 3 p (j 1 j 2 j 3 )log p (j 1 j 2 j 3 ) p (j 1 ) p (j 2 )p (j 3 ) (3) where thesummationindices j 1 j 2 and j 3 rangeoverthesetsofpossibilitiesof I 1 I 2 and I 3 respectively.ThoughNMI(normalizedmutualinformation)andMNMIarenot compatibleinthediscreteandcontinuousformulations(unlikeMIandMMI),inour experiments,weignoredthisfactaswechoseveryspecicintensitylevels. 3.3ExperimentalResults Inthissection,wedescribeourexperimentalresultsfor(a)thecaseofregistration oftwoimagesin2D,(b)thecaseofregistrationofmultipleimagesin2Dand(c)the caseofregistrationoftwoimagesin3D. 3.3.1RegistrationofTwoimagesin2D Forthiscase,wetookpre-registeredMR-T1andMR-T2slicesfromBrainweb[19 ], down-sampledtosize 122 146 (seeFigure 2-12)andcreateda 20 rotatedversion oftheMR-T2slice.Tothisrotatedversion,zero-meanGaussiannoiseofdifferent varianceswasaddedusingthe imnoise functionofMATLAB R r .Thechosenvariances were0.01,0.05,0.1,0.2,0.5,1and2.Allthesevariancesarechosenforanintensity rangebetween0and1.Tocreatetheprobabilitydistributions,wechosebincountsof 16,32,64and128.Foreachcombinationofbin-countandnoise,abrute-forcesearch wasperformedsoastooptimallyalignthesyntheticallyrotatednoisyimagewiththe originalone,asdeterminedbyndingthemaximumofMIorNMIbetweenthetwo images.SixdifferenttechniqueswereusedforMIestimation:(1)simplehistogramswith bilinearinterpolationforimagewarping(referredtoasSimpleHist),(2)ourproposed methodusingiso-contours(referredtoasIso-contours),(3)histogrammingwith partialvolumeinterpolation(referredtoasPVI)(4)histogrammingwithcubicspline interpolation(referredtoasCubic),(5)themethod 2DPointProb proposedin[10 ],and (6)simplehistogrammingwith 10 6 samplestakenfromsub-pixellocationsuniformly 55

PAGE 56

r andomlyfollowedbyusualbinning(referredtoasHistSamples).Theseexperiments wererepeatedfor30noisetrialsateachnoisestandarddeviation.Foreachmethod,the meanandthevarianceoftheerror(absolutedifferencebetweenthepredictedalignment andthegroundtruthalignment)wasmeasured(Figure 3-1).Thesameexperiments werealsoperformedusingaParzen-windowbaseddensityestimatorusingaGaussian kerneland =5 (referredtoasParzen)over30trials.Ineachtrial,10,000samples werechosen.Outofthese,5000werechosenascentersfortheGaussiankerneland therestwereusedforthesakeofentropycomputation.Theerrormeanandvariance wasrecorded(seeTable 3-1). Theadjoiningerrorplots(Figure 3-1)showresultsforallthesemethodsfor allbinscounts,fornoiselevelsof0.05,0.2and1.Theaccompanyingtrajectories (forallmethodsexcepthistogrammingwithmultiplesub-pixelsamples)withMIfor bin-countsof32and128andnoiselevel0.05,0.2and1.00areshownaswell,forsake ofcomparison,foronearbitrarilychosennoisetrial(Figure 3-2).Fromthesegures, onecanappreciatethesuperiorresistancetonoiseshownbybothourmethods,even atveryhighnoiselevels,asevidencedbothbytheshapeoftheMIandNMItrajectories, aswellastheheightofthepeaksinthesetrajectories.Amongsttheothermethods, wenoticedthatPVIismorestablethansimplehistogrammingwitheitherbilinearor cubic-splinebasedimagewarping.Ingeneral,theothermethodsperformbetterwhen thenumberofhistogrambinsissmall,buteventhereourmethodyieldsasmoother MIcurve.However,asexpected,noisedoessignicantlylowerthepeakintheMIas wellasNMItrajectoriesinthecaseofallmethodsincludingours,duetotheincrease injointentropy.Thoughhistogrammingwith 10 6 sub-pixelsamplesperformswell(as seeninFigure 3-1),ourmethodefcientlyanddirectly(ratherthanasymptotically) approachesthetruePDFandhencethetrueMIvalue,undertheassumptionthatwe haveaccesstothetrueinterpolant.Parzenwindowswiththechosen valueof5gave goodperformance,comparabletoourtechnique,butwewishtore-emphasizethatthe 56

PAGE 57

choice oftheparameterwasarbitraryandthecomputationtimewasmuchmorefor Parzenwindows. Alltheaforementionedtechniqueswerealsotestedonafneimageregistration (exceptforhistogrammingwithmultiplesub-pixelsamplesandParzenwindowing, whichwerefoundtobetooslow).Forthesameimageasinthepreviousexperiment, anafne-warpedversionwascreatedusingtheparameters =30 =30, t =-0.3, s =-0.3and =0.Duringourexperiments,weperformedabruteforcesearchon thethree-dimensionalparameterspacesoastondthetransformationthatoptimally alignedthesecondimagewiththerstone.Theexactparameterizationfortheafne transformationisgivenin[ 40].Resultswerecollectedforatotalof20noisetrials andtheaveragepredictedparameterswererecordedaswellasthevarianceof thepredictions.Foralownoiselevelof0.01or0.05,weobservedthatallmethods performedwellforaquantizationupto64bins.With128bins,allmethodsexcept thetwowehaveproposedbrokedown,i.e.yieldedafalseoptimumof around 38 and s and t around0.4.Forhighernoiselevels,allmethodsexceptoursbrokedown ataquantizationofjust64bins.The 2DPointProb techniqueretaineditsrobustness untilanoiselevelof1,whereasthearea-basedtechniquestillproducedanoptimum of =28 s =-0.3, t =-0.4(whichisveryclosetotheidealvalue).Thearea-based techniquebrokedownonlyatanincrediblyhighnoiselevelof1.5or2.Theaverageand standarddeviationofthe estimate oftheparameters s and t ,for32and64bins,forall vemethodsandfornoiselevels0.2and1.00arepresentedinTables 3-2 and 3-3.We alsoperformedtwo-sidedKolmogorov-Smirnovtests[ 41 ]forstatisticalsignicanceon theabsoluteerrors(betweenthetrueandestimatedafnetransformationparameters) yieldedbystandardhistogrammingandtheisocontourmethod,bothfor64bins andanoiseofvariance1.WefoundthatthedifferenceintheerrorvaluesforMI,as computedusingstandardhistogrammingandouriso-contourtechnique,wasstatistically signicant,asascertainedatalevelof0.01. 57

PAGE 58

W ealsoperformedexperimentsondeterminingtheangleofrotationusinglarger imageswithvaryinglevelsofnoise( =0.05,0.2,1 ).ThesameBrainwebimages, asmentionedbefore,wereused,exceptthattheiroriginalsizeof 183 219 was retained.Forabincountupto128,all/mostmethodsperformedquitewell(usinga brute-forcesearch)evenunderhighnoise.Howeverwithalargebincount(256bins), thenoiseresistanceofourmethodstoodout.Theresultsofthisexperimentwith differentmethodsandundervaryingnoisearepresentedinTables 3-4, 3-5 and 3-6. 3.3.2RegistrationofMultipleImagesin2D Theimagesusedwerepre-registeredMR-PD,MR-T1andMR-T2slices(from Brainweb)ofsizes90x109.Thelattertwowererotatedby 1 =20 andby 2 =30 respectively(seeFigure 3-3).Fordifferentnoiselevelsandintensityquantizations, asetofexperimentswasperformedtooptimallyalignthelattertwoimageswiththe formerusingmodiedmutualinformation(MMI)anditsnormalizedversion(MNMI)as criteria.Thesecriteriawerecalculatedusingourarea-basedmethodaswellassimple histogrammingwithbilinearinterpolation.Therangeofangleswasfrom 1 to 40 in stepsof 1 .Theestimatedvaluesof 1 and 2 arepresentedinTable 3-7. 3.3.3RegistrationofVolumeDatasets Experimentswereperformedonsub-volumesofsize 41 41 41 fromMR-PD andMR-T2datasetsfromtheBrainwebsimulator[ 19].TheMR-PDportionwaswarped by 20 aboutthe Y aswellas Z axes.Abrute-forcesearch(from5to 35 instepsof 1 ,withajointPMFof 64 64 bins)wasperformedsoastooptimallyregisterthe MR-T2volumewiththepre-warpedMR-PDvolume.ThePMFwascomputedboth using LengthProb aswellasusingsimplehistogramming,andusedtocomputethe MI/NMIjustasbefore.Thecomputedvalueswerealsoplottedagainstthetwoanglesas indicatedinthetoprowofFigure 3-4.Astheplotsindicate,boththetechniquesyielded theMIpeakatthecorrectpointinthe Y Z plane,i.e.at 20 ,20 .Whenthesame experimentswererunusing VolumeProb,weobservedthatthejointPMFcomputation 58

PAGE 59

f orthesameintensityquantizationwasmorethantentimesslower.Similarexperiments wereperformedforregistrationofthreevolumedatasetsin3D,namely 41 41 41 sub-volumesofMR-PD,MR-T1andMR-T2datasetsfromBrainweb.Thethreedatasets werewarpedthrough )Tj /T1_1 11.955 Tf (2 )Tj /T1_1 11.955 Tf (21 and )Tj /T1_1 11.955 Tf (30 aroundthe X axis.Abruteforcesearchwas performedsoastooptimallyregisterthelattertwodatasetswiththeformerusingMMI astheregistrationcriterion.JointPMFsofsize 64 64 64 werecomputedandthese wereusedtocomputetheMMIbetweenthethreeimages.TheMMIpeakoccurred whentheseconddatasetwaswarpedthrough 2 =19 andthethirdwaswarped through 3 =28 ,whichisthecorrectoptimum.TheplotsoftheMIvaluescalculatedby simplehistogrammingand 3DPointProb versusthetwoanglesareshowninFigure 3-4 (bottomrow)respectively. ThenextexperimentwasdesignedtochecktheeffectofzeromeanGaussian noiseontheaccuracyofafneregistrationofthesamedatasetsusedintherst experiment,usinghistogrammingand LengthProb.AdditiveGaussiannoiseofvariance 2 wasaddedtotheMR-PDvolume.Then,theMR-PDvolumewaswarpedbya 4 4 afnetransformationmatrix(expressedinhomogeneouscoordinatenotation)given as A = SHR z R y R x T where R z R y and R x representrotationmatricesaboutthe Z Y and X axesrespectively, H isashearmatrixand S representsadiagonalscaling matrixwhosediagonalelementsaregivenby 2 s x 2 s y and 2 s z .(Atranslationmatrix T isincludedaswell.Formoreinformationonthisparameterization,pleasesee[ 42].) TheMR-T1volumewasthenregisteredwiththeMR-PDvolumeusingacoordinate descentonallparameters.Theactualtransformationparameterswerechosentobe 7 forallanglesofrotationandshearing,and0.04for s x s y and s z .Forasmallernumber ofbins(32),itwasobservedthatboththemethodsgavegoodresultsunderlownoise andhistogrammingoccasionallyperformedbetter.Table 3-8 showstheperformance ofhistogramsand LengthProb for128bins,over10differentnoisetrials.Summarily, weobservedthatourmethodproducedsuperiornoiseresistanceascomparedto 59

PAGE 60

histog rammingwhenthenumberofbinswaslarger.Toevaluatetheperformanceonreal data,wechosevolumesfromtheVisibleHumanDataset 2 (Male).Wetooksub-volumes ofMR-PDandMR-T1volumesofsize 101 101 41 (slices1110to1151).The twovolumeswerealmostincompleteregistration,sowewarpedtheformerusingan afnetransformationmatrixwith 5 forallanglesofrotationandshearing,andvalue of0.04for s x s y and s z resultinginamatrixwithsumofabsolutevalues3.6686.A coordinatedescentalgorithmfor12parameterswasexecutedonmutualinformation calculatedusing LengthProb soastoregistertheMR-T1datasetwiththeMR-PD dataset,producingaregistrationerrorof0.319(seeFigure 3-5). 3.4Discussion Thusfarinthischapterandthepreviousone,wehavepresentedanewdensity estimatorwhichisessentiallygeometricinnature,usingcontinuousimagerepresentations andtreatingtheprobabilitydensityasareasandwichedbetweeniso-contoursat intensitylevelsthatareinnitesimallyapart.Weextendedtheideatothecaseofjoint densitybetweentwoimages,bothin2Dand3D,asalsothecaseofmultipleimages in2D.Empirically,weshowedsuperiornoiseresistanceonregistrationexperiments involvingrotationsandafnetransformations.Furthermore,wealsosuggesteda faster,biasedalternativebasedoncountingpixelintersectionswhichperformswell, andextendedthemethodtohandlevolumedatasets.Therelationshipbetweenour techniquesandhistogrammingwithmultiplesub-pixelsampleswasalsodiscussed. Thereareafewclaricationsinorderasfollows: 1. Comparisontohistogrammingonanup-sampledimage Ifanimageis up-sampledseveraltimesandhistogrammingisperformedonit,therewillbemore samplesforthehistogramestimate.Atatheoreticallevel,though,thereisstill theissueofnotbeingabletorelatethenumberofbinstotheavailablenumberof 2 Obtained fromtheVisibleHumanProject R r (http://www.nlm.nih.gov/research/ visible/getting_data.html ). 60

PAGE 61

samples .Furthermore,itisrecommendedthattherateofincreaseinthenumber ofbinsbelessthanthesquarerootofthenumberofsamplesforcomputingthe jointdensitybetweentwoimages[ 16],[ 43].Ifthereare d imagesinall,thenumber ofbinsoughttobelessthan N 1 d where N isthetotalnumberofpixels,orsamples tobetaken[ 16],[ 43 ].Considerthatthiscriterionsuggestedthat N sampleswere enoughforajointdensitybetweentwoimageswith bins.Supposethatwenow wishedtocomputeajointdensitywith binsfor d imagesofthesamesize.This wouldrequiretheimagestobeup-sampledbyafactorofatleast N d )Tj /T1_2 5.978 Tf 5.75 0 Td (2 2 which isexponentialinthenumberofimages.Oursimplearea-basedmethodclearly avoidsthisproblem. 2. Choiceofinterpolant Wechosea(piece-wise)linearinterpolantforthesakeof simplicity,thoughinprincipleanyotherinterpolantcouldbeused.Itistruethatwe aremakinganassumptiononthecontinuityoftheintensityfunctionwhichmay beviolatedinnaturalimages.However,givenagoodenoughresolutionofthe inputimage,interpolationacrossadiscontinuitywillhaveanegligibleimpacton thedensityasthosediscontinuitiesareessentiallyameasurezeroset.Onecould evenincorporateanedge-preservinginterpolant[ 44]byrunningananisotropic diffusiontodetectthediscontinuitiesandthentakingcarenottointerpolateacross thetwosidesofanedge. 3. Non-differentiability ThePDFestimatesofourmethodarenotdifferentiable, whichcanposeaproblemfornon-rigidregistrationapplications.Differentiability couldbeachievedbytting(say)asplinetotheobtainedprobabilitytables. However,thisagainrequiressmoothingthedensityestimateinamannerthatis nottiedtotheimagegeometry.Hence,thisgoesagainstthephilosophyofour approach.Forpracticalorempiricalreasons,however,thereisnoreasonwhyone shouldnotexperimentwiththis.Moreover,currently,wedonothaveaclosedform expressionforourdensityestimate.Expressingthemarginalandjointdensities solelyintermsoftheparametersofthechosenimageinterpolantisachallenging problem. Table3-1.Averageandstd.dev.oferrorindegrees(absolutedifferencebetweentrue andestimatedangleofrotation)forMIusingParzenwindows Noise VarianceAvg.ErrorStd.Dev.ofError 0.050.06670.44 0.20.33 0.8 1 3.6 3 2 4.7 12.51 61

PAGE 62

A B C D E F G H I J K L Figure 3-1.Graphsshowingtheaverageerror A (i.e.abs.diff.betweentheestimated andthetrueangleofrotation)anderrorstandarddeviation S withMIasthe criterionfor16,32,64,128bins(row-wise)withanoiseof0.05[from(A)to (D)],withanoiseof0.2[from(E)to(H)]andwithanoiseof1[from(I)to(L)]. Insideeachsub-gure,error-barsareplottedforsixdiff.methods,inthefoll. order:SimpleHistogramming,Iso-contours,PVI,Cubic, 2DPointProb Histogrammingwith 10 6 samples.Error-barsshowthevaluesof A )Tj /T1_1 11.955 Tf 11.96 0 Td (S A, A + S .If S issmall,onlythevalueof A isshown. 62

PAGE 63

Figure 3-2.Firsttwo:MIfor32,128binswithnoiselevelof0.05;Thirdandfourth:witha noiselevelof0.2;Fifthandsixth:withanoiselevelof1.0.Inallplots,dark blue:iso-contours,cyan: 2DPointProb,black:cubic,red:simple histogramming,green:PVI. (Note:Theseplotsshouldbeviewedincolor.) 63

PAGE 64

A B C Figure 3-3.MRslicesofthebrain(A)MR-PDslice,(B)MR-T1slicerotatedby20 degrees,(C)MR-T2slicerotatedby30degrees A B C D Figure 3-4.MIcomputedusing(A)histogrammingand(B) LengthProb (plottedversus Y and Z );MMIcomputedusing(C)histogrammingand(D) 3DPointProb (plottedversus 2 and 3 ) 64

PAGE 65

Figure 3-5.TOPROW:originalPDimage(left),warpedT1image(middle),image overlapbeforeregistration(right),MIDDLEROW:PDimagewarpedusing predictedmatrix(left),warpedT1image(middle),imageoverlapafter registration(right).BOTTOMROW:PDimagewarpedusingidealmatrix (left),warpedT1image(middle),imageoverlapafterregistrationintheideal case(right) 65

PAGE 66

T able3-2.Averagevalueandvarianceofparameters s and t predictedbyvarious methods(32and64bins,noise =0.2 );Groundtruth: =30, s = t = )Tj /T1_3 11.955 Tf (0.3 Method Bins st MI Hist3230,0-0.3,0-0.3,0 NMIHist3230,0-0.3,0-0.3,0 MIIso3230,0-0.3,0-0.3,0 NMIIso3230,0-0.3,0-0.3,0 MIPVI3230,0-0.3,0-0.3,0 NMIPVI3230,0-0.3,0-0.3,0 MISpline3230.8,0.2-0.3,0-0.3,0 NMISpline3230.6,0.7-0.3,0-0.3,0 MI 2DPt. 3230,0-0.3,0-0.3,0 NMI 2DPt. 3230,0-0.3,0-0.3,0 MIHist6429.2,49.70.4,00.27,0.07 NMIHist6428.8,44.90.4,00.33,0.04 MIIso6430,0-0.3,0-0.3,0 NMIIso6430,0-0.3,0-0.3,0 MIPVI6430,0-0.3,0-0.3,0 NMIPVI6430,0-0.3,0-0.3,0 MISpline6424,21.50.4,00.33,0.04 NMISpline6424.3,20.90.4,00.33,0.04 MI 2DPt. 6430,0-0.3,0-0.3,0 NMI 2DPt. 6430,0-0.3,0-0.3,0 66

PAGE 67

T able3-3.Averagevalueandvarianceofparameters s and t predictedbyvarious methods(32and64bins,noise =1);Groundtruth: =30, s = t = )Tj /T1_3 11.955 Tf (0.3 Method Bins s t MI Hist3233.7,18.10.4,00.13,0.08 NMIHist3234.3,15.90.4,00.13,0.08 MIIso3230,0.06-0.3,0-0.3,0 NMIIso3230,0.06-0.3,0-0.3,0 MIPVI3228.1,36.250.26,0.080.19,0.1 NMIPVI3228.1,36.250.3,0.050.21,0.08 MISpline3230.3,49.390.4,00.09,0.1 NMISpline3231.2,48.020.4,00.05,0.1 MI 2DPt. 3230.3,0.22-0.3,0-0.3,0 NMI 2DPt. 3230.3,0.22-0.3,0-0.3,0 MIHist6427.5,44.650.4,00.25,0.08 NMIHist6427,43.860.4,00.246,0.08 MIIso6430.5,0.12-0.27,0.035-0.28,0.02 NMIIso6431.2,0.1-0.27,0.058-0.28,0.02 MIPVI6426.2,36.960.4,00.038,0 NMIPVI6426.8,41.80.4,00.038,0 MISpline6425.9,40.240.4,00.3,0.06 NMISpline6425.7,26.70.4,00.3,0.06 MI 2DPt. 6430.5,0.25-0.24,0.0197-0.23,0.01 NMI 2DPt. 6430.5,0.25-0.26,0.0077-0.22,0.02 T able3-4.Averageerror(absolutediff.)andvarianceinmeasuringangleofrotation usingMI,NMIcalculatedwithdifferentmethods,noise =0.05 Method 128bins256bins MI Hist.0,00.13,0.115 NMIHist.0,00.067,0.062 MIIso.0,00,0 NMIIso.0,00,0 MIPVI0,00,0 NMIPVI0,00,0 MISpline0,00.33,0.22 NMISpline0,00.33,0.22 MI2DPt.0,00,0 NMI2DPt.0,00,0 67

PAGE 68

T able3-5.Averageerror(absolutediff.)andvarianceinmeasuringangleofrotation usingMI,NMIcalculatedwithdifferentmethods,noise =0.2 Method 128bins256bins MI Hist.0.07,0.1960.2,0.293 NMIHist.0.07,0.1960.13,0.25 MIIso.0,00,0 NMIIso.0,00,0 MIPVI0,00,0 NMIPVI0,00,0 MISpline2.77,104.77,10 NMISpline2.77,1018,0.06 MI2DPt.0,00,0 NMI2DPt.0,00,0 T able3-6.Averageerror(absolutediff.)andvarianceinmeasuringangleofrotation usingMI,NMIcalculatedwithdifferentmethods,noise =1 Method 128bins256bins MI Hist.1.26,3127.9,3.1 NMIHist.1.2,3028,3.3 MIIso.0,00,0 NMIIso.0,00,0 MIPVI0,0.2626.9,14.3 NMIPVI0,0.2626.8,14.5 MISpline10,0.218,0.33 NMISpline9.8,0.1518,0.06 MI2DPt.0.07,0.060.07,0.06 NMI2DPt.0.267,0.320.07,0.06 T able3-7.Threeimagecase:anglesofrotationusingMMI,MNMIcalculatedwiththe iso-contourmethodandsimplehistograms,fornoisevariance =0.05,0.1,1 (Groundtruth 20 and 30 ) Noise VarianceMethod32bins64bins 0.05MMI Hist.21,3022,31 0.05MNMIHist.21,3022,31 0.05MMIIso.20,3020,30 0.05MNMIIso.20,3020,30 0.2MMIHist.15,3140,8 0.2MNMIHist.15,3140,8 0.2MMIIso.22,2920,30 0.2MNMIIso.22,2920,30 1MMIHist.40,938,4 1MNMIHist.40,934,4 1MMIIso.22,3035,23 1MNMIIso.22,3040,3 68

PAGE 69

T able3-8.Error(average,std.dev.)validatedover10trialswith LengthProb and histogramsfor128bins; R referstotheintensityrangeoftheimage Noise LevelErrorwith LengthProb Errorwithhistograms 0 0.09, 0.02 0.088,0.009 p 50 R 0.135, 0.029 0.306,0.08 p 100R 0.5, 0.36 1.47,0.646 p 150R 0.56, 0.402 1.945,0.56 69

PAGE 70

CHAPTER 4 APPLICATIONTOIMAGEFILTERING 4.1Introduction Filteringofimageshasbeenoneofthemostfundamentalproblemsstudiedin low-levelvisionandsignalprocessing.Overthepastdecades,severaltechniquesfor datalteringhavebeenproposedwithimpressiveresultsonpracticalapplications inimageprocessing.Asstraightforwardimagesmoothingisknowntobluracross signicantimagestructures,severalanisotropicapproachestoimagesmoothinghave beendevelopedusingpartialdifferentialequations(PDEs)withstoppingtermsto controlimagediffusionindifferentdirections[ 44].ThePDE-basedapproacheshave beenextendedtolteringofcolorimages[ 45]andchromaticityvectorelds[46 ].Other popularapproachestoimagelteringincludeadaptivesmoothing[ 47 ]andkernel densityestimationbasedalgorithms[ 48 ].Allthesemethodsproducesomesortof weightedaverageoveranimageneighborhoodforthepurposeofdatasmoothing, wheretheweightsareobtainedfromthedifferencebetweentheintensityvaluesofthe centralpixelandthepixelsintheneighborhood,orfromthepixelgradientmagnitudes. Beyondthis,techniquessuchasbilateralltering[ 49]produceaweightedcombination thatisalsoinuencedbytherelativelocationofthecentralpixelandtheneighborhood pixels.Thehighlypopularmean-shiftprocedure[ 50],[ 51]isgroundedinsimilarideas asbilateralltering,withtheadditionthattheneighborhoodaroundapixelisallowed tochangedynamicallyuntilaconvergencecriterionismet.Theauthorsprovethatthis convergencecriterionisequivalenttondingthemodeofalocaldensitybuiltjointlyon thespatialparameters(image domain )andtheintensityparameters(image range ). Inthischapter,wepresentanewapproachtodatalteringthatisrootedinsimple yetelegantgeometricintuitions.Atthecoreofourtheoryistherepresentationofan imageasafunctionthatisatleast C 0 continuouseverywhere.Akeypropertyofthe imagelevelsetsisusedtodrivethediffusionprocess,whichwethenincorporateina 70

PAGE 71

fr ameworkofdynamicneighborhoods ala mean-shift.Wedemonstratetherelationship ofourmethodtomanyoftheexistinglteringtechniquessuchasthosedrivenby kerneldensityestimation.Theefcacyofourapproachissupportedwithextensive experimentalresults.Tothebestofourknowledge, oursistherstattempttoexplicitly utilizeimagegeometry(intermsofitslevelcurves) forthisparticularapplication. Thischapterisorganizedasfollows.Section2presentsthekeytheoretical framework.Section3presentsextensionstoourtheory.Insection4,wepresent therelationshipbetweenourmethodandmean-shift.Extensiveexperimentalresultsare presentedinsection5,andwepresentfurtherdiscussionsandconclusionsinsection6. Allormostofthematerialcontainedinthischapterhasbeenpreviouslypublishedbythe authorin[ 52 ] 1 4.2Theory Consideranimageoveradiscretedomain n= f1,..., H gf1,..., W g where theintensityofeachdiscretelocation (x y ) isgivenby I ( x y ).Moreoverconsider aneighborhood N (x i y i ) aroundthepixel (x i y i ).Itiswell-knownthatasimple averagingofallintensityvaluesin N (x i y i ) willbluredges,soaweightedcombinationis calculated,wheretheweightofthe j th pixelisgivenby w (1) (x j y j )= g (jI (x i y i ) )Tj /T1_5 11.955 Tf 9.35 0 Td (I (x j y j )j) foranon-increasingfunction g (.) tofacilitateanisotropicdiffusion,withcommon examplesbeing g (z )= e )Tj /T1_6 5.978 Tf 8.31 3.26 Td (z 2 2 or g (z ) = 2 2 + z 2 ortheirtruncatedversions.Thisapproach isakintothekerneldensityestimation(KDE)approachproposedin[ 48],wherethe 1 P artsofthecontentofthischapterhavebeenreprintedwithpermissionfrom:A. Rajwade,A.BanerjeeandA.Rangarajan,`ImageFilteringbyLevelCurves',Energy MinimizationMethodsinComputerVisionandPatternRecognition(EMMCVPR),2009, pages359-372. c r2009,SpringerVerlag. 71

PAGE 72

ltered valueofthecentralpixeliscalculatedas: ^ I (x i y i )= X (x j ,y j )2N (x i y i ) I (x j y j )K (I (x j y j ) )Tj /T1_2 11.955 Tf 11.96 0 Td (I (x i y i ); W r ) X (x j y j )2N (x i y i ) K (I (x j y j ) )Tj /T1_2 11.955 Tf 11.96 0 Td (I (x i y i ); W r ) (4) Here thekernel K centeredat I (x i y i ) (andparameterizedby W r )isrelatedtothe function g anddeterminestheweights.Themajorlimitationsofthekernelbased approachtoanisotropicdiffusionarethattheentireprocedureissensitivetothe parameter W r andthesizeoftheneighborhood,andmightsufferfromasmall-sample sizeproblem.Furthermore,inadiscreteimplementation,foranyneighborhoodsize largerthan 3 3 ,theproceduredependsonlyontheactualpixelvaluesanddoesnot accountforanygradientinformation,whereasinalteringapplication,itisdesirable toplacegreaterimportanceonthoseregionsoftheneighborhoodwherethegradient valuesarelower. Nowconsiderthattheimageistreatedasacontinuousfunction I (x y ) ofthespatial variables,byinterpolatinginbetweenthepixelvalues.Theearlierdiscreteaverageis replacedbythefollowingcontinuousaveragetoupdatethevalueat (x i y i ): ^ I (x i y i )= ZZ N ( x i ,y i ) I (x y )g (jI (x y ) )Tj /T1_2 11.955 Tf 11.95 0 Td (I (x i y i )j )dxdy Z Z N ( x i ,y i ) g (jI (x y ) )Tj /T1_2 11.955 Tf 11.95 0 Td (I (x i y i )j )dxdy (4) Theaboveformulaisusuallynotavailableinclosedform.Wenowshowaprincipled approximationtothisformula,byresortingtogeometricintuition.Imagineacontourmap ofthisimage,withmultipleiso-intensitylevelcurves C m = f (x y )jI (x y )= m g (referred tohenceforthas`levelcurves')separatedbyanintensityspacingof .Considera portionofthiscontourmapinasmallneighborhoodcenteredaroundthepoint (x i y i ) (seeFigure 4-1A).Thoseregionswherethelevelcurves(separatedbyaxedintensity spacing)arecloselypackedtogethercorrespondtothehigher-gradientregionsofthe neighborhood,whereasinlower-gradientregionsoftheimage,thelevelcurveslie 72

PAGE 73

f arawayfromoneanother.NowasseeninFigure 4-1A,thiscontourmapinducesa tessellationoftheneighborhoodintosome K facets,whereeachfacetcorrespondstoa regioninbetweentwolevelcurvesofintensity m and m +,boundedbytherimofthe neighborhood.Letthearea a k ofthe k th facetofthistessellationbedenotedas a k .Now, ifwemake sufcientlysmall,wecanregardeventhefacetsfromhigh-gradientregions ashavingconstantintensityvalue I k = m .Thisnowleadstothefollowingweighted averageinwhichtheweightingfunctionhasaverycleangeometricinterpretation,unlike thearbitrarychoicefor w (1) intheprevioustechnique: ^ I (x i y i )= K X k =1 a k I k g (jI k )Tj /T1_1 11.955 Tf 11.96 0 Td (I (x i y i )j) K X k =1 a k g (jI k )Tj /T1_1 11.955 Tf 11.95 0 Td (I (x i y i )j ) (4) As thenumberoffacetsistypicallymuchlargerthanthenumberofpixels,andgiven thefactthatthefacetshavearisenfromalocallysmoothinterpolationmethodtoobtain acontinuousfunctionfromtheoriginaldigitalpixelvalues,wenowhaveamorerobust averagethanthatprovidedbyEquation 4.Tointroduceanisotropy,westillrequirethe stoppingterm g (jI k )Tj /T1_1 11.955 Tf 12.15 0 Td (I (x i y i )j) topreventsmearingacrosstheedge,justasinEquation 4. Equation 4 essentiallyperformsanintegrationoftheintensityfunctionoverthe domain N ( x i y i ).Ifwenowperformachangeofvariablestransformingtheintegralon (x y ) toanintegralovertherangeoftheimage,weobtaintheexpression ^ I (x i y i )= ZZ N (x i y i ) I (x y )w (1) (x y )dxdy Z Z N (x i y i ) w (1) (x y )dxdy = Z q = q 2 q =q 1 Z C (q ) qg (jq )Tj /T1_1 11.955 Tf 11.96 0 Td (I (x i y i )j) jrI j dldq Z q =q 2 q = q 1 Z C (q ) g (jq )Tj /T1_1 11.955 Tf 11.96 0 Td (I (x i y i )j) jrI j dldq = lim !0 q 2 X = q 1 Z + q = Z C (q ) qg (jq )Tj /T1_1 11.955 Tf 11.96 0 Td (I ( x i y i ) j) jr I j dldq lim !0 q 2 X = q 1 Z q = + q = Z C (q ) g (jq )Tj /T1_1 11.955 Tf 11.96 0 Td (I (x i y i )j) jr I j dldq (4) 73

PAGE 74

where C (q ) = N (x i y i ) \ f )Tj /T1_6 7.97 Tf (1 (q ), q 1 =inf fI ( x y )j (x y ) 2N (x i y i )g, q 2 = supf I (x y )j(x y ) 2N (x i y i )g and l standsforatangentalongthecurve f )Tj /T1_6 7.97 Tf 6.59 0 Td (1 (q ). Thisapproachisinspiredbythesmoothco-areaformulaforregularfunctions[ 53]which isgivenas Z n ( u )jru jdxdy = Z +1 Length (r q )(q )dq (4) where r q isthelevelsetof u attheintensity q and (u ) representsafunctionof u Notethattheterm R q = + q = R C (q ) dldq jr I j in Equation 4 actuallyrepresentstheareain N (x i y i ) thatistrappedbetweentwocontourswhoseintensityvaluediffersby .Our workdescribedinthepreviouschaptersconsidersthisquantitywhennormalized by jn j tobeactuallyequaltotheprobabilitythattheintensityvalueliesintherange [ +] .Bearingthisinmind,Equation 4 nowacquiresthefollowing probabilistic interpretation: ^ I (x i y i )= q 2 X = q 1 Pr( < I < +jN ) g (j )Tj /T1_3 11.955 Tf 11.96 0 Td (I (x i y i )j) q 2 X = q 1 Pr ( < I < +jN )g (j )Tj /T1_3 11.955 Tf 11.96 0 Td (I ( x i y i )j) (4) As 0,thisproducesanincreasinglybetterapproximationtoEquation 4. Itshouldbepointedoutthatthereexistmethodssuchasadaptiveltering[47],[ 54] inwhichtheweightsinEquation 4 areobtainedas w (2) (x j y j )= g (jr I ( x j y j )j).These methodsplacemoreimportanceonthelower-gradient pixels oftheneighborhood,but donotexploitlevelcurverelationshipsinthewaywedo,andthechoiceoftheweighting functiondoesnothavethegeometricinterpretationthatexistsinourtechnique. Moreovertheoriginalformulationin[ 47]wasdesignedfor 3 3 neighborhoods.For largerneighborhoods,thegradient-basedtermswillhavetobeaugmentedwithan intensity-basedtermtopreventblurringacrossedges. TherealsoexistsanextensiontothestandardneighborhoodlterinEquation 4 reportedin[55 ],whichperformsaweightedleastsquarespolynomialttothe 74

PAGE 75

intensity values(ofthepixels)intheneighborhoodofalocation (x y ).Thevalueof thispolynomialat (x y ) isthenconsideredtobethesmoothedintensityvalue.This techniquediffersfromtheonewepresenthereintwofundamentalways.Unlikeour method,itdoesnotuseareasbetweenlevelsetsasweightstoexplicitlyperforma weightedaveraging.Secondlyasprovedin[ 55 ],itslimitingbehaviorwhen W r 0 and jN (x y )j! 0 resemblesthatofthegeometricheatequationwithalinearpolynomial, andresembleshigherorderPDEswhenthedegreeofthepolynomialisincreased.Our methodisthetruecontinuousformoftheKDE-basedlterfromEquation 4.This KDE-basedlterbehaveslikethePerona-Maliklter,asprovedin[ 55]. 4.3ExtensionsofOurTheory 4.3.1ColorImages Wenowextendourtechniquetocolor(RGB)images.Consideracolorimage denedas I (x y )=( R (x y ), G (x y ), B (x y )):n !R 3 where n R 2 .Incolor images,thereisnoconceptofasingleiso-contourwithconstantvaluesofallthree channels.Henceitismoresensibletoconsideranoverlayoftheindividualiso-contours oftheR,GandBchannels.Thefacetsarenowinducedbyatessellationinvolvingthe intersectionofthreeiso-contoursetswithinaneighborhood,asshowninFigure 4-1B. Eachfacetrepresentsthoseportionsoftheneighborhoodforwhich R < R (x y ) < R + R G < G (x y ) < G + G B < B (x y ) < B + B .Theprobabilistic interpretationfortheupdateontheR,G,Bvaluesisasfollows ^ R ( x i y i ), ^ G (x i y i ), ^ B (x i y i )= X ~ f Pr[ ~ f< ( R G B ) < ~ f + ~ jN ) ~ f g ( R G B ) X ~ f Pr [ ~ f< (R G B ) < ~ f + ~ jN ) g (R G B ) where ~ f =( R G B ), ~ =( R G B ) and g (R G B )= g (j R )Tj /T1_2 11.955 Tf 12.42 0 Td (R (x i y i )j + j G )Tj /T1_2 11.955 Tf -442.29 -23.91 Td (G ( x i y i )j + jB )Tj /T1_2 11.955 Tf 12.35 0 Td (B ( x i y i )j).Notethatinthiscase, I (x y ) isafunctionfromasubsetof R 2 to R 3 ,andhencethethree-dimensionaljoint density isill-denedinthesensethat itisdenedstrictlyona2Dsubspaceof R 3 .Howevergiventhattheimplementation 75

PAGE 76

considers jointcumulativeintervalmeasures,thisdoesnotposeanyproblemina practicalimplementation.WewishtoemphasizethattheaveragingoftheR,G,Bvalues isperformedinastrictlycoupledmanner,allaffectedbythe joint cumulativeinterval measure. 4.3.2ChromaticityFields Previousresearchonlteringchromaticitynoise(whichaffectsonlythedirection andnotthemagnitudeoftheRGBvaluesatimagepixels)includestheworkin[ 46 ] usingPDEsspeciallytunedforunit-vectordata,andtheworkin[ 48 ](page142)using kerneldensityestimationfordirectionaldata.Themorerecentworkonchromaticity lteringin[ 56 ]actuallytreatschromaticityvectorsaspointsonaGrassmannmanifold G 1,3 asopposedtotreatingthemaspointson S 2 ,whichistheapproachpresentedhere andin[ 48 ]and[ 46]. Weextendourtheoryfromtheprevioussectiontounitvectordataandincorporate itinamean-shiftframeworkforsmoothing.Let I ( x y ):n !R 3 betheoriginal RGBimage,andlet J (x y ):n !S 2 bethecorrespondingeldofchromaticity vectors.Apossibleapproachwouldinvolveinterpolatingthechromaticityvectors bymeansofcommonlyusedsphericalinterpolantstocreateacontinuousfunction, followedbytracingthelevelcurvesoftheindividualunit-vectorcomponents ~ v (x y )= (v 1 (x y ), v 2 ( x y ), v 3 (x y )) andcomputingtheirintersection.Howeverforeaseof implementationforthisparticularapplication,weresortedtoadifferentstrategy.Ifthe intensityintervals ~ =( R G B ) arechosentobeneenough,theneachfacet inducedbyatessellationthatusesthelevelcurvesoftheR,GandBchannelvalues, canberegardedashavingaconstantcolorvalue,andhencethechromaticityvector valueswithinthatfacetcanberegardedas(almost)constant.Thereforeitispossible tousejusttheR,G,Blevelcurvesforthetaskofchromaticitysmoothingaswell.The updateequationisverysimilartoEquation 4 withtheR,G,Bvectorsreplacedbytheir unitnormalizedversions.Howeverastheaveragingprocessdoesnotpreservetheunit 76

PAGE 77

nor m,theaveragedvectorneedstoberenormalizedtoproducethesphericalweighted mean. 4.3.3Gray-scaleVideo Forthepurposeofthisapplication,thevideoistreatedasasingle3Dsignal (volume).Theextensioninthiscaseisquitestraightforward,withtheareasbetween levelcurvesbeingreplacedbyvolumesbetweenthelevelsurfacesatnearbyintensities. Howeverwetakeintoaccountthecausalityfactorindeningthetemporalcomponentof theneighborhoodaroundapixel,byperformingtheaveragingateachpixeloverframes onlyfromthepast. 4.4LevelCurveBasedFilteringinaMeanShiftFramework Alltheabovetechniquesarebasedonanaveragingoperationoveronlytheimage intensities(i.e.intherangedomain).Ontheotherhand,techniquessuchasbilateral ltering[ 49]orlocalmode-nding[57]combinebothrangeandspatialdomain,thus usingweightsoftheform w j = g (s ) ((x i )Tj /T1_2 11.955 Tf 12.5 0 Td (x j ) 2 +( y i )Tj /T1_2 11.955 Tf 12.5 0 Td (y j ) 2 )g ( r ) (j(I (x i y i ) )Tj /T1_2 11.955 Tf 12.5 0 Td (I (x j y j ) j) in Equation 4,where g (s ) and g (r ) affectthespatialandrangekernelsrespectively.The mean-shiftframework[ 51 ]isbasedonsimilarprinciples,butchangesthelterwindow dynamicallyforseveraliterationsuntilitndsalocalmodeofthejointdensityofthe spatialandrangeparameters,estimatedusingkernelsbasedonthefunctions g (r ) and g (s ) .Ourlevelcurvebasedapproachtseasilyintothisframeworkwiththeaddition ofaspatialkernel.Onewaytodothiswouldbetoconsidertheimageasasurface embeddedin3D(aMongepatch),asdonein[ 58],andcomputeareasofpatchesin 3Dfortheprobabilityvalues.Howeversuchanapproachmaynotnecessarilyfavor thelowergradientareasoftheimage.Insteadweadoptanothermethodwhereinwe assumetwoadditionalfunctionsof x and y ,namely X ( x y )= x and Y (x y )= y .We computethejointprobabilitiesforarangeofvaluesofthejointvariable (X Y I ) by drawinglocallevelsetsandcomputingareasin2D.Assumingauniformspatialkernel for g (s ) withinaradius W s andarectangularkernelontheintensityfor g (r ) withthreshold 77

PAGE 78

v alue W r (thoughourcoretheoryisunaffectedbyotherchoices),wenowperformthe averagingupdateonthevector (X (x y ), Y (x y ), I (x y )),asopposedtomerelyon I (x y ) aswasdoneinEquation 4.Thisisgivenas: ( X (x i y i ), Y (x i y i ), ^ I (x i y i ))= K X k =1 (x k y k I k )a k g (r ) (jI k )Tj /T1_1 11.955 Tf 11.96 0 Td (I (x i y i )j) K X k =1 a k g (r ) ( jI k )Tj /T1_1 11.955 Tf 11.95 0 Td (I (x i y i )j) (4) In theaboveequation (x k y k ) standsforarepresentativepoint(say,thecentroid)of the k th facetoftheinducedtessellation 2 ,and K isthetotalnumberoffacetswithin thespeciedspatialradius.Notethattheareaofthe k th facet,i.e. a k ,canalsobe interpretedasthejointprobabilityfortheevent ~ x < X (x y ) < ~ x + x ,~ y < Y (x y ) < ~ y + y < I (x y ) < +,ifweassumeauniformdistributionoverthespatialvariables x and y .Here istheusualintensitybin-width, ( x y ) arethepixeldimensions, and (~ x ,~ y ) isapixelgrid-point.Themaindifferencebetweenourapproachandall theaforementionedrange-spatialdomainapproachesisthefactthatwenaturally incorporateaweightinfavorofthelower-gradientareasofthelterneighborhood. Hencethemean-shiftvectorinourcasewillhaveastrongertendencytomovetowards theregionoftheneighborhoodwherethelocalintensitychangeisaslowaspossible (evenifauniformspatialkernelisused).Moreoverjustlikeconventionalmeanshift, ouriterativeprocedureisguaranteedtoconvergetoamodeofthelocaldensityin anitenumberofsteps,byexploitingthefactthattheweightsateachpoint(i.e.the areasofthefacets)arepositive.HenceTheorem5of[ 50]canbereadilyinvoked. ThisisbecauseinEquation 4,thethresholdfunction g (r ) fortheintensityisthe rectangularkernel,andhencethecorrespondingupdateformulaisequivalenttoone 2 The notionofthecentroidwillbecomeclearerinSection 4.5. 78

PAGE 79

with aweightedrectangularkernel,withtheweightsbeingdeterminedbytheareasof thefacets. Amajoradvantageofourtechniqueisthattheparameter canbesettoassmall avalueasdesired(asitjustmeansthatmoreandmorelevelcurvesarebeingused), andtheinterpolationgivesrisetoarobustaverage.Thisisespeciallyusefulinthecase ofsmallneighborhoodsizes,astheintensityquantizationisnownomorelimitedbythe numberofavailablepixels.Inconventionalmean-shift,theproperchoiceofbandwidth isahighlycriticalissue,asveryfewsamplesareavailableforthelocaldensityestimate. Thoughvariablebandwidthproceduresformean-shiftalgorithmshavebeendeveloped extensively,theythemselvesrequireeitherthetuningofotherparametersusingrulesof thumb,orelsesomeexpensiveexhaustivesearchesfortheautomaticdeterminationof thebandwidth[ 59],[ 60].Althoughourmethoddoesrequiretheselectionof W s and W r thelteringresultsarelesssensitivetothechoiceoftheseparametersinourmethod thaninstandardmeanshift. 4.5ExperimentalResults Inthissectionwepresentexperimentalresultstocomparetheperformanceofour algorithminameanshiftframeworkw.r.t.conventionalkernel-basedmeanshift.Forour algorithm,weobtainacontinuousfunctionapproximationtothedigitalimage,bymeans ofpiecewiselinearinterpolantsttoatripleofintensityvaluesinhalf-pixelsoftheimage (inprinciple,wecouldhaveusedanyothersmoothinterpolant).Thecorrespondinglevel setsforsuchafunctionarealsoveryeasytotrace,astheyarejustsegmentswithin eachhalf-pixel.Thelevelsetsinduceapolygonaltessellation.Wechoosetosplitthe polygonsbythesquarepixelboundariesaswellasthepixeldiagonalsthatdelineatethe half-pixelboundaries,therebyconvexifyingallthepolygonsthatwereinitiallynon-convex (seeFigure 4-1C).Eachpolygoninthetessellationcannowbecharacterizedbythe x y coordinatesofitscentroid,theintensityvalueoftheimageatthecentroid,andthearea ofthepolygon.Thus,iftheintensityvalueatgridlocation x i y i istobesmoothed,we 79

PAGE 80

choose awindowofspatialradius W s andintensityradius W r around (x i y i I ( x i y i )), overwhichtheaveragingisperformed.Inotherwords,theaveragingisperformedonly overthoselocations x y forwhich (x )Tj /T1_1 11.955 Tf 11.18 0 Td (x i ) 2 +(y )Tj /T1_1 11.955 Tf 11.19 0 Td (y i ) 2 < W 2 s and jI (x y ) )Tj /T1_1 11.955 Tf 11.19 0 Td (I ( x i y i )j < W r Wewouldliketopointoutthat thoughtheinterpolantusedforcreatingthecontinuous imagerepresentationisindeedisotropicinnature,thisstilldoesnotmakeourltering algorithmisotropic .Thisisbecausepolygonalregions,whoseintensityvaluedoesnot satisfytheconstraint jI (x y ) )Tj /T1_1 11.955 Tf 12.02 0 Td (I (x i y i )j < W r ,donotcontributetotheaveragingprocess (seethestoppingterminEquation 4),andhencethecontributionfrompixelswith verydifferentintensityvalueswillbenullied. 4.5.1Gray-scaleImages Weranourlteringalgorithmoverfourarbitrarilychosenimagesfromthepopular Berkeleyimagedataset[ 61],andtheLenaimage.Toalltheseimages,zeromean Gaussiannoiseofvariance0.003(perunitgray-scalerange)wasadded.Theltering wasperformedusing W s = W r =3 forouralgorithmandcomparedtomean-shift usingGaussianandEpanechnikovkernelswiththesameparameter.Ourmethod producedsuperiorlteringresultstoconventionalmeanshiftwithbothGaussianand Epanechnikovkernels.TheresultsforourmethodandforGaussiankernelmeanshift aredisplayedinFigures 4-2 and 4-3 respectively.Thevisuallysuperiorappearance wasconrmedobjectivelywithmeansquarederror(MSE)valuesinTable 4-1. Itshould benotedthattheaimwastocompareourmethodtostandardmeanshiftfortheexact samesettingoftheparameters W r and W s ,astheyhavethesamemeaninginallthese algorithms.Althoughincreasingthevalueof W r willprovidemoresamplesforaveraging, thiswillallowmoreandmoreintensityvaluestoleakacrossedges. 4.5.2TestingonaBenchmarkDatasetofGray-scaleImages Furtherempiricalresultswithouralgorithm(using W S = W r =5)wereobtained onLansel'sbenchmarkdataset[ 62].Thedatasetcontainsnoisyversionsof13different images.Eachnoisyimageisobtainedfromoneofthreenoisemodels:additive 80

PAGE 81

Gaussian, Poisson,andmultiplicativenoisemodel,foroneofvedifferentvaluesof thenoisestandarddeviation 2f 5 255 10 255 15 255 20 255 25 255 g, leadingtoatotalof195images. Wereportdenoisingresultsonalltheseimageswithouttweakinganyparameters dependingonthenoisemodel(wechose W r = W s =5 forallimagesatallnoiselevels). TheaverageMSEandMSSIM(animagequalitymetricdenedin[ 63])areshownin theplotsinFigures 4-5 and 4-6.Wehavealsodisplayedthedenoisedversionsofa ngerprintimagefromthisdatasetunderthreedifferentvaluesof foradditivenoisein Figures 4-5 and 4-6. 4.5.3ExperimentswithColorImages Similarexperimentswererunoncoloredversionsofthesamefourimagesfromthe Berkeleydataset[ 61].TheoriginalimagesweredegradedbyzeromeanGaussiannoise ofvariance0.003(perunitintensityrange),addedindependentlytotheR,G,Bchannels. Forourmethod,independentinterpolationwasperformedoneachchannelandthejoint densitieswerecomputedasdescribedintheprevioussections.Levelsetsatintensity gapsof R = G = B =1 weretracedineveryhalfpixel.Experimentalresults werecomparedwithconventionalmeanshiftusingaGaussiankernel.Theparameters chosenforbothalgorithmswere W s = W r =6.Despitethedocumentedadvantagesof colorspacessuchasLab[ 48],allexperimentswereperformedintheR,G,Bspacefor thesakeofsimplicity,andalsobecausemanywell-knowncolordenoisingtechniques operateinthisspace[ 45].AsseeninFigures 4-7, 4-8 andTable 4-2,ourmethod producedbetterresultsthanGaussiankernelmeanshiftforthechosenparameter values. 4.5.4ExperimentswithChromaticityVectorsandVideo Twocolorimagesweresyntheticallycorruptedwithchromaticitynoisealteringjust thedirectionofthecolor-triplevector.TheseimagesareshowninFigures 4-9 and 4-10. TheseimageswerelteredusingourmethodandGaussiankernelmeanshiftwitha spatialwindowofsize W s =4 andachromaticitythresholdof W r =0.1 radians.Note 81

PAGE 82

that inthiscase,thedistancebetweentwochromaticityvectors ~ v 1 and ~ v 2 isdenedtobe thelengthofthearcbetweenthetwovectorsalongthegreatcirclejoiningthem,which turnsouttobe =cos )Tj /T1_3 7.97 Tf 6.59 0 Td (1 ~ v 1 T ~ v 2 .Thespecicexpressionforthejointspatial-chromaticity densityusingtheGaussiankernelwas e )Tj /T1_3 5.978 Tf 7.78 4.62 Td ((x )Tj /T1_6 5.978 Tf 5.75 0 Td (x i ) 2 +( y )Tj /T1_6 5.978 Tf 5.75 0 Td (y i ) 2 2 W 2 s e )Tj /T1_8 5.978 Tf 11.15 3.26 Td ( 2 2 W 2 i Thelteredimagesusing bothmethodsareshowninFigures 4-9 and 4-10.Despitethevisualsimilarityofthe output,ourmethodproducedamean-squarederrorof378and980.8,asopposedto 534.9and1030.7forGaussiankernelmeanshift. WealsoperformedanexperimentonvideodenoisingusingtheDavidsequence obtainedfrom http://www.cs.utoronto.ca/ ~ dross/ivt/ .Therst100framesfromthe sequencewereextractedandarticiallydegradedwithzeromeanGaussiannoiseof variance0.006.Twoframesofthecorruptedanddenoised(usingourmethod)sequence areshowninFigure 4-11,asalsoatemporalslicethroughtheentirevideosequence (forthetenthrowofeachframe).Forthisexperiment,thevalueof wassetto8inour method. 4.6Discussion Wehavepresentedanewmethodforimagedenoising,whoseprincipleisrooted inthenotionthatthelower-gradientportionsofanimageinsideaneighborhood aroundapixelshouldcontributemoretothesmoothingprocess.Thegeometryof theimagelevelsets(andthefactthatthespatialdistancebetweenlevelsetsisinversely proportionaltothegradientmagnitudes)isthedrivingforcebehindouralgorithm. Wehavelinkedourapproachtoexistingprobability-densitybasedapproaches,and ourmethodhastheadvantageofrobustdecouplingoftheedgedenitionparameter fromthedensityestimate.Insomesense,ourmethodcanbeviewedasacontinuous versionofmean-shift.Itshouldbenotedthatamodicationtostandardmean-shift basedonsimpleimageup-samplingusinginterpolationwillbeanapproximationto ourarea-basedmethod(giventhesameinterpolant).Wehaveperformedextensive experimentsongray-scaleandcolorimages,chromaticityeldsandvideosequences. 82

PAGE 83

T othebestofourknowledge,oursisthe rstpieceofworkondenoisingwhichexplicitly incorporatestherelationshipbetweenimagelevelcurvesanduseslocalinterpolationbetweenpixelvalues inordertoperformltering.Futureworkwillinvolvea moredetailedinvestigationintotherelationshipbetweenourworkandthatin[ 58 ],by computingtheareasofthecontributingregionswithexplicittreatmentoftheimage I (x y ) asasurfaceembeddedin3D.Secondly,wealsoplantodeveloptopologically inspiredcriteriatoautomatethechoiceofthespatialneighborhoodandtheparameter W r forcontrollingtheanisotropicsmoothing. Itshouldbenotedthatthemainaimofthischapterwastodemonstratetheeffect ofusinginterpolantinformationfordenoising.Ourcontributionsliewithinthemean shiftframework,andthereforewehaveperformedcomparisonswithothermethods thatliewithinthisframework.Forthisreason,wehavenotperformedexperimental comparisonswithsomeleadinglocalconvolutionapproacheslike[ 64]or[ 65]. A B C Figure 4-1.Imagecontourmapsinaneighborhood(A)withhighandlowgradient regionsinaneighborhoodaroundapixel(darkdot);(B)acontourmapofan RGBimageinaneighborhood;red,greenandbluecontourscorrespondto contoursoftheR,G,Bchannelsrespectivelyandthetessellationinducedby theabovelevel-curvepairscontains19facets;(C)Atessellationinducedby RGBlevelcurvepairsandthesquarepixelgrid 83

PAGE 84

T able4-1.MSEforlteredimagesusing(M1)=Ourmethodwith W s = W r =3,using (M2)=MeanshiftwithGaussiankernelswith W s = W r =3 and(M3)=Mean shiftwithGaussiankernelswith W s = W r =5.MSE=mean-squarederrorin thecorruptedimage.Intensityscaleisfrom0to255. Image M1M2M3MSE 1 110.95176.57151.27181.27 253.85170.18106.32193.5 3106.64185.15148.379191.76 4113.8184.77153.577190 Lena78.42184.16128.04194.82 T able4-2.MSEforlteredimagesusing(M1)=Ourmethodwith W s = W r =6,using (M2)=MeanshiftwithGaussiankernelswith W s = W r =6 and(M3)=Mean shiftwithEpanechnikovkernelswith W s = W r =6.MSE=mean-squared errorinthecorruptedimage.Intensityscaleisfrom0to255foreachchannel. Image M1M2M3MSE 1 319.88496.7547.9572.54 2354.76488.7543.4568.69 3129.12422.79525.48584.24 4306.14477.25526.8547.9 84

PAGE 85

Figure 4-2.Foreachimage,topleft:originalimage,topright:degradedimageswith zeromeanGaussiannoiseofstd.dev.0.003,bottomleft:resultsobtained byouralgorithm,andbottomright:meanshiftwithGaussiankernel(right column).Bothbothmethods, W s = W r =3 ; Viewedbestwhenzoomedin thepdfle 85

PAGE 86

Figure 4-3.Foreachimage,topleft:originalimage,topright:degradedimageswith zeromeanGaussiannoiseofstd.dev.0.003,bottomleft:resultsobtained byouralgorithm,andbottomright:meanshiftwithGaussiankernel(right column).Bothbothmethods, W s = W r =3 ; Viewedbestwhenzoomedin thepdfle 86

PAGE 87

Figure 4-4.Topleft:originalimage,topright:degradedimageswithzeromean Gaussiannoiseofstd.dev.0.003,bottomleft:resultsobtainedbyour algorithm,andbottomright:meanshiftwithGaussiankernel(rightcolumn). Bothbothmethods, W s = W r =3 ; Viewedbestwhenzoomedinthepdfle 87

PAGE 88

A B C D E F Figure 4-5.(A),(C)and(E):FingerprintimagesubjectedtoadditiveGaussiannoiseof std.dev. = 5 255 10 255 and 15 255 respectiv ely.(B),(D)and(F):Denoised versionsof(A),(C)and(E)respectively. Viewedbestwhenzoomedinthe pdfle(incolor). 88

PAGE 89

Figure 4-6.Aplotoftheperformanceofouralgorithmonthebenchmarkdataset, averagedoverallimagesfromeachnoisemodel(AdditiveGaussian (AWGN),multiplicativeGaussian(MWGN)andPoisson)andoverallve values,usingMSE(top)andMSSIM(bottom)asthemetric; Viewedbest whenzoomedinthepdfle(incolor) 89

PAGE 90

Figure 4-7.Foreachimage,topleft:originalimage,topright:degradedimageswith zeromeanGaussiannoiseofstd.dev.0.003,bottomleft:resultsobtained byouralgorithm,andbottomright:meanshiftwithGaussiankernel(right column);forbothmethods, W s = W r =3; Viewedbestwhenzoomedinthe pdfle 90

PAGE 91

Figure 4-8.Foreachimage,topleft:originalimage,topright:degradedimageswith zeromeanGaussiannoiseofstd.dev.0.003,bottomleft:resultsobtained byouralgorithm,andbottomright:meanshiftwithGaussiankernel(right column);Forbothmethods, W s = W r =3; Viewedbestwhenzoomedinthe pdfle 91

PAGE 92

Figure 4-9.Animageanditscorruptedversionobtainedbyaddingchromaticitynoise (topleftandtoprightrespectively).Resultsobtainedbylteringwithour method(bottomleft),andwithGaussianmeanshift(bottomright); Viewed bestwhenzoomedinthepdfle(incolor) 92

PAGE 93

Figure 4-10.Animageanditscorruptedversionobtainedbyaddingchromaticitynoise (topleftandtoprightrespectively).Resultsobtainedbylteringwithour method(bottomleft),andwithGaussianmeanshift(bottomright); Viewed bestwhenzoomedinthepdfle(incolor) 93

PAGE 94

Figure 4-11.Firsttwoimages:framesfromthecorruptedsequence.Thirdandfourth: imageslteredbyouralgorithm.Fifthandsixthimages:aslicethroughthe tenthrowofthecorruptedandlteredvideosequences;imagesare numberedlefttoright,toptobottom 94

PAGE 95

CHAPTER 5 ARELATEDPROBLEM:DIRECTIONALSTATISTICSINEUCLIDEANSPACE 5.1Introduction WhenthesamplesdonotresideinEuclideanspace,conventionaldensity estimationtechniquessuchasmixturesofGaussiansorkerneldensityestimation (KDE)usingGaussiankernelsarenotapplieddirectly.Forthespecialcasewhenthe dataresideon S n ,i.e.thesphereembeddedin R n ,thereexistsextensiveliteraturefrom theeldofdirectionalstatisticsthatissummarizedinseveralexemplarybookssuchas [66 ].Conventionally,forKDEormixturemodeldensityestimationofunitvectors,the Gaussiankernelhasbeenreplacedbyvon-Misesorvon-MisesFisher(voMF)kernels forcircularandsphericaldatarespectively.Thesecomputationaltechniqueshave beenappliedforsolvingnumerousproblemsincomputervision,imageprocessing, medicalimagingandcomputergraphics.Mixturemodelingfordirectionaldatawas proposedoriginallybyKim etal. [67 ].Banerjee etal. [ 68]alsoproposedamixture modelfordirectionaldataandapplieditforclusteringproblems.Inmedicalimaging, McGraw etal. [ 69]havemodeledthedisplacementofwatermoleculesinhighangular resolutiondiffusionimagesbymeansofavoMFmixturemodel.Morerecently,mixture modelsofcirculardatahavealsobeenusedfortrajectoryshapeanalysisinstudying objectmotion[ 70 ].KDEofunit-vectordatahasbeenusedinthecontextofsmoothing chromaticityvectorsincolorimages[ 48].Applicationsofsuchdensityestimators incomputergraphicsincludetheworkonapproximationoftheTorrance-Sparrow BidirectionalReectanceFunctions(BRDF)asreportedin[ 71],ortherecentworkin [72 ]forapproximatingthedistributionofsurfacenormals.Eugeciouglu etal. [73 ]usea kernelbasedonpowersofcosinesinsteadofavoMFinKDE,motivatedbythesuperior computationalspeedofthecosineestimator,andapplytheirtechniquefortheanalysis ofowvectorsinuidmechanics. 95

PAGE 96

The abovetechniquesignorethefactthatthedirectionaldataareoftenobtained asatransformationoftheoriginalmeasurementswhicharetypicallyassumedto resideinEuclideanspace.Thereforethetrueprobabilitydensityoftheunitvectordata isrelatedtothatoftheoriginaldatabymeansofarelationshipdictatedbyrandom variabletransformations,akeyconceptinbasicprobabilitytheory[ 74].However, akerneldensityestimateoramixturemodelestimateusing(say)voMFkernels ignoresthisveryfundamentalrelationship.Thetechniqueproposedhereexploits exactlythisrelationshipinthefollowingway:(1)Itperformsdensityestimationinthe originalspace,and(2)Itthentransformsthisdensitytothedirectionalspaceusing randomvariabletransformations.Thereby,itavoidstheaforementionedinconsistency. Secondly,conventionaldensityestimationtechniquesfordirectionaldataalsorequire thesolutionofcomplicatednonlinearequationsforkeyparameterupdatessuchas thecovariance.Thisissueiscompletelycircumventedbythepresentedtechnique. Adensityestimatorisbuiltalsoforanotherdirectionalquantity:hueincolorimages (partoftheHSIorhue-saturation-intensitycolormodel),whichiscomputedfromavery differenttransformationoftheRGBcolorvaluesobtainedfromasensor(camera). Thischapterisorganizedasfollows.Section 5.2 isareviewonthechoiceof kernelfordensityestimationforcircularandsphericaldata.Thedrawbacksofthese approachesareenumeratedandanewapproachtodensityestimationfordirectional dataisintroduced.ThisconceptisextendedforhuedatainSection 5.3.Adiscussionis presentedinSection 5.4 5.2Theory Inthissection,thetheoryofthenewmethodispresented,startingwithareviewon thechoiceofkernelsfordirectionaldensityestimationincontemporaryvisionliterature. 5.2.1ChoiceofKernel Thereexistaplethoraofkernelsusedforestimatingthedensityforunitvector data,andthereasonsforchoosingoneovertheotherrequirecarefulstudy.ForKDE 96

PAGE 97

of directionaldata,thevoMFkernelishighlypopular[ 69].Ithasgreatcomputational conveniencebecause(1)itissymmetric,(2)ityieldselegantclosed-formformulae fortheRenyientropyofavoMFmixturemodel,andforthedistancebetweentwo voMFdistributions[ 69],and(3)theinformation-geometricpropertiesofvoMFmixtures aresimple[ 69 ].Despitethesealgebraicproperties,thereareambiguities[ 75]inthe oft-repeated[ 68]notionthatthevoMFisthe`sphericalanalogueoftheGaussian'.The voMFdistributiondoespossesspropertiessimilartoaGaussiansuchasthoserelated tomaximumlikelihood,andmaximumdifferentialentropyforxedmeanandvariance, besidessymmetry.However,thevoMFalsodiffersfromtheGaussianinthesensethat (1)thecentrallimittheoremonthespheredoesnotinvolvethevoMFbutauniform distributioninstead[ 75],(2)thevoMFisnotthesolutiontotheisotropicheatequation onthesphere[ 67 ]and(3)theconvolutionoftwovoMFdistributionsdoesnotproduce exactlyanothervoMF[ 66].Ifwerestrictourselvestojustthenon-negativeorthantofthe sphere(i.e.axialdata),thentheBinghamdistributionalsopossessesmanyproperties similartotheGaussian[ 75].AnotherpopularkernelforaxialstatisticsistheWatson distribution[ 76].SomepapersevenconsiderasymmetrizedversionofthevoMFkernel, forinstance[ 69 ].However,thechoicebetweenBingham,Watsonandsymmetrized voMFkernelsisunclear,andtheywillproducedifferentdensityestimatesfornite samplesizes.Often,themotivationforchoosingoneovertheotheriscomputational convenience,whichisthechiefreasonbehindthepopularityofthevoMFkernel. 5.2.2UsingRandomVariableTransformation Theaforementioneddensityestimationtechniquesfordirectionaldatatypically assumethatonlythenalunitvectordataareavailable.However,veryoftenin computervisionapplications,theoriginaldataareavailableastheoutputofasensor. Thesearethenconvertedintounitvectorstypically(thoughnotalways-seeSection 5.3)bymeansofaprojectivetransformation(unitnormalization).Thebestinstance thereofisthatcolorimagesareoutputbyacamerausuallyinRGBformatand 97

PAGE 98

the intensitytripleateachpixelisunit-normalizedtoproducechromaticityvectors. Similarly,surfacenormalsoutputbya3Dscannerareunit-normalizedtoproducethe correspondingunitvectors.KDEormixturemodelingtechniquesforsphericaldataare appliedthereafter. Thenewapproachtodensityestimationfordirectionaldatathatdirectlyexploits thefactthattheunitvectorsarea transformation oftheoriginaldata,isnowpresented. Considertheoriginaldatatobearandomvariable X withaprobabilitydensityfunction (PDF) p (X ).Let Y = f (X ) beaknownfunctionof X .ThenthePDFof Y isgivenby p (Y = y )= Z f )Tj /T1_7 5.978 Tf 5.75 0 Td (1 ( y ) p (X = x )dx jf 0 (x ) j (5) Here f )Tj /T1_7 7.97 Tf 6.59 0 Td (1 (y ) represents thesetofallthosevalues x suchthat f (x )= y .Thisisknownas arandomvariabletransformationindensityestimation[ 74],andisaveryfundamental conceptinprobabilitytheory. Thisprincipleforestimatingthedensityofunitvectorsispresentedasfollows. Lettheoriginalrandomvariablein R 2 be ~ W havingdensity p ( ~ W ) andlet ~ V = ~ W jW j = g ( ~ W ) be itsdirectionalcomponent.Clearly, ~ V isdenedon S 1 .Let ~ w =( x y ) bea sampleof ~ W and ~ v = ~ w j ~ w j be thedirectionalcomponentof ~ w .Letthepolarcoordinate representationof ~ w be (r ).Now,thethejointdensityof (r ) isgivenby p ( r )= p (x y ) j @ (r ) @ (x ,y ) j = rp ( x y ). (5) By integratingouttheradius,wehavethedensityof ,i.e.thedensityoftheunit-vector ~ v = ~ w jw j asfollows: p ( ~ V = ~ v )= Z 1 r =0 p ( r ) dr = Z 1 r =0 rp (x y )dr (5) 98

PAGE 99

If ~ w is asamplefromanisotropicGaussiandistributionofvariance 2 andcenteredat (0,0) ,thenitfollowsthat p ( ~ V = ~ v )= 1 2 2 Z 1 r =0 re )Tj /T1_7 5.978 Tf 10.25 3.26 Td (r 2 2 2 dr = 1 2 (5) If ~ w is asamplefromanisotropicGaussiandistributionofvariance 2 andcenteredat (x 0 y 0 ),thenitfollowsthat p ( ~ V = ~ v )= 1 2 2 Z 1 r =0 re )Tj /T1_3 7.97 Tf 6.59 0 Td ((r 2 + r 2 0 )Tj /T1_3 7.97 Tf (2 rr 0 cos( )Tj /T1_9 7.97 Tf 6.59 0 Td ( 0 ))=(2 2 ) dr (5) where (r 0 0 ) isapolarcoordinaterepresentationfor (x 0 y 0 ).Uponsimplication,we have: p ( ~ v )= 1 2 2 f 2 exp )Tj /T1_2 11.955 Tf 14.42 8.08 Td (r 2 0 2 2 + r 2 r 0 cos ( )Tj /T1_1 11.955 Tf 11.95 0 Td ( 0 ) (1+ erf r 0 cos( )Tj /T1_1 11.955 Tf 11.95 0 Td ( 0 ) p 2 ) exp )Tj /T1_2 11.955 Tf (r 2 0 sin 2 ( )Tj /T1_1 11.955 Tf 11.95 0 Td ( 0 ) 2 2 g (5) As seenfromthepreviousequations,arandomvariabletransformationofavector-valued Gaussianrandomvariablefollowedbymarginalizationoverthemagnitudecomponent does not yieldavon-Misesdistribution.Infact,avon-Misesisobtainedby conditioning thevalueof r tosomeconstant(typically r =1)asopposedtointegratingover r (seepages107-108of[ 5],and[77 ]),andthereforerepresentsaconditionalandnota marginaldensity.ThedensityinEquation 5 aboveisknowninthestatisticsliterature asonecorrespondingtoaprojectednormaldistribution[ 78]orangularGaussian distribution[ 66],howeverithasnotbeenintroducedinthecomputervisioncommunity sofartothebestofthisauthor'sknowledge.Furthermore,ithasnotbeenemployedina KDEormixturemodelingframeworksofar(seeSection 5.2.3 andSection 5.2.4). 5.2.3ApplicationtoKernelDensityEstimation Now,suppose ~ w followssomeunknowndistribution.Thedensityof ~ w isconventionally approximatedbymeansofkernelmethodsactingon N samplesoftherandomvariable. IfaGaussiankernelcenteredateachsampleandhavingvariance 2 isused,thenwe 99

PAGE 100

ha ve: p ( ~ w )= 1 2 N 2 N X i =1 exp )Tj 10.5 8.08 Td (j ~ w )Tj /T1_3 11.955 Tf 13.83 0.5 Td (~ w i j 2 2 2 (5) The earlierprocedurewillyieldusthefollowingestimateofthedensityof ~ v : p 1 ( ~ v )= Z 1 r =0 p ( r ) dr = Z 1 r =0 r 2 N 2 N X i =1 e )Tj /T1_4 7.97 Tf ((r 2 + r 2 i )Tj /T1_4 7.97 Tf 6.59 0 Td (2rr i cos( )Tj /T1_9 7.97 Tf ( i ))=(2 2 ) dr (5) where (r i i ) isthestandardpolarcoordinaterepresentationforthesamplepoint ~ w i =( x i y i ).Afterevaluatingtheintegral,weobtainthefollowingexpression: p 1 ( ~ v )= 1 2 N 2 N X i =1 [ 2 exp )Tj /T1_1 11.955 Tf 14.41 8.09 Td (r 2 i 2 2 + r 2 r i cos ( )Tj /T1_3 11.955 Tf 11.95 0 Td ( i ) (1+ erf r i cos( )Tj /T1_3 11.955 Tf 11.95 0 Td ( i ) p 2 ) exp )Tj /T1_1 11.955 Tf (r 2 i sin 2 ( )Tj /T1_3 11.955 Tf 11.96 0 Td ( i ) 2 2 ]. (5) Let p 2 ( ~ v ) be theestimateofthedensityof usingthepopularvon-Miseskernelwitha concentrationparameter .Thenwehave: p 2 ( ~ v )= 1 2 I 0 ()N N X i =1 e ~ v T ~ w i j ~ w i j (5) where I 0 () is themodiedBesselfunctionoforderzero.Itiseasytoseethatfornite samplesizes, p 1 ( ~ v ) 6= p 2 ( ~ v ) ingeneral,evenifasuitablevariablebandwidthkernel densityestimateisusedfor p 2 ( ~ v ).Equation 5 isclearlydifferentfromasuperposition ofvon-Miseskernels,andcanbeconsideredasadirectionaldensityestimatorfor unit-vectordataon S 1 obtainedbyaunit-normalizationoperationoforiginaldatain R 2 usinganewkernel G : p ( ~ v )= 1 N N X i =1 G ( ~ v ; ~ w i ) (5) where G is denedasfollows: G ( ~ v ; ~ w i )= 1 2 exp )Tj 10.49 8.09 Td (j ~ w i j 2 2 2 + 1 2 p 2 ~ v ~ w i 100

PAGE 101

1 + erf ~ v ~ w i p 2 exp j ~ w i j 2 + ( ~ v ~ w i ) 2 2 2 (5) A similarPDFcanbedenedforunitvectordata(denotedas ~ v )on S 2 ,obtainedby projectivetransformationofdatadenotedas ~ w =( x y z ) residingin R 3 belongingto anisotropicGaussiandistributioncenteredat ~ w i =( x i y i z i ).Thisyieldsthefollowing expression: p ( ~ v )= Z 1 r =0 r 2 e )Tj /T1_7 5.978 Tf 7.78 4.63 Td ((x )Tj /T1_6 5.978 Tf 5.75 0 Td (x i ) 2 +( y )Tj /T1_6 5.978 Tf 5.75 0 Td (y i ) 2 +( z )Tj /T1_6 5.978 Tf 5.76 0 Td (z i ) 2 2 2 dr (5) p ( ~ v ) = e )Tj /T1_11 5.978 Tf 9.1 1.91 Td (~ jw i j 2 +( ~ v ~ w i ) 2 2 2 2 2 (2 ) 1.5 p 2 [erf( ~ v ~ w i p 2 ) +1][( ~ v ~ w i ) 2 + 2 ]+ 2 ~ v ~ w i e )Tj /T1_7 5.978 Tf 7.78 4.63 Td (( ~ v ~ w i ) 2 2 2 (5) The keyfeatureofthekerneldensityestimationapproachinthissection(andalsothe pithofthischapter,ingeneral)isthatthemodel-tting(selectionofparameterssuchas )canallbedoneinEuclideanspace.ThenewkernelsproposedinEquations 5 and 5 appearonlyinanemergentwayoutoftherandomvariabletransformation. 5.2.4MixtureModelsforDirectionalData Existingmixturemodelingalgorithmshavedifcultiesassociatedwiththechoice ofthenumberofmixturecomponentsandlocalminimaissuesduringmodeltting. Additionally,thereareotherpracticaldifcultiesinvolvedinmixturemodelingforthecase ofdirectionaldata.Firstly,ifvon-Miseskernels[ 68]areused,themaximum-likelihood estimateofthevariance(orconcentrationparameter,oftendenotedas )isnot availableinclosedformandrequiresthesolutiontoanon-linearequationinvolving Besselfunctions.In[ 68],theparameter isupdatedusingvariousapproximationsfor theBesselfunctionsthatarepartofthenormalizationconstantforvoMFdistributions, followedbytheadditionofanempiricallydiscoveredbiasthatisapolynomialfunctionof theestimatedmeanvectors.ThedifcultiesfacedbyamixtureofvoMFdistributionsin modelingdatathatarespreadoutanisotropicallyaremitigatedbytheuseofamixture 101

PAGE 102

of Watsonkernelsasclaimedin[ 76].Nonetheless,iterativenumericalproceduresto estimate arestillrequired,andthecasewhereafullcovarianceistobeobtained,will beevenmorecomplicated.Moreoverthemethodin[ 76]alsorequiressolvingnon-linear equationsfortheupdateofthe centers oftheindividualcomponentsoverandabove the values.Overandabovethis,theupdateofthemeanvectorsinboth[68]and[76] involvesvectoradditionfollowedbyunitnormalization,whichisunstableifantipodal vectorsareinvolvedasthenormoftheresultantvectorwillbeverysmall. Theapproachbasedonthetheorypresentedintheprevioussubsections overcomesthesedifcultiesbyfollowingatwo-stepprocedure:(1)amixture-modeltin theoriginalEuclideanspacegivenasetof N samples,followedby(2)atransformation ofrandomvariables.IfaGaussianmixturemodelisttotheoriginaldatasamples, using M components,withpriors fp k g,centers f( xk yk )=( r k cos k r k sin k )g and variances f k g ,thenarandomvariabletransformationresultsinthefollowingformof directionalmixturemodel: p ( ~ v )= Z 1 r =0 r 2 2 M X k =1 p k e )Tj /T1_9 5.978 Tf 7.78 5.62 Td ((x )Tj /T1_11 5.978 Tf 5.76 0 Td ( xk ) 2 +( y )Tj /T1_11 5.978 Tf 5.75 0 Td ( yk ) 2 2 2 k dr p ( ~ v ) = M X k =1 p k G ( ~ v ; ~ k k ), (5) where G wasdenedinEquation 5.Sincetheentiremixture-modelingprocedureis performedintheoriginalspace,theaforementioneddifcultiesinestimatingthemean andconcentrationparametersareautomaticallyavoided. Ifwecontinuetofollowthislineofreasoning,wecannowachieveafresh perspectiveonmixturesofvoMFdistributionsaswell.Asmentionedpreviouslyandas clearlydocumentedin[ 5],thevoMFdistributionisobtainedfromaGaussiandistribution by conditioning themagnitudeoftherandomvariabletobesomeconstant.Ifweta Gaussianmixturemodeltotheoriginaldataandexpresseditinpolarcoordinates,we 102

PAGE 103

are leftwiththefollowingexpression: p (r )= M X k =1 p k 2 2 k e )Tj /T1_6 5.978 Tf 7.78 4.62 Td ((r cos )Tj /T1_4 5.978 Tf 5.76 0 Td (r k cos k ) 2 +( r sin )Tj /T1_4 5.978 Tf 5.76 0 Td (r k sin k ) 2 2 2 k (5) By conditioningon r =1,wehave: p ( jr =1)= M X k =1 p k 2 I 0 ( r k 2 k ) e r k cos ( )Tj /T1_8 5.978 Tf 5.76 0 Td ( k ) 2 k (5) This procedurebasicallysuggestsagainthattheentiremixturemodelingalgorithm canbeexecutedinEuclideanspace,andthatamixtureofvoMFdistributionscanbe obtainedby conditioning themagnitudeoftherandomvariabletobe1(orsomeother constant) 1 .Thepolarcoordinatestransformationsyieldaformulafortheconcentration parameter k ofthe k th component,givenas k = r k 2 k Thisprocedurethereforesuggests usaviablealternativetottingamixtureofvoMFdistributionswhentheoriginaldataare available(andnotjusttheunit-vectordata).Similarexpressionscanbederivedforthe caseofdataon S 2 derivedfrom R 3 aswell. 5.2.5PropertiesoftheProjectedNormalEstimator Theprojectednormaldistributionissymmetricandunimodaljustlikethevon-Mises distribution.Figure 5-1 showstheprojectednormaldistributioncorrespondingtoan originalGaussiandistributioncenteredat ~ 0 =(1,0) havingavarianceof 0 =10, andavon-Misesdistributioncenteredat (1,0) with 0 = j ~ 0 j 2 0 = 0.01.Similarly,plots oftheprojectednormaldistributionon S 2 foranoriginalGaussiandistributionwith ~ 0 =(1,0,0) andvariance10,andavoMFdistributionwithmean ~ 0 =(1,0,0) andconcentration 0 = j ~ 0 j 2 are showninFigure 5-2.Asindicatedbytheplots,both distributionshaveadistinctpeakat 54 45 asexpected. 1 Note thatthevoMFdistributionoramixtureofvoMFdistributionsareconditionaland notmarginaldistributions. 103

PAGE 104

F romEquations 5, 5 and 5,itcanbeseenthatthedensityestimatordoes notrequiretheconversionoftheoriginalsamplestounitvectors,butoperatesentirelyin theoriginalspace. 5.3EstimationoftheProbabilityDensityofHue Directionaldataareusuallyobtainedbytheprocessofunit-normalizationof theoriginalvectordatameasuredbyasensor.However,thisisn'talwaysthecase. Forinstance,colorsensorstypicallyoutputvaluesintheRGBcolorformat.These valuesarethenconvertedtoothercolorsystemssuchasHSIusingtransformations ofadifferentkind,presentedbelow.TheHSIcolormodelisbasedonthenotionof separatingacolorintothreequantities-thehueH(whichisthebasiccolorsuchasred orgreen),thesaturationS(whichindicatestheamountofwhitepresentinacolor)and thevalueI(whichindicatestheamountofshadingorblack).Thecomponenthue(H)is anangularquantity.TherulesforconversionbetweentheRGBandHSIcolormodels areasfollows[ 48]: H =cos )Tj /T1_5 7.97 Tf (1 0.5(2R )Tj /T1_2 11.955 Tf 11.96 0 Td (G )Tj /T1_2 11.955 Tf 11.95 0 Td (B ) p (R )Tj /T1_2 11.955 Tf 11.96 0 Td (G ) 2 + ( R )Tj /T1_2 11.955 Tf 11.96 0 Td (B )(G )Tj /T1_2 11.955 Tf 11.95 0 Td (B ) S =1 )Tj /T1_3 11.955 Tf 38.61 8.09 Td (3 R + G + B min( R G B ) I = 1 3 (R + G + B ). (5) The inversetransformationfromHSItoRGB,forhuevalues 0 < H 2 3 is: B = I (1 )Tj /T1_2 11.955 Tf 11.95 0 Td (S ) R = I (1+ S cos H cos( 3 )Tj /T1_2 11.955 Tf 11.96 0 Td (H ) ) G = 3 I )Tj /T1_3 11.955 Tf 11.96 0 Td ((R + B ). (5) Forhuevalues 2 3 < H 4 3 theformulaearegivenby: H = H )Tj /T1_3 11.955 Tf 13.15 8.09 Td (2 3 R = I (1 )Tj /T1_2 11.955 Tf 11.96 0 Td (S ) 104

PAGE 105

G = I (1 + S cos H cos( 3 )Tj /T1_0 11.955 Tf 11.95 0 Td (H ) ) B = 3I )Tj /T1_1 11.955 Tf 11.95 0 Td ((R + B ). (5) andforhuevalues 4 3 < H 2 H = H )Tj /T1_1 11.955 Tf 13.15 8.09 Td (4 3 G = I (1 )Tj /T1_0 11.955 Tf 11.95 0 Td (S ) B = I (1 + S cos H cos( 3 )Tj /T1_0 11.955 Tf 11.96 0 Td (H ) ) R = 3 I )Tj /T1_1 11.955 Tf 11.96 0 Td ((R + B ). (5) If p (R G B ) isthedensityoftheRGBvalues,andtakingintoaccountthefactthatthe RGBtoHSItransformationisone-oneandonto,thedensityoftheHSIvaluesisgiven as: p (H S I )= p (R G B ) j @ (H ,S I ) @ (R ,G B ) j = j @ ( R G B ) @ (H S I ) jp (R G B ). (5) No w,forallhuevalues,wehave: j @ (R G B ) @ (H S I ) j = 2 p 3 sec 2 H (1 + p 3 tan H ) 2 [IS (1 )Tj /T1_0 11.955 Tf 11.95 0 Td (S )+ I 2 S (S +2)]. (5) SupposingtheRGBvaluesweredrawnfromaGaussiandistributioncenteredat (R i G i B i ) havingvariance 2 ,thenthedistributionofHSIisgivenas: p ( H S I )= 2 p 3 sec 2 H (1 + p 3 tan H ) 2 [IS (1 )Tj /T1_0 11.955 Tf 11.95 0 Td (S )+ I 2 S (S +2)] 1 3 (2 ) 1.5 e )Tj /T1_3 5.978 Tf 7.78 4.62 Td ((R )Tj /T1_7 5.978 Tf 5.75 0 Td (R i ) 2 +( G )Tj /T1_7 5.978 Tf 5.76 0 Td (G i ) 2 +( B )Tj /T1_7 5.978 Tf 5.76 0 Td (B i ) 2 2 2 (5) Fur thersimplicationgives p ( H S I )= 2 p 3 sec 2 H (1 + p 3 tan H ) 2 [IS (1 )Tj /T1_0 11.955 Tf 11.95 0 Td (S )+ I 2 S (S +2)] 1 3 (2 ) 1.5 e )Tj /T1_3 5.978 Tf 7.78 4.62 Td ((I +ISk )Tj /T1_7 5.978 Tf 5.75 0 Td (R i ) 2 +( I )Tj /T1_7 5.978 Tf 5.76 0 Td (IS )Tj /T1_7 5.978 Tf 5.75 0 Td (B i ) 2 +( I +IS )Tj /T1_7 5.978 Tf 5.75 0 Td (ISk )Tj /T1_7 5.978 Tf 5.75 0 Td (G i ) 2 2 2 (5) 105

PAGE 106

where k = 2 1+ p 3 tan H .Tondthemarginaldensityofhue,weintegrateoverthevaluesof I and S (bothlyingintheinterval [0,1] ),givingus: p (H )= Z 1 I =0 Z 1 S =0 p (H S I )dSdI (5) Unlikethecasewiththeprecedingsection,thisformulaisnotavailableinclosedform. Howeveritiseasytoapproximatethisformulanumerically,asitjustinvolvesa2D deniteintegraloveraboundedrangeofvalues(of S and I ). Insteadofmarginalizing,ifweconditioned S and I totakeonthevalueof1,then theconditionaldensityof H isobtainedasfollows: p (H jS = I =1)= 6 p 3 sec 2 H 3 (2 ) 1.5 (1 + p 3 tan H ) 2 e )Tj /T1_3 5.978 Tf 7.78 5.34 Td ((1+k )Tj /T1_5 5.978 Tf 5.76 0 Td (R i ) 2 +B 2 i +(2)Tj /T1_5 5.978 Tf 16.6 0 Td (k )Tj /T1_5 5.978 Tf 5.76 0 Td (G i ) 2 2 2 (5) Notice thatequation 5 isanalogoustoEquation 5 inthesensethatbothare conditionaldensities(obtainedbyconditioningothervariablestohaveconstantvalues). Ontheotherhand,Noticethatequation 5 isanalogoustoEquation 5 inthesense thatbotharemarginaldensities(obtainedbyintegratingoutothervariables). Wewouldliketodrawthereader'sattentiontothefactthatboththeseapproaches areradicallydifferentfromthatproposedin[ 79].Thelatterapproachperformsdensity estimationofthehuebyrstconvertingtheRGBsamplestohuevalues.Then,it centersakernelwithadifferentbandwidtharoundeachhuesample.Thevalueofthe bandwidthforthe i th sample H i isdeterminedbythepartialderivatives @ H i @ R @ H i @ G @ H i @ B which indicatesthesensitivityof H i w.r.t.theoriginalRGBvalues.Ashueisanon-linear functionofRGB,thesensitivityinthehuevaluesvarieswiththeRGBvaluesofthe samplesobtainedfromthesensor.Forinstance,hueishighlyunstableatRGBvalues thatareclosetotheachromaticaxis R = G = B 106

PAGE 107

5.4 Discussion MosttechniquesthatestimatethePDFofdirectionaldataassumethatonlythe directionaldataareavailable.Thisfactisexploitedtoderiveanewapproachfordensity estimationofdirectionaldatabyrstestimatingthedensityintheoriginalspacefollowed byarandomvariabletransformation.Therefore,thisistheonlycircular/sphericaldensity estimatorinthecomputervisioncommunity,whichisconsistentwiththeestimateof thedensityoftheoriginaldatafromwhichthedirectionaldataarederived,inthesense ofrandomvariabletransformations,akeyconceptinprobabilitytheory.Secondly, thismethodcircumventsissuesinvolvedinsolvingcomplicatednon-linearequations thatariseinmaximumlikelihoodestimatesfortheparametersofconventionaldensity estimators,asitoperatesintheoriginalspace,andthereforeusesthemuchsimpler mixture-modelingorKDEtechniquesthatarepopularforEuclideandata.Thetheory forthisestimatorisbuiltforunit-normalvectorsaswellasquantitiessuchashuein colorimaging.Thoughthisworkdealsstrictlywithdirectionaldata,theunderlying philosophyofthisapproachiseasilyextensibletodataresidingonotherkindsof manifolds.Thereforeithasthepotentialofposingasaviablealternativetoexisting kerneldensityestimatorsthatrequiretheusageofnon-trivialmathematicaltechniques (suchascomputationofgeodesicdistancesbetweensamplesonagivenmanifold)in ordertobetunedtodatathatresideonnon-Euclideanmanifolds[ 80]. Theapproachpresentedinthischapteralsoraisesthefollowingquestion.Consider arandomvariable f whoseestimatedPDF(sayusingakernelmethod)usingsamples ff 1 f 2 ,..., f n g isgivenby ^ p f ( f = )= 1 n n X i =1 K f ( )Tj /T1_2 11.955 Tf 11.96 0 Td (f i ; f ). (5) No wconsideratransformation T of f ,yieldingthetransformedrandomvariable g = T (f ).Onemethodcouldbetoapplyakerneldensitymethodtodirectlytothe transformedsamples f g 1 = T (f 1 ), g 2 = T (f 2 ),..., g n = T (f n )g,yieldingthedensity 107

PAGE 108

estimate ^ p g (g = f ) = 1 n n X i =1 K g (f )Tj /T1_2 11.955 Tf 11.95 0 Td (g i ; g ) (5) where f = T ( ). Alternatively,onecouldapplyarandomvariabletransformationto ^ p f (f = ) toyield ~ p g (g = f )= Z r = T )Tj /T1_6 5.978 Tf 5.76 0 Td (1 (f ) ^ p f (f = r ) jT 0 ( r )j (5) The relationshipbetween ^ p g (.) and ~ p g (.) willdependuponthechoiceofkernels K f (.) and K g (.) andtheparameters f and g ,whichrequiresfurtherinvestigation.Note thatthePDFestimatorforimageintensitiesfromChapter 2 followstheapproach inEquation 5 asitisanexplicitrandomvariabletransformationfromlocationto intensity,whereasallthesample-basedmethodsreviewedinChapter 2 followtheformer approachinEquation 5. ConsideryetanotherscenariowherethetechniquefromChapter 2 wasused toestimatethedensityoftheintensityvaluesinanimage I ( x y ) .Nowlet J (x y )= T (I (x y )) beatransformationoftheimage I .TherearetwowaystoarriveatthePDF of J (x y ) -oneestimate(denoted ^ p 1 (.))isbyinterpolatingthevalueof I ,andthen applyingtherandomvariabletransformation.Theotherestimate(denoted ^ p 2 (.))is obtainedbyrstcomputingthe J valuesatthediscretelocationsandtheninterpolating thosevaluestoyieldanotherestimateofthedensityof J .Inthiscase,thetwoestimates wouldberelatedbythespecicinterpolantsemployed.Letusconsiderthespeciccase where I (x y ) wasanRGBimage,and J (x y ) wastheimageofchromaticityvectors. Considerthattheinterpolantusedfor I (x y ) wassuchthatthedirectionsofthesubpixel RGBvaluesweresphericallinearfunctionsofthespatialcoordinates,whereasthe magnitudeswerelinearfunctionsofthespatialcoordinates.Consideralsothatthe interpolantusedfor J (x y ) wassphericallinearinnature.Itcanbeseeneasilythatthe estimates ^ p 1 (.) and ^ p 2 (.) usingtheseruleswouldbeequal. 108

PAGE 109

Figure 5-1.Aprojectednormaldistribution(~ 0 =(1,0), 0 =10)andavon-Mises distribution(~ 0 =(1,0), 0 = j ~ 0 j 2 0 = 0.01) A B C Figure 5-2.Plotsof(A)aprojectednormaldensity( ~ 0 =(1,0,0), 0 =10),(B)avoMF density(~ 0 =(1,0,0), 0 = j ~ 0 j 2 0 = 0.01 ),and(C) L 1 normofthedifference betweenthetwodensities 109

PAGE 110

CHAPTER 6 IMAGEDENOISING:ALITERATUREREVIEW 6.1Introduction Inthischapter,wegiveadetailedreviewofcontemporaryliteratureonimage denoising.Wemakeanattempttocoverasmanydiverseapproachesaspossible, thoughacompleteoverviewisbeyondthescopeofthethesis,giventhesheer magnitudeofexistingresearchonthistopic.Tothebestofourknowledge,there existveryfewsurveysonimagedenoising.Thereviewin[ 2]focusesonmathematical characteristicsoftheresidualimages(denedasthedifferencebetweenthegiven noisyandthedenoisedimage)fordifferenttypesofimageltersrangingfrom partialdifferentialequationstowaveletbasedmethods.Asummaryofrecenttrends indenoisingwaspresentedbyDonohoandWeissmanattheIEEEInternational SymposiumonInformationTheory(ISIT)in2007[ 81 ].Thistutorialfocussedonwavelet andothertransformbasedmethods,somelearningbasedmethodsandnon-local methods.Inthepresentreview,wediscussandcritiquemethodsbasedonpartial differentialequations,localconvolutionandregression,transformdomainmethods usingwaveletsandthediscretecosinetransform(DCT),non-localapproaches, methodsbasedonanalysisofthepropertiesofresidualsandmethodsthatuse variousmachinelearningtools.Theaforementionedcategoriesconstitutethebulk ofmodernimagedenoisingliterature.Thefocusofthesurveyisongray-scaleimage denoising,thoughwemakeoccasionalreferencestopapersoncolorimagedenoising. Throughoutthischapterandinsubsequentchapters,weconsidernoisetobearandom signalindependentoftheoriginalsignalthatitcorrupts.Apartfromadescriptive surveyofthecontemporarytechniquesassuch,wealsocoversomecommonissues concerningalmostallcontemporarydenoisingtechniques:methodsforvalidationoflter performanceandmethodsforautomatedparameterselection. 110

PAGE 111

6.2 PartialDifferentialEquations Theisotropicheatequationwasusedforimagesmoothingin[82 ].Itisknown thatexecutingthispartialdifferentialequation(PDE)ontheimageisequivalentto convolutionwithaGaussiankernel,wherethekernelparameter(oftendenoted by )isrelatedtothetimestepandnumberofiterationsofthePDE.However, isotropicsmoothingblursawaysignicantimagefeaturessuchasedgesalongwith thenoise,andhenceisnotusedincontemporarydenoisingalgorithms.Instead,inmost contemporarydiffusionmethods,thediffusionprocessisdirectedbyedgeinformation intheformofadiffusivityfunctionwhichpreventsblurringacrossedgesandallows diffusionalongthem[ 44 ].Thechosendiffusivityfunctionisactuallyamonotonically decreasingfunctionofthegradientmagnitude.TheequationforthePDEcanbewritten asfollows @ I @ t = div(g (jrI j)rI ) (6) where I :n R isagray-scaleimagedenedondomain n and g (jrI j) isadiffusivity functiontypicallydenedas g (jrI j; )= 1 1 + jrI j 2 = 2 (6) Severaldifferentdiffusivityfunctionshavebeenproposed,forinstancethosebyPerona andMalik[ 44 ],Weickert[ 83]andBlack etal. [84 ].Aregularizedversionoftheabove equationhasbeenproposedin[ 85 ].Connectionsbetweenrobuststatisticsand anisotropicdiffusion(whichshowupinthechoiceofdiffusivityfunction)havebeen establishedin[ 84]. SomePDEsareobtainedfromtheEuler-Lagrangeequationscorrespondingto energyfunctionals.Oneexampleistheimagetotalvariationdenedas E (I )= Z n jrI (x y )jdxdy (6) 111

PAGE 112

giving risetothePDE @ I @ t = div( r I jr I j ). (6) It shouldbenotedthattheaforementionedtechniquesarebasedontheassumption thatnaturalimagesarepiecewiseconstant,whichisnotnecessarilyavalidassumption. Theyalsorequirethechoiceoftheparameter inthediffusivity.Thisparameterneed notbeconstantthroughouttheimage.ThenumberofiterationsforwhichthesePDEs areexecutedisanimportantparametercriticalforgoodperformance.Inthelimitof inniteiterations,constantorpiecewise-constantimagesareproduced.Someauthors remedythestoppingtimeselectionissuebyintroducingapriormodelintheenergy formulation,forexamplethefollowingmodicationofthetotalvariationmodel,starting withaninitialimage I 0 E (I )= Z n jrI (x y )jdxdy + Z n (I (x y ) )Tj /T1_2 11.955 Tf 11.96 0 Td (I 0 (x y )) 2 dxdy (6) where isaparameterthattradesdatadelitywithregularity.Theimplicitassumption intheterm (I (x y ) )Tj /T1_2 11.955 Tf 12.64 0 Td (I 0 (x y )) 2 isaGaussiannoisemodel.Assumingthattheimage hasbeencorruptedwithzeromeanGaussiannoiseofknownvariance 2 n ,aconstrained versionoftheobjectivefunctionhasbeenproposedin[ 86]: min I E (I )= Z n jr I (x y )jdxdy (6) subjectto Z n (I )Tj /T1_2 11.955 Tf 11.95 0 Td (I 0 ) 2 dxdy = 2 n (6) Z n I (x y ) dxdy = Z n I 0 (x y )dxdy (6) Fordifferentnoisemodels,suchasPoissonorimpulsenoise,differentpriorscanbe used[ 87].AhighlycomprehensivereviewofseveralsuchPDE-basedapproachescan befoundinexemplarybookssuchas[ 83 ]and[ 53 ],tonameafew.Recently,some authorshavealsointroducedtheconceptofdiffusionwithcomplexnumbers,which 112

PAGE 113

br ingsaboutdenoisinginconjunctionwithedgeenhancement[ 88 ],[ 89].Thelatter techniqueperformsthecomplexdiffusionbytreatingtheimage I :n R asagraphof theform ( x y I (x y )),aframeworkfordiffusiondevelopedin[ 58]. SomeresearchershavedevelopedPDEsbasedonapiecewiselinearassumption onnaturalimages,examplesbeing[ 90]and[ 91 ].Theseturnouttobefourthorder PDEsandtheirenergyfunctionspenalizedeviationintheintensity gradient asopposed todeviationinintensity,andpreserveneshadingbetter.Howeverinsomecasessuch as[ 90 ],speckleartifactshavebeenobservedwhichneedtoberetroactivelyremedied usingmedianlters[ 90].Anotherclassofapproachesconsistsofindependentlyltering thegradientsinthe x and y directions,andthenusingsomepriorassumptiononthe imagegeometrytoreconstructtheimageintensityfromthesmoothedgradientvalues [92 ]. 6.3SpatiallyVaryingConvolutionandRegression Arichclassoftechniquesforimagelteringinvolvetheso-calledspatiallyvarying convolutions.Inthesemethods,animageisconvolvedwithapointwisevaryingmask whichisderivedfromthelocalgeometryextractedfromthesignal.Acloselyrelated ideaisthemodelingofthelocalgeometryofanimage(signal)bymeansofalow-order polynomialfunction.Thesignalisapproximatedlocallybyapointwise-varyingweighted polynomialt.Thecoefcientsofthepolynomialarecomputedbyaleast-squares regression,andthesearethenusedtocomputethevalueofthe(ltered)signalat acentralpoint.Forinstance,thesignalcouldbemodeledasfollows,restrictedtoa neighborhood n aroundapoint x 0 : I ( x )= a 0 + m X i =1 a i (x )Tj /T1_1 11.955 Tf 11.96 0 Td (x 0 ) i (6) where f a i g (0 i m )arecoefcientsofthepolynomial.Thesecoefcientsare obtainedbyleastsquarestting,andthelteredsignalvalueisgivenby I ( x 0 )= a 0 Thisprocedureisnotguaranteedtopreserveedgesasitallowsevendisparateintensity 113

PAGE 114

v aluestoaffectthepolynomialt.Insteadinpractice,thesignalismodeledasfollows: I (x )= a 0 + m X i =1 a i w ( x )Tj /T1_1 11.955 Tf 11.95 0 Td (x 0 I (x ) )Tj /T1_1 11.955 Tf 11.95 0 Td (I (x 0 ); h s h v )(x )Tj /T1_1 11.955 Tf 11.96 0 Td (x 0 ) i (6) Here w (x )Tj /T1_1 11.955 Tf 13.55 0 Td (x 0 I (x ) )Tj /T1_1 11.955 Tf 13.55 0 Td (I (x 0 ); h s h v ) isaweightingschemewhichisbasicallya non-increasingfunctionofthedifferencebetweenspatiallocations,i.e. x )Tj /T1_1 11.955 Tf 12.88 0 Td (x 0 ,and thedifferencebetweenthesignalvaluesatthoselocations,i.e. I (x ) )Tj /T1_1 11.955 Tf 13.13 0 Td (I (x 0 ).The function w isparameterizedby h s and h v whichactasspatialandintensitysmoothing parametersrespectively.Thettingprocedureisnow,ofcourse,a weighted least squaresregression.TheseideastracebacktotheSavitzky-Golaylter[ 93 ],[ 94]and arethesubjectofbeautifulbookssuchas[ 95].Two-dimensionalversionsoftheseideas havebeenrecentlyusedinmodiedformsforimagelteringapplicationsin[ 65 ]and [55 ].In[65 ],theparameter h s isreplacedbyamatrix,whichisselectedinamanner dictatedbylocalimageedgegeometryandnopenaltyisappliedonintensitydeviation. Ontheotherhand,inthelattercase[ 55],theweightsforregressionareaffectedsolely byintensitydifference.Thepopularbilaterallteringtechnique[ 49],[ 96]isagainbased onaweightedlinearcombinationofintensities,withweightsdrivenbybothlocationand intensitydifferences.Infact,thekernelregressionapproachin[ 65]hasbeenframedas ahigher-ordergeneralizationofthebilaterallter.Ifthepolynomialorderisrestrictedto oneandtheweightsareappliedonlyonintensitydifferences,onegetstheso-calledthe kerneldensitybasedlter[ 48 ],alsocalledastheanisotropicneighborhoodlter[ 55]. Aversionwheretheweightsareobtainedfromintensitygradientmagnitudeshasbeen presentedin[ 47]andiscalledastheadaptivelter.Anextensiontotheanisotropic neighborhoodlterusinginterpolationbetweennoisyimageintensityvalues(andthe inducedisocontourmap)hasbeenrecentlypresentedbyusin[ 11]andinChapter 4.Inallthesetechniques,acrucialparameteristhesizeandalsotheshapeofthe neighborhoodforlocalsignalmodeling.Animportantcontributiontowardsolvingthis 114

PAGE 115

prob lemisadata-drivenapproachpresentedin[ 97 ],whichderivesamulti-directional star-shapedneighborhood(oflargestpossiblesize)aroundeachimagepixel. Themean-shiftprocedure,aclusteringtechniqueproposedin[98 ],andappliedto ltering(andsegmentation)in[ 51],canbeconsideredasageneralizationofbilateral ltering,wherethewindowforlocalsignalmodelingisallowedtogrowdynamically. Thisgrowthisdirectedbyanascentonalocaljointdensityfunctionofspatialaswell asintensityvalues.Itshouldbenotedthatbothbilaterallteringandmeanshiftare relatedtotheBeltramiowPDEdevelopedin[ 58].Theserelationshipshavebeen exploredin[ 99].TheconnectionsbetweennonlineardiffusionPDEsoversmallperiods oftimeandspatiallyvaryingconvolutionshavebeenshownin[ 100].In[45],theauthors presentso-calledtrace-basedPDEsforsmoothingofcolorimagesandprovethatthe correspondingdiffusionisexactlyequivalenttoconvolutionswithorientedGaussians, wheretheorientationisdictatedbylocalimagegeometryoredgedirection. Thus,spatiallyvaryingconvolutionsforlteringhavearichhistory.Themostrecent contributioninthisareaistheonepresentedin[ 64]and[101].Thisframeworkisbased upontheJian-Vemuricontinuousmixturemodelfromtheeldofdiffusion-weighted magneticresonanceimaging(DW-MRI)[ 102].In[ 64],complicatedlocalimage geometriessuchasedgesaswellas X, Y or T junctionsaremodeledusingaGabor lterbankatdifferentorientations.ThecollectionofGabor-lterresponsesisexpressed asadiscretemixtureofacontinuousmixtureofGaussians(withWishartmixing densities)oradiscretemixtureofacontinuousmixtureofWatsondistributions(with Binghammixingdensities)torespectivelyyieldtwodifferenttypesofkernelsforlocal geometry-preservingconvolutions.Thenumberofcomponentsofthediscretemixture isgivenbyanappropriatesamplingofthe2Dorientationspaceandtheweightsofthe discretemixturearesolvedbylocalregularizedleastsquarestting.Thenoveltyofthis techniqueis(1)theautomaticsettingofweightsforgeometry-preservingsmoothing, and(2)theabilitytopreservefeaturessuchasimagecornersandjunctions(whichare 115

PAGE 116

ignored bytheotherconvolution-basedmethodsmentionedbefore).Whiletechniques suchascurvature-preservingPDEs[ 103]attemptpreservationofsuchgeometries, theirbehaviorat X Y or T junctions(wherecurvatureisnotdened)mayneedfurther exploration. Themeanshiftprocedureorotherlocalconvolutionlterscanalsobeappliedto theimagegradientstobetterfacilitatethepreservationofshading.Anextensivesurvey ofvariousapplicationswithdifferenttypesoflteringoperationsonimagegradients, followedbyimagereconstructionusingaprojectionontothenearestintegrablesurface [104]orbysolvingthePoissonequation,hasbeenpresentedin[ 105]inashortcourse attheInternationalConferenceonComputerVision,2007,andinpaperssuchas[ 106]. 6.4Transform-DomainDenoising Transform-domaindenoisingapproachestypicallyworkatthelevelofsmallimage patches.Intheseapproaches,theimagepatchisprojectedontoachosenorthonormal basis(suchasawaveletbasisortheDCTbasis)toyieldasetofcoefcients.Itis well-knownthatthecoefcientsinthetransformdomainarehighlycompressibleinthe sensethatthevastmajorityofthesecoefcientsareveryclosetozero.Intheliterature, thispropertyisreferredtoas`sparsity',thoughinastrictsense,sparsitywouldrequire mostcoefcientstobe equal tozero.Intherestofthethesis,weshallsticktothis usageoftheword`sparsity'eventhoughweimplycompressibility.Itisknownthatthe coefcientsinthewaveletorDCTtransformdomainaredecorrelatedfromoneanother [107].Itshouldbenotedthatthesmallercoefcientsusuallycorrespondtothehigher frequencycomponentsofthesignalwhichareoftendominatedbynoise.Toperform denoising,thesmallercoefcientsaremodied(typically,thosecoefcientswhose magnitudeisbelowsome aresettozero,inaprocesstermed`hardthresholding'),and thepatchisreconstructedbyinversionofthetransform.Thisprocedureisrepeatedfor everypatch.Ifthepatchesarechosentobenon-overlapping,onecanobserveseam artifactsatthepatchboundaries.Furthermore,thethresholdingofthecoefcientsis 116

PAGE 117

also knowntoproduceringingartifactsaroundimageedgesorsalientfeatures.Artifacts ofbothtypescanberemediedbyperformingtheaforementionedthreestepsinasliding windowfashionfrompixeltopixel.Thisyieldsanovercompletetransformaseach pixelnowacquiresmultiplehypothesesfromoverlappingpatches.Thesehypotheses areaggregated(typicallybysimpleaveraging)togethertoyieldanalestimate.This processofaveragingofmultiplehypotheseshasbeenreportedtoconsistentlyyield superiorresults[ 108],[ 109],andistermed`translationinvariantdenoising',or`cycle spinning'[ 108]. Theperformanceoftransform-basedtechniquesisaffectedbythefollowing parameters:thechoiceofbasis,thechoiceofathresholdingmechanism,amethod foraggregationofoverlappingestimatesandthepatchsize.Wediscussthesepoints below. 6.4.1ChoiceofBasis Somewhatsurprisingly,ithasbeenobservedthattheslidingwindowDCT outperformsmostwaveletbases[ 109].However,givenalibraryoforthonormalbases, thechoiceofthebestone(fromthepointofviewofdenoising)fromamongstthese,is largelyanopenprobleminsignalprocessing.Inmanyexistingapproaches,theimage patch(ofsize n 1 n 2 )isrepresentedasamatrixandthebasesforrepresentationare obtainedfromtheouterproductofthebasesthatrepresenttherowswiththebases thatrepresentthecolumns[ 109],[ 108].Thisiscalledasaseparablerepresentation.In othercases,theimagepatchisrepresentedasa1Dvectorofsize n 1 n 2 usingabasis ofsize n 1 n 2 n 1 n 2 .Intheseparablecase,ithasbeenobservedthatthetransformmay bebiasedtowardsimageswhosesalientfeaturesarealignedwiththeCartesianaxes. Ifthelocalimagegeometrydeviatesfromtheseaxes,thetransformmaynotbeable torepresentthemcompactlyenough.Thishasbeenremediedbyusingnon-separable basessuchasthesteerablewavelet[ 110 ],orthecurvelettransform[ 111],whichare designedbytakingimagegeometryintoaccount. 117

PAGE 118

6.4.2 ChoiceofThresholdingSchemeandParameters Themostcommonthresholdingmethodishardthresholding,givenasfollows: T (c ; )= 8 > < > : c if jc j 0 if jc j <. (6) Anotherpopularmethod,knownassoftthresholding,notonlynulliescoefcients smallerthanthethresholdbutalsoreducesthevalueofcoefcientsthatarelargerthan thethreshold.Mathematically,softthresholdingisexpressedasfollows: T (c ; )= 8 > > > > < > > > > : c )Tj /T1_4 11.955 Tf 11.95 0 Td ( if j c j >, c > 0 c + if jc j >, c < 0 0 if jc j < (6) Thereexistseveralotherthresholdingschemes(orrather,schemesformodicationof transformcoefcients).Thesemethodscanbeinterpretedastheresultofminimizing differenttypesofriskfunctions.Forexample,thehardthresholdingscheme(sometimes termedasthebestsubsetselectionproblem)istheresultofminimizingthehard thresholdpenalty,softthresholdinghasaninterpretationintermsofminimizing the L 1 penalty,whereasminimizationofthesmoothlyclippedabsolutedeviation (SCAD)leadstoathresholdingschemethatliesintermediatebetweenhardandsoft thresholding(seeFigures1and2andSection(2.1)of[ 112]).Almostallthesemethods ofthresholdingleadtomonotonicfunctionsofthecoefcientmagnitude.Despitethe severalsophisticatedthresholdingfunctionsavailable,thebestdenoisingresultsthat havebeenreportedusingwavelettransformsaretheoneswithhardthresholding, withatranslationinvariantapproach[ 62 ].Thechoiceoftheparameter hasbeen studiedindetailinthecommunity.Forinstance,in[ 113],theauthorsprovethatunder ahardthresholdingscheme,thechoice = n p 2 log N isoptimalfromastatistical riskstandpoint,underzeromeanGaussiannoiseofstandarddeviation n ,where N isthesize(i.e.numberofpixels)oftheimage/imagepatch(seeTheorem(4)and 118

PAGE 119

Equation (31)of[ 113]).IntheexperimentstobepresentedinChapter 7,wehave observedempiricallythatthethreshold =3 producesexcellentdenoisingresultsfora Gaussiannoisemodelwith 8 8 patches,whichapproximatelytallieswiththeresultfrom [113].ThisisintunewithanempiricallyobservedfactthatthecoefcientsofaGaussian randommatrixofstandarddeviation whenprojectedonanorthonormalbasisareless than 3 withaveryhighprobability. 6.4.3MethodforAggregationofOverlappingEstimates Themostcommonapproachforaggregationisasimpleaveragingof(oramedian operationon)allthehypothesesgeneratedforthepixel. 6.4.4ChoiceofPatchSize Thepatchsizechoicepresentstheclassicalbias-variancetradeoff.Verysmall patchesallowpreservationofnerdetailsoftheimagebutmayovert(undersmooth), whereaslargerpatchsizesperformbetterinsmoothinglargerhomogeneousregions butmayoversmoothsomesubtledetails.Verylittleworkexistsonoptimalpatch sizeselection.Infact,thepatchsizeneednotbeconstantthroughouttheimage andcanvaryasperlocalgeometry.Somepaperssuchas[ 114]proposetheuseof multi-scaleapproachesbycombiningestimatesatdifferentscales.Howevertheoptimal combinationofsuchestimatesisstillaproblem,muchlikethatofoptimalaggregationof overlappingestimates.Wepresentacorrelation-coefcientcriterionfortheautomated selectionofasingleglobalpatchsizeinChapter 7. Acommoncriticismoftransform-domainthresholdingtechniques(especiallyhard thresholding)istheirinabilitytodistinguishhighfrequencyinformationfromnoise.Some authorstrytoremedythisbyobservingthatthereexistdependenciesthatariseina transformcoefcientsatthesamespatiallocationbutatdifferentscales[ 115],orat adjacentspatiallocations[ 116].Thesedependenciesareexploitedbyusingmultivariate thresholdingmethods.Forinstancein[ 115],bivariateshrinkagerulesaredeveloped, whichexploittheinterdependencybetweencoefcientsattwoadjacentscalesleading 119

PAGE 120

to superiorimagedenoisingperformance.Anotherpopularwavelet-baseddenoising techniquewhichexploitsinterdependencyofthecoefcientsistheBLS-GSM(Bayesian leastsquaresforGaussianscalemixtures)developedin[ 117].Thismethodassumes thatthedistributionofaneighborhoodofwaveletcoefcients(denedascoefcientsat adjacentscales,orientationsorlocations)canbemodeledasaGaussianscalemixture (apositivehiddenvariablemultipliedbyaGaussianrandomvariable).Assuminga suitableprioronthishiddenrandomvariable,andgivenasetofwaveletcoefcients fromanoisyimage,onecanformanestimateofthetruewaveletcoefcientgivenits neighborsusingaBayesianleastsquaresmethod. Itshouldbenotedthatestimatesusingcoefcientthresholdingschemesareshown tobemaximum aposteriori (MAP)estimatesofthetruesignalcoefcientsgiventhose ofthedegradedsignal,bymakingsuitableassumptionsonthestatisticsofwavelet coefcientsofcleannaturalimages[ 116],[ 118].Typically,thegeneralizedGaussian familyyieldsanexcellentpriorforthedensitiesofnaturalimagewaveletcoefcients [119].Thispriorcanbewrittenasfollows: p (z ; p p ) / e j z p j p (6) A Gaussianprior( p =2 )isknowntoyieldtheempiricalWienerestimateforthe coefcientsofthetrueimage,aLaplacianprior( p =1)correspondstothesoft thresholdingschemeandthehardthresholdingschemeisapproximatedbysmaller valuesof p [116].Doubtingthevalidityofthesepriorsforeverynaturalimagein question,theauthorsof[ 120]learnaminimummeansquareerror(MMSE)estimator forthetruewaveletcoefcientsgiventhecorrespondingnoisycoefcients.Forthis purpose,theybuildatrainingsetofpatchesfromcleannaturalimagesandtheir degradedversion(assumingaxednoisemodel).Followingthis,theysolveasimple regressionproblemtooptimallyperturbcoefcientsofthedegradedpatchessoas toyieldvaluesclosetothoseofthecorrespondingcleanpatches.Forovercomplete 120

PAGE 121

representations ,theauthorsof[ 120]reportthattheregressionprocedureproduces non-monotonicthresholdingfunctions,adeviationfromallearlierthresholdingschemes drivenbyimagepriors. 6.5Non-localTechniques Thesetechniqueswhichwerepopularizedbytherecent`non-localmeans (NL-Means)'algorithm,publishedin[ 2]and[ 121],exploitthefactthatnaturalimages (andespeciallytextures)oftencontainseveralpatchesthatareverysimilartoeach other(asmeasuredinthe L 2 sense,forinstance).NL-Meansobtainsadenoisedimage byminimizingapenaltytermontheaverageweighteddistancebetweenanimage patchandallotherpatchesintheimage,wheretheweightsaredependentonthe squareddifferencebetweentheintensityvaluesinthepatches.Thisisexpressedbelow mathematically: ^ I =argmin I E (I ) (6) E (I )= )Tj /T1_4 11.955 Tf (1 f X x i y i log X x j y j exp )Tj /T1_7 11.955 Tf (f k I (0)Tj /T1_3 7.97 Tf 14.47 0 Td () patc h (x i y i ) )Tj /T1_2 11.955 Tf 11.96 0 Td (I (0)Tj /T1_3 7.97 Tf 14.47 0 Td () patch (x j y j )k 2 # (6) where I (0)Tj /T1_3 7.97 Tf () patch (x i y i ) isapatchcenteredatpixel ( x i y i ) oftheimage I excludingthecentral pixel.Takingthederivativeof E (I ) withrespecttoanypixelvalue I ( x i y i ) andsettingitto zero,yieldsusthefollowingupdateequation: ^ I (x i y i )= P x j ,y j w j I ( x j y j ) P x j ,y j w j (6) w j = exp )Tj /T1_7 11.955 Tf (f k I (0)Tj /T1_3 7.97 Tf 14.47 0 Td () patch (x i y i ) )Tj /T1_2 11.955 Tf 11.95 0 Td (I (0)Tj /T1_3 7.97 Tf 14.47 0 Td () patch (x j y j )k 2 (6) ItcanbeobservedfromthepreviousequationthatNL-Meansisapixel-basedalgorithm andindeedcanbeinterpretedasaspatiallyvaryingconvolutioninwhichtheconvolution maskisderivedusingnon-localimagesimilarity. Usually,justoneupdatestepyieldsgoodresults[2].However,forhighernoise levels,thealgorithmcancertainlybeiteratedseveraltimes[ 122].Theimplicitassumption 121

PAGE 122

in NL-Meansisthatpatchesthataresimilarinanoisyimagewillalsobesimilarinthe originalimage,asnoiseisi.i.d.Theessentialprincipleofimageself-similarityunderlying NL-Meansisthesameastheoneusedinfractalimagecodingmethods[ 123].The NL-Meansalgorithmcanalsobeinterpretedasaminimizeroftheconditionalentropy ofacentralpixelvaluegiventheintensityvaluesinitsneighborhood[ 124],[ 125],and henceitisrootedinsimilarprinciplesasthefamousEfros-Leungalgorithmfortexture synthesis[ 126].Theconditionalentropyisestimatedfromtheconditionaldensity whichisobtainedusingonlythenoisyimagein[ 124]oranexternalpatchdatabasein [125].Otherdenoisingalgorithmsthatexploitimageself-similarityinclude[127]orthe long-rangecorrelationmethodproposedin[ 128]and[129].Avariationalformulation fortheNL-Meanstechniqueispresentedin[ 130].Theconceptofnon-localsimilarity istypicallyusedonlyinthecontextoftranslations,butcanalsobeextendedtohandle changesinrotation,scaleorafnetransformations,asalsochangesinillumination. Suchmodelshavebeenstudiedin[ 131],[ 132]. Critique: TheperformanceoftheNL-Meansalgorithmwillbeaffectedinthose regionsofanimagewhichdonothavesimilarpatcheselsewhereintheimage.The performanceofthetechniqueisalsodependentontheparameter f andthepatchsize. Indeed,forlargevaluesof f ,forlargepatchsizesorifthealgorithmisiteratedseveral times,theresidualsproducedmaydiscernibleimagefeatures(seeFigure9of[ 122]). ItiseasytointerpretoneiterationofNL-meansastheproductofarow-stochastic matrix A 1 ofsize N N withthenoisyimage(representedascolumnvector).Here N isthenumberofpixels.Theentriesof A 1 aregivenbytheweightsfromEqn. 6 IfNL-Meansisexecutediteratively,theweightmatrixwillchange.Letusdenotethe weight-matrixatthe i th iterationby A i .Thereforeaftermultipleiterations,theresulting imageisobtainedfromtheproductofthematrix A withtheoriginalvectorizedimage, where A isgivenby A = n Y i =1 A i (6) 122

PAGE 123

It hasbeenprovedrecently[ 133]thatthelimitingproductofanysequenceofrow-stochastic matricesyieldsamatrixwithallrowsidenticaltooneanother.Whensuchamatrixis multipliedwiththeimagevector,itinvariablyproducesaatimage.Thistheoremis mentionedinAppendixB.ThisprovesthatthelimitoftheNL-Meansalgorithmisaat image. Theaforementionednon-localformulationhasledtothedevelopmentoftheBM3D (blockmatchinginthreedimensions)method[ 134]whichisconsideredthecurrent stateoftheartinimagedenoisingwithexcellentperformanceshownonavarietyof images.Thismethodoperatesatthepatchlevelandforeachreferencepatchinthe image,itcollectsagroupofsimilarpatches.Intheparticularimplementationin[ 134], similarityisdenedintermsoftheEuclideandistancebetweenpre-lteredpatches. Thesesimilarpatchesarethenstackedtogethertoforma3Darray.Theentire3Darray isprojectedontoa3Dtransformbasis,wherecoefcientsbelowaselectedthreshold valuearesettozero.Thelteredpatchesarethenreconstructedbyinversionofthe transform.Thisprocessisrepeatedovertheentireimageinaslidingwindowfashion. Ateachstep,allpatchesinthegrouparelteredandthemultiplehypothesesgenerated forapixelareaveraged.Theauthorstermthecollectivelteringofagroupofpatches as`collaborativeltering'andclaimthatthegroupofpatchesexhibitgreatersparsity collectivelythaneachindividualpatchinthegroup,citingthatasthereasonforthestate oftheartperformanceoftheBM3Dmethod.Inthespecicimplementationin[ 134], the3Dtransformisimplementedinthefollowingway.First,theindividualpatchesare lteredbyprojectionontoa2Dtransformbasis(inthiscasethe2DDCTbasis)followed byhardthresholdingofthecoefcients.Onceallthesepatchesinthegroupareltered individually,eachpixelstack(consistingofthecorrespondingpixelsfromallthepatches) isagainlteredbymeansofa1DHaartransform.Themultiplehypothesesappearingat anypixelareaveragedtoproducethenalsmoothedimage. 123

PAGE 124

The denoisingresultsusingtheBM3Dmethodaretrulyoutstanding.However themethodiscomplexwithseveraltunableparameterssuchaspatchsize,transform thresholds,similaritymeasures,etc.Therefore,itmaynotbeveryeasytoisolate theexacteffectofeachcomponentonthedenoisingperformance.Furthermore,the stackingtogetherofsimilarpatchestoforma3Darrayimposesasignalstructureinthe thirddimension.Infact,onewouldexpecttheorderingoftheindividualpatchesinthe 3Darraytoaffectthelterperformance. Anotherverycompetitive(albeitcomputationallyexpensive)approachforimage denoising,whichmakesuseofnon-localsimilarity,isthetotalleastsquaresregression methodintroducedin[ 135].Inthismethod,foreachreferencepatchinthenoisyimage, agroupofsimilarpatchesiscreated.Thereferencepatchisthenexpressedasalinear combinationofthesimilarpatchesandthecoefcientsofthislinearcombinationare obtainedusingtotalleastsquaresregression.Ascomparedtoasimpleleastsquares regression,thetotalleastsquaresregressionaccountsforthefactthatthenoiseexists inthereferencepatchaswellastheotherpatchesinthegroup.Thecomputational complexityiscubicinthenumberofpatchesinthegroup,whichisadrawbackofthis approach. 6.6UseofResidualsinImageDenoising Thereexistssomeresearchwhichtriestomakeuseofthepropertiesoftheresidual todriveorconstraintheimagelteringprocess.Undertheassumptionofanoisemodel, theoverallideaistodrivethedenoisingtechniqueinsuchawaythattheresidual possessesthesamecharacteristicsasthenoisemodel. 6.6.1ConstraintsonMomentsoftheResidual Oneoftheearliestamongtheseisanapproachfrom[86]whichassumesa Gaussian(i.i.d.)noisemodelofknown andtriestoimposeconstraintsonthestatistics oftheresidual(meanandvariance)ineachiterationofthelteringprocess.Starting fromanoisyimage I 0 ,theiralgorithmtriestondasmoothedimage I (bothdened 124

PAGE 125

on adomain n )thatminimizestheenergyfunctionalgiveninEquation 6.The correspondingEuler-LagrangehasaLagrangemultiplierwhichiscomputedbygradient projection,takingcaretoensurethattheconstraintsarenotviolated[ 86].Asimilar approachhasalsobeenindependentlyproposedin[ 136]. 6.6.2AddingBackPortionsoftheResidual Intraditionaldenoising,thelteringalgorithmisrun(forsome K iterations)to produceasmoothedimage,andtheresidualisignored.IntheapproachbyTadmor, NezzarandVese(called`TNV')[ 137],asmoothedimage J 1 isobtainedfromanoisy image J 0 byminimizinganenergyfunctionalcontainingtwoterms:thetotalvariation of J 1 ,andthemeansquaredifferencebetween J 0 and J 1 integratedoverthedomain (datadelityterm).Thisiscalledastherststepofthealgorithm.Theresidual J 0 )Tj /T1_3 11.955 Tf 12.2 0 Td (J 1 however,isnotdiscarded.Insteadthesamelteringalgorithmisnowagainrunonthe residual,inasecondstep.Thisdecomposes J 0 )Tj /T1_3 11.955 Tf 12.19 0 Td (J 1 intothesumofasmoothedimage J 2 andanotherresidual J 0 )Tj /T1_3 11.955 Tf 12.73 0 Td (J 1 )Tj /T1_3 11.955 Tf 12.74 0 Td (J 2 J 2 isaddedbacktothedenoisedoutputofthe rststep,i.e.to J 1 .Thisprocedureisrepeatedsome ^ K times,yieldinganal`denoised image' J 1 + + J ^ K .As ^ K !1,theauthorsof[ 137]provethattheoriginalnoisyimage isobtainedagain.Inpractice,anupperboundisimposedon ^ K asafreeparameter.A similaralgorithmhasalsobeendevelopedbyOsher etal. [138]withamodieddata delityterm. Critique: Forbothtechniques, ^ K isacrucialfreeparameter.Alsowhenthe smoothedresidualisaddedbackateverystep,somenoisemayalsogetaddedto thesignal.Inexperimentalresultspublishedinacomprehensivesurveyofimage denoisingalgorithms[ 2],theresidualsobtainedontheLenaimageusingmethodsfrom [137]and[138]arenottotallydevoidofimagefeatures. 6.6.3UseofHypothesisTests Thisapproachproposedin[139 ]assumesthattheexactnoisemodelisknown apriori andthattheunderlyingimageispiecewiseat.Tolteranoisyimage,the 125

PAGE 126

algor ithmtriestoapproximateitlocally(i.e.intheneighborhoodofsomeradius W aroundeachpoint)byaconstantvalueinsuchawaythattheresidualsatisesthenoise hypothesis.Infact,itchoosesthemaximumvalueof W forwhichthelocaldistribution oftheresidualinasmallneighborhoodaroundanyimagepixelisclosetotheassumed noisedistribution.Here,`closeness'isdenedusingoneofthecanonicalhypothesis tests.Thisprocedureisrepeatedateverypointintheimagedomain.Itshouldbe notedthatthisproblemisdifcultintwoormoredimensions,whereasin1Ditcanbe solvedeasilyusingadynamicprogrammingmethodlikeasegmentedleastsquares approach[ 140].Anotherrelatedpaper[ 141]presentsanalgorithmthatissimilartothe TNVapproachdescribedearlier,withoneimportantchange:the`denoised'residualis addedbackonlyatthosepoints (^ x ,^ y ) suchthattheresidualinaneighborhoodaround (^ x ,^ y ) violatesthehypothesisthatitconsistsofasetofsamplesfromtheassumed noisedistribution.Experimentalresultsaredemonstratedwithaverysimpleisotropic Gaussiansmoothingalgorithmwithadecreaseintheamountoffeaturesthatarevisible intheresidual. 6.6.4ResidualsinJointRestorationofMultipleImages Theauthorsof[142 ]observethatwhenmultipleimagesofanobjectacquired, thenoiseaffectingtheindividualimagesisoftenindependentacrosstheimages,even ifthenoisemodelisnotindependentoftheunderlyingsignal.Theyexploitthisina denoisingframeworkthatenforcestheindividualresidualsforeachoftheimagestobe independentofoneanother.Theparticularindependencemeasurechosenisthesum ofpairwisemutualinformationvalues.Aniterativeoptimizationprocedureisproposed. Critique: Asarguedinsection 6.6.1 ,merestatisticalconstraintsdonotguarantee `noiseness'oftheresidual,especiallyifmorecomplicatedimagemodelsaretobe considered.Abiggerproblemisthatmerelysatisfyingthepropertiesoftheresidualis notguaranteedtoleadtoadequaterestorationoftheimagegeometry.Infact,adirect enforcementofnoise-likepropertiesoftheresidualcanleadtoseriousundersmoothing. 126

PAGE 127

The propertiesoftheresiduals,canhoweverbeusedforautomaticallyndingindividual smoothingparameters,aswillbediscussedinChapter 8 6.7DenoisingTechniquesusingMachineLearning InthetransformdomainmethodsdiscussedinSection 6.4 ,axedtransformbasis ischosenforsignalrepresentation.Thereexistseveralpaperswhichattempttotune thetransformbasisbasedonthestatisticsofimagefeaturesorpatches.Forinstance in[ 118],theauthorsusenoise-freetrainingdatatolearnindependentcomponents ofthetrainingvectors.ThelearnedICAbasisisthenusedtodenoisenoisyimage patchesusingamaximumlikelihoodmodel,leadingtoasoftshrinkageoperation.In thisparticularcase,thelearnedbasisisorthonormal.However,therehasbeenrecent interestinlearningovercompletebases(alsocalleddictionaries),wherethenumber ofvectorsinthedictionaryexceedstheirdimension.Thishaslargelybeenpioneered byworkssuchas[ 143],[ 144],[ 145].Theseapproachesareofinterestbecausethe inherentredundancyofthevectorsinthedictionaryleadstomorecompact(sparser) representationofnaturalsignals.Infact,thesepapersspeciallytunethedictionariesin suchawaythatnaturalimagepatchespossesssparserepresentationswhenprojected ontothedictionary. Inthemorerecentliterature,theKSVDalgorithm[146],[ 147]hasgainedpopularity intheimagedenoisingcommunity.Inthistechnique,startingfromoverlappingpatches fromanoisyimage,anovercompletedictionaryaswellassparserepresentationsof thepatchesinthatdictionaryarelearnedinanalternatingminimizationframework. Thealgorithmhasproducedexcellentresultsondenoising[ 146].ThenameKSVD stemsfromthefactthatthe K columnsofthedictionaryareupdatedoneatatime usingasingularvaluedecomposition(SVD)operation.Amulti-scalevariantofthis algorithm(knownasMS-KSVD)learnsdictionariestorepresentpatchesattwoormore scalesleadingtofurtherredundancy[ 114].Thisalgorithmhasyieldedstateoftheart performance,onparwiththeBM3Dalgorithm[ 134]describedintheprevioussection 127

PAGE 128

[114]. HoweverKSVDandMS-KSVDbothrequireanexpensiveiteratedoptimization procedureforwhichnoconvergenceproofhasbeenestablishedsofar.Thealternating minimizationframeworkissubjecttolocalminima[ 114]andrequiresparameterssuch aslevelofsparsity.Someoftheseparametersarechosentobeadirectfunctionof thenoisevariance.Howeverinsuccessiveiterationsoftheoptimization,theimage ispartiallysmoothed,andthisthereforeaffectsthequalityofsubsequentparameter updateswhichareaffectedbythechangesinthenoisevariance(seesections3and4 of[ 146]). IntheKSVDapproach,asingleovercompletedictionaryislearnedfortheentire image.Asopposedtothis,theauthorsof[ 148]performaclusteringsteponthepatches fromthenoisyimageandthenrepresentthepatchesfromeachclusterseparately usingprincipalcomponentsanalysis(PCA).Inpractice,theclusteringstep(K-means)is performedoncoarselypre-lteredpatchesandthelearnedPCAbasesarenecessarily oflowerrank.Thedenoisedintensityvaluesareproducedbymeansofakernel regressionframeworkfrom[ 65].Theentireprocedureisiteratedforbetterperformance. TheauthorscallthisastheKLLD( K locallylearneddictionaries)approach[ 148].The ideaofusingaunionofdifferentorthonormal(PCA)basesforeachcluster(asopposed toasinglecomplexbasisfortheunionofallclusters)isinteresting.However,the methodhasfreeparameterssuchasthepre-lteringprocedure,theclusteringalgorithm andthenumberofclusters. ItshouldbenotedthatbothKSVDandKLLDusenon-localpatchsimilarityin learningthebases.Hence,theycanalsobeclassiedasnon-localapproaches describedinSection 6.5 .Thesparsedictionarybasedmethodshavegainedpopularity notonlyinimagedenoisingbutalsootherrestorationproblemssuchassuper-resolution [149]. 128

PAGE 129

6.8 CommonProblemswithContemporaryDenoisingTechniques Therearecommonissuesconcerningmostcontemporarydenoisingtechniques whichwebrieyreviewinthissection. 6.8.1ValidationofDenoisingAlgorithms Thereisnoclearconsensusonthemethodsforvalidationoftheperformanceof denoisingalgorithms.Giventwodenoisingalgorithms A and B (ofsize N pixels)and theiroutputonanoisyinputimage,theprimaryrequirementisthatofavalidquality measureforcomparingtheirrelativeperformance.Thequalitymeasuredecidesthe proximitybetweenthedenoisedimageandthetrueimage(i.e.thecleanimage,devoid ofanydegradation).Themostcommonqualitymeasureisthemeansquarederror (MSE)denedasfollows MSE (A, B )= 1 N N X i =1 (A i )Tj /T1_2 11.955 Tf 11.96 0 Td (B i ) 2 (6) and thepeaksignaltonoiseratio(PSNR)whichiscomputedasfollowsfromtheMSE PSNR ( A, B )=10log 10 255 2 MSE ( A, B ) (6) The lowertheMSE(orhigherthePSNR),thebettertheperformanceofthedenoising algorithm. Now,theidealqualitymeasureshouldbeintunewithwhatweashumansperceive tobea`better'image.Thisisatrickyissueasitisaffectedbyseveralfactorsincluding thetypeofdisplaysystem.WhiletheMSEisaveryintuitivemeasure(andindeedis alsoametric),itisnotnecessarilyintunewithperceptualqualitybecauseitweighs errorsineverypixelequally.However,thehumaneyeishardlysensitivetominorerrors inhigh-frequencytexturedregionssuchasthefurofthemandrillinFigure 6-1,or carpettexture.Therefore,evenifanimagecontainsperturbationsinthehigh-frequency texturedportionsofanimageandconsequentlyhashighMSE,itmaystillberegarded 129

PAGE 130

as agoodqualityimagefromaperceptualpointofview.Severalsuchlimitationsofthe MSE/PSNRhavebeendocumentedin[ 150]withnumerousexamples. Furthermore,theauthorsof[150],[ 151]proposeanewqualitymeasuretermed thestructuredsimilarityindex(SSIM)whichmeasuresthesimilaritybetweenthe correspondingpatchesofimages A and B .Thesimilarityismeasuredintermsof theproximityofthemeanvaluesofthepatches,theirvariancesandalsoastructural similarityintermsofthecorrelationcoefcient.Giventwopatches A ( i ) and B (i ) from images A and B respectively,thisisrepresentedasfollows: SSIM (A (i ) B ( i ) )= 2 a b 2 a + 2 b 2 a b 2 a + 2 b ab a b (6) = 2 a b 2 a + 2 b ab 2 a + 2 b (6) where a and b are themeanvalueofpatches A (i ) and B ( i ) respectively, a and b are theirrespectivestandarddeviationsand ab isthecovariancebetweenthepatches.For comparisonbetweenthecompleteimages,themeasureisdenedas SSIM (A, B )= 1 NP X i SSIM (A (i ) B (i ) ) (6) where NP is thenumberofnon-overlappingpatches.Inpractice,thestatisticsfrom allpatchlocationsarenotweighedequally,butusingasymmetricGaussianwindow ofsomechosen(small)standarddeviation[ 151].TheSSIMisknowntocorrelatewell withthehumanvisualsystem[ 151],howeveritisunstablewhenanyofthetermsin thedenominatorsapproachzero,andrequiresappropriatescaleselection.Whilethere existsamulti-scaleequivalent[ 63](denotedasMSSIM)whichcombinesSSIMvalues atdifferentimagescales,thechoiceofscaleformeasurementofthestatisticsisstill anopenissue.IntheexperimentalresultsreportedinChapter 7,weperformvalidation usingPSNR,usingSSIMwiththedefaultparametersettingsusedbytheauthorsof[ 63 ] (forinstancewindowsizeof 11 11 ). 130

PAGE 131

6.8.2 AutomatedFilterParameterSelection Whileresearchonimagedenoisinghasbeenveryextensive,theliteratureon automatedmethodsforselectingappropriatelterparametersisnotverylarge.Most techniquesselectthebestparameterretroactivelyintermsofoptimizingafull-reference imagequalitymeasure.WedeferfurtherdiscussiononthistopictoChapter 8 A B C Figure 6-1.Mandrillimage:(A)withnonoise,(B)withnoiseof =10,(C)withnoiseof =20;thenoiseishardlyvisibleinthetexturedfurregion(viewedbest whenzoomedinthepdfle) 131

PAGE 132

CHAPTER 7 BUILDINGUPONTHESINGULARVALUEDECOMPOSITIONFORIMAGE DENOISING 7.1Introduction Thischapterdescribestwonewalgorithmsforgray-scaleimagedenoising.Our methodsarelargelybasedupontheclassicaltechniqueofsingularvaluedecomposition (SVD),apopularconceptinlinearalgebra.TheSVDwasrstappliedforimage lteringandcompressionapplicationsin[ 152]and[ 153].Onastand-alonebasis,its performanceonlteringleavesmuchtobedesired.Howeverduringthisthesis,we haveexploredseveralideaswhichbuildupontheSVD,leadingtosimpleandelegant techniqueswithexcellentperformance.Manyoftheintermediateideasthatwere explored,failedtoproducegoodresultsintermsofdenoisingperformance.Whilethe vastmajorityofcontemporaryresearchliteraturefocusesonlyonpositiveresults,we choosetoadoptadifferentphilosophy.Weshallpresentnegativeresults(andwherever possible,analyzethereasonsforthenegativeresults)inadditiontothepositiveones thatareonparorbetterthanthestateoftheart.Wehopethatthiswillprovidereaders ofthisthesiswithbetterinsightandopenupideasforfutureresearch. Weobservethataprincipleddenoisingtechniquecanbemotivatedbythefollowing considerations.Whatconstitutesagoodmodelfortheimagesbeingdealtwith?What isknownaboutthenoisemodel?Whatpropertiesdistinguishacleanimagefrom onecontainingpurenoise?Wemakethefollowingassumptionsinthetheoretical descriptionandexperimentalresults.Weassumeagray-scaleimageintheintensity range [0,255] denedonadiscreterectangulardomain n .Inthetechniquesthatwe investigate,weexploitdifferentwell-knownpropertiesofnaturalimages.Weassume azeromeani.i.d.(independentandidenticallydistributed)Gaussiannoisemodelofa xedstandarddeviation ,asthecommondegradationprocess.Wedonotcoverthe caseofsignal-dependentnoiseorthosewithoutpreciseprobabilisticcharacteristics (suchasnoiseinducedbylossycompressionalgorithms)inthisthesis. 132

PAGE 133

7.2 MatrixSVD ThematrixSVDisapopulartechniqueinlinearalgebrawithawidevarietyof applicationsinsignalprocessing,suchasltering,compressionandleast-squares regressiontonameafew.Givenamatrix A ofsize m n denedontheeldofreal numbers,therealwaysexistsafactorizationofthefollowingform[ 154] A = USV T (7) where U isa m m orthonormalmatrix, S isa m n diagonalmatrixofpositive `singular'valuesand V isa n n orthonormalmatrix.Conventionally,theentriesof S arearrangedindescendingorderofmagnitude.Moreover,ifthesingularvalueshappen tobeunique,thematrixSVDisalwaysunique,modulosignchangesonthecolumnsof U and V .Thecolumnsof V (calledtherightsingularvectors)aretheeigenvectorsof A T A,whereasthecolumnsof U (calledtheleftsingularvectors)aretheeigenvectorsof AA T andthesingularvaluesin S turnouttobethesquarerootsoftheeigenvaluesof A T A (equivalently AA T ).AgeometricinterpretationoftheSVD(assumingrealvector spaces)ispresentedin[ 154].ThematrixSVDhasbeautifulmathematicalproperties suchasprovidingaprincipledmethodforthenearestorthonormalmatrix,andthebest lower-rankapproximationtoamatrix(bothinthesenseoftheFrobeniusnorm)[ 154]. 7.3SVDforImageDenoising Itiswell-knownthatthesingularvaluesofnaturalimagesfollowanexponential decayrule[ 155].ThispropertyalsoholdstrueforFouriercoefcientmagnitudes.In fact,theSVDbaseshaveafrequencyinterpretation.Thesmallersingularvaluesofthe imagecorrespondtohigherfrequenciesandthelargevaluescorrespondtothelower frequencycomponents.ThispropertyoftheSVDhasbeenusedbothindenoising[ 152] aswellasincompression[ 153]. Now,consideranoisyimage A (adegradedversionofanunderlyingclean image A c )affectedbyadditiveGaussiannoiseofstandarddeviation .Filteringis 133

PAGE 134

accomplished bycomputingthedecomposition A = USV T andthennullifyingthe smallervaluesof A,whicheffectivelydiscardshigherfrequencycomponents(whichare knowntocorrespondmostlytonoise)[ 152].Anexampleofthisprocedureisillustrated inFigure 7-1,whereallsingularvaluessmallerthansome k th singularvalueweresetto zero.Itisclearlyseenthatlowranktruncation(i.e.iftheindex k ischosentobesmall) producesblurryimagesandincreasingtherankaddsinimagedetailsbutintroduces moreandmorenoise.Takingthissub-parperformanceintoaccount,thisdecomposition isinsteadperformedatthelevelofimagepatches.Indeed,smallpatchescapturelocal informationwhichcanbecompactlyrepresentedwithsmall-sizedbases.TheSVDis computedinaslidingwindowfashionandlteredversionsofoverlappingpatchesare averagedinordertoproduceanallteredimage.Theaveragingisusefulforremoving seamartifactsatpatchboundariesandalsobringsinmultiplehypotheses.Theseresults areshowninFigure 7-2 fordifferentsettings:(1)rank1andrank2truncationofeach patch,(2)nullicationofpatchsingularvaluesbelowaxedthresholdof p 2 log N (where N isthenumberofimagepixels),and(3)truncationofsingularvaluesinsuch awaythattheresidualateachpatchhasastandarddeviationof (i.e.astandard deviationequaltothatofthenoise). 7.4OracleDenoiserwiththeSVD Despitetheimprovementinresultswiththepatch-basedmethod(asseenupon comparingFigures 7-2 and 7-1),thelteringperformanceisstillfarfromdesirable. Themainreasonforthisisthatthesingularvectorsareunabletoadequatelyseparate signalfromnoise.Therearetwokeyobservationswemakehere.Firstly,let Q and Q n becorrespondingpatchesfromacleanimageanditsnoisyversionrespectively. Giventhedecomposition Q n = U n S n V T n ,theprojectionof Q (thetruepatch)ontothe bases (U n V n ) isgivenas S Q = U T n QV n .Thismatrix S Q isnon-diagonalandhence containsmorenon-zeroelementsthan S n .Notethefactthat S Q is`denser'than S n Despitethis,ifwecouldsomehowchangetheentriesin S n tomatchthosein S Q ,we 134

PAGE 135

w ouldnowhaveaperfectdenoisingtechnique.Nevertheless,SVD-basedltering techniquesemphasizelow-ranktruncationorothermethodsofincreasingthesparsity ofthematrixofsingularvalues.WeshalldwellmoreonthispointinSection 7.5 in thecontextoflteringwithSVDbasesaswellasuniversalbasessuchastheDCT. Thesecondimportantobservationisthattheadditivenoisedoesn'tjustaffectthe singularvaluesofthepatchbutthesingularvectors(whicharetheeigenvectorsofthe row-rowandcolumn-columncorrelationmatricesofthepatch)aswell.Bearingthisin mind,itisstrangethatSVD-baseddenoisingtechniquesdonotseektomanipulatethe orthonormalbasesandinsteadfocusonlyonchangingthesingularvalues.Wenow performthefollowingexperimentwhichstartswithanoisyimageandassumesthat thetruesingularvectorsofthecleanpatchunderlyingeverynoisypatchintheimage areknownorprovidedtousbyanoracle.Thedenoisingtechniquenowproceedsas follows: 1.LettheSVDforapatch Q (i ) fromacleanimagebe Q (i ) = USV T .Projectthenoisy patch Q (i ) n ontothesebasestoproduceamatrix S Q ( i ) = U T Q (i ) V 2.Settozeroallelementsin S Q (i ) suchthat jS Q (i ) j < 3.Producethedenoisedversionof Q (i ) n byinvertingtheprojection. 4.Repeattheaboveprocedureinslidingwindowfashionandaverageallthe hypothesesateverypixeltoyieldadenoisedimage. Wetermthismethodasthe`oracledenoiser'.Given 8 8 patches,wechoosethe thresholdof =3 forthefollowingreasons.Firstly,zeromeanGaussianrandom variableswithstandarddeviation (i.e.belongingto N (0, ))havevaluelessthan 3 withhighprobability,andprojectionsofmatricesofGaussianrandomvariablesonto orthonormalbasesalsoobeythisrule(experimentally,thisprobabilitywasobservedto beverycloseto1).Secondly, =3 comesclosetotheideathresholdof = p 2 log n 2 from[ 113]forpatchesofsize n n 135

PAGE 136

Sample experimentalresultswiththeabovetechniqueareshowninFigure 7-3 fortwonoiselevels:20and40.TheresultingPSNRvaluesofthisidealdenoiserfar exceedthestateoftheartmethodssuchasBM3D[ 134].Clearly,thisexperimentis notpossibleinpractice,howeveritservesasabenchmark,driveshomeanimportant deciencyofcontemporarySVDlteringapproaches,andchalksoutapathforusto explore:manipulatingtheSVDbasesofanoisypatch,orsomehowusingbasesthatare `better'thantheSVDbasesofthenoisypatch,maybethekeytoimprovingdenoising performance. 7.5SVD,DCTandMinimumMeanSquaredErrorEstimators AshasbeendescribedinSection 6.4,nullicationofthesmallercoefcientvalues fromtheprojectionofanoisypatchontoabasisisactuallyaMAPestimatorofthe coefcientsofthetruepatch.TheMAPestimatorisdrivenbysparsity-promotingimage priorswhichholdforimageensemblesbutnotnecessarilyforeveryindividualimage. Wethereforeexploreminimummeansquareerror(MMSE)estimatorsforestimationof thetrueprojectioncoefcients. 7.5.1MMSEEstimatorswithDCT TheideaofusingMMSEestimatorsisinspiredbytheworkin[120].However,there isonemajordifferencebetweentheapproachfrom[ 120]andtheonewepresenthere. In[ 120],theauthorslearnagenericruletooptimallyperturbtheDCTcoefcientsof anensembleofnoisyimagepatchessoastoreducethemeansquarederrorwiththe DCTcoefcientsoftheircorrespondingunderlyingcleanpatches.Adifferentruleis learnedforeachDCTcoefcient(thenumberofcoefcientsisequaltothepatch-size) orforeachsub-band,thoughalltherulesarecommonacrosspatches.However,we haveobservedexperimentallythattheoptimalrulesforpatchesofdifferentgeometric structuresdiffersignicantlyfromoneanother(seeFigure 7-5).Therefore,wemove awayfromthenotionofasinglesetofrulesfortheentireensembleandinsteadlearn adifferentsetofrulesforeachtrainingpatch.Wemakethedenitionoftheword`rule' 136

PAGE 137

more preciseinthefollowing.Considerthe i th patchfromadatabase P D of N patches. Weshalldenotethepatchas I i .Letitssizebe n n andletits k th DCTcoefcientbe ^ I i (k ) where 1 k n 2 .Letusdenote J ij asthe j th noisyinstanceofpatch I i where 1 j M ,andlet ^ J ij ( k ) beits k th DCTcoefcient.Thenforeach 1 k n 2 andfor eachpatch 1 i N ,wemayseekaperturbation ki suchthat ki = min M X j =1 ( ^ J ij (k ) + )Tj /T1_5 11.955 Tf 12 2.66 Td (^ I i (k ) ) 2 (7) Unfortunately,thevaluesofcorrespondingDCTcoefcientsbelongingtomultiplenoisy instancesofapatchshowconsiderablevariance,whichpreventsthelearningofany meaningfulperturbationrule.Toalleviatethisproblem,wequantizethevaluesofeach DCTcoefcientintoaxednumberofbins,say B .Thusforthe k th coefcientofthe i th patch,wenomorelearnasinglescalarvalue,butasetof B perturbationvalues f kib g oneforeachbin.Thiscanbemathematicallyexpressedas kib = min M X j =1 b ( ^ J ij (k ) )( ^ J ij (k ) + )Tj /T1_5 11.955 Tf 12 2.65 Td (^ I i (k ) ) 2 (7) where b ( ^ J ij (k ) )= 8 > < > : 1 if b m (1) ik )Tj /T1_2 7.97 Tf 6.59 0 Td (m (2) ik B c = b 0 otherwise. (7) In theaboveequation,wedenethefollowingterms: m (1) ik = max j ^ J ij (k ) (7) m (2) ik = min j ^ J ij (k ) (7) Notethatthequantizationofthecoefcientsismotivatedbythefactthattheperturbation ofthecoefcientsowingtocorruptionbyGaussiannoiseshowssomeregularity,since randomvariablesfrom N (0, ) liewithinaboundedinterval [)Tj /T1_5 11.955 Tf (3 ,+3 ] withveryhigh probability. 137

PAGE 138

No w,givenanoisyimage(whichdoesnotappearinthedatabase P D ),wedivide itintopatches.Foreachnoisypatch P ,wesearchforitsnearestneighborfromthe trainingpatchdatabase P D .Lettheindexofthisnearestneighborbe s .Wenowapply thecorrespondingrulesalreadylearned,i.e.theperturbations f ksb g (1 k n 2 ), todenoisethepatch P .Asperthisrule,the k th coefcientof P ,denotedby ^ P (k ) ,is changedto ^ P (k ) = ^ P (k ) + ksb (7) where b isthebinforwhich b ( ^ P ( k ) )=1 ItisquitepossiblethatthevalueofaparticularDCTcoefcient ^ P (k ) fallsoutside therange [m (2) sk m (1) sk ].Insuchcaseswefollowtheheuristicmethodofapplyingthe perturbationfromthebinthatliesclosesttothevalue ^ P (k ) .FromEquation 7,wealso seeanimplicitassumptionthattheperturbationvaluesareconstantwithinanybin. Whilemoresophisticatedperturbationfunctions(say,linearwithinanybin)arepossible, westicktopiecewiseconstantfunctionsforsimplicity. 7.5.2MMSEEstimatorswithSVD WehavepreviouslymotivatedthefactthatusingbetterSVDbasescanhelpin improvingdenoisingresults.Supposethatforeachpatch I i inthepatchdatabase P D wecomputeitsSVDas I i = U i S i V T i .Weconjecturethatthebases (U i V i ) canserveas effectivedenoisinglters.Again,let J ij bethe j th noisyinstanceofpatch I i (1 j M ). Theprojectionof J ij onto (U i V i ) is S ij = U T i J ij V i .Weseektolearnrules f kib g forvalues of ^ J ij (k ) quantizedinto B binsjustasinEquation 7.Now,givenapatch P fromanoisy image,letitsnearestneighborfromthedatabasebepatch I s .Wethenproject P onto (U s V s ) givingusthematrix S s = U T s PV s andwemodifythecoefcientsin S s usingthe perturbationrules f ksb g (1 k n 2 ,1 b B )alreadylearnedfor I s .Theperturbation iscarriedoutinthesamewayasinEquation 7. 138

PAGE 139

7.5.3 ResultswithMMSEEstimatorsUsingDCT 7.5.3.1Syntheticpatches Werstexperimentwithasetof15syntheticallygeneratedpatches(allofsize 8 8)ofdifferentgeometricstructures.Wegenerated500noiseinstancesofeach patchfrom N (0,20) .Aquantizationof20binswasusedforeveryDCTcoefcient. ThesyntheticpatchesareshowninFigure 7-4.Thestatisticsofthemeansquared errorsbetweenthetrueandreconstructedpatches(namelytheaverage,maximum andmedianreconstructionerrors,allmeasuredacrossthedifferentnoiseinstances) areshowninTable 7-1 fortwomethods:theMAPestimatorwhichsetstozeroallDCT coefcientswhoseabsolutevalueisbelow 3 (towhichweshallhenceforthrefertoas theMAPestimator),andtheMMSEestimatordescribedpreviously.Clearly,theMMSE errorsareconsistentlylower.Forsomepatches(suchastheX-shapedpatchinFigure 7-4),weobtainedperturbationfunctionsthatwerenotstrictlymonotonic,ascanbeseen inFigure 7-8. 7.5.3.2Realimagesandalargepatchdatabase Next,webuiltacorpusof12000patchesofsize 8 8 takenfromtherstve imagesoftheBerkeleydatabase[ 61],allconvertedtogray-scale.Thesizeofeach imagewasabout 320 480.Wegenerated500noiseinstancesofeachpatchfrom N (0,20) .TheperturbationvalueswerelearnedasindicatedinEquation 7 fora quantizationof30binspercoefcient.Duringtraining,weagainconsistentlyobserved lowerreconstructionerrorsfortheMMSEestimatorthantheMAPestimator.Next, givenanoisyimage,wedivideditintonon-overlappingpatchesanddenoisedeach patchaspertheperturbationfunctionslearnedforthenearestneighbor(inthecorpus) correspondingtoeachpatch.ThereconstructionresultswiththisMMSEmethodas wellastheMAPestimatorareshowninFigures 7-6 and 7-7.Aquickglancereveals thatreconstructionwiththeMAPestimatorexhibitsconsiderablymoreringingartifacts thantheMMSEestimator.Butowingtothenon-overlappingnatureofthepatches,both 139

PAGE 140

the MMSEandMAPreconstructionsshowpatchseamartifacts.Theseseamartifacts canbeeliminatedbydenoisingoverlappingpatchesandthenaveragingtheresultsas showninFigures 7-6 and 7-7.Surprisingly,weobtainlowerPSNRvaluesfortheMMSE methodwithoverlapthanforMAPwithoverlap.Weascribethisdropinperformanceof theMMSEestimatortotwofactors:errorsintheresultsofthenearestneighborsearch fornoisyadjacentpatches(theaccuracyofwhichwillbeaffectedbynoise),andmuch moreimportantly,errorsduetothelimitedpatchrepresentationinthedatabase.Indeed, thenearestneighborfromthedatabasemaynotbecloseenoughtoproduceanMMSE estimatorthatproducesareconstructioncloseenoughtothetrueunderlyingpatch. 7.5.4ResultswithMMSEEstimatorsUsingSVD WenowexplorewhathappensifsimilarexperimentsareperformedonSVDbases (whicharepropertiesofindividualpatches)ratherthanonuniversalbases. 7.5.4.1Syntheticpatches Werantheexperimentonthesame15syntheticpatchesasinSection 7.5.3.1 with500noisyinstancesofeachpatchdrawnfrom N (0,20) .Aquantizationof20 binswasusedforeverySVDcoefcient.ThesyntheticpatchesareshowninFigure 7-4.Thestatisticsofthemeansquarederrorsbetweenthetrueandreconstructed patches(namelytheaverage,maximumandmedianreconstructionerrors,allmeasured acrossthedifferentnoiseinstances)areshowninTable 7-2 fortwomethods:theMAP estimator,andtheMMSEestimatordescribedpreviously.TheMMSEerrorsareagain consistentlylowerthantheMAPerrors.Forsomepatches(suchastheX-shapedpatch inFigure 7-4),weagainobtainedperturbationfunctionsthatwerenotstrictlymonotonic, ascanbeseeninFigure 7-4.NoticethattheerrorswithMMSEestimatorsonSVDare muchlowerthanthosewithDCT(compareTables 7-1 and 7-2),thereasonbeingthat inthisexperiment,wehaveaccesstotheSVDbasesofthetrueunderlyingpatches (whereastheDCTwasauniversalbasis). 140

PAGE 141

7.5.4.2 Realimagesandalargepatchdatabase WeusedthesamecorpusofpatchesgeneratedinSection 7.5.3.2.TheSVD baseswerecomputedforall12000patches.Perturbationruleswerelearnedtochange thevaluesoftheprojectionmatrixtooptimizeaverageMSEacrossnoiseinstances andtheseruleswerestored.Next,patchesfromagivennoisyimage(again,different fromanyofthetrainingimages)wereprojectedontotheSVDbasesofthenearest neighborinthecorpus.ThecoefcientsweremanipulatedwiththeMAPruleaswell asthelearnedMMSErulestoproducetwoseparateoutputs.Tooursurprise,the performanceoftheMMSEestimatorwasverypoor.TheMAPestimatorwithSVD performedreasonablywellbutnotaswellastheoneappliedonDCTbases.These resultsareshowninFigure 7-9 ontheBarbaraimagewhichwassubjectedtonoise from N (0,20) (startingPSNR21.5).ThePSNRvalueswithMMSEonSVD,MAPon SVDandtheoracleestimatorwere25.2,28.85and36.6respectively.Basedonthis, wedrawthefollowingconclusions.TheMMSEerrorswereverylowduringtrainingbut highduringtesting.ThisclearlyindicatesanoverttingproblemwhendealingwithSVD bases,whichwasmuchmoreseverethanwhiledealingwithDCTbases.Considerthat wearegivenanarbitrarytrainingdatabase,andanarbitrarilychosennoisyimagefor testing.Itishighlyunlikelythatwecouldndanexactmatchinthedatabaseforevery imagepatch.Therulesthatwerelearnedonthenoisyinstancesoftheexactsame patchdonotseemtoapplyverywelltoother`similar'patches. However,wewishtoemphasizethatthereisstillmeritintheideaofattemptingto manipulatetheSVDbases.Thisisevidencedbytheimprovementintheperformanceof theMAPestimatorappliedonprojectionsontotheSVDbasesofthenearestneighbor fromthedatabase,overthatofthesameestimatorappliedtotheSVDbasesofthe noisypatchitself. 141

PAGE 142

7.6 FilteringofSVDBases WehaveobservedthattheSVDbasesofadjacentpatches(i.e.patcheswiththeir top-leftcornersatadjacentpixels)fromcleannaturalimagestendtoexhibitgreater similaritythanthosefromnoisyversionsofthoseimages.Thesimilarityisquantiedin termsoftheanglesbetweenunitvectorsfromcorrespondingcolumnsofthe U matrices (orthoseofthe V matrices)oftheadjacentpatches.Thisobservationisclearlya propertyofnaturalimagepatches(andnotamereconsequenceofthefactthatwe computedSVDbasesofmatricesthathadseveralrowsorcolumnsincommon).With thisinmind,weexploredtheeffectofsmoothingthe U and V basesofadjacentpatches fromtheimageusingsomeaveragingtechniques.Therearethreewaysthiscouldbe done: 1.Smooth(saybysomesortofaveragingscheme)thecorrespondingcolumnsfrom the U matricesofadjacentpatches,andthecorrespondingcolumnsfromthe V matricesofadjacentpatches. 2.Smooth(saybysomesortofaveragingscheme)theouterproducts U i V T i (1 i n ,1 j n ),i.e.outerproductsofthecorrespondingcolumnsfromthe U and V basescomputedfromadjacentpatches. 3.RunadiffusionPDEdenedspecicallyfororthonormalmatricesonthe U bases andalsoonthe V bases(independently). Therearemathematicalcomplicationsthatariseintherstmethod:theaveragingreally oughttobedonebyrespectingthegeometryofthespaceoforthonormalmatrices. However,theorthonormalmatriceswithdeterminant+1aredisjointfromthosewith determinant-1.Thisisproblematicfromthepointofviewofcomputingintrinsic averages.Furthermore,independentaveragingofthe U and V matricesignoresthe inherentcouplingbetweenthem(asgivenapatch P ,theyareeigenvectorsof P T P and PP T respectively).Takingaveragesofouterproductsofcorrespondingcolumnsfrom U and V helpsbringinthisdependence.However,itstillignoresthedependencebetween thedifferentouterproductsthemselves. 142

PAGE 143

Ignor ingtheabovemathematicalissues,wecomputedEuclideanaverages.Asthe resultantmatriceswerenomoreorthonormal,weorthonormalizedthemusingaQR decomposition.Whilecomputingaveragesofouterproducts(inmethod2),thereare considerablecomplicationsinforcingtheaveragedouter-producttolieinthespaceof matricesoftheform v 1 v T 2 where jv 1 j = j v 2 j =1,whichwereignoredinourexperiments. Weperformedimagedenoisingexperimentsbyrstsmoothingthebasescomputed from 8 8 patchesusingeitherofthethreetechniques,projectingthepatchesonto thebases,applyingtheMAPruleonthecoefcientsoftheprojectionmatrixand reconstructingthepatchbyinvertingthetransform.IncaseofthediffusionPDEdened fororthonormalmatrices,weusedthefollowingisotropicheatequationdenedin[ 156] formatrix U 2 SO (p p ): dU k dt = L k + p X i =1 ( L i U k )U i (7) where L k = U k xx + U k yy (7) and U k stands forthe k th columnof U Notethatcouplingbetweenthe U and V matricescanbeimposedindirectlyby introductionofadatadelityconstraintonthepatch P inadditiontothesmoothness termonthe U and V matrices,andthenexecutingalternatingPDEs(Euler-Lagrange equations)on U and V .However,experimentalresultsonaveragingoftheSVDbases wereingeneralnotsatisfactory.Similarexperimentswererepeatedwithnonlocal averagingofsimilar U and V matricesfromdifferentregionsoftheimage,andtherewas noimprovementintheresults.Weconjecturethatthesmoothnessof U and V bases fromadjacentpatchesmaynotbeastrongenoughpropertyofnaturalimages. 7.7NonlocalSVDwithEnsemblesofSimilarPatches Wenowpresentanalgorithmforimagedenoisingusinganon-localextensionofthe SVD.Wecallthisalgorithmnon-localSVDorNL-SVD. 143

PAGE 144

W eknowthattheSVDofamatrix P 2 R m n isgivenas P = USV T wherethe columnsof U consistoftheeigenvectorsofthematrix Cr = PP T (7) wheretheelementof Cr fromthe i th rowand j th columnisgivenas Cr ij = X k P ik P jk =< P i P j > (7) where P i and P j standforthe i th and j th rowsof P respectively.Similarlythecolumnsof V consistoftheeigenvectorsofthematrix Cc = PP T (7) wheretheelementof Cc fromthe i th rowand j th columnisgivenas Cc ij = X k P ki P kj =< P T i P T j > (7) where P T i and P T j standforthe i th and j th columnsof P respectively.Notethat Cr and Cc aretherow-rowandcolumn-columncorrelationmatricesof P respectively.Wealso knowthattheSVDgivesustheoptimallow-rankdecompositionof P .Inotherwords,the optimalsolutionto E ( ~ P )= kP )Tj /T1_5 11.955 Tf 13.24 2.66 Td (~ P k 2 (7) subjecttotheconstraint rank ( ~ P )= kk < m k < n (7) isgivenby ~ P = kU k ~ SV T k k 2 (7) where U k and V k aretherst k columnsof U and V respectivelyand ~ S containsthe k largestsingularvaluesof S .ThisisoftencalledastheEckhart-Youngtheorem[ 154]. 144

PAGE 145

Giv entheinadequateperformanceofthelocalpatchSVD,wecontinueoursearch for`better'basestorepresenteachpatch.Withthisinmind,wenowexplorewhatwould happenifweweretoconsideranon-localgeneralizationoftheSVD.Givenapatch P fromthenoisyimage,welookforotherpatchesintheimagethatare`similar'to P .We willgiveaprecisedenitionofsimilaritylaterinSection 7.7.1.Letusconsiderthatthere are K suchsimilarpatches(including P )whichwelabelas fP i g where 1 i K Next,weaskthefollowingquestion:what single pairoforthonormalmatrices U k and V k willprovidethebestrank-k approximationtoallthepatches fP i g ?Inotherwords,what (U k V k ) minimizesthefollowingenergy? E (U k f S i g, V k )= K X i =1 kP i )Tj /T1_1 11.955 Tf 11.96 0 Td (U k S i V T k k 2 (7) where U T k U k = I (7) V T k V k = I (7) 8 i S i 2 R k k (7) Thesolutiontothisproblemisgivenbyaniterativeminimization(startingfromrandom initialconditions)presentedin[ 157].Notethatthematrices fS i g inthiscaseare not diagonal.Notealsothatthebasis U k V k doesnotcorrespondtotheindividualSVD basesbuttoabasispairthatiscommontoallthechosenpatches.Relatedworkin [155]presentsanalternatingminimizationframeworkwiththeadditional(heuristically driven)constraintthatallthematrices fS i g arediagonal.Thisconstraintisimposedat everystepofthealternatingminimizationframework.Anapproximatesolutiontothe energyfunctioninEquation 7 ispresentedin[ 158].Thissolution,whichiscalled asthe2D-SVD,canbecomputedinclosedformandobviatestheneedforexpensive iterativeoptimizations.The2D-SVDforthepatchcollection fP i g isgivenasfollows. 145

PAGE 146

Consider therow-rowandcolumn-columncorrelationmatrices Cr = K X i =1 P i P T i (7) Cc = K X i =1 P T i P i (7) Then U k containstherst k eigenvectorsof Cr correspondingtothe k largesteigenvalues of Cr ,and V k containstherst k eigenvectorsof Cc correspondingtothe k largest eigenvaluesof Cc .Thepreciseerrorboundsfortheapproximatesolutionw.r.t.thetrue globalsolutionarederivedin[ 158]. Weusethisnon-localSVDframeworkinadenoisingalgorithmandweshallshow laterthatthisproducesresultscompetitivewiththestartoftheart.Westartoffby dividingthegivennoisyimageintopatches.Foreach`reference'patch,wecollect patchessimilartoitandobtainthecommonbasisforthemusingthenon-localSVD method.However,thisleavesopentheproblemofdecidingonthebestrank k forthe bases,whichneednotbeconstantacrosspatchesofdifferentgeometricstructure. Weobviatetheneedforselectionofthisparameterbyfollowingadifferentapproach. Wecomputethefull-rankorthonormalbases U and V ,i.e.wechoose k = n for n n patches.Nowthegivennoisypatch P isprojectedontothepair (U V ) producingthe matrix S ( P ) = U T PV .Essentially,wecanwritetheentriesof P as P ij = X kl S (P ) kl U ki V lj (7) whichisequivalenttoalinearcombinationofouterproductsoftheform U k V T l (1 k n ,1 l n ).Weconjecturethatthisformulationhasaninterpretationintermsof2D spatialfrequencieswhereinthesmallercoefcientvaluescorrespondtohighervalues ofatleastoneofthefrequencies.Therefore,wechoosetonullifythecoefcientswith smallervalues(asdecidedbyathreshold).Givensucha`ltered'projectionmatrix S (P ) wereconstructthepatch.Thisoperationisrepeatedonoverlappingpatchesinasliding 146

PAGE 147

windo wfashionandtheoverlappinghypothesesareaggregatedbyaveragingleadingto anallteredimage.Crucialtotheperformanceofthislteristhechoiceofanotionof patchsimilarityandalsothechoiceofthresholdsforremovingsmallercoefcients.We discussthesechoicesbelow. 7.7.1ChoiceofPatchSimilarityMeasure Givenareferencepatch P ref inanoisyimage,wecancomputeits K nearest neighborsfromtheimage,butthisrequiresachoiceof K whichmaynotbethesame acrossdifferentimagepatches.Hence,wereverttoadistancethreshold d andselect allpatches P i suchthatthetotalsquareddifferencebetween P ref and P i isbelow d NotethatwehavethroughoutassumedaxedandknownnoisemodelN (0, ).Ifwe weretoassumethat P ref and P i weredifferentnoisyversionsofthesameunderlying patch,weobservethatthefollowingrandomvariablehasa 2 densitywith z = n 2 degreesoffreedom: x = n 2 X k =1 (P ref k )Tj /T1_2 11.955 Tf 11.95 0 Td (P ik ) 2 2 2 (7) The cumulativeofa 2 randomvariablewith z degreesoffreedomisgivenbythe expression F (x ; z )= r ( x 2 z 2 ) (7) where r (x a ) stands fortheincompletegammafunctiondenedasfollows: r ( x a )= 1 \(a ) Z x t =0 e )Tj /T1_3 7.97 Tf 6.59 0 Td (t t a )Tj /T1_7 7.97 Tf 6.59 0 Td (1 dt (7) with \(a ) being theGammafunctiondenedas \(a )= Z 1 0 e )Tj /T1_3 7.97 Tf (t t (a )Tj /T1_7 7.97 Tf (1) dt (7) Weobservethatif z 3 ,forany x 3z ,wehave F (x ; z ) 0.99.Thereforefora patch-sizeof n n andunderthegiven ,wechoosethefollowingthresholdforthe total 147

PAGE 148

squared differencebetweenthepatches: d =6 2 n 2 (7) Thus if twopatchesarenoisyversionsofthesamecleanpatch,thisthresholdwill pickthemwithaveryhighprobability.Buttheconverseisnottrue,andthereforewemay endupcollectingpatchpairsthatsatisfythethresholdbutarequitedifferentstructurally. Toeliminatesuch`falsepositives',weobservethatif P ref and P i arenoisyversionsof thesamepatch,thevaluesin P ref )Tj /T1_5 11.955 Tf 12.09 0 Td (P i belongto N (0, p 2 ). Thismotivatesustousea hypothesistest,inthisparticularcasetheone-sidedKolmogorov-Smirnov(K-S)test.To avoidhavingtochooseaxedsignicancelevel,weusethe p -valuesoutputbytheK-S testsasaweightingfactorinthecomputationofthecorrelationmatrices.Thereforewe rewritethemasfollows: Cr = K X i =1 p KS (P ref P i )P i P T i (7) Cc = K X i =1 p KS (P ref P i )P T i P i (7) with p KS (P ref P i ) beingthe p -valuefortheK-Stesttocheckhowwellthevaluesin P ref )Tj /T1_5 11.955 Tf 12.72 0 Td (P i conformto N (0, p 2 ). Thisthusgivesusarobustversionofthe2D-SVD. ThereisadifferencebetweenourapproachandrobustversionsofPCA,suchasthe L 1 -norm(robust)PCAin[159 ].Wedonotneedtochooseanarbitraryrobustnorm, butuseaweightingfunctiondirectedbyahypothesistestinstead.Thisisakinto computationoffuzzycovariancematricesinfuzzyrobustPCA[ 160]. Inpractice,weobservedthatthethreshold d =6 2 n 2 wastooconservative. Thatis,mostpatches P i whichdifferedfromthereferencepatchbymorethan 3 2 n 2 yieldedp-values p KS (P ref P i ) thatwereveryclosetozero.Henceweusedtheless conservativebound d =3 2 n 2 inourexperiments.Thisalsoledtosomeimprovement incomputationalspeed.Weimplementedavariantofourmethodinwhichonlythe 148

PAGE 149

threshold d w asusedforpatchselection,andthehypothesistestwasentirelyignored. Surprisingly,wedidnotexperienceanysignicantdropinperformanceonourdatasets ifthehypothesistestwasneglected.Nonethelessinallreportedresults,westillused thehypothesistestbecauseitisaprincipledwayofmitigatingtheeffectoffalse positives.AnexampleofthephenomenonoffalsepositivesisillustratedinFigure 7-10.ThetwoimagesinFigure 7-10 arestructurallyverydifferent(containinggraylevels of10and40),andyettheMSEbetweentheirnoisyversions( =20)isonly4075which fallsbelowthethresholdof 3 2 =4800.HowevertheKS-testyieldsap-valueveryclose to0,therebyprovidingabetterindicationofstructuraldissimilarity. Itshouldbefurthernotedthateventhebound d 3 2 n 2 isquiteconservative.It canberenedusingthefactthatthe 2 densitycanbeapproximatedas N (n 2 n p 2) if n 2 is large.Thisresultsfollowsfromthecentrallimittheoremandholdsgoodfor n 2 64. Thisgivesusthefollowingrenedbound: d =( n 2 + p 2 2.362 n )2 2 = 2( n 2 +3.29 n ) 2 (7) fromtheinversecumulativefor N (n 2 p 2n ) at 0.99. 7.7.2ChoiceofThresholdforTruncationofTransformCoefcients Asournoisemodelis N (0, ) ,weobservethatthecorrespondingrandomvariables in n n patcheshavemagnitudelessthan p 2 log n 2 withveryhighprobability, asalsotheentriesofthecorrespondingprojectionmatrices(ontoorthonormal bases/basis-pairs).Henceweassumethatcoefcientslessthanthisthresholdhave beenproducedduetonoise.Thisthresholdhappenstobetheuniversallyoptimal thresholdforwaveletdenoisingwithhardthresholding[ 113](alsoseeSection 6.4),and holdsspecicallyfori.i.d.Gaussiannoiseforanygivenorthonormalbasis.Whilehard thresholdingmayleadtoeliminationofsomeusefulhigh-frequencyinformation,thisloss iscompensatedthroughtheredundancyfromoverlappingpatches[ 108 ]. 149

PAGE 150

7.7.3 OutlineofNL-SVDAlgorithm TheNL-SVDisoutlinedherebelow: 1.Dividetheimageintooverlappingpatches. 2.Foreachpatch P ref (calledas`referencepatch'),ndpatches fP i g fromtheimage thataresimilartoitinthesenseexplainedinSection 7.7.1. 3.Computetheweightedrow-rowandcolumn-columncorrelationmatrices Cr and Cc fP i g asperEquation 7 4.Findtheeigenvectorsof Cr togiveorthonormalmatrix U andthoseof Cc togive orthonormalmatrix V 5.Project P ref onto ( U V ) togive S ref = U T P ref V 6.Setallsmallentriesof S ref tozero,asdiscussedinSection 7.7.2 7.Reconstruct P ref using P ref = US ref V T andaccumulatethepixelvaluestothe appropriatelocationintheimage. 8.Repeatabovestepsforallimagepatches. 9.Aggregateallthehypothesesandaveragethemtoproducethenallteredimage. 7.7.4AveragingofHypotheses Notethattheprocedureforaveragingofhypothesesproducedforapatchis commontocontemporarypatch-basedalgorithmsnotonlyforimagedenoising applications[ 134],[146 ],[ 108](whereitiscalledas`translationinvariantdenoising'), butalsoforseveralotherapplicationssuchastexturesynthesis[ 161].Wehave experimentedwithotheraggregationproceduressuchasndingthemedianofall availablehypothesisvalues,re-lteringofpixelvaluesorlearningweightsforweighted linearcombinations.Whilebeingmorecomputationallyexpensive,noneofthese proceduresimprovedtheperformancebeyondsimpleaveraging. 7.7.5VisualizingtheLearnedBases Wenowpresenttwoexamplesofthebaseslearnedtoshowtheeffectofthe structureofthepatchandtovisualizethecorrespondingbases.Therstexample(in Figure 7-11 )isapatchofsize 8 8 containingorientedtexturefromtheBarbaraimage. 150

PAGE 151

The patchessimilartoit(asmeasuredinthenoisyversionofthatimage)areshown alongside,asalsothelearnedbases.Thebasesthatwevisualizeareactually64outer productsoftheform U i V T j (1 i j 8).Wepresentasecondexamplewhichcontains ahigh-frequencyfurtexturefromthemandrillimage,inFigure 7-12.InFigure 7-13,we showouter-productsof 8 8 DCTbasesforcomparisonwiththoseinFigure 7-11 and Figure 7-12 7.7.6RelationshipwithFourierBases Itisawell-knownresultthattheprincipalcomponentsofnaturalimagepatches (inthiscase,justrowsorcolumnsfromimagepatches)aretheFourierbases[ 107]. Furthermore,thispropertyisaconsequenceofthetranslationinvariancepropertyofthe covariancebetweennaturalimages.Infact,itisprovedin[ 162](seesection5.8.2)that undertheassumptionoftranslationinvariance,theeigenvectorsofthecovariancematrix ofnaturalimagepatchesturnouttobesinusoidalfunctionsofdifferentfrequencies. Theaforementionedfactcanbeexperimentallyobservedbycomputingtheprincipal componentsofalargeensembleofpatchesofxedsize-theresultsareverycloseto DCTbases(therealcomponentsoftheFourierbases).Wecomputedtherow-rowand column-columncovariancematricesof 8 8 patchessampledatevery4pixelsfrom allthe300imagesoftheBerkeleydatabase[ 61 ]convertedtogray-scale(i.e.atotalof 2.88 10 6 patches).TheeigenvectorsofthesematriceswereverysimilartoDCTbases asmeasuredbytheanglesbetweencorrespondingbasisvectors:0.2,4,4,6.8,5.6,6,4 and3degrees. ForNL-SVD,theconsequenceofthisresultisasfollows.Ifforeveryreference patch,thecorrelationmatriceswerecomputedfromseveralpatcheswithoutattention tosimilarity,wecouldgetalterverysimilartotheslidingwindowDCTlter(modulo asymptotics,andbarringthedifferenceduetorobustPCA). 151

PAGE 152

7.8 ExperimentalResults Wenowdescribeourexperimentalresults.Forournoisemodel(i.e.additiveand i.i.d. N (0, )),wepick 2f5,10,15,20,25,30,35 g.Weperformexperimentson Lansel'sbenchmarkdataset[ 62]consistingof13commonlyusedimagesallofsize of 512 512.WepitNL-SVDagainstthefollowing:NL-Means[ 2 ],KSVD[146],our implementationofa3D-DCTalgorithm(seeSection 7.8.5),BM3D[134]andtheoracle denoiserfromSection 7.4.Forcomparisonateachnoiselevel,weusePSNRvalues aswellasSSIMvaluesatpatch-size 11 11 (aspertheimplementationin[63]).(For thedenitionofSSIM,refertoSection 6.8.1.)Allthesemetricsweremeasuredbyrst writingtheimagesintoaleinoneofthestandardimageformats(usually,pgm)and thenreadingthembackintomemory.Thoughthisintroducesminorquantizationeffects (andusuallyreducesthePSNR/SSIMvaluesslightlyforallmethods),wefollowthis approachasitrepresentsrealisticdigitalstorageofimages. InthecaseofBM3DandNL-Means,weusedthesoftwareprovidedbytheauthors online.ForKSVD,weusedtheresultsalreadyreportedbytheauthorsonthedenoising benchmark[ 62].Theseresultswereavailableonlyfornoiselevelsuptoandincluding =25 .ForBM3D,wereportresultsonbothstagesoftheiralgorithm:theintermediate stage,aswellasthenalstagewhichperformsempiricalWienerlteringontheoutput oftheearlierstage.Werefertothesestagesas`BM3D1'and`BM3D2'respectively. Toeachoftheabovealgorithms,thenoise isspeciedasinput(whichisuseful foroptimalparameterselectionintheirprovidedsoftwares).ForNL-SVD,weused 8 8 patchesinallexperimentsandasearchwindowradiusof20aroundeachpoint.The searchwindowradiusisnotafreeparameterasitaffectsonlycomputationalefciency andnotaccuracy.Infactlargersizesofthesearchwindowdidnotimprovetheresults inourexperiments.Therearenootherfreeparametersinourtechnique,apartfromthe patch-sizewhichisalsotrueofallotherpatch-basedalgorithmsintheeld.Later,in Section 7.9,wepresentacriterionforpatch-sizeselectionbymeasuringthecorrelation 152

PAGE 153

coefcient betweenpatchesfromtheresidualimage(i.e.differencebetweennoisyand denoisedimages).ForNL-means,weused 9 9 patchesthroughout,withasearch windowradiusof20.ForBM3Dimplementation,weusedthedefaultsettingsofallthe variousparametersasobtainedfromtheauthors'software(theirselectedpatch-sizeis again 8 8).TheresultsforKSVDhavebeenreportedbytheauthorsthemselves,and henceweassumethattheoptimalparametersettingswerealreadyusedforgenerating thoseresults. 7.8.1DiscussionofResults FromthePSNRresultspresentedinTables 7.12, 7.12, 7-7, 7-9, 7-12, 7-14 and 7-16,andthecorrespondingSSIMresultsinTables 7-4, 7-6, 7-8, 7-10, 7-13, 7-15 and 7-17,wemakeseveralobservations.NL-SVDisconsistentlysuperiortoNL-Meansin termsofPSNRandSSIM.ThesetablesalsocontainresultsoftheHOSVDalgorithm, oursecondtechnique,whichweshallbepresentinglaterinSection 7.10.Inallthe tablesattheendofthechapter,wehaveusednumberstorefertoimagenamestosave space.Thenumbersandthecorrespondingnamesareasfollows:13-airplane,12Barbara,11-boats,10-couple,9-elaine,8-ngerprint,7-goldhill,6-Lena,5-man, 4-mandrill,3-peppers,2-stream,1-Zelda. 7.8.2ComparisonwithKSVD OurPSNRandSSIMvaluesarecomparabletothosereportedforKSVD.However, NL-SVDhasseveralotheradvantagesascomparedtoKSVDfromaconceptualaswell asimplementationpointofview.KSVDlearnsanovercompletedictionaryonthey fromthenoisyimage.Thisprocedurerequiresiteratedoptimizationsandisexpensive. Themethodisalsopronetolocalminimaandthisputsarticiallimitsonthesizeofthe dictionarythatshould(orcan)belearned[ 114].Thealgorithmrequiresparametersthat arenoteasytotune:thenumberofdictionaryvectors( K ),parameterforthestopping criterionforthepursuitprojectionalgorithmandthetradeoffbetweendatadelityand sparsityterms.Ontheotherhand,NL-SVDderivesaspatiallyadaptivebasisateach 153

PAGE 154

pix elinonestepandrequiresnofurtheriterations.Moreover,givenpatchesofsize p p welearnmatrixbasesofsize p p ateachpoint(seeSection 7.12),whereasKSVD learnsonedictionaryofsize p 2 K where K >> p 2 .Thereexistsamultiscaleversion ofKSVD[ 114 ]whichhasproducedimprovementintheperformanceoftheoriginal algorithmfrom[ 146](seeTable3of[ 114]),butwehaven'tincludeditinthecomparisons aswewereunabletoobtainanefcientimplementationforthesame. 7.8.3ComparisonwithBM3D ThecurrentstateofthearttechniqueinimagedenoisingistheBM3Dmethod from[ 134].TheBM3Dalgorithmworksonanensembleofpatchesfromtheimage thataresimilartoeachreferencepatch.Ittreatstheensembleasa3Darray,anda3D transformisappliedtothispatchensembleforthepurposeofltering.Thistreatmentof asequenceof(possiblyoverlapping)patchesasasignalisconceptuallystrange. Thespecicimplementationin[134]adoptsthefollowingsteps.Firstly,thesimilarity betweenapatchfromthenoisyimageandotherpatchesfromthesameimageis measuredusingtheL2distancebetweentheirrespectiveDCTcoefcientsafterrst settingtozeroallcoefcientsbelowathreshold.Inotherwords,thepatchesare pre-ltered(solely)forthepurposeofsimilaritycomputation.Next,theindividualnoisy patchesinthegrouparelteredusinga2D-DCTor2Dbiorthogonalwaveletswithhard thresholding(withthethresholdforcoefcientschosenasaxedmultipleoftheknown noise ).Finally,theindividualpixelstackscreatedfromthelteredpatches(fromthe earlierstep)arefurtherlteredbyusingaHaartransform.Themultiplehypotheses appearingateachpixelareaggregatedtoproducealteredimage.Thisiscalledas the`intermediatestage'oftheBM3Dalgorithm(whichwerefertoas`BM3D1').Thisis followedbyasecondstagewhichfurtherlterstheoutputofBM3D1.Patchesfromthe outputimageofBM3D1thataresimilartoareferencepatchfromthatimageareagain stackedtogether,anda3Dtransformisapplied.Thetransformcoefcientsaremodied usinganempiricalWienerlter.Thetransformisinvertedfollowedbyaggregation 154

PAGE 155

of multiplehypothesestoproducethenallteredimage.Thisnalstageistermed `BM3D2'.Theexactow-chartforallthesestepsisgivenin[ 134]. TheoverallBM3Dalgorithmcontainsanumberofparameters:thechoiceof transformfor2Dand1Dltering(whetherHaar/DCT/Biorthogonalwavelet),the distancethresholdforpatchsimilarity,thethresholdsfortruncationoftransformdomain coefcients,aparametertorestrictthemaximumnumberofpatchesthataresimilar toanyonereferencepatch,andthechoiceofpre-lterwhilecomputingthesimilarity betweenpatchesintherststage(BM3D1).Thereisananalogoussetofparameters forthesecondstagethatusesempiricalWienerltering(BM3D2)overandabovethe resultsfromstage1.Infact,giventhecomplexnatureofthisalgorithm,itmaybedifcult toisolatetherelativecontributionofeachofitscomponents.NotethatNL-SVDtoo requiresthresholdsforpatchsimilarityandtruncationoftransformdomaincoefcients, buttheseareobtainedinaprincipledmannerfromthenoisemodelasexplainedin Section 7.7.1 and 7.7.2.TheBM3Dimplementationin[ 134]usesxedthresholds withanimpreciserelationshiptothenoisevariance.Forinstance,itusesadistance thresholdof2500ifthenoise 40 andathresholdof5000otherwise,atransform domainthresholdof 2.7 ,andapatchsizeof 32 32 andadistancethresholdof400in theWienerlteringstep.UnlikeBM3D,wedonotresorttoanypre-lteringmethodsfor ndingthedistancebetweennoisypatches,butinsteaduseprincipledapproacheslike hypothesistests.Furthermore,NL-SVDtunesthebasesinaspatiallyadaptivemanner insteadofusingxedbases.ItmustalsobementionedthattheWienerlteringstepin BM3D2makestheimplicitassumptionthatthetransformcoefcientsoftheunderlying imageareGaussiandistributed.ItisthisGaussianassumptionalonethatmakesa Wienerlter(oralinearminimummeansquaresestimator)theoptimalleastsquares estimator[ 163].TheGaussianassumptionisgenerallynottrueforDCTorother transformcoefcientsofnaturalimagesorimagepatches.Intermsofempiricalresults, thePSNRvaluesforNL-SVDwerelessthanthoseproducedbyBM3D1(amarginof 155

PAGE 156

0.3 dB)andBM3D2(amarginof0.7dB).However,ouralgorithmhastheadvantageof beingsimpletoimplement,beingconceptuallycleanandhavingparametersthatare obtainedinaprincipledmanner. 7.8.4ComparisonofNon-LocalandLocalConvolutionFilters AsdescribedinSection 6.3,convolutionltersarearichclassofdenoising techniques.Someofthese[ 101],[ 156]makeexplicituseoflocalimagegeometries. Forinstance,theworkin[ 101 ]presentsaninnovativemethodofexploitingrichlocal geometricstructuresforderivingconvolutionlters,andpaysspecialattentionto structuressuchascorners/junctionsinadditiontoedges.TheNL-SVDtechniquein thisthesistakesadifferentpath:itisbasedonlearningspatiallyadaptivebasesthat sparselyrepresentimagepatches.Indeed,NL-SVDdrawsitsprimaryinspirationfrom NL-Meanswhichdiffersinitsfoundationsfromlocalconvolutionltersonatleasttwo counts:(1)itdrawsinformationfromdifferentpartsoftheimagewhichexhibitsome measureofsimilaritytothepixelintensityatthecurrentprocessinglocationandthen (2)usesthisnon-localinformationtomodulatethediffusionatthecurrentpixel.The non-localnatureofNL-Meansisexpectedtogiveitanedgeincomparisontopurely localtechniquesliketheaforementionedconvolutionlters[ 101]. Followingthislineofreasoning,itcomesassomewhatofasurprisethatthepurely localconvolutiontechniquein[ 101]isabletoempiricallyoutperformNL-Meansonthe commonlyused`house'imagewhendegradedbynoisedrawnfrom N (0,20) .Onve noiserealizationsataxed =20,thetechniquefrom[101]producedadenoised imagehavinganaveragePSNRof33.447and33.464(MSE29.402and29.284) respectivelydependingonthetypeofkernelused 1 ,whereasNL-Meansproduced aPSNRofonly32.72(MSE34.760).Thissuggeststhattheinclusionofadditional 1 W egratefullyacknowledgetheeffortsofSileHuincollectingthisresult. 156

PAGE 157

geometr icinformationsuchascorners/junctionsallowspurelylocalconvolutionmethods tocompeteoncertainimageswithnon-localtechniquessuchasNL-Means. 7.8.5Comparisonwith3D-DCT InSection 7.8.3,westatedthatgiventhemultitudeofstepsintheBM3Dalgorithm, itmaybedifculttoisolatetheindividualcontributionofeachstep.Weseektoillustrate thispointbycomparingNL-SVDwithourimplementationofBM3Dinvolvingpurelythe DCTin3D(ontheensembleofnoisypatchesthataresimilartothereferencepatch,the ensemblebeingrepresentedasa3Dstack).Weputanupperlimitof K =30 onthe numberofsimilarpatchesinanensemble,whichissimilartotheBM3Dimplementation in[ 134].Wetermthisvariantasa`3D-DCT'.Thehardthresholdforthe3D-DCT coefcientsis p log n 2 K Ascanbeseeninthetablesattheendofthechapter, NL-SVDconsistentlyoutperforms3D-DCT.Webelievethissufcientlyillustratesthe advantagesofmethodfornon-localbasislearning. 7.8.6ComparisonwithFixedBases Thechoiceof`best'basisoptimizedfordenoisingperformanceisstilllargelyan openissueinsignalprocessing.Asaconsequence,itmaybedifculttocomparethe relativemeritsanddemeritsoflearnedbasesoveruniversalbases.Learnedbaseshave theadvantagethattheyallowfortunabilitytothecharacteristicsoftheunderlyingdata. Inourexperiments,wehaveobservedbetterperformancewithNL-SVDas comparedtoltersusingaslidingwindow2D-DCT.Wepresentafewexamples:the boatsimageandBarbaraimageinFigure 7-22,forwhichweobtainedupto1dBPSNR improvementoverDCT.Wealsopresentanexamplewithalargenumberofrepeating patterns,whichclearlyillustratesthevirtuesofnonlocalbasislearningoverusinga xedbasis.ThisisillustratedwiththecheckerboardimageinFigure 7-23.Comparative guresoverthebenchmarkdatabasearepresentedinTable 7-11. 157

PAGE 158

7.8.7 VisualComparisonoftheDenoisedImages Theoriginalandnoisyimage(from N (0,20) ),andthedenoisedimagesproduced byouralgorithm,NL-Means,BM3D1andBM3D2canbeviewedinFigures 7-14, 7-16, 7-18 and 7-20.Thereaderisurgedtozoomintothe.pdfletoviewtheimagesmore carefully.ThecorrespondingresidualimagescanbeviewedinFigures 7-15, 7-17, 7-19 and 7-21 .Notethattheresidualiscalculatedasthedifferencebetweenthenoisyand denoisedimage,withthedifferenceimagenormalizedbetween0and255.Clearly, NL-Meansproducesresidualswithadiscernibleamountofstructure.Finerstructural detailscanbeobservedintheresidualsproducedbyouralgorithmaswellasthose byBM3D1.BM3D2doesproduceverynoisyresiduals.Iftheimagesarezoomedin, onecanhoweverobservesomestrangeshock-likeartifactsincertainportionsofthe denoisedimagesproducedbyBM3D,especiallybyBM3D2.OneexampleisBarbara's facefromFigure 7-14 -seeFigure 7-25 forazoomed-inview.Theseartifactsare absentinNL-SVD.HoweverBM3Dseemstopreservesomeneredgessomewhat betterthanourtechnique.See,forinstance,theportionofthetableclothlyingonthe tableintheBarbaraimage.Weperformedamoredetailedcomparisonbetweenour outputandtheBM3DoutputontheBarbaraimage.Forthiswecomputedtheabsolute differencebetweenthetrueimageandouroutput,andtheabsolutedifferencebetween thetrueimageandtheoutputofBM3D1/BM3D2.Thesedifferenceimagesareshown inFigure 7-24.Themeanabsoluteerrorvaluesovertheentireimagewere5.36(for NL-SVD),5.28(BM3D1)and4.87(BM3D2).ThemeanL2errorswere53.12,51.34 and44.36respectively.TheerrorsproducedbyNL-SVDweregreaterthanthoseby BM3D1/BM3D2forroughlyonly50percentofthepixels.WealsoranaCannyedge detector(withthedefaultparametersfromtheMATLABimplementation)onthetrue image,andcomputedtheerrorsonlyontheedgepixels.Themeanabsoluteerrors onedgepixelswere6.7,6.7and6.4forNL-SVD,BM3D1andBM3D2respectively, whereasthemeanL2errorsonedgepixelswere77.4,76.7and70.5respectively. 158

PAGE 159

Ho wever,foronlyaround45percentoftheedgepixels,wastheerrorforNL-SVD greaterthanthatforBM3D1/BM3D2. 7.9SelectionofGlobalPatchSize AllresultsinSection 7.8 werereportedforaxedpatch-sizeof 8 8,asthis isacommonlyusedparameterinpatch-basedalgorithms(includingJPEG).Here, wepresentanobjectivecriterionforselectingthepatch-sizethatwillyieldthebest denoisingperformance.Forthis,weconsidertheresidualimagesafterdenoisingwitha xedpatch-size p p ,withthethresholdfordiscardingthesmallercoefcientschosen tobe p 2 log p 2 .Eachresidualimageisdividedintonon-overlappingpatchesofsize q q where q 2f8,9,...,15,16 g .Foreachvalueof q ,wecomputetheaverageabsolute correlationcoefcientbetweenallpairsofpatchesintheresidualimage,andthen calculatethetotaloftheseaveragevalues.Theabsolutecorrelationcoefcientbetween vectors v 1 and v 2 (ofsize q 2 1)isdenedasfollows: pq ( v 1 v 2 )= 1 q 2 j(v 1 )Tj /T1_5 11.955 Tf 11.96 0 Td ( 1 ) T (v 2 )Tj /T1_5 11.955 Tf 11.96 0 Td ( 2 )j v 1 v 2 (7) where 1 and 2 are themeanvaluesofvectors v 1 and v 2 ,and v 1 and v 2 aretheir correspondingstandarddeviations. Ourintuitionisthatanoptimaldenoiserwillproduceresidualpatchesthatarehighly decorrelatedwithoneanotherasmeasuredby pq .However pq iscertainlydependent uponthepatch-size q q thatisusedforcomputationofthestatistics.Hence,wesum upthecross-correlationvaluesover q andoverallpatchpairs,thusgivingus p = X i 2 n,j 2n, q pq (v i v j ) (7) asthenalmeasure.Here v i and v j denotepatches(invectorform)withtheirupperleft corneratlocations i and j (respectively)intheimagedomain n.Thepatch-size p p whichproducestheleastvalueof p isselectedastheoptimalparametervalue.Inour experiments,wevaried p from3to16.WehaveobservedthatthePSNRcorresponding 159

PAGE 160

to theoptimal p isveryclosetotheoptimalPSNR.ThiscanbeseeninTable 7-18 whereforeachimageinthebenchmarkdatabase,wereportthefollowing:(1)the highestPSNRacross p 2f3,4,5,...,15,16 g ,(2)thepatch-sizewhichproducedthat PSNR,(3)thelowest p valueacross p ,(4)thepatch-sizewhichproducedthelowest p valueand(5)thePSNRforthebestpatch-sizeasperthecriterion p .Onecanseefrom Table 7-18 thatthedropinPSNR(ifany)isverylow.Thedenoisedimagesandtheir residualsfordifferentpatch-sizesarealsoshownalongsideinFigures 7-26 and 7-27. Thenoise-levelforalltheseresultsis =20. Ideally,theremaynotbeasingleoptimalpatch-sizefortheentireimage.Abetter approachwouldbetoadaptthepatch-sizebasedonthelocalstructureoftheimage. However,giventheaggregationofhypothesesfrom(andconsequentdependenceon) neighboringpatches,thisturnsouttobeanon-trivialproblem. 7.10DenoisingwithHigherOrderSingularValueDecomposition Wenowpresentasecondalgorithmforimagedenoising,whichisalsorootedin thenon-localbasislearningframework.Themaindifferenceisthatthisalgorithmnow groupstogethersimilarpatchesasa3Dstackandlterstheentirestackusinga3D transform-namely,thehigherordersingularvaluedecomposition(HOSVD)ofthestack. Thecoreideaofgroupingtogethersimilarpatchesandapplying3Dtransformsistaken fromtheBM3DalgorithmwhichwasdescribedinSection 6.5 andingreaterdetailin Section 7.8.3.Themaindifferenceisthatweincorporatethisnotioninabasislearning strategyunlikeBM3D. 7.10.1TheoryoftheHOSVD Thehigherordersingularvaluedecomposition(HOSVD)istheextensionofthe SVDof(2D)matricestohigher-ordermatrices(oftencalledtensors).TheHOSVDwas rstproposedinthepsychologyliteraturebyTuckerforthecaseof3Dmatriceswhereit wascalledtheTucker3decomposition[ 164 ].Averyextensivedevelopmentofthetheory 160

PAGE 161

of HOSVDformatricesofallordersispresentedinthethesisofLathauwer[ 165],from wherethefollowingbriefdescriptionissummarized. Givenahigherordermatrix A 2 R N 1 N 2 ...N D ,theHOSVDdecomposesitinthe followingmanner A = S 1 U (1) 2 U (2) 3 ... D U (D ) (7) where U (1) 2 R N 1 N 1 U (2) 2 R N 2 N 2 ,...,U (D ) 2 R N D N D areallorthonormalmatrices,and S 2 R N 1 N 2 ... N D isahigherordermatrixthatsatisessomespecialproperties.Here, thesymbol n standsforthe n th modetensorproductdenedin[165].Fixingthe n th indexto ,letthesubtensorof S bedenotedas S n .Then S satises < S n S n ,f >=0 8 f n where 6= f .Thisiscalledastheall-orthogonalityproperty.Furthermore,we alsohave kS n ,1 kkS n ,2 k ... kS n ,N n k forall n Letusvisualize A asahypercubewhoseedgesarecoincidentwiththeCartesian axes.The n th unfoldingof A canbevisualizedasthetensorobtainedbyslicing A paralleltotheplanespannedbytheCartesianaxesoftherstand n th dimensionsand thenarrangingtheslicesinsuccessiontoyielda2Dmatrix.Inpractice,theHOSVDcan becomputedfromtheSVDofsuitableunfoldingsofthehigher-ordermatrix A.Itturns outthatEquation 7 hasthefollowingequivalentrepresentationintermsoftensor unfoldings[ 165]: A (n ) = U (n ) S (n ) (U (n +1) n U (n +2) n ... n U (D ) n U (1) n U (2) ... n U (n )Tj /T1_4 7.97 Tf 6.59 0 Td (1) ) T (7) Forathoroughintroductiontomulti-linearalgebraandtheHOSVD,wereferthereader to[ 165].AninterestingapplicationoftheHOSVDtofacerecognitionispresentedin [166]. 7.10.2ApplicationofHOSVDforDenoising WenowdescribehowtheHOSVDisappliedforjointdenoisingofmultipleimage patches.Foreachreferencepatchinthenoisyimage,allpatchessimilartoitare collectedandrepresentedasa3Darray Z2 R p p K ,wherethepatcheshavesize p p 161

PAGE 162

and K is thenumberofsimilarpatchesintheensemble(notethat K isspatiallyvarying). Apatch P issaidtobesimilartothereferencepatchif kP )Tj /T1_1 11.955 Tf 12.65 0 Td (P ref k 2 2 d where d is denedearlierinSection 7.7.1.TheHOSVDof Z isgivenasfollows Z = S 1 U (1) 2 U (2) 3 U (3) (7) wheretheorthonormalmatrices U (1) 2 R p p U (2) 2 R p p and U (3) 2 R K K canbe computedfromtheSVDoftheunfoldings Z (1) Z (2) and Z (3) respectively.Theexact equationsareasfollows: Z (1) = U (1) S (1) ( U (2) n U (3) ) T (7) Z (2) = U (2) S (2) ( U (3) n U (1) ) T (7) Z (3) = U (3) S (3) (U (1) n U (2) ) T (7) However,thecomplexityoftheSVDcomputationsfor K K matricesis O (K 3 ) Topreventthecomputationsfromgettingunwieldy,weputanuppercaponthenumber ofallowedsimilarpatches,i.e.weimposetheconstraintthat K 30.Thepatches from Z arethenprojectedontotheHOSVDtransform.Theparameterforthresholding thetransformcoefcientsispickedtobe p 2 log p 2 K ,againaspertherulefrom[113 ]. Thestack Z isthenreconstructedafterinvertingthetransformtherebylteringall theindividualpatches.NotethatunlikeNL-SVD(seeSection 7.7.3 ),welter all the individualpatchesintheensembleandnotjustthereferencepatch.Thisafforded additionalsmoothingonallthepatcheswhichwasrequiredduetotheupperlimitof K 30 unlikethecasewithNL-SVD.Again,thereferencepatchismovedinasliding windowfashionandthehypothesesappearingateachpixelareaveragedtoproduce thenallteredimage. 7.10.3OutlineofHOSVDAlgorithm TheHOSVDfordenoisingisoutlinedherebelow: 1.Dividetheimageintooverlappingpatchesofsize p p 162

PAGE 163

2.F oreachpatch P ref (calledas`referencepatch'),ndpatches fP i g fromtheimage thataresimilartoitinthesenseexplainedinSection 7.7.1. 3.Stackthesimilarpatchesina3Darray Z2 R p p K 4.Computetheunfoldings Z (1) Z (2) and Z (3) andthencomputetheirSVDtoyieldthe matrices U (1) U (2) and U (3) respectively. 5.Computeanyoneunfoldingofthetensor S ,say S (1) 6.Settozeroallentriesof S (1) thataresmaller(inabsolutevalue)than p 2 log p 2 K 7.ReconstructtheentirestackusingEquation 7 whichlterseverypatchinthe ensemble. 8.Repeatabovestepsforallimagepatches. 9.Aggregateallthehypothesesandaveragethemtoproducethenallteredimage. WewouldliketoemphasizethattherearetwokeydifferencesbetweenourHOSVD algorithmandBM3D.Firstly,welearnaspatiallyvaryingbasiswhereasBM3Duses universalbases(2D-DCTorbiorthogonalwaveletsdependinguponthenoiselevel, followedbyaHaarbasisinthethirddimension).Secondly,asBM3Dstackstogether similarpatchesandperformsaHaartransforminthethirddimension,itthusimplicitly treatsthepatchesasasignalinthethirddimension.Ontheotherhand,ourHOSVD methoddoesnotimposeanysuch`signalness'inthethirddimension.Infact,scrambling theorderofthepatchesinthethirddimensionwillproducethesamevaluesofthe projectioncoefcients,exceptforcorrespondingpermutationoperations.Indeed,both HOSVDandNL-SVDdonottreatthepatchesassignalsinanydimensionunlikebases suchastheDCT.TheSVDofapatchisitselfinvarianttorowandcolumnpermutations. Howeverthisisnotaproblem,becauseitisunlikelytoencounterpatchesfromreal imagesthatarerow/columnpermutationsofoneanother.Ontheotherhand,the orderingofpatchesinthethirddimension(thechoiceofwhichisafreeparameter)may potentiallyaltertheoutputofadenoisingalgorithmsuchasBM3D,whereasourmethod willstillremaininvarianttothischange. 163

PAGE 164

7.11 ExperimentalResultswithHOSVD ThePSNRresultsforHOSVDarepresentedinTables 7.12, 7.12, 7-7, 7-9, 7-12, 7-14 and 7-16.ThecorrespondingSSIMresultscanbefoundinTables 7-4, 7-6, 7-8, 7-10, 7-13, 7-15 and 7-17.Fromthesetables,itcanbeobservedthatHOSVD issuperiortoKSVD,NL-Means,3D-DCTandNL-SVD.Indeed,itisalsosuperiorto BM3D1athighernoiselevels( 20)onmostimagesintermsofPSNR/SSIMvalues, thoughitlagsslightlybehindBM3D2.TheaveragedifferencebetweenthePSNRvalues producedbyHOSVDandBM3D2atnoiselevels10,20and30is0.346,0.281and 0.343respectively(seeTables 7.12, 7-9 and 7-14). AcomparisonbetweenNL-SVDandHOSVDrevealsthatthelatteroutperforms theformerontheweakerorneredgesortextures.Wehaveobservedthattheimages denoisedbyHOSVDsometimestendtohaveafaintgrainyappearance.Thereason forthisisthatHOSVDsmoothesanensembleofpatchesbyprojectionontoacommon basisfollowedbytruncationoftransformcoefcients.Wehaveobservedexperimentally thatthistendstoslightlyunder-smooththepatches,whencomparedtopatchesthat aresmoothed individually asintechniqueslikeNL-SVD.Theundermsoothingis compensatedforbytheaveragingoperationsandthelteringofallpatchesfromthe stack.Thefaintgrainyappearancecanbemitigatedbyrunningalinearsmoothing lter,suchasaPCAinthethirddimensionfrompatchstacksfromthelteredoutput ofHOSVD(similartotheWienerlterideaimplementedinBM3D2butwithalearning componentinvolvingPCAonthestackofcorrespondingpixelsfromsimilarpatches), whichseemstoimprovesubjectivevisualquality(inouropinion).Weleavearigorous testingofthisissueforfuturework. 7.12ComparisonofTimeComplexity Wenowpresentatimecomplexityanalysisofallthecompetingalgorithms.For this,assumethatthenumberofimagepixelsis N andtheaveragetimetocompute similarpatchesperreferencepatchis T S .Letusassumethattheaveragenumberof 164

PAGE 165

patches similartothereferencepatchis K .Letthesizeofthepatchbe n n .Thetime complexityofNL-SVDisthen O ([ T S + Kn 3 ]N ) becausetheeigendecompositionofa n n matrixis O ( n 3 ) andmultiplicationoftwo n n matricesisalsoan O (n 3 ) operation. TheBM3Dimplementationin[ 134]requires O (Kn 3 ) timeforthe2Dtransformsand O (K 2 n 2 ) timeforthe1Dtransforms,ifthetransformsareimplementedusingsimple matrixmultiplication.Thisleadstoatotalcomplexityof O ([ T S + Kn 3 + K 2 n 2 ]N ). IfalgorithmssuchasthefastFouriertransformsareused,thiscomplexityreduces to O ([ T S + Kn 2 log n + n 2 K log K ]N ).Ifweassumethat n is o (K ) (i.e.thatthe averagenumberof`similar'patchesismuchgreaterthanthepatchwidth/height,a veryreasonableassumptiontomake),thenNL-SVDisinfactbetterintermsoftime complexitythanBM3D.ThecomplexityofHOSVDisobtainedasfollows.Givena patchstackofsize n n K ,thesizeoftwoofitsunfoldingsis n nK ,theSVD ofwhichconsumes O (Kn 3 ) time.Thethirdunfoldinghassize K n 2 ,theSVDof whichconsumes O (min( K 2 n 2 Kn 4 )) time.Hencethetotalcomplexityofthemethodis O ([ T S + Kn 3 + min(K 2 n 2 Kn 4 )] N ). NoteagainthatNL-SVDandHOSVDfollowtheconceptofmatrixbasedpatch representations,intunewiththephilosophyfollowedby[ 155],[ 152],[ 158],[ 167]and [168].Wecouldhaverepresentedeach n n patchasa n 2 1 vectorandbuilta covariancematrixofsize n 2 n 2 toproducethespatiallyadaptivebases.Infact, suchanapproachwastakenin[ 169].Howeverthecomplexityofsuchamethodis O ([ T S + Kn 4 + n 6 ]N ) whichisgreaterthanours.TheKSVDtechniquealsofollows similarvector-basedpatchrepresentationandthe K learnedbaseshavesize n 2 1 (with K >> n 2 ).AnimportantpointtobementionedisthattheSVDisacharacteristic ofamatrix/patch.ThereisnoanalogtotheSVDforthevectorialrepresentationofthe patch. 165

PAGE 166

A B C D E F Figure 7-1.GlobalSVDFilteringontheBarbaraimage:(A)cleanBarbaraimage,(B) noisyimagewithGaussiannoiseof =20 (PSNR=22.11),(C)ltered imagewithrank1truncation(PSNR=14.7),(D)lteredimagewithrank10 truncation(PSNR=20.17),(E)lteredimagewithrank100truncation (PSNR=24.3),(F)lteredimagewithrank200truncation(PSNR=23.03) 166

PAGE 167

A B C D E F Figure 7-2.Patch-basedSVDlteringontheBarbaraimage,(A)cleanBarbaraimage, (B)noisyimagewithGaussiannoiseof =20 (PSNR=22.11),(C)ltered imagewithrank1truncationineachpatch(PSNR=23.9),(D)lteredimage withrank2truncationineachpatch(PSNR=25.05),(E)lteredimagewith nullicationofsingularvaluesbelow 3 ineachpatch(PSNR=23.42),(F) lteredimagewithtruncationofsingularvaluesineachpatchsoastomatch noisevariance(PSNR=25.8) 167

PAGE 168

A B C D E Figure 7-3.OraclelterwithSVD,(A)cleanBarbaraimage,(B)noisyimagewith Gaussiannoiseof =20 (PSNR=22.11),(C)lteredimagewith nullicationofallthevaluesintheprojectionmatrixbelow 3 ineachpatch (PSNR=36.9),(D)noisyimagewithGaussiannoiseof =40 (PSNR= 22.11),(E)lteredimagewithnullicationofallthevaluesintheprojection matrixbelow 3 ineachpatch(PSNR=31.34) Figure 7-4.Fifteensyntheticpatches 168

PAGE 169

A B Figure 7-5.ThresholdfunctionsforDCTcoefcientsof(A)thesixthand(B)theseventh patchfromFigure 7-4 169

PAGE 170

A B C D E F Figure 7-6.DCTlteringwithMAPandMMSEmethods,(A)cleanBarbaraimage,(B) noisyimagewithGaussiannoiseof =20 (PSNR=22.11),(C)ltered imagewithMMSEestimatoronnon-overlapping 8 8 patches(PSNR= 26.26),(D)lteredimagewithMAPestimatoronnon-overlapping 8 8 patches(PSNR=26.19),(E)lteredimagewithMMSEestimatoron overlapping 8 8 patches(PSNR=28.03),(F)lteredimagewithMAP estimatoronoverlapping 8 8 patches(PSNR=29.94) 170

PAGE 171

A B C D E F Figure 7-7.DCTlteringwithMAPandMMSEmethods,(A)cleanBarbaraimage,(B) noisyimagewithGaussiannoiseof =20 (PSNR=22.11),(C)ltered imagewithMMSEestimatoronnon-overlapping 8 8 patches(PSNR= 27.12),(D)lteredimagewithMAPestimatoronnon-overlapping 8 8 patches(PSNR=26.9),(E)lteredimagewithMMSEestimatoron overlapping 8 8 patches(PSNR=29.1),(F)lteredimagewithMAP estimatoronoverlapping 8 8 patches(PSNR=29.94) 171

PAGE 172

A B Figure 7-8.Thresholdfunctionsforcoefcientsof(A)thesixthand(B)theseventhpatch fromFigure 7-4 whenprojectedontoSVDbasesofpatchesfromthe database 172

PAGE 173

A B C D E Figure 7-9.SVDlteringwithMAPandMMSEmethods,(A)cleanBarbaraimage,(B) noisyimagewithGaussiannoiseof =20 (PSNR=22.11),(C)ltered imagewithMMSEestimatorwithSVDbasesofpatchesfromthedatabase, onoverlapping 8 8 patches(PSNR=25.2),(D)lteredimagewithMAP estimatorwithSVDbasesofpatchesfromthedatabase,onoverlapping 8 8 patches(PSNR=28.85),(E)lteredimagewithMAPestimatorwith SVDbasesofthetruepatches,onoverlapping 8 8 patches(PSNR=36.6) A B C D Figure 7-10.MotivationforrobustPCA:thoughthepatchesarestructurallydifferent,the differencebetweenthetwonoisypatchesfallsbelowthethresholdof 3 2 n 2 173

PAGE 174

A B C Figure 7-11.Barbaraimage,(A)referencepatch,(B)patchessimilartothereference patch(similaritymeasuredonnoisyimagewhichisnotshownhere),(C) correlationmatrices(toprow)andlearnedbases 174

PAGE 175

A B C Figure 7-12.Mandrillimage,(A)referencepatch,(B)patchessimilartothereference patch(similaritymeasuredonnoisyimagewhichisnotshownhere),(C) correlationmatrices(toprow)andlearnedbases 175

PAGE 176

Figure 7-13.DCTbases(8 8). 176

PAGE 177

A B C D E F G Figure 7-14.Barbaraimage:(A)cleanimage,(B)noisyversionwith =20,PSNR=22, (C)outputofNL-SVD,(D)outputofNL-Means,(E)outputofBM3D1,(F) outputofBM3D2,(G)outputofHOSVD 177

PAGE 178

A B C D E Figure 7-15.Residualwith(A)NL-SVD,(B)NL-Means,(C)BM3D1,(D)BM3D2,(E) HOSVD 178

PAGE 179

A B C D E F G Figure 7-16.Boatimage:(A)cleanimage,(B)noisyversionwith =20,PSNR=22, (C)outputofNL-SVD,(D)outputofNL-Means,(E)outputofBM3D1,(F) outputofBM3D2,(G)outputofHOSVD 179

PAGE 180

A B C D E Figure 7-17.Residualwith(A)NL-SVD,(B)NL-Means,(C)BM3D1,(D)BM3D2,(E) HOSVD 180

PAGE 181

A B C D E F G Figure 7-18.Streamimage:(A)cleanimage,(B)noisyversionwith =20,PSNR=22, (C)outputofNL-SVD,(D)outputofNL-Means,(E)outputofBM3D1,(F) outputofBM3D2,(G)outputofHOSVD 181

PAGE 182

A B C D E Figure 7-19.Residualwith(A)NL-SVD,(B)NL-Means,(C)BM3D1,(D)BM3D2,(E) HOSVD 182

PAGE 183

A B C D E F G Figure 7-20.Fingerprintimage:(A)cleanimage,(B)noisyversionwith =20,PSNR= 22,(C)outputofNL-SVD,(D)outputofNL-Means,(E)outputofBM3D1, (F)outputofBM3D2,(G)outputofHOSVD 183

PAGE 184

A B C D E Figure 7-21.Residualwith(A)NL-SVD,(B)NL-Means,(C)BM3D1,(D)BM3D2,(E) HOSVD 184

PAGE 185

A B C D Figure 7-22.For =20,denoisedBarbaraimagewithNL-SVD(A)[PSNR=30.96]and DCT(C)[PSNR=29.92].Forthesamenoiselevel,denoisedboatimage withNL-SVD(B)[PSNR=30.24]andDCT(D)[PSNR=29.95]. 185

PAGE 186

A B C D Figure 7-23.(A)Checkerboardimage,(B)Noisyversionoftheimagewith =20,(C) DenoisedwithNL-SVD(PSNR=34)and(D)DCT(PSNR=27).Zoomin forbetterview. 186

PAGE 187

A B C Figure 7-24.AbsolutedifferencebetweentrueBarbaraimageanddenoisedimage producedby(A)NL-SVD,(B)BM3D1,(C)BM3D2.Allthreealgorithms wererunonimagewithnoise =20. A B C Figure 7-25.AzoomedviewofBarbara'sfacefor(A)theoriginalimage,(B)NL-SVD and(C)BM3D2.NotetheshockartifactsonBarbara'sfaceproducedby BM3D2. 187

PAGE 188

A B C D E F G Figure 7-26.ReconstructedimageswhenBarbara(withnoise =20)isdenoisedwith NL-SVDrunonpatchsizes(A) 4 4,(B) 6 6,(C) 8 8,(D) 10 10,(E) 12 12,(F) 14 14 and(G) 16 16 188

PAGE 189

A B C D E F G Figure 7-27.ResidualimageswhenBarbara(withnoise =20)isdenoisedwith NL-SVDrunonpatchsizes(A) 4 4,(B) 6 6,(C) 8 8,(D) 10 10,(E) 12 12,(F) 14 14 and(G) 16 16 189

PAGE 190

T able7-1.Avg,maxandmedianerroronsyntheticpatchesfromFigure 7-4 withMAP andMMSEestimatorsforDCTbases P atchMAPMAPMAPMMSEMMSEMMSE (avg)(max)(med.)(avg)(max)(med.) 1 16.38172.453.453.8028.443.16 281.63234.0674.9044.19196.1833.88 377.54268.9267.7443.20163.4934.33 457.23232.2149.253.7226.773.11 560.63221.4851.173.9460.673.19 6799.241192.52795.08277.22468.22269.79 737.46207.2224.613.8541.913.19 860.05272.3265.8933.3998.5028.70 962.18220.5366.9034.27103.8828.23 1063.17200.4551.2029.6255.4029.61 1139.59172.3531.143.8452.333.15 1242.01250.6832.067.21313.863.26 13430.12815.37422.66223.45421.27220.23 14425.47768.06416.78221.48409.32214.90 15900.201494.50890.41318.90599.51314.54 T able7-2.Avg,maxandmedianerroronsyntheticpatchesfromFigure 7-4 withMAP andMMSEestimatorsforSVDbasisofthecleansyntheticpatch P atchMAPMAPMAPMMSEMMSEMMSE (avg)(max)(med.)(avg)(max)(med.) 1 17.04200.254.393.7634.983.10 218.19157.714.143.8324.203.17 317.13159.674.603.7742.493.07 461.40234.5751.093.7715.053.23 561.63303.5553.953.7727.483.07 624.43229.0611.823.8837.623.21 737.41182.0726.013.7118.873.10 817.05153.974.514.0941.733.13 920.75185.965.373.9819.893.19 1041.95200.2920.7923.49120.359.22 1116.43171.913.883.7316.823.19 1216.88200.244.044.0150.553.14 1352.93177.5844.3523.66133.7416.56 1456.33291.7144.3023.44141.3117.12 1539.89162.0131.0614.5294.975.34 190

PAGE 191

T able7-3.PSNRvaluesfornoiselevel =5 onthebenchmarkdataset Image #NL-SVDNL-MeansKSVDHOSVD3DDCTBM3D1BM3D2Oracle 13 38.33938.26839.13138.60938.33938.98139.14645.347 1237.43437.11938.04437.69337.43637.96338.14345.023 1136.57836.72137.17936.59436.26336.93937.14144.183 1036.62336.83337.26036.82536.59737.26037.37944.203 935.62336.37737.28335.85835.64336.26136.64144.398 835.91335.30036.62436.10936.10836.22736.41045.964 736.31436.75137.05536.38736.16836.86137.07644.341 637.85037.90338.55437.99737.66238.45938.53444.162 536.44936.67837.00836.67636.36537.02537.18745.078 434.69934.96835.18134.73934.66035.00735.14845.412 336.92137.45537.70737.00736.60237.47837.54043.672 235.00435.32535.54435.16135.11435.52935.64345.534 138.64638.43439.33638.77038.39039.12239.22443.876 T able7-4.SSIMvaluesfornoiselevel =5 onthebenchmarkdataset Image #NL-SVDNL-MeansKSVDHOSVD3DDCTBM3D1BM3D2Oracle 13 0.9560.9490.9580.9580.9530.9580.9590.986 120.9610.9570.9630.9630.9580.9630.9640.989 110.9360.9380.9400.9330.9220.9350.9380.986 100.9460.9460.9490.9460.9400.9480.9500.988 90.9000.9160.9320.9030.8940.9080.9180.987 80.9860.9840.9880.9870.9870.9870.9870.999 70.9370.9400.9430.9360.9300.9390.9430.988 60.9410.9390.9450.9410.9340.9430.9430.981 50.9470.9440.9480.9490.9410.9490.9510.989 40.9530.9580.9580.9520.9490.9550.9580.995 30.9200.9280.9280.9180.9060.9220.9220.978 20.9590.9610.9620.9600.9590.9620.9640.995 10.9410.9360.9440.9420.9350.9420.9430.978 191

PAGE 192

T able7-5.PSNRvaluesfornoiselevel =10 onthebenchmarkdataset Image #NL-SVDNL-MeansKSVDHOSVD3DDCTBM3D1BM3D2Oracle 13 35.13734.21335.66435.14434.90535.54435.86741.725 1234.03233.04434.38634.45934.05034.53634.88241.177 1133.32032.74333.62333.39232.84733.63533.85539.183 1033.23532.67433.49333.37733.08233.78133.99339.882 933.00332.76433.94233.32032.54833.28733.30437.903 831.63131.46432.38631.61431.93832.13132.42740.652 733.00932.74833.39833.06632.45133.37933.61339.371 635.16633.96535.46035.33635.01535.57635.82540.577 532.51432.33932.83532.47432.13232.95033.20840.131 429.98930.26230.48629.48429.88630.35330.53439.226 334.52133.78134.80734.72834.29134.91335.00339.316 230.38030.76030.93129.80730.26230.83131.09939.530 136.24234.39936.54236.44736.06236.52736.80840.864 T able7-6.SSIMvaluesfornoiselevel =10 onthebenchmarkdataset Image #NL-SVDNL-MeansKSVDHOSVD3DDCTBM3D1BM3D2Oracle 13 0.9280.8870.9310.9270.9250.9290.9350.975 120.9340.9030.9340.9380.9330.9370.9420.978 110.8790.8660.8830.8860.8620.8840.8880.956 100.8950.8780.8980.9030.8860.9040.9080.969 90.8180.8220.8530.8320.7890.8220.8190.935 80.9630.9610.9680.9610.9640.9660.9690.995 70.8730.8620.8790.8780.8500.8790.8850.964 60.9080.8700.9090.9120.9040.9110.9150.963 50.8860.8690.8840.8900.8700.8890.8950.971 40.8850.8960.8960.8760.8680.8910.8970.977 30.8790.8580.8820.8850.8720.8830.8820.946 20.8920.8970.9020.8770.8770.8980.9060.983 10.9110.8620.9130.9140.9090.9130.9160.961 192

PAGE 193

T able7-7.PSNRvaluesfornoiselevel =15 onthebenchmarkdataset Image #NL-SVDNL-MeansKSVDHOSVD3DDCTBM3D1BM3D2Oracle 13 33.26232.22333.59733.34132.82533.50633.94939.502 1232.28331.36332.37532.83632.00832.58733.05738.766 1131.45430.56631.70631.70430.89931.71032.03936.910 1031.32030.34231.39431.57630.96331.72532.04937.637 931.85431.35532.27132.20131.60332.09532.13235.364 829.53729.15930.05129.79829.22929.90430.26237.562 731.29630.61731.50831.65130.70331.60431.86536.835 633.48732.16633.71233.68833.22233.73734.13338.506 530.38829.83530.48230.56829.75930.70030.97337.013 427.55727.46127.96927.10127.30527.88128.16635.793 333.13832.10733.19933.32632.85233.37533.59437.534 228.12928.08028.56427.76127.61928.40528.71835.905 134.62832.81734.74834.84834.37534.77135.27039.248 T able7-8.SSIMvaluesfornoiselevel =15 onthebenchmarkdataset Image #NL-SVDNL-MeansKSVDHOSVD3DDCTBM3D1BM3D2Oracle 13 0.9050.8550.9100.8980.9020.9010.9160.967 120.9100.8760.9090.9160.9090.9130.9230.969 110.8360.8100.8410.8490.8220.8450.8530.935 100.8530.8210.8520.8660.8440.8640.8740.955 90.7710.7730.7890.7910.7580.7820.7760.886 80.9400.9340.9460.9430.9310.9440.9490.990 70.8200.7990.8240.8380.7990.8320.8410.942 60.8810.8370.8840.8830.8800.8830.8930.949 50.8280.7990.8230.8380.8040.8340.8420.950 40.8230.8260.8350.8140.7930.8310.8420.956 30.8550.8220.8550.8570.8520.8570.8610.929 20.8250.8180.8370.8140.7850.8310.8450.964 10.8870.8360.8880.8880.8880.8870.8970.952 193

PAGE 194

T able7-9.PSNRvaluesfornoiselevel =20 onthebenchmarkdataset Image #NL-SVDNL-MeansKSVDHOSVD3DDCTBM3D1BM3D2Oracle 13 31.93630.54032.26632.01531.43332.02832.55237.695 1230.87829.42130.76231.53330.54331.02631.66036.603 1130.18728.91130.36030.49129.59630.39530.80235.510 1029.96128.38729.92930.29929.42230.25230.69836.066 931.13529.92431.34131.35430.88731.28431.43334.178 828.05327.42428.45428.56327.38928.40328.79435.318 730.09828.93130.16630.53629.53230.39730.72635.241 632.24030.47332.37132.41131.90332.37532.95036.975 528.93927.99528.85329.29128.25029.20029.46434.989 425.97625.93326.37225.72025.54326.26026.58233.434 332.00930.35732.00532.16631.74032.13832.49836.260 226.80026.37527.06226.72225.89226.91827.19233.485 133.40130.90233.49433.52533.16933.43034.07537.971 T able7-10.SSIMvaluesfornoiselevel =20 onthebenchmarkdataset Image #NL-SVDNL-MeansKSVDHOSVD3DDCTBM3D1BM3D2Oracle 13 0.8850.8020.8930.8690.8850.8750.8990.959 120.8820.8210.8770.8970.8840.8840.9030.956 110.8010.7530.8030.8140.7890.8090.8240.922 100.8160.7550.8120.8310.8060.8280.8450.945 90.7470.7230.7550.7610.7400.7550.7540.859 80.9140.9030.9220.9260.8990.9220.9300.984 70.7780.7360.7760.8000.7610.7930.8070.924 60.8580.7820.8610.8520.8610.8550.8750.938 50.7750.7290.7680.7920.7530.7840.7960.930 40.7650.7600.7800.7640.7220.7760.7920.936 30.8350.7690.8350.8300.8360.8310.8430.918 20.7640.7460.7730.7670.7000.7710.7860.942 10.8670.7770.8690.8590.8710.8620.8800.944 194

PAGE 195

T able7-11.PSNRvalues:NL-SVDversusDCTfornoiselevel =20 onthebenchmark dataset Image #NL-SVDDCT chec kerboard34.527.2 1331.9331.76 1230.8829.93 1130.2029.95 1029.9629.73 931.1331.06 828.0528.08 730.1029.90 632.2432.09 529.9428.57 425.9825.82 332.0031.67 226.8026.49 133.4033.48 T able7-12.PSNRvaluesfornoiselevel =25 onthebenchmarkdataset Image #NL-SVDNL-MeansKSVDHOSVD3DDCTBM3D1BM3D2Oracle 13 30.84528.83531.14531.03830.32430.95531.51236.212 1229.76628.13529.55230.43929.36729.87030.59535.030 1129.20727.49529.18829.48228.47629.30629.78234.292 1028.73626.91628.68929.29728.21529.12029.63934.794 930.48428.90630.62930.69830.28730.54030.88033.389 826.83425.76427.22527.42226.21827.26227.71933.669 729.09427.60829.15229.58828.69029.43329.83434.074 631.32929.20231.27931.47030.85231.29431.95735.678 527.76326.36727.61728.20027.13628.06128.34033.344 424.92924.48825.20425.01924.18625.09725.43531.635 330.99328.80230.80131.34930.78531.17031.62635.239 225.79224.89425.90025.82324.74125.87626.10631.623 132.45629.58432.36732.68532.12732.34333.12036.856 195

PAGE 196

T able7-13.SSIMvaluesfornoiselevel =25 onthebenchmarkdataset Image #NL-SVDNL-MeansKSVDHOSVD3DDCTBM3D1BM3D2Oracle 13 0.8680.7600.8760.8540.8700.8520.8850.951 120.8500.7800.8490.8700.8600.8570.8840.943 110.7710.7000.7690.7840.7580.7770.7990.909 100.7770.6970.7730.8000.7690.7920.8160.935 90.7290.6870.7330.7370.7270.7320.7390.841 80.8860.8640.8960.9060.8740.9020.9130.977 70.7400.6810.7390.7660.7320.7590.7780.910 60.8400.7380.8420.8340.8420.8280.8580.928 50.7300.6640.7230.7490.7110.7410.7560.911 40.7130.6900.7250.7200.6520.7230.7450.915 30.8150.7250.8150.8140.8210.8080.8280.909 20.7070.6670.7130.7150.6360.7190.7350.919 10.8490.7350.8500.8440.8540.8360.8640.936 196

PAGE 197

T able7-14.PSNRvaluesfornoiselevel =30 onthebenchmarkdataset Image #NL-SVDNL-MeansHOSVD3DDCTBM3D1BM3D2Oracle 13 29.87527.68030.07929.43730.10130.71135.098 1228.63926.85329.46228.49828.95229.79333.696 1128.30526.36828.65027.65628.46629.01733.380 1027.74025.66528.29027.26828.15428.75933.635 929.99727.86529.97629.77029.94730.42032.770 825.86324.55226.67625.43426.38226.87432.251 728.35726.57628.79827.99328.65429.14533.160 630.23328.08030.41130.00030.41731.19434.597 526.77825.17627.27826.24827.10927.35332.041 424.13923.39624.29323.09424.20824.55130.233 329.99627.43830.15029.82430.16430.67334.259 224.88423.85825.27824.04125.11525.33630.289 131.54928.38831.38531.16631.26732.13035.861 T able7-15.SSIMvaluesfornoiselevel =30 onthebenchmarkdataset Image #NL-SVDNL-MeansHOSVD3DDCTBM3D1BM3D2Oracle 13 0.8530.7170.8150.8560.8320.8730.944 120.8240.7300.8360.8400.8330.8680.930 110.7460.6500.7520.7360.7500.7790.901 100.7420.6390.7600.7370.7570.7900.925 90.7150.6440.7100.7160.7120.7270.829 80.8600.8260.8900.8510.8810.8950.968 70.7130.6310.7330.7070.7280.7530.898 60.8150.6880.7910.8250.8030.8430.918 50.6900.6100.7100.6770.7040.7220.891 40.6640.6240.6860.5870.6770.7010.896 30.7960.6760.7710.8040.7820.8110.900 20.6490.6090.6870.5920.6750.6900.897 10.8290.6830.7970.8370.8060.8450.929 197

PAGE 198

T able7-16.PSNRvaluesfornoiselevel =35 onthebenchmarkdataset Image #NL-SVDNL-MeansHOSVD3DDCTBM3D1BM3D2Oracle 13 28.90526.54229.32228.62529.25929.89733.834 1227.49725.66328.54727.55927.94428.91432.188 1127.41925.33027.90326.86927.66428.28732.523 1026.93124.75027.57226.52227.34228.01032.796 929.43626.89529.53329.24729.31329.91332.197 824.92523.62825.82224.70125.51626.06531.128 727.69225.73228.11627.35427.93028.44032.378 629.46627.05729.84729.38629.67630.54533.778 525.94824.23726.47725.53126.31226.59031.071 423.38522.56523.53722.39423.40823.80429.055 329.10126.32129.46928.99229.30229.90033.358 224.16723.07824.57523.50424.46624.73029.155 130.82427.27530.82230.46230.45031.34934.869 T able7-17.SSIMvaluesfornoiselevel =35 onthebenchmarkdataset Image #NL-SVDNL-MeansHOSVD3DDCTBM3D1BM3D2Oracle 13 0.8380.6740.8100.8430.8140.8600.937 120.7900.6810.8150.8130.8010.8460.912 110.7160.6000.7270.7090.7180.7550.892 100.7110.5880.7360.7100.7240.7660.918 90.7020.6020.6980.7050.6910.7150.821 80.8310.7930.8690.8290.8590.8770.960 70.6870.5880.7070.6840.6980.7280.888 60.7980.6400.7830.8130.7790.8300.909 50.6550.5640.6780.6470.6690.6930.875 40.6110.5690.6320.5380.6300.6570.875 30.7780.6310.7610.7890.7590.7970.892 20.6020.5640.6420.5590.6370.6540.874 10.8140.6320.7900.8240.7820.8300.922 198

PAGE 199

T able7-18.Patch-sizeselectionfor =20 Image #BestPSNRBestscaleBest BestscaleBestPSNR (byPSNR) (by )by 1332.00089.648 1431.690 1230.990109.6501130.958 1130.26089.6401429.910 1030.03099.6501129.988 931.21089.7131630.968 828.12089.680628.110 730.19089.6391430.024 632.350109.6411432.190 529.19059.644928.890 426.16649.756626.044 332.02089.6391231.808 227.02059.673627.020 133.510109.6351133.499 199

PAGE 200

CHAPTER 8 AUTOMATEDSELECTIONOFFILTERPARAMETERS 8.1Introduction Despitethevastbodyofliteratureonimagedenoising,relativelylittleworkhas beendoneintheareaofautomaticallychoosingthelterparametersthatyieldoptimal lterperformance.Thetypicaldenoisingtechniquerequirestuningparametersthatare criticalforitsoptimalperformance.Indenoisingexperimentsreportedincontemporary literature,thelterperformanceisusuallymeasuredusingafull-referenceimage qualitymeasure(suchastheMSE/PSNRorSSIM)betweenthedenoisedandthe originalimage.Theparametersarepickedsoastoyieldtheoptimalvalueofthe qualitymeasureforaparticularlter,butthisrequiresknowledgeoftheoriginalimage andisnotextensibletoreal-worlddenoisingsituations.Henceweneedcriteriafor parameterselectionthatdonotrefertotheoriginalimage.Inthischapter,weclassify thesecriteriaintotwotypes:(1)independence-basedcriteriathatmeasurethedegree ofindependencebetweenthedenoisedimageandtheresidual,and(2)criteriathat measurehownoisytheresidualimageis,withoutreferringtothedenoisedimage.We contributetoandcritiquecriteriaoftype(1),andproposedthoseoftype(2).Ourcriteria maketheassumptionthatthenoiseisi.i.d.andadditive,andthatalooselower-bound onthenoisevarianceisknown. Thematerialinthischapterisbasedontheauthor'spublishedworkin[ 170 ] 1 .This chapterisorganizedasfollows:Section 8.2 reviewsexistingliteratureforlterparameter selection,followedbyadescriptionoftheproposedcriteriainSection 8.3 ,experimental resultsinSection 8.4 anddiscussioninSection 8.5. 1 P artsofthecontentsofthischapterhavebeenreprintedwithpermissionfrom:A. Rajwade,A.RangarajanandA.Banerjee,`AutomatedFilterParameterSelectionusing MeasuresofNoiseness',CanadianConferenceonComputerandRobotVision,pages 86-93,June2010. c r2010,IEEE. 200

PAGE 201

8.2 LiteratureReviewonAutomatedFilterParameterSelection InPDE-baseddenoising,thechoiceofstoppingtimeforthePDEevolutionisa crucialparameter.SomeresearchersproposetostopthePDEwhenthevarianceof theresidualequalsthevarianceofthenoise,whichisassumedtobeknown[ 86],[ 171]. Thismethodignoreshigherorderstatisticsofthenoise.Othersuseahypothesistest betweentheempiricaldistributionoftheresidualandthetruenoisedistribution[ 139]for polynomialorderselectioninregression-basedsmoothing.Howevertheexactvariance ofthenoiseoritscompletedistributionisusuallynotknowninpracticalsituations. Adecorrelation-basedcriterionindependentlyproposedin[ 172]and[ 173 ]doesnot requireanyknowledgeofthenoisedistributionexceptthatthenoiseisindependent oftheoriginalsignal.Asperthiscriterion,theoptimallterparameterischosentobe onewhichminimizesthecorrelationcoefcientbetweenthedenoisedandtheresidual images,regardlessofthenoisevariance.Thiscriterionhoweverhassomeproblems:(1) inthelimitofextremeover-smoothingorunder-smoothing,thecorrelationcoefcientis undenedasthedenoisedimagecouldbecomeaconstantimage,(2)itistooglobala criterion(thoughusingasumoflocalmeasuresisareadyalternative)and(3)itignores higher-orderdependencies.AsolutiontothethirdissueissuggestedbyusinSection 8.3. Itshouldbenotedthatalltheaforementionedcriteria(asalsotheoneswesuggest inthischapter)arenecessarybutnotsufcientforparameterselection.Gilboa et al. [174]attempttoalleviatethisbyselectingastoppingtimethatseekstomaximize thesignal-to-noise-ratio(SNR)directly.Theirmethodhoweverrequiresanestimate oftherateofchangeofthecovariancebetweentheresidualandthenoisew.r.t.the lteringparameter.Thisestimateinturnrequiresfullknowledgeofthenoisedistribution. Saddledwiththismethodistheassumptionthatthecovariancebetweentheresidualfor anyimageandtheactualnoise,canbeestimatedfromasinglenoise-image(generated fromthesamenoisedistribution)onwhichthelteringalgorithmisrun.Thisassumption 201

PAGE 202

is notjustiedtheoreticallythoughexperimentalresultsareimpressive(see[ 174]for moredetails).Vanhamel etal. [ 175]proposeacriterionthatmaximizesanestimate ofthecorrelationbetweenthedenoisedimageandthetrue,underlyingimage.This estimate,however,canbecomputedonlybyusingsomeassumptionsthathaveonly experimentaljustication.Inwaveletthresholdingmethods,riskbasedcriteriahave beenproposedfortheoptimalchoiceofthethresholdforthewaveletcoefcients.These methodssuchasthosein[ 113 ],ortheSURE-Stein'sunbiasedriskestimatorfrom [176],againrequireknowledgeofthenoisemodelincludingthenoisevariancevalue. Recently,Brunet etal. havedevelopedno-referencequalityestimatesoftheMSE betweenthedenoisedimageandthetrueunderlyingimage[ 141].Theseestimatesdo notrequireknowledgeoftheoriginalimage,buttheydorequireknowledgeofthenoise varianceandobtainarough,heuristicestimateofthecovariancebetweentheresidual andthenoise.Moreovertheperformanceoftheseestimateshasbeentestedonlyon Gaussiannoisemodels. 8.3Theory 8.3.1IndependenceMeasures Inwhatfollows,weshalldenotedthedenoisedimageobtainedbylteringanoisy image I as D ,itscorrespondingresidualas R (notethat R = I )Tj /T1_3 11.955 Tf 13.01 0 Td (D )andthetrue imageunderlying I as J .Asmentionedearlier,independence-basedcriteriahavebeen developedinimageprocessingliterature.Incaseswhereanoisysignalisoversmoothed (locallyorglobally),theresidualimageclearlyshowsthedistinctfeaturesfromthe image(referredtoas`methodnoise'in[ 2]).Thisistrueeveninthosecaseswhere thenoiseisindependentofthesignal.Independence-basedcriteriaarebasedonthe assumptionthatwhenthenoisyimageislteredoptimally,theresidualwouldcontain mostlynoiseandverylittlesignalandhenceitwouldbeindependentofthedenoised image.Ithasbeenexperimentallyreportedin[ 172]thattheabsolutecorrelation coefcient(denotedas CC )between D and R decreasesalmostmonotonicallyas 202

PAGE 203

the ltersmoothingparameterisincreased(indiscretesteps)fromalowerboundto acertain`optimal'value,afterwhichitsvalueincreasessteadilyuntilanupperbound. However,CCignoresanythinghigherthansecond-orderdependencies.Toalleviate thisproblem,weproposetominimizethemutualinformation(MI)between D and R asacriterionforparameterselection.Thishasbeenproposedasa(local)measure ofnoisenessearlierin[ 141],butithasbeenusedinthatpaperonlyasanindicator ofareasintheimagewheretheresidualisunfaithfultothenoisemodel,ratherthan asanexplicitparameter-selectioncriterion.Inthischapter,wealsoproposetouse thefollowinginformation-theoreticmeasuresofcorrelationfrom[ 39](seepage47)as independencecriteria: 1 (R D )=1 )Tj /T1_1 11.955 Tf 13.15 8.09 Td (H ( R jD ) H (D ) = MI (R D ) H ( D ) (8) 2 (R D ) =1 )Tj /T1_1 11.955 Tf 13.15 8.09 Td (H (D jR ) H (R ) = MI (R D ) H (R ) (8) Here H (X ) ref erstotheShannonentropyof X ,and H (X jY ) referstotheconditional Shannonentropyof X given Y 1 and 2 bothhavevaluesboundedfrom0(full independence)to1(fullindependence). Aproblemwithallthesecriteria(CC,MI, 1 2 )liesintheinherentprobabilistic notionofindependenceitself.Intheextremecaseofoversmoothing,the`denoised' imagemayturnouttohaveaconstantintensity,whereasinthecaseofextreme undersmoothing(nosmoothingorverylittlesmoothing),theresidualwillbeaconstant (zero)signal.Insuchcases, CC 1 2 areill-denedwhereas MI turnsouttobe zero(itsleastpossiblevalue).Whatthisindicatesisthatthesecriteriahavethe innatetendencytofavorextremecasesofunder-orover-smoothing.Inpractical applications,onemaychoosetogetaroundthisissuebychoosingalocalminimumof thesemeasureswithinaheuristicallychosenintervalintheparameterlandscapefrom 0to 1,butwewishtodrivehomeamorefundamentalpointabouttheinherentawin usingindependencemeasures.Moreover,itshouldbenotedthatlocalizedversionsof 203

PAGE 204

these measures(i.e.sumoflocalindependencemeasures)mayproducefalseoptimaif thelteringalgorithmsmoothesoutlocalregionswithnetextures. 8.3.2CharacterizingResidual`Noiseness' Giventhefactthattheassumednoisemodelisi.i.d.andsignalindependent, weexpecttheresidualproducedbyanidealdenoisingalgorithmtoobeythese characteristics.Therefore,patchesfromresidualimagesareexpectedtohavesimilar distributionsifthelteringalgorithmhasperformedwell.Ourcriterionforcharacterizing theresidual`noiseness'isrootedintheframeworkofstatisticalhypothesistesting.We choosethetwo-sampleKolmogorov-Smirnov(KS)testtocheckstatisticalhomogeneity. Thetwo-sampleKStest-statisticisdenedas K =sup x jF 1 (x ) )Tj /T1_2 11.955 Tf 11.95 0 Td (F 2 (x )j (8) where F 1 (x ) and F 2 (x ) aretherespectiveempiricalcumulativedistributionfunctions (ECDF)ofthetwosamples,computedwith N 1 and N 2 points.Underthenullhypothesis when N 1 !1 N 2 !1 ,thedistributionof K tendstotheKolmogorovdistribution,and isthereforeindependentoftheunderlyingtrueCDFsthemselves.Thereforethe K value hasaspecialmeaninginstatistics.Fora`signicancelevel' (theprobabilityoffalsely rejectingthenullhypothesisthatthetwoECDFswerethesame),let K bethestatistic valuesuchthat P (K K )=1 )Tj /T1_7 11.955 Tf 10.89 0 Td ( .Thenullhypothesisissaidtoberejectedatlevel if p N 1 N 2 N 1 + N 2 K > K Givenavalueofthetest-statisticcomputedempiricallyfromthesamples (denotedas ^ K ),weterm P (K ^ K ) (underthenull-hypothesis)asthep-value. Mostnaturalimages(apartfromhomogenoustextures)showaconsiderabledegree ofstatisticaldissimilarity.Todemonstratethis,weperformedthefollowingexperiment onall300imagesfromtheBerkeleydatabase[ 61].Eachimageatfourscaleswith successivedownsamplingfactorof 2 3 w astiledintonon-overlappingpatchesofsizes s s where s 2f 16,24,32g .Thetwo-sampleKStestfor =0.05 wasperformed forpatchesfromtheseimages.Theaveragerejectionratewas 81% whichindicates 204

PAGE 205

that differentregionsfromeachimagehavedifferentdistributions.Itshouldbenoted thatthetilingoftheimageintopatcheswasveryimportant:aKStestbetweensample subsetsfromrandom(non-contiguous)locationsproducedverylowrejectrates.A similarexperimentwiththesamescalesandpatchsizesrunonpureGaussiannoise imagesresultedinarejectionrateofonly 7% for =0.05.Next,Gaussiannoiseof =0.005 (forintensityrange[0,1])wasaddedtoeachimage.Eachimagewasltered usingthePerona-Maliklter[ 44]for90iterationswithastepsizeof0.05andedgeness criterionof r =40 andtheresidualimageswerecomputedafterthelastiteration.The KS-testwasperformedat =0.05 betweenpatchpairsfromeachresidualimage.The resultingrejectionratewas 41%,indicatingstrongheterogeneityintheresidualvalues. Asstructuralpatternswereclearlyvisibleinalltheseresidualimages,wetherefore conjecturethatstatisticalheterogeneityisastrongindicatorofthepresenceofstructure. Moreoverthepercentagerejectrate(denotedas h ),theaveragevalueoftheKS-statistic (i.e. K )andtheaveragenegativelogarithmofthep-valuesfromeachpairwisetest (denotedas P )areallindicatorsofthe`noiseness'ofaresidual(thelowerthevalue,the noisierandhencemoredesirabletheresidual).Hencethesemeasuresactascriteria forlterparameterselection 2 .Wepreferthecriteria P and K to h becausetheydonot requireasignicanceleveltobespecied apriori TheadvantageoftheKS-basedmeasureoverMIorCCisthatvaluesof P and K arehighincasesofimageoversmoothing(astheresidualwillthencontainmore andmorestructure).ThisisunlikeMIorCCwhichwillattainfalseminima.Thisis demonstratedinFigure 8-1 wherethedecreaseinthevaluesofMIorCCathigh smoothinglevelsisquiteevident.JustlikeMIorCC,theKS-basedcriteriadonot requireknowledgeofthenoisedistributionoreventheexactnoisevariance.However 2 F orcomputing P ,thereistheassumptionthatthepairwisetestsbetweenindividual patchesareallindependent,forthesakeofsimplicity. 205

PAGE 206

all thesecriteriacouldbefooledbythepathologicalcaseofzeroorverylowdenoising. Thisisbecauseintheveryinitialstagesofdenoising(obtainedby,say,runninga PDEwithaverysmallstepsizeforveryfewiterations),theresidualislikelytobe devoidofstructureandindependentofthe(under)smoothedimage.Consequently, allmeasures:MI,CC, K and P willacquire(falsely)lowvalues.Thisproblemcanbe avoidedbymakingassumptionsoftherangeofvaluesforthenoisevariance(oraloose lower-bound),withoutrequiringexactknowledgeofthevariance.Thishasbeenthe strategyfollowedimplicitlyincontemporaryparameterselectionexperiments(e.g.in [172]thePDEstepsizesarechosentobe0.1and1).Inallourexperiments,wemake similarassumptions.TheexceptionisthatKS-basedmeasuresdonotrequireanyupper boundonthevariancetobeknown:justalowerboundsufces. 8.4ExperimentalResults Todemonstratetheeffectivenessoftheproposedcriteria,weperformedexperiments onallthe13imagesfromthebenchmarkdataset.Allimagesfromthedatasetwere down-sizedfrom 512 512 to 256 256.Weexperimentedwith6noiselevels 2 n 2f10 )Tj /T1_6 7.97 Tf (4 ,5 10 )Tj /T1_6 7.97 Tf 6.59 0 Td (4 ,10 )Tj /T1_6 7.97 Tf 6.59 0 Td (3 ,5 10 )Tj /T1_6 7.97 Tf 6.59 0 Td (3 ,0.01,0.05 g onanintensityrangeof[0,1],and withtwoadditivezero-meannoisedistributions:Gaussian(themostcommonlyused assumption)andboundeduniform(noiseduetoquantization).Thelower-bound assumedonthenoisevariancewas 10 )Tj /T1_6 7.97 Tf 6.59 0 Td (6 inallexperiments.Twolteringalgorithms weretested:thenon-localmeans(NL-Means)lter[ 2]andtotalvariation(TV)[ 86].The equationfortheNL-Meanslterisasfollows: ^ I (x )= P x k 2 N (x ; SR ) w k ( x )I (x k ) P x k 2 N (x ;SR ) w k (x ) (8) w k (x ) =exp )Tj /T1_4 11.955 Tf 8.14 -9.68 Td ()Tj 13.15 8.09 Td (kq (x ; QR ) )Tj /T1_1 11.955 Tf 11.96 0 Td (q (x k ; QR )k 2 ) (8) where ^ I (x ) is theestimatedsmoothedintensity, N ( x ; SR ) isasearchwindowofdiameter SR aroundpoint x w k (x ) isaweightfactor, q (x ; QR ) isapatchofdiameter QR centered 206

PAGE 207

at x and is asmoothingparameter 3 .Inourexperiments,apatchsizeof 12 12 wasused,withasearchwindowof50x50. waschosenbyrunningtheNL-Means algorithmfor55different valuesforsmoothing,fromtheset f1:1:10,20:20: 640,650:50:1200 g.Theoptimal valueswerecomputedusingthefollowing criteria: CC ( D R ) MI (D R ), 1 (D R ) 2 (D R );sumoflocalizedversionsofall abovemeasuresona 12 12 window; h P and K usingtwo-sampleKStestson non-overlapping 12 12 patches;and h n P n and K n valuescomputedusingKS-test betweentheresidualandthetruenoisesamples(whichweknowhereastheseare syntheticexperiments).Allinformationtheoreticquantitieswerecomputedusing40bins astheimagesizewas 256 256 (thethumbrulefortheoptimalnumberofbinsfor n samplesis O (n 1 = 3 )). Thetotalvariation(TV)lterisobtainedbyminimizingthefollowingenergy: E (I )= Z n jrI (x )jdx (8) foranimagedenedonthedomain n ,whichgivesageometricheatequationPDE thatisiteratedsome T timesstartingfromthegivennoisyimageasaninitialcondition. Thestoppingtime T istheequivalentofthesmoothingparameterhere.FortheTV lter,inadditiontoallthecriteriamentionedbefore,wealsotestedthemethodin[ 174] (assumingknowledgeofthenoisedistribution). 8.4.1ValidationMethod Inordertovalidatethe or T estimatesproducedbythesecriteria,itisimportant toseehowwelltheyareintunewiththoselterparametervaluesthatareoptimalwith regardtodifferentwell-knownqualitymeasuresbetweenthedenoisedimageandthe originalimage.ThemostcommonlyusedqualitymeasureistheMSE.Howeveras 3 Note thatweuse todenotethesmoothingparameterofthelteringalgorithmand 2 n todenotethevarianceofthenoise. 207

PAGE 208

documented in[ 150]andmentionedinChapter 6 ,MSEhasseverallimitations.Hence wealsoexperimentedwithstructuredsimilarityindex(SSIM)developedin[ 151 ];withthe L1differencebetweenthedenoisedandtheoriginalimage;andwiththeCC,MI, 1 and 2 valuesbetweenthedenoisedandtheoriginalimage(aswellaswiththesumoftheir localversions). 8.4.2ResultsonNL-Means ResultsonNL-MeansforGaussiannoiseofsixdifferentvariancesareshownin Tables 8-1 throughto 8-6. Inallthesetables, X =absolutedifferencebetween` X valuesaspredictedbythecriterion,andtheoptimal` X 'value.The` X 'valueisdened tobethequalitymeasure` X 'betweenthedenoisedandthetrueimage,chosenhere tobe L 1 ortheSSIM. d X istheabsolutedifferencebetweenthe valueforNL-Means predictedbythecriterionandthe valuewhenthequalitymeasure X wasoptimal. Theotherqualitymeasuresarenotshownheretosavespace.Thelasttworowsofthe tablesindicatetheminimumandmaximumoftheoptimalqualitymeasurevaluesacross allthe13imagesonwhichtheexperimentswererun(whichgivesanideaaboutthe rangeoftheoptimal X values). Someresultsonthe`stream'and`mandrill'imagesareshowninFigure 8-2 and 8-3 withthecorrespondingresiduals.Experimentswerealsoperformedonimages degradedwithboundeduniformnoiseoftotalwidth 2 5 10 )Tj /T1_2 7.97 Tf 6.59 0 Td (4 and 2 5 10 )Tj /T1_2 7.97 Tf (3 (onan intensityrangeof[0,1])withresultsshowninTables 8-7 and 8-8. Forlowandmoderatenoiselevels,itwasobservedthatthecriteria P or K producederrorsanorderofmagnitudebetterthanMI, 1 and 2 (whichweretheclosest competitors)andeventwoordersofmagnitudebetterthanCC.Ourobservationwas thatCCandinformationtheoreticcriteriatendtocauseundersmoothingforNL-Means. Athighnoiselevels,wesawthatallcriteriaproducedahigherrorinpredictionofthe optimalparameter.AnexplanationforthisisthattheNL-Meansalgorithmbyitself doesnotproduceverygoodresultsathighnoiselevels,andrequireshigh values 208

PAGE 209

which producehighlystructuredresiduals.Forlow values,itproducesresidualsthat resemblenoiseinthesenseofvariouscriteria,butthisleadstohugelyundersmoothed estimates. AninterestingphenomenonweobservedwasthatthesameKS-testbased measures(i.e. P n and K n )betweentheresidualsandtheactualnoisesamples(which weknow,asthesearesyntheticexperiments)oftendidnotperformaswellasthe KS-testmeasures(i.e. P and K )betweenpairsofpatchesfromtheresidual.We conjecturethatthisisowingtobiasesinherentintheNL-Meansalgorithm(asinmany others-see[ 148])duetowhichtheresidualshavedifferentmeansandvariancesas comparedtotheactualnoise,eventhoughtheresidualsmaybehomogenous.We checkedexperimentallythatthevarianceoftheresidualsproducedbyNL-Meansunder valuesoptimalinanL1-sensewassignicantlydifferentfromthenoisevariance. 8.4.3EffectofPatchSizeontheKSTest TheKStestemployedhereoperatesonimagepatches.Thepatch-sizecan beacrucialparameter:toolowapatchsize(say 2 2)willleadtoreductioninthe discriminatorypoweroftheKStestforthisapplicationandcause(false)rejectionfor alllterparameters,whereastoohighapatchsizewillleadto(false)acceptanceforall lterparameters.Wechoseapatchsizesothatthenumberofsamplesforestimation ofthecumulativewassufcient.Thiswasdeterminedinsuchawaythatthemaximum absoluteerrorbetweentheestimatedandtrueunderlyingCDFswasnomorethan0.1 withaprobabilityof0.9.Then,usingtheDvoretzky-Kiefer-Wolfowitzinequality,itfollows thatthereshouldbeatleast149samples[ 177],[178 ].Hencewechoseapatchsizeof 12 12.However,wealsoperformedanexperimentwithNL-MeanswheretheKS-test wasperformedacrossmultiplescalesfrom12to60instepsof8(foranimageofsize 256 256),andaverage h P and K valueswerecalculated.Howeverfortheseveral experimentsdescribedintheprevioussections,wejustusedthepatchsizeof 12 12, asthemultiscalemeasuredidnotproducesignicantlybetterresults. 209

PAGE 210

8.4.4 ResultsonTotalVariation ResultsfortotalvariationdiffusionwithGaussiannoiseofvariance 5 10 )Tj /T1_5 7.97 Tf (4 and 5 10 )Tj /T1_5 7.97 Tf 6.59 0 Td (3 areshowninTables 8-9 and 8-10.Forthismethod,theKS-basedmeasures performedwellintermsoferrorsinpredictingthecorrectnumberofiterationsandthe correctqualitymeasures,butnotaswellasMIwithintherestrictedstoppingtimerange. Theresultswerealsocomparedtothoseobtainedfrom[ 174]whichperformedthebest, thoughwewouldliketoremindthereaderthatmethodfrom[ 174]requiresknowledgeof thefullnoisedistribution.Also,inthecaseoftotalvariation,theKS-basedmeasuresdid notoutperformMI.Anexplanationforthisisthatthetotalvariationmethodisunableto producehomogenousresidualsforitsoptimalparameterset,asitisspecicallytuned forpiecewiseconstantimages.Thisassumptiondoesnotholdgoodforcommonly occurringnaturalimages.Asagainstthis,NL-Meansisalterexpresslyderivedfrom theassumptionthatpatchesin`clean'naturalimages(andthosewithlowormoderate noise)haveseveralsimilarpatchesindistantpartsoftheimage. 8.5DiscussionandAvenuesforFutureWork Inthischapter,wehavecontributedtoandcritiquedindependence-basedcriteriafor lterparameterselectionandpresentedacriterionthatmeasuresthehomogeneityof theresidualstatistics.Onthewhole,wehavecontributedtotheparadigmofexploiting statisticalpropertiesoftheresidualimagesfordrivingthedenoisingalgorithm.The proposednoisenessmeasuresrequirenootherassumptionsexceptthat(1)thenoise shouldbei.i.d.andadditive,andthat(2)alooselowerboundonthenoisevariance isknowntopreventfalseminimawithextremeundersmoothing.UnlikeCCorMI,the KS-basednoisenessmeasuresareguaranteednottobeyieldfalseminimaincaseof oversmoothing. TheKS-basednoisenesscriteriarequireaveragingofthe P or K valuesfrom differentpatches.Forfuturework,thiscanbereplacedbyperforming k -sampleversions oftheKolmogorov-SmirnovorrelatedtestssuchasCramervon-Mises[ 179]between 210

PAGE 211

individual patchesversusapooledsamplecontainingtheentireresidualimage.Thiswill produceasingle K or P valueforthewholeimage. Theassumptionofi.i.d.noisemaynotholdinsomedenoisingscenarios.Incase ofazero-meanGaussiannoisemodelwithintensitydependentvariances,aheuristic approachwouldbetonormalizetheresidualssuitablyusingfeedbackfromthedenoised intensityvalues(regardingthemasthe`true'imagevalues)andthenrunningthe KS-tests.Theefcacyofthisapproachneedstobetestedondenoisingalgorithmsthat arecapableofhandlingintensitydependentnoise.IncasethenoiseobeysaPoisson distribution(whichisneitherfullyadditivenormultiplicative),therearetwowaysto proceed:eitherapplyavariancestabilizertransformation[ 180]whichconvertsthedata intothatcorruptedbyGaussiannoisewithvarianceofone,orelsesuitablychangethe denitionoftheresidualitself. Moreover,theexistenceofauniversallyoptimalparameterselectorisnotyet established:differentcriteriamayperformbetterorworsefordifferentdenoising algorithmsorwithdifferentassumptionsonthenoisemodel.Thisis,asperour surveyoftheliterature,anunsolvedprobleminimageprocessing.Lastly,despite encouragingexperimentalresults,thereisnoestablishedtheoreticalrelationship betweentheperformanceofnoisenesscriteriaforlterparameterselectionandthe `ideal'parametersintermsofimagequalitycriterialikeMSE.Adetailedstudyof risk-basedcriteriasuchasthosein[ 113]maybeimportantinthiscontext. 211

PAGE 212

Figure 8-1.PlotsofCC,MI, P andMSEonanimagesubjectedtoupto16000iterations oftotalvariationdenoising Table8-1.(NL-Means)Gaussiannoise 2 n =0.0001 L 1 d L1 SSIM d SSIM h 0.088 10.4620.0027.692 P 0.0317.5380.00512.462 K 0.0407.8460.00410.308 CC 0.0859.8460.01017.846 MI 0.18917.0770.01118.615 1 0.18917.0770.01118.615 2 0.176180.01120.769 LocalMI0.0558.7690.00716.154 h NM 0.85131.4620.00916.385 P NM 0.08713.2310.0024.923 K NM 0.21519.3850.0015.538 Min2.225-0.888Max9.147-0.986212

PAGE 213

A B C D E F G H I J K L M N Figure 8-2.ImageswithGaussiannoisewith 2 n =5 10 )Tj /T1_2 7.97 Tf 6.59 0 Td (3 denoisedbyNL-Means. Parameterselectedforoptimalnoisenessmeasures:(a): P ,(c) K ,(e) CC (g)MI;andoptimalqualitymeasures:(i) L 1,(k)SSIM,(m)MIbetween denoisedimageandresidual.Correspondingresidualsin (b),(d),(f),(h),(j),(l),(n);Zoominpdfleforbetterview 213

PAGE 214

A B C D E F G H I J K L M N Figure 8-3.ImageswithGaussiannoisewith 2 n =5 10 )Tj /T1_2 7.97 Tf 6.59 0 Td (3 denoisedbyNL-Means. Parameterselectedforoptimalnoisenessmeasures:(a): P ,(c) K ,(e) CC (g)MI;andoptimalqualitymeasures:(i) L 1,(k)SSIM,(m)MIbetween denoisedimageandresidual.Correspondingresidualsin (b),(d),(f),(h),(j),(l),(n);Zoominpdfleforbetterview 214

PAGE 215

T able8-2.(NL-Means)Gaussiannoise 2 n =0.0005 L 1 d L1 SSIM d SSIM h 0.029 7.6920.00312.769 P 0.0104.6150.00617.385 K 0.0145.3850.00415.077 CC 0.0688.4620.00715.077 MI 0.08714.6150.00919.692 1 0.08714.6150.00919.692 2 0.23219.9230.01526.538 LocalMI0.15514.6150.00513.538 h n 0.43628.1540.0038 P n 0.04710.7690.0039.692 K n 0.12816.1540.0015.846 Min2.683-0.884Max9.383-0.981T able8-3.(NL-Means)Gaussiannoise 2 n =0.001 L 1 d L1 SSIM d SSIM h 0.041 9.2310.00416.154 P 0.0245.3850.00516.923 K 0.0357.6920.00414.615 CC 0.15116.1540.00412.308 MI 0.12619.2310.00818.462 1 0.12619.2310.00818.462 2 0.15726.9230.01329.231 LocalMI0.19120.3080.00310.308 h n 0.21822.3080.0016.154 P n 0.041100.00212.308 K n 0.10015.3850.0016.923 Min3.069-0.879Max9.601-0.976215

PAGE 216

T able8-4.(NL-Means)Gaussiannoise 2 n =0.005 L1 d L1 SSIM d SSIM h 0.206 33.8460.00324.615 P 0.20733.8460.00324.615 K 0.488430.00533.769 CC 2.25392.3080.03455.385 MI 2.67779.5380.05467.231 1 2.72081.0770.05468.769 2 2.119105.2310.053105.231 LocalMI3.889107.8460.06974 h n 1.337380.03238 P n 1.33533.5380.03336.615 K n 1.33633.5380.03442.769 Min4.838-0.791Max10.695-0.955T able8-5.(NL-Means)Gaussiannoise 2 n =0.01 L1 d L1 SSIM d SSIM h 12.121 226.6920.202177.462 P 12.121226.6920.202177.462 K 12.121226.6920.202177.462 CC 8.886207.1540.149157.923 MI 11.701224.4620.195175.231 1 11.701224.4620.195175.231 2 6.2182000.119163.077 LocalMI12.121226.6920.202177.462 h n 1.89160.9230.04564 P n 3.64286.4620.081108 K n 3.649880.082115.692 Min6.285-0.735Max11.661-0.933216

PAGE 217

T able8-6.(NL-Means)Gaussiannoise 2 n =0.05 L1 d L 1 SSIM d SSIM h 14.704 906.1540.183643.846 P 11.290838.4620.140576.154 K 9.597805.3850.118543.077 CC 18.95910200.249757.692 MI 19.4351026.1540.253763.846 1 19.5501027.6920.255765.385 2 19.4351026.1540.253763.846 LocalMI19.7831030.7690.258768.462 h n 24.5161028.4620.305796.923 P n 26.7211120.7690.325858.462 K n 26.7211120.7690.325858.462 Min11.748-0.555Max17.478-0.806T able8-7.(NL-Means)Uniformnoisewidth=0.001 L 1 d L1 SSIM d SSIM h 0.055 9.6920.00310.923 P 0.0135.6920.00614.923 K 0.0216.6150.00514.000 CC0.08711.0770.00917.231 MI0.18813.3850.00717.077 1 0.18813.3850.00717.077 2 0.26716.8460.00919.000 LocalMI0.24417.0770.01121.692 h n 0.77030.2310.00814.538 P n 0.05410.3080.0037.231 K n 0.11414.4620.0014.615 Min2.339-0.887-1.195Max9.176-0.985-2.011217

PAGE 218

T able8-8.(NL-Means)Uniformnoisewidth=0.01 L 1 d L1 SSIM d SSIM h 0.034 10.7690.00518.462 P 0.0279.2310.00520.000 K 0.04212.3080.00516.923 CC0.43034.6150.00414.615 MI0.13716.9230.00727.692 1 0.13716.9230.00727.692 2 0.12521.5380.01135.385 LocalMI0.47736.3080.00416.308 h n 0.0259.2310.00420.000 P n 0.0209.2310.00626.154 K n 0.02510.7690.00627.692 Min3.522-0.860-1.148Max9.835-0.970-1.834T able8-9.(TV)Gaussiannoise 2 n =0.0005 L1 dt L1 SSIM dt SSIM h 0.558 53.4620.00656.538 P 0.52248.4620.00652.308 K 0.51346.5380.00650.385 CC3.487365.0000.088371.923 MI0.10320.7690.00123.077 1 0.10320.7690.00123.077 2 2.478267.6920.062274.615 LocalMI0.47936.9230.00532.308 h n 0.53869.6150.00776.538 P n 0.52368.8460.00775.769 K n 0.52869.6150.00776.538 Gilboa etal. [ 174]0.05010.2310.00116.385 Min2.622-0.975Max4.426-0.995218

PAGE 219

T able8-10.(TV)Gaussiannoise 2 n =0.005 L1 dt L1 SSIM dt SSIM h 0.665 129.6150.008102.692 P 0.493109.6150.00680.385 K 0.430102.3080.00673.077 CC2.156350.7690.073376.923 MI0.42288.8460.012118.077 1 0.42288.8460.012118.077 2 1.849296.9230.063331.538 LocalMI5.475270.7690.084240.769 h n 0.19459.6150.00896.538 P n 0.22174.2310.008115.000 K n 0.21675.7690.008116.538 Gilboa etal. [ 174]0.09460.0000.00242.000 Min4.995-0.892Max11.284-0.980219

PAGE 220

CHAPTER 9 CONCLUSIONANDFUTUREWORK 9.1ListofContributions Wehavepresentedcontributionstotwomajorproblemsfundamentaltoimage processing:probabilitydensityestimationandimagedenoising.Thecontributionsto probabilitydensityestimationareasfollows: 1.DevelopmentofanewPDFestimatorforimageswhichaccountsforthefactthat theimageisnotjustabunchofsamples,butadiscreteversionofanunderlying continuoussignal. 2.ExtensionoftheaboveconceptforjointPDFsoftwoormoreimages,denedon 2Dor3Ddomains. 3.Extensionoftheaboveconceptstodevelopthreedifferentbiaseddensity estimatorsthatfavorthehighergradientregionsorpointsofasingleimage(in 2D/3D),apairofimages(in2D/3D)oratripleofimages(in3D). 4.ApplicationofalltheabovePDFestimatorstoafneimageregistration. 5.ApplicationofallunbiasedPDFestimatorstolteringofgrayscaleandcolor images,chromaticityeldsandgrayscalevideo,inamean-shiftframework. 6.Developmentofdensityestimatorsforunit-vectordatasuchaschromaticityand hueincolorimagesbymakingexplicituseofthefactthattheyareobtainedas transformationofcolormeasurementsthatcanbeassumedtolieinEuclidean space. Thecontributionstoimagedenoisingareasfollows: 1.Wehavedevelopedanon-localimagedenoisingalgorithm(NL-SVD)aftera seriesofexperimentsonthepatchSVD.OurtechniquelearnsSVDbasesforan ensembleofpatchesthataresimilartoareferencepatchlocatedateachpixel. Thesespatiallyadaptivebasesareshowntoproduceexcellentperformanceon imagedenoising,comparabletothestateoftheart. 2.Ourmethodhasparameterswhichareobtainedinaprincipledmannerfromthe noisemodel.Themethodisthuselegantandefcientasitdoesnotneedany complicatedoptimizationprocedure. 3.WehaveextendedtheNL-SVDtechniquetoperformjointlteringofimage patches,leadingtotheHOSVDbasedlteringtechniquethatyieldsevenbetter imagequalityvalues. 220

PAGE 221

4.W ehavealsopresentedanewstatisticalcriterionforautomatedlterparameter selectionandusedittoobtainthesmoothingparameterintheNLMeansalgorithm withoutreferencetothetrueimage. 9.2FutureWork FutureworkontheprobabilitydensityestimatorhasbeenoutlinedinSection 3.4. Here,weleavebehindpointerstopossiblefutureextensionsofourworkinimage denoising. 9.2.1TryingtoReachtheOracle TheultimateaimofseveraloftheproceduresreportedinChapter 7 wastoobtain theSVDbasesoftheunderlyingpatch.ThebasesobtainedbyNL-SVDandHOSVD yieldexcellentperformancebutarestillfarbehindtheoracledenoiser.Isitpossible toobtainthetruebasesorbasesthatareveryclosetothetruebases?Arethere otherbasesthatwouldyieldequivalentperformance?Thesequestionsremainopen problems. 9.2.2BlindandNon-blindDenoising Inmanycontemporarydenoisingalgorithms[2 ],[ 146],[ 134],oneassumes knowledgeofthetruenoisevarianceasthisallowsprincipledselectionofvarious parameters.However,thenoisevarianceisoftennotknowninpracticeandthisiscalled asa`blinddenoisingscenario'.Insuchcases,onecanuseknowledgeaboutthesensor deviceingettinganideaofthenoisevariance.However,environmentalfactorstoocan affectimagequality,andinsuchcases,onecannotmerelyusesensorproperties.In practice,thenoisevariancecanbeestimatedfromthenoisydataavailable.Oneof themostcommonlyusedtechniquesfornoisevarianceestimationcomputestheHaar wavelettransformoftheimage.ThemaximumabsolutedeviationoftheHHsub-band (highfrequencycomponentsinbothxandydirections)isconsideredtobeareasonable estimateofthenoisevariance[ 181].Threetraining-basedmethodsarepresentedin [182]:twowhichmakeuseofaLaplacianpriorfornaturalimages,andanotherwhich measuresnoisevariancefromthevarianceofhomogenousregionsinanoisyimage. 221

PAGE 222

A statisticalcriterionfordistinguishingbetweenhomogenousregionsandregionswith edges/orientedtextureispresentedin[ 183].Developmentofarobustnoisevariance estimatorandusingitinconjunctionwiththedenoisingmethodpresentedinthisthesis, isaninterestingdirectionforfuturework.Furthermore,onecanalsoside-stepthe problemofnoisevarianceestimationasfollows:ourdenoisingalgorithmcanberun assumingseveraldifferentvaluesforthenoisestandarddeviation .Thisaffectsthe criticalparametersfortransform-domainthresholdingormeasurementofsimilarity betweenpatches.Afterdenoising,onecancomputeoneofthenoisenessmeasures discussedinthepreviouschaptersandselectthe valuethatproducedthe`noisiest' residual. 9.2.3ChallengingDenoisingScenarios Ourdenoisingalgorithmhasbeentestedthoroughlyon(i.i.d.andadditive) zero-meanGaussiannoiseatdifferentvaluesof .Mostcontemporaryalgorithms fromtheliteraturehavealsobeentestedonlyonGaussiannoise.Thismodelisknown toholdtrueforthermalnoiseandalsoforlmgrainnoiseundersomeconditions[ 184]. However,thereexistseveralothernoisemodelssuchasthenegativeexponentialmodel whichaffectsimagesacquiredthroughsyntheticapertureradar,Poissonnoisewhich isavalidmodelforimagesacquiredwithcamerashavinglowshutterspeedorunder poorillumination,orspecklenoiseinultrasound[ 184].Thepatchsimilaritymeasure, therelativebehaviorofthetruesignalandthenoiseinstancesinthetransformdomain, andthechoiceofnormsorenergycriteriatooptimizeforsuitabledenoisingbases,are allaffectedbytheassumednoisemodel.InthecaseofdistributionslikePoissonwhich arenotreallyadditive,characterizationofthebehaviorofthenoiseinstancesinthe transformdomainposesadifcultproblem.Tocomplicatemattersfurther,thenoise affectingtheimagemaybeintensitydependentordrawnfromnoisedistributionsthat arespatiallyvarying.Infact,thePoissonmodelisonesuch,thenoiseinducedbylossy compressionalgorithmsisanother.Alltheseproblemspresentrichavenuesforfuture 222

PAGE 223

research. Ultimately,actualcameranoiseisthecumulativeeffectofseveralfactors: shutterspeed,ambientillumination,stabilityofthecameratakingthepicture,motion oftheobjectsinthescene,thebehavioroftheelectroniccircuitryinsidethecamera, andthelossycompressionalgorithmtostoretheimages.Acarefulstudyofallthese factorsandtheinterplaybetweenthemisanimportantopenprobleminpracticalimage processing. 223

PAGE 224

APPENDIX A DERIVATIONOFMARGINALDENSITY Inthissection,wederivetheexpressionforthemarginaldensityoftheintensityofa single2Dimage.WebeginwithEq.( 2)derivedinSection 2.2.1: p ( )= 1 A Z I (x y )= f f f f f f f @ x @ I @ y @ I @ x @ u @ y @ u f f f f f f f du (A) Consider thefollowingtwoexpressionsthatappearwhileperformingachangeof variablesandapplyingthechainrule: dxdy =[ dIdu ] 2 6 4 @ x @ I @ y @ I @ x @ u @ y @ u 3 7 5 (A) dI du =[ dxdy ] 2 6 4 @ I @ x @ u @ x @ I @ y @ u @ y 3 7 5 = [ dxdy ] 2 6 4 I x u x I y u y 3 7 5 (A) Takingtheinverseinthelatter,wehave dxdy = 1 I x u y )Tj /T1_1 11.955 Tf 11.95 0 Td (I y u x 2 6 4 u y )Tj /T1_1 11.955 Tf (u x )Tj /T1_1 11.955 Tf (I y I x 3 7 5 [ dI du ]. (A) Comparingtheindividualmatrixcoefcients,weobtain f f f f f f f @ x @ I @ y @ I @ x @ u @ y @ u f f f f f f f = I x u y )Tj /T1_1 11.955 Tf 11.96 0 Td (u x I y (I x u y )Tj /T1_1 11.955 Tf 11.95 0 Td (I y u x ) 2 = 1 I x u y )Tj /T1_1 11.955 Tf 11.96 0 Td (I y u x (A) No w,clearlytheunitvector ~ u isperpendicularto ~ I ,i.e.wehavethefollowing: u y = I x p I 2 x + I 2 y and (A) u x = )Tj /T1_1 11.955 Tf (I y p I 2 x + I 2 y (A) 224

PAGE 225

This nallygivesus f f f f f f f @ x @ I @ y @ I @ x @ u @ y @ u f f f f f f f = 1 p I 2 x + I 2 y (A) Hence wearriveatthefollowingexpressionforthemarginaldensity: p ( )= 1 A Z I ( x ,y )= du p I 2 x + I 2 y (A) This isthesameexpressionasinEq.( 2 ). 225

PAGE 226

APPENDIX B THEOREMONTHEPRODUCTOFACHAINOFSTOCHASTICMATRICES Thespecictheoremfrom[133]ontheproductofachainofstochasticmatricesis producedhere(verbatim)forcompleteness: Let n beanarbitrarysetandletforeach 2 n P = 2 6 6 6 6 6 6 6 6 6 6 4 p 11 ... p 1N ..... ..... ..... p N 1 ... p NN 3 7 7 7 7 7 7 7 7 7 7 5 (B) bearow-stochasticmatrix,i.e.amatrixwith P j p ij =1 and p ij 0 forall (i j ).Then supposeallmatrices P satisfytheconditionthatthereexistsaconstant c > 0 suchthat P j c j min c where c j min denotestheminimumvalueoftheelementsinthe j th column of P .Let = f! 1 2 ,... g beanarbitrarysequenceofelementsfrom n.Thenthelimit M =lim n !1 P n P n )Tj /T1_8 5.978 Tf 5.76 0 Td (1 ... P 1 existsandisamatrixwithidenticalrowsgivenas M = 2 6 6 6 6 4 1 ... N ..... 1 ... N 3 7 7 7 7 5 (B) Moreoverif M n =lim n !1 P n P n )Tj /T1_8 5.978 Tf 5.76 0 Td (1 ... P 1 ,thenforany i 1 2 N X j =1 jM n (i j ) )Tj /T1_3 11.955 Tf 11.95 0 Td ( j j (1 )Tj /T1_5 11.955 Tf 11.96 0 Td (c ) n n 0. (B) forsomeprobabilityvector f 1 2 ,..., N g.Theconvergencerateisthusupper boundedby (1 )Tj /T1_5 11.955 Tf 11.95 0 Td (c ) n 226

PAGE 227

REFERENCES [1]Tina IsNoAcronym(TINA)ImageDatabase,Availablefrom http://www.tina-vision. net/ilib.php ,2008,UniversityofManchesterandUniversityofShefeld,UK. [2]A.Buades,B.Coll,andJ.-M.Morel,Areviewofimagedenoisingalgorithms,with anewone, Multiscalemodellingandsimulation ,vol.4,no.2,pp.490,2005. [3]B.Silverman, DensityEstimationforStatisticsandDataAnalysis .London,UK: ChapmanandHall,1986. [4]J.Simonoff, SmoothingMethodsinStatistics .Berlin,Germany:SpringerVerlag, 1996. [5]C.Bishop, PatternRecognitionandMachineLearning .SpringerVerlag,2006. [6]D.Herrick,G.Nason,andB.Silverman,Somenewmethodsforwaveletdensity estimation, Sankhya ,vol.63,pp.394,2001. [7]A.PeterandA.Rangarajan,Maximumlikelihoodwaveletdensityestimationwith applicationstoimageandshapematching, IEEETrans.ImageProcess.,vol.17, no.4,pp.458,April2008. [8]D.Donoho,I.Johnstone,G.Kerkyacharian,andD.Picard,Densityestimationby waveletthresholding, Ann.Stat.,vol.24,pp.508,1996. [9]A.Rajwade,A.Banerjee,andA.Rangarajan,Newmethodofprobabilitydensity estimationwithapplicationtomutualinformationbasedimageregistration, in IEEEConf.ComputerVisionandPatternRecognition ,vol.2,2006,pp. 1769. [10],Continuousimagerepesentationsavoidthehistogrambinningproblem inmutualinformationbasedimageregistration,in IEEEInt.Symp.Biomedical Imaging,2006,pp.840. [11],Probabilitydensityestimationusingisocontoursandisosurfaces: applicationstoinformation-theoreticimageregistration, IEEETrans.Pattern Anal.Mach.Intell.,vol.31,no.3,pp.475,2009. [12]T.KadirandM.Brady,Estimatingstatisticsinarbitraryregionsofinterest,in BritishMach.VisionConf. ,2005,pp.589. [13]N.JoshiandM.Brady,Nonparametricmixturemodelbasedevolutionoflevel sets,in Int.Conf.Computing:TheoryandApplications(ICCTA),2007,pp. 618. [14]E.Hadjidemetriou,M.Grossberg,andS.Nayar,Histogrampreservingimage transformations, Int.J.Comput.Vis. ,vol.45,no.1,pp.5,2001. 227

PAGE 228

[15]J .BoesandC.Meyer,Multi-variatemutualinformationforregistration,in Med. ImageComput.Computer-AssistedIntervention ,ser.LNCS,vol.1679.Springer, 1999,pp.606. [16]J.ZhangandA.Rangarajan,Multimodalityimageregistrationusinganextensible informationmetric,in Inf.Process.Med.Img. ,ser.LNCS,vol.3565.Springer, 2005,pp.725. [17]M.deBerg,M.vanKreveld,M.Overmars,andO.Schwarzkopf, Computational Geometry:AlgorithmsandApplications.Berlin,Germany:SpringerVerlag,1997. [18]L.Rudin,S.Osher,andE.Fatemi,Nonlineartotalvariationbasednoiseremoval algorithms, PhysicaD ,vol.60,pp.259,1992. [19]D.L.Collinsetal.,Designandconstructionofarealisticdigitalbrainphantom, IEEETrans.Med.Imag.,vol.17,no.3,pp.463,1998. [20]J.Pluim,J.Maintz,andM.Viergever,Mutualinformationbasedregistrationof medicalimages:Asurvey, IEEETrans.Med.Imag. ,vol.22,no.8,pp.986, 2003. [21]H.Chen,M.Arora,andP.Varshney,Mutualinformation-basedimageregistration forremotesensingdata, J.RemoteSensing,vol.24,no.18,pp.3701, 2003. [22]P.ViolaandW.Wells,Alignmentbymaximizationofmutualinformation, Int.J. Comput.Vis. ,vol.24,no.2,pp.137,1997. [23]F.Maes,A.Collignon,D.Vandermeulen,G.Marchal,andP.Suetens, Multimodalityimageregistrationbymaximizationofmutualinformation, IEEE Trans.Med.Imag.,vol.16,no.2,pp.187,1997. [24]F.Maes,D.Vandermeulen,andP.Suetens,Medicalimageregistrationusing mutualinformation, Proc.IEEE ,vol.91,no.10,pp.1699,2003. [25]M.Rao,Y.Chen,B.Vemuri,andF.Wang,Cumulativeresidualentropy:Anew measureofinformation, IEEETrans.Inf.Theory ,vol.50,no.6,pp.1220, 2004. [26]F.WangandB.Vemuri,Non-rigidmulti-modalimageregistrationusing cross-cumulativeresidualentropy, Int.J.Comput.Vis.,vol.74,no.2,pp. 201,2007. [27]F.Wang,B.Vemuri,M.Rao,andY.Chen,Cumulativeresidualentropy,anew measureofinformationanditsapplicationtoimagealignment,in IEEEInt.Conf. ComputerVision,2003,pp.548. 228

PAGE 229

[28]J .Beirlant,E.Dudewicz,L.Gy or,andE.C.vanderMeulen,Nonparametric entropyestimation:Anoverview, Int.J.Math.Stat.Sci. ,vol.6,no.1,pp.17, June1997. [29]P.Viola,Alignmentbymaximizationofmutualinformation,Ph.D.dissertation, MassachussetsInstituteofTechnology,1995. [30]C.Yang,R.Duraiswami,N.Gumerov,andL.Davis,ImprovedfastGauss transformandefcientkerneldensityestimation,in IEEEInt.Conf.Computer Vision,vol.1,2003,pp.464. [31]M.LeventonandW.Grimson,Multi-modalvolumeregistrationusingjointintensity distributions,in Med.ImageComput.Computer-AssistedIntervention,vol.1496, 1998,pp.1057. [32]T.DownieandB.Silverman,Awaveletmixtureapproachtotheestimationof imagedeformationfunctions, SankhyaSeriesB,vol.63,pp.181,2001. [33]B.Ma,A.Hero,J.Gorman,andO.Michel,Imageregistrationwithminimum spanningtreealgorithm,in IEEEInt.Conf.ImageProcess. ,vol.1,2000,pp. 481. [34]J.CostaandA.Hero,Entropicgraphsformanifoldlearning,in IEEEAsilomar Conf.Sign.,Sys.andComp. ,vol.1,2003,pp.316. [35]M.SabuncuandP.Ramadge,GradientbasedoptimizationofanEMSTimage registrationfunction,in IEEEInt.Conf.Acoust.,Speech,Sig.Proc. ,vol.2,2005, pp.253. [36]N.Dowson,R.Bowden,andT.Kadir,Imagetemplatematchingusingmutual informationandNP-Windows,in Int.Conf.PatternRecognition,vol.2,2006,pp. 1186. [37]B.Karacali,Informationtheoreticdeformableregistrationusinglocalimage information, Int.J.Comput.Vis. ,vol.72,no.3,pp.219,2007. [38]P.Th evenazandM.Unser,Optimizationofmutualinformationformultiresolution imageregistration, IEEETrans.ImageProcess. ,vol.9,no.12,pp.2083, 2000. [39]T.CoverandJ.Thomas, ElementsofInformationTheory .NewYork,USA:Wiley Interscience,1991. [40]J.ZhangandA.Rangarajan,Afneimageregistrationusinganewinformation metric,in IEEEConf.ComputerVisionandPatternRecognition ,vol.1,2004,pp. 848. [41]W.Feller,OntheKolmogorov-Smirnovlimittheoremsforempiricaldistributions, TheAnnalsofMathematicalStatistics,vol.19,no.2,pp.177,1948. 229

PAGE 230

[42]R. ShekharandV.Zagrodsky,Mutualinformation-basedrigidandnonrigid registrationofultrasoundvolumes, IEEETrans.Med.Imag.,vol.21,no.1,pp. 9,2002. [43]L.Devroye,L.Gyor,andG.Lugosi, AProbabilisticTheoryofPatternRecognition.Berlin,Germany:SpringerVerlag,1996. [44]P.PeronaandJ.Malik,Scale-spaceandedgedetectionusinganisotropic diffusion, IEEETrans.PatternAnal.Mach.Intell.,vol.12,no.7,pp.629, 1990. [45]D.Tschumperl eandR.Deriche,Vector-valuedimageregularizationwithPDEs: Acommonframeworkfordifferentapplications, IEEETrans.PatternAnal.Mach. Intell.,vol.27,no.4,pp.506,2005. [46]B.TangandG.Sapiro,Colorimageenhancementviachromaticitydiffusion, IEEETrans.ImageProcess. ,vol.10,pp.701,1999. [47]P.Saint-Marc,J.Chen,andG.Medioni,Adaptivesmoothing:ageneraltoolfor earlyvision, IEEETrans.PatternAnal.Mach.Intell. ,vol.13,no.6,pp.514, 1991. [48]K.PlataniotisandA.Venetsanopoulos, Colorimageprocessingandapplications NewYork,USA:SpringerVerlag,2000. [49]C.TomasiandR.Manduchi,Bilaterallteringforgrayandcolorimages,in IEEE Int.Conf.ComputerVision,1998,pp.839. [50]Y.Cheng,Meanshift,modeseekingandclustering, IEEETrans.PatternAnal. Mach.Intell.,vol.17,no.8,pp.790,1995. [51]D.ComaniciuandP.Meer,Meanshift:arobustapproachtowardfeaturespace analysis, IEEETrans.PatternAnal.Mach.Intell. ,vol.24,no.5,pp.603, 2002. [52]A.Rajwade,A.Banerjee,andA.Rangarajan,Imagelteringdrivenbylevel curves,in Int.Conf.EnergyMin.MethodsComputerVisionPatternRecognition 2009,pp.359. [53]T.ChanandJ.Shen, ImageProcessingandAnalysis:Variational,PDE,wavelets, andstochasticmethods.SIAM,2005. [54]D.BarashandD.Comaniciu,Acommonframeworkfornonlineardiffusion, adaptivesmoothing,bilaterallteringandmeanshift, ImageVis.Comput. ,vol.22, pp.73,2004. [55]A.Buades,B.Coll,andJ.-M.Morel,NeighborhoodltersandPDEs, Numerische Mathematik,vol.105,no.1,pp.1,2006. 230

PAGE 231

[56]R. SubbaraoandP.Meer,Discontinuitypreservinglteringoveranalytic manifolds,in IEEEConf.ComputerVisionandPatternRecognition ,2007,pp. 1. [57]J.vandeWeijerandR.vandenBloomgard,Localmodeltering,in IEEEConf. ComputerVisionandPatternRecognition,vol.2,2001,pp.428. [58]N.Sochen,R.Kimmel,andR.Malladi,Ageneralframeworkforlowlevelvision, IEEETrans.ImageProcess. ,vol.7,pp.310,1998. [59]D.Comaniciu,Analgorithmfordata-drivenbandwidthselection, IEEETrans. PatternAnal.Mach.Intell.,vol.25,pp.281,2003. [60]D.Comaniciu,V.Ramesh,andP.Meer,Thevariablebandwidthmeanshift anddata-drivenscaleselection,in IEEEInt.Conf.ComputerVision,2001,pp. 438. [61]D.Martin,C.Fowlkes,D.Tal,andJ.Malik,Adatabaseofhumansegmented naturalimagesanditsapplicationtoevaluatingsegmentationalgorithmsand measuringecologicalstatistics,in IEEEInt.Conf.ComputerVision ,vol.2,2001, pp.416. [62]S.Lansel,AboutDenoiseLab,Availablefrom http://www.stanford.edu/ slansel/ DenoiseLab/documentation.htm,2006. [63]Z.Wang,E.Simoncelli,andA.Bovik,Multi-scalestructuralsimilarityforimage qualityassessment,in IEEEAsilomarConf.Signals,Sys.Comp. ,2003,pp. 1398. [64]O.Subakan,J.Bing,B.Vemuri,andE.Vallejos,Featurepreservingimage smoothingusingacontinuousmixtureoftensors,in IEEEInt.Conf.Computer Vision,2007,pp.1. [65]H.Takeda,S.Farsiu,andP.Milanfar,Kernelregressionforimageprocessingand reconstruction, IEEETrans.ImageProcess.,vol.16,no.2,pp.349,2007. [66]K.MardiaandP.Jupp, DirectionalStatistics.Chichester,UK:WileyInterscience, 2000. [67]P.KimandJ.Koo,Directionalmixturemodelsandoptimalestimationofthe mixingdensity, Can.J.Stat.,pp.383,1998. [68]A.Banerjee,I.Dhillon,J.Ghosh,andS.Sra,Clusteringontheunithypersphere usingvonMises-Fisherdistributions, J.Mach.LearningRes.,vol.6,pp. 1345,2005. [69]T.McGraw,B.Vemuri,R.Yezierski,andT.Mareci,VonMises-Fishermixture modelofthediffusionODF,in IEEEInt.Symp.BiomedicalImaging ,2006,pp. 65. 231

PAGE 232

[70]A. Prati,S.Calderara,andR.Cucchiara,Usingcircularstatisticsfortrajectory shapeanalysis,in IEEEConf.ComputerVisionandPatternRecognition ,June 2008,pp.1. [71]K.Hara,K.Nishino,andK.Ikeuchi,Multiplelightsourcesandreectance propertyestimationbasedonamixtureofsphericaldistributions, IEEEInt.Conf. ComputerVision,vol.2,pp.1627,Oct.2005. [72]C.Han,B.Sun,R.Ramamoorthi,andE.Grinspun,Frequencydomainnormal mapltering, ACMTrans.Graphics,vol.26,no.3,pp.28,2007. [73]O.EugeciougluandA.Srinivasan,Efcientnonparametricdensityestimation onthespherewithapplicationsinuidmechanics, SIAMJournalonScientic Computing,vol.22,no.1,pp.152,2000. [74]A.Papoulis, Probability,RandomVariablesandStochasticProcesses .McGraw Hill,1984. [75]H.Schaeben,Normalorientationdistributions, TexturesandMicrostructures vol.19,pp.197,1992. [76]A.Bijral,M.Breitenbach,andG.Grudic,MixtureofWatsondistributions:A generativemodelforhypersphericalembeddings,in AIandStatistics ,2007,pp. 1. [77]T.DownsandA.L.Gould,Somerelationshipsbetweenthenormalandvon Misesdistributions, Biometrika ,vol.54,no.3,pp.684,1967. [78]B.Presnell,S.Morrison,andR.Littell,Projectedmultivariatelinearmodelsfor directionaldata, J.Am.Stat.Assoc. ,vol.93,no.443,pp.1068,1998. [79]T.GeversandH.Stokman,Robusthistogramconstructionfromcolorinvariants forobjectrecognition, IEEETrans.PatternAnal.Mach.Intell.,vol.25,no.10,pp. 113,2003. [80]B.Pelletier,KerneldensityestimationonRiemannianmanifolds, Stat.Prob. Letters,vol.73,pp.297,2005. [81]D.DonohoandT.Weissman,Recenttrendsindenoisingtutorial,ISIT2007, Availablefrom http://www.stanford.edu/ slansel/tutorial/summary.htm ,2007. [82]B.M.terHaarRomeny, Geometry-drivendiffusionincomputervision.Utrecht, Netherlands:Kluwer,1994. [83]J.Weickert, AnisotropicDiffusioninImageProcessing .Stuttgart,Germany: Teubner,1998. [84]M.Black,G.Sapiro,D.Marimont,andD.Heeger,Robustanisotropicdiffusion, IEEETrans.ImageProcess. ,vol.7,no.3,pp.421,1998. 232

PAGE 233

[85]F .Catte,P.Lions,J.Morel,andT.Coll,Imageselectivesmoothingandedge detectionbynonlineardiffusion, SIAMJ.Numer.Anal.,vol.29,no.1,pp. 182,1992. [86]L.RudinandS.Osher,Totalvariationbasedimageresorationwithfreelocal constraints,in IEEEInt.Conf.ImageProcess. ,1994,pp.31. [87]T.Le,R.Chartrand,andT.Asaki,Avariationalapproachtoreconstructing imagescorruptedbyPoissonnoise, J.Math.Imag.Vis.,vol.27,pp.257, 2007. [88]G.Gilboa,N.Sochen,andY.Zeevi,Imageenhancementanddenoisingby complexdiffusionprocesses, IEEETrans.PatternAnal.Mach.Intell.,vol.26, no.8,pp.1020,2004. [89]D.SeoandB.Vemuri,Complexdiffusiononscalarandvectorvaluedimage graphs,in Int.Conf.EnergyMin.MethodsComput.VisionPatternRecognition 2009,pp.98. [90]Y.YouandM.Kaveh,Fourthorderpartialdifferentialequationsfornoiseremoval, IEEETrans.ImageProcess. ,vol.9,no.10,pp.1723,2000. [91]M.Hajiaboli,Ananisotropicfourth-orderpartialdifferentialequationfornoise removal,in ScaleSpaceandVariationalMethodsinComputerVision,2009,pp. 356. [92]P.Mr azek,Monotonicityenhancingnonlineardiffusion, J.VisualCommun.Image Representation,vol.13,no.1,pp.313,2000. [93]A.SavitzkyandM.Golay,Smoothinganddifferentiationofdatabysimpliedleast squaresprocedures, Anal.Chem.,vol.36,no.8,pp.1627,1964. [94]W.Press,A.Teukolsky,W.Vetterling,andB.Flannery, NumericalrecipesinC (2nded.):theartofscienticcomputing .NewYork,NY,USA:Cambridge UniversityPress,1992. [95]J.FanandI.Gijbels, Localpolynomialmodelinganditsapplication .London,UK: ChapmanandHill,1996. [96]S.M.SmithandJ.M.Brady,SUSAN-anewapproachtolowlevelimage processing, Int.J.Comput.Vis. ,vol.23,pp.45,1995. [97]V.Katkovnik,A.Foi,K.Egiazarian,andJ.Astola,Directionalvaryingscale approximationsforanisotropicsignalprocessing,in Eur.SignalProcess.Conf. 2004,pp.1. [98]K.FukunagaandL.Hostetler,Theestimationofthegradientofadensity function,withapplicationsinpatternrecognition, IEEETrans.Inf.Theory ,vol.21, no.1,pp.32,1975. 233

PAGE 234

[99]M. Elad,Ontheoriginofthebilaterallterandwaystoimproveit, IEEETrans. ImageProcess. ,vol.11,no.10,pp.1141,2002. [100]N.Sochen,R.Kimmel,andA.Bruckstein,Diffusionsandconfusionsinsignaland imageprocessing, J.Math.Imag.Vis. ,vol.14,pp.195,2001. [101]O.Subakan,Continuousmixturemodelsforfeaturepreservingsmoothingand segmentation,Ph.D.dissertation,UniversityofFlorida,2009. [102]B.Jian,B.Vemuri,E.Ozarslan,P.Carney,andT.Mareci,Anoveltensor distributionmodelforthediffusion-weightedMRsignal, Neuroimage,vol.37, no.1,pp.164,2007. [103]D.Tschumperl e,Fastanisotropicsmoothingofmulti-valuedimagesusing curvature-preservingPDEs, Int.J.Comput.Vis.,vol.68,no.1,pp.65,2006. [104]R.FrankotandR.Chellappa,Amethodforenforcingintegrabilityinshapefrom shadingalgorithms, IEEETrans.PatternAnal.Mach.Intell. ,vol.10,no.4,pp. 439,1988. [105]A.AgrawalandR.Raskar,Shortcourse(ICCV2007):Gradientdomain manipulationtechniquesinvisionandgraphics, http://www.umiacs.umd.edu/ aagrawal/ICCV2007Course/index.html,2007. [106]H.Wang,Y.Chen,T.Fang,J.Tyan,andN.Ahuja,Gradientadaptiveimage restorationandenhancement,in Int.Conf.ImageProc.,2006,pp.2893. [107]P.Hancock,R.Baddeley,andL.Smith,Theprincipalcomponentsofnatural images, Network:ComputationinNeuralSystems ,vol.3,pp.61,1992. [108]R.CoifmanandD.Donoho,Translation-invariantdenoising,YaleUniversity,Tech. Rep.,1995. [109]L.Yaroslavsky,K.Egiazarian,andJ.Astola,Transformdomainimagerestoration methods:review,comparisonandinterpretation,in SPIEProceedingsSeries, NonlinearProcessingandPatternAnalysis,2001,pp.1. [110]W.FreemanandE.Adelson,Thedesignanduseofsteerablelters, IEEETrans. PatternAnal.Mach.Intell.,vol.13,no.9,pp.891,1991. [111]J.-L.Starck,E.Candes,andD.Donoho,Thecurvelettransformforimage denoising, IEEETrans.ImageProcess. ,vol.11,no.6,pp.670,2002. [112]J.FanandR.Li,Variableselectionvianonconcavepenalizedlikelihoodandits oracleproperties, J.Am.Stat.Assoc.,vol.96,no.456,pp.1348,December 2001. [113]D.DonohoandI.Johnstone,Idealspatialadaptationbywaveletshrinkage, Biometrika,vol.81,pp.425,1993. 234

PAGE 235

[114]J .Mairal,G.Sapiro,andM.Elad,Learningmultiscalesparserepresentationsfor imageandvideorestoration, MultiscaleModelingandSimulation,vol.7,no.1,pp. 214,2008. [115]L.SendurandI.Selesnick,Bivariateshrinkagefunctionsforwavelet-based denoisingexploitinginterscaledependency, IEEETrans.SignalProcess. ,vol.50, no.11,pp.2744,2002. [116]E.Simoncelli,Bayesiandenoisingofvisualimagesinthewaveletdomain,in LectureNotesinStatistics,vol.141,1999,pp.291. [117]J.Portilla,V.Strela,M.Wainwright,andE.Simoncelli,Imagedenoisingusing scalemixturesofGaussiansinthewaveletdomain, IEEETrans.ImageProcess. vol.12,no.11,pp.1338,2003. [118]A.Hyvarinen,P.Hoyer,andE.Oja,Imagedenoisingbysparsecodeshrinkage, in IntelligentSignalProcessing,1999,pp.1. [119]J.HuangandD.Mumford,Statisticsofnaturalimagesandmodels, IEEEConf. ComputerVisionandPatternRecognition,vol.1,pp.1541,1999. [120]Y.Hel-OrandD.Shaked,Adiscriminativeapproachforwaveletdenoising, IEEE Trans.ImageProcess. ,vol.17,no.4,pp.443,2008. [121]A.Buades,B.Coll,andJ.-M.Morel,Nonlocalimageandmoviedenoising, Int.J. Comput.Vis. ,vol.76,no.2,pp.123,2008. [122]T.Brox,O.Kleinschmidt,andD.Cremers,Efcientnonlocalmeansfordenoising oftexturalpatterns, IEEETrans.ImageProcess.,vol.17,no.7,pp.1083, 2008. [123]M.Ghazel,G.Freeman,andE.Vrscay,Fractalimagedenoising, IEEETrans. ImageProcess. ,vol.12,no.12,pp.1560,2003. [124]S.AwateandR.Whitaker,Unsupervised,information-theoretic,adaptiveimage lteringforimagerestoration, IEEETrans.PatternAnal.Mach.Intell. ,vol.28, no.3,pp.364,2006. [125]K.PopatandR.Picard,Cluster-basedprobabilitymodelanditsapplicationto imageandtextureprocessing, IEEETrans.ImageProcess.,vol.6,no.2,pp. 268,1997. [126]A.EfrosandT.Leung,Texturesynthesisbynonparametricsampling,in IEEEInt. Conf.ComputerVision,1999,pp.1033. [127]J.D.Bonet,Noisereductionthroughdetectionofsignalredundancy,MIT,AILab, Tech.Rep.,1997. 235

PAGE 236

[128]D .ZhangandZ.Wang,Restorationofimpulsenoisecorruptedimagesusing long-rangecorrelation, IEEESignalProcess.Letters ,vol.5,no.1,pp.4,1998. [129],Imageinformationrestorationbasedonlong-rangecorrelation, IEEETrans. CircuitSyst.VideoTechnol. ,vol.12,no.5,pp.331,2002. [130]S.Kindermann,S.Osher,andP.Jones,Deblurringanddenoisingofimagesby nonlocalfunctionals, SIAMInterdisc.J. ,vol.4,no.4,pp.1091,2005. [131]M.EbrahimiandE.Vrscay,Self-similarityinimaging,20yearsafter`fractals everywhere',in Int.WorkshopLocalNon-LocalApprox.ImageProcess. ,2008, pp.165. [132]O.Kleinschmidt,T.Brox,andD.Cremers,Nonlocaltexturelteringwithefcient treestructuresandinvariantpatchsimilaritymeasures,in Int.WorkshopLocal Non-LocalApprox.ImageProcess. ,2008,pp.1. [133]O.Steno,Perfectsamplingfromthelimitofdeterministicproductsofstochastic matrices, ElectronicCommun.Prob.,vol.13,pp.474,2008. [134]K.Dabov,A.Foi,V.Katkovnik,andK.Egiazarian,Imagedenoisingbysparse 3-dtransform-domaincollaborativeltering, IEEETrans.ImageProcess. ,vol.16, no.8,pp.2080,2007. [135]K.HirakawaandT.Parks,Imagedenoisingusingtotalleastsquares, IEEE Trans.ImageProcess. ,vol.15,no.9,pp.2730,2006. [136]L.Dascal,M.Zibulevsky,andR.Kimmel,Signaldenoisingbyconstrainingthe residualtobestatisticallynoise-similar,Technion,Israel,Tech.Rep.,2008. [137]E.Tadmor,S.Nezzar,andL.Vese,Amultiscaleimagerepresentationusing hierarchical(BV,L2)decompositions, Multiscalemodellingandsimulation ,vol.2, pp.554,2004. [138]S.Osher,M.Burger,D.Goldfarb,J.Xu,andW.Yin,Aniterativeregularization methodfortotalvariation-basedimagerestoration, Multiscalemodellingand simulation ,vol.4,pp.460,2005. [139]J.PolzehlandV.Spokoiny,Imagedenoising:apointwiseadaptiveapproach, Ann.Stat.,vol.31,no.1,pp.30,2003. [140]J.KleinbergandE.Tardos, AlgorithmDesign.Boston,USA:Addison-Wesley Longman,2005. [141]D.Brunet,E.Vrscay,andZ.Wang,Theuseofresidualsinimagedenoising,in Int.Conf.ImageAnal.Recognition,2009,pp.1. 236

PAGE 237

[142]Y .Chen,H.Wang,T.Fang,andJ.Tyan,Mutualinformationregularizedbayesian frameworkformultipleimagerestoration,in IEEEInt.Conf.ComputerVision, 2005,pp.190. [143]B.OlshausenandD.Field,Emergenceofsimple-cellreceptive-eldpropertiesby learningasparsecodefornaturalimages, Nature,vol.381,no.6583,p.607609, 1996. [144]M.Lewicki,T.Sejnowski,andH.Hughes,Learningovercomplete representations, NeuralComputation ,vol.12,pp.337,1998. [145]M.LewickiandB.Olshausen,Aprobabilisticframeworkfortheadaptationand comparisonofimagecodes, J.Opt.Soc.Am. ,vol.16,pp.1587,1999. [146]M.EladandM.Aharon,Imagedenoisingvialearneddictionariesandsparse representation,in IEEEConf.ComputerVisionandPatternRecognition ,vol.1, 2006,pp.17. [147]M.Aharon,M.Elad,andA.Bruckstein,TheK-SVD:analgorithmfordesigningof overcompletedictionariesforsparserepresentation, IEEETrans.SignalProcess., vol.54,no.11,pp.4311,2006. [148]P.ChatterjeeandP.Milanfar,Clustering-baseddenoisingwithlocallylearned dictionaries, IEEETrans.ImageProcess.,vol.18,no.7,pp.1438,2009. [149]J.Yang,J.Wright,T.Huang,andY.Ma,Imagesuper-resolutionassparse representationofrawimagepatches,in IEEEInt.Conf.Comp.Vis.PatternRec. 2008,pp.1. [150]Z.WangandA.Bovik,Meansquarederror:Loveitorleaveit?Anewlookat signaldelitymeasures, IEEESignalProcess.Mag.,vol.26,no.1,pp.98, 2009. [151]Z.Wang,A.Bovik,H.Sheikh,andE.Simoncelli,Imagequalityassessment: Fromerrorvisibilitytostructuralsimilarity, IEEETrans.ImageProcess.,vol.13, no.4,pp.600,2004. [152]H.AndrewsandC.Patterson,Singularvaluedecompositionsanddigitalimage processing, IEEETrans.Acoust.,SpeechandSignalProcess.,vol.24,no.1,pp. 425,1976. [153],Singularvaluedecomposition(SVD)imagecoding, IEEETrans.Commun., vol.24,no.4,pp.425,1976. [154]L.TrefethenandD.Bau, NumericalLinearAlgebra .SIAM:SocietyforIndustrial andAppliedMathematics,1997. [155]A.Rangarajan,Learningmatrixspaceimagerepresentations,in Int.Conf. EnergyMin.MethodsComputerVisionPatternRecognition,2001,pp.153. 237

PAGE 238

[156]D .Tschumperl eandR.Deriche,Orthonormalvectorsetsregularizationwith PDEsandapplications, Int.J.Comput.Vis. ,vol.50,pp.237,2002. [157]J.Ye,Generalizedlowrankapproximationsofmatrices, Mach.Learning ,vol.61, no.1,pp.167,2005. [158]C.DingandJ.Ye,Two-dimensionalsingularvaluedecomposition(2DSVD)for2D mapsandimages,in SIAMInt.Conf.DataMining ,2005,pp.32. [159]N.Kwak,PrincipalcomponentanalysisbasedonL1-normmaximization, IEEE Trans.PatternAnal.Mach.Intell.,vol.30,no.9,pp.1672,2008. [160]G.Heo,P.Gader,andH.Frigui,RKF-PCA:RobustkernelfuzzyPCA, Neural Networks,vol.22,no.5-6,pp.642,2009. [161]A.EfrosandW.Freeman,Imagequiltingfortexturesynthesisandtransfer,in SIGGRAPH:AnnualConf.Computergraphicsandinteractivetechniques ,2001, pp.341. [162]A.Hyvarinen,J.Hurri,andP.Hoyer, NaturalImageStatistics:AProbabilistic ApproachtoEarlyComputationalVision.Springer,Heidelberg,2009. [163]A.RosenfeldandA.Kak, DigitalPictureProcessing .Orlando,USA:Academic Press,1982. [164]L.Tucker,Somemathematicalnotesonthree-modefactoranalysis, Psychometrika,vol.31,no.3,pp.279,1966. [165]L.deLathauwer,Signalprocessingbasedonmultilinearalgebra,Ph.D. dissertation,KatholiekeUniversiteitLeuven,Belgium,1997. [166]M.VasilescuandD.Terzopoulos,Multilinearanalysisofimageensembles: Tensorfaces,in Int.Conf.PatternRecognition,2002,pp.511. [167]K.Gurumoorthy,A.Rajwade,A.Banerjee,andA.Rangarajan,BeyondSVD: Sparseprojectionsontoexemplarorthonormalbasesforcompactimage representation,in Int.Conf.PatternRecognition,2008,pp.1. [168],Amethodforcompactimagerepresentationusingsparsematrixandtensor projectionsontoexemplarorthonormalbases, IEEETrans.ImageProcess., vol.19,no.2,pp.322,2010. [169]D.MuresanandT.Parks,Adaptiveprincipalcomponentsandimagedenoising, in IEEEInt.Conf.ImageProcess. ,2003,pp.101. [170]A.Rajwade,A.Rangarajan,andA.Banerjee,Automatedlterparameter selectionusingmeasuresofnoiseness,in Can.Conf.Comput.RobotVision, 2010,pp.86. 238

PAGE 239

[171]J .Weickert,Coherenceenhancingdiffusionltering, Int.J.Comput.Vis. ,vol.31, no.3,pp.111,1999. [172]P.Mr azekandM.Navara,Selectionofoptimalstoppingtimefornonlinear diffusionltering, Int.J.Comput.Vision ,vol.52,no.2-3,pp.189,2003. [173]J.-F.Aujol,G.Gilboa,T.Chan,andS.Osher,Structure-textureimage decompositionmodeling,algorithmsandparameterselection, Int.J.Comput. Vis. ,vol.67,no.1,pp.111,2006. [174]G.Gilboa,N.Sochen,andY.Zeevi,EstimationofoptimalPDE-baseddenoising intheSNRsense, IEEETrans.ImageProcess.,vol.15,no.8,pp.2269, Aug.2006. [175]I.Vanhamel,C.Mihai,H.Sahli,A.Katartzis,andI.Pratikakis,Scaleselectionfor compactscale-spacerepresentationofvector-valuedimages, Int.J.Comput.Vis. vol.84,no.2,pp.194,2009. [176]D.DonohoandI.Johnstone,Adaptingtounknownsmoothnessviawavelet shrinkage, J.Am.Stat.Assoc. ,vol.90,no.432,pp.1200,1995. [177]A.Dvoretzky,J.Kiefer,andJ.Wolfowitz,Asymptoticminimaxcharacterofthe sampledistributionfunctionandoftheclassicalmultinomialestimator, Ann.Math. Stat.,vol.27,no.3,pp.642,1956. [178]Wikipedia,DvoretzkyKieferWolfowitzinequality,Availablefrom http://en. wikipedia.org/wiki/Dvoretzky-Kiefer-Wolfowitz inequality, 2010. [179]J.Kiefer,K-sampleanaloguesoftheKolmogorov-SmirnovandCramer-Von Misestests, Ann.Math.Stat.,vol.30,no.2,pp.420,1959. [180]F.Anscombe,ThetransformationofPoisson,binomialandnegative-binomial data, Biometrika ,vol.35,pp.246,1948. [181]D.Donoho,Denoisingbysoftthresholding, IEEETrans.Inf.Theory ,vol.41, no.3,pp.613,1995. [182]A.D.Stefano,P.White,andW.Collis,Trainingmethodsforimagenoiselevel estimationonwaveletcomponents, EURASIPJ.Appl.SignalProcess.,vol.2004, no.16,pp.2400,2004. [183]X.ZhuandP.Milanfar,Ano-referenceimagecontentmetricanditsapplicationto denoising,in IEEEInt.Conf.ImageProcess. ,2010,pp.1. [184]C.Boncelet,Imagenoisemodels,in HandbookofImageandVideoProcessing A.Bovik,Ed.NewYork,USA:AcademicPress,2005,pp.397. 239

PAGE 240

BIOGRAPHICAL SKETCH AjitRajwadewasbornandbroughtupinthecityofPune,India.Hecompleted hisbachelor'sdegreeincomputerengineeringfromtheGovernmentCollegeof Engineering,Pune(afliatedtotheUniversityofPune)in2001,hismaster'sdegree incomputersciencefromMcGillUniversity,Montreal,Canadain2004,andhisdoctoral degreeincomputerengineeringfromtheUniversityofFloridain2010.Hisresearch interestsareincomputervisionandimageprocessing,andcomputationalgeometry. 240