<%BANNER%>

Image Retargeting and Smart Projectors

MISSING IMAGE

Material Information

Title:
Image Retargeting and Smart Projectors
Physical Description:
1 online resource (64 p.)
Language:
english
Creator:
Qi, Shaoyu
Publisher:
University of Florida
Place of Publication:
Gainesville, Fla.
Publication Date:

Thesis/Dissertation Information

Degree:
Doctorate ( Ph.D.)
Degree Grantor:
University of Florida
Degree Disciplines:
Computer Engineering, Computer and Information Science and Engineering
Committee Chair:
Ho, Jeffrey Yih Chian
Committee Members:
Peters, Jorg
Vemuri, Baba C
Entezari, Alireza
Hager, William Ward

Subjects

Subjects / Keywords:
applications -- camera -- image -- retargeting -- stereo
Computer and Information Science and Engineering -- Dissertations, Academic -- UF
Genre:
Computer Engineering thesis, Ph.D.
bibliography   ( marcgt )
theses   ( marcgt )
government publication (state, provincial, terriorial, dependent)   ( marcgt )
born-digital   ( sobekcm )
Electronic Thesis or Dissertation

Notes

Abstract:
Image retargeting algorithms aim to adjust the image to the display screen whose aspect ratio or size differ significantly from that of the image by applying a non-uniform and content-aware image resizing. Compared to uniform image scaling, retargeting provides considerably better results in terms of its preservation of the image’s content and integrity. Due to recent advances in display technology, image retargeting has emerged recently as an active field of research in computer vision and graphics. In my thesis, I will investigate two new types of image retargeting problems and their solutions. First, I will propose a novel algorithm that can retarget images to domains with non-rectangular boundary. This improvement over the existing image retargeting algorithms allow greater flexibility in choosing the retargeted image domain and consequently, it permits a wider range of applications. Second, I will propose a novel algorithm that can simultaneously retarget 3D images and adjust the resulting disparity values. I will propose the novel idea that retargeting 3D images should consider the dual objectives of preserving image content as well as maximizing viewing comfort. At the last part I will introduce a smart projector system to adaptively project images in accordance of the detected display area.
General Note:
In the series University of Florida Digital Collections.
General Note:
Includes vita.
Bibliography:
Includes bibliographical references.
Source of Description:
Description based on online resource; title from PDF title page.
Source of Description:
This bibliographic record is available under the Creative Commons CC0 public domain dedication. The University of Florida Libraries, as creator of this bibliographic record, has waived all rights to it worldwide under copyright law, including all related and neighboring rights, to the extent allowed by law.
Statement of Responsibility:
by Shaoyu Qi.
Thesis:
Thesis (Ph.D.)--University of Florida, 2013.
Local:
Adviser: Ho, Jeffrey Yih Chian.

Record Information

Source Institution:
UFRGP
Rights Management:
Applicable rights reserved.
Classification:
lcc - LD1780 2013
System ID:
UFE0045374:00001

MISSING IMAGE

Material Information

Title:
Image Retargeting and Smart Projectors
Physical Description:
1 online resource (64 p.)
Language:
english
Creator:
Qi, Shaoyu
Publisher:
University of Florida
Place of Publication:
Gainesville, Fla.
Publication Date:

Thesis/Dissertation Information

Degree:
Doctorate ( Ph.D.)
Degree Grantor:
University of Florida
Degree Disciplines:
Computer Engineering, Computer and Information Science and Engineering
Committee Chair:
Ho, Jeffrey Yih Chian
Committee Members:
Peters, Jorg
Vemuri, Baba C
Entezari, Alireza
Hager, William Ward

Subjects

Subjects / Keywords:
applications -- camera -- image -- retargeting -- stereo
Computer and Information Science and Engineering -- Dissertations, Academic -- UF
Genre:
Computer Engineering thesis, Ph.D.
bibliography   ( marcgt )
theses   ( marcgt )
government publication (state, provincial, terriorial, dependent)   ( marcgt )
born-digital   ( sobekcm )
Electronic Thesis or Dissertation

Notes

Abstract:
Image retargeting algorithms aim to adjust the image to the display screen whose aspect ratio or size differ significantly from that of the image by applying a non-uniform and content-aware image resizing. Compared to uniform image scaling, retargeting provides considerably better results in terms of its preservation of the image’s content and integrity. Due to recent advances in display technology, image retargeting has emerged recently as an active field of research in computer vision and graphics. In my thesis, I will investigate two new types of image retargeting problems and their solutions. First, I will propose a novel algorithm that can retarget images to domains with non-rectangular boundary. This improvement over the existing image retargeting algorithms allow greater flexibility in choosing the retargeted image domain and consequently, it permits a wider range of applications. Second, I will propose a novel algorithm that can simultaneously retarget 3D images and adjust the resulting disparity values. I will propose the novel idea that retargeting 3D images should consider the dual objectives of preserving image content as well as maximizing viewing comfort. At the last part I will introduce a smart projector system to adaptively project images in accordance of the detected display area.
General Note:
In the series University of Florida Digital Collections.
General Note:
Includes vita.
Bibliography:
Includes bibliographical references.
Source of Description:
Description based on online resource; title from PDF title page.
Source of Description:
This bibliographic record is available under the Creative Commons CC0 public domain dedication. The University of Florida Libraries, as creator of this bibliographic record, has waived all rights to it worldwide under copyright law, including all related and neighboring rights, to the extent allowed by law.
Statement of Responsibility:
by Shaoyu Qi.
Thesis:
Thesis (Ph.D.)--University of Florida, 2013.
Local:
Adviser: Ho, Jeffrey Yih Chian.

Record Information

Source Institution:
UFRGP
Rights Management:
Applicable rights reserved.
Classification:
lcc - LD1780 2013
System ID:
UFE0045374:00001


This item has the following downloads:


Full Text

PAGE 1

IMAGERETARGETINGANDSMARTPROJECTORSBySHAOYUQIADISSERTATIONPRESENTEDTOTHEGRADUATESCHOOLOFTHEUNIVERSITYOFFLORIDAINPARTIALFULFILLMENTOFTHEREQUIREMENTSFORTHEDEGREEOFDOCTOROFPHILOSOPHYUNIVERSITYOFFLORIDA2013

PAGE 2

c2013ShaoyuQi 2

PAGE 3

Idedicatethisthesistomybelovedparentsandotherfamilymembers.Thankyouforallofyourconstantsupport,encouragementandlovethroughoutmylife. 3

PAGE 4

ACKNOWLEDGMENTS Iwouldliketoexpressmysincerethankstoeveryonesurroundsme.Theirhelp,supportandencouragementareindispensableformetonishmyPh.Dresearch.FirstofallIamgratefultomyadvisor,Dr.JeffreyHo.Iappreciateallhiscontributionsoftime,ideasandfundingtosupportmenishmyPhDdegree.Asmymentor,colleagueandfriend,Ilearnedfromhimnotonlyknowledgeandideasinmyeldofresearch,butmoreimportantly,thewayofdevelopingavagueideaintoamatureresearchtopicwithstabletheorybackgroundandvalidexperimentresults.Withouthisguidance,patienceandsincerehelp,itwouldneverbepossibleformetonishmyPh.Dwork.IwouldalsoliketothankprofessorBabaVemuri,professorJorgPeters,professorArirezaEntezariandprofessorWilliamHagerforservingasmycommitteemembers.Thankyouforallyourpreciousideas,commentsandsuggestionsduringmyproposalexamandnaldissertation.VeryspecialthankstotheDepartmentofComputerandInformationScienceandTechnology,UniversityofFlorida,forthenancial,academicandtechnicalsupport.AlsoIamverythankfultothetwograduateadvisorsinCISEdepartment,Mr.BowersandMrs.Crisman.Allyourtimeandeffortinhelpingmeoutintheseyearsarehighlyappreciated.Toallmylabmates,JasonChi,MohsenAli,MuhammadRushdi,ShahedNejhum,andManuSethi,thankyouforalltheideas,helpandjoysyoubroughttome.ToallmyfriendsinGainesville,thankyouforthetimeandfunyousharewithmealltheseyears.Lastly,Iwillgratefullyacknowledgemyparentsforalltheirloveandencouragement.DuringtheyearsIaminthepursuitofmyPhDdegreeaswellasallmylife,theyconstantlyactasmyrmbackground,andshowfullsupporttowhateverdecisionImade. 4

PAGE 5

TABLEOFCONTENTS page ACKNOWLEDGMENTS .................................. 4 LISTOFFIGURES ..................................... 7 ABSTRACT ......................................... 9 CHAPTER 1INTRODUCTION ................................... 11 2RELATEDWORKS .................................. 16 3SEAMSEGMENTCARVING ............................ 19 3.1SeamCarving ................................. 19 3.2ImageRetargetingInaIrregularShapedDomain .............. 19 3.3TheSeamSegmentCarvingMethod ..................... 21 3.3.1SeamSegments ............................ 22 3.3.2SeamSegmentExtraction ....................... 23 3.3.3Shape-awareSeamSegmentSelection ............... 25 3.3.4EnergyFunctionForSeamSegmentCarving ............ 28 4SHIFT-MAPBASEDIMAGERETARGETINGWITHDISPARITYADJUSTMENT 31 4.1Shift-MapAndImportanceFilteringForStereoImages ........... 31 4.1.1ImprovedImportanceFilteringForStereoImages .......... 33 4.1.1.1Pixelsaliencyandshift-mapgradient ........... 33 4.1.1.2Improvedimportanceltering ................ 33 4.1.2DisparityAdjustment .......................... 34 4.1.2.1Trimming ........................... 35 4.1.2.2Adjustmentfunction(dn)andadjustmentmap ...... 35 4.1.2.3Smoothingtheadjustmentmap ............... 37 5ASMARTPROJECTOR/CAMERASYSTEM ................... 40 5.1ASmartProjector/CameraSystem ...................... 40 5.2TheProjector-CameraDistortionModel ................... 41 5.3DetectingDisplayArea ............................. 42 5.3.1ProjectedPatterns ........................... 43 5.3.2DetectingDisplayAreasAndCornerPointsInCapturedImage .. 43 5.3.3FindingTheProjectiveTransformationMatrixH ........... 44 5.3.3.1Matchingcorrespondingfeaturepoints ........... 44 5.3.3.2FindingHfrompairsofcorrespondingcornerpoints ... 46 5.3.4DisplayAreaMAndSeamSegmentCarving ............ 46 5

PAGE 6

6EXPERIMENTALRESULTSANDDISCUSSIONS ................ 47 6.1ExperimentalResultsAndApplications ................... 47 6.1.1Applications ............................... 48 6.1.2Limitations ................................ 49 6.2StereoImageRetargetingWithDisparityAdjustment ............ 50 6.2.1StereoImageRetargeting ....................... 50 6.2.2DisparityAdjustment .......................... 53 6.3SmartProjector/CameraSystem ....................... 57 7CONCLUSIONS ................................... 59 REFERENCES ....................................... 60 BIOGRAPHICALSKETCH ................................ 64 6

PAGE 7

LISTOFFIGURES Figure page 1-1Introductiontoimageretargeting .......................... 12 1-2Imageretargetingwithirregularly-shapedboundary ............... 13 1-33Dimageretargetinganddisparityadjustment .................. 14 3-1Thepipelineofirregularshapeimageretargetingalgorithm. ........... 21 3-2Directionafliatedtoaseamsegment ....................... 22 3-3Boundaryconstructionandseamsegmentextraction ............... 23 3-4Exampleofboundaryconstruction ......................... 24 3-5Threetypesofcroppedareas ............................ 25 3-6Effectivenessoftheshape-awareseamsegmentselectionapproach ...... 26 4-1Outlineoftheproposedstereoimageretargetingalgorithm ........... 32 4-2Thecalculationofadjustmentfunctions ...................... 37 4-3Effectivenessofadjustment-mapsmoothing .................... 38 5-1Twocomponentsandworkowoftheproposedsmartprojector/camerasystem. 41 5-2Settingofthesmartprojector/camerasystemandthedistortionmodel ..... 42 5-3Projectedpatternsandcapturedimages ...................... 43 5-4Calculatingcorrespondences ............................ 44 5-5Illustrationofcornerpointsregistration ....................... 45 6-1Effectiveofseamsegmentcarving ......................... 47 6-2Comparisonofseamsegmentcarvingandotheralgorithms ........... 48 6-3Limitationofseamsegmentcarving ........................ 49 6-4Correctnessofourshift-mapalgorithm ....................... 51 6-5Comparisonofouralgorithmandcurrentstate-of-the-artalgorithm ....... 52 6-6Effectivenessofdisparityadjustment(1) ...................... 53 6-7Effectivenessofdisparityadjustment(2) ...................... 54 6-8Resultsofdisplayareadetection .......................... 55 7

PAGE 8

6-9EffectivenessofSmartProjection(1) ........................ 56 6-10EffectivenessofSmartProjection(2) ........................ 57 8

PAGE 9

AbstractofDissertationPresentedtotheGraduateSchooloftheUniversityofFloridainPartialFulllmentoftheRequirementsfortheDegreeofDoctorofPhilosophyIMAGERETARGETINGANDSMARTPROJECTORSByShaoyuQiAugust2013Chair:JeffreyHoMajor:ComputerEngineeringImageretargetingalgorithmsaimtoadjusttheimagetothedisplayscreenwhoseaspectratioorsizediffersignicantlyfromthatoftheimagebyapplyinganon-uniformandcontent-awareimageresizing.Comparedtouniformimagescaling,retargetingprovidesconsiderablybetterresultsintermsofitspreservationoftheimagescontentandintegrity.Duetorecentadvancesindisplaytechnology,imageretargetinghasemergedrecentlyasanactiveeldofresearchincomputervisionandgraphics.Inthisdissertation,Iwillpresenttheresultsfrommyinvestigationoftwonewtypessofimageretargetingproblemsandtheirsolutions.Specically,Iwillrstproposeanovelalgorithmthatcanretargetimagestodomainswithnon-rectangularboundary.Thisimprovementovertheexistingimageretargetingalgorithmsallowsagreaterexibilityinchoosingtheretargetedimagedomainandconsequently,itpermitsawiderrangeofapplications.Second,Iwillproposeanovelalgorithmthatcansimultaneouslyretarget3Dimagesandadjusttheirdisparityvalues,andinthecontextof3Dimageretargeting,Iwilldiscussanewviewpointthatconsidersthesimultaneousobjectivesofpreservingimagecontentaswellasmaximizingviewingcomfort.Inthelastpartofthedissertation,Iwillpresentthedesignandimplementationofasmartprojectorsystemthatincorporatesoneoftheproposedimageretargetingalgorithm.Forthesmartprojectorsystem,itcanautomaticallydeterminethedisplayareaandretargetthe 9

PAGE 10

displayimageaccordinglyinordertomaximizethequalityanddelityofthedisplayedimage. 10

PAGE 11

CHAPTER1INTRODUCTIONAsdigitalimagesarenowdisplayedonanincreasingvarietyofdisplaydevices,contest-awareimageretargetinghasattractedconsiderableamountofattentionintheresearchcommunityinthelastfewyears.Briey,imageretargetingattemptstoadjusttheimageinordertodisplaytheimageonadisplayscreen(area)thatissignicantlydifferentinsizeoraspectratio.Differentfromthepreviousimageresizingmethodsthatuniformlyscaleanimage,currentstate-of-the-artimageretargetingmethodsresizetheimageinanon-uniformwaywiththeexplicitaimofpreservingtheintegrityandstructureofsalientregionsintheimageandminimizingtheirdistortions.Figure1-1displaysseveralexamplesofimageretargeting,andtheyshowthattheretargetingalgorithmisabletoautomaticallydeterminethesalientareasintheimages,typicalforegroundobjectsfoundintheimagessuchaspeople,animals,cars,etc.andthenon-salientareasthatusuallyformtheimagebackgroundsliketheskyandmeadow.Thealgorithmappropriatesdifferentamountofscalingtothesalientandnon-salientregions,withmostresizingconcentratedinthenon-salientbackgroundareas.Currently,imageretargetinghasbeenstudiedmostlyundertheassumptionthatthedisplayscreensarerectangularandthedisplayedimagesaretheusualplanarRGBimages.Unfortunately,thesetwoassumptionsseverelylimitthecapacityandexibilityofexistingimageretargetingalgorithmsandrenderthemcumbersomeifnotentirelyinappropriateformanyanticipatedfutureapplications.Inparticular,twonoteworthyrecentemergingtrendsintechnologyhavegivenusaglimpseofpotentialfutureapplicationsthatmakeexistingimageretargetingalgorithmsseeminadequateandinsufcient.First,theideaofubiquitousdisplayhasprovidedagreatimpetusforthedevelopmentandpopularizationofminiatureprojectors,devicesthatallowimagestobedisplayedandshownonalmostlimitlessvarietyofdisplayareas.Inthismuchbroaderapplicationcontext,thesizeofthedisplayareaisnolongerthesole 11

PAGE 12

OriginalImageResizedImageRetargetedImageABCFigure1-1. Introductiontoimageretargeting.A)Originalimage.ThedisplaydomainhasdifferentaspectratioandB)uniformscalingoftheimageproducesresultthatmakesitdifculttoviewtheforegroundsalientobject.Incomparison,C)retargetedimageprovidesaconsiderableimprovementandclarityinpresentingtheforegroundsalientobjectintheimage.(Imagesbetterviewedincolor)(PicturescourtesyofMikiRubinstain[ 24 ].) variablethatdeterminesthequalityofthedisplayedimage,whichcanbeaffectedby,amongmanyotherfactors,theshapeofthedisplayarea.Inparticular,forthesimplestextensionwithnon-rectangularplanardisplaydomain,noneofthecurrentexistingimageretargetingalgorithmsiscapableofretargetingimagestosuchdomain.Second,therecentproliferationof3Dcontentsinentertainmentandgamingindustryhassuggestedadigitalfuturedominatedby3Dimagesand3Dviewingexperience.Withoutdoubt,therealizationofsuchfuturisticvisionwouldrequirethedevelopmentofnewalgorithmsandtechniquesthatcanprocessanddisplay3Dimagesefcientlyandinformativelyinwaysthatmaximizetheviewingexperience.Inparticular,retargeting3Dimagesprovidesanovelandinterestingdirectionforsubstantiallyextendingcurrenttechnologicalboundaryforimageretargeting.Inthisdissertation,Iwillinvestigatenewimageretargetingalgorithmsthataredesignedforthetwotypesofanticipatedfutureapplicationsmentionedabove.Specically,Iwillpresenttwonewimageretargetingalgorithmsthat,respectively,1)retargetsimagestonon-rectangular(planar)domainsand2)retargets3Dimages: 12

PAGE 13

ImageRetargetingwithnon-rectangulardomain[ 21 ].Althoughimage OriginalImageNewBoundaryCroppedImageMyResultABCDFigure1-2. ImageRetargetingwithIrregularly-shapedboundary.A)TheoriginalimageandB)aheart-shapedtargetdomain(highlightedinred).ComparedtothedirectcroppingshowninC),theretargetedimageD)preservesmorerelevantimagedetailssuchasthebeach,pebbles/stones,wholehumanbodies,etc.Theproposedalgorithmwillallowphotoeditorsandartisticusersmorefreedomandexibilityinchoosingtargetedimagedomains.(Imagesbetterviewedincolor)(PicturescourtesyofMicrosoftResearch.) retargetingisarelativelynascentareaofresearch,itissomewhatsurprisingthattheproblemofretargetingimagestonon-rectangulardomainhasnotbeenstudiedandreportedintheliterature.Inadditiontotheaforementionedapplicationusingpicoprojectors,imageretargetingwithnon-rectangulardomainalsohavemanyinterestingapplicationssuchasphotoediting.InmydissertationIwilldevelopanefcientalgorithmthatcanautomaticallyretargeta(typical)rectangularimagetoshapeswithirregularboundaries.Comparedwithcurrentalgorithms,theexibilityandversatilityofthenovelalgorithmisclearlydemonstratedinFigure1-1asitprovidesgreaterdegreesoffreedom,choicesandcontrolsfortheusers StereoImageRetargeting[ 22 ].Thesecondtechnicalcontributionofmydissertationisanefcientandaccuratealgorithmforretargeting3Dimages.3Dimagesareusuallygivenasapairofstereoimagesandthedisparityvaluescomputedfromthetwoimages.Mystartingpointforinvestigating3Dimageretargetingisthenovelideaandviewpointthatthemainaimof3Dimageretargetingshouldbeaboutmaximizing 13

PAGE 14

ABC DEFFigure1-3. 3DimageretargetingandDisparityAdjustment.Fromlefttoright:A,DOriginalstereoimagepairs,B,ERetargetedimagepairs,C,FAnaglyphbeforeandafterdisparityadjustment.(betterviewingincolor)(PicturescourtesyofTaliBasha[ 4 ].) viewingexperience.ThisnovelviewpointisindirectoppositeofBashaetal.[ 4 ]thatadvocatesthepreservationof3Dgeometryintermsofdepthvaluesasthemainpriority.Inparticular,thegoalfor3Dimageretargetingshouldbelessabouttheprecisionoftheactualgeometricdetailsbutmoreabouttheirperception,andthelattercanbeaffectedbyvariousfactorsincludingbasichumanperceptionsandpsychologicalresponses.Extensiveworkshavebeendoneonevaluatingvariousdifferentfactorsthatcanaffecthumanperceptionof3Dimages,e.g.,[ 12 ],andtheyhaveshownthatthereisacertainrangeofdisparityvalues,thecomfortzone,thatmaximizestheacuityofhumanperception.Mygoalistodesignandimplementanimageretargetingalgorithmthatcansimultaneouslyretargetthe3Dimagesandreadjustthedisparityvaluesintheretargetedimageswiththeexplicitaimofmaximizingviewingexperienceandcomfort. 14

PAGE 15

Theaforementionedtwoalgorithmswillconstitutethemaintechnicalcontributionsofmydissertation.Inthelastpartofmydissertation,Iwillpresentanovelapplicationofimageretargetingintheimplementationofasmartprojectorsystem.Asmartcamera/projectorsystemisimportantforubiquitousdisplay,afuturistictechnologythathasstartedgainingpopularityamongresearchersandtechno-enthusiasts.Inparticular,theprevalenceofpicoprojectors,therecently-developedhighlycompactLED-drivenprojectors,hasgreatlyexpandedtheapplicationscopeoftheprojectors.Comparedtothetraditionalprojectors,picoprojectorsareabletoprojectpictures,videosorslidesonvariouskindsofdisplaysurfaces,andtakingusafewstepcloserinrealizingthefuturisticvisionofubiquitousdisplay.Animportanttechnicalproblemistoprovideasmuchdelityforthedisplayedimagesgivenapotentiallyless-than-idealdisplaysurface,anditiswhereimageretargetingbecomesusefulandindispensable.Specically,thesmartprojectorsystemincludesaprojector,animagecapturingdevicesuchasacameraorwebcam,andacomputer.Capturedimageservedasthefeedbackfordetectingthedisplayarea,andoncethedisplayareaisdetermine,theimageretargetingisappliedtoretargettheimagebeforetheimageisdisplayedbytheprojector.Theremainingsectionsofthisproposalarestructuredasfollows.InChapter 2 ,relatedworkwillbesurveyed.Chapters 3 and 4 presentthedetailsofthetwoimageretargetingalgorithms.ThedetailofthedesignandimplementationofthesmartprojectorsystemwillbepresentedinChapter 5 .Chapter 6 containstheexperimentalresultsandevaluations,andtheconcludingchapterpresentstheplanforfuturework. 15

PAGE 16

CHAPTER2RELATEDWORKSImage/Videoretergetingalgorithmshasattractedconsiderableamountofattentionrecentlyinthevisioncommunity,duetotheabilitytopreservetheimportantareasandstructuresofthesalientobjects.Earlyretargetingwork,suchas[ 32 ][ 29 ],aimtopreserveimportantandsalientobjectsintheimagebysegmentation,croppingandpasting.Whilelayingthegroundworkforfuturedevelopment,theeffectivenessofthesemethods,particularlyinanautomatedandunsupervisedsetting,issomewhatlimitedduetotheirsimplicity.Forinstance,importantnotionssuchaspreservingsceneconsistencyaremoredifculttomanageusingoperationssuchascroppingandpastingthatusuallyaltertheimageinanon-incrementalway.Seamcarvingintroducedin[ 2 ]isperhapstherstimageretargetingmethodtohavegainedwidepopularity.Themethodmethodofseamcarvingincrementallyalterstheimagebycarvingawayseams,sequencesoflowenergypixelsrunningverticallyacrosstheimage.Itssimplicityallowsmanyextensionsindifferentways.Rubinsteinetal.appliesseamcarvingtovideoretargeting[ 25 ],andfurtherincorporateseamcarvingwithcroppingandhomogeneousresizingtodevelopamulti-operatorimageretargetingalgorithm[ 26 ].Furtherextensionsoftheseamcarvingalgorithmincludeseamcarvingwithgraph-cut-basedoptimization[ 10 ],algorithmusingdiscontinuousseams[ 8 ],andmorespecicapplicationsofimageretargetingsuchasbuildingthumbnailimages[ 35 ].Imagewarpinghasbeenappliedtoimageretargetingandithasgeneratedawholenewfamilyofretargetingalgorithms.Theideaofimagewarpingandretargetingistoachieveanoptimaldeformationoftheimagethatpreservessalientregionsaccordingtoamashbuiltontopoftheimage,theverticesofwhichareacquiredbycomputingtheglobalminimaofaquadraticenergyfunction.Thedeformationcanbecomputedusingaquadmesh[ 12 37 39 ],oratrianglemesh[ 9 ].Theshift-mapmethod,rstintroducedbyPritch,etal.[ 20 ],aimstomanipulatevaluesinthe'shift-map'torearrangetheimage 16

PAGE 17

content,andthisparticularfeaturemakesitideallysuitedforimageretargeting[ 13 20 ].Veryrecently,Ding,etal.[ 6 ]proposean'importancelter'tocomputetheshift-mapbyintegratingshift-mapgradientacrossscanline.Intermsofrunningtime,theimportancelterprovidesasignicantimprovementoverpreviousgraph-cut-basedmethodswithoutnoticeabledegradationinoutputimagequality.Thefamilyofseamcarvingmethodsdecreasetheimagesizebydirectlyremovingpixels.Underthesettingofseams,itiseasytogetridofthetwistedunimportantareas.Moreover,theideaofseamcarvingiseasytounderstandandimplement,anditsresultsarelessreplying(evennotreplyingforsomeinputimages)onthesaliencymap.Comparedtoseamcarving,thetwointerpolationbasedmethods,warpingandshift-map,areabletoprovidesmootherresults.Butthequalityoftheirresultshighlydependsonthesaliencymap.Moreover,formostofthealgorithmsinthisfamily,itmightbeexpensiveintimeifadensemashisapplied.Imageretargetingwithdepththedepthinformationincorporatedisanothertypeofimageretargetingthathasbeeninvestigatedpreviouslyintheliterature.Manseld,etal.[ 17 ]introducethe'scenecarving'algorithmthatusesaroughdepthmapaslayerlabelsforobjectsintheimage.Tothebestofmyknowedge,therstworkonstereoimageretargetingbyUtsugi,etal.,[ 36 ].Withthehelpofadensedisparitymap,theiralgorithmisabletoretargetapairofstereoimages.Basha,etal.furtherdevelopamorecompleteandadvancedstereoimageretargetingalgorithm[ 4 ],withtheabilitytoretaingeometricconsistency.Andnally,inthecontextofstereoimageprocessing,[ 14 ]usessparsefeaturematching,linear/nonlineardisparitymappingandmesh-basedwarpingtoadjustthedisparityvalueforbetterviewingexperience.However,theirmethoddoesnotretargetstereoimages.Duringrecentyearsseveraldescentresearchworkshavebeenproposedontheprojector/camerahybridsystemforenhancedviewingeffect.Colorcorrectionmethodslike[ 33 ],[ 27 ],[ 16 ]trytocorrectcolordistortioncausedbythecolorsonthedisplay 17

PAGE 18

area,byndingthetransformationofthecolorspacebetweentheoriginalimageandtheprojectedimage,orusingthecapturedimageasfeedbacktocompensatethenon-whitebackground.Geometrydistortioncompensationtriestowarptheimagebeforeprojectiontocorrectdistortionscausedbyprojectors.Therstsystemisthe'iLamps'fromRaskar,et,al[ 23 ],whichappliesanapproximated3Dreconstructionandmapthetextures.DualPhotography[ 31 ]describestherelationshipofthetwoscenesviewedattheprojectorandcameraside.InverseLightTransport[ 30 ]providesamodelonhowapointlightwouldbouncebetweensurfaces.Basedonthesetheories,Ng,et.al[ 19 ]andDing,et.al.[ 5 ]developedalgorithmstocorrectgeometrydistortionbyapplyinganimagetransformationontheoriginalimageaccordingtoatransformmatrixM.Afterprojectingthetransformedimageinsteadoftheoriginalone,distortioncausedbyirregulargeometryofthedisplayareacouldbecorrected.Thelastfamilyofproblemistodoautomaticimageregistrationandstitchingformulti-projectorsystems,anexampleisshownin[ 28 ].Allofthepreviousresearchworksprovidepreciousideasandguidestomydevelopmentofasmartprojector/camerasystem. 18

PAGE 19

CHAPTER3SEAMSEGMENTCARVING 3.1SeamCarvingTheschemeofseamcarvingisrstdescribedin[ 2 ].Apartfromtheothertwofamilyofimageretargetingmethods,seamcarvingdecreasesthesizeofimagebyiterativelyremovingpixels.SupposethesizeofanimageIisMH,aseamSisasetofHinterconnectedpixels,s1,s2,s3,,sH,withonlyonepixelineachrow,andowingfromtoptobottom.Afterremoving(W)]TJ /F5 11.955 Tf 12.3 0 Td[(W0)seamsitispossibletoacquiretheimagewiththedesiredsize.Generallyspeaking,adesiredseamissupposedtopasslessimportantpartsoftheimage,likethesky,thesea,orotherareaswithlittleinformation.Toquantizetheamountoflossofinformationandstructurebyremovingacertainseam,thesumofenergyvalueofallthepixelsincludedinthisseamisthencalculated.Whenaseamistobecalculated,eachpixelisassignedanenergyvalueshowingthepenaltyofremovingthatcertainpixel.Theenergytermcouldbeaweightedcombinationofthefollowing:asaliencymapfromsomehigh/lowlevelcomputervisionalgorithms,the'forwardenergy'[ 25 ]consideringtheeffectofdeletingapairofpixelswithintwoneighboringrows,orsomeuser-denedtermsormapstohighlightaspeciedregion,etc.Duringeachloop,thecalculatedseamistheonewiththelowestsumofenergyamongallpossibleseams.Dynamicprogrammingisappliedtosolvetheseamcarvingproblembyndingoneseamatatime. 3.2ImageRetargetingInaIrregularShapedDomainIrregularshapeimageretargetingisamorechallengingproblemcomparedtoformerrectangularimageretargeting.1First,thenewalgorithmshouldbeexibleenoughtoreshapetheoriginalimagetovariouskindsofshapes.Secondandmore 1Thischapterhasbeenpublishedin[ 21 ]. 19

PAGE 20

important,inthecontextofrectangularimageretargeting,theamountofsizedecreaseineachrowisidentical,thusitispossibletosearchthewholeimagetodetectandgetridofthelesssalientareas.However,theamountofdeformationwillvaryinlocationsincaseofirregularshapeimageretargeting.Thisinconsistencywillbringaboutadditionaldistortionsatthoserowsaffectedbynon-identicalsizedecrease,thusadditionalconstraintsshouldbeintroduced(section 3.3.4 ).Withtheseinmind,thequalityofanirregularshaperetargetedimagecouldbejustiedinthefollowingtwoaspects,fromwhichitiscleartoseethattrivialmethods,likecropping,arenotsufcienttohandlethisproblem. 1. Thesalientcontentsandstructuresinthenon-croppedareasshouldbemostlypreserved. 2. Theobjectsinthecroppedareasshouldberetrievedwiththeleastamountofdistortionofthenearbysalientobjects.Andmoreimportant,theresultshouldbevisuallyacceptable.Threefamiliesofapproaches,seamcarving(e.g.[ 25 ]),warping(e.g.[ 38 ])andshift-mapediting(e.g.[ 6 ]),havebeensuccessfullydevelopedintheliteratureofimageretargeting.Warpingmethodsrearrangetheimagebywarpingtheimageaccordingtoanoptimizedmesh.Unfortunatelycurvyboundariesarehardtosimulatewithasparsemesh,whileusingadensemeshwillsignicantlyincreasetherunningtime.Currentshift-mapalgorithmcouldonlymovethepixelsalongonedimension,whichisproveninsufcientincaseofreshaping(section 3.3.3 ).Onthecontrary,underthesettingofseamremoval,seamcarvingapproachnotonlyprovidesastraightforwardwaytochangetheshape,butalsoholdsthecapabilityofutilizingtinyortwistednon-salientareas,whichisextremelydesiredinthecontextofirregularshapeimageretargeting.InthispaperIhaveproposedanovelseamsegmentcarvingmethodforirregularshapeimageretargeting,whichisproventoprovideefcientsolutionsforreshapinganimagetoavarietyofshapes[ 21 ].Ialsodevelopedanewapplication,theadaptivecamera-projectorsystem,toautomaticallydetecttheshapeofthedisplayarea,and 20

PAGE 21

Figure3-1. Thepipelineofirregularshapeimageretargetingalgorithm. retargetandprojecttheimagewithrespecttothenewshape.Finally,Iconcludetheintroductionwithasummaryofthetwospeciccontributionsofthispaper: 1. Amoregeneralformofseamsthan[ 2 ],calledtheseamsegment,isintroduced.Seamsegmentisalsoaconnectedlowenergypath,butitisnotrequiredtorunfromtherstrow/columntothelast.Thenewshapecouldbecomposedbycarvingasequenceoftheseamsegmentsiteratively(section 3.3.1 ). 2. Myalgorithmcarvebothhorizontalandverticalseamsegmentssimultaneouslytoadaptdifferentshapes.Anautomaticshape-awareseamsegmentselectionapproach(section 3.3.3 )isdevelopedforbetterresultquality. 3.3TheSeamSegmentCarvingMethodTheinputsofmymethodareanimageIwithheightHandwidthW,andamaskMBoundtohighlightthenewshape.TheoutputisaretargetedimageI`withtheshapedenedinMBound.TheoverallpipelineofmymethodisshowninFig. 3-1 .Therststepistocarveallthethroughseamstottheboundingboxofthenewshape,ifnecessary.Thenoneoptimalseamsegmentisselectedandcarvedfromalltheextractedseamsegmentsineachloop,untiltheretargetedimagehasthedesiredshape.Atlasttheretargetedimageispastedbacktottheoriginalmask.Justnotethatalthoughthethroughseamscouldalsoberegardedasseamsegments,hereIchoosetotaketurnstocarvethehorizontal/verticalcuttingthroughseamsbeforestartingtoprocess 21

PAGE 22

Figure3-2. Directionafliatedtoaseamsegment.Eachseamsegmentisafliatedwithonedirectionindicatingthesetofpixelstobeshifted.A)Ina5x5image,aseamsegmenthasbeencalculated(labelledin'X').B)(upper)Iftheseamsegmentistoformtheleftboundary,thepixelsonitsleft(inpurple)aremovedrightwardsfor1pixel.(lower)Thecorrespondentpixelsareshiftedleftwardsincaseoftheconstructionoftherightboundary. theseamsegments,sothatabalanceofsizereductioninbothdimensionscouldbeachieved. 3.3.1SeamSegmentsTheadvantageofusingseamsegmentsisthattheshapeoftheboundarycouldbesimplymodiedbyndingandcarvingasequenceofseamsegments,andshiftthecorrespondingpixels.Fig. 3-3 showsthewaytoformaspecialleftboundarybycarvingtwoseamsegments.Oncealltheseamsegmentsarecarved,theleftboundarywillhavethedesiredshape.Theotherthreeboundariescanbereshapedsimilarlybyndingtheproperseamsegmentsandshiftthecorrespondentpixelsaccordingly.Similartotheformersettingofseams[ 2 ],aseamsegmentSisalsoasetofinterconnectedpixels,s1,s2,s3,,sn,withonlyonepixelineachrow(verticalseamsegment)oreachcolumn(horizontalseamsegment),butnotnecessarilystartsandendsbytheimageboundary.Bothacutting-throughseamandanisolatepixelcouldberegardedasoneseamsegment.Moreover,sincenotalltherows/columnsareaffectedduringseamsegmentremoval,differentdirectionsofpixelmovementwillsurelybringaboutdifferentresults(seeFig. 3-2 ). 22

PAGE 23

Figure3-3. Boundaryconstructionandseamsegmentextraction.A)Theredlinehighlightsthedesirednewboundary.B)(upper)Findtherstseamsegment(labelledin'X'),and(lower)carveoutthatseamsegmentbyshiftingallthepurplepixelstotheright.C)Afterndingandcarvingthesecondseamsegment,thenewboundaryissuccessfullyformed.AnexampleoftheauxiliaryarrayALforseamextractionisalsoshowninthisgure. InthefollowingsectionsIwillbeusingthetermSX,n(st,ed)todenoteacertainseamsegment.Xdenesthedirectionafliatedtotheseamsegment,whosevaluecouldbeeitherL(eft)/R(ight)(verticalseamsegments),orT(op)/B(ottom)(horizontalseamsegments).nisaserialnumbertolabelacertainseamsegmentanddepicttheorderofseamsegmentremoval.(st,ed)showsthestartingandendingrow/columnofthisseamsegment.Forexample,SL,5(4,7)denotesthe5thseamsegmentstartingatthe4throwandendingatthe7throw,aimingtobuildtheleftboundary.SL,n(st,st)indicatesthatthisseamsegmentcontainsonly1pixelinrowst,andSL,n(1,H)denesaverticalcutting-throughseam. 3.3.2SeamSegmentExtractionSeamsegmentextractiontendstondthestartingandendingrows/columns(st,ed)beforeaseamsegmentisactuallycalculated.Thepixelsofaseamsegmentwilllaterbesearchedonlywithintheserows/columns. 23

PAGE 24

ABCD EFGHOriginalImage&180SeamSegments360SeamSegmentsAll543SeamDirectCroppingCarvedCarvedSegmentsCarvedFigure3-4. Exampleofboundaryconstruction.A-DOriginalimagewithnewboundaryhighlighted,rst180seamsegments(green,blue,yellowandvioletseamsegmentsareusedforbuildingtheleft,right,topandbottomboundary),rst360seamsegments,all543seamsegments.E-HResultfromdirectcropping,resultbycarvingtherst180seamsegments,resultbycarvingtherst360seamsegments,nalresult.Comparedtodirectcroppingmymethodisabletopreservemorecontentsoftheislands.likethesandybeach.(betterviewingincolor)(PicturescourtesyofMikiRubinstain[ 24 ].) ToefcientlyextracttheseamsegmentsIuseanauxiliaryarrayAX,whereXdenotesthecorrespondentboundary.Withoutlostofgeneralityhereonlythewaytoextractseamsegmentsfortheleftboundaryisillustrated.TheauxiliaryarrayALisa1Hvector,andAL(j)denotesthenumberofpixelsremainedtobecarvedinrowj(Fig. 3-3 (B,C)).OnceALisconstructed,allseamsegmentscouldbeextractedbytheleftboundaryasfollows:ForagivenassistancearrayAL,scanfromAL(1)toAL(H)tryingtondalltheintervals[st,ed]thatAL(j)>0,8j2[st,ed],andsubtractAL(j)by1forallcorrespondingrowsuntilALisall0. 24

PAGE 25

Figure3-5. A)Threetypesofcroppedareas.Type(1)areacanbecroppedusingonly1typeofseamsegment,while2-3typesofseamsegmentscouldbeusedtoconstructthenewboundaryalongtype(2)area.Type(3)areacroppingisnotapplicableundermysetting.B)Boundariesbytype(2)areacanbeconstructedbycarvingrandomcombinationofvertical(orange)andhorizontal(green)seamsegments. 3.3.3Shape-awareSeamSegmentSelectionFig. 3-5 (A)concludesthedifferenttypesofboundariesmyseamsegmentcarvingalgorithmissupposedtohandle.InFig. 3-5 (A)arectangularimageistobereshapedaccordingtoagivenmask.Thecolouredareasaretothosebecarved,andthewhiteareasdepictsthenalshape.Aftercarefullyinspectingthecroppedareaitispossibletodistinguishthemintothreetypes,ashighlightedinthegure.Byshiftingverticallyorhorizontallythepixelsintype(1)areacouldreachonly1boundary,whilethepixelsintype(2)areacouldreachmultipleboundaries.Thisindicatesthenewboundariesalongtype(1)areawillbeconstructedbycarvingoutonecertaintypeofseamsegment(eitherL(eft),R(ight),T(op)orB(ottom),asmentionedinsection3),whilemultipletypesofseamsegmentscanbeutilizedtofoundtheboundariesalongtype(2)area.Type(3)areapixelsareregardedtobeinnerpixelswithoutanystraightpathstotheboundary,whichindicatesthatboundariesbytype(3)areacouldnotbecomposedbyshiftingpixelsinwardsundermycurrentsettings.InfactIactuallytriedtocarveaholebyndingseamsegmentsandmovepixelstowardtheboundary.Unfortunatelyinmost 25

PAGE 26

ABCD EFGHOriginalImage&AllHorizontalAllVerticalShape-awareSeamDirectCroppingSeamSegmentsSeamSegmentsSegmentSelectionFigure3-6. Effectivenessoftheshape-awareseamsegmentselectionapproach.A-DOriginalimagewithnewboundaryhighlighted,seamsegmentswhenonlyhorizontalseamsegmentsareused,seamsegmentswhenonlyverticalseamsegmentsareused,seamsegmentswithautomaticseamselection(green,blue,yellow,violetseamsegmentsdenotethefourtypeofseamsegments,L,R,T,B,respectively).E-HResultfromdirectcropping,resultfromndingandcarvingonlyhorizontalseamsegments,resultfromndingandcarvingonlyverticalseamsegments,resultfromshape-awareseamsegmentselection.Onlyintheretargetedimagewiththeseamsegmentselectionapproachthetreebranchandreectionofthebridgeisnotdistorted.(PicturescourtesyofMikiRubinstain[ 24 ].) ofthecasessalientobjectsresideinthemiddleoftheimage,thusitisnotworthwhiletoretrievesomeinformationbutdistortsalientobjects/areas.Idon'tconsidersplittinganimageintomultiplepiecesduetosimilarreasons.Aninterestingfactfortype(2)areaisthatitsboundarycouldbeachievedbycarvingarandomcombinationofverticalandhorizontalseamsegments,asshowninFig. 3-5 (B).Itistrivialtoseethatbycarvingseamsegmentsindifferentordersthenalresultswouldnotbeidentical.Fig. 3-6 (B,C)showstwofailureexamplesfromsearchingandcarvingverticalorhorizontalseamsegmentsonly.Inthesetwoexamples,althoughbothofthemareabletoretrievecontentsoutsidethenewboundary,visibledistortionsoccur,liketheshadowsofthebridgeandtreebranchesininbothofthem. 26

PAGE 27

AstotheexampleinFig. 3-5 (B),usinghorizontalseamsegmentsarebetterfortworeasons.First,pixelsofaseamsegmentaresearchedonlywithinthecorrespondingrows/columns.Sincethetotalnumberofpixelstobecroppedisthesame,longerseamsegmentsallowustosearchforlowenergypixelsinabroaderarea,thuslowertheamountofdeformationrateandinformationloss.Second,seamsegmentcarvingalgorithmshiftsthecurrentboundariesinwardstomatchthenewboundaries.Inthisexample,duetothelargeslopedifferencebetweentheleftverticalboundaryandthenewboundary,eventhoughanobjectinsidethecroppedareacouldberetrievedbyverticalseamsegments,itsstructureishighlyunlikelytopreserve.Theattempttondtheoptimalcarvingorderishardandunrealistic.Inan400300imagewitharound20%ofcroppedareasusuallyover500seamsegmentsarecarved,andinmostofthecasesthenumberofchoiceofthenextseamsegmenttobecarvedisover6,whichmakesthebranchandboundsearchingtreeextremelyhuge,thusbothtimeandspacecomplexitytondthesolutionarehigh.Moreover,itisdifculttoquantizethequalityoftheretargetedimage.Basedonthediscussionsabove,inthispaperIdevelopedagreedybasedapproachtowiselyselectthenextseamsegment:theshape-awareseamsegmentselectionapproach.Atthebeginningofeachiterationfourboundariesarescannedandallthepossibleseamsegmentsareextractedtocarve.Amongallthepossibleseamsegments,itisalwaysthelongestone2tobeselected,calculatedandcarved.Thisapproachisproventobringaboutsmootherandlessdistortedresultswithouttheneedtopayspecialattentiontotheboundaryshape.AsshowninFig. 3-6 (D),mynovelshape-awareseamsegmentselectionapproachretrievesobjectsoutsidethenewboundary,e.g.branchandbank,aswillaspreservestheoverallstructuralconsistency. 2Herethe'longest'seamsegmentindicatestheonewiththelargest(ed)]TJ /F5 11.955 Tf 11.95 0 Td[(st)value. 27

PAGE 28

3.3.4EnergyFunctionForSeamSegmentCarvingSimilartotheotherseamcarvingmethodslike[ 2 ],inseamsegmentcarvingthecarvedseamsegmentisalsotheonewiththeleastsumofenergy.ThewaytocalculatetheenergyfortheseamsegmentSL,nisillustratedin( 3 ).InthefollowingillustrationsIwilluseSL,ninsteadofthefullformSL,n(st,ed)tokeeptheequationshortandtidy. Etotal(SL,n)=Eseam(SL,n)+Epos(SL,n)(3)TheenergyfunctionforndingtheseamsegmentSL,niscomposedoftwoparts,EseamandEpos.Eseam(SL,n)isthewell-denedenergytermusedforseamcarvingin[ 25 ],whichconsistsoftheforwardenergytermsEvandEh,andEposisaimedspeciallyforthepositionsofthetwoterminationsofaseamsegment.isabalancefactor,andissetto0.6inmyexperiments.Ifasaliencymapisuseditwouldthenbeabletoadditsvalueatonepixeltimesabalancefactorto( 3 ).Eseamiscalculatedinthefollowingmanner: Eseam(SL,n)=X(i,j)2SL,nEh(i,j)+X(i,j),(i+,j+1)2SL,nEv(i,i+,j)(3)ThehorizontalenergyEhofremovingapixel(i,j)is: Eh(i,j)=jI(i+1,j))]TJ /F5 11.955 Tf 11.96 0 Td[(I(i)]TJ /F4 11.955 Tf 11.96 0 Td[(1,j)j(3)Intheverticaldirectiontheenergyvalueisrelatednotonlytothepixel(i,j)butalsotheselectedpixelinrowj+1.Supposethepixel(i+,j+1)istoberemoved,sinceinmymethodIconstraintheseamsegmentstobecontinuous,theverticalforwardenergycouldbecomputedby: Ev(i,i+,j)=8>>>><>>>>:jI(i)]TJ /F4 11.955 Tf 11.95 0 Td[(1,j))]TJ /F5 11.955 Tf 11.95 0 Td[(I(i,j+1)ji+=i)]TJ /F4 11.955 Tf 11.95 0 Td[(10i+=ijI(i+1,j))]TJ /F5 11.955 Tf 11.95 0 Td[(I(i,j+1)ji+=i+1(3) 28

PAGE 29

Eposin( 3 )ismynovel'positionenergy'fortheseamsegments.Thereasonforaddingthistermisthatunlikethecutting-throughseamswhichremoveexactly1pixelperrow,theseamsegmentsSL,i(st,ed)onlydeletepixelsbetweenthestthandedthrow.Inthiscase,additionaldistortionswilloccurbetweentherow(st)]TJ /F4 11.955 Tf 12.37 0 Td[(1)andst,andalsobetweentherowedand(ed+1).Thisisbecausethepixelslefttotheseamsegmentinstthandedthrowwillmoveleftwardsfor1pixel,butthere'snopixelmovementinrow(st)]TJ /F4 11.955 Tf 12.52 0 Td[(1)and(ed+1).Itistheinconsistencyinpixelmovementthatbringsaboutadditionaldistortions,whichIcalled'positiondistortion'inthispaper.Moreover,thefartheraseamsegmentisfromtheboundary,themoreseverepositiondistortionitmightbringaboutsincemorepixelswouldbeaffectedbyinconsistentpixelmovement.Withthisunderstandinginmind,Idenethepositionenergyasfollows:(assumingthestartingpointisoftheseamsegmentis(ist,st)and(ied,ed)) Epos(SL,n(st,ed))=Ehead(ist,st)+Etail(ied,ed)(3)whereEheadandEtailaredenedbelow:iftherstcarvedpointisnotlyingontheleftboundary(ist=1),orthestartingrowisnotthetoprowst6=1,then Ehead(ist,st)=ist)]TJ /F9 7.97 Tf 6.59 0 Td[(1Xi=1jI(i,st))]TJ /F5 11.955 Tf 11.96 0 Td[(I(i+1,st)]TJ /F4 11.955 Tf 11.95 0 Td[(1)j(3)otherwiseEheadis0.Similarly,ifaseamsegmentdoesnotterminateattheleftboundaryorthebottomrow,thepositionenergyEtailis: Etail(ied,ed)=ied)]TJ /F9 7.97 Tf 6.59 0 Td[(1Xi=1jI(i,ed))]TJ /F5 11.955 Tf 11.96 0 Td[(I(i+1,ed+1)j(3)Etailissetto0ified=1ored=H.FromthissettingIcanseethatifaseamstartsfromthetopandendsinthebottom,thenEposis0,andmymethodreducestotheformerseamcarvingmethod.Similarly,iftheseamsegmentcontainsonly1pixel,thenEvis0,whichistondonepixelatstthrowthatthedistortioncausedbyremovingthatcertainpixelistheleast. 29

PAGE 30

Withtheenergyfunctionmentionedabove( 3 )theseamsegmentcanbecalculatedefcientlyusingdynamicprogramming.Theseamsegmentsforbuildingtheotherthreeboundariescouldbecalculatedsimilarly. 30

PAGE 31

CHAPTER4SHIFT-MAPBASEDIMAGERETARGETINGWITHDISPARITYADJUSTMENT 4.1Shift-MapAndImportanceFilteringForStereoImagesTheshift-mapwasrstintroducedin[ 20 ]andIreferthereadertotheoriginalpaperformorespecicdetails.Theshift-mapS(x,y)isconsideredasaWHreal-valuedmatrix,elementsofwhichdenethelocationsofthepixelsafterretargeting.S(x,y)essentiallyprovidesatransformation(deformation)betweenWHandW`H-imagegrids,andtheretargetedimageisthengeneratedbyimageinterpolationaccordingtothe2DdeformationgivenbyS(x,y).TherearetwobasicconstraintsforS(x,y):boundaryconstraintS(1,y)=1andS(W,y)=W`andmonotonicityconstraintthatS(x,y)6S(x+1,y)formaintainingthescanlineorder.FurtherconstraintscanbereadilyformulatedforS(x,y).Forinstance,suppose(xL,y)and(xR,y)areapairofcorrespondingpixelsfromapairofrectiedstereoimageswithdisparityvalued=xL)]TJ /F5 11.955 Tf 12.09 0 Td[(xR.ThisparticulardisparityvaluecanbepreservedbyenforcingtheconstraintSL(xL,y))]TJ /F5 11.955 Tf 12.61 0 Td[(SR(xR,y)=d=xL)]TJ /F5 11.955 Tf 12.61 0 Td[(xR.Inactualcomputation,itisthegradientsoftheshift-mapsthatarecomputeddirectlyfromimagedataandthenalshift-mapisobtainedbyintegratingthegradientusingimportanceltering[ 6 ].Anewalgorithmissuccessfullyproposedforbothstereoimageretargetinganddisparityadjustment[ 22 ].1Threemainnovelfeaturesofmyalgorithmareillustratedasfollows: 1. Ihaveincorporatedtwodifferenttasks,stereoimageretargetinganddisparityadjustment,intooneframework.Mynalshift-mapiscapableofbothretargetinganddisparityadjusting,andiseasilycomputedbyaddingtheshift-mapforretargeting(section 4.1 )andadjustment-map(section 4.1.2 )together. 2. Ihaveimprovedtheimportanceltering[ 6 ]algorithmforstereoimageretargeting,withspecialconcernofoccludedpixelsanddisparitypreservation. 1Thischapterhasbeenpublishedin[ 22 ]. 31

PAGE 32

Figure4-1. OutlineofTheProposedStereoImageRetargetingAlgorithm:Theinputisapairofstereoimages,IL,IR,andadisparitymapId.Theshift-mapsSLandSRforretargeting,andtheadjustment-mapsAL,ARfordisparityadjustmentareinitiallycomputedindependently.Thetwopairsofmapsarecombinedatthelaststeptoyieldthenalshift-mapsforgeneratingtheretargetedimages. 3. Ihaveinventedanotherwaytoadjustthedisparityvaluesforbetterviewingexperienceotherthan[ 14 ]byusingtrimmingandadjustment-map.Figure 4-1 illustratesthegeneraloutlineoftheproposedalgorithm.Theinputsareapairofrectiedstereoimages,ILandIR,andadisparitymapId.Imageretargetinganddisparityadjustment,originallyproceedindependently,producetheshift-mapsfortheimagepairSL,SRandthedisparityadjustment-mapsAL,ARindependently.Thenalstepcombinesbothestimatestogetherbyaddingthecorrespondingshift-mapandadjustment-maptoyieldthenalshift-mapSLadjandSRadj.Thenaloutputstereoimagepairs,ILoutandIRout,isthengeneratedbyimageinterpolationusingthetwonalshift-maps.Ievaluatetheproposedmethodextensivelyintheexperimentalsection,andmyresultsdemonstratemymethod`sefciencyaswellasitspotentialforproducinghigh-qualityoutputs,withaconsiderablyshorterrunningtime. 32

PAGE 33

4.1.1ImprovedImportanceFilteringForStereoImages 4.1.1.1Pixelsaliencyandshift-mapgradientDenoteG=rxSthehorizontalgradientoftheshiftmapS.G(x,y)=1willindicatethatnodeformationoccursat(x,y),whileG(x,y)=0meansthat(x,y)willberemoved.OthervaluesofG(x,y)2(0,1)representdifferentamountsofshrinkageatthegivenpixel.Foroccludedpixels,theshift-mapgradientisalwayssettooneandtheformulaforG(x,y)isgivenby G(x,y,Es(x,y))=8><>:Cye)]TJ /F9 7.97 Tf 6.59 0 Td[(22(1)]TJ /F13 5.978 Tf 5.76 0 Td[(Es(x,y) 2)2(x,y)=2O1(x,y)2O(4)where=W`=W,EsisthenormalizedsaliencymapandOisthesetoccludedpixels.isatuningparameterwhosevalueisbetween0.2to0.5.Tosatisfytheboundaryconstraint,thesumofthegradientineachrowshouldbeW` X(x,y)2O1+X(x,y)=2OG(x,y,Es(x,y))=W`8y,(4)andthisdeterminesthenormalizationconstantCyforrowyinEquation( 4 ) Cy=(W`)]TJ /F12 11.955 Tf 11.95 8.96 Td[(P(x,y)2O1) P(x,y)=2OG(x,y,Es(x,y)).(4)Sinceimportancelteringcouldnotenforcetheboundaryconstraintwithoccludedpixelsincorporated,inEquation( 4 )abalancefactorisaddedtoadjustthesumofshift-mapgradient.Thealueofisrelatedtotheamountofocclusion,andformostcases2[0.8,1]. 4.1.1.2ImprovedimportancelteringImportanceltering[ 6 ]computestheshift-mapSbyweightedintegrationofagivengradientmapG.Bysettingthespanoftheltertobeonequarteroftheimageheight,theup/bottomneighboringshift-mapvalueswillbesimilar.Withtheextraconstraintof 33

PAGE 34

preservingoccludedareas,myimprovedimportancelterisasfollows: S(x,y)=8><>:Py+rj=y)]TJ /F13 5.978 Tf 5.76 0 Td[(rw(x,j)[S(x)]TJ /F9 7.97 Tf 6.59 0 Td[(1,j)+G(x,j)] Py+rj=y)]TJ /F13 5.978 Tf 5.76 0 Td[(rw(x,j)(x,y)=2OS(x)]TJ /F4 11.955 Tf 11.96 0 Td[(1,y)+1(x,y)2O(4)withtheweightw(x,y)givenby w(x,y)=eEs(x,y).(4)Equation( 4 )impliesthatshift-mapvaluesofnon-occludedpixelswillbecomputedfromaonedimensionlterofsize(2r+1)bytakingtheweightedaverageofthepredictedvalues,whichisthesumofshift-mapvaluefromthepreviouscolumnandtheshift-mapgradient.Meanwhile,foroccludedpixels,Iusethesameassumptionas[ 4 ]thatnosizeshrinkagewouldhappenwithinthesepixels,thustheirshift-mapvaluesarecomputeddirectlybyadding1toshift-mapvaluesoftheirleftneighbours.Iuseatwo-stepapproachtoobtaintheleftandrightshift-mapSLandSR:FirstIcomputetheleftandrightshift-mapsindependentlybyEquation( 4 ).Second,Irenetheshift-mapvaluesasfollows:Supposepandqareapairofcorrespondingpixelsintheleftandrightimages,andfpandfqaretheirpreliminaryshift-mapvaluesfromtherststep,thentheirnalshift-mapvaluescouldbecomputedby( 4 ): 8><>:SL(p)=1 2(fp+fq+dp,q)SR(q)=1 2(fp+fq)]TJ /F5 11.955 Tf 11.95 0 Td[(dp,q)(4)Equation( 4 )ensuresthatinformationfrombothILandIRareutilizedtocomputeshift-maps,anddisparityvaluesareexplicitlypreserved.Shift-mapvalueofoccludedpixelswillbelledinaccordingtoequation( 4 )afterwards. 4.1.2DisparityAdjustmentFordisparityadjustment,thegoalistomaptheoriginaldisparityvalued2`=[dmin`,dmax`]toatargetedranged2=[dmin,dmax].Adirectbutunsatisfactorysolutionistondadirectmappingfunctiond='(d`),andwarptheimagebymoving 34

PAGE 35

correspondingpixelpairstoadjustthedisparityvalue.However,ifthedifferencebetweenand`islarge,thismethodinvariablyrequireslargepixeldeformationthatcancauseseriousimagedistortions.Instead,Iproposedanalternativesolutionthatfollowstheideaoftrimmingandmappingusingthenon-linearmappingfunctiond='(d`)]TJ /F5 11.955 Tf 11.95 0 Td[(dt).Thebasicideafordealingwithlargedifferenceindisparityvaluesistorstcorrectitwithalargeconstantshift(dt)indisparityvaluesfollowedbyanonlinearmapping'tomapfromn`=[dnmin`,dnmax`]=[dmin`)]TJ /F5 11.955 Tf 12.46 0 Td[(dt,dmax`)]TJ /F5 11.955 Tf 12.45 0 Td[(dt]to.Thelargeconstantshiftindisparityvaluescanbeaccomplished,specicallyforimageretargetingapplications,bytrimmingawayimageregionsneartheborders.Withaproperlychosenvalueofdt,n`andwillhavesignicantoverlaps,whichminimizestheamountofpixelmovement(usuallytolessthan10).Notethattrimmingispeculiartoimageretargetingandunderadifferentcontext(e.g.,[ 14 ]),largedeformationandtheresultingdistortionareusuallyunavoidable. 4.1.2.1TrimmingTrimmingisdonebycroppingimagecolumns.Ifdtislargerthanzero,thentheleftmostdtcolumnsinILandtherightmostdtcolumnsinIRarecropped,andviceversa.Thisisreasonablebecauseinmostcasessalientregionsintheimageareusuallyawayfromtheboundary.Furthermore,thenumberofcroppedcolumnsalsocontributetothetotalsizereduction.Theoffsetconstantdtiscloselyrelatedtothedisparityrangesand`.Inmyimplementation,dmin`anddmax`in`arexedtobethedisparityvaluerankedexactlyonthetop1%andbottom1%ofalldisparityvaluesinId(disparityvaluesatoccludedpixelsdonotcount),anddtissettodmin)]TJ /F5 11.955 Tf 12 0 Td[(d`mintomatchand`roughly. 4.1.2.2Adjustmentfunction(dn)andadjustmentmapThedisparitymappingfunctionsarefomulatedusingoperatorsin[ 14 ],andbyapplyingdifferentoperators(linear/nonlinear,continuous/discontinuous),itispossible 35

PAGE 36

toadjustthedisparityvalueswithgreatexibility.Inmymethod,thenonlinearmappingfunction'(x)iscalculatedas '(dn)=d`min+Zdndnmin'`(x)dx=d`min+1 nZdndnminh(x)dx(4)In( 4 ),h(dn)isahistogramcountingthenumberofpixelswithacertaindisparityvaluedn.Thevalueofh(dn)isregardedasproportionaltotherstderivativeof'(dn),andnisthenormalizationfactortoensurethat '(dnmin)='(dmin)]TJ /F5 11.955 Tf 11.95 0 Td[(dt)=d`min (4) '(d`nmax)='(dmax)]TJ /F5 11.955 Tf 11.96 0 Td[(dt)=d`max.Theadjustment-functionisdenedthatquantiestheamountofadditionalpixelmovementneededforapixelwithdisparitydnas(dn)=dn)]TJ /F3 11.955 Tf 12.91 0 Td[('(dn).AnexampleisshowninFigure 4-2 .Inmyalgorithm,nonlineardisparityadjustmentisdonebyaddinganoveladjustment-maptotheshift-map.Supposep:(xL,y)andq:(xR,y)areapairofcorrespondentpixelsintheleftandrightimage,wherexandyisthecolumnandrowcoordinate,thedisparityvaluebetweenthemaredenedasdp,q=xL)]TJ /F5 11.955 Tf 11.97 0 Td[(xR.Thedisparityadjustmentmodelthencouldbeillustratedasfollows: d`p,q='(dp,q)=dp,q)]TJ /F3 11.955 Tf 11.95 0 Td[((dp,q)=xL)]TJ /F5 11.955 Tf 11.95 0 Td[(xR)]TJ /F3 11.955 Tf 11.95 0 Td[((dp,q)(4)Tominimizethedistortioncausedbydisparityadjustment,theamountofshiftisevenlydistributedtopandq.IdeneL(d)=R(d)=(d)=2,thentheshiftingmodelis: 8><>:x`L=xL)]TJ /F3 11.955 Tf 11.96 0 Td[(L(dp,q)x`R=xR+R(dp,q)(4)Theadjustment-mapisconstructedbyapplying(d)toallthepixels: 36

PAGE 37

Figure4-2. Thisgureshowshowadjustment-functioniscalculated.(d),dminanddmaxarelabelledbythereddots.A)thehistogramofthedisparityvalue,B)themappingfunction'(dn)byintegratingandnormalizingthehistogram,C)theadjustment-function(dn)=dn)]TJ /F3 11.955 Tf 11.95 0 Td[('(dn). AL(p)=8><>:L(dp,q)p2IL^p=2OL0p2IL^p2OL(4) AR(q)=8><>:R(dp,q)q2IR^q=2OR0q2IR^q2OR(4)OLandORarethesetofoccludedpixelsinILandIR.Oncetheshift-mapsSLandSRareobtained,thenalshift-mapforbothretargetinganddisparityadjustmentcouldbegenerateddirectlyby: 8><>:SLadj=SL)]TJ /F5 11.955 Tf 11.95 0 Td[(ALSRadj=SR+AR(4)Byinterpolationfromthenalshift-mapSLadjandSRadj,theretargetedanddisparityadjustedstereoimagepairILoutandIRoutcouldbeobtained. 4.1.2.3SmoothingtheadjustmentmapApplyingtheadjustment-mapfrom( 4 )and( 4 )directlywithoutconsideringsmoothnessusuallyresultsinsignicantvisualdefects,andexperimentalresultshavesuggestedthatsmoothingtheadjustment-mapcanprovidesignicantimprovementinthequalityofthenalresult.Horizontalsmoothingispreformedonthegradientdomainoftheadjustment-map,rxA,bysmoothingthepixelswithabsolutegradientsdeemed 37

PAGE 38

AB CDFigure4-3. EffectivenessofAdjustment-MapSmoothing.A,Bdisplaystheun-smoothedadjustment-mapandvisibledefectscanbeobservedaroundhumantorsosandlegs.C,Dshowsthesmoothedadjustment-mapwithamuchbetterimprovementinvisualquality.(PhotoscourtesyofTaliBasha[ 4 ].) toolarge.TherststepistoscanthepixelsinrxArowbyrowfromtoptobottom.Onceapixel(x0,y0)wherejrxA(x0,y0)j>isspotted,thefollowinghorizontalsmoothingfunctionisapplied,whilekeepingthesumofeachrowinrxAunchanged: rxA(x,y0)=1 2r+1Px0+ri=x0)]TJ /F6 7.97 Tf 6.59 0 Td[(rrxA(i,y0)x=x0)]TJ /F5 11.955 Tf 11.95 0 Td[(r,x0)]TJ /F5 11.955 Tf 11.95 0 Td[(r+1,,x0+r(4)In( 4 ),rdenesthesizeofthehorizontalsmoothingwindowandinmyexperimentitissettoceil(jrxA(x,y)j ).denotestheamountofsmoothing.Largerwillenhanceadjustmentaccuracy,butimpairvisualeffects.Areasonablevalueofisfoundto 38

PAGE 39

be0.3.Thehorizontally-smoothedadjustment-mapAshcanbeeasilyrecomputedbyintegratingthesmoothedversionofrxA.Verticalsmoothingisdoneeasilybyapplyingalterwiththesizeof1(2r+1)toAshsothateachpointswilltaketheaverageofbothitselfanditsrupper/lowerneighbors.Meanwhile,sincetheadjustment-mapissmoothedinsteadofthedisparitymap,thedepthrelationshipwouldnotbemessedup.Figure 4-3 showsthesignicantimprovementintheoutputqualityafterhorizontalandverticalsmoothing.Itisinevitablethataftersmoothingthedisparityvaluesforsomepixelswillnolongerequalto'(dp,q).Butsinceonlypixelswithabruptchangesinadjustment-mapwillbeaffected,thetrade-offbetweeninaccurate'(dp,q)andimagequalityclearlyfavorsthelatter. 39

PAGE 40

CHAPTER5ASMARTPROJECTOR/CAMERASYSTEMUbiquitousdisplayisanexcitingfuturistictechnologythatiscurrentlyunderintenseinvestigationanddevelopment.Inparticular,smartprojectors,projectorsystemsthathavetheabilitytoonlineadjustandadapttheirdisplay,havebecomepopularrecently,andmanypersonalprojectorsystemsarecommerciallyavailable.Forubiquitousdisplays,theprojectsystemsnolongeroperateincontrolledenvironmentssuchasclassrooms,conferenceroomsormovietheaters,anditcanbeassumedthattheavailabledisplayareaswillfarfrombeingideal.Therefore,animportanttechnicalproblemistoprovidetheprojectorsystemwithcertainabilitytoonlineadjustitsdisplayinordertoprovideasmuchdelityaspossibility,giventhelessthanoptimalconstraints.Inthischapter,wewillshowhowtheseamsegmentcarvingalgorithmpresentedinchapter 3 canbeincorporatedintoaprojectorsystem,andtheresultingsmartprojectorsystemthathastheabilitytodetectthedisplayareaandretargettheimagesaccordingly. 5.1ASmartProjector/CameraSystemTheproposedsmartprojectorsystemhastwocomponents:thedisplayareadetectionmoduleandtheimageretargetingmodule.Aschematicillustrationofthesystem'sworkowisshowninFigure 5-1 .Thedisplayareadetectionmoduleisresponsiblefordetectingthedisplayareabyprojectingtwopredesignedpatterns,andmatchingthecorrespondingfeaturepointsbetweentheoriginalpatternandthecapturedimage.Usingthedisplayimageasthereference,theareadetectionmodulecomputeabinarymaskMthatdenesthedetectedarearelativelytotheimage.Detailsofthisprocesswillbepresentedinsection 5.3 .Withthedisplayareasuccessfullydetected,themaskMwillbeincorporatedintheimageretargetingmoduleastheinputfortheaforementionedseamsegmentcarving 40

PAGE 41

Figure5-1. Twocomponentsandworkowoftheproposedsmartprojector/camerasystem. algorithm.TheimageIisthenregargetedandprojected,automaticallyreshapedandalignedtothedisplayarea. 5.2TheProjector-CameraDistortionModelTheproposedprojector/camerasystemiscomposedofaprojector,acameraandacomputer.Controlledbythecomputer,theprojectorprojectstheimageinfull-screensize,andthecameracapturestheentiredisplayarea.Figure 5-2 (left)showsoneexampleofthesystemsetting.Thedistortionmodelintheprojector/camerasystem,rstintroducedin[ 34 ],canbeillustratedasfollows:SupposetheoriginalimagetobeprojectedisI,thecorrespondingcapturedimageofthewholedisplayareaisIcap,thentwocorrespondingpointsPandPcapbetweenIandIcapcouldberelatedbyprojectivetransformation: pcap=TcamTprojp=Hp,p=H)]TJ /F9 7.97 Tf 6.59 0 Td[(1pcap,H=TcamTproj.(5) 41

PAGE 42

ABFigure5-2. Settingofthesmartprojector/camerasystemandthedistortionmodel.A)Therealsettingofthesmartprojector/camerasystem.B)Thedistortionmodel.(PhotoscourtesyofShaoyuQi.) Figure 5-2 (right)describesthedistortionmodel.Inthewholeprocedure,twotransformations,TcamandTprojareincorporatedtomodelthedistortionfromtheprojectorandcamera,asillustratedinequation( 5 ).CombiningTcamandTproj,thetransformationbetweenIandIcapcanbemodeledasprojectivetransformationH.Withhomogeneouscoordinates,therelationbetweenpairsofcorrespondingpointsPandP0inIandIcapare: 266664h11h12h13h21h22h23h31h32h33377775266664xy1377775=266664x0y0z0377775266664x0=z0y0=z01377775(5)Inthesettingoftheproposedsmartprojector/camerasystem,itisreasonabletoassumethattheprojectorhasbeencalibratedinternally,andthedistortioncausedbytheprojectorwillbeignoredinthefollowingpresentation. 5.3DetectingDisplayAreaAccordingtotheaboveprojector/cameramodel,theshapeofdisplayarea,representedasabinarymaskM,canbedetectedbyndingtheprojectivetransformation 42

PAGE 43

ABCDFigure5-3. Projectedpatternsandcapturedimages.A,BTwoprojectedpatterns.C,DCapturedImagewithbothPatterns. Hfrompairsofcorrespondingfeaturepoints,andwarpingthedetectedboundaryMcapinIcapaccordingtoH. 5.3.1ProjectedPatternsInthesmartprojector/camerasystem,twopatternswiththesamesizeofprojectorresolution(Figure 5-3 )areprojectedandcapturedsequentially.TherstoneisacheckerboardIpcpatternwithblocksize5050underascreenresolutionof600800.Thecornerpointsofalltheblocksareusedasfeaturepointstogetthecorrespondencesbetweenthepatternandcapturedimage.ThesecondpatternIpgisagraybackgroundwithtworedandbluecoloredblocks.Thegraybackgroundlightsupthedisplayarea,andthetwocolorblocksrefertothepositionoftwoparticularcornerpointsinthecheckerboard.Thedimmed(gray)backgroundensuresthatallthecornerpointsandcolorblocksareclearlyvisibleinthecapturedimage(Figure 5-3 ).Theonlyrequirementofprojectorsettingisthatthetwocolorblocksshouldbewithinthedisplayarea.Figure 5-3 (C-D)showstwocapturedimagewithcorrespondingaforementionedpatterns.InthefollowingsectionsIwillbeusingIccandIcgtodenotethetwocapturedimagewith'checkerboard'and'gray'patterns. 5.3.2DetectingDisplayAreasAndCornerPointsInCapturedImageWithoutlostofgenerality,itisreasonabletoassumethesubstantialdifferenceincolor/intensitybetweenthedisplayareaandthebackground,thusthedisplayareainthecapturedimageIcgcouldbeeasilyidentiedbythresholdingtheimageaccordingtothe 43

PAGE 44

ABFigure5-4. Calculatingcorrespondences.A)FindingCorrespondencesoftherstfourcornerpoints.B)Findingnewcorrespondencesfrompairsofcomputedcorrespondenceinastepwisemanner. intensityvalue,andselecttheconnectedcomponentcoveringthetwocolorblocks.ThedetecteddisplayareainIcgisstoredasabinarymaskMcap.Projectivetransformationcanbecalculatedfromthecornerpointsofthecheckerboardwithinthedisplayarea.ThesepointsaredetectedbyapplyingtheHarriscornerdetector[ 11 ]toIcc,removingthecornersbytheboundary,andmergingpositiveresponsesfromtheHarrisDetectorthatoriginatedfromthesamepairofcornerpoints. 5.3.3FindingTheProjectiveTransformationMatrixHTocalculatetheprojectivetransformationmatrixH,rstcorrespondencesbetweencornerpointsCfromIpcandCcapfromIccshouldbecomputed,thenequation( 5 )couldbesolvedbythenormalizedDirectLinearTransform(DLT)algorithm[ 40 ]. 5.3.3.1MatchingcorrespondingfeaturepointsNotethatinourpatterns,positionsoftwocornerpointshavealreadybeenhighlightedbythetwocoloredblock,andcorrespondencesofthesetwocanbeacquireddirectlybydetectingthepositionofredandblueblocks.AmoredetailedillustrationisinFigure 5.3.3.1 (A).SupposePrandPbaretwocornerpointscorrespondingtothetwocolorblocks,thentheothertwopointswithinthesameblock,P1andP2,canbeeasilyselectedbyndingthefourclosestcornerpointsoftheaveragepositionofPrandPb(labeledas'x'intheimage).P1andP2canbe 44

PAGE 45

ABCDFindingcorrespondencesfromfourpairsofcornerpoints. EFComputedcorrespondencesbetweenpairsofcornerpointsinFigure 5-3 Figure5-5. Illustrationofcornerpointsregistration.(A-D)Correspondencesarecomputedfromfourregisteredpoints.(E-F)Allthecorrespondentpointsarecalculated. distinguishedviandingrelativepositionsbytakingcrossproduct,andverifyingthesignofzcomponentCorrespondencesoftherestofcornerpointsinCcaparecomputedinthefollowingmanner:givenatupleofneighboringpoints(P1,P2),theestimatedpositionofthenextcornerpointPestalongthesamedirectionof)485()222()484(!P1P2isPest=P2+)484()222()485(!P1P2.ThetruepositionofthenextcornerpointcanthenbeachievedbysearchingfortheclosestcornerpointPinCcapsothatkP)]TJ /F5 11.955 Tf 12.33 0 Td[(Pestk<0.3k)484()222()485(!P1P2k,ifPestisinthedetecteddisplayareainMcap.AnexampleofndingthepositionoftwoneighboringpointsP3andP4isshowninFigure 5.3.3.1 (b).TheApproximateNearestNeighbor(ANN)algorithm[ 1 3 18 ]isincorporatedtoboostthespeedofpointsearching.Correspondencesforalltherestcornerpointsarecomputedfromtherstfourpairsofcornerpointsasthe'seed',asillustratedinFigure 5-5 (A-D).Inthebeginning8 45

PAGE 46

tuplesofneighboringpointsareaddedtotheprocessingqueueQfromthefourpairsofregisteredneighboringpoints((P1,P2)and(P2,P1)aretwotupleswiththesamepointsbutdifferentdirections).Onceanewcornerpointhasitscorrespondence,newtuplesbetweenthecurrentpointandallitsregisteredneighborsareaddedtothequeue.ThealgorithmterminateswhenQbecomesempty,bywhichmomentallthecornerpointsinCcaphavetheircorrespondence(Figure 5-5 (E-F)). 5.3.3.2FindingHfrompairsofcorrespondingcornerpointsNormalizedDirectLinearTransform(DLT)algorithm[ 40 ]isefcienttosolvefortheprojectivetransformationmatrixamongNpairsofcorresponding2-Dpoints(x,y,1)and(x0,y0,1),byrstnormalizingthetwosetofpairedpointscenteringtotheoriginwithmaximumdistanceofp 2,andthensolvingthefollowinglinearsystemA)777(!h=)777(!0: 2666666666666666664x1y11000)]TJ /F5 11.955 Tf 9.3 0 Td[(x1x01)]TJ /F5 11.955 Tf 9.29 0 Td[(y1x01)]TJ /F5 11.955 Tf 9.3 0 Td[(x01000x1y11)]TJ /F5 11.955 Tf 9.3 0 Td[(x1y01)]TJ /F5 11.955 Tf 9.29 0 Td[(y1y01)]TJ /F5 11.955 Tf 9.3 0 Td[(y01x2y21000)]TJ /F5 11.955 Tf 9.3 0 Td[(x2x02)]TJ /F5 11.955 Tf 9.29 0 Td[(y2x02)]TJ /F5 11.955 Tf 9.3 0 Td[(x02000x2y21)]TJ /F5 11.955 Tf 9.3 0 Td[(x2y02)]TJ /F5 11.955 Tf 9.29 0 Td[(y2y02)]TJ /F5 11.955 Tf 9.3 0 Td[(y02...........................xNyN1000)]TJ /F5 11.955 Tf 9.3 0 Td[(xNx0N)]TJ /F5 11.955 Tf 9.29 0 Td[(yNx0N)]TJ /F5 11.955 Tf 9.3 0 Td[(x0N000xNyN1)]TJ /F5 11.955 Tf 9.3 0 Td[(xNy0N)]TJ /F5 11.955 Tf 9.29 0 Td[(yNy0N)]TJ /F5 11.955 Tf 9.3 0 Td[(y0N377777777777777777526666666666666666666666664h11h12h13h21h22h23h31h32h3337777777777777777777777775=)778(!0(5)Equation( 5 )canbesolvedbydoingsingularvaluedecompositiononA=USV,and)777(!histhecolumnofVcorrespondingtotheminimumsingularvalue. 5.3.4DisplayAreaMAndSeamSegmentCarvingThegroundtruthofdisplayareaMiscomputedbywarpingMcapwithaccordancetotheprojectivetransformationH.ImagesarethenretargetedwiththemaskM,andprojectedtotthedisplayarea. 46

PAGE 47

CHAPTER6EXPERIMENTALRESULTSANDDISCUSSIONSIhaveproposedtwonovelimageretargetingrelatedalgorithmsinthechapter3and4.InthefollowingpartofthispaperIwillshowtheresultsshowingthecapabilityandeffectivenessofmyalgorithms. 6.1ExperimentalResultsAndApplications ABCDE FGHIJ KLMNOFigure6-1. Effectiveofseamsegmentcarving.Threeexamplesshowingthatmyalgorithmcannotonlyrestoreimportantcontentsfromthecroppedarea,butalsocanpreservethedetails.(Fromlefttoright)A,F,KOriginalimagewithnewboundaryhighlighted,B,G,LCutting-throughseams,C,H,MSeamsegments,D,I,NIrregularshaperetargetedimage,E,J,OClose-upviewofthedetails.(betterviewingincolor)(PicturescourtesyofMikiRubinstain[ 24 ]andMicrosoftResearch.) ItestmyalgorithmonacollectionofimagesontheRetargetMe[ 24 ]dataset,theMSRASalientObjectDatabase[ 15 ]aswellassomeavailableimagesonline,andchangetheimageintodifferentkindsofshapes.Inmostofthecases,myalgorithmhastheabilitytoretrievetheimportantinformationfromthecroppedareas.Fig. 6-1 showsthecorrectnessandefciencyofmymethod.InFig. 6-1 ,forthe'boat'example 47

PAGE 48

ABCD EFGH IJKLOriginalImageDirectCroppingRetarget&CropOurMethodFigure6-2. Comparisonoftheresultsfromtheseamsegmentcarvingtodirectcroppingand'retarget&crop'.(Lefttoright)A,E,IOriginalimagewithnewboundaryhighlighted,B,F,JResultsfromdirectcropping,C,G,KResultsfrom'retarget&crop',D,H,LResultsfrommymethod.(betterviewingincolor)(PicturescourtesyofMikiRubinstain[ 24 ]andMicrosoftResearch.) theheadpartoftheboat,aswellassomeotherobjectsbythebank,arestillkeptintheretargetedimage.Inthe'orchid'example,notonlythewholerightpetaliskept,butthetinycurvesonthepetalissuccessfullypreservedaswell. 6.1.1ApplicationsInthissectionIpresenttwopossibleapplicationsfortheirregularimageshaperetargetingalgorithmwithexperimentalresults. AestheticImageEditing.Oneofthepossibleapplicationofirregularimageretargetingisaestheticimageediting.Invarioussituationsincludingmagazine,posterandicondesign,imagesareintentionallyreshapedforbetterviewingeffects.Moreover, 48

PAGE 49

ABCImage&BoundarySeamSegmentCarvingCroppingFigure6-3. Limitationofseamsegmentcarving.Forthehighlystructuredobjectsoutsidethenewboundary,croppingbecomesabettersolution,forthatstructurepreservationismoredesiredinthiscase.A)Theimageandthenewboundary.B)Resultfromourmethod.C)Resultfromcropping.(PicturescourtesyofMikiRubinstain[ 24 ].) intheproductionofE-cardsanddigitalphotoalbums,sometimesitisalsoneededtoreshapetheimagetotthenon-rectangularimageframe.InFig. 6-2 imagesareretergetedintodifferentshapes.ForcomparisonIalsoshowresultsfromothertwomethods,directcroppingand'retarget&crop',thelatterisdonebycarvingonlythecutting-throughseams,andcroppingaccordingtothenewboundary.Asshownintheexamples,both'retarget&crop'andmyapproachcouldretrievemorecontentsthandirectcropping,butmymethodoutperformsthe'retarget&crop'methodinthatinformationsinthecroppedareascouldbefurtherretrieved,liketheowerandbutterywingsinrow1,andsandsandsmallrocksinthelastrow. 6.1.2LimitationsAccordingtothepreviousdiscussions,theamountofdeformationsdifferinlocations,whichindicatesthatthenewboundaryshouldbereasonablyshaped.Iflargepartofsalientarearesidesoutsidethenewboundary,thecroppedareacontainsstructuredobjects,orifthenewshapeisverycurvy,twistedorseverelydecreasedinsize,thenitishighlyunlikelytorestorethatobjectproperly,orproduceavisuallypleasingresult,andcroppingwouldinturnbecomeabettersolution(Fig. 6-3 ). 49

PAGE 50

6.2StereoImageRetargetingWithDisparityAdjustmentIhaveimplementedtheproposedalgorithminMATLABandCwithoutseriouscodeoptimizations,andittakeslessthan3stogeneratea300400retargetedstereoimagepairsona3.4GHzcomputer.Incomparison,Bashaetal.'smethod[ 4 ]willtakearound20stoiterativelycarvealltheseams.Inthefollowingsubsection,Iwillrstdemonstratethecorrectnessofmystereoimageretargetingalgorithmwithoutdisparityadjustment,andthendemonstratehowviewingeffectcouldbeenhancedusingwithdisparityadjustment. 6.2.1StereoImageRetargetingItestmyalgorithmonthestereoretargetingdatasetfrom[ 4 ]andMiddleburystereodatasets.Sincecomputingthesaliencymapdirectlyfromintensityvalueshasalwaysbeentricky,inthisexperiment,Idonotuseanyparticularsaliencymapinordertoavoiddegradingthequalityoftheoutputimageswithsuboptimalsaliencymap.Instead,IusethenormalizeddisparitymapcombinedwiththeimagegradientasthesaliencymaptocomputethetwogradientmapsGLandGRrequiredfortheimportanceltering.Figure 6-4 showsthecorrectnessofmyalgorithmforregularstereoimageretargetingproblem(withoutdisparityadjustment).Notethatthefaceiskeptintactwithminimalamountofdistortion.Furthermore,moredetailedfeaturessuchastheshapeofeyebrowsandhairsarealsowell-preserved.Inmyalgorithm,thedisparityvaluesarekeptapproximatelyexact,asthealgorithmmanipulatesthepositionsofeachpairofcorrespondingpixelstoforcethemtoassumeaparticulardisparityvalue.ThethirdcolumnofFigure 6-4 showsthedisparitymapscomputedbyELASalgorithm[ 7 ]thatveriestheagreementbetweenthedisparityvaluesbeforeandafterretargeting.In[ 4 ]theyalreadyprovedthatitisnotsufcienttopreservethedepth(disparity)byuniformscaling,thusIdonotincorporateuniformscalingintothecomparisons.Ihavealsocomparedmyalgorithmwiththecurrentstate-of-artmethod[ 4 ].ResultsshowninFigure 6-5 indicatethatmymethodcanprovideresultsthatarevisuallymore 50

PAGE 51

ABC DEFFigure6-4. CorrectnessoftheProposedAlgorithm.Columnsfromlefttoright:A,DOriginalpairofstereoimages.B,ERetargetedpairofstereoimages.C,FThedisparitymapcomputedbeforeandafterretargetingusingLibelasalgorithm[ 7 ].Thesizereductioninwidthis25%.(PicturescourtesyofTaliBasha[ 4 ].) appealingwithbetterpreservationofimagestructures.Inthe'man'example(rsttworows),thetracklinesonthegroundareclearlynotstraightintheirresultbecausenon-horizontal/verticallinesaredifculttopreserveusingseamcarving,whichcanbebetterhandledusinganinterpolation-basedmethod.The'car'example(thirdandforthrow)furtherdemonstratesthedifferencebetweenmyalgorithmand[ 4 ].Seamcarvingcanremovemorepixelsinsmallandtwistedareas,likethegapbetweenthewhitePolointhecenterandthebluecarontheright.However,thestructuralintegralityofanobjectcannotbeguaranteedonceaseamhascutthroughanobject.ImportancelteringgiveninEquation( 4 )([ 6 ])canmaketheneighboringpixelsinthesamecolumntohavesimilarshift-mapvalues,andithelpstodecreasetheamountofunnecessarydistortion.Inthe'car'example,thebackendoftheblackwagonontherightsuffersseriousandvisibledistortionafterseamcarving,butitisessentiallyintactinmyresult. 51

PAGE 52

ABC DEF GHI JKLOriginalImageBashaet.al.MyResultFigure6-5. Comparisonbetweenmymethodand[ 4 ].Imagewidthisdecreasedby20%inthisexample.(LefttoRight):A,D,G,JOriginalimage.B,E,H,KResultsfrom[ 4 ].C,F,I,LMyresults.Notethestraighttracklinesproducedbytheproposedalgorithmintherstexample.(PicturescourtesyofTaliBasha[ 4 ].) 52

PAGE 53

ABC DEF GHIFigure6-6. EffectivenessofDisparityAdjustment.Toptworows(lefttoright):A,DOriginalstereoimages.B,ERetargetedstereoimages.C,FAdjustment-maps.Bottomrow(lefttoright):G)Anaglyphwithnodisparityadjustment,H)Anaglyphwithtrimmingonly,I)Anaglyphwithtrimminganddisparityadjustment.(Bestviewedincolorandred-cyan3Dglasses.)(PicturescourtesyofTaliBasha[ 4 ].) 6.2.2DisparityAdjustmentThetermcomfortzonediscussedinSection 4.1.1.1 and[ 14 ]impliesthatnotallthestereoimagesaredisplay-ready.Inmyexperiment,itisobservedthatfora23-inchLEDscreen,reasonabledisparityvaluesfordisplayisbetween-15(behindthescreen)and+5(infrontofthescreen).Forthedisparityvaluesoutsidethisrange,theobjectswillappeartoonearortoofaraway,whichcauseunpleasantphysiologicalresponsessuchasdizziness.Moreover,itisreasonabletoplacetheimportantforegroundobjectsatthescreenlevel(withdisparityvaluecloseto0)forbettervisualeffect.Basedonthisknowledge,inthefollowingexperimentsIsetthedesiredrangeofdisparitytobethe 53

PAGE 54

ABC DEF GHIFigure6-7. AnotherExampleofDisparityAdjustment.Fortheregionwhereabruptdifferencesoccurinthenon-smoothadjustment-map,distortionbecomesvisibleduetosmoothing.However,thisvisualdefectisnotparticularlynoticeablebyhumanwhileviewingtheimagesin3D.Bycomparingthethreeanaglyphsinthebottomrow(G-I),theonewithtrimmingandnonlineardisparityadjustment(I)clearlyprovidesthebestviewingresult.(Bestviewedincolorwithred-cyan3Dglasses.)(PicturescourtesyofTaliBasha[ 4 ].) interval`=[)]TJ /F4 11.955 Tf 9.3 0 Td[(15,5],andusethenonlinearmappingfunctionmentionedin 4.1.2 tocalculatetheadjustmentmap.TwoexamplesofthedisparityadjustmentalgorithmareshowninFigure 6-6 and 6-7 .Theoriginalimagesarenotreadyfordisplaybecausethedisparityvaluesaretoolarge.Peopleintheanaglyphmadebydirectcombinationoftheretargetedstereoimages(withoutadjustingdisparityvalues)appeartooneartotheviewer.Aftertrimming,theentiresceneisuniformlymovedawayfromtheviewer;however,itisstillnotoptimalsincethedepthdifferencebetweentheforegroundandbackgroundistoo 54

PAGE 55

ABC DEF GHICheckerboardPatternGrayPatternDetectedDisplayAreaFigure6-8. Resultsofdisplayareadetection.A,D,GCapturedcheckerboardpattern.B,E,HCapturedgraypattern.C,F,IDetecteddisplayarea.(PhotoscourtesyofShaoyuQi.) largeforcomfortableviewing.Byadjustingthedisparityvalues,thedisparityrangeiscompressedbymovingpeoplebehindthescreen,whilekeepingthebackgrounddisparityunchanged.Thedetailandpatternofpixelshiftsareshownintheadjustment-map,whichindicatesthatalargeamountofpixelmovementoccursintheforegroundregionwhilethebackgroundmostlystaysintheoriginalposition.Inevitably,fortheareaswhereabruptchangesoccurintheadjustment-map(e.g.,theregionaroundtheman'shead),visibledistortionappearsbecauseoftheinconsistentpixelmovementduetosmoothing.Similarproblemhasalsobeen 55

PAGE 56

ABC DEF GHIOriginalImageDirectProjectionSmartProjectionFigure6-9. EffectivenessofSmartProjection(1).A,D,GOriginalimages.B,E,HResultsfromdirectprojection.C,F,IResultsfromsmartprojection.(PicturescourtesyofMikiRubinstain[ 24 ]andMicrosoftResearch.) reportedin[ 14 ]anditwasconsideredtobeunavoidablewithoutanyhigh-levelfeaturedescriptorsorfurtherconstraints.However,itisinterestingtoseethatthisdistortionisnotparticularlynoticeablewhenviewedin3D.Forevaluationofpleasingviewingin3DIactuallyholdauserstudyamongasmallgroupof7people,and6ofthemsupporttheideathattheretargeted3Dimageswithtrimming+nonlineardisparityadjustmentprovidethebestviewingexperiencein3Damongthethreealternativesshowninthegures. 56

PAGE 57

ABC DEF GHIOriginalImageDirectProjectionSmartProjectionFigure6-10. EffectivenessofSmartProjection(2).A,D,GOriginalimages.B,E,HResultsfromdirectprojection.C,F,IResultsfromsmartprojection.(PicturescourtesyofMikiRubinstain[ 24 ]andMicrosoftResearch.) 6.3SmartProjector/CameraSystemTovalidatetheeffectivenessofmysmartprojector/camerasystem,rstthreedifferentshapesofdisplayareasareusedtotesttheaccuracyofdisplayareadetection,withtheresultinFigure 6-8 .InFigure 6-8 ,Itestthreedifferentshapeofdisplayareas.Therstandsecondcolumnsshowtheprojectedtwopatternstothedisplayarea,andthelastcolumncontainstheresultofdetecteddisplayareas.Itcanbeseenfromtheresultsthatthedetecteddisplayareasarecorrectlydetectedandwarped. 57

PAGE 58

Figure 6-9 andFigure 6-10 comparestheresultofdirectprojection(projecttheimageregardlessofthedisplayarea)andmynovel'smartprojection'.Comparedtodirectprojection,itcanbeeasilyjudgedvisuallythatsmartprojectionbringsaboutbetterviewingeffectbytakingtheadvantageofimageretargeting. 58

PAGE 59

CHAPTER7CONCLUSIONSInthisdissertation,Ihavestudiedthecontent-awareimageresizingproblem,imageretargeting,andpresentedtwoimageretargetingalgorithmsforsolvingtwonovelimageretatrgetingproblems:irregular-shapeimageretargetingandstereoimageretargetingwithdisparityadjustment.Inaddition,Ihaveinvestigatedanovelapplicationofimageretargetingintheformofdesigningandimplementingasmartprojectorsystemthatcanautomaticallydetectthedisplayareaandretargettheimageaccordingly.InChapter 3 Ihavedemonstratedanewimageretargetingmethodbasedontheproposednotionof'seamsegments'.Theseam-segmentcarvingcanbeconsideredasanextensionofthewell-knownseamcarvingalgorithm.Byremovingtheseamsegmentsandshiftingthepixels,theimagecanbereshapedwithoutdistortingmuchoftheoriginalsalientcontentsandstructures.Thisalgorithmcanalsobeappliedtoworkwithanexternalsaliencymapwithadditionaltermsaddedtotheframeworkforpreservingapplication-dependentimageregions.InChapter 4 ,Ihavepresentedthestereoimageretargetingalgorithm.Animportanttechnicalcomponentoftheproposedstereoimageretargetingalgorithmisanimprovedmethodforimportancelteringthatcanperformtherequiredintegrationsimultaneouslyandovercomethepossibilityofdeadlock.Ihavealsointroducedthenotionofadjustmentmapforadjustingthedisparityvaluesinordertoaccommodatecertainknownconstraintsrequiredforpleasantandcomfortableviewingof3Dimages.ThevalidityandeffectivenessofbothalgorithmsaresupportedbyexperimentalresultspresentedinChapter 6 .Finally,inChapter 5 ,Ihavepresentedthedesignandimplementationofafunctioningsmartcamerasystem,andtheproposedsystemmakesasmallbutsubstantialcontributionforrealizingthefuturisticvisionofubiquitousdisplay. 59

PAGE 60

REFERENCES [1] Arya,SunilandMount,DavidM.ApproximateNearestNeighborQueriesinFixedDimensions.Proc.4thAnn.ACM-SIAMSymposiumonDiscreteAlgorithms(SODA).1993,271. [2] Avidan,S.andShamir,A.Seamcarvingforcontent-awareimageresizing.ACMTransactionsonGraphics(TOG).vol.26.ACM,2007,10. [3] Bagon,Shai.MatlabclassforANN.2009.URL http://www.wisdom.weizmann.ac.il/~bagon/matlab.html [4] Basha,Tali,Moses,Yael,andAvidan,Shai.Geometricallyconsistentstereoseamcarving.ICCV.2011,1816. [5] Ding,Y.,Xiao,J.,Tan,K.H.,andYu,J.Catadioptricprojectors.ComputerVisionandPatternRecognition,2009.CVPR2009.IEEEConferenceon.IEEE,2009,2528. [6] Ding,Y.,Xiao,J.,andYu,J.Importancelteringforimageretargeting.ComputerVisionandPatternRecognition(CVPR),2011IEEEConferenceon.IEEE,2011,89. [7] Geiger,Andreas,Roser,Martin,andUrtasun,Raquel.EfcientLarge-ScaleStereoMatching.AsianConferenceonComputerVision(ACCV).Queenstown,NewZealand,2010. [8] Grundmann,M.,Kwatra,V.,Han,M.,andEssa,I.Discontinuousseam-carvingforvideoretargeting.ComputerVisionandPatternRecognition(CVPR),2010IEEEConferenceon.IEEE,2010,569. [9] Guo,Y.,Liu,F.,Shi,J.,Zhou,Z.H.,andGleicher,M.Imageretargetingusingmeshparametrization.Multimedia,IEEETransactionson11(2009).5:856. [10] Han,D.,Wu,X.,andSonka,M.Optimalmultiplesurfacessearchingforvideo/imageresizing-agraph-theoreticapproach.ComputerVision,2009IEEE12thInternationalConferenceon.IEEE,2009,1026. [11] Harris,ChrisandStephens,Mike.Acombinedcornerandedgedetector.Alveyvisionconference.vol.15.Manchester,UK,1988,50. [12] Karni,Z.,Freedman,D.,andGotsman,C.Energy-BasedImageDeformation.ComputerGraphicsForum.vol.28.WileyOnlineLibrary,2009,1257. [13] Kav-Venaki,E.andPeleg,S.FeedbackRetargeting.MediaRetargetingWorkshopatECCV2010.Crete,2010. 60

PAGE 61

[14] Lang,M.,Hornung,A.,Wang,O.,Poulakos,S.,Smolic,A.,andGross,M.Nonlineardisparitymappingforstereoscopic3D.ACMTransactionsonGraphics(TOG)29(2010).4:75. [15] Liu,T.,Yuan,Z.,Sun,J.,Wang,J.,Zheng,N.,Tang,X.,andShum,H.Y.Learningtodetectasalientobject.PatternAnalysisandMachineIntelligence,IEEETransactionson33(2011).2:353. [16] Majumder,A.,Brown,R.G.,andEl-Ghoroury,H.S.Displaygamutreshapingforcoloremulationandbalancing.ComputerVisionandPatternRecognitionWorkshops(CVPRW),2010IEEEComputerSocietyConferenceon.IEEE,2010,17. [17] Manseld,A.,Gehler,P.,VanGool,L.,andRother,C.Scenecarving:Sceneconsistentimageretargeting.ComputerVisionECCV2010(2010):143. [18] Mount,DavidM.andArya,Sunil.ANN:ALibraryforApproximateNearestNeighborSearching.2006.Version1.1.1.URL http://www.cs.umd.edu/~mount/ANN/ [19] Ng,T.T.,Pahwa,R.S.,Bai,J.,Quek,T.Q.S.,andTan,K.H.Radiometriccompensationusingstratiedinverses.ComputerVision,2009IEEE12thIn-ternationalConferenceon.IEEE,2009,1889. [20] Pritch,Y.,Kav-Venaki,E.,andPeleg,S.Shift-mapimageediting.ComputerVision,2009IEEE12thInternationalConferenceon.IEEE,2009,151. [21] Qi,ShaoyuandHo,Jeffrey.SeamSegmentCarving:RetargetingImagestoIrregularly-ShapedImageDomains.ECCV(6).2012,314. [22] .Shift-MapBasedStereoImageRetargetingwithDisparityAdjustment.ACCV(4).2012,457. [23] Raskar,R.,VanBaar,J.,Beardsley,P.,Willwacher,T.,Rao,S.,andForlines,C.iLamps:geometricallyawareandself-conguringprojectors.ACMSIGGRAPH2003.ACM,2003,10. [24] Rubinstein,M.,Gutierrez,D.,Sorkine,O.,andShamir,A.Acomparativestudyofimageretargeting.ACMTransactionsonGraphics(TOG).vol.29.ACM,2010,160. [25] Rubinstein,M.,Shamir,A.,andAvidan,S.Improvedseamcarvingforvideoretargeting.ACMTransactionsonGraphics(TOG).vol.27.ACM,2008,16. [26] .Multi-operatormediaretargeting.ACMTransactionsonGraphics(TOG).vol.28.ACM,2009,23. 61

PAGE 62

[27] Sajadi,B.,Lazarov,M.,Gopi,M.,andMajumder,A.Colorseamlessnessinmulti-projectordisplaysusingconstrainedgamutmorphing.VisualizationandComputerGraphics,IEEETransactionson15(2009).6:1317. [28] Sajadi,B.andMajumder,A.Auto-calibrationofcylindricalmulti-projectorsystems.VirtualRealityConference(VR),2010IEEE.IEEE,2010,155. [29] Santella,A.,Agrawala,M.,DeCarlo,D.,Salesin,D.,andCohen,M.Gaze-basedinteractionforsemi-automaticphotocropping.ProceedingsoftheSIGCHIconferenceonHumanFactorsincomputingsystems.ACM,2006,771. [30] Seitz,S.M.,Matsushita,Y.,andKutulakos,K.N.Atheoryofinverselighttransport.ComputerVision,2005.ICCV2005.TenthIEEEInternationalConferenceon.vol.2.IEEE,2005,1440. [31] Sen,P.,Chen,B.,Garg,G.,Marschner,S.R.,Horowitz,M.,Levoy,M.,andLensch,H.Dualphotography.ACMTransactionsonGraphics(TOG).vol.24.ACM,2005,745. [32] Setlur,V.,Takagi,S.,Raskar,R.,Gleicher,M.,andGooch,B.Automaticimageretargeting.Proceedingsofthe4thinternationalconferenceonMobileandubiquitousmultimedia.ACM,2005,59. [33] Steele,R.M.,Ye,M.,andYang,R.Colorcalibrationofmulti-projectordisplaysthroughautomaticoptimizationofhardwaresettings.ComputerVisionandPatternRecognitionWorkshops,2009.CVPRWorkshops2009.IEEEComputerSocietyConferenceon.IEEE,2009,55. [34] Sukthankar,Rahul,Stockton,RobertG,andMullin,MatthewD.Smarterpresentations:Exploitinghomographyincamera-projectorsystems.ComputerVision,2001.ICCV2001.Proceedings.EighthIEEEInternationalConferenceon.vol.1.IEEE,2001,247. [35] Sun,JinandLing,Haibin.Scaleandobjectawareimageretargetingforthumbnailbrowsing.ICCV.2011,1511. [36] Utsugi,K.,Shibahara,T.,Koike,T.,Takahashi,K.,andNaemura,T.Seamcarvingforstereoimages.3DTV-Conference:TheTrueVision-Capture,TransmissionandDisplayof3DVideo(3DTV-CON),2010.IEEE,2010,1. [37] Wang,Y.S.,Lin,H.C.,Sorkine,O.,andLee,T.Y.Motion-basedvideoretargetingwithoptimizedcrop-and-warp.ACMTransactionsonGraphics(TOG)29(2010).4:90. [38] Wang,Y.S.,Tai,C.L.,Sorkine,O.,andLee,T.Y.Optimizedscale-and-stretchforimageresizing.ACMTransactionsonGraphics(TOG)27(2008).5:118. 62

PAGE 63

[39] Wolf,L.,Guttmann,M.,andCohen-Or,D.Non-homogeneouscontent-drivenvideo-retargeting.ComputerVision,2007.ICCV2007.IEEE11thInternationalConferenceon.IEEE,2007,1. [40] Zisserman,Andrew.Multipleviewgeometryincomputervision.CambridgeUniversityPress,2003. 63

PAGE 64

BIOGRAPHICALSKETCH ShaoyuQireceivedhisdoctoratedegreeinthesummerof2013,fromtheDepartmentofComputerandInformationScienceandEngineering,UniversityofFlorida,Gainesville,FL,USA.BeforethathewasstudyinginNanjingUniversity,Chinaandreceivedthebachelor'sdegreeinJune2009,attheDepartmentofComputerScienceandTechnology.Shaoyu'sresearchinterestincludesimageretargeting,computervisionandmachinelearning. 64