<%BANNER%>

Topics in Global Optimization

Permanent Link: http://ufdc.ufl.edu/UFE0041470/00001

Material Information

Title: Topics in Global Optimization Ellipsoidal Bisection, Graph Partitioning and Sparse Reconstruction
Physical Description: 1 online resource (100 p.)
Language: english
Creator: Phan, Dung
Publisher: University of Florida
Place of Publication: Gainesville, Fla.
Publication Date: 2010

Subjects

Subjects / Keywords: bound, branch, convex, global, graph, image, nonmonotone, nonsmooth, optimization, partitioning, quadratic, reconstruction, sparse, sublinear, underestimation
Mathematics -- Dissertations, Academic -- UF
Genre: Mathematics thesis, Ph.D.
bibliography   ( marcgt )
theses   ( marcgt )
government publication (state, provincial, terriorial, dependent)   ( marcgt )
born-digital   ( sobekcm )
Electronic Thesis or Dissertation

Notes

Abstract: In this dissertation, we develop theories and practical algorithms for optimization problems which arise from science, economics and engineering. First, an ellipsoidal branch and bound algorithm is developed for global optimization. The algorithm starts with a known ellipsoid containing the feasible set. Branching in the algorithm is accomplished by subdividing the feasible set using ellipses. Lower bounds are obtained by replacing the concave part of the objective function by an affine underestimate. A ball approximation algorithm, obtained by generalizing of a scheme of Lin and Han, is used to solve the convex relaxation of the original problem. Numerical experiments are given for a randomly generated quadratic objective function and randomly generated convex, quadratic constraints. Second, we propose an exact algorithm for solving the graph partitioning problem with upper and lower bounds on the size of each set in the partition. The algorithm is based on a branch and bound method applied to a continuous quadratic programming formulation of the problem. Lower bounds are achieved by decomposing the objective function into convex and concave parts and replacing the concave part by the best affine underestimate. It is shown that the best affine underestimate can be expressed in terms of the center and the radius of the smallest sphere containing the feasible set. The concave term is obtained either by a constant diagonal shift associated with the smallest eigenvalue of the objective function Hessian, or by a diagonal shift obtained by solving a semidefinite programming problem. Numerical results show that the proposed algorithm is competitive with state-of-the-art graph partitioning codes. Finally, the convergence rate is analyzed for the SpaSRA algorithm (Sparse Reconstruction by Separable Approximation) of Wright, Nowak and Figueiredo for minimizing a sum f(x) + g(x) where f is smooth and g is convex, but possibly nonsmooth. We prove that if f is convex, then the error in the objective function at iteration k, for k sufficiently large, is bounded by a/(b+k) for suitable choices of a and b. Moreover, if the objective function is strongly convex, then the convergence is R-linear. An improved version of the algorithm based on a cycle version of the Barzilai-Borwein iteration and an adaptive line search is given. The performance of the algorithm is investigated using applications in the areas of signal processing and image reconstruction.
General Note: In the series University of Florida Digital Collections.
General Note: Includes vita.
Bibliography: Includes bibliographical references.
Source of Description: Description based on online resource; title from PDF title page.
Source of Description: This bibliographic record is available under the Creative Commons CC0 public domain dedication. The University of Florida Libraries, as creator of this bibliographic record, has waived all rights to it worldwide under copyright law, including all related and neighboring rights, to the extent allowed by law.
Statement of Responsibility: by Dung Phan.
Thesis: Thesis (Ph.D.)--University of Florida, 2010.
Local: Adviser: Hager, William W.

Record Information

Source Institution: UFRGP
Rights Management: Applicable rights reserved.
Classification: lcc - LD1780 2010
System ID: UFE0041470:00001

Permanent Link: http://ufdc.ufl.edu/UFE0041470/00001

Material Information

Title: Topics in Global Optimization Ellipsoidal Bisection, Graph Partitioning and Sparse Reconstruction
Physical Description: 1 online resource (100 p.)
Language: english
Creator: Phan, Dung
Publisher: University of Florida
Place of Publication: Gainesville, Fla.
Publication Date: 2010

Subjects

Subjects / Keywords: bound, branch, convex, global, graph, image, nonmonotone, nonsmooth, optimization, partitioning, quadratic, reconstruction, sparse, sublinear, underestimation
Mathematics -- Dissertations, Academic -- UF
Genre: Mathematics thesis, Ph.D.
bibliography   ( marcgt )
theses   ( marcgt )
government publication (state, provincial, terriorial, dependent)   ( marcgt )
born-digital   ( sobekcm )
Electronic Thesis or Dissertation

Notes

Abstract: In this dissertation, we develop theories and practical algorithms for optimization problems which arise from science, economics and engineering. First, an ellipsoidal branch and bound algorithm is developed for global optimization. The algorithm starts with a known ellipsoid containing the feasible set. Branching in the algorithm is accomplished by subdividing the feasible set using ellipses. Lower bounds are obtained by replacing the concave part of the objective function by an affine underestimate. A ball approximation algorithm, obtained by generalizing of a scheme of Lin and Han, is used to solve the convex relaxation of the original problem. Numerical experiments are given for a randomly generated quadratic objective function and randomly generated convex, quadratic constraints. Second, we propose an exact algorithm for solving the graph partitioning problem with upper and lower bounds on the size of each set in the partition. The algorithm is based on a branch and bound method applied to a continuous quadratic programming formulation of the problem. Lower bounds are achieved by decomposing the objective function into convex and concave parts and replacing the concave part by the best affine underestimate. It is shown that the best affine underestimate can be expressed in terms of the center and the radius of the smallest sphere containing the feasible set. The concave term is obtained either by a constant diagonal shift associated with the smallest eigenvalue of the objective function Hessian, or by a diagonal shift obtained by solving a semidefinite programming problem. Numerical results show that the proposed algorithm is competitive with state-of-the-art graph partitioning codes. Finally, the convergence rate is analyzed for the SpaSRA algorithm (Sparse Reconstruction by Separable Approximation) of Wright, Nowak and Figueiredo for minimizing a sum f(x) + g(x) where f is smooth and g is convex, but possibly nonsmooth. We prove that if f is convex, then the error in the objective function at iteration k, for k sufficiently large, is bounded by a/(b+k) for suitable choices of a and b. Moreover, if the objective function is strongly convex, then the convergence is R-linear. An improved version of the algorithm based on a cycle version of the Barzilai-Borwein iteration and an adaptive line search is given. The performance of the algorithm is investigated using applications in the areas of signal processing and image reconstruction.
General Note: In the series University of Florida Digital Collections.
General Note: Includes vita.
Bibliography: Includes bibliographical references.
Source of Description: Description based on online resource; title from PDF title page.
Source of Description: This bibliographic record is available under the Creative Commons CC0 public domain dedication. The University of Florida Libraries, as creator of this bibliographic record, has waived all rights to it worldwide under copyright law, including all related and neighboring rights, to the extent allowed by law.
Statement of Responsibility: by Dung Phan.
Thesis: Thesis (Ph.D.)--University of Florida, 2010.
Local: Adviser: Hager, William W.

Record Information

Source Institution: UFRGP
Rights Management: Applicable rights reserved.
Classification: lcc - LD1780 2010
System ID: UFE0041470:00001


This item has the following downloads:


Full Text

PAGE 1

TOPICSINGLOBALOPTIMIZATION:ELLIPSOIDALBISECTION,GRAPHPARTITIONINGANDSPARSERECONSTRUCTIONByDUNGTIENPHANADISSERTATIONPRESENTEDTOTHEGRADUATESCHOOLOFTHEUNIVERSITYOFFLORIDAINPARTIALFULFILLMENTOFTHEREQUIREMENTSFORTHEDEGREEOFDOCTOROFPHILOSOPHYUNIVERSITYOFFLORIDA2010 1

PAGE 2

c2010DungTienPhan 2

PAGE 3

Tomyfamily 3

PAGE 4

ACKNOWLEDGMENTS Thereareanumberofpeoplewhosesupportandexpertisehavegreatlycontributedtothecompletionofthiswork.Withoutthegeneroushelpoftheseindividuals,thisdissertationwouldnothavebeenpossible.Firstandforemost,Iwouldliketoexpressmysinceregratitudetomyadvisor,Dr.WilliamHager,forhissupport,invaluableguidanceandencouragementoverthecourseofmystudiesattheUniversityofFlorida.Ihavelearnedalotofprofessionalskillsfromhisimmenseknowledgeandinterminableenthusiasm.Ithasbeenafantasticpleasureandaprivilegeworkingwithhim.Ialsowouldliketothankmysupervisorycommitteemembers,Dr.PanosPardalos,Dr.YunmeiChen,Dr.BernardMairandDr.JayGopalakrishnanfortheircommentsandinsightsintomywork.Inparticular,Iamdeeplygratefultomyfamily.Iwanttothankmyparents,b^oKha'iandme.Kim,andyoungerbrother,Khoa,fortheirunconditionalloveandsupportthroughthislongandarduousprocess.Mostimportantly,Iwanttothankmywonderfulwife,Ngo.cDi^e.p,forherconstantsourceofencouragement,loveandunwaveringbeliefinme.Finally,IamthankfultotheUFFoundationfortheAlumniFellowshipthatsupportedmeformyrstfouryears.Inaddition,thisworkwaspartiallysupportedbytheNationalScienceFoundationunderGrant0619080and0620286. 4

PAGE 5

TABLEOFCONTENTS page ACKNOWLEDGMENTS ................................. 4 LISTOFTABLES ..................................... 7 LISTOFFIGURES .................................... 8 LISTOFSYMBOLS .................................... 9 ABSTRACT ........................................ 10 CHAPTER 1INTRODUCTION .................................. 12 1.1GlobalOptimization .............................. 12 1.2GraphPartitioningProblem .......................... 13 1.3Gradient-basedAlgorithmsforSparseRecovery ............... 14 2ANELLIPSOIDALBRANCHANDBOUNDALGORITHMFORGLOBALOPTIMIZATION ................................... 16 2.1Introduction ................................... 16 2.2EllipsoidalBisection .............................. 18 2.3BoundingProcedure .............................. 19 2.4PhaseOne .................................... 22 2.5BranchandBoundAlgorithm ......................... 23 2.6BallApproximationAlgorithmforConvexOptimization .......... 26 2.7NumericalExperiments ............................. 32 2.7.1RateofConvergenceforBAA ..................... 33 2.7.2ComparisonwithOtherAlgorithmsforProgramswithConvexCost 34 2.7.3ProblemswithNonconvexCost .................... 39 3ANEXACTALGORITHMFORGRAPHPARTITIONING ........... 42 3.1IntroductiontoGraphPartitioning ...................... 42 3.2ContinuousQuadraticProgrammingFormulation .............. 43 3.3ConvexLowerBoundsforObjectiveFunction ................ 46 3.3.1LowerBoundBasedonMinimumEigenvalue ............. 47 3.3.2LowerBoundBasedonSemideniteProgramming .......... 52 3.4BranchandBoundAlgorithm ......................... 54 3.5NecessaryandSucientOptimalityConditions ............... 57 3.6NumericalResults ................................ 63 3.6.1LowerBoundComparison ....................... 65 3.6.2AlgorithmPerformance ......................... 65 5

PAGE 6

4GRADIENT-BASEDMETHODSFORSPARSERECOVERY .......... 70 4.1IntroductiontoSpaRSAAlgorithm ...................... 70 4.2ConvergenceEstimateforConvexFunctions ................. 74 4.3ConvergenceEstimateforStronglyConvexFunctions ............ 78 4.4MoreGeneralReferenceFunctionValues ................... 80 4.5ComputationalExperiments .......................... 83 4.5.1`2)]TJ /F3 11.955 Tf 11.95 0 Td[(`1Problems ............................. 84 4.5.2ImageDeblurringProblems ....................... 85 4.5.3Group-separableRegularizer ...................... 88 4.5.4Total-variationPhantomReconstruction ............... 89 5CONCLUSIONSANDFUTURERESEARCHDIRECTIONS .......... 91 REFERENCES ....................................... 94 BIOGRAPHICALSKETCH ................................ 100 6

PAGE 7

LISTOFTABLES Table page 2-1Positivedenitecases ................................ 36 2-2Positivesemidenitecases .............................. 38 2-3Theperformanceofbranchandboundalgorithmform=2 ............ 41 2-4Theperformanceofbranchandboundalgorithmform=6 ............ 41 3-1Comparisonoftwolowerbounds .......................... 65 3-2Meshinstances .................................... 66 3-3Compilerdesign .................................... 66 3-4deBruijnnetworks .................................. 67 3-5Toroidalgrid ..................................... 67 3-6Mixedgridgraphs .................................. 68 3-7Planargrid ...................................... 68 3-8Randomlygeneratedgraphs ............................. 69 4-1Averageover10runsfor`2)]TJ /F3 11.955 Tf 11.95 0 Td[(`1problems ..................... 85 4-2Deblurringimages .................................. 86 4-3Total-variationphantomreconstruction ....................... 90 7

PAGE 8

LISTOFFIGURES Figure page 2-1KKTerrorversusiterationnumberforn=200,m=4,A0positivedenite .. 33 2-2KKTerrorversusiterationnumberforn=200,m=4,A0positivesemidenite 34 3-1Supposec62C ..................................... 50 3-2Branchandboundtree ................................ 56 4-1Numberofmatrix-vectormultiplicationsversuserror ............... 86 4-2Deblurringtheresolutionimage ........................... 87 4-3Deblurringthecameramanimage .......................... 87 4-4Group-separablereconstruction ........................... 88 4-5Phantomreconstruction ............................... 89 8

PAGE 9

LISTOFSYMBOLS,NOMENCLATURE,ORABBREVIATIONS Thelistshownbelowgivesabriefdescriptionofthemajormathematicalsymbolsdenedinthiswork. Rn Thespaceofrealn-dimensionalvectorskk TheEuclideannormofavectork Integer,oftenusedtodenotetheiterationnumberinanalgorithmx Rnvectorofunknownvariablesei Thei-thcolumnoftheidentitymatrixfk fk=f(xk)rf(x) Thegradientoffatx,arowvectorg(x) Acollumvector,thetransposeofthegradientatx,i.e.,g(x)=rf(x)Tgk gk=g(xk)A0 MeansthatAisapositivesemidenitematrixjSj Standsforthenumberofelements(cardinality)ofS,foranysetS(S) ThediameterofthesetS,i.e.,(S)=supfkx)]TJ /F6 11.955 Tf 11.95 0 Td[(yk:x;y2SgSc ThecomplementofS 9

PAGE 10

AbstractofDissertationPresentedtotheGraduateSchooloftheUniversityofFloridainPartialFulllmentoftheRequirementsfortheDegreeofDoctorofPhilosophyTOPICSINGLOBALOPTIMIZATION:ELLIPSOIDALBISECTION,GRAPHPARTITIONINGANDSPARSERECONSTRUCTIONByDungTienPhanMay2010Chair:WilliamW.HagerMajor:Mathematics Inthisdissertation,wedeveloptheoriesandpracticalalgorithmsforoptimizationproblemswhicharisefromscience,economicsandengineering. First,anellipsoidalbranchandboundalgorithmisdevelopedforglobaloptimization.Thealgorithmstartswithaknownellipsoidcontainingthefeasibleset.Branchinginthealgorithmisaccomplishedbysubdividingthefeasiblesetusingellipses.Lowerboundsareobtainedbyreplacingtheconcavepartoftheobjectivefunctionbyananeunderestimate.Aballapproximationalgorithm,obtainedbygeneralizingofaschemeofLinandHan,isusedtosolvetheconvexrelaxationoftheoriginalproblem.Numericalexperimentsaregivenforarandomlygeneratedquadraticobjectivefunctionandrandomlygeneratedconvex,quadraticconstraints. Second,weproposeanexactalgorithmforsolvingthegraphpartitioningproblemwithupperandlowerboundsonthesizeofeachsetinthepartition.Thealgorithmisbasedonabranchandboundmethodappliedtoacontinuousquadraticprogrammingformulationoftheproblem.Lowerboundsareachievedbydecomposingtheobjectivefunctionintoconvexandconcavepartsandreplacingtheconcavepartbythebestaneunderestimate.Itisshownthatthebestaneunderestimatecanbeexpressedintermsofthecenterandtheradiusofthesmallestspherecontainingthefeasibleset.TheconcavetermisobtainedeitherbyaconstantdiagonalshiftassociatedwiththesmallesteigenvalueoftheobjectivefunctionHessian,orbyadiagonalshiftobtainedbysolvingasemidenite 10

PAGE 11

programmingproblem.Numericalresultsshowthattheproposedalgorithmiscompetitivewithstate-of-the-artgraphpartitioningcodes. Finally,theconvergencerateisanalyzedfortheSpaSRAalgorithm(SparseReconstructionbySeparableApproximation)ofWright,NowakandFigueiredoforminimizingasumf(x)+ (x)wherefissmoothand isconvex,butpossiblynonsmooth.Weprovethatiffisconvex,thentheerrorintheobjectivefunctionatiterationk,forksucientlylarge,isboundedbya=(b+k)forsuitablechoicesofaandb.Moreover,iftheobjectivefunctionisstronglyconvex,thentheconvergenceisR-linear.AnimprovedversionofthealgorithmbasedonacycleversionoftheBarzilai-Borweiniterationandanadaptivelinesearchisgiven.Theperformanceofthealgorithmisinvestigatedusingapplicationsintheareasofsignalprocessingandimagereconstruction. 11

PAGE 12

CHAPTER1INTRODUCTION Themainfocusofthisdissertationisonthedevelopmentofmathematicaltheoryandcomputeralgorithmsforoptimizationproblemswhicharisefromscience,economicsandengineering.Ourresearchhasbeenconductedinthreeprincipalareas:globaloptimization,graphpartitioningandsparsereconstruction. 1.1GlobalOptimization Wehavedevelopedanellipsoidalbranchandboundalgorithmfortheglobaloptimizationoftheproblem minf(x)subjecttox2; whereRniscompactsetandf:Rn!Risaweaklyconvexfunction;thatis,f(x)+kxk2isconvexforsome0.Globaloptimizationhasavarietyofimportantapplications[ 48 ].Forexample,manyrealworldapplications,includingproblemsinplanningandscheduling,economicsofscale,andengineeringdesignandcontrolcanoftenbeformulatedasquadraticprogrammingproblemswhichisasubclassofthegeneralproblem. ThealgorithmstartswithaknownellipsoidEcontaining.ThebranchingprocessinthebranchandboundalgorithmisbasedonsuccessiveellipsoidalbisectionsoftheoriginalE.Alowerboundfortheobjectivefunctionvalueoveranellipseisobtainedbywritingfasthesumofaconvexandaconcavefunctionandreplacingtheconcavepartbyananeunderestimate.Thisunderestimatealongwiththeellipsoidalbisectionisusedtogetthelowerboundforthesolutionoftheoriginalproblem.Wedevelopanalgorithmtosolvethisconvexproblemwhichisbasedonthesuccessiveapproximationofthefeasiblesetbyballs,andwhichtiesinnicelywiththeellipsoidal-basedbranchandboundalgorithm.Theballapproximationalgorithmisabletoecientlyhandletheconvexrelaxationoftheproblem.Itisnumericallyshownthatthenovelalgorithmoutperformstheinterior-point 12

PAGE 13

andgradientprojectionmethodsforsolvingtheconvexproblem.Numericalexperimentsforproblemswithaquadraticobjectivefunctionandellipsoidalconstraintsdemonstratetheeciencyoftheglobalalgorithm. 1.2GraphPartitioningProblem Thegraphpartitioningproblemistondthesmallestsetofedgeswhich,whenremovedfromthegraph,splitverticesintotwosetswithagivennumberofelementsineachset.Theproblemhasmanypracticalapplicationssuchasparallelcomputing(taskassignment,loadbalancing),circuitboardandmicro-chipdesignandmoleculardynamicssimulations.In[ 37 ],HagerandKrylyukshowedthatthediscreteproblemcanbeformulatedasthefollowingcontinuousquadraticoptimizationproblem minimizef(x):=(1)]TJ /F6 11.955 Tf 11.96 0 Td[(x)T(A+D)xsubjectto0x1;l1Txu;(QP) whereAistheadjacencymatrixofthegraph,Disadiagonalmatrix,landuaregivenintegers.Ingeneral,thegraphpartitioningproblemisextremelydiculttosolvesinceitisoneoftheNPcompleteproblems.Manyheuristicmethodshavebeendevelopedintheliteraturewhichcanndanapproximatesolutionforlargescaleproblems.However,itisagreatchallengetocomeupwithexactmethodsforgeneralgraphsevenwitha100vertices[ 52 ].Wehavedevelopedaconvexquadraticbranchandboundalgorithmbyexploitingtherecentoptimization-basedformulation.Tightlowerboundsareobtainedbysolvingaconvexoptimizationproblemderivedfromthesolutionofasemideniteprogram.Weshowhowtotransformafeasiblexforthegraphpartitioning(QP)toabinaryfeasiblepointywithanobjectivefunctionvaluewhichsatisesf(y)f(x).Thebinaryfeasiblepointcorrespondstoapartitionofthegraphverticesandf(y)istheweightofthecutedges.Atanystationarypointof(QP)whichisnotalocalminimizer,wepresentatheoremthatprovidesadescentdirectionwhichcanbeusedtostrictlyimprovetheobjectivefunctionvalue. 13

PAGE 14

Numericalresults[ 40 ]showthattheproposedexactalgorithmiscompetitivewithstate-of-the-artgraphpartitioningmethods;therelativeperformanceofalgorithmisbetterforsparsegraphsthanfordensegraphs.Inaddition,forthequadraticprogramassociatedwiththegraphpartitioningproblem,wegiveanecessaryandsucientoptimalityconditionsforalocaloptimality.Theseconditionsrelatethegraphstructureandtherst-orderoptimalityconditionsatthegivenpoint.Theseresultsarestrikingsinceforageneralquadraticprogramming,decidingwhetheragivenpointisalocalminimizerisNP-hard. 1.3Gradient-basedAlgorithmsforSparseRecovery Manysignal/imagereconstructionproblemshavetheform minx2Rn(x):=f(x)+ (x);(SR) wheref:Rn!Risasmoothfunction,and :Rn!Risconvex.Thefunction ,usuallycalledtheregularizerorregularizationfunction,isniteforallx2Rn,butpossiblynonsmooth.Animportantapplicationof(SR),foundinthesignalprocessingliterature,isthewell-known`2)]TJ /F3 11.955 Tf 11.95 0 Td[(`1problem(calledbasispursuitdenoising) minx2Rn1 2kAx)]TJ /F6 11.955 Tf 11.96 0 Td[(bk22+kxk1; whereA2Rkn(usuallykn),b2Rk;2R,0,andkk1isthe1-norm. Recently,Wright,NowakandFigueiredo[ 86 ]introducedtheSparseReconstructionbySeparableApproximationalgorithm(SpaRSA)forsolvingtheproblem.Thealgorithmhasbeenshowntoworkwellinpractice.In[ 86 ],theauthorsestablishglobalconvergenceSpaRSA.In[ 41 ],weproveanestimateoftheforma=(b+k)fortheerrorintheobjectivefunctionwhenfisconvex.Ifisstronglyconvex,thentheconvergenceoftheobjectivefunctionandtheiteratesisatleastR-linear.AstrategyispresentedforimprovingtheperformanceofSpaRSAbasedonacyclicBarzilai-Borweinstep[ 20 ]andanadaptivechoice[ 42 ]forthereferencefunctionvalueinthelinesearch.Weprovetheexistenceof 14

PAGE 15

aconvergentsubsequenceofiteratesthatapproachesastationarypoint.Foraslightlystrongerversionofthereferencefunction,weshowthatsublinearorlinearconvergenceagainholdwhenisconvexorstronglyconvexrespectively.Inaseriesofnumericalexperimentsintheareasofsignalprocessingandimagereconstruction,itisshownthatanAdaptiveSpaRSAbasedonarelaxedchoiceofthereferencefunctionvalueandacyclicBarzilai-BorweiniterationoftenyieldsmuchfasterconvergenceforSpaRSA,especiallywhentheerrortoleranceissmall. 15

PAGE 16

CHAPTER2ANELLIPSOIDALBRANCHANDBOUNDALGORITHMFORGLOBALOPTIMIZATION 2.1Introduction Inthischapterwedevelopabranchandboundalgorithmfortheglobaloptimizationoftheproblem minf(x)subjecttox2;(P) whereRnisacompactsetandf:Rn!Risaweaklyconvexfunction[ 83 ];thatis,f(x)+kxk2isconvexforsome0.ThealgorithmstartswithaknownellipsoidEcontaining.ThebranchingprocessinthebranchandboundalgorithmisbasedonsuccessiveellipsoidalbisectionsoftheoriginalE.Alowerboundfortheobjectivefunctionvalueoveranellipseisobtainedbywritingfasthesumofaconvexandaconcavefunctionandreplacingtheconcavepartbyananeunderestimate.See[ 29 48 ]fordiscussionsconcerningglobaloptimizationapplications. Asaspecicapplicationofourglobaloptimizationalgorithm,weconsiderproblemswithaquadraticobjectivefunctionandwithquadratic,ellipsoidalconstraints.Globaloptimizationalgorithmsforproblemswithquadraticobjectivefunctionandquadraticconstraintsincludethosein[ 1 61 72 ].In[ 72 ]Raberconsidersproblemswithnonconvex,quadraticconstraintsandwithann-simplexenclosingthefeasibleregion.Hedevelopsabranchandboundalgorithmbasedonasimplicial-subdivisionofthefeasiblesetandalinearprogrammingrelaxationoverasimplextoestimatelowerbounds.Inasimilarsettingwithboxconstraints,Linderoth[ 61 ]developsabranchandboundalgorithminwhichthethefeasibleregionissubdividedusingtheCartesianproductoftwo-dimensionaltrianglesandrectangles.Explicitformulaefortheconvexandconcaveenvelopsofbilinearfunctionsovertrianglesandrectangleswerederived.ThealgorithmofLe[ 1 ]focusesonproblemwithconvexquadraticconstraints;Lagrangedualityisusedtoobtainlowerboundsfortheobjectivefunction,whileellipsoidalbisectionisusedtosubdividethefeasibleregion. 16

PAGE 17

Thechapterisorganizedasfollows.InSection 2.2 wereviewtheellipsoidalbisectionschemeof[ 1 ]whichisusedtosubdividethefeasibleregion.Section 2.3 developstheconvexunderestimateusedtoobtainalowerboundfortheobjectivefunction.Sincefisweaklyconvex,wecanwriteitasthesumofaconvexandconcavefunctions: f(x)=)]TJ /F3 11.955 Tf 5.48 -9.68 Td[(f(x)+kxk2+)]TJ /F2 11.955 Tf 5.48 -9.68 Td[()]TJ /F3 11.955 Tf 9.3 0 Td[(kxk2;(2.1) where0.AdecompositionofthisformisoftencalledaDC(dierenceconvex)decomposition(see[ 48 ]).Forexample,iffisaquadratic,thenwecouldtake=)]TJ /F1 11.955 Tf 11.29 0 Td[(minf0;1g; where1isthesmallesteigenvalueoftheHessianr2f.Theconcaveterm)]TJ /F3 11.955 Tf 9.3 0 Td[(kxk2in( 2.1 )isunderestimatedbyananefunction`whichleadstoaconvexunderestimatefLoffgivenby fL(x)=)]TJ /F3 11.955 Tf 5.48 -9.68 Td[(f(x)+kxk2+`(x):(2.2) WeminimizefLoverthesetE\toobtainalowerboundfortheobjectivefunctiononasubsetofthefeasibleset.Anupperboundfortheoptimalobjectivefunctionvalueisobtainedfromthebestfeasiblepointproducedwhencomputingthelowerbound,orfromanylocalalgorithmappliedtothisbestfeasiblepoint.Notethatweakconvexityforareal-valuedfunctionistheanalogueofhypomonotonicityforthederivativeoperator[ 18 49 70 ]. InSection 2.4 wediscussthephaseoneproblemofndingapointinwhichalsoliesintheellipsoidE.Section 2.5 givesthebranchandboundalgorithmandprovesitsconvergence.Section 2.6 focusesonthespecialcasewherefandareconvex.TheballapproximationalgorithmofLinandHan[ 59 60 ]forprojectingapointontoaconvexsetisgeneralizedtoreplacethenormobjectivefunctionbyanarbitraryconvexfunction.Numericalexperiments,reportedinSection 2.7 ,comparetheballapproximationalgorithmtoSEDUMI1.1aswellastogradientprojectionalgorithms.Wealsocomparethebranch 17

PAGE 18

andboundalgorithmtoaschemeofAn[ 1 ]inwhichthelowerboundisobtainedbyLagrangeduality. Throughoutthechapter,theinteriorofasetSisdenotedintS,while@Sistheboundary.Givenx;y2Rn,[x;y]isthelinesegmentconnectingxandy:[x;y]=f(1)]TJ /F3 11.955 Tf 11.95 0 Td[(t)x+ty:0t1g: Theopenlinesegment,whichexcludestheendsxandy,isdenoted(x;y).Thegradientrf(x)isarowvectorwith(rf(x))i=@f(x) @xi: 2.2EllipsoidalBisection Inthissection,wegiveabriefoverviewoftheellipsoidalbisectionschemeintroducedbyAn[ 1 ].ThisideaoriginatesfromtheellipsoidmethodforsolvingconvexoptimizationproblemsbyShor,NemirovskiandYudin[ 76 88 ].ConsideranellipsoidEwithcentercintheform E=fx2Rn:(x)]TJ /F6 11.955 Tf 11.96 0 Td[(c)TB)]TJ /F5 7.97 Tf 6.59 0 Td[(1(x)]TJ /F6 11.955 Tf 11.95 0 Td[(c)1g;(2.3) whereBisasymmetric,positivedenitematrix.Givenanonzerovectorv2Rn,thesetsH)]TJ /F1 11.955 Tf 10.41 1.79 Td[(=fx2E:vTxvTcgandH+=fx2E:vTxvTcg partitionEintotwosetsofequalvolume.Thecentersc+andc)]TJ /F1 11.955 Tf 10.98 1.8 Td[(andthematrixBoftheellipsoidsEofminimumvolumecontainingHaregivenasfollows:c=cd n+1;B=n2 n2)]TJ /F1 11.955 Tf 11.96 0 Td[(1B)]TJ /F1 11.955 Tf 13.24 8.08 Td[(2ddT n+1;d=Bv p vTBv: Asmentionedin[ 1 ],ifthenormalvalwayspointsalongthemajoraxisofE,thenanestedsequenceofbisectionsshrinkstoapoint. 18

PAGE 19

2.3BoundingProcedure Inthissection,weobtainananeunderestimate`fortheconcavefunctionkxk2ontheellipsoid E=fx2Rn:xTAx)]TJ /F1 11.955 Tf 11.96 0 Td[(2bTxg;(2.4) whereAisasymmetric,positivedenitematrix,b2Rn,and2R.Thesetofaneunderestimatesforkxk2isgivenby U=f`:Rn!R;`isane,)-222(kxk2`(x)forallx2Eg:(2.5) Thebestunderestimateisasolutionoftheproblem min`2Umaxx2E)]TJ /F14 11.955 Tf 11.29 9.68 Td[()]TJ /F2 11.955 Tf 5.48 -9.68 Td[(kxk2+`(x):(2.6) Theorem2.3.1. Asolutionof( 2:6 )is`(x)=)]TJ /F1 11.955 Tf 9.3 0 Td[(2cTx+,wherec=A)]TJ /F5 7.97 Tf 6.59 0 Td[(1bisthecenteroftheellipsoid,=2cT)-222(kk2,and =argmaxx2Ekx)]TJ /F6 11.955 Tf 11.95 0 Td[(ck2:(2.7) If(E)isthediameterofE,thenmin`2Umaxx2E)]TJ /F14 11.955 Tf 11.29 9.68 Td[()]TJ /F2 11.955 Tf 5.48 -9.68 Td[(kxk2+`(x)=(E)2 4: Proof. Tobegin,wewillshowthattheminimizationin( 2.6 )canberestrictedtoacompactset.Clearly,whencarryingouttheminimizationin( 2.6 ),weshouldrestrictourattentiontothose`whichtouchthefunctionh(x)=kxk2atsomepointinE.Lety2Edenotethepointofcontact.Sinceh(x)`(x)andh(y)=`(y),alowerboundfortheerrorh(x))]TJ /F3 11.955 Tf 11.96 0 Td[(`(x)overx2Eish(x))]TJ /F3 11.955 Tf 11.95 0 Td[(`(x)j`(x))]TJ /F3 11.955 Tf 11.95 0 Td[(`(y)j)-223(jh(x))]TJ /F3 11.955 Tf 11.95 0 Td[(h(y)j: 19

PAGE 20

IfMisthedierencebetweenthemaximumandminimumvalueofhoverE,thenwehave h(x))]TJ /F3 11.955 Tf 11.95 0 Td[(`(x)j`(x))]TJ /F3 11.955 Tf 11.95 0 Td[(`(y)j)]TJ /F3 11.955 Tf 17.93 0 Td[(M:(2.8) Anupperboundfortheminimumin( 2.6 )isobtainedbythefunction`0whichisconstantonE,withvalueequaltotheminimumofh(x)overx2E.IfwisapointwherehattainsitsminimumoverE,thenwehavemaxx2Eh(x))]TJ /F3 11.955 Tf 11.95 0 Td[(`0(x)=maxx2Eh(x))]TJ /F3 11.955 Tf 11.96 0 Td[(h(w)=M: Forx2E,wehave h(x))]TJ /F3 11.955 Tf 11.96 0 Td[(`(x)maxx2Eh(x))]TJ /F3 11.955 Tf 11.95 0 Td[(`(x)maxx2Eh(x))]TJ /F3 11.955 Tf 11.95 0 Td[(`0(x)=M(2.9) whenwerestrictourattentiontoanefunctions`whichachieveanobjectivefunctionvaluein( 2.6 )whichareatleastasgoodas`0.Combining( 2.8 )and( 2.9 )gives j`(x))]TJ /F3 11.955 Tf 11.96 0 Td[(`(y)j2M(2.10) when`achievesanobjectivefunctionvaluein( 2.6 )whichisatleastasgoodas`0.Thus,whenwecarryouttheminimizationin( 2.6 ),weshouldrestricttoanefunctionswhichtouchhatsomepointy2Eandwiththechangein`acrossEsatisfyingthebound( 2.10 )forallx2E.Thistellsusthattheminimizationin( 2.6 )canberestrictedtoacompactset,andthataminimizermustexist. Supposethat`attainstheminimumin( 2.6 ).LetzbeapointinEwhereh(x))]TJ /F3 11.955 Tf 12.01 0 Td[(`(x)achievesitsmaximum.ATaylorexpansionaroundx=zgives h(x))]TJ /F3 11.955 Tf 11.96 0 Td[(`(x)=h(z))]TJ /F3 11.955 Tf 11.95 0 Td[(`(z)+(rh(z))-221(r`)(x)]TJ /F6 11.955 Tf 11.96 0 Td[(z))-221(kx)]TJ /F6 11.955 Tf 11.96 0 Td[(zk2(2.11) 20

PAGE 21

sinceh(x)=kxk2.Since`2U,thesetgivenin( 2.5 ),wehaveh(x))]TJ /F3 11.955 Tf 12.35 0 Td[(`(x)0forallx2E,so( 2.11 )yields 0h(z))]TJ /F3 11.955 Tf 11.96 0 Td[(`(z)+(rh(z))-222(r`)(x)]TJ /F6 11.955 Tf 11.96 0 Td[(z))-221(kx)]TJ /F6 11.955 Tf 11.96 0 Td[(zk2(2.12) forallx2E.Bytherst-orderoptimalityconditionsforz,wehave(rh(z))-222(r`)(x)]TJ /F6 11.955 Tf 11.96 0 Td[(z)0 forallx2E.Itfollowsfrom( 2.12 )that0h(z))]TJ /F3 11.955 Tf 11.96 0 Td[(`(z))-222(kx)]TJ /F6 11.955 Tf 11.95 0 Td[(zk2; orh(z))]TJ /F3 11.955 Tf 11.95 0 Td[(`(z)kx)]TJ /F6 11.955 Tf 11.95 0 Td[(zk2 forallx2E.Sincethereexistsx2Esuchthatkx)]TJ /F6 11.955 Tf 11.95 0 Td[(zk(E)=2,wehave maxx2Eh(x))]TJ /F3 11.955 Tf 11.95 0 Td[(`(x)=h(z))]TJ /F3 11.955 Tf 11.96 0 Td[(`(z)(E)2=4:(2.13) Wenowobservethatforthespecicanefunction`giveninthestatementofthetheorem,( 2.13 )becomesanequality,whichimpliestheoptimalityof`in( 2.6 ).ExpandinaTaylorseriesaroundx=c,wherec=A)]TJ /F5 7.97 Tf 6.58 0 Td[(1bisthecenteroftheellipsoidE,toobtainh(x)=kck2)]TJ /F1 11.955 Tf 11.96 0 Td[(2cT(x)]TJ /F6 11.955 Tf 11.95 0 Td[(c))-222(kx)]TJ /F6 11.955 Tf 11.95 0 Td[(ck2=)]TJ /F1 11.955 Tf 9.29 0 Td[(2cTx+kck2)-222(kx)]TJ /F6 11.955 Tf 11.96 0 Td[(ck2: Hence,for`,wehaveh(x))]TJ /F3 11.955 Tf 11.96 0 Td[(`(x)=kck2)]TJ /F3 11.955 Tf 11.95 0 Td[()-222(kx)]TJ /F6 11.955 Tf 11.95 0 Td[(ck2=k)]TJ /F6 11.955 Tf 11.95 0 Td[(ck2)-222(kx)]TJ /F6 11.955 Tf 11.96 0 Td[(ck2=maxy2Eky)]TJ /F6 11.955 Tf 11.96 0 Td[(ck2)-222(kx)]TJ /F6 11.955 Tf 11.96 0 Td[(ck2: Clearly,h(x))]TJ /F3 11.955 Tf 12.05 0 Td[(`(x)0forallx2E,andthemaximumoverx2Eisattainedatx=c.Moreover,h(c))]TJ /F3 11.955 Tf 11.95 0 Td[(`(c)=maxy2Eky)]TJ /F6 11.955 Tf 11.96 0 Td[(ck2=(E)2=4: 21

PAGE 22

Consequently,( 2.13 )becomesanequalityfor`=`,whichimpliestheoptimalityof`in( 2.6 ). ToevaluatethebestaneunderestimategivenbyTheorem 2.3.1 ,weneedtosolvetheoptimizationproblem( 2.7 ).Thisamountstondingthemajoraxisoftheellipsoid.Thesolutionis=c+sy; whereyisauniteigenvectorofAassociatedwiththesmallesteigenvalue,andsischosensothatliesontheboundaryoftheE.FromthedenitionofE,weobtains=p (cTAc+)=: WeminimizethefunctionfLin( 2.2 )overE\,with`thebestaneunderestimateofkxk2,toobtainalowerboundfortheobjectivefunctionoverE\.Anupperboundfortheoptimalobjectivefunctionvalueisobtainedbystartinganylocaloptimizationalgorithmfromthebestiterategeneratedduringthecomputationofthelowerbound.Forthenumericalexperimentsreportedlater,thegradientprojectionalgorithm[ 42 ]isthelocaloptimizationalgorithm.Ofcourse,byusingafasterlocalalgorithm,theoverallspeedoftheglobaloptimizationalgorithmwillincrease. 2.4PhaseOne Ineachstepofthebranchandboundalgorithmfor(P),weneedtosolveaproblemoftheform minf(x)subjecttox2E\;(2.14) inthespecialcasewherefisconvex(thefunctionfLin( 2.2 ))andEisanellipsoid.Inordertosolvethisproblem,weoftenneedtondafeasiblepoint.Oneapproachforndingafeasiblepointistoconsidertheminimizationproblem minxTAx)]TJ /F1 11.955 Tf 11.96 0 Td[(2bTxsubjecttox2;(2.15) 22

PAGE 23

whereAandbareassociatedwiththeellipsoidEin( 2.4 ).Assumingweknowafeasiblepointx02,wecouldapplyanoptimizationalgorithmto( 2.15 ).Iftheobjectivefunctionvaluecanbereducedbelow,thenweobtainapointinE.Iftheoptimalobjectivefunctionvalueisstrictlylargerthan,thentheproblem( 2.14 )isinfeasible. Ifthesetisitselftheintersectionofellipsoids,thentheprocedurewehavejustdescribedcouldbeusedinarecursivefashiontodetermineafeasiblepointforeitherorE\,ifitexists.Inparticular,suppose=\mj=1Ejistheintersectionofmellipsoids,whereEj=fx2Rn:xTAjx)]TJ /F1 11.955 Tf 11.95 0 Td[(2bTjxjg: Apointx12E1isreadilydetermined.Proceedingbyinduction,supposethatwehaveapointxk)]TJ /F5 7.97 Tf 6.58 0 Td[(12\k)]TJ /F5 7.97 Tf 6.58 0 Td[(1j=1Ej.AnygloballyconvergentiterativemethodisappliedtotheconvexoptimizationproblemminxTAkx)]TJ /F1 11.955 Tf 11.96 0 Td[(2bTkxsubjecttox2\k)]TJ /F5 7.97 Tf 6.59 0 Td[(1j=1Ej: Iftheobjectivefunctionvalueisreducedbelowk,thenafeasiblepointin\kj=1Ejhasbeendetermined.Conversely,iftheoptimalobjectivefunctionvalueisabovek,then\kj=1Ejisempty. 2.5BranchandBoundAlgorithm Ourbranchandboundalgorithmispatternedafterageneralbranchandboundalgorithm,asappearsin[ 48 ]forexample.ForanyellipseE,dene ML(E)=minffL(x):x2E\g;(2.16) wherefListhelowerbound( 2.2 )correspondingtothebestaneunderestimateofkxk2onE.Weassumethatanalgorithmisavailabletosolvetheoptimizationproblem( 2.16 ). Ellipsoidalbranchandboundwithlinearunderestimate(EBL) 1. LetE0beanellipsoidwhichcontainsandsetS0=fE0g. 23

PAGE 24

2. EvaluateML(E0)andletx02denotethefeasiblepointgeneratedduringtheevaluationofML(E0)withthesmallestfunctionvalue. 3. Fork=0;1;2;::: (a) ChooseEk2SksuchthatML(Ek)=minfML(E):E2Sg.BisectEk2SkwithtwoellipsoidsdenotedEk1andEk2(seeSection 2.2 ).EvaluateML(Ek1)andML(Ek2). (b) Letxk+1denoteafeasiblepointassociatedwiththesmallestfunctionvaluethathasbeengenerateduptothisiterationanduptothisstep.Hence,ifyk1andyk2aresolutionsto( 2.16 )associatedwithE=Ek1andE=Ek2respectively,thenwehavef(xk+1)f(yki),i=1;2. (c) SetSk+1=fE2Sk[fEk1g[fEk2g:ML(E)f(xk+1);E6=Ekg Theorem2.5.1. Supposethatthefollowingconditionshold: A1. ThefeasiblesetiscontainedinsomegivenellipsoidE,iscompact,andfisweaklyconvexoverE. A2. Anestedsequenceofellipsoidalbisectionsshrinkstoapoint(seeSection 2.2 ). Theneveryaccumulationpointofthesequencexkisasolutionof(P). Proof. Letydenoteanyglobalminimizerfor(P).Wenowshowthatforeachk,thereexistsE2Skwithy2E.SinceE0,y2E0.Proceedingbyinduction,supposethatforeachj,0jk,thereexistsanellipsoidFj2Sjwithy2Fj.WenowwishtoshowthatthereexistFk+12Sk+1withy2Fk+1.InStep3c,Fk2SkcanonlybedeletedfromSk+1ifML(Fk)>f(xk+1)orFk=Ek.TheformercasecannotoccursinceML(Fk)f(y)f(xk+1); duetotheglobaloptimalityofy.IfFk=Ek,thenyliesineitherEk1orEk2.Ify2Eki,thenEki2Sk+1sinceML(Eki)f(y)f(xk+1): 24

PAGE 25

Letxdenoteanaccumulationpointofthesequencexk.Sinceisclosedandxk2foreachk,x2.By[ 83 ,Prop.4.4],aweaklyconvexfunctionislocallyLipschitzcontinuous.Hence,fiscontinuousonandf(xk)approachesf(x).Ifxisasolutionof(P),thentheproofiscomplete.Otherwise,f(y)
PAGE 26

2.6BallApproximationAlgorithmforConvexOptimization Inthissectionwegiveanalgorithmtosolve(P)inthespecialcasethatfandareconvex.Thisalgorithm,whichisbasedonthesuccessiveapproximationofthefeasiblesetbyballs,tiesinnicelywiththeellipsoidal-basedbranchandboundalgorithm.Thealgorithmisageneralizationoftheballapproximationalgorithm[ 60 ]ofLinandHan.ThealgorithmofLinandHandealswiththespecialcasewheretheobjectivefunctionhastheformkx)]TJ /F6 11.955 Tf 9.74 0 Td[(ak2andisanintersectionofellipsoids.Lingeneralizesthisalgorithmin[ 59 ]totreatconvexconstraints.Theanalysisin[ 59 60 ]istightlycoupledtothenormobjectivefunction.InourfurthergeneralizationoftheLin/Hanalgorithm,thenormobjectivefunctionisreplacedbyanarbitraryconvexfunctionalfandanadditionalconstraintsetRnisincluded,whichmightrepresentboundconstraintsforexample.Moreprecisely,weconsidertheproblem minf(x)subjecttox2F:=fx2:g(x)0g;(C) wheref:Rn!R,g:Rn!Rm,andthefollowingconditionshold: C1. fandgareconvexanddierentiable,isclosedandconvex,andFiscompact. C2. Thereexistsxintherelativeinteriorofwithg(x)<0. C3. Thereexists>0suchthatkrgi(x)kwhengi(x)=0forsomei2[1;m]andx2. TheconditionC2isreferredtoastheSlatercondition. Wewillgiveanewanalysiswhichhandlesthismoregeneralconvexproblem(C).IneachiterationofLin'salgorithmin[ 59 ],theconvexconstraintsareapproximatedbyballconstraints.Leth:Rn!Rbeaconvex,dierentiablefunctionwhichdenesaconvex,nonemptysetH=fx2Rn:h(x)0g: 26

PAGE 27

TheballapproximationBh(x)atx2Hisexpressedintermsofacentermapc:H!Rnandaradiusmapr:H!R:Bh(x)=fy2Rn:ky)]TJ /F6 11.955 Tf 11.95 0 Td[(c(x)kr(x)g: Thesetwomapsmustsatisfythefollowingconditions: B1. BothcandrarecontinuousonH. B2. Ifh(x)<0,thenx2intBh(x),theinteriorofBh(x). B3. Ifh(x)=0,thenx2@Bh(x),andc(x)=x)]TJ /F3 11.955 Tf 11.95 0 Td[(rh(x)Tforsomexed>0: MapswhichsatisfyB1,B2,andB3arethefollowing,assuminghiscontinuouslydierentiable:c(x)=x)]TJ /F3 11.955 Tf 11.96 0 Td[(rh(x)T;r(x)=krh(x)k)]TJ /F3 11.955 Tf 20.59 0 Td[(h(x); whereandarexedpositivescalars. Letciandridenotecenterandradiusmapsassociatedwithgi,letBibetheassociatedballgivenbyBi(x)=fy2Rn:ky)]TJ /F6 11.955 Tf 11.95 0 Td[(ci(x)kri(x)g; anddeneB(x)=\mi=1Bi(x).OurgeneralizationofthealgorithmofLinandHanisthefollowing: Ballapproximationalgorithm(BAA) 1. Letx0beafeasiblepointfor(C). 2. Fork=0;1;::: (a) Letykbeasolutionoftheproblem minf(x)subjecttox2\B(xk):(2.21) (b) Setxk+1=x(k)wherex()=(1)]TJ /F3 11.955 Tf 12.24 0 Td[()xk+ykandkisthelargest2[0;1]suchthatx()2Fforall2[0;]. 27

PAGE 28

In[ 59 ,Lem.3.1]itisshownthatintB(x)6=;foreachx2FwhenthecenterandradiusmapsciandrisatisfyB2andB3andthereexistsxsuchthatg(x)<0.Lin'sproofisbasedonthefollowingobservation:For>0sucientlysmall,x+(x)]TJ /F6 11.955 Tf 12.16 0 Td[(x)liesintheinteriorofBi(x)foreachi.InC2wealsoassumethatx2ri,where\ri"denotesrelativeinterior.Hence,for>0sucientlysmall,x+(x)]TJ /F6 11.955 Tf 11.21 0 Td[(x)liesinbothriandintheinteriorofBi(x)foreachi.Consequently,wehave ri\intB(x)6=;foreveryx2F:(2.22) Thisimpliesthatthesubproblems( 2.21 )ofBAAarealwaysfeasible.Anoptimalsolutionykexistsduetothecompactnessofthefeasiblesetandthecontinuityoftheobjectivefunction. Theorem2.6.1. IfC1,C2,andC3holdandthecentermapciandtheradiusmaprisatisfyB1,B2,andB3,i=1;2;:::;m,thenthelimitxofanyconvergentsubsequenceofiteratesxkofBAAisasolutionof(C). Proof. Initially,x02F.Proceedingbyinduction,itfollowsfromthelinesearchinStep2aofBAAthatxk2Fforeachk.ByB2andB3,x2Bh(x)ifh(x)0.Consequently,xk2\B(xk)foreachk.Thisshowsthatxkisfeasiblein( 2.21 )foreachk,andtheminimizerykin( 2.21 )satises f(yk)f(xk)foreachk:(2.23) Bytheconvexityoffandby( 2.23 ),wehave f(xk+1)kf(yk)+(1)]TJ /F3 11.955 Tf 11.96 0 Td[(k)f(xk)f(xk);(2.24) wherek2[0;1]isdenedinStep2bofBAA.Hence,f(xk)approachesalimitmonotonically.SinceFiscompactandxk2Fforeachk,anaccumulationpointx2Fexists.Sincethecentermapsciandtheradiusmapsriarecontinuous,theballsBi(xk)areuniformlybounded,andhence,theykarecontainedinboundedset.Lety 28

PAGE 29

denoteanaccumulationpointoftheyk.Tosimplifytheexposition,let(xk;yk)denoteaprunedversionoftheoriginalsequencewhichapproachesthelimit(x;y). Wenowshowthat y=argminf(x)subjecttox2\B(x):(2.25) Suppose,tothecontrary,thatthereexists~y2\B(x)suchthatf(~y)0issmallenoughthat^y2ri,^y2intBi(x)foreachi,andf(^y)f(~y),wecontradicttheoptimalityofykin( 2.21 ).Thisestablishes( 2.25 ). Again,byB2andB3,xisfeasiblein( 2.25 ).Sinceyisoptimalin( 2.25 ),wehavef(y)f(x).Wewillshowthat f(y)=f(x):(2.26) Suppose,tothecontrary,thatf(y)0: ByaTaylorexpansionaroundx,weseethatthereexisti2(0;1)suchthat gi(x+(y)]TJ /F6 11.955 Tf 11.96 0 Td[(x))<0forall2(0;i]:(2.27) (ii) gi(x)<0:Inthiscase,theretriviallyexistsi2(0;1)suchthat( 2.27 )holds. 29

PAGE 30

Letbetheminimumofi,1im.Bytheconvexityoff,wehave f(x+(y)]TJ /F6 11.955 Tf 11.96 0 Td[(x))f(x)+(f(y))]TJ /F3 11.955 Tf 11.95 0 Td[(f(x))
PAGE 31

Sinceyisasolutionof( 2.25 )andtheSlatercondition( 2.22 )holds,therst-orderoptimalityconditionholdsaty.Thatis,thereexist2Rmsuchthat 0;i(ky)]TJ /F6 11.955 Tf 11.95 0 Td[(ci(x)k2)]TJ /F3 11.955 Tf 11.96 0 Td[(r2i(x))=0;i=1;2;:::;m;rxL(;y)(x)]TJ /F6 11.955 Tf 11.96 0 Td[(y)0forallx2:9>=>;(2.31) Ifrf(y)(x)]TJ /F6 11.955 Tf 11.98 0 Td[(y)0forallx2,thenyistheglobalminimizeroftheconvexfunctionfover.Sincef(y)=f(x)by( 2.26 ),itfollowsthatxisasolutionof(C),andtheproofwouldbecomplete.Hence,wesupposethatrf(y)(x)]TJ /F6 11.955 Tf 12.42 0 Td[(y)<0forsomex2,whichimpliesthat6=0by( 2.31 ). Sincefisconvex,wehave f(x)f(y)+rf(y)(x)]TJ /F6 11.955 Tf 11.96 0 Td[(y):(2.32) Weexpandtheexpression1 2mXi=1i)]TJ /F2 11.955 Tf 5.48 -9.68 Td[(kx)]TJ /F6 11.955 Tf 11.95 0 Td[(ci(x)k2)]TJ /F3 11.955 Tf 11.96 0 Td[(r2i(x) inaTaylorseriesaroundx=yandevaluateatx=xtoobtain1 2mXi=1i)]TJ /F2 11.955 Tf 5.48 -9.68 Td[(kx)]TJ /F6 11.955 Tf 11.95 0 Td[(ci(x)k2)]TJ /F3 11.955 Tf 11.96 0 Td[(r2i(x)=1 2mXi=1i)]TJ /F2 11.955 Tf 5.48 -9.68 Td[(ky)]TJ /F6 11.955 Tf 11.95 0 Td[(ci(x)k2)]TJ /F3 11.955 Tf 11.96 0 Td[(r2i(x)+mXi=1i(y)]TJ /F6 11.955 Tf 11.96 0 Td[(ci(x))T(x)]TJ /F6 11.955 Tf 11.95 0 Td[(y)+1 2kx)]TJ /F6 11.955 Tf 11.95 0 Td[(yk2mXi=1i: Weaddthisequationto( 2.32 )toobtain L(x;)L(y;)+rxL(y;)(x)]TJ /F6 11.955 Tf 11.96 0 Td[(y)+1 2kx)]TJ /F6 11.955 Tf 11.95 0 Td[(yk2mXi=1i:(2.33) 31

PAGE 32

Bycomplementaryslacknessandby( 2.26 ),wehaveL(y;)=f(y)=f(x).Hence,( 2.33 )yields 1 2kx)]TJ /F6 11.955 Tf 11.95 0 Td[(yk2mXi=1irxL(y;)(x)]TJ /F6 11.955 Tf 11.95 0 Td[(y)+1 2mXi=1i)]TJ /F2 11.955 Tf 5.48 -9.68 Td[(kx)]TJ /F6 11.955 Tf 11.95 0 Td[(ci(x)k2)]TJ /F3 11.955 Tf 11.96 0 Td[(r2i(x): (2.34) By( 2.31 )andthefactthatx2,wehaverxL(y;)(x)]TJ /F6 11.955 Tf 12.13 0 Td[(y)0.Sincex2B(x),thelasttermin( 2.34 )isnonpositive.Hence,theentirerightsideof( 2.34 )isnonpositive.Since0and6=0,( 2.34 )impliesthaty=x. Replacingybyxintherst-orderconditions( 2.31 )gives rf(x)+mXi=1i(x)]TJ /F6 11.955 Tf 11.95 0 Td[(ci(x))T!(x)]TJ /F6 11.955 Tf 11.96 0 Td[(x)0forallx2:(2.35) Ifgi(x)<0,thenbyB2,x2intBi(x)andi=0bycomplementaryslackness.Ifgi(x)=0,thenbyB3,ci(x)=x)]TJ /F3 11.955 Tf 11.96 0 Td[(rgi(x)T.Withthesesubstitutions,( 2.35 )yields rf(x)+mXi=1irgi(x)!(x)]TJ /F6 11.955 Tf 11.96 0 Td[(x)0forallx2: Hence,therst-orderoptimalityconditionsfor(C)aresatisedatx.Sincetheobjectivefunctionandtheconstraintsof(C)areconvex,xisasolutionof(C).Thiscompletestheproof. 2.7NumericalExperiments Weinvestigatetheperformanceofthealgorithmsoftheprevioussectionsusingrandomlygeneratedquadraticallyconstrainedquadraticprogrammingproblemsoftheform minxTA0x+bT0xsubjecttog(x)0;(QPP) wherex2Rnandgi(x)=xTAix+bTix+ci,i=1;2;:::;m.Herebi2Rnandciisascalarforeachi.ThematricesAiaresymmetric,positivedenitefori1.Inourexperimentswiththeballapproximationalgorithm,wetakeA0symmetric,positive 32

PAGE 33

semidenite.Inourexperimentswiththebranchandboundalgorithm,weconsidermoregeneralindeniteA0.ThecodesarewrittenineitherCorFortran.TheexperimentswereimplementedusingaMatlab7.0.1interfaceonaPCwith2GBmemoryandIntelCore2Duo2GhzprocessorsrunningtheWindowsVistaoperatingsystem. 2.7.1RateofConvergenceforBAA ThetheoryofSection 2.6 establishestheconvergenceofBAA.Experimentally,weobservethattheconvergencerateislinear.Figure 2-1 showsthatthebehavioroftheKKTerrorasafunctionoftheiterationnumberforarandomlygeneratedpositivedenitematrixA0ofdimension200andfor4ellipsoidalconstraints(m=4).TheKKTerroris Figure2-1. KKTerrorversusiterationnumberforn=200,m=4,A0positivedenite computedusingtheformulagiveninSection4of[ 36 ].Roughly,thisformulaamountstotheinnitynormofthegradientoftheLagrangianplustheinnitynormoftheviolationincomplementaryslackness.IfA0isconstructedtohavepreciselyonezeroeigenvalue,thentheconvergencerateagainappearstobelinear,asseeninFigure 2-2 33

PAGE 34

Figure2-2. KKTerrorversusiterationnumberforn=200,m=4,A0positivesemidenite 2.7.2ComparisonwithOtherAlgorithmsforProgramswithConvexCost Togainsomeinsightintotherelativeperformanceoftheballapproximationalgorithm(BAA),wesolvedrandomlygeneratedproblemswithconvexcostusingthreeotheralgorithms: SEDUMI,foroptimizationoversymmetriccones. Thegradientprojectionalgorithm.Wetriedboththenonmonotonegradientprojectalgorithm(NGPA)givenin[ 42 ]andthenonmonotonespectralprojectedgradientmethod(SPG)ofBirgin,Martnez,andRaydan[ 6 7 ](ACMAlgorithm813). Wenowdiscussindetailhoweachofthesealgorithmswasimplemented.TheBAAsubproblems( 2.21 )havetheform minxTA0x+bT0xsubjecttokx)]TJ /F6 11.955 Tf 11.96 0 Td[(cik2r2i;1im:(2.36) Wesolvethesesubproblemsbyapplyingtheactivesetalgorithm(ASA)developedin[ 42 ]tothedualproblem.Tofacilitatetheevaluationofthedualfunction,wecomputethediagonalizationA0=QDQTwhereDisdiagonalandQisorthogonal.Substituting 34

PAGE 35

x=Qyin( 2.36 )yieldstheequivalentproblemminyTDy+bT0Qysubjecttoky)]TJ /F6 11.955 Tf 11.95 0 Td[(QTcik2r2i;1im: Thedualproblemis max0miny2RnyTDy+bT0Qy+mXi=1i)]TJ /F2 11.955 Tf 5.48 -9.68 Td[(ky)]TJ /F6 11.955 Tf 11.96 0 Td[(QTcik2)]TJ /F3 11.955 Tf 11.96 0 Td[(r2i:(2.37) Thei-thcomponentofthegradientofthedualfunctionwithrespecttoissimplyky())]TJ /F6 11.955 Tf 12.12 0 Td[(QTcik2)]TJ /F3 11.955 Tf 12.12 0 Td[(r2iwherey()achievestheminimumin( 2.37 ).Thisminimumiseasilyevaluatedsincethequadratictermintheobjectivefunctionisdiagonal. SEDUMIcouldbeapplieddirectlyto(QPP)whenthecostfunctionisstronglyconvex.WeusedVersion1.1ofthecodeobtainedfrom http://sedumi.mcmaster.ca/ Inimplementingthegradientprojectionalgorithmfor(QPP),weneedtoprojectavectorontothefeasibleset.Thisamountstosolvingaproblemoftheformminkx)]TJ /F6 11.955 Tf 11.96 0 Td[(ak2subjecttog(x)0: WesolvedthisproblemusingBAA.AniterationofBAAreducestothesolutionofaproblemwiththefollowingstructure: minkx)]TJ /F6 11.955 Tf 11.96 0 Td[(ak2subjecttokx)]TJ /F6 11.955 Tf 11.95 0 Td[(cik2r2i;i=1;2;:::;m:(2.38) Asin[ 59 ],wesolvetheseproblemsbyformingthedualproblemmax0minx2Rnkx)]TJ /F6 11.955 Tf 11.95 0 Td[(ak2+mXi=1i)]TJ /F2 11.955 Tf 5.48 -9.68 Td[(kx)]TJ /F6 11.955 Tf 11.96 0 Td[(cik2)]TJ /F3 11.955 Tf 11.96 0 Td[(r2i: Aftercarryingouttheinnerminimization,thisreducesto max0)]TJ 10.49 8.15 Td[(ka+Pmi=1icik2 1+Pmi=1i+mXi=1i(kcik2)]TJ /F3 11.955 Tf 11.95 0 Td[(r2i):(2.39) 35

PAGE 36

Table2-1. Positivedenitecases n;m SED success BAA success NGPA success SPG success time 0.52 0.07 4.70 5.94 100,4 iter 10.06 28 19.00 30 172.43 17 200.06 17 time 2.75 0.32 10.68 11.76 200,4 iter 9.56 26 12.70 30 203.23 21 348.36 20 time 8.14 0.72 128.19 122.23 300,4 iter 9.60 27 21.46 30 269.86 20 431.83 20 time 20.13 1.64 404.28 438.84 400,4 iter 10.26 27 49.26 29 352.66 18 545.23 18 time 44.07 2.54 579.21 647.56 500,4 iter 12.33 28 29.90 30 369.20 15 574.13 13 time 57.28 4.27 648.79 611.60 600,4 iter 9.30 26 36.80 29 309.43 19 306.50 19 time 3.51 0.08 86.89 81.67 100,40 iter 10.26 28 19.00 30 150.76 21 165.66 21 time 26.56 0.32 268.63 250.22 200,40 iter 12.70 30 12.70 30 199.70 17 218.50 16 time 54.56 0.81 732.50 727.92 200,100 iter 10.66 30 9.50 30 295.80 20 327.26 20 time 23.84 0.72 579.62 530.02 100,200 iter 14.96 30 20.06 30 249.43 18 261.46 19 time 0.093 0.002 3.02 2.75 4,100 iter 9.06 29 6.96 30 19.46 26 19.40 25 time 0.114 0.004 6.23 5.70 4,200 iter 9.56 27 8.73 30 16.26 26 16.46 26 time 0.148 0.012 13.45 11.58 4,300 iter 11.06 25 12.56 30 15.26 24 15.33 23 time 0.195 0.014 16.27 12.87 4,400 iter 13.33 28 12.26 30 16.26 28 15.70 28 time 0.221 0.017 21.08 18.16 4,500 iter 13.83 26 11.50 30 13.83 26 13.83 26 time 0.235 0.018 31.65 34.83 4,600 iter 12.13 26 11.00 30 15.40 24 16.33 24 Ifsolvesthedualproblem( 2.39 ),thentheassociatedsolutionoftheprimalproblem( 2.38 )isx=a+Pmi=1ici 1+Pmi=1i: Again,thedualproblem( 2.39 )issolvedusingtheactivesetalgorithm(ASA)of[ 42 ]. ThetestproblemsusedinTables 2-1 and 2-2 weregeneratedasfollows:LetRand(n;l;u)denoteavectorinRnwhoseentriesarechosenrandomlyintheinterval 36

PAGE 37

(l;u).RandompositivedenitematricesAaregeneratedusingtheproceduregivenin[ 60 ],whichwenowsummarize.Letwi2Rand(n;)]TJ /F1 11.955 Tf 9.3 0 Td[(1;1)fori=1,2,3,anddeneQi=I)]TJ /F1 11.955 Tf 11.95 0 Td[(2vivTi;vi=wi=kwik: LetDbeadiagonalmatrixwithdiagonalinRand(n;0;100).Finally,A=UDUTwithU=Q1Q2Q3.Toobtainarandomlygeneratedpositivesemidenitematrix,weusethesameprocedure,however,werandomlysetonediagonalelementofDtozero. Wemakeaspecialchoiceforcitoensurethatthefeasiblesetfor(QPP)isnonempty.Werstgeneratep2Rand(n;)]TJ /F1 11.955 Tf 9.3 0 Td[(50;50)andwesetci=)]TJ /F1 11.955 Tf 9.3 0 Td[((pTAip+bTip+si); wheresiisrandomlygeneratedintheinterval[0;10]andbi2Rand(n;)]TJ /F1 11.955 Tf 9.3 0 Td[(100;100).Withthischoiceforci,thefeasiblesetfor(QPP)isnonemptysincepliesintheinteriorofthefeasibleset.Thestoppingcriterioninourexperimentswas kP(xk)]TJ /F6 11.955 Tf 11.96 0 Td[(gk))]TJ /F6 11.955 Tf 11.95 0 Td[(xkk10)]TJ /F5 7.97 Tf 6.58 0 Td[(4;(2.40) wherePdenotesprojectionintothefeasiblesetfor(QPP)andgk=2A0xk+b0isthegradientoftheobjectivefunctionatxk.Whenthecostisconvex,theleftsideof( 2.40 )vanishesifandonlyifxkisasolutionof(QPP). Tables 2-1 and 2-2 reporttheaverageCPUtimeinseconds(time),theaveragenumberofiterations(iter),andthenumberofsuccessesin30randomlygeneratedtestproblems.Thealgorithmwasconsideredsuccessfuliftheerrortolerance( 2.40 )wassatised. Basedonournumericalexperiments,itappearsthatBAAcanachieveanerrortoleranceontheorderofthesquarerootofthemachineepsilon[ 35 80 ],similartothecomputingprecisionwhichisachievedbyinteriorpointmethodsforlinearprogrammingpriortosimplexcrossover.Theconvergencetolerance( 2.40 )waschosensinceitseemsto 37

PAGE 38

Table2-2. Positivesemidenitecases n;m BAA success NGPA success SPG success time 0.11 15.04 17.38 100,4 iter 42.16 30 328.53 22 408.20 19 time 0.55 44.32 45.82 200,4 iter 99.33 30 313.10 20 356.43 22 time 1.02 290.19 304.89 300,4 iter 74.23 30 374.13 22 417.60 21 time 2.28 501.14 572.63 400,4 iter 111.83 30 404.66 19 492.83 19 time 5.37 620.13 657.61 500,4 iter 200.03 27 382.66 16 478.60 17 time 0.61 356.40 321.92 100,40 iter 82.30 30 276.56 22 237.23 21 time 2.74 398.54 415.28 200,40 iter 127.23 30 369.43 17 416.060 17 time 3.19 1030.02 949.29 100,200 iter 108.63 28 311.40 19 352.23 18 time 0.054 16.74 14.25 4,100 iter 38.66 30 31.13 16 31.23 16 time 0.075 44.74 33.24 4,200 iter 43.46 27 26.20 13 23.36 18 time 0.076 111.03 100.38 4,300 iter 31.50 29 29.33 14 28.60 11 time 0.049 205.17 237.62 4,400 iter 36.73 29 27.20 18 31.23 18 time 0.065 229.77 247.82 4,500 iter 41.86 28 24.46 16 26.30 17 approachthemaximumaccuracywhichcouldbeachievedbyBAAinthesetestproblems.Numerically,BAAseemstoterminatewhenthesolutiontothesubproblem( 2.21 )yieldsadirectionwhichdepartsfromthefeasibleset,andhence,thestepsizeinthelinesearchStep2biszero.Wewereabletoachieveafurtherimprovementinthesolutionbytakingapartialstepinthisinfeasibledirectionsincetheincreaseinconstraintviolationwasmuchlessthantheimprovementinobjectivefunctionvalue.Nonetheless,theimprovementinaccuracyachievedbypermittinginfeasibilitywasatmostonedigitinourexperiments. InTables 2-1 and 2-2 weseethatBAAgavethebestresultsforthistestset,bothintermsofCPUtimeandintermsofsuccesses(thenumberoftimesthattheconvergencetolerance( 2.40 )wasachieved).Recallthatthegradientprojectionalgorithmsinour 38

PAGE 39

experimentsusedBAAtocomputetheprojectedgradient.TheconvergencefailuresforthegradientprojectionalgorithmsinTables 2-1 and 2-2 wereduetothefactthatBAAwasunabletocomputetheprojectedgradientwithenoughaccuracytoyielddescentinthegradientprojectionalgorithm. 2.7.3ProblemswithNonconvexCost WetestedourellipsoidalbranchandboundalgorithmusingsomerandomlygeneratedtestproblemswithA0indenite.Tocomputein( 2.7 ),weusedthepowermethod(see[ 80 ])tondtheeigenvectorassociatedwiththelargesteigenvalue.Wechosein( 2.2 )tobe0.1minusthesmallesteigenvalueofA0.NGPAwasusedtolocallysolve(QPP)andupdatetheupperbound. Wetookm=2andrandomlygeneratedtestproblemusingtheprocedurein[ 1 ].Thatis,theellipsoidalconstraintfunctionsin(QPP)havetheformgi(x)=(x)]TJ /F6 11.955 Tf 11.96 0 Td[(ci)TB)]TJ /F5 7.97 Tf 6.59 0 Td[(1i(x)]TJ /F6 11.955 Tf 11.95 0 Td[(ci))]TJ /F1 11.955 Tf 11.96 0 Td[(1; whereBi=UDiUTandUisasgivenearlier.DiisadiagonalmatrixwithitsdiagonalinRand(n;0;60),c12Rand(n;0;100),andc2=c1+:8vwherevisthesemi-majoraxisoftheellipsoidg1(x)0.Forthischoiceofc2,theellipsoidsg1(x)0andg2(x)0havenonemptyintersectionatx=c2.Intheobjectivefunction,A0=UDUTwhereDisadiagonalmatrixwithdiagonalinRand(n;)]TJ /F1 11.955 Tf 9.3 0 Td[(30;30)andb02Rand(n;)]TJ /F1 11.955 Tf 9.3 0 Td[(1;1).Thecasem=2isespeciallyimportantsincequadraticproblemswithtwoellipsoidalconstraintsbelongtotheclassofCelis-Dennis-Tapiasubproblems[ 12 ]whicharisefromtheapplicationofthetrustregionmethodforequalityconstrainednonlinearprogramming[ 15 16 44 58 62 69 87 ]. IfUBkandLBkaretherespectiveupperandlowerboundsfortheoptimalobjectivefunctionvalueatiterationk,thenourstoppingcriterionwasUBk)]TJ /F1 11.955 Tf 11.95 0 Td[(LBkmaxfa;rjLBkjg; 39

PAGE 40

witha=10)]TJ /F5 7.97 Tf 6.59 0 Td[(5andr=10)]TJ /F5 7.97 Tf 6.59 0 Td[(2. Weconsideredproblemsof8dierentdimensionsrangingfrom30upto300asshowninTable 2-3 .Foreachdimension,wesolved4randomlygeneratedproblems.Table 2-3 showsthenumericalresultsforourtestinstances,where\neigs"isthenumberofnegativeeigenvaluesoftheobjectivefunction,\lb1"and\ub1"arethelowerboundandupperboundsattherststep,\val"isthecomputedoptimalvalueand\it"isthenumberofiterations.Wealsoreporttheperformanceofthealgorithmform=6inTable 2-4 Incomparingourellipsoidalbranchandboundalgorithmbasedonlinearunderestimation(EBL)totheellipsoidalbranchandboundalgorithmofLeThiHoaiAn[ 1 ]basedondualunderestimation(EBD),anadvantageofEBDisthattheunderestimatesareoftenquitetightinthedual-basedapproach.AsseeninTable 2-3 ,EBLrequiredupto273bisectionsforthistestsetwhileEBDin[ 1 ]wasabletosolverandomlygeneratedtestproblemswithoutanybisections.Ontheotherhand,adisadvantageofEBDisthatthedualproblemsarenondierentiablewhenA0isindenite.Consequently,theevaluationofthelowerboundusingEBDentailssolvinganoptimizationproblemwhich,ingeneral,isnondierentiable.WithEBL,however,computingalowerboundinvolvessolvingaconvexoptimizationproblem.Tosummarize,EBDprovidestightlowerboundsusinganondierentiableoptimizationproblemforthelowerbound,whileEBLprovideslesstightlowerboundsusingaconvexoptimizationproblemforthelowerbound. 40

PAGE 41

Table2-3. Theperformanceofbranchandboundalgorithmform=2 n neigs lb1 ub1 val it time 30 12 34827.3 35256.3 35254.8 5 1.75 17 -41212.1 -40746.2 -40748.8 21 3.58 14 38601.4 38977.6 38977.6 0 0.72 17 -31534.2 -31108.4 -31119.8 92 8.82 50 22 -357168.8 -356828.8 -356828.8 0 0.52 21 -33792.9 -33447.9 -33447.9 1 0.84 21 -29694.6 -29254.1 -29255.2 247 23.08 26 35034.0 35416.6 35414.8 5 2.12 60 29 17783.6 18227.4 18227.4 78 22.41 26 -27498.2 -27110.5 -27110.5 69 18.22 30 -69845.7 -69463.1 -69463.1 0 0.56 28 20408.7 20963.1 20927.1 273 42.65 100 50 -11495.2 -11196.9 -11218.9 56 30.72 51 17539.6 17909.3 17909.3 4 1.84 52 -46065.5 -45653.2 -45653.2 0 0.88 40 970829.8 971326.6 971326.6 0 0.92 150 75 -302382.2 -302071.0 -302071.0 0 0.95 83 29089.4 29500.8 29500.8 64 31.45 72 16580.5 16904.9 16904.9 1 1.98 73 -32461.9 -32036.1 -32036.1 1 1.37 200 100 10798.5 11226.1 11226.1 81 56.58 95 -27242.9 -26792.1 -26792.1 2 2.27 100 35293.0 35862.1 35862.1 1 1.63 96 -31712.8 -31138.3 -31138.3 77 47.06 250 135 37015.8 37477.6 37477.6 1 2.9 131 -27278.9 -26563.0 -26780.0 86 88.40 128 -9979.6 -9683.9 -9683.9 59 131.54 121 -371385.9 -370991.9 -370991.9 0 2.03 300 145 -162041.5 -161645.7 -161645.7 0 5.33 152 -48085.4 -47529.3 -47529.3 1 7.56 138 226345.6 226377.8 226377.8 0 4.79 148 -17649.5 -17013.7 -17323.2 109 257.52 Table2-4. Theperformanceofbranchandboundalgorithmform=6 n neigs lb1 ub1 val it time 30 17 22717.1 22993.1 22993.1 1 2.41 21 -22847.0 -22520.4 -22524.5 14 10.58 20 -17858.2 -17573.2 -17573.2 1 1.84 60 33 -21818.1 -21489.1 -21489.2 33 21.64 27 47683.8 47826.0 47826.0 0 2.82 31 -4926.5 -4652.0 -4728.7 4 7.62 100 56 -35438.9 -35411.0 -35411.0 0 0.78 52 -1740.1 -1187.5 -1198.2 354 283.25 49 -6756.5 -6148.9 -6148.9 3 8.06 41

PAGE 42

CHAPTER3ANEXACTALGORITHMFORGRAPHPARTITIONING 3.1IntroductiontoGraphPartitioning Givenagraphwithedgeweights,thegraphpartitioningproblemistopartitiontheverticesintotwosetssatisfyingspeciedsizeconstraints,whileminimizingthesumoftheweightsoftheedgesthatconnecttheverticesinthetwosets.GraphpartitioningproblemsariseinmanyareasincludingVLSIdesign,datamining,parallelcomputing,andsparsematrixfactorizations[ 37 51 57 79 ].ThegraphpartitioningproblemisNP-complete[ 31 ]. Therearetwogeneralclassesofmethodsforthegraphpartitioningproblem,exactmethodswhichcomputetheoptimalpartition,andheuristicmethodswhichtrytoquicklycomputeanapproximatesolution.Heuristicmethodsincludespectralmethods[ 46 ],geometricmethods[ 32 ],multilevelschemes[ 47 ],optimization-basedmethods[ 24 ],andmethodsthatemployrandomizationtechniquessuchasgeneticalgorithms[ 77 ].SoftwarewhichimplementsheuristicmethodsincludesMetis([ 53 { 55 ]),Chaco[ 45 ],Party[ 71 ],PaToH[ 11 ],SCOTCH[ 68 ],Jostle[ 85 ],Zoltan[ 22 ],andHUND[ 33 ]. Thischapterdevelopsanexactalgorithmforthegraphpartitioningproblem.Inearlierwork,Brunetta,Conforti,andRinaldi[ 9 ]proposeabranch-and-cutschemebasedonalinearprogrammingrelaxationandsubsequentcutsbasedonseparationtechniques.AcolumngenerationapproachisdevelopedbyJohnson,Mehrotra,andNemhauser[ 51 ],whileMitchell[ 63 ]developsapolyhedralapproach.Karisch,Rendl,andClausen[ 52 ]developabranch-and-boundmethodutilizingasemideniteprogrammingrelaxationtoobtainalowerbound.Sensen[ 75 ]developsabranch-and-boundmethodbasedonalowerboundobtainedbysolvingamulticommodityowproblem. Inthischapter,wedevelopabranch-and-boundalgorithmbasedonaquadraticprogramming(QP)formulationofthegraphpartitioningproblem.TheobjectivefunctionoftheQPisexpressedasthesumofaconvexandaconcavefunction.Weconsidertwodierenttechniquesformakingthisdecomposition,onebasedoneigenvaluesandtheother 42

PAGE 43

basedonsemideniteprogramming.Ineachcase,wegiveananeunderestimatefortheconcavefunction,whichleadstoatractablelowerboundinthebranchandboundalgorithm. Thechapterisorganizedasfollows.InSection 3.2 wereviewthecontinuousquadraticprogrammingformulationofthegraphpartitioningproblemdevelopedin[ 37 ]andweexplainhowtoassociateasolutionofthecontinuousproblemwiththesolutiontothediscreteproblem.InSection 3.3 wediscussapproachesfordecomposingtheobjectivefunctionfortheQPintothesumofconvexandaconcavefunctions,andineachcase,weshowhowtogenerateananelowerboundfortheconcavepart.Section 3.4 givesthebranch-and-boundalgorithm,whileSection 3.5 providesnecessaryandsucientconditionsforalocalminimizer.Section 3.6 comparestheperformanceofthenewbranch-and-boundalgorithmtoearlierresultsgivenin[ 52 ]and[ 75 ]. Inthischapter,1denotesthevectorwhoseentriesareall1.Thedimensionwillbeclearfromcontext.IfA2Rnn,A0meansthatAispositivesemidenite.Weleteidenotethei-thcolumnoftheidentitymatrix;again,thedimensionwillbeclearfromcontext. 3.2ContinuousQuadraticProgrammingFormulation LetGbeagraphwithnverticesV=f1;2;;ng; andletaijbeaweightassociatedwiththeedge(i;j).Whenthereisnoedgebetweeniandj,wesetaij=0.Foreachiandj,weassumethataii=0andaij=aji;inotherwords,weconsideranundirectedgraphwithoutselfloops(asimple,undirectedgraph).Thesignoftheweightsisnotrestricted,andinfact,aijcouldbenegative,asitwouldbeinthemax-cutproblem.Givenintegerslandusuchthat0lun,wewishtopartitiontheverticesintotwodisjointsets,withbetweenlanduverticesinoneset,whileminimizingthesumoftheweightsassociatedwithedgesconnectingverticesindierent 43

PAGE 44

sets.Theedgesconnectingthetwosetsinthepartitionarereferredtoasthecutedges,andtheoptimalpartitionminimizesthesumoftheweightsofthecutedges.Hence,thegraphpartitioningproblemisalsocalledthemin-cutproblem. In[ 37 ],HagerandKrylyukshowthatforasuitablechoiceofthediagonalmatrixD,thegraphpartitioningproblemisequivalenttothefollowingcontinuousquadraticprogrammingproblem: minimizef(x):=(1)]TJ /F6 11.955 Tf 11.96 0 Td[(x)T(A+D)xsubjectto0x1;l1Txu;(3.1) whereAisthematrixwithelementsaij.Supposexisbinaryandletusdenethesets V0=fi:xi=0gandV1=fi:xi=1g:(3.2) Itcanbecheckedthatf(x)isthesumoftheweightsofthecutedgesassociatedwiththepartition( 3.2 ).Hence,ifweaddtherestrictionthatxisbinary,then( 3.1 )isexactlyequivalenttondingthepartitionwhichminimizestheweightofthecutedges.Note,though,thattherearenobinaryconstraintsin( 3.1 ).Theequivalencebetween( 3.1 )andthegraphpartitioningproblemisasfollows(see[ 37 ,Thm.2.1]): Theorem3.2.1. IfthediagonalmatrixDischosensothat dii+djj2aijanddii0(3.3) foreachiandj,then( 3:1 )hasabinarysolutionxandthepartitiongivenby( 3:2 )isamin-cut. Thegeneralizationofthisresulttomultisetpartitioningisgivenin[ 38 ].Thecondition( 3.3 )issatised,forexample,bythechoicedjj=maxf0;a1j;a2j;:::;anjg 44

PAGE 45

foreachj.TheproofofTheorem 3.2.1 wasbasedonshowingthatanysolutionto( 3.1 )couldbetransformedtoabinarysolutionwithoutchangingtheobjectivefunctionvalue.Withamodicationofthisidea,anyfeasiblepointcanbetransformedtoabinaryfeasiblepointwithoutincreasingtheobjectivefunctionvalue.Wenowgiveaconstructiveproofofthisresult,whichisusedwhenwesolve( 3.1 ). Corollary3.2.2. Ifxisfeasiblein( 3:1 )andthediagonalmatrixDsatises( 3:3 ),thenthereexistsabinaryywithf(y)f(x)andyi=xiwheneverxiisbinary. Proof. Werstshowhowtondzwiththepropertythatzisfeasiblein( 3.1 ),f(z)f(x),1Tzisinteger,andtheonlycomponentsofzandxwhichdierarethefractionalcomponentsofx.If1Tx=uor1Tx=l,thenwearedonesincelanduareintegers;hence,weassumethatl<1Tx0,f(x+ei)
PAGE 46

Supposethatxisnotbinary.Since1Txisaninteger,xmusthaveatleasttwononbinarycomponents,sayxiandxj.Again,expandingfisaTaylorseriesgivesf(x+(ei)]TJ /F6 11.955 Tf 11.95 0 Td[(ej))=f(x)+(rf(x)i)-222(rf(x)j)+2(2aij)]TJ /F3 11.955 Tf 11.95 0 Td[(dii)]TJ /F3 11.955 Tf 11.95 0 Td[(djj): By( 3.3 ),thequadratictermisnonpositiveforanychoiceof.Iftherstderivativetermisnegative,thenweincreaseabove0untileitherxi+reaches1orxj)]TJ /F3 11.955 Tf 12.36 0 Td[(reaches0.Sincetherstderivativetermisnegativeand>0,wehavef(x+(ei)]TJ /F6 11.955 Tf 12.34 0 Td[(ej))
PAGE 47

3.3.1LowerBoundBasedonMinimumEigenvalue Letusdecomposetheobjectivefunctionf(x)=(1)]TJ /F6 11.955 Tf 12.33 0 Td[(x)T(A+D)xinthefollowingway:f(x)=(f(x)+kxk2))]TJ /F3 11.955 Tf 11.95 0 Td[(kxk2; whereisthemaximumof0andthelargesteigenvalueofA+D.ThisrepresentsaDC(dierenceconvex)decomposition(see[ 48 ])sincef(x)+kxk2andkxk2arebothconvex.Theconcavetermkxk2isunderestimatedbyananefunction`toobtainaconvexunderestimatefLoffgivenby fL(x)=)]TJ /F3 11.955 Tf 5.48 -9.68 Td[(f(x)+kxk2+`(x): Wenowconsidertheproblemofndingthebestaneunderestimate`fortheconcavefunctionkxk2overagivencompact,convexsetdenotedC.Thesetofaneunderestimatorsforkxk2isgivenbyS1=f`:Rn!Rsuchthat`isaneand)-222(kxk2`(x)forallx2Cg: Thebestaneunderestimateisasolutionoftheproblem min`2S1maxx2C)]TJ /F14 11.955 Tf 11.29 9.68 Td[()]TJ /F2 11.955 Tf 5.48 -9.68 Td[(kxk2+`(x):(3.4) ThefollowingresultgeneralizesTheorem 2.3.1 inChapter 2 (seealsoTheorem3.1in[ 39 ])wherewedeterminethebestaneunderestimateforkxk2overanellipsoid. Theorem3.3.1. LetCRnbeacompact,convexsetandletcbethecenterandrbetheradiusofthesmallestspherecontainingC.Thissmallestsphereisuniqueandasolutionof( 3:4 )is`(x)=)]TJ /F1 11.955 Tf 9.3 0 Td[(2cTx+kck2)]TJ /F3 11.955 Tf 11.96 0 Td[(r2: Furthermore,min`2S1maxx2C)]TJ /F14 11.955 Tf 11.29 9.68 Td[()]TJ /F2 11.955 Tf 5.48 -9.68 Td[(kxk2+`(x)=r2: 47

PAGE 48

Proof. First,wewillshowthattheminimizationin( 3.4 )canberestrictedtoacompactset.Clearly,whencarryingouttheminimizationin( 3.4 ),weshouldrestrictourattentiontothose`whichtouchthefunctionh(x):=kxk2atsomepointinC.Lety2Cdenotethepointofcontact.Sinceh(x)`(x)andh(y)=`(y),alowerboundfortheerrorh(x))]TJ /F3 11.955 Tf 11.95 0 Td[(`(x)overx2Cish(x))]TJ /F3 11.955 Tf 11.95 0 Td[(`(x)j`(x))]TJ /F3 11.955 Tf 11.95 0 Td[(`(y)j)-223(jh(x))]TJ /F3 11.955 Tf 11.95 0 Td[(h(y)j: IfMisthedierencebetweenthemaximumandminimumvalueofhoverC,thenwehave h(x))]TJ /F3 11.955 Tf 11.95 0 Td[(`(x)j`(x))]TJ /F3 11.955 Tf 11.95 0 Td[(`(y)j)]TJ /F3 11.955 Tf 17.93 0 Td[(M:(3.5) Anupperboundfortheminimumin( 3.4 )isobtainedbythelinearfunction`0whichisconstantonC,withvalueequaltotheminimumofh(x)overx2C.IfwisapointwherehattainsitsminimumoverC,thenwehavemaxx2Ch(x))]TJ /F3 11.955 Tf 11.95 0 Td[(`0(x)=maxx2Ch(x))]TJ /F3 11.955 Tf 11.96 0 Td[(h(w)=M: Letusrestrictourattentiontothelinearfunctions`whichachieveanobjectivefunctionvaluein( 3.4 )whichisatleastassmallasthatof`0.Forthese`andforx2C,wehave h(x))]TJ /F3 11.955 Tf 11.95 0 Td[(`(x)maxx2Ch(x))]TJ /F3 11.955 Tf 11.95 0 Td[(`(x)maxx2Ch(x))]TJ /F3 11.955 Tf 11.95 0 Td[(`0(x)=M:(3.6) Combining( 3.5 )and( 3.6 )gives j`(x))]TJ /F3 11.955 Tf 11.96 0 Td[(`(y)j2M:(3.7) Thus,whenwecarryouttheminimizationin( 3.4 ),weshouldrestrictourattentiontolinearfunctionswhichtouchhatsomepointy2Candwiththechangein`acrossCsatisfyingthebound( 3.7 )forallx2C.Thistellsusthattheminimizationin( 3.4 )canberestrictedtoacompactset,andthataminimizermustexist. 48

PAGE 49

Supposethat`attainstheminimumin( 3.4 ).LetzbeapointinCwhereh(x))]TJ /F3 11.955 Tf 12.03 0 Td[(`(x)achievesitsmaximum.ATaylorexpansionaroundx=zgivesh(x))]TJ /F3 11.955 Tf 11.95 0 Td[(`(x)=h(z))]TJ /F3 11.955 Tf 11.95 0 Td[(`(z)+(rh(z))-222(r`)(x)]TJ /F6 11.955 Tf 11.95 0 Td[(z))-222(kx)]TJ /F6 11.955 Tf 11.95 0 Td[(zk2: Since`2S1,h(x))]TJ /F3 11.955 Tf 11.96 0 Td[(`(x)0forallx2C.Itfollowsthat h(z))]TJ /F3 11.955 Tf 11.95 0 Td[(`(z))]TJ /F1 11.955 Tf 21.92 0 Td[((rh(z))-222(r`)(x)]TJ /F6 11.955 Tf 11.95 0 Td[(z)+kx)]TJ /F6 11.955 Tf 11.96 0 Td[(zk2:(3.8) SinceCisconvex,therst-orderoptimalityconditionsforzgive(rh(z))-222(r`)(x)]TJ /F6 11.955 Tf 11.96 0 Td[(z)0 forallx2C.Itfollowsfrom( 3.8 )that h(z))]TJ /F3 11.955 Tf 11.95 0 Td[(`(z)kx)]TJ /F6 11.955 Tf 11.95 0 Td[(zk2(3.9) forallx2C.Thereexistsx2Csuchthatkx)]TJ /F6 11.955 Tf 12.15 0 Td[(zkrorelsezwouldbethecenterofasmallerspherecontainingC.Hence,( 3.9 )impliesthath(z))]TJ /F3 11.955 Tf 11.96 0 Td[(`(z)r2: Itfollowsthat maxx2Ch(x))]TJ /F3 11.955 Tf 11.96 0 Td[(`(x)h(z))]TJ /F3 11.955 Tf 11.96 0 Td[(`(z)r2:(3.10) Wenowobservethatforthespeciclinearfunction`giveninthestatementofthetheorem,( 3.10 )becomesanequality,whichimpliestheoptimalityof`in( 3.4 ).ExpandhinaTaylorseriesaroundx=ctoobtain h(x)=kck2)]TJ /F1 11.955 Tf 11.96 0 Td[(2cT(x)]TJ /F6 11.955 Tf 11.95 0 Td[(c))-222(kx)]TJ /F6 11.955 Tf 11.95 0 Td[(ck2=)]TJ /F1 11.955 Tf 9.3 0 Td[(2cTx+kck2)-222(kx)]TJ /F6 11.955 Tf 11.95 0 Td[(ck2: 49

PAGE 50

Figure3-1. Supposec62C Subtract`(x)=)]TJ /F1 11.955 Tf 9.3 0 Td[(2cTx+kck2)]TJ /F3 11.955 Tf 11.95 0 Td[(r2frombothsidestoobtain h(x))]TJ /F3 11.955 Tf 11.95 0 Td[(`(x)=r2)-221(kx)]TJ /F6 11.955 Tf 11.95 0 Td[(ck2:(3.11) Ifc2C,thenthemaximumin( 3.11 )overx2Cisattainedbyx=cforwhichh(c))]TJ /F3 11.955 Tf 11.96 0 Td[(`(c)=r2: Consequently,( 3.10 )becomesanequalityfor`=`,whichimpliestheoptimalityof`in( 3.4 ). Wecanshowthatc2Casfollows:Supposec62C.SinceCiscompactandconvex,thereexistsahyperplaneHstrictlyseparatingcandC{seeFigure 3-1 Ifc0istheprojectionofcontoH,then kx)]TJ /F6 11.955 Tf 11.96 0 Td[(c0k
PAGE 51

TheuniquenessofthesmallestspherecontainingCisasfollows:SupposethatthereexisttwodierentsmallestspheresS1andS2containingC.LetS3bethesmallestspherecontainingS1\S2.SincethediameteroftheintersectionisstrictlylessthanthediameterofS1orS2,wecontradicttheassumptionthatS1andS2werespheresofsmallestradiuscontainingC. Remark3.3.2. AlthoughthesmallestspherecontainingCinTheorem 3.3.1 isunique,thebestlinearunderestimatorofh(x)=kxk2isnotunique.Forexample,supposeaandb2RnandCisthelinesegmentC=fx2Rn:x=a+(1)]TJ /F3 11.955 Tf 11.95 0 Td[()b;2[0;1]g: Alongthislinesegment,hisaconcavequadraticinonevariable.Thebestaneunderes-timatealongthelinesegmentcorrespondstothelineconnectingtheendsofthequadraticrestrictedtothelinesegment.Hence,inRn+1,anyhyperplanewhichcontainsthepoints(h(a);a)and(h(b);b)leadstoabestaneunderestimate. Remark3.3.3. LetCbetheboxB=fx2Rn:pxqg: ThediameterofB,thedistancebetweenthepointsinBwithgreatestseparation,iskp)]TJ /F6 11.955 Tf 12.07 0 Td[(qk.Hence,thesmallestspherecontainingBhasradiusatleastkp)]TJ /F6 11.955 Tf 12.07 0 Td[(qk=2.Ifx2B,thenjxi)]TJ /F1 11.955 Tf 11.96 0 Td[((pi+qi)=2j(qi)]TJ /F3 11.955 Tf 11.95 0 Td[(pi)=2 foreveryi.Consequently,kx)]TJ /F1 11.955 Tf 12.82 0 Td[((p+q)=2kkp)]TJ /F6 11.955 Tf 12.82 0 Td[(qk=2andthespherewithcenterc=(p+q)=2andradiusr=kp)]TJ /F6 11.955 Tf 12.4 0 Td[(qk=2containsB.ItfollowsthatthisisthesmallestspherecontainingBsinceanyotherspheremusthaveradiusatleastkp)]TJ /F6 11.955 Tf 11.95 0 Td[(qk=2. Remark3.3.4. FindingthesmallestspherecontainingCmaynotbeeasy.However,thecenterandradiusofanyspherecontainingCyieldsananeunderestimateforkxk2over 51

PAGE 52

C.Thatis,ifSisaspherewithCS,thenthebestaneunderestimateforkxk2overSisalsoananeunderestimateforkxk2overC. 3.3.2LowerBoundBasedonSemideniteProgramming AdierentDCdecompositionoff(x)=(1)]TJ /F6 11.955 Tf 11.96 0 Td[(x)T(A+D)xisthefollowing:f(x)=(f(x)+xTx))]TJ /F6 11.955 Tf 11.95 0 Td[(xTx; whereisadiagonalmatrixwithi-thdiagonalelementi0.WewouldliketomakethesecondtermxTxassmallaspossiblewhilekeepingthersttermf(x)+xTxconvex.Thissuggeststhefollowingsemideniteprogrammingproblem minimizePni=1isubjectto)]TJ /F1 11.955 Tf 11.96 0 Td[((A+D)0;0;(3.13) whereisthediagonalof.IfthediagonalofA+Disnonnegative,thentheinequality0canbedroppedsinceitisimpliedbytheinequality)]TJ /F1 11.955 Tf 11.96 0 Td[((A+D)0. Asbefore,weseekthebestlinearunderestimateoftheconcavefunction)]TJ /F6 11.955 Tf 9.3 0 Td[(xTxoveracompact,convexsetC.Ifanyoftheivanish,thenreorderthecomponentsofxsothatx=(y;z)wherezcorrespondstothecomponentsofithatvanish.Let+betheprincipalsubmatrixofcorrespondingtothepositivediagonalelements,anddenethesetC+=fy:(y;z)2Cforsomezg: Theproblemofndingthebestlinearunderestimatefor)]TJ /F6 11.955 Tf 9.29 0 Td[(xTxoverCisessentiallyequivalenttondingthebestlinearunderestimatefor)]TJ /F6 11.955 Tf 9.3 0 Td[(yT+yovertheC+.Hence,thereisnolossofgeneralityinassumingthatthediagonalofisstrictlypositive.AsaconsequenceofTheorem 3.3.1 ,wehave Corollary3.3.5. Supposethediagonalofisstrictlypositiveandletcbethecenterandrtheradiusoftheuniquesmallestspherecontainingtheset1=2C:=f1=2x:x2Cg: 52

PAGE 53

Thebestlinearunderestimateof)]TJ /F6 11.955 Tf 9.3 0 Td[(xTxoverthecompact,convexsetCis`(x)=)]TJ /F1 11.955 Tf 9.3 0 Td[(2cT1=2x+kck2)]TJ /F3 11.955 Tf 11.95 0 Td[(r2: Furthermore,min`2S2maxx2C)]TJ /F14 11.955 Tf 11.29 9.68 Td[()]TJ /F6 11.955 Tf 5.48 -9.68 Td[(xTx+`(x)=r2; whereS2=f`:Rn!Rsuchthat`isaneand)]TJ /F6 11.955 Tf 11.95 0 Td[(xTx`(x)forallx2Cg: Proof. Withthechangeofvariablesy=1=2x,ananefunctioninxistransformedtoananefunctioninyandconversely,ananefunctioninyistransformedtoananefunctioninx.Hence,theproblemofndingthebestaneunderestimatefor)]TJ /F6 11.955 Tf 9.3 0 Td[(xTxoverCisequivalenttotheproblemofndingthebestaneunderestimateforkyk2over1=2C.ApplyTheorem 3.3.1 tothetransformedprobleminy,andthentransformbacktox. Remark3.3.6. IfCistheboxfx2Rn:0x1g,then1=2CisalsoaboxtowhichwecanapplytheobservationinRemark 3.3.3 .Inparticular,wehave c=1 21=21=1 21=2andr=k1=21k=2=k1=2k=2:(3.14) Hence,kck2)]TJ /F3 11.955 Tf 11.95 0 Td[(r2=0andwehave`(x)=)]TJ /F15 11.955 Tf 9.3 0 Td[(Tx. Remark3.3.7. LetusconsiderthesetC=fx2Rn:0x1;1Tx=bg; where0
PAGE 54

AsobservedinRemark 3.3.6 ,thecentercandradiusrofthesmallestsphereScontainingthesetfy2Rn:0y1=2g aregivenin( 3.14 ).TheintersectionofthisspherewiththehyperplaneyT)]TJ /F5 7.97 Tf 6.59 0 Td[(1=2=bisalowerdimensionalsphereSwhosecentercistheprojectionofcontothehyperplane.ScontainsCsinceCiscontainedinboththeoriginalsphereSandthehyperplane.Withalittlealgebra,weobtainc=1 21=2+b)]TJ /F3 11.955 Tf 11.96 0 Td[(:5n Pni=1)]TJ /F5 7.97 Tf 6.59 0 Td[(1i)]TJ /F5 7.97 Tf 6.58 0 Td[(1=2: BythePythagoreanTheorem,theradiusrofthelowerdimensionalsphereSisr=vuut :25 nXi=1i!)]TJ /F1 11.955 Tf 13.15 8.09 Td[((b)]TJ /F3 11.955 Tf 11.95 0 Td[(:5n)2 Pni=1)]TJ /F5 7.97 Tf 6.59 0 Td[(1i: Hence,byCorollary 3.3.5 ,anunderestimateof)]TJ /F6 11.955 Tf 9.3 0 Td[(xTxisgivenby`(x)=)]TJ /F15 11.955 Tf 9.29 0 Td[(Tx+n)]TJ /F1 11.955 Tf 11.95 0 Td[(2b Pni=1)]TJ /F5 7.97 Tf 6.59 0 Td[(1i1Tx+kck2)]TJ /F1 11.955 Tf 11.96 0 Td[((r)2: Since1Tx=bwhenx2C,itcanbeshown,aftersomealgebra,that`(x)=)]TJ /F15 11.955 Tf 9.3 0 Td[(Tx(alltheconstantsintheanefunctioncancel).Hence,theaneunderestimate`computedinRemark 3.3.6 fortheunitboxandtheaneunderestimate`computedinthisremarkfortheunitboxintersectthehyperplane1Tx=barethesame. 3.4BranchandBoundAlgorithm Sincethecontinuousquadraticprogram( 3.1 )hasabinarysolution,thebranchingprocessinthebranchandboundalgorithmisbasedonsettingvariablesto0or1andreducingtheproblemdimension(wedonotemploybisectionsofthefeasibleregionasin[ 39 ]).Webeginbyconstructingalinearorderingoftheverticesofthegraphaccordingtoanestimateforthedicultyinplacingthevertexinthepartition.Forthenumericalexperiments,theorderwasbasedonthetotalweightoftheedgesconnectingavertexto 54

PAGE 55

theadjacentvertices.Iftwoverticesv1andv2haveweightsw1andw2respectively,thenv1precedesv2ifw1>w2. Letv1,v2,:::,vndenotetheorderedvertices.Leveliinthebranchandboundtreecorrespondstosettingthevi-thcomponentofxtothevalues0or1.Eachleafatlevelirepresentsaspecicselectionof0and1valuesforthev1throughvi-thcomponentsofx.Hence,aleafatlevelihasalabeloftheform =(b1;b2;:::;bi);bj=0or1for1ji:(3.15) Correspondingtothisleaf,thevalueofthevj-thcomponentofxisbjfor1ji. LetTkdenotethebranchandboundtreeatiterationkandletE(Tk)denotetheleavesinthetree.Suppose2E(Tk)liesatleveliinTkasin( 3.15 ).Letxdenotethevectorgottenbyremovingcomponentsvj,1ji,fromx.Thevj-thcomponentofxhasthepre-assignedbinaryvaluebjfor1ji.Aftertakingintoaccounttheseassignedbinaryvalues,thequadraticproblemreducestoalowerdimensionalprobleminthevariablexoftheformminimizef(x)subjectto0x1;l1Txu; whereu=u)]TJ /F8 7.97 Tf 19.14 14.94 Td[(iXj=1bjandl=l)]TJ /F8 7.97 Tf 19.15 14.94 Td[(iXj=1bj: UsingthetechniquesdevelopedinSection 3.3 ,wereplacefbyaconvexlowerbounddenotedfLandweconsidertheconvexproblem minimizefL(x)subjectto0x1;l1Txu:(3.16) LetM()denotetheoptimalobjectivefunctionvaluefor( 3.16 ).Atiterationk,theleaf2E(Tk)forwhichM()issmallestisusedtobranchtothenextlevel.Ifhastheform 55

PAGE 56

Figure3-2. Branchandboundtree ( 3.15 ),thenthebranchingprocessesgeneratesthetwonewleaves (b1;b2;:::;bi;0)and(b1;b2;:::;bi;1):(3.17) Anillustrationinvolvinga3-levelbranchandboundtreeappearsinFigure 3-2 Duringthebranchandboundprocess,wemustalsocomputeanupperboundfortheminimalobjectivefunctionvaluein( 3.1 ).Thisupperboundisobtainedusingaheuristictechniquebasedonthegradientprojectionalgorithmandsphereapproximationstothefeasibleset.Theheuristicforgeneratinganupperboundwasdescribedin[ 67 ].Aspointedoutearlier,manyheuristictechniquesareavailable(forexample,Metis([ 53 { 55 ]),Chaco[ 45 ],andParty[ 71 ]).Anadvantageofthequadraticprogrammingbasedheuristicisthatwestartatthesolutiontothelowerboundingproblem,asolutionwhichtypicallyhasfractionalentriesandwhichisafeasiblestartingpointfor( 3.1 ).Consequently,theupperboundisnolargerthantheobjectivefunctionvalueassociatedwiththeoptimalpointinthelower-boundproblem. Convexquadraticbranchandbound(CQB) 1. InitializeT0=;andk=0.Evaluatebothalowerboundforthesolutionto( 3.1 )andanupperdenotedU0. 2. Choosek2E(Tk)suchthatM(k)=minfM():2E(Tk)g.IfM(k)=Uk,thenstop,anoptimalsolutionof( 3.1 )hasbeenfound. 56

PAGE 57

3. Assumingthatkhastheform( 3.15 ),letTk+1bethetreeobtainedbybranchingatkandaddingtwonewleavesasin( 3.17 );alsoseeFigure 3-2 .Evaluatelowerboundsforthequadraticprogrammingproblems( 3.16 )associatedwiththetwonewleaves,andevaluateanimprovedupperbound,denotedUk+1,byusingsolutionstothelowerboundproblemsasstartingguessesinadescentmethodappliedto( 3.1 ). 4. Replacekbyk+1andreturntostep2. Convergenceisassuredsincethereareanitenumberofbinaryvaluesforthecomponentsofx.Intheworstcase,thebranchandboundalgorithmwillbuildall2n+1)]TJ /F1 11.955 Tf 11.03 0 Td[(1nodesofthetree. 3.5NecessaryandSucientOptimalityConditions Weusethegradientprojectionalgorithmtoobtainanupperboundforasolutionto( 3.1 ).Sincethegradientprojectionalgorithmcanterminateatastationarypoint,weneedtobeabletodistinguishbetweenastationarypointandalocalminimizer,andatastationarypointwhichisnotalocalminimizer,weneedafastwaytocomputeadescentdirection. Webeginbystatingtherst-orderoptimalityconditions.Givenascalar,denethevector(x;)=(A+D)1)]TJ /F1 11.955 Tf 11.95 0 Td[(2(A+D)x+1; andtheset-valuedmapsN:R!2RandM:R!2RN()=8>>>><>>>>:Rif=0f1gif<0f0gif>0;M()=8>>>><>>>>:Rif=0fugif>0flgif<0: Foranyvector,N()isavectorofsetswhosei-componentisthesetN(i).Therst-orderoptimality(Karush-Kuhn-Tucker)conditionsassociatedwithalocalminimizer 57

PAGE 58

xof( 3.1 )canbewritteninthefollowingway:Forsomescalar,wehave 0x1;x2N((x;));l1Txu;and1Tx2M():(3.18) Therstandthirdconditionsin( 3.18 )aretheconstraintsin( 3.1 ),whiletheremainingtwoconditionscorrespondtocomplementaryslacknessandstationarityoftheLagrangian. In[ 37 ],HagerandKrylyukgiveanecessaryandsucientoptimalityconditionsfor( 3.1 ),whichwenowreview.Givenanyxthatisfeasiblein( 3.1 ),letusdenethesetsU(x)=fi:xi=1g;L(x)=fi:xi=0g;andF(x)=fi:0
PAGE 59

(b) xi>0and1Tx=u. (c) xi<1and1Tx=l. Corollary3.5.2. (Hager-Krylyuk)Supposethatl0,thend=ei,withanappropriatechoiceofsign,isadescentdirectioninanyofthecases(a){(c)of(P4). Proof. TheLagrangianLassociatedwith( 3.1 )hastheform L(x)=f(x)+(1Tx)]TJ /F3 11.955 Tf 11.95 0 Td[(b))]TJ /F14 11.955 Tf 11.96 11.36 Td[(Xi2Lixi)]TJ /F14 11.955 Tf 11.96 11.36 Td[(Xi2Ui(xi)]TJ /F1 11.955 Tf 11.96 0 Td[(1);(3.21) whereb=uif>0,b=lif<0,andstandsfor(x;).ThesetsLandUdenoteL(x)andU(x)respectively.Bytherst-orderoptimalityconditions( 3.18 ),wehaveL(x)=f(x)andrL(x)=0.ExpandingtheLagrangianaroundxgivesL(x+y)=L(x)+rL(x)y+1 2yTr2L(x)y=f(x))]TJ /F6 11.955 Tf 11.96 0 Td[(yT(A+D)y: 59

PAGE 60

WesubstituteforLusing( 3.21 )toobtain f(x+y)=L(x+y))]TJ /F3 11.955 Tf 11.95 0 Td[((1T(x+y))]TJ /F3 11.955 Tf 11.96 0 Td[(b)+Xi2Li(xi+yi)+Xi2Ui(xi+yi)]TJ /F1 11.955 Tf 11.96 0 Td[(1)=f(x))]TJ /F3 11.955 Tf 11.95 0 Td[(1Ty)]TJ /F6 11.955 Tf 11.96 0 Td[(yT(A+D)y+Xi2Liyi+Xi2Uiyi: (3.22) If(P2)isviolated,thenthereareindicesiandj2F(x)suchthatdii+djj>2aij.Weinserty=(ei)]TJ /F6 11.955 Tf 11.96 0 Td[(ej)in( 3.22 )toobtain f(x+(ei)]TJ /F6 11.955 Tf 11.95 0 Td[(ej))=f(x)+2(2aij)]TJ /F3 11.955 Tf 11.95 0 Td[(dii)]TJ /F3 11.955 Tf 11.95 0 Td[(djj):(3.23) Sincethecoecientof2isnegative,d=ei)]TJ /F6 11.955 Tf 12.3 0 Td[(ejisadescentdirectionfortheobjectivefunction.Since00issucientlysmall. Finally,supposethatl0.Substitutingy=eiin( 3.22 )yieldsf(x+ei)=f(x))]TJ /F3 11.955 Tf 11.95 0 Td[(2dii: Sincethecoecientdiiof2ispositive,d=eiisadescentdirection.Moreover,inanyofthecases(a){(c)of(P4),x+disfeasibleforsome>0withanappropriatechoiceofthesignofd. Wenowgiveanecessaryandsucientconditionforalocalminimizertobestrict.Whenalocalminimizerisnotstrict,itmaybepossibletomovetoaneighboringpointwhichhasthesameobjectivefunctionvaluebutwhichisnotalocalminimizer. Corollary3.5.4. Ifxisalocalminimizerfor( 3:1 )and( 3:20 )holds,thenxisastrictlocalminimizerifandonlyifthefollowingconditionshold: (C1) F(x)isempty. 60

PAGE 61

(C2) rf(x)i>rf(x)jforeveryi2L(x)andj2U(x). (C3) Iflf(x)whenyisafeasiblepointnearx.IfFhasatleasttwoelements,thenby(P2)ofTheorem 3.5.1 ,dii+djj=2aijforeachiandj2F.Sincetherst-orderoptimalityconditions( 3.18 )holdatx,itfollowsfrom( 3.23 )that f(x+(ei)]TJ /F6 11.955 Tf 11.96 0 Td[(ej))=f(x)(3.24) forall.Sincethisviolatestheassumptionthatxisastrictlocalminimizer,weconcludethatjFj1.If1Tx=uor1Tx=l,thensinceuandlareintegers,itisnotpossibleforxtohavejustonefractionalcomponent.Consequently,Fisempty.Ifl<1Tx
PAGE 62

violatestheassumptionthatxisastrictlocalminimizer.Hence,oneoftheinequalitiesin( 3.25 )isstrict.Theoneachsideof( 3.25 )iscancelledtoobtain(C2). Supposethatl0and1Tx=uorxi<1and1Tx=l,theidentityf(x+ei)=f(x)impliesthatweviolatethestrictlocaloptimalityofx.Thisestablishes(C3). Conversely,supposethatxisalocalminimizerand(C1){(C3)hold.Wewillshowthat rf(x)y>0whenevery6=0andx+yfeasiblein( 3:1 ):(3.26) Asaresult,bythemeanvaluetheorem,f(x+y)>f(x)whenyissucientlysmall.Hence,xisastrictlocalminimizer. Whenx+yisfeasiblein( 3.1 ),wehave yi0foralli2Landyi0foralli2U:(3.27) Bytherst-orderoptimalitycondition( 3.18 ),i0foralli2Landi0foralli2U.Hence,wehave (rf(x)+1T)y=Ty=Xi2Liyi+Xi2Uiyi0:(3.28) Wenowconsiderthreecases. First,supposethat1Ty=0andy6=0.By(C1)Fisemptyandhence,by( 3.27 ),yi>0forsomei2Landyj<0forsomej2U.Afteraddingtoeachsideintheinequalityin(C2),itfollowsthateither mini2Li0>maxj2Uj(3.29) 62

PAGE 63

or mini2Li>0maxj2Uj:(3.30) Combining( 3.28 ),( 3.29 ),and( 3.30 )givesrf(x)yiyi)]TJ /F3 11.955 Tf 12.02 0 Td[(jyj>0sinceeitheri>0orj<0,andyi>0>yj. Second,supposethat1Ty6=0and6=0.Tobespecic,supposethat>0.Bycomplementaryslackness,1Tx=u.Sincex+yisfeasiblein( 3.1 )and1Ty6=0,wemusthave1Ty<0.Hence,by( 3.28 ),rf(x)y>0.Thecase<0issimilar. Finally,considerthecase1Ty6=0and=0.Inthiscase,wemusthavel0.IfZ6=;,thenby(C3),either1Tx=uandxi=0foralli2Zor1Tx=landxi=1foralli2Z.Tobespecic,supposethat1Tx=uandxi=0foralli2Z.Again,sincex+yisfeasiblein( 3.1 )and1Ty6=0,wehave1Ty<0.IfU=;,thenx=0sinceF=;.Since1Ty<0,wecontradictthefeasibilityofx+y.Hence,U6=;.Since1Ty<0,thereexistsj2Usuchthatyj<0.SinceZL,itfollowsfrom( 3.29 )thatj<0.By( 3.28 )rf(x)yjyj>0.Thecase1Tx=landxi=1foralli2Zissimilar.Thiscompletestheproofof( 3.26 ),andthecorollaryhasbeenestablished. 3.6NumericalResults WeinvestigatetheperformanceofthebranchandboundalgorithmbasedonthelowerboundsinSection 3.3 usingaseriesoftestproblems.ThecodeswerewritteninCandtheexperimentswereconductedonanIntelXeonQuad-CoreX53552.66GHzcomputerusingtheLinuxoperatingsystem.Onlyoneofthe4processorswasusedintheexperiments.Toevaluatethelowerbound,wesolve( 3.16 )bythegradientprojectionmethodwithanexactlinesearchandBarzilai-Borweinsteplength[ 2 ].ThestoppingcriterioninourexperimentswaskP(xk)]TJ /F6 11.955 Tf 11.96 0 Td[(gk))]TJ /F6 11.955 Tf 11.95 0 Td[(xkk10)]TJ /F5 7.97 Tf 6.58 0 Td[(4; 63

PAGE 64

wherePdenotestheprojectionontothefeasiblesetandgkisthegradientoftheobjectivefunctionatxk.Thesolutionofthesemideniteprogrammingproblem( 3.13 )wasobtainedusingVersion6.0.1oftheCSDPcode[ 8 ]availableat https://projects.coin-or.org/Csdp/ WecomparetheperformanceofouralgorithmwithresultsreportedbyKarisch,Rendl,andClausenin[ 52 ]andbySensenin[ 75 ].Sincetheseearlierresultswereobtainedondierentcomputers,weobtainedestimatesforthecorrespondingrunningtimeonourcomputerusingtheLINPACKbenchmarks[ 23 ].Sinceourcomputerisroughly30timesfasterthantheHP9000/735usedin[ 52 ]anditisroughly7timesfasterthantheSunUltrSPARC-II400Mhzmachineusedin[ 75 ],theearlierCPUtimesweredividedby30and7respectivelytoobtaintheestimatedrunningtimeonourcomputer.Notethatthesameinterior-pointalgorithmthatweuse,whichisthemainroutineintheCSDPcode,wasusedtosolvethesemideniterelaxationin[ 52 ]. Thetestproblemswerebasedonthegraphbisectionproblemwherel=u=n=2.TwodierentdatasetswereusedfortheAmatricesinthenumericalexperiments.MostofthetestproblemscamefromthelibraryofBrunetta,Conforti,andRinaldi[ 9 ]whichisavailableat ftp://ftp.math.unipd.it/pub/Misc/equicut. SomeofthetestmatriceswerefromtheUFSparseMatrixLibrarymaintainedbyTimothyDavis: http://www.cise.u.edu/research/sparse/matrices/ Sincethissecondsetofmatricesisnotdirectlyconnectedwithgraphpartitioning,wecreateanAforgraphpartitioningasfollows:IfthematrixSfromthelibrarywassymmetric,thenAwastheadjacencymatrixdenedasfollows:thediagonalofAiszero,aij=1ifsij6=0,andaij=0otherwise.IfSwasnotsymmetric,thenAwastheadjacencymatrixofSTS. 64

PAGE 65

3.6.1LowerBoundComparison OurnumericalstudybeginswithacomparisonofthelowerboundofSection 3.3.1 basedontheminimumeigenvalueofA+Dandthebestaneunderestimate,andthelowerboundofSection 3.3.2 basedonsemideniteprogramming.WelabelthesetwolowerboundsLB1andLB2respectively.InTable 3-1 ,therst5graphscorrespondtomatricesfromtheUFSparseMatrixLibrary,whilethenext5graphswerefromthetestsetofBrunetta,Conforti,andRinaldi.Thecolumnlabeled\Opt"istheminimumcutandwhilenistheproblemdimension.ThenumericalresultsindicatethatthelowerboundLB2basedonsemideniteprogrammingisgenerallybetter(larger)thanLB1.InTable 3-1 thebestlowerboundishighlightedinbold.Basedontheseresults,weusethesemideniteprogramming-basedlowerboundinthenumericalexperimentswhichfollow. Table3-1. Comparisonoftwolowerbounds Graph n LB1 LB2 Opt Tina Discal 11 0.31 0.86 12 jg1009 9 1.55 1.72 16 jg1011 11 1.48 0.94 24 Stranke94 10 1.76 1.77 24 Hamrle1 32 -1.93 1.12 17 4x5t 20 -21.71 5.43 28 8x5t 40 -16.16 2.91 33 t050 30 0.90 18.54 397 2x17m 34 1.33 1.27 316 s090 60 -9.84 13.10 238 3.6.2AlgorithmPerformance Unlessstatedotherwise,theremainingtestproblemscamefromthelibraryofBrunetta,Conforti,andRinaldi[ 9 ].Table 3-2 givesresultsformatricesassociatedwiththeniteelementmethod[ 78 ].ThethreemethodsarelabeledCQB(ourconvexquadraticbranchandboundalgorithm),KRC(algorithmofKarisch,Rendl,andClausen[ 52 ]),andSEN(algorithmofSensen[ 75 ]).\n"istheproblemdimension,\%"isthepercentofnonzerosinthematrix,and\#nodes"isthenumberofnodesinthebranchandboundtree.TheCPUtimeisgiveninseconds.Thebesttimeishighlightedinbold.Ascanbe 65

PAGE 66

seeninTable 3-2 ,CQBwasfastestin6outofthe10problemseventhoughthenumberofnodesinthebranchandboundtreewasmuchlarger.ThusbothKRCandSENprovidedmuchtighterrelaxations,however,thetimetosolvetheirrelaxedproblemswasmuchlargerthanthetimetooptimizeourconvexquadratics. Table 3-3 givesresultsforcompilerdesignproblems[ 26 50 ].Forthistestset,KRCwasfastestin3outof5testproblems.NotethoughthatthetimesforCQBwerecompetitivewithKRC. Table 3-4 givesresultsforbinarydeBruijngraphswhichariseinapplicationsrelatedtoparallelcomputerarchitecture[ 17 25 ].Thesegraphsareconstructedbythefollowingprocedure.WerstbuildadirectedgraphusingtheMathematicacommand: A=TableForm[ToAdjacencyMatrix[DeBruijnGraph[2,n]]] Table3-2. Meshinstances CQBKRCSEN graph n % #nodestime #nodestime #nodestime m4 32 10 220.05 10.03 10.14 ma 54 5 80.16 10.10 10.28 me 60 5 130.20 10.13 10.28 m6 70 5 2050.47 11.23 11.43 mb 74 4 950.43 10.98 11.14 mc 74 5 4120.52 11.53 11.43 md 80 4 1010.55 10.96 11.28 mf 90 4 990.79 10.80 11.85 m1 100 3 2001.04 1536.50 13.00 m8 148 2 35166.62 110.70 14.14 Table3-3. Compilerdesign CQBKRCSEN graph n % #nodestime #nodestime #nodestime cd30 30 13 110.05 10.03 10.00 cd45 45 10 350.27 10.23 10.57 cd47a 47 9 450.34 10.33 71.00 cd47b 47 9 670.29 353.73 31.43 cd61 61 10 950.86 10.67 66.00 66

PAGE 67

Toobtainthegraphpartitioningtestproblem,weaddtheMathematicageneratedmatrixtoitstransposeandsetthediagonalto0.Forthistestset,SENhadbyfarthebestperformance. Table3-4. deBruijnnetworks CQBKRCSEN graph n % #nodestime #nodestime #nodestime debr5 32 12 570.11 30.20 10.00 debr6 64 6 73272.25 5515.63 11.00 debr7 128 3 161409451:22:45 71146:36 110.28 Table 3-5 givesresultsfortoroidalgridgraphs.Thesegraphsareconnectedwithanhkgrid,thenumberofverticesinthegraphisn=hkandthereare2hkedgeswhoseweightsarechosenfromauniformdistributionontheinterval[1;10].SinceSensendidnotsolveeitherthistestset,ortheremainingtestsets,wenowcomparebetweenCQBandKRC.WeseeinTable 3-5 thatCQBwasfasterthanKRCin9ofthe10toroidalgridcases. Table3-5. Toroidalgrid CQBKRC graph n % #nodestime #nodestime 4x5t 20 21 130.01 10.03 6x5t 30 14 460.05 10.10 8x5t 40 10 1410.16 10.20 21x2t 42 10 180.02 10.17 23x2t 46 9 780.15 334.16 4x12t 48 9 690.17 30.56 5x10t 50 8 1290.24 10.20 6x10t 60 7 9920.54 4311.66 7x10t 70 6 8440.68 4719.06 10x8t 80 5 4200.91 4531.46 Table 3-6 givesresultsformixedgridgraphs.Thesearecompletegraphsassociatedwithanplanarhkplanargrid;theedgesintheplanargridreceivedintegerweightsuniformlydrawnfrom[1,100],whilealltheotheredgesneededtocompletethegraphreceivedintegerweightsuniformlydrawnfrom[1,10].Forthesegraphs,KRCwasmuch 67

PAGE 68

fasterthanCQB.Noticethatthegraphsinthistestsetarecompletelydense.Onetrendthatisseeninthesenumericalexperimentsisthatasthegraphdensityincreases,theperformanceofCQBrelativetotheothermethodsdegrades. Table3-6. Mixedgridgraphs CQBKRC graph n % #nodestime #nodestime 2x10m 20 100 1500.03 10.03 6x5m 30 100 24760.20 10.03 2x17m 34 100 424102.12 210.96 10x4m 40 100 517133.74 20.06 5x10m 50 100 3588797296.19 10.06 ResultsforplanargridgrapharegiveninTable 3-7 .Thesegraphsareassociatedwithanhkgrid.Therearehkverticesand2hk)]TJ /F3 11.955 Tf 12.54 0 Td[(h)]TJ /F3 11.955 Tf 12.54 0 Td[(kedgeswhoseweightsareintegersuniformlydrawnfrom[1,10].Forthisrelativelysparsetestset,CQBwasfasterin7outof10problems. Table3-7. Planargrid CQBKRC graph n % #nodestime #nodestime 10x2g 20 15 100.01 10.03 5x6g 30 11 440.05 10.10 2x16g 32 9 230.06 10.13 18x2g 36 8 190.08 10.06 2x19g 38 8 530.29 491.83 5x8g 40 9 240.08 10.06 3x14g 42 8 310.14 50.60 5x10g 50 7 1780.34 10.30 6x10g 60 6 2240.35 5710.63 7x10g 70 5 2710.63 6118.56 Table 3-8 givesresultsforrandomlygeneratedgraphs.Forthesegraphs,thedensityisrstxedandthentheedgesareassignedintegerweightsuniformlydrawnfrom[1,10].Forthistestset,CQBisfastestin11of20cases.Again,observethattherelativeperformanceofCQBdegradesasthedensityincreases,mainlyduetothelargenumberofnodesinthebranchandboundtree. 68

PAGE 69

Table3-8. Randomlygeneratedgraphs CQBKRC graph n % #nodestime #nodestime v090 20 10 120.01 10.03 v000 20 100 9520.02 10.03 t090 30 10 100.05 10.03 t050 30 50 50810.32 170.73 t000 30 100 1226703.79 30.20 q090 40 10 890.14 10.13 q080 40 20 9140.24 312.30 q030 40 70 55465232.23 232.06 q020 40 80 136451772.58 70.83 q010 40 90 4344123217.16 131.36 q000 40 100 8186984380.72 10.13 c090 50 10 3970.29 10.33 c080 50 20 142902.20 456.13 c070 50 30 13629015.70 498.06 c030 50 70 228587292756.26 515.46 c290 52 10 3400.34 10.40 c490 54 10 14430.54 153.30 c690 56 10 34050.82 31.00 c890 58 10 133852.66 7117.53 s090 60 10 82832.01 379.90 69

PAGE 70

CHAPTER4GRADIENT-BASEDMETHODSFORSPARSERECOVERY 4.1IntroductiontoSpaRSAAlgorithm Inthischapterweconsiderthefollowingoptimizationproblem minx2Rn(x):=f(x)+ (x);(4.1) wheref:Rn!Risasmoothfunction,and :Rn!Risconvex.Thefunction ,usuallycalledtheregularizerorregularizationfunction,isniteforallx2Rn,butpossiblynonsmooth.Animportantapplicationof( 4.1 ),foundinthesignalprocessingliterature,isthewell-known`2)]TJ /F3 11.955 Tf 11.95 0 Td[(`1problem(calledbasispursuitdenoisingin[ 14 ]) minx2Rn1 2kAx)]TJ /F6 11.955 Tf 11.96 0 Td[(bk22+kxk1;(4.2) whereA2Rkn(usuallykn),b2Rk;2R,0,andkk1isthe1-norm. Recently,Wright,NowakandFigueiredo[ 86 ]introducedtheSparseReconstructionbySeparableApproximationalgorithm(SpaRSA)forsolving( 4.1 ).Thealgorithmhasbeenshowntoworkwellinpractice.In[ 86 ]theauthorsestablishglobalconvergenceofSpaRSA.Inthischapter,weproveanestimateoftheforma=(b+k)fortheerrorintheobjectivefunctionwhenfisconvex.Iftheobjectivefunctionisstronglyconvex,thentheconvergenceoftheobjectivefunctionandtheiteratesisatleastR-linear.AstrategyispresentedforimprovingtheperformanceofSpaRSAbasedonacyclicBarzilai-Borwein(BB)step[ 19 20 30 73 ]andanadaptivechoice[ 42 ]forthereferencefunctionvalueinthelinesearch.Thechapterconcludeswithaseriesofnumericalexperimentsintheareasofsignalprocessingandimagereconstruction. Throughoutthechapter,thesubscriptkoftenrepresentstheiterationnumberinanalgorithm.@ (y)isthesubdierentialaty,asetofrowvectors.Ifp2@ (y),then (x) (y)+p(x)]TJ /F6 11.955 Tf 11.95 0 Td[(y) 70

PAGE 71

forallx2Rn. TheSpaRSAalgorithm,aspresentedin[ 86 ],isasfollows: SparseReconstructionbySeparableApproximation(SpaRSA) Given>1,2(0;1),[min;max](0;1),andstartingguessx1.Setk=1.Step1.Choose02[min;max]Step2.Set=j0wherej0isthesmallestintegersuchthat(xk+1)Rk)]TJ /F3 11.955 Tf 11.95 0 Td[(kxk+1)]TJ /F6 11.955 Tf 11.96 0 Td[(xkk2wherexk+1=argminfrf(xk)z+kz)]TJ /F6 11.955 Tf 11.95 0 Td[(xkk2+ (z):z2Rng.Step3.Ifxk+1=xk,terminate.Step4.Setk=k+1andgotostep1. Theparameter0in[ 86 ]wastakentobetheBBparameter[ 2 ]withsafeguards: 0=BBk=minfksk)]TJ /F6 11.955 Tf 11.95 0 Td[(ykk:minmaxg(4.3) wheresk=xk)]TJ /F6 11.955 Tf 12.04 0 Td[(xk)]TJ /F5 7.97 Tf 6.59 0 Td[(1andyk=gk)]TJ /F6 11.955 Tf 12.04 0 Td[(gk)]TJ /F5 7.97 Tf 6.59 0 Td[(1.Also,in[ 86 ],thereferencevalueRkistheGLL[ 34 ]referencevaluemaxkdenedby maxk=maxf(xk)]TJ /F8 7.97 Tf 6.58 0 Td[(j):0j
PAGE 72

employsaxedchoiceforrelatedtotheLipschitzconstantforf,whileSpaRSAemploysanonmonotonelinesearch.AsublinearconvergenceresultforamonotonelinesearchversionofISTAisgivenbyBeckandTeboulle[ 3 ]andbyNesterov[ 65 ].InSection 4.2 wegiveasublinearconvergenceresultforthenonmonotoneSpaRSA,whileSection 4.3 givesalinearconvergenceresultwhentheobjectivefunctionisstronglyconvex. In[ 86 ]itisshownthatthelinesearchinStep2terminatesforanitejwhenfisLipschitzcontinuouslydierentiable.HereweweakenthisconditionbyonlyrequiringLipschitzcontinuityoveraboundedset. Proposition4.1.1. LetLbethelevelsetdenedby L=fx2Rn:(x)(x1)g:(4.5) Wemakethefollowingassumptions: (A1) ThelevelsetLiscontainedintheinteriorofacompact,convexsetK,andfisLipschitzcontinuouslydierentiableonK. (A2) isconvexand (x)isniteforallx2Rn. If(xk)Rk(x1),thenthereexistswiththepropertythat(xk+1)Rk)]TJ /F3 11.955 Tf 11.96 0 Td[(kxk+1)]TJ /F6 11.955 Tf 11.95 0 Td[(xkk2 wheneverwherexk+1isobtainedasinStep2ofSpaRSA. Proof. Letkbedenedbyk(z)=f(xk)+rf(xk)(z)]TJ /F6 11.955 Tf 11.95 0 Td[(xk)+kz)]TJ /F6 11.955 Tf 11.95 0 Td[(xkk2+ (z); where0.Sincekisastronglyconvexquadratic,itslevelsetsarecompact,andtheminimizerxk+1inStep2exists.Sincexk+1istheminimizerofk,wehavek(xk+1)=f(xk)+rf(xk)(xk+1)]TJ /F6 11.955 Tf 11.96 0 Td[(xk)+kxk+1)]TJ /F6 11.955 Tf 11.96 0 Td[(xkk2+ (xk+1)k(xk)=f(xk)+ (xk): 72

PAGE 73

Thisisrearrangedtoobtainkxk+1)]TJ /F6 11.955 Tf 11.96 0 Td[(xkk2rf(xk)(xk)]TJ /F6 11.955 Tf 11.96 0 Td[(xk+1)+ (xk))]TJ /F3 11.955 Tf 11.96 0 Td[( (xk+1)rf(xk)(xk)]TJ /F6 11.955 Tf 11.96 0 Td[(xk+1)+pk(xk)]TJ /F6 11.955 Tf 11.96 0 Td[(xk+1); wherepk2@ (xk).Takingnormsyields kxk+1)]TJ /F6 11.955 Tf 11.96 0 Td[(xkk(kgkk+kpkk)=:(4.6) ByTheorem23.4andCorollary24.5.1in[ 74 ]andbythecompactnessofL,thereexistsaconstantc,independentofxk2L,suchthatkgkk+kpkkc.Consequently,wehavekxk+1)]TJ /F6 11.955 Tf 11.95 0 Td[(xkkc=: SinceKiscompactandLliesintheinteriorofK,thedistancefromLtotheboundaryofKispositive.Choose2(0;1)sothatc=.Hence,when,xk+12Ksincexk2L. LetdenotetheLipschitzconstantforfonKandsupposethat.Sincexk2LKandkxk+1)]TJ /F6 11.955 Tf 12.5 0 Td[(xkk,wehavexk+12K.Moreover,duetotheconvexityofK,thelinesegmentconnectingxkandxk+1liesinK.Proceedingasin[ 86 ],aTaylorexpansionaroundxkyieldsf(xk+1)f(xk)+rf(xk)(xk+1)]TJ /F6 11.955 Tf 11.95 0 Td[(xk)+:5kxk+1)]TJ /F6 11.955 Tf 11.96 0 Td[(xkk2: Adding (xk+1)tobothsides,wehave (xk+1)k(xk+1)+(:5)]TJ /F3 11.955 Tf 11.96 0 Td[()kxk+1)]TJ /F6 11.955 Tf 11.96 0 Td[(xkk2 (4.7) k(xk)+(:5)]TJ /F3 11.955 Tf 11.96 0 Td[()kxk+1)]TJ /F6 11.955 Tf 11.95 0 Td[(xkk2=(xk)+(:5)]TJ /F3 11.955 Tf 11.95 0 Td[()kxk+1)]TJ /F6 11.955 Tf 11.96 0 Td[(xkk2Rk+(:5)]TJ /F3 11.955 Tf 11.95 0 Td[()kxk+1)]TJ /F6 11.955 Tf 11.96 0 Td[(xkk2since(xk)RkRk)]TJ /F3 11.955 Tf 11.95 0 Td[(kxk+1)]TJ /F6 11.955 Tf 11.96 0 Td[(xkk2if:5)]TJ /F3 11.955 Tf 11.95 0 Td[()]TJ /F3 11.955 Tf 21.92 0 Td[(: 73

PAGE 74

Hence,thepropositionholdswith=max; 2(1)]TJ /F3 11.955 Tf 11.96 0 Td[(): Remark4.1.2. SupposeRk(x1).InStep2ofSpaRSA,xk+1ischosensothat(xk+1)Rk.Hence,thereexistsRk+1suchthat(xk+1)Rk+1(x1).Inotherwords,ifthehypothesis\(xk)Rk(x1)"ofProposition 4.1.1 issatisedatstepk,thenachoiceforRk+1existswhichsatisesthishypothesisatstepk+1. Remark4.1.3. WenowshowthattheGLLreferencevaluemaxksatisesthecondition(xk)Rk(x1)ofProposition 4:1:1 foreachk.Theconditionmaxk(xk)isatrivialconsequenceofthedenitionofmaxk.Also,bythedenition,wehavemax1=(x1).Fork1,(xk+1)maxkaccordingtoStep2ofSpaRSA.Hence,maxkisadecreasingfunctionofk.Inparticular,maxkmax1=(x1). 4.2ConvergenceEstimateforConvexFunctions Inthissectionwegiveasublinearconvergenceestimatefortheerrorintheobjectivefunctionvalue(xk)assumingfisconvexandtheassumptionsofProposition 4.1.1 hold. By(A1)and(A2),( 4.1 )hasasolutionx2Landanassociatedobjectivefunctionvalue:=(x).Theconvergenceoftheobjectivefunctionvaluestoisaconsequenceoftheanalysisin[ 86 ]: Lemma4.2.1. If(A1)and(A2)holdandRk=maxkforeveryk,thenlimk!1(xk)=: Proof. By[ 86 ,Lemma4],theobjectivefunctionvalues(xk)approachalimitdenoted.By[ 86 ,Theorem1],allaccumulationpointsoftheiteratesxkarestationarypoints.AnaccumulationpointexistssinceKiscompactandtheiteratesareallcontainedinLK,asshowninRemark 4.1.3 .Sincefand arebothconvex,astationarypointisaglobalminimizerof.Hence,=. 74

PAGE 75

Oursublinearconvergenceresultisthefollowing: Theorem4.2.2. If(A1)and(A2)hold,fisconvex,andRk=maxkforallk,thenthereexistconstantsaandbsuchthat(xk))]TJ /F3 11.955 Tf 11.95 0 Td[(a b+k forksucientlylarge. Proof. By( 4.7 )withk+1replacedbyk,wehave (xk)k)]TJ /F5 7.97 Tf 6.59 0 Td[(1(xk)+b0kskk2;b0=:5;(4.8) wheresk=xk)]TJ /F6 11.955 Tf 11.96 0 Td[(xk)]TJ /F5 7.97 Tf 6.58 0 Td[(1.Sincexkminimizesk)]TJ /F5 7.97 Tf 6.58 0 Td[(1andfisconvex,itfollowsthat k)]TJ /F5 7.97 Tf 6.59 0 Td[(1(xk)=minz2Rnff(xk)]TJ /F5 7.97 Tf 6.59 0 Td[(1)+rf(xk)]TJ /F5 7.97 Tf 6.59 0 Td[(1)(z)]TJ /F6 11.955 Tf 11.96 0 Td[(xk)]TJ /F5 7.97 Tf 6.58 0 Td[(1)+k)]TJ /F5 7.97 Tf 6.59 0 Td[(1kz)]TJ /F6 11.955 Tf 11.96 0 Td[(xk)]TJ /F5 7.97 Tf 6.59 0 Td[(1k2+ (z)gminff(z)+ (z)+k)]TJ /F5 7.97 Tf 6.58 0 Td[(1kz)]TJ /F6 11.955 Tf 11.95 0 Td[(xk)]TJ /F5 7.97 Tf 6.59 0 Td[(1k2:z2Rng=minf(z)+k)]TJ /F5 7.97 Tf 6.59 0 Td[(1kz)]TJ /F6 11.955 Tf 11.96 0 Td[(xk)]TJ /F5 7.97 Tf 6.58 0 Td[(1k2:z2Rng; (4.9) wherek)]TJ /F5 7.97 Tf 6.59 0 Td[(1istheterminatingvalueofatstepk)]TJ /F1 11.955 Tf 11.95 0 Td[(1.Combining( 4.8 )and( 4.9 )gives (xk)minf(z)+kz)]TJ /F6 11.955 Tf 11.96 0 Td[(xk)]TJ /F5 7.97 Tf 6.58 0 Td[(1k2:z2Rng+b0kskk2;(4.10) where=isanupperboundforthekimpliedbyProposition 4.1.1 .Bytheconvexityofandwithz=(1)]TJ /F3 11.955 Tf 11.96 0 Td[()xk)]TJ /F5 7.97 Tf 6.59 0 Td[(1+xforany2[0;1],wehaveminz2Rn(z)+kz)]TJ /F6 11.955 Tf 11.95 0 Td[(xk)]TJ /F5 7.97 Tf 6.59 0 Td[(1k2((1)]TJ /F3 11.955 Tf 11.95 0 Td[()xk)]TJ /F5 7.97 Tf 6.59 0 Td[(1+x)+2kxk)]TJ /F5 7.97 Tf 6.58 0 Td[(1)]TJ /F6 11.955 Tf 11.96 0 Td[(xk2(1)]TJ /F3 11.955 Tf 11.95 0 Td[()(xk)]TJ /F5 7.97 Tf 6.59 0 Td[(1)++2kxk)]TJ /F5 7.97 Tf 6.58 0 Td[(1)]TJ /F6 11.955 Tf 11.96 0 Td[(xk2=(1)]TJ /F3 11.955 Tf 11.95 0 Td[()(xk)]TJ /F5 7.97 Tf 6.59 0 Td[(1)++bk2; 75

PAGE 76

wherebk=kxk)]TJ /F5 7.97 Tf 6.59 0 Td[(1)]TJ /F6 11.955 Tf 11.96 0 Td[(xk2.Combiningthiswith( 4.10 )yields (xk)(1)]TJ /F3 11.955 Tf 11.96 0 Td[()(xk)]TJ /F5 7.97 Tf 6.59 0 Td[(1)++bk2+b0kskk2(1)]TJ /F3 11.955 Tf 11.96 0 Td[()Rk)]TJ /F5 7.97 Tf 6.59 0 Td[(1++bk2+b0kskk2 (4.11) forany2[0;1].Dene i=maxf(xk):(i)]TJ /F1 11.955 Tf 11.96 0 Td[(1)M
PAGE 77

AsaconsequenceofLemma 4.2.1 ,i)]TJ /F5 7.97 Tf 6.59 0 Td[(1convergesto.Hence,theminimizingalsoapproaches0asitendsto1.Chooseklargeenoughthattheminimizingislessthan1.Itfollowsfrom( 4.16 )thatforthisminimizingchoiceof,wehave ii)]TJ /F5 7.97 Tf 6.59 0 Td[(1)]TJ /F1 11.955 Tf 13.15 8.08 Td[((i)]TJ /F5 7.97 Tf 6.59 0 Td[(1)]TJ /F3 11.955 Tf 11.95 0 Td[()2 4b2+b3(i)]TJ /F5 7.97 Tf 6.59 0 Td[(1)]TJ /F3 11.955 Tf 11.96 0 Td[(i):(4.18) Deneei=i)]TJ /F3 11.955 Tf 11.96 0 Td[(.Subtractingfromeachsideof( 4.18 )giveseiei)]TJ /F5 7.97 Tf 6.59 0 Td[(1)]TJ /F3 11.955 Tf 11.95 0 Td[(e2i)]TJ /F5 7.97 Tf 6.59 0 Td[(1=(4b2)+b3(ei)]TJ /F5 7.97 Tf 6.58 0 Td[(1)]TJ /F3 11.955 Tf 11.96 0 Td[(ei)=(1+b3)ei)]TJ /F5 7.97 Tf 6.58 0 Td[(1)]TJ /F3 11.955 Tf 11.96 0 Td[(e2i)]TJ /F5 7.97 Tf 6.58 0 Td[(1=(4b2))]TJ /F3 11.955 Tf 11.96 0 Td[(b3ei: Wearrangethistoobtain eiei)]TJ /F5 7.97 Tf 6.59 0 Td[(1)]TJ /F3 11.955 Tf 11.96 0 Td[(b4e2i)]TJ /F5 7.97 Tf 6.58 0 Td[(1whereb4=1 4b2(1+b3):(4.19) By( 4.19 )eiei)]TJ /F5 7.97 Tf 6.59 0 Td[(1,whichimpliesthateiei)]TJ /F5 7.97 Tf 6.59 0 Td[(1)]TJ /F3 11.955 Tf 11.95 0 Td[(b4ei)]TJ /F5 7.97 Tf 6.59 0 Td[(1eioreiei)]TJ /F5 7.97 Tf 6.58 0 Td[(1 1+b4ei)]TJ /F5 7.97 Tf 6.58 0 Td[(1: Weformthereciprocalofthislastinequalitytoobtain1 ei1 ei)]TJ /F5 7.97 Tf 6.59 0 Td[(1+b4: Applyingthisinequalityrecursivelygives1 ei1 ej+(i)]TJ /F3 11.955 Tf 11.96 0 Td[(j)b4oreiej 1+(i)]TJ /F3 11.955 Tf 11.95 0 Td[(j)b4ej; wherejischosenlargeenoughtoensurethattheminimizingin( 4.17 )islessthan1forallij. Supposethatk2((i)]TJ /F1 11.955 Tf 11.96 0 Td[(1)M;iM]withi>j.Sinceik=M,wehave(xk))]TJ /F3 11.955 Tf 11.95 0 Td[(eiej 1+(i)]TJ /F3 11.955 Tf 11.96 0 Td[(j)b4ejej 1)]TJ /F3 11.955 Tf 11.96 0 Td[(jb4ej+kb4ej=M: Theproofiscompletedbytakinga=M=b4andb=M=(b4ej))]TJ /F3 11.955 Tf 11.95 0 Td[(Mj. 77

PAGE 78

4.3ConvergenceEstimateforStronglyConvexFunctions InthissectionweprovethatSpaRSAconvergesR-linearlywhenfisaconvexfunctionandsatises (y)(x)+ky)]TJ /F6 11.955 Tf 11.96 0 Td[(xk2(4.20) forally2Rn,where>0.Hence,xisauniqueminimizerof.Forexample,iffisastronglyconvexfunction,then( 4.20 )holds. Theorem4.3.1. If(A1)and(A2)hold,fisconvex,satises( 4:20 ),andRk=maxkforeveryk,thenthereexistconstants2(0;1)andcsuchthat (xk))]TJ /F3 11.955 Tf 11.95 0 Td[(ck((x1))]TJ /F3 11.955 Tf 11.96 0 Td[()(4.21) foreveryk. Proof. Letibedenedasin( 4.12 ).Wewillshowthatthereexist2(0;1)suchthat i)]TJ /F3 11.955 Tf 11.96 0 Td[((i)]TJ /F5 7.97 Tf 6.58 0 Td[(1)]TJ /F3 11.955 Tf 11.96 0 Td[():(4.22) Letc1bechosentosatisfytheinequality 0
PAGE 79

Weutilizetheinequality( 4.13 )butwithdierentboundsforthebkiandskiterms.Fork2((i)]TJ /F1 11.955 Tf 11.95 0 Td[(1)M;iM],wehave bk:=kxk)]TJ /F5 7.97 Tf 6.59 0 Td[(1)]TJ /F6 11.955 Tf 11.96 0 Td[(xk2 ((xk)]TJ /F5 7.97 Tf 6.59 0 Td[(1))]TJ /F3 11.955 Tf 11.96 0 Td[() (Rk)]TJ /F5 7.97 Tf 6.59 0 Td[(1)]TJ /F3 11.955 Tf 11.96 0 Td[() (R(i)]TJ /F5 7.97 Tf 6.59 0 Td[(1)M)]TJ /F3 11.955 Tf 11.95 0 Td[()=b5(i)]TJ /F5 7.97 Tf 6.58 0 Td[(1)]TJ /F3 11.955 Tf 11.96 0 Td[();b5= : Therstinequalityisdueto( 4.20 )andthelastinequalityissinceRkismonotonedecreasing.Bythedenitionofkibelow( 4.12 ),itfollowsthatki2((i)]TJ /F1 11.955 Tf 11.96 0 Td[(1)M;iM]and bkib5(i)]TJ /F5 7.97 Tf 6.59 0 Td[(1)]TJ /F3 11.955 Tf 11.95 0 Td[():(4.24) Insertingin( 4.13 )thebound( 4.24 )andtheCase2requirementkskik21=2andtheminimizingcoecientis=1+b0c1)]TJ /F1 11.955 Tf 18.01 8.08 Td[(1 4b5<1 79

PAGE 80

since1=(4b5)==(4)>b0c1by( 4.23 ).Thiscompletestheproofof( 4.22 ). Fork2((i)]TJ /F1 11.955 Tf 11.95 0 Td[(1)M;iM],wehave(xk))]TJ /F3 11.955 Tf 11.95 0 Td[(eii)]TJ /F5 7.97 Tf 6.59 0 Td[(1e11 )]TJ /F3 11.955 Tf 5.48 -9.69 Td[(1=Mk((x1))]TJ /F3 11.955 Tf 11.96 0 Td[(): Hence,( 4.21 )holdswithc=1=and=1=M.Thiscompletestheproof. Remark4.3.2. Thecondition( 4:20 )whencombinedwith( 4.21 )showsthattheiteratesxkconvergeR-linearlytox. 4.4MoreGeneralReferenceFunctionValues TheGLLreferencefunctionvaluemaxk,denedin( 4.4 ),oftenleadstogreatereciencywhenM>1,whencomparedtothemonotonechoiceM=1.Inpractice,itisfoundthatevenmoreexibilityinthereferencefunctionvaluecanfurtheraccelerateconvergence.In[ 42 ]HagerandZhangproveconvergenceofthenonmonotonegradientprojectionmethodwheneverthereferencefunctionRksatisesthefollowingconditions: (R1) R1=(x1). (R2) (xk)RkmaxfRk)]TJ /F5 7.97 Tf 6.59 0 Td[(1;maxkgforeachk>1. (R3) Rkmaxkinnitelyoften. In[ 42 ]theyprovideaspecicchoiceforRkwhichsatises(R1){(R3)andwhichgavemorerapidconvergencethanthechoiceRk=maxk.Tosatisfy(R3),wecouldchooseanintegerL>0andsimplysetRk=maxkeveryLiterations.Anotherstrategy,closerinspirittowhatisusedinthenumericalexperiments,istochooseadecreaseparameter>0andsetRk=maxkif(xk)]TJ /F8 7.97 Tf 6.58 0 Td[(L))]TJ /F3 11.955 Tf 12.38 0 Td[((xk).WenowgiveconvergenceresultsforSpaRSAwheneverthereferencefunctionvaluesatises(R1){(R3).Intherstconvergenceresultwhichfollows,convexityoffisnotrequired. Theorem4.4.1. If(A1)and(A2)holdandthereferencefunctionvalueRksatises(R1){(R3),thentheiteratesxkofSpaRSAhaveasubsequenceconvergingtoalimitxsatisfying02@(x). 80

PAGE 81

Proof. WerstapplyProposition 4.1.1 toshowthatStep2ofSpaRSAisfullledforsomechoiceofj.ThisrequiresthatweshowRk(x1)foreachk.Thisholdsfork=1by(R1).Also,fork=1,wehavemax1=(x1).Proceedingbyinduction,supposethatRi(x1)andmaxi(x1)fori=1,2,:::,k.ByProposition 4.1.1 ,Step2ofSpaRSAterminatesatanitejandhence,(xk+1)Rk(x1): Itfollowsthatmaxk+1(x1)andRk+1maxfRk;maxk+1g(x1).Thiscompletestheinductionstep,andhence,byProposition 4.1.1 ,itfollowsthatineveryiteration,Step2ofSpaRSAisfullledforanitej. ByStep2ofSpaRSA,wehave(xk)Rk)]TJ /F5 7.97 Tf 6.59 0 Td[(1)]TJ /F3 11.955 Tf 11.95 0 Td[(minkskk2; wheresk=xk)]TJ /F6 11.955 Tf 12.83 0 Td[(xk)]TJ /F5 7.97 Tf 6.59 0 Td[(1.InthethirdparagraphoftheproofofTheorem2.2in[ 42 ],itisshownthatwhenaninequalityofthisformissatisedforareferencefunctionvaluesatisfying(R1){(R3),thenliminfk!1kskk=0: Letkidenoteastrictlyincreasingsequencewiththepropertythatskitendsto0andxkiapproachesalimitdenotedx.Thatis,limi!1ski=0andlimi!1xki=x: Sinceskitendsto0,itfollowsthatxki)]TJ /F5 7.97 Tf 6.59 0 Td[(1alsoapproachesx.Bytherst-orderoptimalityconditionsforxki,wehave 02rf(xki)]TJ /F5 7.97 Tf 6.59 0 Td[(1)+2ki(xki)]TJ /F6 11.955 Tf 11.95 0 Td[(xki)]TJ /F5 7.97 Tf 6.59 0 Td[(1)+@ (xki);(4.26) wherekidenotesthevalueofinStep2ofSpaRSAassociatedwithxki.Again,byProposition 4.1.1 ,wehavetheuniformboundki=.Takingthelimitasitendsto 81

PAGE 82

1,itfollowsfromCorollary24.5.1in[ 74 ]that02rf(x)+@ (x): Thiscompletestheproof. Withasmallchangein(R3),weobtaineithersublinearorlinearconvergenceoftheentireiterationsequence. Theorem4.4.2. Supposethat(A1)and(A2)hold,fisconvex,thereferencefunctionvalueRksatises(R1)and(R2),andthereisL>0withthepropertythatforeachk, Rjmaxjforsomej2[k;k+L):(4.27) Thenthereexistconstantsaandbsuchthat(xk))]TJ /F3 11.955 Tf 11.95 0 Td[(a b+k forksucientlylarge.Moreover,ifsatisesthestrongconvexitycondition( 4:20 ),thenthereexist2(0;1)andcsuchthat(xk))]TJ /F3 11.955 Tf 11.95 0 Td[(ck((x1))]TJ /F3 11.955 Tf 11.96 0 Td[() foreveryk. Proof. Letki,i=1;2;:::,denoteanincreasingsequenceofintegerswiththepropertythatRjmaxjforj=kiandRjRj)]TJ /F5 7.97 Tf 6.58 0 Td[(1whenki
PAGE 83

Givenj,choosekisuchthatj2[ki;ki+1).Sincej)]TJ /F3 11.955 Tf 12.55 0 Td[(ki
PAGE 84

Thetestproblemsareassociatedwithapplicationsintheareasofsignalprocessingandimagereconstruction.AllexperimentswerecarriedoutonaPCusingMatlab7.6withaAMDAthlon64X2dualcore3Ghzprocessorand3GBofmemoryrunningWindowsVista.Version2.0ofSpaSRAwasobtainedfromMarioFigueiredo'swebpage(http://www.lx.it.pt/mtf/SpaRSA/).Thecodewasrunwithdefaultparameters.AdaptiveSpaRSAwaswritteninMatlabwiththefollowingparametervaluesmin=10)]TJ /F5 7.97 Tf 6.59 0 Td[(30;max=1030;=5;=10)]TJ /F5 7.97 Tf 6.59 0 Td[(4;M=10: Thetestproblems,suchasthebasispursuitdenoisingproblem( 4.2 ),involveaparameter.Thechoiceofthecyclelengthwasbasedonthevalueof:m=1if10)]TJ /F5 7.97 Tf 6.59 0 Td[(2;otherwisem=3: Asapproacheszero,theoptimizationproblembecomesmoreillconditionedandtheconvergencespeedimproveswhenthecyclelengthisincreased. ThestoppingconditionforbothSpaRSAandAdaptiveSpaRSAwaskkxk+1)]TJ /F6 11.955 Tf 11.96 0 Td[(xkk1; wherekdenotesthenalvalueforinStep2ofSpaRSA,kk1isthemax-norm,andistheerrortolerance.ThisterminationconditionissuggestedbyVandenberghein[ 82 ].Aspointedoutearlier,xkisastationarypointwhenxk+1=xk.Forotherstoppingcriteria,see[ 43 ]or[ 86 ].Inthefollowingtables,\Ax"denotesthenumberoftimesthatavectorismultipliedbyAorAT,\cpu"istheCPUtimeinseconds,and\Obj"istheobjectivefunctionvalue. 4.5.1`2)]TJ /F3 11.955 Tf 11.95 0 Td[(`1Problems WecomparetheperformanceofAdaptiveSpaRSAwithSpaRSAbysolving`2)]TJ /F3 11.955 Tf 12.42 0 Td[(`1problemsofform( 4.2 )usingtherandomlygenerateddataintroducedin[ 56 86 ].ThematrixAisarandomknmatrix,withk=28andn=210.TheelementsofA 84

PAGE 85

arechosenfromaGaussiandistributionwithmeanzeroandvariance1=(2n).Theobservedvectorisb=Axtrue+n,wherethenoisenissampledfromaGaussiandistributionwithmeanzeroandvariance10)]TJ /F5 7.97 Tf 6.58 0 Td[(4.xtrueisavectorwith160randomlyplaced1spikeswithzerosintheremainingelements.Thisisatypicalsparsesignalrecoveryproblemwhichoftenarisesincompressedsensing[ 27 ].Wesolvedtheproblem( 4.2 )correspondingtotheerrortolerance10)]TJ /F5 7.97 Tf 6.58 0 Td[(5withdierentregularizationparametersbetween10)]TJ /F5 7.97 Tf 6.59 0 Td[(1and10)]TJ /F5 7.97 Tf 6.59 0 Td[(5.Table 4-1 reportstheaveragecputimes(seconds)andthenumberofmatrix-vectormultiplicationsover10runsforboththeoriginalSpaRSAalgorithmandanimplementationbasedonacontinuationmethod(see[ 43 ]).Theimplementationsusingthecontinuationmethodareindicatedby\/c"inTable 4-1 .TheseresultsshowthattheAdaptiveSpaRSAissignicantlyfasterthanSpaSRAwhennotusingthecontinuationtechnique.Theperformancegapdecreaseswhenthecontinuationtechniqueisapplied.Nonetheless,AdaptiveSpaRSAyieldsbetterperformance. Figure 4-1 plotserrorversusthenumberofmatrix-vectormultiplicationfor=10)]TJ /F5 7.97 Tf 6.58 0 Td[(4andtheimplementationwithoutcontinuation.Whentheerrorislarge,bothalgorithmhavethesameperformance.Astheerrortolerancedecreases,theperformanceoftheadaptivealgorithmissignicantlybetterthantheoriginalimplementation. Table4-1. Averageover10runsfor`2)]TJ /F3 11.955 Tf 11.95 0 Td[(`1problems 1e-1 1e-2 1e-3 1e-4 1e-5 Axcpu Axcpu Axcpu Axcpu Axcpu SpaRSA 65.3.07 706.4.56 3467.52.73 8802.96.86 5925.54.65 Adaptive 65.4.07 582.8.44 1998.81.58 4394.03.50 2911.92.36 SpaRSA/c 65.3.07 626.7.48 2172.11.67 684.9.52 474.8.36 Adaptive/c 65.4.07 569.0.44 1928.31.51 636.0.50 453.7.34 4.5.2ImageDeblurringProblems Inthissubsection,wepresentresultsfortwoimagerestorationproblemsbasedonimagesreferredtoasResolutionandCameraman.Theimagesare256256grayscaleimages;thatis,n=2562=65536.Theimagesareblurredbyconvolutionwithan88blurringmaskandnormallydistributednoisewithstandarddeviation0:0055isaddedto 85

PAGE 86

Figure4-1. Numberofmatrix-vectormultiplicationsversuserror thenalsignal(seeproblem701in[ 81 ]).Theimagerestorationproblemhastheform( 4.2 )where=0:00005andA=HWisthecompositionoftheblurmatrixandtheHaardiscretewavelettransform(DWT)operator.Forthesetestproblems,thecontinuationapproachisnofaster,andinsomecasessignicantlyslower,thantheimplementationwithoutcontinuation.Therefore,wesolvedthesetestproblemswithoutthecontinuationtechnique.TheresultsinTable 4-2 againindicatethattheadaptiveschemeyieldsmuchbetterperformanceastheerrortolerancedecreases. Table4-2. Deblurringimages error 1e-2 1e-3 1e-4 1e-5 AxcpuObj AxcpuObj AxcpuObj AxcpuObj Resolution SpaRSA 492.57.4843 884.80.3525 45824.74.2992 167988.27.2970 Adaptive 371.93.5619 734.02.3790 31617.28.2981 68135.90.2970 Cameraman SpaRSA 341.66.3491 773.99.2181 33217.08.1880 135669.45.1868 Adaptive 351.71.3380 633.31.2232 21511.20.1880 59931.4.1868 86

PAGE 87

Figure4-2. Deblurringtheresolutionimage Figure4-3. Deblurringthecameramanimage 87

PAGE 88

Figure4-4. Group-separablereconstruction 4.5.3Group-separableRegularizer Inthissubsection,weexamineperformanceusingthegroupseparableregularizers[ 86 ]forwhich (x)=nXi=1kx[i]k2; wherex[1];x[2];:::;x[m]aremdisjointsubvectorsofx.Thesmoothpartofcanbeexpressedasf(x)=1 2kAx)]TJ /F6 11.955 Tf 12.08 0 Td[(bk2,whereA2R10244096wasobtainedbyorthonormalizingtherowsofamatrixconstructedinSubsection 4.5.1 .Thetruevectorxtruehas4096componentsdividedintom=64groupsoflengthli=64.xtrueisgeneratedbyrandomlychoosing8groupsandllingthemwithnumberschosenfromaGaussiandistributionwithzeromeanandunitvariance,whileallothergroupsarelledwithzeros.Thetargetvectorisb=Axtrue+n,wherenisGaussiannoisewithmeanzeroandvariance10)]TJ /F5 7.97 Tf 6.59 0 Td[(4.Theregularizationparameterischosenassuggestedin[ 86 ]:=0:3kATbk1.Weran10testproblemswitherrortolerance=10)]TJ /F5 7.97 Tf 6.58 0 Td[(5andcomputetheaverageresults.AdaptiveSpaRSAsolvedthetestproblemin0.8420secondswith67.4matrix/vectormultiplications,whiletheSpaRSAobtainedsimilarperformance:0.8783secondsand69.1matrix/vectormultiplications.Figure 4-4 showstheresultobtainedbybothmethodsforonesample. 88

PAGE 89

Figure4-5. Phantomreconstruction 4.5.4Total-variationPhantomReconstruction Inthisexperiment,theimageistheShepp-Loganphantomofsize256256(see[ 4 10 ]).Theobjectivefunctionwas(x)=1 2kA(x))]TJ /F6 11.955 Tf 11.96 0 Td[(bk2+:01TV(x) whereAisa61362562matrixcorrespondingto6136locationsinthe2DFourierplane(masked_FFTinMatlab).Thetotalvariation(TV)regularizationisdenedasfollowsTV(x)=Xiq )]TJ /F2 11.955 Tf 5.48 -9.69 Td[(4hix2+(4vix)2 89

PAGE 90

Table4-3. Total-variationphantomreconstruction error 1e-2 1e-3 1e-4 AxcpuObj AxcpuObj AxcpuObj SpaRSA 142.5536.7311 14330.0614.7457 2877938.2514.1433 Adaptive 142.5736.7311 13627.3214.6840 731185.6214.1730 where4hiand4viarelinearoperatorscorrespondingtohorizontalandverticalrstorderdierences(see[ 5 ]).AsseeninTable 4-3 ,AdaptiveSpaRSAwasfasterthantheoriginalSpaRSAwhentheerrortolerancewassucientlysmall. 90

PAGE 91

CHAPTER5CONCLUSIONSANDFUTURERESEARCHDIRECTIONS Wehaveinvestigatedanellipsoidalbranchandboundalgorithmforglobaloptimization,anexactmethodforgraphpartitioning,andconvergenceanalysisandanimprovedversionforSpaRSAalgorithm.Themainresultsofthisdissertationwerepresentedinthreechapters. InChapter 2 ,agloballyconvergentbranchandboundalgorithmwasdevelopedinwhichtheobjectivefunctionwaswrittenasthedierenceofconvexfunctions.ThealgorithmwasbasedonananeunderestimategiveninTheorem 2.3.1 fortheconcavepartoftheobjectivefunctionrestrictedtoanellipsoid.AnalgorithmofLinandHan[ 59 60 ]forprojectingapointontoaconvexsetwasgeneralizedsoastoreplacetheirnormobjectivebyanarbitraryconvexfunction.Thisgeneralizationcouldbeemployedinthebranchandboundalgorithmforageneralobjectivefunctionwhentheconstraintsareconvex.Numericalexperimentsweregivenforarandomlygeneratedquadraticobjectivefunctionandrandomlygeneratedconvex,quadraticconstraints. InChapter 3 ,wepresentedanexactalgorithmforsolvingthegraphpartitioningproblemwithupperandlowerboundsonthesizeofeachsetinthepartition.Thealgorithmisbasedonacontinuousquadraticprogrammingformulationofthediscretepartitioningproblem.WeshowhowtotransformafeasiblexforthegraphpartitioningQP( 3.1 )toabinaryfeasiblepointywithanobjectivefunctionvaluewhichsatisesf(y)f(x).Thebinaryfeasiblepointcorrespondstoapartitionofthegraphverticesandf(y)istheweightofthecutedges.Atanystationarypointof( 3.1 )whichisnotalocalminimizer,Proposition 3.5.3 providesadescentdirectionthatcanbeusedtostrictlyimprovetheobjectivefunctionvalue.Wealsogavenecessaryandsucientconditionsforalocalminimizertobestrict. Inthebranchandboundalgorithm,theobjectivefunctionisdecomposedintothesumofaconvexandaconcavepart.Alowerboundfortheobjectivefunctionisachieved 91

PAGE 92

byreplacingtheconcavepartbyananeunderestimate.Twodierentdecompositionswereconsidered,onebasedontheminimumeigenvalueofthematrixintheobjectivefunction,andtheotherbasedonthesolutiontoasemideniteprogrammingproblem.Thesemideniteprogrammingapproachgenerallyledtomuchtighterlowerbounds.Inaseriesofnumericalexperiments,thenewalgorithmCQB(convexquadraticbranchandbound)wascompetitivewithstate-of-the-artpartitioningmethods;therelativeperformanceofCQBwasbetterforsparsegraphsthanfordensegraphs. InChapter 4 ,theconvergencepropertiesoftheSpaRSAalgorithm(SparseReconstructionbySeparableApproximation)ofWright,Nowak,andFigueiredo[ 86 ]wereanalyzed.WeestablishedsublinearconvergencewhenisconvexandtheGLLreferencefunctionvalue[ 34 ]isemployed.Whenisstronglyconvex,theconvergenceisR-linear.Forareferencefunctionvaluewhichsatises(R1){(R3),weprovedtheexistenceofaconvergentsubsequenceofiteratesthatapproachesastationarypoint.Foraslightlystrongerversionof(R3),givenin( 4.27 ),weshowedthatsublinearorlinearconvergenceagainholdwhenisconvexorstronglyconvexrespectively.Inaseriesofnumericalexperiments,itwasshownthatanAdaptiveSpaRSA,basedonarelaxedchoiceofthereferencefunctionvalueandacyclicBBiteration[ 20 42 ],oftenyieldedmuchfasterconvergence,especiallywhentheerrortolerancewassmall. Basedontherecentoptimization-basedformulation,wehavedevelopedanexactmethodforsolvingthegraphpartitioningproblemsofsmallormediumsize.Ourfutureworkwillcombineoptimization-basedtechniquestoobtainhighqualitypartitionswithmultilevelcombinatorictechniquestoachievefastcomputingtimesforlarge-scalegraphs. Itisknownfromtheareasofsignalandimagereconstructionthatthesolutionofthebasispursuitdenoisingproblemisoftenverysparse,consistingofmanyzerocomponents.Currently,weareworkingonexploitingtheactivesettechniquetosolvethenonsmoothoptimizationproblem. 92

PAGE 93

WealsowouldliketoextendthecyclicBarzilai-Borweinstepsizetodealwiththetotalvariationimagereconstruction,whichiscurrentlyoneofthemostactiveandpromisingtoolsinimagerecovery.Theproblemcanbeexpressedas minu2RNkukTV+ 2kAu)]TJ /F6 11.955 Tf 11.96 0 Td[(fk22; wheref2Rmisgiven,u2RNandkukTVistheisotropictotalvariation(TV)semi-normofuinthediscreteform.Inparticular,thematrixA2RmNisoftenoflargesizeandill-conditioned.Weareinterestedinthedevelopmentoffast,robustalgorithmsforsolvingthisproblem.Thenwecouldapplythendingtotheproblemofimagereconstructioninpartialparallelimagingtosignicantlyreducethescantimeandimprovethequalityofreconstructedimages. 93

PAGE 94

REFERENCES [1] L.T.H.An,Anecientalgorithmforgloballyminimizingaquadraticfunctionunderconvexquadraticconstraints,Math.Program.,87(2000),pp.401{426. [2] J.BarzilaiandJ.M.Borwein,Twopointstepsizegradientmethods,IMAJ.Numer.Anal.,8(1988),pp.141{148. [3] A.BeckandM.Teboulle,Afastiterativeshrinkage-thresholdingalgorithmforlinearinverseproblems,SIAMJournalonImagingSciences,2(2009),pp.183{202. [4] J.Bioucas-DiasandM.Figueiredo,TwIST:Two-stepiter-ativeshrinkage/thresholdingalgorithmforlinearinverseproblems.http://www.lx.it.pt/bioucas/TwIST/TwIST.htm. [5] J.Bioucas-Dias,M.Figueiredo,andJ.P.Oliveira,Totalvariation-basedimagedeconvolution:amajorization-minimizationapproach.,inProceedingsoftheIEEEInternationalConferenceonAcoustics,SpeechandSignalProcessing,vol.2,2006,pp.861{864. [6] E.G.Birgin,J.M.Martnez,andM.Raydan,Nonmonotonespectralprojectedgradientmethodsforconvexsets,SIAMJ.Optim.,10(2000),pp.1196{1211. [7] ,Algorithm813:SPG-softwareforconvex-constrainedoptimization,ACMTrans.Math.Softw.,27(2001),pp.340{349. [8] B.Borchers,CSDP,AClibraryforsemideniteprogramming,OptimizationMethodsandSoftware,11(1)(1999),pp.613{623. [9] L.Brunetta,M.Conforti,andG.Rinaldi,Abranch-and-cutalgorithmfortheequicutproblem,Math.Program.,78(1997),pp.243{263. [10] E.J.CandesandJ.Romberg,Practicalsignalrecoveryfromrandomprojections,WaveletApplicationsinSignalandImageProcessingXI,Proc.SPIEConf.,5914(2005). [11] U.V.CatalyurekandC.Aykanat,Hypergraph-partitioningbaseddecompositionforparallelsparse-matrixvectormultiplication,IEEETransactiononParallelandDistributedSystems,10(1999),pp.673{693. [12] M.Celis,J.E.Dennis,andR.A.Tapia,Atrustregionstrategyfornonlinearequalityconstrainedoptimization,inNumericalOptimization1984,Philadelphia,PA,1985,SIAM,pp.71{82. [13] A.Chambolle,R.A.DeVore,N.Y.Lee,andB.J.Lucier,Nonlinearwaveletimageprocessing:Variationalproblems,compression,andnoiseremovalthroughwaveletshrinkage,IEEETrans.ImageProcess.,7(1998),pp.319{335. 94

PAGE 95

[14] S.Chen,D.Donoho,andM.Saunders,Atomicdecompositionbybasispursuit,SIAMJ.Sci.Comput.,20(1998),pp.33{61. [15] X.D.ChenandY.Yuan,Anoteonquadraticforms,Math.Program.,86(1999),pp.187{197. [16] ,Onlocalsolutionsofthecdtsubproblem,SIAMJ.Optim.,10(1999),pp.359{383. [17] O.Collins,S.Dolinar,R.McEliece,andF.Pollara,AVLSIdecomposi-tionofthedeBruijngraph,J.ACM,39(1992),pp.931{948. [18] P.L.CombettesandT.Pennanen,Proximalmethodsforcohypomonotoneotperators,SIAMJ.Control,43(2004),pp.731{742. [19] Y.H.Dai,Alternatestepsizegradientmethod,Optimization,52(2003),pp.395{415. [20] Y.H.Dai,W.W.Hager,K.Schittkowski,andH.Zhang,ThecyclicBarzilai-Borweinmethodforunconstrainedoptimization,IMAJ.Numer.Anal.,26(2006),pp.604{627. [21] I.Daubechies,M.Defrise,andC.D.Mol,Aniterativethresholdingalgorithmforlinearproblemswithasparsityconstraint,Comm.PureAppl.Math.,57(2004),pp.1413{1457. [22] K.Devine,E.Boman,R.Heaphy,R.Bisseling,andU.Catalyurek,Parallelhypergraphpartitioningforscienticcomputing,inProc.of20thInternationalParallelandDistributedProcessingSymposium(IPDPS'06),IEEE,2006. [23] J.J.Dongarra,Performanceofvariouscomputersusingstandardlinearequationssoftware,Tech.Reportcs-89-85,UniversityofTennessee,Knoxville,TN,USA,2008. [24] J.Falkner,F.Rendl,andH.Wolkowicz,Acomputationalstudyofgraphpartitioning,Math.Programming,66(1994),pp.211{240. [25] R.Feldmann,B.Monien,P.Mysliwietz,andS.Tschoke,AbetterupperboundonthebisectionwidthofdeBruijnnetworks,inSTACS97,vol.1200ofLectureNotesinComputerScience,Springer,Berlin,1997,pp.511{522. [26] C.E.Ferreira,A.Martin,C.C.deSouza,R.Weismantel,andL.A.Wolsey,Thenodecapacitatedgraphpartitioningproblem:Acomputationalstudy,Math.Programming,81(1998),pp.229{256. [27] M.A.T.Figueiredo,R.D.Nowak,andS.J.Wright,Gradientprojectionforsparsereconstruction:Applicationtocompressedsensingandotherinverseprob-lems,IEEEJournalonSelectedTopicsinSignalProcessing,1(2007),pp.586{597. [28] T.FigueiredoandR.D.Nowak,AnEMalgorithmforwavelet-basedimagerestoration,IEEETrans.ImageProcess.,12(2003),pp.906{916. 95

PAGE 96

[29] C.A.FloudasandV.Visweswaran,Quadraticoptimization,inHandbookofGlobalOptimization,R.HorstandP.Pardalos,eds.,KluwerAcademic,1994,pp.217{270. [30] A.Friedlander,J.M.Martnez,B.Molina,andM.Raydan,Gradi-entmethodwithretardsandgeneralizations,SIAMJ.Numer.Anal.,36(1999),pp.275{289. [31] M.R.Garey,D.S.Johnson,andL.Stockmeyer,SomesimpliedNP-completegraphproblems,TheoreticalComputerScience,1(1976),pp.237{267. [32] J.R.Gilbert,G.L.Miller,andS.H.Teng,Geometricmeshpartitioning:Implementationandexperiments,SIAMJ.Sci.Comput.,19(1998),pp.2091{2110. [33] L.Grigori,E.Boman,S.Donfack,andT.A.Davis,Hypergraph-basedunsymmetricnesteddissectionorderingforsparseLUfactorization,SIAMJ.Sci.Comput.,(2008).undersubmission. [34] L.Grippo,F.Lampariello,andS.Lucidi,Anonmonotonelinesearchtech-niqueforNewton'smethod,SIAMJ.Numer.Anal.,23(1986),pp.707{716. [35] W.W.Hager,AppliedNumericalLinearAlgebra,Prentice-Hall,EnglewoodClis,NJ,1988. [36] W.W.HagerandS.Gowda,Stabilityinthepresenceofdegeneracyanderrorestimation,Math.Program.,85(1999),pp.181{192. [37] W.W.HagerandY.Krylyuk,Graphpartitioningandcontinuousquadraticprogramming,SIAMJ.Disc.Math.,12(1999),pp.500{523. [38] ,Multisetgraphpartitioning,Math.Meth.Oper.Res.,55(2002),pp.1{10. [39] W.W.HagerandD.T.Phan,Anellipsoidalbranchandboundalgorithmforglobaloptimization,SIAMJ.Optim.,20(2009),pp.740{758. [40] W.W.Hager,D.T.Phan,andH.Zhang,Anexactalgorithmforgraphpartitioning,Math.Program.,(2009,submitted). [41] ,Gradient-basedalgorithmsforsparserecovery,SIAMJournalonImagingSciences,(2009,submitted). [42] W.W.HagerandH.Zhang,Anewactivesetalgorithmforboxconstrainedoptimization,SIAMJ.Optim.,17(2006),pp.526{557. [43] E.Hale,W.Yin,andY.Zhang,Axed-pointcontinuationmethodfor`1-regularizedminimizationwithapplicationstocompressedsensing,tech.report,RiceUniversity,July2007. [44] M.Heinkenschloss,Onthesolutionofatwoballtrustregionsubproblem,Math.Programming,64(1994),pp.249{276. 96

PAGE 97

[45] B.HendricksonandR.Leland,TheChacouser'sguide-Version2.0,SandiaNationalLaboratories,TechnicalReportSAND94-2692,1994. [46] ,Animprovedspectralgraphpartitioningalgorithmformappingparallelcomputa-tions,SIAMJ.Sci.Comput.,16(1995),pp.452{469. [47] ,Amultilevelalgorithmforpartitioninggraphs,inProc.Supercomputing'95,ACM,Nov.1995. [48] R.Horst,P.M.Pardalos,andN.V.Thoai,IntroductiontoGlobalOptimiza-tion,KluwerAcademicPublishers,Dordrecht,Holland,1995. [49] A.N.Iusem,T.Pennanen,andB.F.Svaiter,Inexactvariantsoftheproximalpointalgorithmwithoutmonotonicity,SIAMJ.Optim.,13(2003),pp.1080{1097. [50] E.L.Johnson,A.Mehrotra,andG.L.Nemhauser,Min-cutclustering,Math.Program.,62(1993),pp.133{151. [51] T.Johnson,Aconcurrentdynamictaskgraph,inProc.1993Intl.Conf.onParallelProcessing,1993.(TR-93-011,CISEDept.,Univ.ofFlorida). [52] S.E.Karisch,F.Rendl,andJ.Clausen,Solvinggraphbisectionproblemswithsemideniteprogramming,INFORMSJournalonComputing,12(2000),pp.177{191. [53] G.KarypisandV.Kumar,Afastandhighqualitymultilevelschemeforpartition-ingirregulargraphs,SIAMJ.Sci.Comput.,20(1998),pp.359{392. [54] ,Parallelmultilevelk-waypartitioningschemeforirregulargraphs,SIAMReview,41(1999),pp.278{300. [55] ,Multilevelk-wayhypergraphpartitioning,VLSIDesign,11(2000),pp.285{300. [56] S.-J.Kim,K.Koh,M.Lustig,S.Boyd,andD.Gorinevsky,Aninterior-pointmethodforlarge-scale`1-regularizedleastsquares,IEEEJournalonSelectedTopicsinSignalProcessing,1(2007),pp.606{617. [57] T.Lengauer,CombinatorialAlgorithmsforIntegratedCircuitLayout,JohnWiley,Chichester,1990. [58] G.D.LiandY.Yuan,Computingacelis-dennis-tapiastep,J.Comput.Math.,23(2005),pp.463{478. [59] A.Lin,Aclassofmethodforprojectiononaconvexset,AdvancedModelingandOptimization,5(2003),pp.211{221. [60] A.LinandS.P.Han,Aclassofmethodsforprojectionontheintersectionofseveralellipsoids,SIAMJ.Optim.,15(2005),pp.129{138. [61] J.Linderoth,Asimplicialbranchandboundalgorithmforsolvingquadraticallyconstrainedquadraticprograms,Math.Program.,103(2005),pp.251{282. 97

PAGE 98

[62] J.M.MartinezandS.A.Santos,Atrust-regionstrategyforminimizationonarbitrarydomains,Math.Program.,68(1995),pp.267{301. [63] J.Mitchell,Branch-and-cutforthek-wayequipartitionproblem,tech.report,DepartmentofMathematicalSciences,RensselaerPolytechnicInstitute,2001. [64] K.G.MurtyandS.N.Kabadi,SomeNP-completeproblemsinquadraticandlinearprogramming,Math.Program.,39(1987),pp.117{129. [65] Y.Nesterov,Gradientmethodsforminimizingcompositeobjectivefunction,COREDiscussionPapers2007/76,UniversitcatholiquedeLouvain,CenterforOperationsResearchandEconometrics(CORE),Sept.2007. [66] P.M.PardalosandS.A.Vavasis,QuadraticprogrammingwithonenegativeeigenvalueisNP-hard,J.GlobalOptim.,1(1991),pp.15{22. [67] S.C.Park,T.A.Davis,W.W.Hager,andH.Zhang,Quadraticprogram-mingtechniquesingraphpartitioning,SIAMJ.Sci.Comput.,(2006,submitted). [68] F.Pellegrini,J.Roman,andP.R.Amestoy,Hybridizingnesteddissec-tionandhaloapproximateminimumdegreeforecientsparsematrixordering,Concurrency:Pract.Exp.,12(2000),pp.68{84. [69] J.M.PengandY.Yuan,Optimalityconditionsfortheminimizationofaquadraticwithtwoquadraticconstraints,SIAMJ.Optim.,7(1997),pp.579{594. [70] T.Pennanen,Localconvergenceoftheproximalpointalgorithmandmultipliermethodswithoutmonotonicity,Math.Oper.Res.,27(2002),pp.170{191. [71] R.PreisandR.Diekmann,ThePARTYpartitioninglibraryuserguide-version1.1,1996. [72] U.Raber,Asimplicialbranchandboundmethodforsolvingnonconvexall-quadraticprograms,J.GlobalOptim.,13(1998),pp.417{432. [73] M.RaydanandB.F.Svaiter,RelaxedsteepestdescentandCauchy-Barzilai-Borweinmethod,Comput.Optim.Appl.,21(2002),pp.155{167. [74] R.T.Rockafellar,Convexanalysis,PrincetonUniv.Press,1970. [75] N.Sensen,Lowerboundsandexactalgorithmsforthegraphpartitioningproblemusingmulticommodityows,inLectureNotesinComputerScience,vol.2161,Springer-Verlag,2001,pp.391{403. [76] N.Z.Shor,Cut-omethodwithspaceextensioninconvexprogrammingproblems,CyberneticsandSystemAnalysis,1(1977),pp.94{97. [77] A.J.Soper,C.Walshaw,andM.Cross,Acombinedmultilevelsearchandmultileveloptimizationapproachtograph-partition,J.GlobalOptim.,29(2004),pp.225{241. 98

PAGE 99

[78] C.Souza,R.Keunings,L.Wolsey,andO.Zone,Anewapproachtomini-mizingthefrontwidthinniteelementcalculations,ComputerMethodsinAppliedMechanicsandEngineering,111(1994),pp.323{334. [79] S.-H.Teng,ProvablygoodpartitioningandloadbalancingalgorithmsforparalleladaptiveN-bodysimulation,SIAMJ.Sci.Comput.,19(1998),pp.635{656. [80] L.N.TrefethenandD.BauIII,NumericalLinearAlgebra,SIAM,Philadelphia,1997. [81] E.vandenBerg,M.P.Friedlander,G.Hennenfent,F.J.Herrmann,R.Saab,andO.Yilmaz,Algorithm890:Sparco:Atestingframeworkforsparsereconstruction,ACMTrans.Math.Softw.,35(2009),pp.1{16. [82] L.Vandenberghe,Gradientmethodsfornonsmoothproblems(lecturenote-spring2009).http://www.ee.ucla.edu/vandenbe/ee236c.html. [83] J.P.Vial,Strongandweakconvexityofsetsandfunctions,Math.Oper.Res.,8(2)(1983),pp.231{259. [84] C.VoneschandM.Unser,Fastiterativethresholdingalgorithmforwavelet-regularizeddeconvolution,inProceedingsoftheSPIEOpticsandPhotonics2007ConferenceonMathematicalMethods:WaveletXII,vol.6701,SanDiego,CA,2007,pp.1{5. [85] C.Walshaw,M.Cross,andM.Everett,Paralleldynamicgraphpartitioningforadaptiveunstructuredmeshs,J.ParallelDistrib.Comput.,47(1997),pp.102{108. [86] S.J.Wright,R.D.Nowak,andM.A.T.Figueiredo,Sparsereconstructionbyseparableapproximation,IEEETrans.SignalProcess.,57(2009),pp.2479{2493. [87] Y.YeandS.Zhang,Newresultsonquadraticminimization,SIAMJ.Optim.,14(2003),pp.245{267. [88] D.B.YudinandA.S.Nemirovski,Informationalcomplexityandeectivemethodsforthesolutionofconvexextremalproblems,Ekonom.Mat.Metody.,12(1976),pp.550{559(inRussian). 99

PAGE 100

BIOGRAPHICALSKETCH D~ungPhanwasborninHatinhprovince,Vietnam.Heobtainedhismaster'sdegreeinHighPerformanceComputationforEngineeredSystemsattheSingapore-MITAlliance,NationalUniversityofSingapore(NUS)in2004.BeforearrivingatNUS,hewasanundergraduateattheCollegeofSciencebelongingtotheVietnamNationalUniversity(VNU)atHanoi,andearnedaBachelorofSciencedegreeinComputationalMathematicsin2001.Allhissecondaryeducationhappenedatthehighschoolforgiftedstudentsinmathematicsandinformatics,VNU. 100