Citation
Accelerated Bundle Level Type Methods for Large Scale Convex Optimization

Material Information

Title:
Accelerated Bundle Level Type Methods for Large Scale Convex Optimization
Creator:
Zhang, Wei
Publisher:
University of Florida
Publication Date:
Language:
English

Thesis/Dissertation Information

Degree:
Doctorate ( Ph.D.)
Degree Grantor:
University of Florida
Degree Disciplines:
Mathematics
Committee Chair:
CHEN,YUNMEI
Committee Co-Chair:
LAN,GUANGHUI
Committee Members:
MCCULLOUGH,SCOTT A
ALLADI,KRISHNASWAMI
JURY,MICHAEL THOMAS
WU,SAMUEL SHANGWU

Subjects

Subjects / Keywords:
accelerated-algorithm
bundle-level-method
convex-optimization

Notes

General Note:
This Ph.D. dissertation is focused on developing accelerated bundle-level (BL) type methods for solving large-scale convex optimization problems (COPs) arising from many data analysis and machine learning problems. Comparing to gradient-descent type methods, the BL type methods not only achieve the optimal iteration complexity for smooth, weakly smooth and nonsmooth COPs, but also have the advantages of utilizing historical information and restricted memory to speed up convergence in practice. In the first part, we proposed several fast accelerated BL methods for ball-constrained and unconstrained COPs to improve the efficiency and applicability of the existing BL type methods. By reducing the number of involved subproblems and introducing an efficient approach to exactly solve the subproblem, the proposed methods could achieve higher accuracy of the solution with faster convergence speed. The applications to quadratic programming and image processing are also presented. In the second part, we developed the inexact accelerated BL methods for solving smooth COPs and a class of saddle-point problem, where the exact first-order information of the objective function is either not available or very expensive to compute. We further extended the proposed algorithms to solve strongly convex functions with better iteration complexities, and discussed the dependence between the accuracy of the inexact oracle and the best accuracy of the approximate solution the algorithm could achieve. In the last part, we introduced several accelerated BL methods for functional constrained optimization: $\min_{x\in X} f(x), s.t.\ g(x)\le 0$. This problem is considered to be difficult if the projection onto the feasible set $\{x\in X: g(x) \le 0\}$ is hard to compute. We reduced the original problem to a non-functional constrained COP and applied the fast accelerated BL methods with some important modifications to solve the non-functional constrained COP. Moreover, by incorporating Nesterov's smoothing technique, we further extended the proposed methods to solve constrained saddle-point problem. Both the iteration complexity and practical performance of the proposed algorithms are studied.

Record Information

Source Institution:
UFRGP
Rights Management:
All applicable rights reserved by the source institution and holding location.
Embargo Date:
8/31/2018

Downloads

This item has the following downloads:


Full Text

PAGE 1

ACCELERATEDBUNDLELEVELTYPEMETHODSFORLARGESCALECONVEXOPTIMIZATIONByWEIZHANGADISSERTATIONPRESENTEDTOTHEGRADUATESCHOOLOFTHEUNIVERSITYOFFLORIDAINPARTIALFULFILLMENTOFTHEREQUIREMENTSFORTHEDEGREEOFDOCTOROFPHILOSOPHYUNIVERSITYOFFLORIDA2017

PAGE 2

c2017WeiZhang

PAGE 3

Tomyadvisorsandmyfamily

PAGE 4

ACKNOWLEDGMENTSIwouldliketoexpressmysincereappreciationtomyadvisorDr.YunmeiChen.Withoutherprofessionalguidance,invaluableadvicesandconsistentsupportthroughoutmyentiregraduatestudy,thisthesiswouldnothavebeenpossible.Sheisveryknowledgeableandfriendly.Ireallyenjoytheexperiencelearningfromandworkingwithher.IamdeeplygratefultomyCo-advisor,Dr.GuanghuiLan,forsharingmehisgreatknowledgeinnonlinearprogrammingandoptimizationandgivingmeinvaluableadvicesandhelpformyresearch.Next,IwouldliketothankDr.ScottMccullough,Dr.MichaelJury,Dr.KrishnaswamiAlladiandDr.SamuelWuforservingasmydoctoralcommitteemembers,attendingmyoralexamandPh.Ddefenseandprovidingconstructiveadvicestomyresearchandthesis.Furthermore,IwouldliketothankDr.JeanLarsonandDr.PerterSinforservingasourgraduatecoordinate,Dr.ScottMcculloughandDr.YunmeiChenforwritingrecommendationlettersforme,Ireallyappreciatetheirhelpandkindness.IwouldalsoliketothankallDr.Chen'sgroupmembers,OuyangYuyuan,HaoZhang,XianqiLiandChenxiChenfortheirsupportanddiscussionthroughoutmygraduatestudy.Finally,andmostimportantly,Iwouldexpressmyspecialthankstomyfamilyfortheirconsistentloveandsupport. 4

PAGE 5

TABLEOFCONTENTS page ACKNOWLEDGMENTS ................................... 4 LISTOFTABLES ...................................... 7 LISTOFFIGURES ..................................... 8 ABSTRACT ......................................... 9 CHAPTER 1INTRODUCTION ................................... 11 1.1ProblemofInterest ............................... 11 1.2DevelopmentofBundleLevelTypeMethods .................. 13 1.3OutlineoftheDissertation ............................ 16 2FASTACCELERATEDBUNDLELEVELMETHODSFORCONVEXOPTIMIZATION 19 2.1Background ................................... 19 2.2FastAcceleratedBundleLevelMethodsforBall-ConstrainedConvexOptimization 20 2.2.1FAPLMethodforBall-ConstrainedConvexOptimization ........ 21 2.2.2FUSLMethodforBall-ConstrainedSaddle-PointProblem ....... 28 2.2.3SolvingtheSubproblemofFAPLandFUSLMethods .......... 36 2.3FastAcceleratedBundleLevelMethodsforUnconstrainedConvexOptimization 38 2.3.1ExpansionAlgorithmforUnconstrainedConvexOptimization ...... 38 2.3.2ExtendingFAPLandFUSLforUnconstrainedConvexOptimization .. 46 2.4GeneralizationtoStronglyConvexOptimization ................ 48 2.5NumericalExperiments ............................. 53 2.5.1QuadraticProgramming ......................... 53 2.5.2TotalVariationBasedImageReconstruction .............. 58 2.5.3PartiallyParallelImaging ......................... 61 3ACCELERATEDBUNDLELEVELMETHODSWITHINEXACTORACLE ..... 65 3.1Background ................................... 65 3.2FastAcceleratedProx-LevelMethodswithInexactOracle ........... 69 3.2.1IFAPLMethodforSmoothBall-ConstrainedConvexOptimization ... 70 3.2.2IFAPLSMethodforUnconstrainedStronglyConvexOptimization ... 79 3.3InexactFastUniformSmoothingLevelMethodsforSaddle-PointProblem ... 83 3.3.1IFUSLMethodforBall-ConstrainedSaddle-PointProblem ....... 84 3.3.2IFUSLSMethodforStronglyConvexSaddle-PointProblem ...... 95 4ACCELERATEDBUNDLELEVELMETHODSFORFUNCTIONALCONSTRAINEDOPTIMIZATION .................................... 100 4.1Background ................................... 100 5

PAGE 6

4.2ACBLMethodforFunctionalConstrainedConvexOptimization ........ 103 4.3ACPLMethodforFunctionalConstrainedConvexOptimization ........ 114 4.4ACSLMethodforConstrainedSaddle-PointProblem .............. 121 REFERENCES ........................................ 128 BIOGRAPHICALSKETCH ................................. 132 6

PAGE 7

LISTOFTABLES Table page 2-1UniformlydistributedQPinstances .......................... 54 2-2GaussiandistributedQPinstances ........................... 55 2-3ComparisontoMatlabsolver ............................. 55 2-4UnconstrainedQPinstances .............................. 57 7

PAGE 8

LISTOFFIGURES Figure page 2-1TV-basedreconstruction(Shepp-Loganphantom) .................. 60 2-2TV-basedreconstruction(brainimage) ........................ 61 2-3SensitivitymapandCartesianmasks ......................... 63 2-4PPIimagereconstruction(acquisitionrate:14%) .................. 63 2-5PPIimagereconstruction(acquisitionrate:10%) .................. 64 8

PAGE 9

AbstractofDissertationPresentedtotheGraduateSchooloftheUniversityofFloridainPartialFulllmentoftheRequirementsfortheDegreeofDoctorofPhilosophyACCELERATEDBUNDLELEVELTYPEMETHODSFORLARGESCALECONVEXOPTIMIZATIONByWeiZhangAugust2017Chair:YunmeiChenCochair:GuanghuiLanMajor:MathematicsThisPhDdissertationisfocusedondevelopingacceleratedbundle-level(BL)typemethodsforsolvinglarge-scaleconvexoptimizationproblems(COPs)arisingfrommanydataanalysisandmachinelearningproblems.Comparingtogradient-descenttypemethods,theBLtypemethodsnotonlyachievetheoptimaliterationcomplexitiesforsmooth,weaklysmoothandnonsmoothCOPs,butalsohavetheadvantagesofutilizinghistoricalinformationandrestrictedmemorytospeedupconvergenceinpractice.Intherstpart,weproposedseveralfastacceleratedBLmethodsforball-constrainedandunconstrainedCOPstoimprovetheeciencyandapplicabilityoftheexistingBLtypemethods.Byreducingthenumberofinvolvedsubproblemsandintroducinganecientapproachtoexactlysolvethesubproblem,theproposedmethodscouldachievehigheraccuracyofsolutionwithfasterconvergencespeed.Theapplicationstoquadraticprogrammingandimageprocessingarealsopresented.Inthesecondpart,wedevelopedtheinexactacceleratedBLmethodsforsolvingsmoothCOPsandaclassofsaddle-pointproblem,wheretheexactrst-orderinformationoftheobjectivefunctioniseithernotavailableorveryexpensivetocompute.Wefurtherextendedtheproposedalgorithmstosolvestronglyconvexfunctionswithbetteriterationcomplexities,anddiscussedthedependencebetweentheaccuracyoftheinexactoracleandthebestaccuracyoftheapproximatesolutionthealgorithmscouldachieve. 9

PAGE 10

Inthelastpart,weintroducedseveralacceleratedBLmethodsforfunctionalconstrainedoptimization:minx2Xf(x);s:t:g(x)0.Thisproblemisconsideredtobedicultiftheprojectionontothefeasiblesetfx2X:g(x)0gishardtocompute.Wereducedtheoriginalproblemtoanon-functionalconstrainedCOPandappliedthefastacceleratedBLmethodswithsomeimportantmodicationstosolvethenon-functionalconstrainedCOP.Moreover,byincorporatingNesterov'ssmoothingtechnique,wefurtherextendedtheproposedmethodstosolveaclassofconstrainedsaddle-pointproblem.Boththeiterationcomplexityandpracticalperformanceoftheproposedalgorithmsarestudied. 10

PAGE 11

CHAPTER1INTRODUCTION 1.1ProblemofInterestManydataanalysisproblemsareoftenmodeledasthefollowingbroadclassofconvexoptimizationproblem(COP): f:=minx2Xf(x);(1{1)whereXRnisaclosedconvexset,andf:X!Risaconvexfunctionsatisfying f(y))]TJ /F6 11.955 Tf 11.95 0 Td[(f(x))-222(hf0(x);y)]TJ /F6 11.955 Tf 11.96 0 Td[(xiM 1+ky)]TJ /F6 11.955 Tf 11.96 0 Td[(xk1+;8x;y2B(x;R);(1{2)forsomeM>0and2[0;1].Wecallf()isnonsmoothif=0,smoothif=0,andweaklysmoothif0<<1.Forexample,theclassicridgeregressionmodel(namely,Tikhonovregularization)instatisticallearningestimatesparametersbymin2Rnky)]TJ /F6 11.955 Tf 11.96 0 Td[(Ak2subjecttokk; (1{3)wherekkdenotestheEuclideannorm,ydescribestheobservedoutcome,Aarethepredictorsintheobserveddata,andisaregularizationparameter.Theabovemodelcanbeviewedasaspecialcaseof( 1{1 )withx=,f(x)=ky)]TJ /F6 11.955 Tf 12.59 0 Td[(Axk2,andX=fx2Rn:kxkg.Anotherimportantexampleistheclassicaltwo-dimensionaltotalvariation(TV)basedimagereconstructionproblemgivenby:minu2Rn1 2kAu)]TJ /F6 11.955 Tf 11.96 0 Td[(bk2+kukTV; (1{4)whereAisthemeasurementmatrix,uisthen-vectorformofatwo-dimensionalimagetotheconstructed,brepresentstheobserveddata,kkTVisthediscreteTVsemi-norm,andistheregularizationparameter.Problem( 1{4 )canalsobecastedas( 1{1 )bysettingx=u,f(x)=kAx)]TJ /F6 11.955 Tf 12.03 0 Td[(bk2=2+kxkTV,andX=Rn.Itisworthnotingthatwhileproblem( 1{3 )hasanEuclideanballconstraint,problem( 1{4 )isanunconstrainedCOP.Moreover,theobjective 11

PAGE 12

functionin( 1{3 )issmooth,whiletheobjectivefunctionin( 1{4 )isdenedasthesummationofasmoothtermkAx)]TJ /F6 11.955 Tf 11.95 0 Td[(bk2=2andanonsmoothtermkxkTV.Duetothehighdimensionalityofxformanyapplicationsindataanalysisandimaging,muchrecentresearcheorthasbeendirectedtothedevelopmentofecientrst-ordermethodsforsolving( 1{1 ).First-ordermethodsusegradients(orsubgradients)offexclusivelyandhencepossesssignicantlyreducediterationcostthansecond-ordermethods.Theeciencyofthesealgorithmsareoftenmeasuredbytheiriterationcomplexityintermsofthenumberof(sub)gradientevaluationsrequiredtondanapproximatesolutionof( 1{1 ).Inviewoftheclassiccomplexitytheory[ 1 ],foranyrst-ordermethodsthenumberof(sub)gradientevaluationsrequiredtondan-solutionof( 1{1 )(i.e.,apointx2Rnsatisfyingf(x))]TJ /F6 11.955 Tf 10.75 0 Td[(f)cannotbesmallerthanO(1=2)iffisnonsmooth.Thiscanbeachieved,forexample,bytraditionalsubgradientmethods.Forasmoothf,theoptimaliterationcomplexityisO(1=p ),whichcanbeachieved,forexample,byNesterov'sacceleratedgradient(AG)algorithms[ 2 { 4 ].Recently,byadaptingNesterov'sAGschemesandsmoothingtechnique[ 2 ],severalpopularclassesofrst-ordermethodshavebeendevelopedtosignicantlyimprovetheiriterationcomplexitybounds.Forinstance,theacceleratedprimaldual(APD)algorithm[ 5 ]andacceleratedhybridproximalextragradientalgorithm[ 6 ],whichexhibittheoptimaliterativecomplexityforsolvingabroadclassofsaddle-point(SP)problems,andseveralvariantsofalternatingdirectionmethodofmultipliers(ADMM)[ 7 { 12 ],haveimprovedtheiterationcomplexityregardingtothesmoothcomponent.Inthisthesis,wefocusonadierentclassofrst-ordermethods,i.e.,bundle-level(BL)typemethods,whichnotonlycanachievetheoptimaliterationcomplexitiesforsolvingsmooth,weaklysmoothandnonsmoothCOPsbutalsohavetheadvantagesofutilizingrestrictedmemoryandhistoricalrst-orderinformationthroughcuttingplanemodelstosignicantlyacceleratethenumericalperformanceofthegradientdescenttypemethodsasmentionedabove.WerstgiveareviewonseveraldierenttypesofBLmethods. 12

PAGE 13

1.2DevelopmentofBundleLevelTypeMethodsTheBLmethodoriginatedfromthewell-knownKelley'scutting-planemethodin1960[ 13 ].ConsidertheCOPfX:=minx2Xf(x); (1{5)whereXisacompactconvexsetandfisaclosedconvexfunction.ThefundamentalideaofthecuttingplanemethodistogenerateasequenceofpiecewiselinearfunctionstoapproximatefonX.Inparticular,givenx1;x2;:::;xk2X,thismethodapproximatesfby mk(x):=maxfh(xi;x);1ikg;(1{6)andcomputestheiteratexk+1by xk+12Argminx2Xmk(x);(1{7)where h(z;x):=f(z)+hf0(z);x)]TJ /F6 11.955 Tf 11.95 0 Td[(zi;(1{8)andf0(x)2@f(x),where@f(x)denotesthesubdierentialoffatx.Clearly,thefunctionsmi,i=1;2;:::,satisfymi(x)mi+1(x)f(x)foranyx2X,andareidenticaltofatthosesearchpointsxi,i=1;:::;k.However,theinaccuracyandinstabilityofthepiecewiselinearapproximationmkoverthewholefeasiblesetXmayaecttheselectionofnewiterates,andtheaboveschemeconvergesslowlyboththeoreticallyandpractically[ 1 4 ].SomeimportantimprovementsofKelley'smethodhavebeenmadein1990sunderthenameofbundlemethods(see,e.g.,[ 14 { 16 ]).Inparticular,byincorporatingthelevelsetsintoKelley'smethod,Lemarechal,NemirovskiiandNesterov[ 16 ]proposedin1995theclassicBLmethodbyperformingaseriesofprojectionsovertheapproximatelevelsets.Givenx1;x2;:::;xk,theclassicBLiterationconsistsofthefollowingthreesteps: a) Set fk:=minff(xi);1ikgandcomputealowerboundonfXbyf k=minx2Xmk(x): 13

PAGE 14

b) Setthelevellk=f k+(1)]TJ /F6 11.955 Tf 11.95 0 Td[() fkforsome2(0;1). c) SetXk:=fx2X:mk(x)lkganddeterminethenewiterateby xk+1=argminx2Xkkx)]TJ /F6 11.955 Tf 11.96 0 Td[(xkk2:(1{9)IntheBLmethod,thelocalizerXkisusedtoapproximatethelevelsetLk:=fx:f(x)lkg,becausetheprojectionoverLkisoftentoodiculttocompute.Intuitively,askincreases,thevalueoflkwillconvergetofX,andconsequentlybothLkandXkwillconvergetothesetofoptimalsolutionstoproblem( 3{1 ).Itisshownin[ 16 ]thatthenumberofBLiterationsrequiredtondan-solutiontoproblem( 3{1 ),i.e.,apoint^x2Xs.t.f(^x))]TJ /F6 11.955 Tf 12.36 0 Td[(fX,canbeboundedbyO(1=2),whichisoptimalforgeneralnonsmoothconvexoptimizationintheblack-boxmodel.ObservethatfortheaboveBLmethods,thelocalizerXkaccumulatesconstraints,andhencethesubprobleminStepc)becomesmoreandmoreexpensivetosolve.Inordertoovercomethisdiculty,somerestricted-memoryBLalgorithmshavebeendevelopedin[ 15 17 ].Inparticular,Ben-TalandNemirovski[ 17 ]introducedthenon-Euclideanrestrictedmemorylevel(NERML)method,inwhichthenumberofextralinearconstraintsinXkcanbeassmallas1or2,withoutaectingtheoptimaliterationcomplexity.Moreover,theobjectivefunctionkk2in( 1{9 )isreplacedbyageneralBregmandistanced()forexploitingthegeometryofthefeasiblesetX.NERMLisoftenregardedasstate-of-the-artforlarge-scalenonsmoothconvexoptimizationasitsubstantiallyoutperformssubgradienttypemethodsinpractice.SomemorerecentdevelopmentofinexactproximalbundlemethodsandBLmethodscouldbefoundin[ 18 { 25 ].WhiletheclassicBLmethodwasoptimalforsolvingnonsmoothCOPsonly,Lan[ 26 ]recentlysignicantlygeneralizedthismethodsothatitcanoptimallysolveanyblack-boxCOPs,includingnonsmooth,smoothandweaklysmoothCOPs.Inparticular,forproblem( 3{1 )overcompactfeasiblesetX,thetwonewBLmethodsproposedin[ 26 ],i.e.,theacceleratedbundle-level(ABL)andacceleratedprox-level(APL)methods,cansolvethese 14

PAGE 15

problemsoptimallywithoutrequiringanyinformationonproblemparameters.TheABLmethodcanbeviewedasanacceleratedversionoftheclassicBLmethod.SameastheclassicBLmethod,thelowerboundonfXisestimatedfromthecuttingplanemodelmkin( 1{6 ),theupperboundonfXisgivenbythebestobjectivevaluefoundsofar.ThenoveltyoftheABLmethodexistsinthatthreedierentsequences,i.e.,fxlkg;fxkgandfxukg,areusedforupdatingthelowerbound,prox-center,andupperboundrespectively,whichleadstoitsacceleratediterationcomplexityforsmoothandweaklysmoothproblems.TheAPLmethodisamorepractical,restrictedmemoryversionoftheABLmethod,whichalsoemploysnon-Euclideanprox-functionstoexplorethegeometryofthefeasiblesetX.WenowprovideabriefdescriptionoftheAPLmethod.Thismethodconsistsofmultiplephases,andeachphasecallsagapreductionproceduretoreducethegapbetweenthelowerandupperboundsonfbyaconstantfactor.Morespecically,atphases,giventheinitiallowerboundlbsandupperboundubs,theAPLgapreductionproceduresetsf 0=lbs; f0=ubs;l=lbs+(1)]TJ /F6 11.955 Tf 11.95 0 Td[()ubs,anditerativelyperformsthefollowingsteps. a) Setxlk=(1)]TJ /F6 11.955 Tf 11.96 0 Td[(k)xuk)]TJ /F9 7.97 Tf 6.58 0 Td[(1+kxk)]TJ /F9 7.97 Tf 6.59 0 Td[(1andf k:=maxff k)]TJ /F9 7.97 Tf 6.58 0 Td[(1;minfl;h kgg,where h k:=minx2Xk)]TJ /F12 5.978 Tf 5.76 0 Td[(1h(xlk;x):(1{10) b) Updatetheprox-centerxkby xk=argminx2X kd(x);(1{11)whereX k:=fx2Xk)]TJ /F9 7.97 Tf 6.59 0 Td[(1:h(xlk;x)lg. c) Set fk=minf fk)]TJ /F9 7.97 Tf 6.58 0 Td[(1;f(~xk)g,where~xuk=kxk+(1)]TJ /F6 11.955 Tf 11.49 0 Td[(k)xuk)]TJ /F9 7.97 Tf 6.59 0 Td[(1,andxukischosenaseither~xukorxuk)]TJ /F9 7.97 Tf 6.59 0 Td[(1suchthatf(xuk)= fk. d) ChooseXksuchthatX kXk Xk,where Xk:=fx2X:hrd(xk);x)]TJ /F6 11.955 Tf 11.95 0 Td[(xki0g:Hererd()denotesthegradientofd().Observethattheparameterskandarexedapriori,anddonotdependonanyproblemparameters.Moreover,thelocalizerXkischosenbetweenX kand Xk,sothatthenumbersoflinearconstraintsinthetwosubproblems( 1{10 )and( 1{11 )canbefullycontrolled.Itisshownin[ 26 ]thatboththeABLandAPLmethods 15

PAGE 16

achievetheoptimaliterationcomplexityuniformlyforsmooth,weaklysmoothandnonsmoothconvexfunctionsforsolvingproblem( 3{1 ).Moreover,byincorporatingNesterov'ssmoothingtechnique[ 2 ]intotheAPLmethod,Lanalsopresentedin[ 26 ]thattheuniformsmoothinglevel(USL)methodwhichcanachievetheoptimalcomplexityforsolvinganimportantclassofnonsmoothstructuredSPproblemswithoutrequiringinputofanyproblemparameters.Becauseoftheuseofcuttingplanemodelandthetechniqueofrestrictedmemorytoecientlyutilizehistoricalinformation,theacceleratedBLmethodsoftenndasolutionwithmuchloweriterationcostandlessnumberofiterationsthanmanygradientdecenttypemethods.Moreover,theBLtypemethodsdonotinvolveanystepsize,sodonotrequiretheinputontheLipschitzconstantoftheobjectivefunction.Thesebuilt-infeaturesofBLtypemethodsareveryimportantforsolvinglarge-scaleCOPs.However,thedevelopmentofacceleratedBLmethodsforvariousofCOPsinvolvingsmoothorweaklysmoothfunctionsarestillinsucient.Inparticular,mostexistingBLtypemethodsrequirethefeasiblesettobeboundedandthereisnoBLtypemethodsdevelopedforunconstrainedoptimizationwithiterationcomplexityproved.Inmanypracticalproblems,theexactrst-orderinformationoftheobjectivefunctionmaybeunavailableorveryexpensivetocompute,avarietyofinexactBLmethods[ 18 { 21 23 { 25 27 { 32 ]havebeendevelopedtodealwithinexactoracle,butallthesemethodsarenon-acceleratedBLmethodsandlimitedtosolvenonsmoothCOPs.TheBLtypemethodsalsohavecertainadvantagesforsolvingfunctionalconstrainedoptimization(FCO),however,alltheexistingconstrainedBL(CBL)methods[ 15 16 33 { 35 ]canonlysolvenonsmoothFCOwhereboththeobjectiveandconstraintarenonsmooth.MotivatedbyinsucientdevelopmentandtheadvantagesandchallengesoftheacceleratedBLtypemethods,thisthesisaimstodevelopvariousacceleratedBLtypemethodstodealwithgeneralCOPs,COPswithinexactoracleandFCOthathavetheacceleratediterationcomplexitiesfortheseproblemsandfastconvergencespeedinpractice. 1.3OutlineoftheDissertationThisthesisisorganizedasfollows. 16

PAGE 17

InChapter 2 ,weproposetheFAPLandtheFUSLmethodstoimprovetheperformanceoftheexistingacceleratedBLmethodsforsolvingsmooth,weaklysmoothandnonsmoothball-constrainedCOPsandaclassofSPproblems,respectively.TheFAPLandFUSLmethodshavesignicantlyreducedthecomputationalcostperiterationandincreasedtheaccuracyofthesolution.Then,weintroduceageneralframeworktoextendtheoptimalmethodsforball-constrainedCOPstounconstrainedCOPs.TheproposedframeworksolvesunconstrainedCOPsthroughsolutionstoaseriesofball-constrainedCOPsandachievesthesameorderoftheiterationcomplexityasthecorrespondingball-constrainedCOPs.Theapplicationstolarge-scaleleastsquaresproblemsandtotalvariationbasedimagereconstructionproblemswithcomparisonstorelatedstate-of-the-artalgorithmsarealsopresented.InChapter 3 ,wediscusstheacceleratedBLtypemethodswithinexactoraclestodealwiththeCOPswheretheexactrst-orderinformationoftheobjectivefunctioniseithernotavailableortooexpensivetocompute.SeveralinexactacceleratedBLtypemethodsareproposedtosolvesmoothCOPsandaclassofSPproblems,withextensiontostronglyconvexfunctions.Dierenttonon-acceleratedalgorithms,[ 36 ]showsthattheacceleratedrst-ordermethodsmustnecessarilysuerfromerroraccumulationoftheinexactoracle.Therefore,westudythedependencebetweentheaccuracyoftheinexactoracleandthebestaccuracytheproposedalgorithmscouldachieve,andestablishthecorrespondingiterationcomplexityboundsforconvergingtoasolutionwithsuchaccuracyguaranteed.InChapter 4 ,weconsidertheFCOgivenasminx2Xf(x);s:t:g(x)0.Thisproblemisconsideredtobedicultiftheprojectionontothefeasiblesetfx2X;g(x)0gishardtocompute.TheproposedACBLmethodiscapabletouniformlysolveFCOswithsmooth,weaklysmoothandnonsmoothobjectiveandconstraint,andachievestheiterationcomplexityO)]TJ /F9 7.97 Tf 21.49 -4.98 Td[(1 2=(1+3)log1 ,where=minff;gg.Hence,iff=g=0,theACBLmethodachievesthesamebestiterationcomplexityastheCBLmethodsin[ 15 16 33 ]fornonsmoothFCO,andiff=g=1,theACBLmethodrstestablishestheiterationcomplexityO1 p log1 forsmoothFCO.Furthermore,inordertoimprovethepracticalperformance 17

PAGE 18

oftheACBLmethod,wedeveloptheACPLmethod.TheACPLmethodincorporatestherestricted-memorytechniquetocontrolthenumberoflinearapproximationsusedinthelevelset,andusesasimplestrategytoupdatethelowerestimateonf.Lastly,byemployingNesterov'ssmoothingtechniquefornonsmoothfunctions,weextendtheACPLmethodtosolveaclassofconstrainedSPproblemwheretheobjectiveisaSPproblemandtheconstraintissmooth. 18

PAGE 19

CHAPTER2FASTACCELERATEDBUNDLELEVELMETHODSFORCONVEXOPTIMIZATION 2.1BackgroundOnecrucialproblemassociatedwithmostexistingBLtypemethods,includingAPLandUSL,isthateachiterationofthesealgorithmsinvolvessolvingtwooptimizationproblems:rstalinearprogrammingproblemtocomputethelowerbound,andthenaconstrainedquadraticprogrammingproblemtoupdatetheprox-centerornewiterate.Infact,theeciencyofthesealgorithmsreliesonthesolutionstothesetwoinvolvedsubproblems,andthelatteroneisoftenmorecomplicatedthantheprojectionsubprobleminthegradientprojectiontypemethods.Moreover,mostexistingBLtypemethodsrequiretheassumptionthatthefeasiblesetisboundedduetothefollowingtworeasons.Firstly,thefeasiblesethastobeboundedtocomputeameaningfullowerboundbysolvingtheaforementionedlinearprogrammingproblem.Secondly,theiterationcomplexityanalysisofthosemethods,suchastheclassicalBLmethod[ 16 ],NERML[ 17 ],ABL,APLandUSL[ 26 ],reliesontheassumptionthatthefeasiblesetiscompact.ItshouldbenotedthatthereexistsomevariantsofBLmethods[ 18 24 37 38 ]forsolvingnonsmoothCOPsinwhichthecomputationofthesubproblemforupdatingthelowerboundisskipped,sothatthefeasiblesetXisallowedtobeunbounded.Forinstance,thelevelbundlemethodin[ 38 ]updatesthelevelparameterdirectlywhenthedistancefromthestabilitycentertothenewlygeneratediteratebecomeslargerthanachosenparameter.However,allthesemethodsarelimitedtosolvenonsmoothCOPsandthereisnoiterationcomplexityanalysisprovidedinanyoftheseworks.TheseissueshavesignicantlyhinderedtheapplicabilityofexistingBLtypemethods.ThemainpurposeofthischapteristoimprovethepracticalperformanceoftheexistingacceleratedBLtypemethodsforsolvingball-constrainedCOPs,andfurtherextendtheBLtypemethodstosolveunconstrainedCOPswithiterationcomplexityboundsestablished.InSection 2.2 ,weproposetheFAPLandFUSLmethodsthatgreatlyreducethecomputationalcostperiterationoftheAPLandUSLmethodsforsolvingball-constrainedCOPs.In 19

PAGE 20

particular,asimpleapproachisintroducedtoexactlysolvethedualproblemoftheonlysubprobleminouralgorithms.InSection 2.3 ,wepresentageneralframeworktoextendtheoptimalmethodsforball-constrainedCOPstounconstrainedCOPs.TheproposedframeworksolvesunconstrainedCOPsthroughsolutionstoaseriesofball-constrainedCOPsandachievesthesameorderofiterationcomplexityasthecorrespondingball-constrainedCOPs.InSection 2.4 ,wefurtherextendtheFAPLandFUSLmethodtosolvestronglyconvexoptimizationproblemwithbetteriterationcomplexitiesestablished.Inthelastsection,weapplytheFAPLandFUSLmethodstosolvelarge-scaleleastsquaresproblemsandtotalvariationbasedimagereconstructionproblems. 2.2FastAcceleratedBundleLevelMethodsforBall-ConstrainedConvexOptimizationInthissection,wediscussthefollowingball-constrainedCOP:f x;R:=minx2B( x;R)f(x); (2{1)whereB(x;R):=fx2Rn:kx)]TJ ET q 0.478 w 172.95 -330.92 m 179.6 -330.92 l S Q BT /F6 11.955 Tf 172.95 -337.74 Td[(xkRgdenotestheEuclideanballcenteredatxwithradiusR,andf:Rn!Risaconvexfunctionsatisfying f(y))]TJ /F6 11.955 Tf 11.95 0 Td[(f(x))-222(hf0(x);y)]TJ /F6 11.955 Tf 11.96 0 Td[(xiM 1+ky)]TJ /F6 11.955 Tf 11.96 0 Td[(xk1+;8x;y2B(x;R);(2{2)forsomeM>0and2[0;1].Thisf()canbenonsmooth(=0),smooth(=1)andweaklysmooth(0<<1).Thissectioncontainsthreesubsections.WerstpresentamuchsimpliedAPLmethod,referredtothefastAPL(FAPL)method,forsolvingball-constrainedblack-boxCOPsinSubsection 2.2.1 ,andthenpresentthefastUSL(FUSL)methodforsolvingaspecialclassofball-constrainedSPproblemsinSubsection 2.2.2 .WeshowhowtosolvethesubproblemsinthesetwoalgorithmsinSubsection 2.2.3 20

PAGE 21

2.2.1FAPLMethodforBall-ConstrainedConvexOptimizationOurgoalinthissubsectionistopresenttheFAPLmethod,whichcansignicantlyreducetheiterationcostfortheAPLmethodappliedtoproblem( 2{1 ).Inparticular,weshowthatonlyonesubproblem,ratherthantwosubproblemsasintheAPLmethod,isrequiredintheFAPLmethodfordeninganewiterate(orprox-center)andupdatinglowerbound.Wealsodemonstratethattheballconstraintin( 2{1 )canbeeliminatedfromthesubproblembyproperlyspecifyingtheprox-function.SimilarlytotheAPLmethod,theFAPLmethodconsistsofouter-innerloops,andineachouteriteration,aninnerloop,theFAPLgapreductionproceduredenotedbyGFAPL,iscalledtoreducethegapbetweentheupperandlowerboundsonf x;Rin( 2{1 )byaconstantfactor.WestartbydescribingtheFAPLgapreductionprocedureinProcedure 2.2.1 .ThisprocedurediersfromthegapreductionprocedureusedintheAPLmethodinthefollowingseveralaspects.Firstly,thelocalizersQ kand QkinprocedureGFAPL(seeSteps 1 and 4 )onlycontainlinearconstraintsandhencearepossiblyunbounded,whilethelocalizersintheAPLmethodmustbecompact.Secondly,weeliminatethesubproblemthatupdatesthelowerboundonf x;RintheAPLmethod.Instead,intheFAPLmethod,thelowerboundisupdatedtothelevelldirectlywheneverQ k=;orkxk)]TJ ET q 0.478 w 246.3 -388.19 m 252.96 -388.19 l S Q BT /F6 11.955 Tf 246.3 -395.01 Td[(xk>R.Thirdly,wechooseaspecicprox-functiond(x)=1 2kx)]TJ ET q 0.478 w 145.34 -412.1 m 151.99 -412.1 l S Q BT /F6 11.955 Tf 145.34 -418.92 Td[(xk2,andasaresult,allthethreesequencesfxkg;fxlkgandfxukgwillresideintheballB( x;R).Atlast,aswewillshowinnextsubsection,sincethesubproblem( 2{6 )onlycontainsalimitednumberoflinearconstraints(depthofmemory),wecansolveitveryeciently,orevenexactlyifthedepthofmemoryissmall(e.g.,lessthan10).WenowaddafewmoreremarksaboutthetechnicaldetailsofProcedure 2.2.1 .Firstly,Procedure 2.2.1 isterminatedatStep 2 ifQk=;orkxk)]TJ ET q 0.478 w 297.27 -531.64 m 303.92 -531.64 l S Q BT /F6 11.955 Tf 297.27 -538.46 Td[(xk>R,whichcanbecheckedautomaticallywhensolvingthesubproblem( 2{6 )(seeSubsection 2.2.3 formoredetails).Secondly,inStep 4 ,whileQkcanbeanypolyhedralsetbetweenQ kand Qk,inpracticewecansimplychooseQktobetheintersectionofthehalf-spacefx2Rn:hxk)]TJ ET q 0.478 w 386.98 -603.36 m 393.63 -603.36 l S Q BT /F6 11.955 Tf 386.98 -610.18 Td[(x;x)]TJ /F6 11.955 Tf 11.95 0 Td[(xki0gandafewmostrecentlygeneratedhalf-spaces,eachofwhichisdenedbyfx2Rn: 21

PAGE 22

TheFAPLgapreductionprocedure:(x+;lb+)=GFAPL(^x;lb;R; x;;). 0: Setk=1, f0=f(^x),l=lb+(1)]TJ /F6 11.955 Tf 11.95 0 Td[() f0,Q0=Rn,andletxu0=^x,x0= x. 1: Updatethecuttingplanemodel:xlk=(1)]TJ /F6 11.955 Tf 11.95 0 Td[(k)xuk)]TJ /F9 7.97 Tf 6.59 0 Td[(1+kxk)]TJ /F9 7.97 Tf 6.59 0 Td[(1; (2{3)h(xlk;x)=f(xlk)+f0(xlk);x)]TJ /F6 11.955 Tf 11.96 0 Td[(xlk; (2{4)Q k=fx2Qk)]TJ /F9 7.97 Tf 6.59 0 Td[(1:h(xlk;x)lg: (2{5) 2: Updatetheprox-centerandlowerbound: xk=argminx2Q kd(x):=1 2kx)]TJ ET q 0.478 w 314.14 -157.37 m 320.8 -157.37 l S Q BT /F6 11.955 Tf 314.14 -164.19 Td[(xk2:(2{6)IfQ k=;orkxk)]TJ ET q 0.478 w 116.35 -191.64 m 123 -191.64 l S Q BT /F6 11.955 Tf 116.35 -198.46 Td[(xk>R,thenterminatewithoutputsx+=xuk)]TJ /F9 7.97 Tf 6.59 0 Td[(1;lb+=l. 3: Updatetheupperbound:set~xuk=(1)]TJ /F6 11.955 Tf 11.95 0 Td[(k)xuk)]TJ /F9 7.97 Tf 6.59 0 Td[(1+kxk; (2{7)xuk=(~xuk;iff(~xuk)< fk)]TJ /F9 7.97 Tf 6.58 0 Td[(1;xuk)]TJ /F9 7.97 Tf 6.59 0 Td[(1;otherwise; (2{8)and fk=f(xuk).If fkl+( f0)]TJ /F6 11.955 Tf 11.96 0 Td[(l),thenterminatewithx+=xuk;lb+=lb. 4: ChooseanypolyhedralsetQksatisfyingQ kQk Qk,where Qk:=fx2Rn:hxk)]TJ ET q 0.478 w 261.19 -343.49 m 267.84 -343.49 l S Q BT /F6 11.955 Tf 261.19 -350.31 Td[(x;x)]TJ /F6 11.955 Tf 11.96 0 Td[(xki0g:(2{9)Setk=k+1andgotoStep 1 h(xl;x)lgforsome1k.Finally,inordertoguaranteetheterminationofprocedureGFAPLandtheoptimaliterationcomplexity,theparametersfkgusedinthisprocedureneedtobeproperlychosen.OnesetofconditionsthatfkgshouldsatisfytoguaranteetheconvergenceofprocedureGFAPLisgivenasfollows: 1=1;00and82[0;1].Thefollowinglemmaprovidestwoexamplesfortheselectionoffkg. Lemma1. a) Ifk=2=(k+1),k=1;2;:::,thenthecondition( 2{10 )issatisedwithc=2. 22

PAGE 23

b) Iffkgisrecursivelydenedby 1=1;2k+1=(1)]TJ /F6 11.955 Tf 11.96 0 Td[(k+1)2k;8k1;(2{11)thenthecondition( 2{10 )holdswithc=2. Proof. Part a followsimmediatelybypluggingk=2=(k+1),k=1;2;:::,into( 2{10 ).Forpart b ,itiseasytoprovebyinductionthatk2(0;1];k+1
PAGE 24

convexandclosed.Therefore,thesubproblem( 2{6 )alwayshasauniquesolutionaslongasQ kisnon-empty.TonishtheproofitsucestoshowthatEf(l)=;wheneitherQ k=;orkxk)]TJ ET q 0.478 w 426.61 -52.95 m 433.26 -52.95 l S Q BT /F6 11.955 Tf 426.61 -59.77 Td[(xk>R,whichcanbeprovedbycontradiction.Firstly,ifQ k=;butEf(l)6=;,thenbypart a provedabove,wehaveEf(l)Q k,whichcontradictstheassumptionthatQ kisempty.Ontheotherhand,supposethatkxk)]TJ ET q 0.478 w 129.47 -124.67 m 136.12 -124.67 l S Q BT /F6 11.955 Tf 129.47 -131.49 Td[(xk>RandEf(l)6=;,letxR2Argminx2B( x;R)f(x),itisclearthatxR2Ef(l)Q kby a ,howeverkxR)]TJ ET q 0.478 w 192.98 -148.58 m 199.63 -148.58 l S Q BT /F6 11.955 Tf 192.98 -155.4 Td[(xkR
PAGE 25

Proposition1. Ifthestepsizesfkgk1arechosensuchthat( 2{10 )holds,thenthenumberofiterationsperformedbyprocedureGFAPLdoesnotexceed N():=c1+MR1+ (1+)2 1+3+1;(2{13)where:=ub)]TJ /F3 11.955 Tf 11.96 0 Td[(lb. Proof. SupposethatprocedureGFAPLdoesnotterminateattheKthiterationforsomeK>0.Thenobservingthatxk2Q kQk)]TJ /F9 7.97 Tf 6.58 0 Td[(1 Qk)]TJ /F9 7.97 Tf 6.59 0 Td[(1forany2kKdueto( 2{5 )and( 2{6 ),andx12Q0;x0=argminx2Q0d(x),wehavehrd(xk)]TJ /F9 7.97 Tf 6.59 0 Td[(1);xk)]TJ /F6 11.955 Tf 11.95 0 Td[(xk)]TJ /F9 7.97 Tf 6.59 0 Td[(1i0;81kK.Sinced(x)isstronglyconvexwithmodulus1,wealsohaved(xk)d(xk)]TJ /F9 7.97 Tf 6.59 0 Td[(1)+hrd(xk)]TJ /F9 7.97 Tf 6.59 0 Td[(1);xk)]TJ /F6 11.955 Tf 11.95 0 Td[(xk)]TJ /F9 7.97 Tf 6.58 0 Td[(1i+1 2kxk)]TJ /F6 11.955 Tf 11.96 0 Td[(xk)]TJ /F9 7.97 Tf 6.59 0 Td[(1k2:Combiningthetworelationsabove,itimplies1 2kxk)]TJ /F6 11.955 Tf 12.06 0 Td[(xk)]TJ /F9 7.97 Tf 6.58 0 Td[(1k2d(xk))]TJ /F6 11.955 Tf 12.06 0 Td[(d(xk)]TJ /F9 7.97 Tf 6.59 0 Td[(1).Summinguptheseinequalitiesfor1kK,weconcludethat 1 2KXk=1kxk)]TJ /F6 11.955 Tf 11.95 0 Td[(xk)]TJ /F9 7.97 Tf 6.58 0 Td[(1k2d(xK)=1 2kxK)]TJ ET q 0.478 w 304.64 -334.87 m 311.3 -334.87 l S Q BT /F6 11.955 Tf 304.64 -341.69 Td[(xk21 2R2:(2{14)By( 2{2 ),( 4{18 ),( 4{19 )andtheconvexityoff,wehaveforany1kK,f(xuk)f(~xuk)h(xlk;~xuk)+M 1+k~xuk)]TJ /F6 11.955 Tf 11.95 0 Td[(xlkk1+ (2{15)=(1)]TJ /F6 11.955 Tf 11.96 0 Td[(k)h(xlk;xuk)]TJ /F9 7.97 Tf 6.58 0 Td[(1)+kh(xlk;xk)+M 1+k~xuk)]TJ /F6 11.955 Tf 11.95 0 Td[(xlkk1+: (2{16)Sinceh(xlk;xuk)]TJ /F9 7.97 Tf 6.59 0 Td[(1)f(xuk)]TJ /F9 7.97 Tf 6.59 0 Td[(1),h(xlk;xk)ldueto( 2{5 )and( 2{6 ),and~xuk)]TJ /F6 11.955 Tf 13.14 0 Td[(xlk=k(xk)]TJ /F6 11.955 Tf 11.95 0 Td[(xk)]TJ /F9 7.97 Tf 6.58 0 Td[(1)dueto( 4{11 )and( 4{18 ),wehave f(xuk))]TJ /F6 11.955 Tf 11.95 0 Td[(l(1)]TJ /F6 11.955 Tf 11.96 0 Td[(k)(f(xuk)]TJ /F9 7.97 Tf 6.59 0 Td[(1))]TJ /F6 11.955 Tf 11.95 0 Td[(l)+1+kM 1+kxk)]TJ /F6 11.955 Tf 11.96 0 Td[(xk)]TJ /F9 7.97 Tf 6.59 0 Td[(1k1+(2{17) 25

PAGE 26

Dividingbothsidesof( 2{17 )by1+k,andthensumminguptheseinequalitiesfor1kK,by( 2{10 )andthefactf(xuk))]TJ /F6 11.955 Tf 11.96 0 Td[(l0,81kK,wehave f(xuK))]TJ /F6 11.955 Tf 11.96 0 Td[(l1+KM 1+KXk=1kxk)]TJ /F6 11.955 Tf 11.95 0 Td[(xk)]TJ /F9 7.97 Tf 6.58 0 Td[(1k1+(2{18)ApplyHolder'sinequality,anduse( 4{76 ),wehaveKXk=1kxk)]TJ /F6 11.955 Tf 11.96 0 Td[(xk)]TJ /F9 7.97 Tf 6.59 0 Td[(1k1+K1)]TJ /F10 5.978 Tf 5.76 0 Td[( 2 KXk=1kxk)]TJ /F6 11.955 Tf 11.95 0 Td[(xk)]TJ /F9 7.97 Tf 6.59 0 Td[(1k2!1+ 2K1)]TJ /F10 5.978 Tf 5.75 0 Td[( 2R1+;anda1+Kc1+K)]TJ /F9 7.97 Tf 6.58 0 Td[((1+)dueto( 2{10 ).Therefore( 4{44 )gives f(xuK))]TJ /F6 11.955 Tf 11.96 0 Td[(lMR1+ (1+)c1+ K1+3 2(2{19)InviewofStep 3 inprocedure 2.2.1 ,andusingthefactthatl=lb+(1)]TJ /F6 11.955 Tf 11.49 0 Td[()ubinStep 0 ,wealsohavef(xuK))]TJ /F6 11.955 Tf 11.96 0 Td[(l>(ub)]TJ /F6 11.955 Tf 11.95 0 Td[(l)=:Combiningtheabovetworelations,andusing( 2{10 )and( 4{76 ),weobtain 0andparameters;2(0;1). 26

PAGE 27

1: Setp12Argminx2B( x;R)h(p0;x),lb1=h(p0;p1);ub1=minff(p0);f(p1)g,let^x1beeitherp0orp1suchthatf(^x1)=ub1,ands=1. 2: Ifubs)]TJ /F3 11.955 Tf 11.95 0 Td[(lbs,terminateandoutputapproximatesolution^xs. 3: Set(^xs+1;lbs+1)=GFAPL(^xs;lbs;R; x;;)andubs+1=f(^xs+1). 4: Sets=s+1andgotoStep 2 .Forthesakeofsimplicity,eachiterationofProcedureGFAPLisalsoreferredtoasaniterationoftheFAPLmethod.ThefollowingtheoremestablishesthecomplexityboundsonthenumbersofgapreductionproceduresGFAPLandtotaliterationsperformedbytheFAPLmethod,itsproofissimilartothatofTheorem4in[ 26 ]. Theorem1. Foranygiven>0,ifthestepsizesfkginprocedureGFAPLarechosensuchthat( 2{10 )holds,thenthefollowingstatementsholdfortheFAPLmethodtocomputean-solutiontoproblem( 2{1 ). a) ThenumberofgapreductionproceduresGFAPLperformedbytheFAPLmethoddoesnotexceed S:=max0;log1 q(2R)1+M (1+):(2{22) b) ThetotalnumberofiterationsperformedbytheFAPLmethodcanbeboundedby N():=S+1 1)]TJ /F6 11.955 Tf 11.95 0 Td[(q2 1+3c1+MR1+ (1+)2 1+3;(2{23)whereqisdenedin( 3{32 ). Proof. Werstproveparta).Lets:=ubs)]TJ /F3 11.955 Tf 12.1 0 Td[(lbs,withoutlossofgenerality,weassumethat1>.InviewofStep 0 inAlgorithm 1 and( 2{2 ),wehave 1f(p1))]TJ /F6 11.955 Tf 11.96 0 Td[(h(p0;p1)=f(p1))]TJ /F6 11.955 Tf 11.96 0 Td[(f(p0))-222(hf0(p0);p1)]TJ /F6 11.955 Tf 11.95 0 Td[(p0i(2R)1+M 1+:(2{24)Also,byLemma 3 wecanseethats+1qsforanys1,whichimpliesthats+1qs1;8s0: 27

PAGE 28

Moreover,ifan-solutionisfoundafterperformingSgapreductionproceduresGFAPL,thenwehave S>S+1:(2{25)Combiningtheabovethreeinequalities,weconcludethat
PAGE 29

where^fisasmoothconvexfunction,i.e.,9L^f>0s.t.^f(y))]TJ /F3 11.955 Tf 14.5 3.16 Td[(^f(x))-222(hr^f(x);y)]TJ /F6 11.955 Tf 11.95 0 Td[(xiL^f 2ky)]TJ /F6 11.955 Tf 11.95 0 Td[(xk2; (2{29)and F(x):=maxy2YfhAx;yi)]TJ /F3 11.955 Tf 19.68 0 Td[(^g(y)g:(2{30)Here,YRmisacompactconvexset,^g:=Y!Risarelativelysimpleconvexfunction,andA:Rn!Rmisalinearoperator.WewillintegrateNesterov'ssmoothingtechniqueforminimizingnonsmoothfunctions[ 2 ]intotheFAPLmethodtosolveproblem( 2{1 )-( 2{28 )(i.e.,problem( 2{1 )withfdenedin( 2{28 )),whichcansignicantlyreducetheiterationcostoftheUSLmethod[ 26 ].Letv:Y!Rbeaprox-functionwithmodulusvanddenotingcv:=argminv2Yv(y),byemployingtheideaintheimportantworkofNesterov[ 2 ],wecanapproximateFin( 2{30 )bythesmoothfunction F(x):=maxy2YfhAx;yi)]TJ /F3 11.955 Tf 19.68 0 Td[(^g(y))]TJ /F6 11.955 Tf 11.95 0 Td[(V(y)g;(2{31)where>0iscalledthesmoothingparameter,andV()istheBregmandivergencedenedby V(y):=v(y))]TJ /F6 11.955 Tf 11.95 0 Td[(v(cv))-221(hrv(cv);y)]TJ /F6 11.955 Tf 11.95 0 Td[(cvi:(2{32)Itwasshownin[ 2 ]thatthegradientofF()givenbyrF(x)=Ay(x)isLipschitzcontinuouswithconstant L:=kAk2=(v);(2{33)wherekAkistheoperatornormofA,Aistheadjointoperator,andy(x)2Yisthesolutiontotheoptimizationproblem( 2{31 ).Moreover,the\closeness"ofF()toF()dependslinearlyonthesmoothingparameter,i.e., F(x)F(x)F(x)+Dv;Y;8x2X;(2{34) 29

PAGE 30

where Dv;Y:=maxy;z2Yfv(y))]TJ /F6 11.955 Tf 11.96 0 Td[(v(z))-222(hrv(z);y)]TJ /F6 11.955 Tf 11.95 0 Td[(zig:(2{35)Therefore,ifwedenote f(x):=^f(x)+F(x);(2{36)then f(x)f(x)f(x)+Dv;Y:(2{37)Applyinganoptimalgradientmethodtominimizethesmoothfunctionfin( 2{36 ),Nesterovprovesin[ 2 ]thattheiterationcomplexityforcomputingan-solutiontoproblem( 2{1 )-( 2{28 )isboundedbyO(1 p +1 ).However,thevaluesofquiteafewproblemparameters,suchaskAk;vandDv;Y,arerequiredfortheimplementationofNesterov'ssmoothingscheme.ByincorporatingNesterov'ssmoothingtechnique[ 2 ]intotheAPLmethod,Landevelopedin[ 26 ]anewbundle-leveltypemethod,namelytheUSLmethod,tosolvestructuredproblemsgivenintheformof( 2{1 )-( 2{28 ).WhiletheUSLmethodachievesthesameoptimaliterationcomplexityasNesterov'ssmoothingschemein[ 2 ],oneadvantageoftheUSLmethodoverNesterov'ssmoothingschemeisthatthesmoothingparameterisadjusteddynamicallyduringtheexecution,andanestimateofDv;Yisobtainedautomatically,whichmakestheUSLmethodproblemparameterfree.However,similartotheAPLmethod,eachiterationoftheUSLmethodinvolvesthesolutionstotwosubproblems.BasedontheUSLmethodin[ 26 ]andouranalysisoftheFAPLmethodinSection 2.2.1 ,weproposeafastUSL(FUSL)methodthatsolvesproblem( 2{1 )-( 2{28 )withthesameoptimaliterationcomplexityastheUSLmethod,butrequiringonlytosolveonesimplersubproblemineachiteration.SimilartotheFAPLmethod,theFUSLmethodhasanouter-innerloopsstructure,andeachouteriterationcallsaninnerloop,agapreductionproceduredenotedbyGFUSL,toreducethegapbetweentheupperandlowerboundsonf x;Rin( 2{1 )-( 2{28 )byaconstantfactorunlessan-solutionisfound.WestartbydescribingprocedureGFUSL. 30

PAGE 31

TheFUSLgapreductionprocedure:(x+;D+;lb+)=GFUSL(^x;D;lb;R; x;;). 0: Letk=1, f0=f(^x);l=lb+(1)]TJ /F6 11.955 Tf 11.95 0 Td[() f0,Q0=Rn,xu0=^x,x0= x,and :=( f0)]TJ /F6 11.955 Tf 11.95 0 Td[(l)=(2D):(2{38) 1: Updatethecuttingplanemodel:Setxlkto( 4{11 ),Q kto( 2{5 ),andh(xlk;x)=h(xlk;x)=f(xlk)+f0(xlk);x)]TJ /F6 11.955 Tf 11.95 0 Td[(xlk: (2{39) 2: Updatetheprox-center:Setxkto( 2{6 ).IfQ k=;orkxk)]TJ ET q 0.478 w 319.79 -127.77 m 326.44 -127.77 l S Q BT /F6 11.955 Tf 319.79 -134.59 Td[(xk>R,thenterminatewithoutputsx+=xuk)]TJ /F9 7.97 Tf 6.59 0 Td[(1;D+=D;lb+=l. 3: UpdatetheupperboundandtheestimateofDv;Y:Set~xukto( 4{18 ),xukto( 4{19 ),and fk=f(xuk).Checkthefollowingconditions: 3a) iff(xuk)l+( f0)]TJ /F6 11.955 Tf 11.95 0 Td[(l),thenterminatewithoutputsx+=xuk;D+=D;lb+=lb; 3b) iff(xuk)>l+( f0)]TJ /F6 11.955 Tf 12.84 0 Td[(l)andf(xuk)l+ 2( f0)]TJ /F6 11.955 Tf 12.85 0 Td[(l),thenterminatewithoutputsx+=xuk;D+=2D;lb+=lb. 4: ChooseQkassameasStep 4 inGFAPL.Setk=k+1,andgotostep1. AfewremarksaboutprocedureGFUSLareinplace.Firstly,sincethenonsmoothobjectivefunctionfisreplacedbyitssmoothedapproximationf,wereplacethecuttingplanemodelin( 1{8 )withtheoneforf(see( 3{78 )).AlsonotethatfortheUSLmethodin[ 26 ],^fisassumedtobeasimpleLipschitzcontinuousconvexfunction,andonlyFisapproximatedbythelinearestimation.However,intheFUSLmethod,weassume^fisgeneralsmoothandconvex,andlinearizeboth^fandFin( 3{78 ).Secondly,thesmoothingparameterisspeciedasafunctionoftheparameterD,f0andl,whereDisanestimatorofDv;Ydenedby( 4{96 )andgivenasaninputparameterforprocedureGFUSL.Thirdly,sameastheFAPLmethod,theparametersfkgarechosenaccordingto( 2{10 ).SuchconditionsarerequiredtoguaranteetheoptimaliterationcomplexityoftheFUSLmethodforsolvingproblem( 2{1 )-( 2{28 ).Fourthly,similartotheFAPLmethod,thelocalizersQ k;Qk; Qkonlycontainalimitednumberoflinearconstraints,andthereisonlyonesubproblem(i.e.,( 2{6 ))involvedinprocedureGFUSL,whichcanbesolvedexactlywhenthedepthofmemoryissmall.ThefollowinglemmaprovidessomeimportantobservationsaboutprocedureGFUSL,whicharesimilartothosefortheUSLgapreductionprocedurein[ 26 ]. 31

PAGE 32

Lemma4. ThefollowingstatementsholdforprocedureGFUSL. a) IfprocedureGFUSLterminatesatSteps 2 or 3 a ,thenwehaveub+)]TJ /F3 11.955 Tf 12.35 0 Td[(lb+q(ub)]TJ /F3 11.955 Tf 12.35 0 Td[(lb),whereqisdenedby( 3{32 )andub:=f(^x);ub+:=f(x+). b) IfprocedureGFUSLterminatesatStep 3 b ,thenD 2( f0)]TJ /F6 11.955 Tf 11.95 0 Td[(l):Weconcludefromtheaboverelation,( 3{71 ),and( 2{38 )thatDv;Yf(xuk))]TJ /F6 11.955 Tf 11.95 0 Td[(f(xuk) >( f0)]TJ /F6 11.955 Tf 11.96 0 Td[(l) 2=D:Finally,D+<2Dv;YfollowsimmediatelyfromtheaboverelationandthedenitionofD+inStep 3 b ThefollowingresultprovidesaboundonthenumberofiterationsperformedbyprocedureGFUSL. Proposition2. Supposethatfkgk1inprocedureGFUSLarechosensuchthat( 2{10 )holds.Then,thenumberofiterationsperformedbythisproceduredoesnotexceed N(;D):=cR0@s L^f +p 2kAk r D v1A+1;(2{40)where:=f(^x))]TJ /F3 11.955 Tf 11.95 0 Td[(lb. Proof. Itiseasytoseethatthegradientoffin( 2{28 )hasLipschitzcontinuousgradientwithconstantL=L^f+L,whereLandL^faredenedin( 4{94 )and( 2{29 ),respectively.SupposethatprocedureGFUSLdoesnotterminateatiterationK.AstheGFUSLprocedurecouldbeviewedasapplyingtheGFAPLproceduretof,similartothediscussionon( 2{19 ), 32

PAGE 33

andnoticethat=1asfissmooth,wehave f(xuK))]TJ /F6 11.955 Tf 11.95 0 Td[(lc2Ld(xK) K2c2LR2 2K2;(2{41)wherecisdenedin( 2{10 ).Also,sinceprocedureGFUSLdoesnotterminateatiterationK,inviewoftheterminationconditioninStep 3 b andthedenitionoflinStep 0 ,wehave f(xuK))]TJ /F6 11.955 Tf 11.96 0 Td[(l> 2:(2{42)By( 2{41 ),( 2{42 ),( 4{94 )and( 2{38 ),weconcludethat K0,andparameters;2(0;1). 1: Setp12Argminx2B( x;R)h(p0;x),lb1=h(p0;p1);ub1=minff(p0);f(p1)g,let^x1beeitherp0orp1suchthatf(^x1)=ub1,ands=1. 2: Ifubs)]TJ /F3 11.955 Tf 11.95 0 Td[(lbs,terminateandoutputapproximatesolution^x. 3: Set(^xs+1;Ds+1;lbs+1)=GFUSL(^xs;Ds;lbs;R; x;;)andubs+1=f(^x). 4: Sets=s+1andgotoStep 2 .Forsimplicity,wesaythataphase(i.e.,anouteriteration)oftheFUSLmethodoccurswhensincreasesby1.Morespecically,similartotheUSLmethod,weclassifytwotypesofphasesintheFUSLmethod.AphaseiscalledsignicantifthecorrespondingGFUSLprocedureterminatesatSteps 2 or 3 a ,otherwiseitiscallednon-signicant.Clearly,ifthe 33

PAGE 34

valueofDv;yisprovided,whichistheassumptionmadeinNesterov'ssmoothingscheme[ 2 ],thenwecansetD1=Dv;YintheschemesofboththeUSLandFUSLmethods,andconsequently,allthephasesofbothmethodsbecomesignicant.AsthesameastheFAPLmethod,aniterationofprocedureGFUSLisalsoreferredtoaniterationoftheFUSLmethod.ThefollowingresultestablishesaboundonthetotalnumberofiterationsperformedbytheFUSLmethodtondan-solutiontoproblem( 2{1 )-( 2{28 ).NotethattheproofofthisresultissimilartothatofTheorem7in[ 26 ]. Theorem2. SupposethatfkginprocedureGFUSLarechosensuchthat( 2{10 )holds.Then,thetotalnumberofiterationsperformedbytheFUSLmethodforcomputingan-solutiontoproblem( 2{1 )-( 2{28 )isboundedby N():=S1+S2+(2 p 2)]TJ /F3 11.955 Tf 11.96 0 Td[(1+p 2 1)]TJ /F6 11.955 Tf 11.95 0 Td[(q)cRkAk s ~D v+(S1+1 1)]TJ 11.96 7.44 Td[(p q)cRs L^f ;(2{44)whereqandDv;Yaredenedby( 3{32 )and( 4{96 )respectively,and ~D:=maxfD1;2Dv;Yg;S1:=maxlog2Dv;Y D1;0andS2:=2666log1 q4p 2RkAkq Dv;Y v+2R2L^f 3777:(2{45) Proof. Weprovethisresultbyestimatingthenumbersofiterationsperformedwithinnon-signicantandsignicantphasesseparately.Supposethatthesetsofindicesofthenon-signicantandsignicantphasesarefm1;m2;:::;ms1gandfn1;n2;:::;ns2grespectively.Foranynon-signicantphasemk,1ks1,wecaneasilyseefromStep 3 b thatDmk+1=2Dmk,bypartb)inLemma 4 ,thenumberofnon-signicantphasesperformedbytheFUSLmethodisboundedbyS1denedabove,i.e.,s1S1.Inaddition,sinceDms1~D,wehaveDmk(1=2)s1)]TJ /F4 7.97 Tf 6.59 0 Td[(k~D,where~Disdenedabove.Combiningtheaboveestimatesons1andDmk,andinviewofthefactmk>forall1ks1,wecanboundthenumberofiterationsperformedwithinthenon-signicant 34

PAGE 35

phasesby N1=s1Xk=1 N(mk;Dmk)s1Xk=1 N;~D=2s1)]TJ /F4 7.97 Tf 6.58 0 Td[(kS10@cRs L^f +11A+p 2cRkAk s ~D vS1Xk=12)]TJ /F12 5.978 Tf 7.78 4.32 Td[((S1)]TJ /F10 5.978 Tf 5.75 0 Td[(k) 2S10@cRs L^f +11A+2cRkAk (p 2)]TJ /F3 11.955 Tf 11.96 0 Td[(1)s ~D v: (2{46)ApplyingLemma8in[ 26 ]andrelation( 2{29 ),andinviewofthefactthatp0;p12B( x;R)inAlgorithm 2 ,theinitialgapisboundedby1:=ub1)]TJ /F3 11.955 Tf 11.96 0 Td[(lb1[F(p0))]TJ /F6 11.955 Tf 11.95 0 Td[(F(p1))-222(hF0(p1);p0)]TJ /F6 11.955 Tf 11.95 0 Td[(p1i]+h^f(p0))]TJ /F3 11.955 Tf 14.5 3.15 Td[(^f(p1))]TJ /F13 11.955 Tf 11.96 13.27 Td[(D^f0(p1);p0)]TJ /F6 11.955 Tf 11.96 0 Td[(p1Ei (2{47)4p 2RkAkr Dv;Y v+2R2L^f; (2{48)whereF0(p1)2@F(p1).Thenforthesignicantphases,similartotheproofofTheorem 1 ,wehaves2S2.Moreover,foranynk,1ks2,usingLemmas 3 4 ,wehaveDnk~D,nk+1qnk,andns2>,whichimpliesnk>=qs2)]TJ /F4 7.97 Tf 6.58 0 Td[(k.CombinetheseestimatesonDnk;nkandboundons2,wecanseethatthetotalnumberofiterationsperformedwithinthesignicantphasesisboundedby N2=s2Xk=1 N(nk;Dnk)s2Xk=1 N(=qs2)]TJ /F4 7.97 Tf 6.59 0 Td[(k;~D)S2+cRs L^f S2Xk=1qS2)]TJ /F10 5.978 Tf 5.76 0 Td[(k 2+p 2cRkAk s ~D vS2Xk=1qS2)]TJ /F4 7.97 Tf 6.59 0 Td[(kS2+cR 1)]TJ 11.96 7.45 Td[(p qs L^f +p 2cRkAk (1)]TJ /F6 11.955 Tf 11.96 0 Td[(q)s ~D v: (2{49)Finally,thetotalnumberofiterationsperformedbytheFUSLmethodisboundedby N1+ N2,andthus( 2{44 )holds. 35

PAGE 36

From( 2{44 )intheabovetheorem,wecanseethattheiterationcomplexityoftheFUSLmethodforsolvingproblem( 2{1 )-( 2{28 )isboundedbyO r L^f +kAk !: (2{50)TheaboveiterationcomplexityisthesameasthatoftheNesterovsmoothingschemein[ 2 ]andtheUSLmethodin[ 26 ].However,boththeUSLandFUSLmethodsimproveNesterov'ssmoothingschemeinthatbothofthemareproblemparameterfree.Inaddition,asdetailedinSubsection 2.2.3 below,theFUSLmethodfurtherimprovestheUSLmethodbysignicantlyreducingitsiterationcostandimprovingtheaccuracyforsolvingitssubproblems. 2.2.3SolvingtheSubproblemofFAPLandFUSLMethodsInthissubsection,weintroduceanecientmethodtosolvethesubproblems( 2{6 )intheFAPLandFUSLmethods,whicharegivenintheformof xc:=argminx2Q1 2kx)]TJ /F6 11.955 Tf 11.96 0 Td[(pk2:(2{51)Here,Qisaclosedpolyhedralsetdescribedbymlinearinequalities,i.e.,Q:=fx2Rn:hAi;xibi;i=1;2;:::;mg;whereAi2Rnandbi2Rfor1im.NowletusexaminetheLagrangedualof( 2{51 )givenby max0minx2Rn1 2kx)]TJ /F6 11.955 Tf 11.96 0 Td[(pk2+mXi=1i[hAi;xi)]TJ /F6 11.955 Tf 19.27 0 Td[(bi]:(2{52)Itcanbecheckedfromthetheoremofalternativesthatproblem( 2{52 )issolvableifandonlyifQ6=;.Indeed,ifQ6=;,itisobviousthattheoptimalvalueof( 2{52 )isnite.Ontheotherhand,ifQ=;,thenthereexists0suchthatTA=0andTb<0,whichimpliesthattheoptimalvalueof( 2{52 )goestoinnity.Moreover,if( 2{52 )issolvableandisone 36

PAGE 37

ofitsoptimaldualsolutions,then xc=p)]TJ /F4 7.97 Tf 16.85 14.95 Td[(mXi=1iAi:(2{53)Itcanalsobeeasilyseenthat( 2{52 )isequivalentto max0)]TJ /F3 11.955 Tf 10.5 8.09 Td[(1 2TM+CT;(2{54)whereMij:=hAi;Aji;Ci:=hAi;pi)]TJ /F6 11.955 Tf 19.71 0 Td[(bi;8i;j=1;2;:::;m:Hence,wecandeterminethefeasibilityof( 2{51 )orcomputeitsoptimalsolutionbysolvingtherelativelysimpleproblem( 2{54 ).Manyalgorithmsarecapableofsolvingtheabovenonnegativequadraticprogrammingin( 2{54 )eciently.Duetoitslowdimension(usuallylessthan10inourpractice),weproposeabrute-forcemethodtocomputetheexactsolutionofthisproblem.ConsidertheLagrangedualassociatedwith( 2{54 ):min0max0L(;):=1 2TM)]TJ /F3 11.955 Tf 11.96 0 Td[((CT+);wherethedualvariableis:=(1;2;:::;m).ApplyingtheKKTcondition,wecanseethat0isasolutiontoproblem( 2{54 )ifandonlyifthereexists0suchthatrL(;)=0andh;i=0: (2{55)Notethattherstidentityin( 2{55 )isequivalenttoalinearsystem:M)]TJ /F6 11.955 Tf 9.3 0 Td[(I0BBBBBBBBBBBBBB@1...m1...m1CCCCCCCCCCCCCCA=0BBBBBBB@C1C2...Cm1CCCCCCCA; (2{56) 37

PAGE 38

whereIisthemmidentitymatrix.Theabovelinearsystemhas2mvariablesandmequations.Butforanyi=1;:::;m,wehaveeitheri=0ori=0,andhenceweonlyneedtoconsider2mpossiblecasesonthenon-negativityofthesevariables.Sincemisrathersmallinpractice,itispossibletoexhaustallthese2mcasestondtheexactsolutionto( 2{55 ).Foreachcase,werstremovethemcolumnsinthematrix(M)]TJ /F6 11.955 Tf 12.58 0 Td[(I)whichcorrespondtothemvariablesassumedtobe0,andthensolvetheremainingdeterminedlinearsystem.Ifallvariablesofthecomputedsolutionarenon-negative,thensolution(;)to( 2{55 )isfound,andtheexactsolutionxcto( 2{51 )iscomputedby( 2{53 ),otherwise,wecontinuetoexaminethenextcase.Itisinterestingtoobservethatthesedierentcasescanalsobeconsideredinparalleltotaketheadvantagesofhighperformancecomputingtechniques. 2.3FastAcceleratedBundleLevelMethodsforUnconstrainedConvexOptimizationInthissectionwediscussthefollowingunconstrainedCOP: f:=minx2Rnf(x);(2{57)wheref:Rn!Risconvex,andforanyclosedsets2Rn,thereexistsM()>0and()2[0;1],suchthat f(y))]TJ /F6 11.955 Tf 11.96 0 Td[(f(x))-222(hf0(x);y)]TJ /F6 11.955 Tf 11.95 0 Td[(xiM() 1+()ky)]TJ /F6 11.955 Tf 11.95 0 Td[(xk1+();8x;y2:(2{58)Theaboveassumptiononf()coversunconstrainednonsmooth(()=0),smooth(()=1)andweaklysmooth(0<()<1)COPs.WerstpresentagenericalgorithmicframeworktosolveunconstrainedCOPthroughsolutionstoaseriesofball-constrainedCOPsinSubsection 2.3.1 ,thenextendtheFAPLandFUSLmethodstounconstrainedCOPswithiterationcomplexityanalysisinSubsection 2.3.2 2.3.1ExpansionAlgorithmforUnconstrainedConvexOptimizationInordertoanalyzethecomputationalcomplexityofBLmethods,itiscommonlyassumedthatthefeasiblesetofinterestiscompact(e.g.,[ 16 17 26 ]).Whilethecompactness 38

PAGE 39

assumptionisessentialforperformingcomplexityanalysis,theapplicabilityofexistingBLmethodsonunconstrainedCOPmaybeimpaired.Inpractice,onepossibleworkaroundforBLmethodstosolve( 2{57 )mayinvolveanassumptiononthedistancefromapointxtoanoptimalsolutionx,namely,x2B(x;R):=fx2Rn:kx)]TJ ET q 0.478 w 309.88 -76.86 m 316.53 -76.86 l S Q BT /F6 11.955 Tf 309.88 -83.68 Td[(xkRgforsomexandR.Withsuchanassumption,wecansolve( 2{57 )byconsideringaball-constrainedCOPf x;R:=minx2B( x;R)f(x): (2{59)Itshouldbenotedthat,whiletheaboveequivalentreformulationseemsstraightforward,itscomputationalcomplexityreliesalmostexclusivelyontheradiusR.Inparticular,ifRisclosetothedistancefromxtoX,theoptimalsolutionsetof( 2{57 ),i.e.,R)]TJ /F6 11.955 Tf 12.03 0 Td[(Dissmallenough,where x:=argminxfk x)]TJ /F6 11.955 Tf 11.96 0 Td[(xk:x2XgandD:=k x)]TJ /F6 11.955 Tf 11.95 0 Td[(xk;(2{60)thenthecomputationalcomplexityforcomputingapproximatesolutionstotheball-constrainedCOP( 2{59 )andtheoriginalunconstrainedCOP( 2{57 )areclose,anditisdenitelyreasonabletosolve( 2{59 )instead.However,ifRisseverelyoverestimated,thecomputationalcomplexityforsolvingtheball-constrainedCOP( 2{59 )maybecomemuchhigherthantheoptimalcomplexityboundthatdependsonD.Basedontheabovediscussion,wecanconcludethatagoodBLmethodshouldtackletheunconstrainedCOP( 2{57 )fromtwoperspectives.Firstly,withoutanysatisableknowledgeregardingD,suchBLmethodshouldstillbeabletosolve( 2{57 )withoptimalcomplexityboundthatdependsonD.Secondly,ifthereexistsanapproximatedistanceRthatisclosetoD,agoodBLmethodshouldsolvetheball-constrainedproblem( 2{59 )eciently.Wewillconsideragenericframeworkthatfollowstheformerperspectiveinthissubsection,andthenextendtheBLmethodtosolve( 2{57 )inthenextsubsection.Inthissubsection,ourgoalistodesignagenericalgorithmicframeworkthatfollowstheformerperspectiveintheabovediscussion,andsolvetheunconstrainedCOP( 2{57 )throughsolutionstoaseriesofball-constrainedCOPs.Itshouldbenotedthatsuchconceptindeed 39

PAGE 40

followsnaturallyfromthetwoperspectivesdiscussedabove:ifaBLmethodiscomputationallyecientforball-constrainedCOPsofform( 2{59 ),startingfromcertainRfor( 2{59 )andenlargingitbytwoeachtime,wewilleventuallyreachacloseenoughestimateofDafterlogarithmicamountoftimes.Given x2Rn;R>0;>0,letusassumethatthereexistsarst-orderalgorithm,denotedbyA( x;R;),whichcanndan-solutionto( 2{59 ).Inotherwords,weassumethateachcalltoA( x;R;)willcomputeapointz2B( x;R)suchthatf(z))]TJ /F6 11.955 Tf 11.98 0 Td[(f x;R.Moreover,throughoutthissection,weassumethatthenumberof(sub)gradientevaluationsrequiredbyA( x;R;)forndingan-solutionto( 2{59 )isboundedbyN x;R;:=C1( x;R;f)R1 1+C2( x;R;f)R2 2; (2{61)where11>0and22>0.C1( x;R;f)andC2( x;R;f)areconstantsthatdependonfin( 2{59 )andnondecreasingwithrespecttoR.Forexample,iffisasmoothconvexfunction,rfisLipschitzcontinuousinRnwithconstantL,i.e.,( 2{58 )holdswith(Rn)=1andM(Rn)=L,andweapplytheAPLmethodto( 2{59 ),thenwehaveonlyonetermwith1=1,1=1=2,andC1( x;R;f)=cp Lin( 2{61 ),wherecisauniversalconstant.Observethatthetwocomplexitytermsin( 2{61 )willbeusefulforanalyzingsomestructuredCOPproblemsinSection 2.2.2 .ItshouldalsobenotedthatamoreaccurateestimateofC1( x;R;f)iscM(B( x;R)),sincetheLipschitzconstantL=M(Rn)throughoutRnislargerthanorequaltothelocalLipschitzconstantonB( x;R).LetAbethealgorithmthatcanndan-solutiontoball-constrainedproblem( 2{59 ).Byutilizinganovelguessandcheckprocedure,wepresentanexpansionalgorithmforunconstrainedconvexoptimizationsasfollows: Algorithm3. ExpansionalgorithmforunconstrainedCOPChooseanarbitraryr1>1andcomputetheinitialgap1:=f( x))]TJ /F3 11.955 Tf 9.3 0 Td[(minx2B( x;r1)h( x;x).Fork=1;2;:::, 40

PAGE 41

Step1.Solve x0k=A( x;rk;k),thenusex0kastheinitialpointtosolve x00k=A( x;2rk;k). Step2.Iff( x0k))]TJ /F6 11.955 Tf 11.96 0 Td[(f( x00k)>k,updaterk 2rkandgotoStep 1 Step3.Otherwise,output xk= x00k,andletk+1=k=2andrk+1=rk.Notethatinordertoreecttheexibilitywechooser1asanypositiveconstantinAlgorithm 3 .However,fromthetheoreticalcomplexitypointofview,weprefertostartingwithasmallerr1(i.e.,alowerboundonD).Infact,giventhetargetaccuracyonecanderiveatheoreticallyviableselectionofr1givenasfollows.Forsimplicity,assumethatfhasLipschitzcontinuousgradientwithconstantL,wehavef(x0))]TJ /F6 11.955 Tf 12.79 0 Td[(fL 2kx0)]TJ /F6 11.955 Tf 12.8 0 Td[(xk2.Ifkx0)]TJ /F6 11.955 Tf 12.75 0 Td[(xkp 2=L,thenx0alreadysatisesf(x0))]TJ /F6 11.955 Tf 12.75 0 Td[(f.Therefore,wecansetr1=p 2=L.SuchanestimaterequiressomepriorinformationofL.ItshouldbenotedthatanoverestimateonLdoesnothurtthecomplexitybound,sincer1isupdatedexponentiallyfastandwillapproachDquickly.Steps 1 and 2 inAlgorithm 3 constitutealoopforndingapairofsolutions( x0k; x00k)satisfying0f( x0k))]TJ /F6 11.955 Tf 11.96 0 Td[(f( x00k)kforanyk1.Since x0kand x00karek-optimalsolutionstominx2B( x;rk)f(x)andminx2B( x;2rk)f(x)respectively,thisloopmustterminateinnitetime,becauseitwillterminatewheneverrkD.Forsimplicity,wecallitanexpansionifwedoubletheradiusinStep 2 .Eachiterationmaycontainseveralexpansionsbeforeoutputtingsolution xkinStep 3 .Notethat,fortheimplementationofAlgorithm 3 ,mostcomputationalworkexistsinStep 1 ,whichinvolvesasequenceofcallstoalgorithmA.However,noticethatthegaps(k)andradiuses(rk)forthesecallsaremonotonicallydecreasingandincreasing,respectively,numerouscomputationalcostcouldbesavedbyusingresultsfrompreviousexpansionsanditerations.Forinstance,theoutputsolutionsofpreviouscallstoAassociatedwithlargergapsorsmallerradiusescouldalwaysbeusedasthestartingpointforthecurrentcalltoA.Forsuccessiveexpansions,ifthepreviousexecutionofStep 1 calledA(x;rk;k)andA(x;2rk;k)andthecurrentexecutionofStep 1 callsA(x;2rk;k)andA(x;4rk;k),thecomputationofcallA(x;2rk;k)could 41

PAGE 42

besaved.Moreover,ifAisreferredtosomespecicalgorithms,suchasBLtypemethods,morepreviousresults,likethelowerboundsandprox-centers,couldalsobeutilizedtosavecomputationalcost.Notethatthereareproximalbundlemethodsincorporatingwithtrustregiontechniquesforupdatingthenewiterate[ 39 40 ].In[ 39 ]anquadraticcuttingplanemodelbasedmethodandin[ 40 ]theideaofChebychevcenterwereusedtogeneratethetrustregions.Thesetrustregionmethodsrestricttheiteratesinthetrustregionforbetterconvergencetotheoptimalsolution,whiletheapproximatesolutionsinthesearchingballsgeneratedbyourexpansionalgorithmareusedonlyforcheckingwhetherornotthecurrentsearchingballneedstobeexpandedinordertogetabetterestimateofD.BeforeanalyzingtheiterationcomplexityofAlgorithm 3 ,wediscusssomeimportantobservationsrelatedtotheaforementionedexpansions. Lemma5. Letx2Rnandr>0beaconstant, x1and x2be-solutionstoCOPs f x;r:=minx2B( x;r)f(x)andf x;2r:=minx2B( x;2r)f(x);(2{62)respectively.If0f( x1))]TJ /F6 11.955 Tf 11.95 0 Td[(f( x2),thenwehave f( x2))]TJ /F6 11.955 Tf 11.95 0 Td[(f3+2D r;(2{63)wherefandDaredenedin( 2{57 )and( 2{60 )respectively. Proof. Clearly,bydenition,wehavek x1)]TJ ET q 0.478 w 217.08 -465.37 m 223.73 -465.37 l S Q BT /F6 11.955 Tf 217.08 -472.19 Td[(xkr,k x2)]TJ ET q 0.478 w 290.51 -465.37 m 297.16 -465.37 l S Q BT /F6 11.955 Tf 290.51 -472.19 Td[(xk2r,0f( x1))]TJ /F6 11.955 Tf 11.98 0 Td[(f x;rand0f( x2))]TJ /F6 11.955 Tf 9.77 0 Td[(f x;2r:Itsucestoconsiderthecasewhenfx;2r>fandkx)]TJ ET q 0.478 w 396.61 -489.27 m 403.27 -489.27 l S Q BT /F6 11.955 Tf 396.61 -496.09 Td[(xk>2r,sinceotherwise( 2{63 )holdstrivially.Supposex1andx2arethesolutionstotherstandsecondproblemsin( 2{62 )respectively,let^xbetheintersectionofthelinesegment(x;x1)withtheballB( x;2r),anddenoteR1:=k^x)]TJ /F6 11.955 Tf 10.56 0 Td[(x1kandR2:=kx)]TJ /F6 11.955 Tf 10.57 0 Td[(x1k.Clearly,^x=(1)]TJ /F4 7.97 Tf 11.76 4.82 Td[(R1 R2)x1+R1 R2x.Bytheconvexityoff(),wehave f(^x)(1)]TJ /F6 11.955 Tf 13.15 8.09 Td[(R1 R2)f(x1)+R1 R2f(x);(2{64) 42

PAGE 43

whichimpliesthat R1 R2[f(x1))]TJ /F6 11.955 Tf 11.96 0 Td[(f(x)]f(x1))]TJ /F6 11.955 Tf 11.96 0 Td[(f(^x);(2{65)andthatf(^x)f(x1)duetothefactthatf(x)f(x1).Also,wehavef(^x)f(x2)since^x2B( x;2r).Inaddition,f(x1))]TJ /F6 11.955 Tf 11.95 0 Td[(f(x2)=[f(x1))]TJ /F6 11.955 Tf 11.95 0 Td[(f( x1)]+[f( x1))]TJ /F6 11.955 Tf 11.96 0 Td[(f( x2)]+[f( x2))]TJ /F6 11.955 Tf 11.95 0 Td[(f(x2)] (2{66)0++=2: (2{67)Combiningthepreviousinequalities,weobtain f(x1))]TJ /F3 11.955 Tf 11.95 0 Td[(2f(x2)f(^x)f(x1);(2{68)whichimpliesthatf(x1))]TJ /F6 11.955 Tf 13.12 0 Td[(f(^x)2.Using( 2{65 ),andthefactthatR1randR2D+r,wehavef(x1))]TJ /F6 11.955 Tf 11.95 0 Td[(f(x)2R2 R12+2D r:Therefore,f( x2))]TJ /F6 11.955 Tf 11.96 0 Td[(f(x)f( x1))]TJ /F6 11.955 Tf 11.95 0 Td[(f(x)[f( x1))]TJ /F6 11.955 Tf 11.95 0 Td[(f(x1)]+[f(x1))]TJ /F6 11.955 Tf 11.96 0 Td[(f(x)]3+2D r: WearenowreadytoprovetheiterationcomplexityofAlgorithm 3 forsolvingtheunconstrainedCOP( 2{57 ). Theorem3. Supposethatthenumberof(sub)gradientevaluationsrequiredbyA( x;R;)forndingan-solutionto( 2{59 )isboundedby( 2{61 ).Foranyk1,denotek:=f( xk))]TJ /F6 11.955 Tf 12.09 0 Td[(ffortheoutput xkinAlgorithm 3 .Thenwehave a) rkr:=maxfr1;2Dgforallk1,whereDisdenedin( 2{60 ); b) limk!1k=0; 43

PAGE 44

c) Thetotalnumberof(sub)gradientevaluationsperformedbyAlgorithm 3 forndingthek-solutionxktoproblem( 2{57 )isboundedby O C1( x;2r;f)r1 1k+C2( x;2r;f)r2 2k!:(2{69) Proof. Westartbyprovinga),ifr1D,thenf(x0k))]TJ /F6 11.955 Tf 12.37 0 Td[(f(x00k)kforanyk1andnoexpansiontakesplace,hencerk=r1=r.Ifr1D,fromAlgorithm 3 ,weseethatexpansionoccursatStep 2 ifandonlyiff( x0k))]TJ /F6 11.955 Tf 12.27 0 Td[(f( x00k)>k.Hence,ifrkD,thisconditionisnotsatisedandnomoreexpansionisperformed.Thisimpliesrk<2D.Toproveb),observethatx0kisusedastheinitialpointforcomputingx00kinAlgorithm 3 andhencef(x00k)f(x0k).CombiningthisobservationwiththeconditioninStep 3 ,wehave0f(x0k))]TJ /F6 11.955 Tf 11.96 0 Td[(f(x00k)k.ApplyingLemma 5 implies k=f( x00k))]TJ /F6 11.955 Tf 11.95 0 Td[(f3+2D rkk:(2{70)Notethatthetotalnumberofexpansionisboundedby S1:=log2r r1;(2{71)hencekdecreasesto0askincreases,wehavelimk!1k=0.Toprovec),assumethatthenumberofexecutionsofStep 1 inAlgorithm 3 fornding xkisK.Forany1jK,letA( x;Rj;j)andA( x;2Rj;j)becalledinthejthexecutionofStep 1 .Byusing( 2{61 )andnotingthatC1( x;R;f)andC2( x;R;f)arenondecreasingwithrespecttoR,wehavethenumberof(sub)gradientevaluationsperformedbythejthexecutionofStep 1 isboundedby Nj:=(1+21)C1( x;2Rj;f)R1j 1j+(1+22)C2( x;2Rj;f)R2j 2j:(2{72)LetN0jandN00jbetherstandsecondtermsontherightof( 2{72 ),respectively.The(j+1)thexecutionofStep 1 eitherdoublestheradiusorreducesthegapbyhalfcomparingtothejthexecution,i.e.,Rj+1=2Rjorj+1=j=2respectively.Thereforewehaveeither 44

PAGE 45

N0j+121N0jandN00j+122N00j,orN0j+121N0jandN00j+122N00j.Since2121>1and2222>1,wecancombinethesetwocasesandhave N0j2)]TJ /F4 7.97 Tf 6.59 0 Td[(1N0j+1andN00j2)]TJ /F4 7.97 Tf 6.59 0 Td[(2N00j+1;for1jK)]TJ /F3 11.955 Tf 11.96 0 Td[(1;(2{73)whichfurtherimplies N0j2)]TJ /F4 7.97 Tf 6.59 0 Td[(1(K)]TJ /F4 7.97 Tf 6.58 0 Td[(j)N0KandN00j2)]TJ /F4 7.97 Tf 6.59 0 Td[(2(K)]TJ /F4 7.97 Tf 6.59 0 Td[(j)N00K;for1jK:(2{74)Thenthetotalnumberof(sub)gradientevaluationsperformedbytheseKexecutionsofStep 1 inAlgorithm 3 isboundedbyN:=KXj=1(N0j+N00j)N0KKXj=12)]TJ /F4 7.97 Tf 6.59 0 Td[(1(K)]TJ /F4 7.97 Tf 6.59 0 Td[(j)+N00KKXj=12)]TJ /F4 7.97 Tf 6.59 0 Td[(2(K)]TJ /F4 7.97 Tf 6.59 0 Td[(j) (2{75)0,Ci( x;2rk;f)rik(3+2D rk)i=Ci( x;2rk;f)ri)]TJ /F4 7.97 Tf 6.59 0 Td[(ik(3rk+2D)ifori=1;2aremonotonicallyincreasingwithrespecttork,which,inviewofthefactrk<2rprovedbyparta),thereforeclearlyimplies N<2Xi=1(23i+2i+3i)Ci( x;2r;f) 2i)]TJ /F3 11.955 Tf 11.95 0 Td[(1ri ik:(2{79)Hencetheproofiscomplete. NotethattosolvetheunconstrainedCOP( 2{57 ),theterminationcriterionsofmostrst-orderalgorithmsarebasedontheresidualofthe(sub)gradient,whichwouldleadto 45

PAGE 46

dierentcomplexityanalysis.Tothebestofourknowledge,withoutanypriorinformationonD,thereisnoveriableterminationcriterionbasedonfunctionaloptimalitygapthatcouldguaranteetheterminationofalgorithmsforndingan-solutionto( 2{57 ).ComparingtoNesterov'soptimalgradientmethodforunconstrainedproblemsin[ 3 ],Algorithm 3 onlyprovideseciencyestimatesaboutk:=f( xk))]TJ /F6 11.955 Tf 12.54 0 Td[(fwhentheoutput xkisupdated,whiletheoptimalgradientmethodcouldhaveestimatesabout k:=f(xk))]TJ /F6 11.955 Tf 12.55 0 Td[(fforeachiteratexk.ForbothmethodstheeciencyestimatesinvolveD.SinceAlgorithm 3 extendsmethodsforball-constrainedCOPstosolve( 2{57 ),andtheiterationsintheexpansionsofAlgorithm 3 couldberegardedasaguessandcheckproceduretoestimateD,itisreasonablethattheeciencyestimatesareonlyprovidedforunexpansivesteps,i.e.,Step 3 ofAlgorithm 3 ,whichoutput xk.Ithasbeenshownin[ 26 ]thattheAPLmethodanditsvariant,theUSLmethod,achievetheoptimaliterationcomplexitiesin( 2{59 )forsmooth,nonsmoothandweaklysmoothCOPsandaclassofstructuredsaddlepointproblemsrespectively,onanyconvexandcompactfeasibleset.SoAlgorithm 3 couldbeincorporatedtosolve( 2{57 )withtheoptimaliterationcomplexitiestoo.Therefore,theremainingpartofthispaperwillfocusonhowtoimprovetheeciencyoftheseBLtypemethodsforsolvingball-constrainedCOPs. 2.3.2ExtendingFAPLandFUSLforUnconstrainedConvexOptimizationInthissubsection,westudyhowtoutilizetheFAPLandFUSLmethodstosolvetheunconstrainedproblemsbasedonourresultsinSection 2.3 .Letusrstconsiderthecasewhenfin( 2{57 )satises( 2{58 ).IfthemethodAinStep 1 ofAlgorithm 3 isgivenastheFAPLmethod,thenbyTheorem 1 ,andthefactthatonlyone(sub)gradientoffiscomputedineachiterationoftheFAPLmethod,thenumberof(sub)gradientevaluationswithinonecalltoA( x;R;)isboundedbyN()givenby( 2{30 ).Therefore,fortheFAPLmethod,wehave1=2(1+) 3+2;1=2 1+3;C1( x;R;f)=C0M2 1+3andC2( x;R;f)=2=2=0in( 2{61 ),whereC0isaconstantdependingontheparametersq;;;andcintheFAPLmethod.Lettingk:=f( xk))]TJ /F6 11.955 Tf 12.32 0 Td[(ffork1inAlgorithm 3 and 46

PAGE 47

applyingTheorem 3 ,wethenconcludeforndingthek-solution xktoproblem( 2{57 ),thetotalnumberof(sub)gradientevaluationsperformedbyAlgorithm 3 isboundedbyO M(2 r)1+ k2 1+3!; (2{80)whereM:=M(B( x;2 r)),:=(B( x;2 r))and risdenedinparta)ofTheorem 3 .ItshouldbenotedthattheconstantsMandarelocalconstantsthatdependonthesizeoftheinitialball,i.e.,r1,andthedistancefrom xandx,whicharenotrequiredfortheFAPLmethodandAlgorithm 3 ,andalsogenerallysmallerthantheconstantsM(Rn)and(Rn),respectively,fortheglobalHoldercontinuitycondition.Moreover,iffin( 2{57 )isgivenintheformof( 2{28 )asastructurednonsmoothCOP,thentheFUSLmethodcouldbeappliedtosolvethecorrespondingstructuredball-constrainedproblemsinAlgorithm 3 .ByTheorem 2 ,thenumberof(sub)gradientevaluationsoffwithinonecalltoA( x;R;)isboundedby S1+S2+C0Rr L^f +C00RkAk ;(2{81) whereC0;C00aresomeconstantsdependingontheparametersq;;;v;c;D0andDv;YintheFUSLmethod.ApplyingTheorem 3 with1=2=1,1=1 2,2=1,C1( x;R;f)=C0p L^f,andC2( x;R;f)=C00kAk,weconcludethatthetotalnumberof(sub)gradientevaluationsperformedbyAlgorithm 3 tondthek-solution xktoproblem( 2{57 )-( 2{28 )isboundedby O0@2C0 rs L^f k+2C00 rkAk k1A:(2{82)SimilartotheFAPLmethod,hereLf:=Lf(B( x;2 r))isalowerboundofLf(Rn). 47

PAGE 48

2.4GeneralizationtoStronglyConvexOptimizationInthissection,wegeneralizetheFAPLandFUSLmethodsforsolvingconvexoptimizationproblemsintheformof( 1{1 )whoseobjectivefunctionfsatises f(y))]TJ /F6 11.955 Tf 11.96 0 Td[(f(x))-221(hf0(x);y)]TJ /F6 11.955 Tf 11.95 0 Td[(xi 2ky)]TJ /F6 11.955 Tf 11.95 0 Td[(xk2;8x;y2Rn;(2{83)forsome>0.Forthesakeofsimplicity,weassumethroughoutthissectionthataninitiallowerboundlb0fisavailable1.Underthisassumption,itfollowsfrom( 3{51 )thatkp0)]TJ /F6 11.955 Tf 12.03 0 Td[(xk22[f(p0))]TJ /F3 11.955 Tf 12.03 0 Td[(lb0]=foragiveninitialpointp0,andhencethattheFAPLandFUSLmethodsforball-constrainedproblemscanbedirectlyapplied.However,sincethelowerandupperboundsonfareconstantlyimprovedineachphaseofthesealgorithms,wecanshrinktheballconstraintsbyaconstantfactoronceeveryphaseaccordingly.WeshowthattherateofconvergenceoftheFAPLandFUSLmethodscanbesignicantlyimprovedinthismanner.WerstpresentamodiedFAPLmethodforsolvingblack-boxCOPswhichsatisfyboth( 3{2 )and( 3{51 ).Morespecically,wemodifytheballconstraintsusedintheFAPLmethodbyshiftingtheprox-centerxandshrinkingtheradiusRinprocedureGFAPL.Clearly,suchamodicationdoesnotincuranyextracomputationalcost.Thisalgorithmisformallydescribedasfollows. ThemodiedFAPLgapreductionprocedure:(x+;lb+)=~GFAPL(^x;lb;r;;)InFAPLgapreductionprocedure,set x=^x,andconsequentlytheprox-functiondin( 2{6 )isreplacedbykx)]TJ /F3 11.955 Tf 12.68 0 Td[(^xk2=2. Algorithm4. ThemodiedFAPLmethodforstronglyconvexoptimizationInAlgorithm 1 ,changesteps 0 1 and 3 tothefollowingsteps. 0: Chooseinitiallowerboundlb1f,initialpointp02Rn,initialupperboundub1=f(p0),tolerance>0andparameters;2(0;1). 1Otherwise,weshouldincorporateaguess-and-checkproceduresimilartotheoneinSection 2.3 48

PAGE 49

1: Set^x1=p0,ands=1. 3: Set(^xs+1;lbs+1)=~GFAPL(^xs;lbs;p 2(f(^xs))]TJ /F3 11.955 Tf 11.95 0 Td[(lbs)=;;)andubs+1=f(^xs+1).AfewremarksontheabovemodiedFAPLmethodareinplace.Firstly,letxbetheoptimalsolutionofproblem( 1{1 )anddenes=ubs)]TJ /F3 11.955 Tf 11.98 0 Td[(lbs.Bythedenitionofubsandlbs,wehavef(^xs))]TJ /F6 11.955 Tf 11.95 0 Td[(f(x)s,which,inviewof( 3{51 ),thenimpliesthat k^xs)]TJ /F6 11.955 Tf 11.96 0 Td[(xk22s =:r2;(2{84)andx2B(^xs;r).Secondly,similartoprocedureGFAPL,ifprocedure~GFAPLterminatesatstep2,wehaveEf(l)\B(^xs;r)=;.Combiningthiswiththefactx2B(^xs;r),weconcludethatlisavalidlowerboundonf.Therefore,nomatterwhetherprocedure~GFAPLterminatesatstep2orstep4,thegapbetweenupperandlowerboundsonfhasbeenreducedands+1qs,wheretheqisdenedin( 3{32 ).WeestablishinTheorem 4 theiterationcomplexityboundsofthemodiedFAPLmethodforminimizingstronglyconvexfunctions. Theorem4. Supposethatfkgk1inprocedure~GFAPLarechosensuchthat( 2{10 )holds.ThenthetotalnumberofiterationsperformedbythemodiedFAPLmethodforcomputingan-solutionofproblem( 1{1 )isboundedbyeS cs M +1!andeS+1 1)]TJ /F6 11.955 Tf 11.95 0 Td[(q1)]TJ /F10 5.978 Tf 5.76 0 Td[( 1+3 21+ 2c1+M (1+)1+ 21)]TJ /F10 5.978 Tf 5.75 0 Td[( 2!2 1+3;respectively,forsmoothstronglyconvexfunctions(i.e.,=1)andnonsmoothorweaklysmoothstronglyconvexfunctions(i.e.,2[0;1)),whereqisdenedin( 3{32 ),lb1andub1aregiveninitiallowerboundandupperboundonf,and eS:=log1 qub1)]TJ /F3 11.955 Tf 11.96 0 Td[(lb1 :(2{85) 49

PAGE 50

Proof. Supposethatprocedure~GFAPLdoesnotterminateatthekthinneriteration.Itthenfollowsfrom( 2{19 )and( 2{84 )that f(xuk))]TJ /F6 11.955 Tf 11.95 0 Td[(lMr1+ 1+c1+ k1+3 2:(2{86)Moreover,inviewoftheterminationconditioninStep 3 andrelation( 2{84 ),wehavef(xuk))]TJ /F6 11.955 Tf -454.71 -23.91 Td[(l(ubs)]TJ /F6 11.955 Tf 12.12 0 Td[(l)=sandr=p 2s=.Combiningalltheaboveobservationsweconcludethat k 21+ 2c1+M (1+)1+ 21)]TJ /F10 5.978 Tf 5.76 0 Td[( 2s!2 1+3:(2{87)Sothenumberofinneriterationsperformedineachcalltoprocedure~GFAPLisboundedby 21+ 2c1+M (1+)1+ 21)]TJ /F10 5.978 Tf 5.76 0 Td[( 2s!2 1+3+1:(2{88)Sincethegapbetweentheupperandlowerboundsonfisreducedbyaconstantfactorineachphase,i.e.,s+1qs,iteasytoseethatthetotalnumberofphasesisboundedby~Sdenedabove.Usingtheprevioustwoconclusionsandthefactthats=qeS)]TJ /F4 7.97 Tf 6.58 0 Td[(s,wecanshowthatthetotalnumberofiterationsperformedbythemodiedFAPLmethodisboundedby eS+ 21+ 2c1+M (1+)1+ 21)]TJ /F10 5.978 Tf 5.76 0 Td[( 2!2 1+3eSXs=1q(eS)]TJ /F4 7.97 Tf 6.59 0 Td[(s)1)]TJ /F10 5.978 Tf 5.75 0 Td[( 1+3:(2{89)Specically,iffissmooth(=1),thentheaboveboundisreducedto eS cs M +1!:(2{90)Iffisnonsmooth(=0)orweaklysmooth(2(0;1)),thentheaboveboundisequivalenttoeS+ 21+ 2c1+M (1+)1+ 21)]TJ /F10 5.978 Tf 5.76 0 Td[( 2!2 1+3eSXs=1q(eS)]TJ /F4 7.97 Tf 6.58 0 Td[(s)1)]TJ /F10 5.978 Tf 5.76 0 Td[( 1+3eS+1 1)]TJ /F6 11.955 Tf 11.95 0 Td[(q1)]TJ /F10 5.978 Tf 5.76 0 Td[( 1+3 21+ 2c1+M (1+)1+ 21)]TJ /F10 5.978 Tf 5.75 0 Td[( 2!2 1+3: (2{91) 50

PAGE 51

NowletusconsidertheSPproblemgivenby( 2{28 ),wherethesmoothcomponent^fisstronglyconvexwithmodulus.SimilartothemodiedFAPLmethod,wepresentamodiedFUSLmethodforsolvingthisstronglyconvexSPproblemasfollows. ThemodiedFUSLgapreductionprocedure:(x+;D+;lb+)=~GFUSL(^x;D;lb;r;;)InFUSLgapreductionprocedure,set x=^x,andconsequentlytheprox-functiondisreplacedbykx)]TJ /F3 11.955 Tf 12.68 0 Td[(^xk2=2. Algorithm5. ThemodiedFUSLmethodforstronglyconvexSPproblemInAlgorithm 2 ,changesteps 0 1 and 3 tothefollowingsteps. 0: Chooseinitiallowerboundlb1f,initialpointp02Rn,initialupperboundub1=f(p0),prox-functionv(),initialguessD1onthesizeDv;Y,tolerance>0andparameters;2(0;1). 1: Set^x1=p0,ands=1. 3: Set(^xs+1;Ds+1;lbs+1)=~GFUSL(^xs;Ds;lbs;p 2(f(^xs))]TJ /F3 11.955 Tf 11.95 0 Td[(lbs)=;;)andubs+1=f(^xs+1).Inthefollowingtheorem,wedescribetheconvergencepropertiesofthemodiedFUSLmethodforsolving( 1{1 )-( 2{28 )withstronglyconvexsmoothcomponent^f. Theorem5. Supposethatfkgk1inprocedure~GFUSLarechosensuchthat( 2{10 )holds.ThenwehavethefollowingstatementsholdforthemodiedFUSLmethod. a) ThetotalnumberofiterationsperformedbythemodiedFUSLmethodforcomputingan-solutionofproblem( 1{1 )-( 2{28 )isboundedby (S1+eS)0@s 2cL^f +11A+4kAkp ~D (1)]TJ 11.96 7.45 Td[(p q)r c v;(2{92)whereqisdenedin( 2{10 ),S1and~Daredenedin( 2{45 ),andeSisdenedin( 2{85 ). b) Inparticular,ifDv;Yisknown,andsetD1=Dv;YatStep 0 ,thenthenumberofiterationsperformedbythemodiedFUSLmethodisreducedto N():=eS0@s 2cL^f +11A+2kAk (1)]TJ 11.96 7.45 Td[(p q)s cDv;Y v:(2{93) 51

PAGE 52

Proof. SimilarlytothediscussioninTheorem 2 ,weclassifythenon-signicantandsignicantphasesandestimatesthenumbersofiterationsperformedbyeachtypeofphases.Supposethatthesetofindicesofthenon-signicantandsignicantphasesarefm1;m2;:::;ms1gandfn1;n2;:::;ns2grespectively.ThenthenumberofnonsignicantphasesisboundedbyS1,i.e.,s1S1.Andsince1=ub1)]TJ /F3 11.955 Tf 12.58 0 Td[(lb1,sothenumberofsignicantphasesisboundedbyeSdenedabove,i.e.,s2S2.InviewofProposition 2 ,andsubstituter=q 2 ,wehaveforanyphase eN(;D):=s 2cL^f +2kAk s cD v+1:(2{94)FollowingsimilardiscussioninTheorem 2 ,wehavethenumberofiterationsperformedbynon-signicantphasesinthemodiedFUSLmethodisboundedbyeN1=s1Xk=1eN(mk;Dmk)s1Xk=1eN(;~D=2s1)]TJ /F4 7.97 Tf 6.58 0 Td[(k) (2{95)S10@s 2cL^f +11A+2kAk s c~D vS1Xk=1qS1)]TJ /F10 5.978 Tf 5.75 0 Td[(k 2 (2{96)S10@s 2cL^f +11A+2kAk (1)]TJ 11.96 7.45 Td[(p q)s c~D v: (2{97)AndtheboundonnumberofiterationsperformedbyallsignicantphasesisgivenbyeN2=s2Xk=1eN(nk;Dnk)s2Xk=1eN(=qs2)]TJ /F4 7.97 Tf 6.59 0 Td[(k;~D) (2{98)eS0@s 2cL^f +11A+2kAk s c~D veSXk=1qeS)]TJ /F10 5.978 Tf 5.76 0 Td[(k 2 (2{99)eS0@s 2cL^f +11A+2kAk (1)]TJ 11.95 7.45 Td[(p q)s c~D v: (2{100)Therefore,thetotalnumberofiterationsisboundedbyeN1+eN2,andthusparta)holds. 52

PAGE 53

Forpartb),inviewofLemma 4 ,wecanseethatifD1=Dv;Y,thenDsDv;Yforalls1,andallphasesofthemodiedFUSLmethodaresignicant.Therefore,replaceDnkand~Din( 2{98 ),wecanconcludepartb)holds. InviewoftheaboveTheorem 5 ,wecanseethattheiterationcomplexityofthemodiedFUSLmethodforsolvingthestronglyconvexSPproblem( 2{28 )isboundedbyO(kAk=p ): 2.5NumericalExperimentsInthissectionweapplytheFAPLandFUSLmethodstosolveafewlarge-scaleCOPs,includingthequadraticprogrammingproblemswithlargeLipschitzconstants,syntheticandrealworldtotalvariationbasedimagereconstructionproblems,thencomparethemwithsomeotherrst-orderalgorithms.AllthealgorithmswereimplementedinMATLAB,VersionR2011aandallexperimentswereperformedonadesktopwithanInterDualCore2Duo3.3GHzCPUand8Gmemory. 2.5.1QuadraticProgrammingThemainpurposeofthissectionistoinvestigatetheperformanceoftheFAPLmethodforsolvingsmoothCOPsespeciallywithlargeLipschitzconstantsanddemonstratetheimprovementoftheFAPLmethodcomparingtosomeotherstate-of-the-artBLtypemethods.Andsincemoststate-of-the-artBLtypemethodsrequirecompactfeasiblesets,weconsiderthefollowingquadraticprogrammingproblem: minkxk1kAx)]TJ /F6 11.955 Tf 11.95 0 Td[(bk2;(2{101)whereA2Rmnandb2Rm.WecomparetheFAPLmethodwithNesterov'soptimalmethod(NEST)forsmoothfunctions[ 2 ],NERML[ 17 ],andAPL[ 26 ].WealsocomparetheFAPLmethodwiththebuilt-inMatlablinearsystemsolverinviewofitsgoodpracticalperformance.IntheAPLmethod,thesubproblemsaresolvedbyMOSEK[ 41 ],anecientsoftwarepackageforlinearandsecond-orderconeprogramming.TwocaseswithdierentchoicesofinitiallowerboundLBintheseexperimentsareconducted:(1).LB=0and(2).LB=. 53

PAGE 54

Inourexperiments,givenmandn,twotypesofmatrixAaregenerated.ThersttypeofmatrixAisrandomlygeneratedwithentriesuniformlydistributedin[0,1],whiletheentriesofthesecondtypearenormallydistributedaccordingtoN(0;1).WethenrandomlychooseanoptimalsolutionxwithintheunitballinRn,andgeneratethedatabbyb=Ax.Weapplyallthesefourmethodsaforementionedtosolve( 2{101 )withthissetofdataAandb,andtheaccuracyofthegeneratedsolutionsaremeasuredbyek=kAxk)]TJ /F6 11.955 Tf 12.2 0 Td[(bk2.TheresultsareshowninTables 2-1 2-2 and 2-3 Table2-1. UniformlydistributedQPinstances A:n=4000;m=3000;L=2:0e6;e0=2:89e4 Alg LB Iter.TimeAcc. Iter.TimeAcc. FAPL 0 1033.069.47e-7 1423.768.65e-9 2776.555.78e-7 80019.182.24e-11 APL 0 12837.109.07e-7 21060.859.82e-9 30085.656.63e-6 800234.692.59e-9 NERML 0 21858.329.06e-7 500134.621.63e-8 30084.011.02e-2 800232.141.71e-3 NEST 10000220.13.88e-5 20000440.023.93e-6 A:n=8000;m=4000;L=8:0e6;e0=6:93e4 Alg LB Iter.TimeAcc. Iter.TimeAcc. FAPL 0 704.677.74e-7 956.466.85e-10 1498.996.27e-7 27616.946.10e-10 APL 0 7971.247.79e-7 144129.523.62e-9 248205.488.16e-7 416358.968.68e-9 NERML 0 153128.717.30e-7 300251.794.03e-9 300257.541.18e-3 800717.139.24e-5 NEST 10000681.035.34e-5 200001360.524.61e-6 FAPLmethodforlargedimensionmatrix MatrixA:mn LB Iter.TimeAcc. Iter.TimeAcc. 1000020000 0 9736.656.41e-11 18569.317.29e-21 L=5.0e7 20773.708.28e-8 800292.062.32e-15 1000040000 0 6749.959.21e-11 12291.497.27e-21 L=1.0e8 13088.407.11e-8 421295.151.95e-16 1000060000 0 5258.067.68e-11 95106.148.43e-21 L=1.5e8 156160.939.84e-8 394422.47.48e-16 InordertoinvestigatetheeciencyofsolvingunconstrainedCOPsusingtheproposedexpansionalgorithm(i.e.,Algorithm 3 ),weconducttwosetsofexperimentstocomparethe 54

PAGE 55

Table2-2. GaussiandistributedQPinstances A:n=4000;m=3000;L=2:32e4;e0=2:03e3 Alg LB Iter.TimeAcc. Iter.TimeAcc. FAPL 0 1052.788.43e-7 1534.107.84e-10 3388.026.86e-7 69616.589.74e-10 APL 0 12835.129.01e-7 17247.499.28e-9 639200.677.92e-7 800258.251.03e-7 NERML 0 19248.447.05e-7 27670.311.09e-8 30093.323.68e-1 800257.256.41e-2 NEST 10000211.307.78e-4 20000422.781.95e-4 A:n=8000;m=4000;L=2:32e4;e0=2:03e3 Alg LB Iter.TimeAcc. Iter.TimeAcc. FAPL 0 493.258.34e-7 684.377.88e-10 1659.775.17e-7 28016.185.06e-10 APL 0 5948.918.59e-7 7864.951.70e-8 300268.479.81e-7 670637.709.42e-10 NERML 0 105181.239.14e-7 133102.681.39e-8 300282.569.92e-3 800760.268.32e-4 NEST 10000567.593.88e-4 200001134.389.71e-5 FAPLmethodforlargedimensionmatrix MatrixA:mn LB Iter.TimeAcc. Iter.TimeAcc. 1000020000 0 7827.887.22e-11 14551.816.81e-21 L=5.7e4 22878.579.92e-8 800280.191.37e-15 1000040000 0 4834.365.97e-11 8762.248.26e-21 L=9e4 156106.127.18e-8 390271.154.29e-16 1000060000 0 3436.309.88e-11 6569.567.24e-21 L=1.2e5 9898.119.50e-8 350361.838.34e-16 Table2-3. ComparisontoMatlabsolver MatrixA:mn MatlabAnb FAPLmethod TimeAcc. Iter.TimeAcc. Uniform20004000 4.415.48e-24 2043.596.76e-23 Uniform20006000 7.129.04e-24 1554.109.73e-23 Uniform20008000 9.809.46e-24 1354.459.36e-23 Uniform200010000 12.431.04e-23 1084.237.30e-23 Gaussian30005000 11.175.59e-25 2076.257.18e-23 Gaussian30006000 13.961.43e-24 1525.509.59e-23 Gaussian30008000 19.571.66e-24 1054.838.17e-23 Gaussian300010000 25.181.35e-24 955.435.81e-23 55

PAGE 56

performanceofsolvingtheunconstrainedQPproblemminx2RnkAx)]TJ /F6 11.955 Tf 12.24 0 Td[(bk2usingtwodierentstrategies:onestartswithsmallinitialfeasibleball,thenappliestheunconstrainedFAPLmethod(i.e.,incorporatingtheexpansionalgorithmwiththeFAPLmethodassubroutineA),whiletheotherone,undertheassumptionthataboundonDdenedin( 2{60 )isknown,appliestheball-constrainedFAPLmethoddirectlybychoosingsomelargeballthatcontainsatleastoneoptimalsolution.Sincetheperformanceofbothmethodswouldbeaectedbythedistancebetweentheinitialpointx0andtheoptimalsolutionset,inbothexperiments,wesetx=0;D1,andforanyinitialballB(x;R),wechoosethestartingpointx0randomlywithintheballandthennormalizex0andsetkx0k=R 2.Forbothmethods,theinitiallowerboundissetto,andtheparametersoftheFAPLmethodarethesame.Inthisrstexperiment,Aisgeneratedrandomlywithentriesuniformlydistributedin[0;1].Inthesecondexperiment,weusetheworst-caseQPinstanceforrst-ordermethodswhicharegeneratedbyA.Nemirovski(seetheconstructionschemein[ 4 ]and[ 1 ]).WhenweapplytheunconstrainedFAPLmethod,theradiusesoftheinitialballsarechosenas10)]TJ /F9 7.97 Tf 6.58 0 Td[(5D,10)]TJ /F9 7.97 Tf 6.59 0 Td[(4D,10)]TJ /F9 7.97 Tf 6.59 0 Td[(3D,10)]TJ /F9 7.97 Tf 6.58 0 Td[(2Dand10)]TJ /F9 7.97 Tf 6.59 0 Td[(1D,respectively.Whiletheball-constrainedFAPLmethodisemployed,theradiusesoftheballsareselectedas105D,104D,103D,102Dand10D,respectively.TheresultsareshowninTable 2-4 .TheadvantagesoftheFAPLmethodcanbeobservedfromtheseexperiments.Firstly,itisevidentthatBLtypemethodshavemuchlessiterationsthanNESTespeciallywhentheLipschitzconstantoftheobjectivefunctionislarge.AmongthesethreeBLtypemethods,NERMLrequiresmuchmoreiterationsthanAPLandFAPL,whichhaveoptimaliterationcomplexityforthisproblem.Secondly,comparingtotheexistingBLtypemethods(APLandNERML),FAPLhasmuchlowercomputationalcostforeachiteration.ThecomputationalcostofFAPLforeachiterationisjustslightlylargerthanthatofNEST.However,thecostsofeachiterationofAPLandNERMLare10timeslargerthanthatofNEST. 56

PAGE 57

Table2-4. UnconstrainedQPinstances Uniforminstance:A=rand(4000;8000) FAPL-Unconstrained FAPL-Ballconstrained RadiusIter.TimeAcc. RadiusIter.TimeAcc. 1e-5140573.382.14e-11 1e51888104.167.07e-10 1e-4118762.702.14e-11 1e4141477.245.55e-10 1e-3112859.346.72e-11 1e3117264.519.69e-11 1e-293355.389.38e-11 1e281243.993.71e-11 1e-183545.895.38e-11 1e161733.016.94e-11 Worst-caseinstance:A=Bdata(2062;4124) FAPL-Unconstrained FAPL-Ballconstrained RadiusIter.TimeAcc. RadiusIter.TimeAcc. 1e-5140923.639.93e-8 1e5400077.175.32e-1 1e-4126221.579.97e-8 1e4400078.203.29e-3 1e-3113522.819.98e-8 1e3350068.986.38e-5 1e-284815.109.99e-8 1e2250046.671.01e-6 1e-184715.309.89e-8 1e1160228.929.89e-8 Thirdly,considerthedierencebetweentheperformanceofsettingtheinitiallowerboundequalto0and,itisalsoevidentthatFAPLismorerobusttothechoiceoftheinitiallowerboundanditupdatesthelowerboundmoreecientlythantheothertwoBLtypemethods.ThoughsettingtheinitiallowerboundequaltoincreasesthenumbersofiterationsforallthesethreeBLtypemethods,acloseexaminationrevealsthatthedierencebetweensettingthelowerboundtozeroandforFAPLisnotsosignicantasthatforAPLandNERML,especiallyforlargematrix,forexample,thesecondoneinTable 2-1 .Fourthly,FAPLneedslessnumberofiterationsthanAPL,especiallywhentherequiredaccuracyishigh.Aplausibleexplanationisthatexactlysolvingthesubproblemsprovidesbetterupdatingfortheprox-centers,andconsequently,moreaccurateprox-centersimprovetheeciencyofalgorithmsignicantly.Theexperimentsshowthat,forAPLandNERML,itishardtoimprovetheaccuracybeyond10)]TJ /F9 7.97 Tf 6.59 0 Td[(10.However,FAPLcankeepalmostthesamespeedfordeceasingtheobjectivevaluefrom106to10)]TJ /F9 7.97 Tf 6.59 0 Td[(21.Fifthly,wecanclearlyseefromTable 2-3 thatFAPLiscomparabletoorsignicantlyoutperformsthebuilt-inMatlabsolverforrandomlygeneratedlinearsystems,eventhoughourcodeisimplementedinMATLABratherthanlower-levellanguages,suchasCorFORTRAN. 57

PAGE 58

WecanexpectthattheeciencyofFAPLwillbemuchimprovedbyusingCorFORTRANimplementation,whichhasbeenusedintheMATLABsolverforlinearsystems.Finally,fromTable 2-4 itisevidentthattheperformanceofboththeunconstrainedFAPLmethodandtheball-constrainedFAPLmethodareaectedbythedistancebetweenthestartingpointx0andtheoptimalsolutionset.AndimproperestimationsonDwouldincreasethecomputationalcostsignicantly.ComparingtheresultspresentedinthesamerowsoftheleftandrightcolumnsinTable 2-4 ,onecanseethatwhentheestimationsareproper(i.e.,RisclosetoD),theperformanceofbothmethodsarecomparable,andwhentheestimationsareimproper,theunconstrainedFAPLmethodrequiresmuchlessiterationsandCPUtimeandcanachievehigheraccuracythantheball-constrainedFAPLmethod.Especially,fortheworst-caseinstance,theadvantagesaremuchmoresignicant.Insummary,duetoitslowiterationcostandeectiveusageofthememoryofrst-orderinformation,theFAPLmethodisapowerfultoolforsolvingball-constrainedsmoothCOPsespeciallywhenthenumberofvariablesishugeand/orthevalueofLipschitzconstantislarge.AndbyincorporatingtheproposedExpansionAlgorithm,theunconstrainedFAPLmethodisveryecientforsolvingunconstrainedCOPsespeciallywhenaproperestimationontheoptimalsolutionsetisnotavailable. 2.5.2TotalVariationBasedImageReconstructionInthissubsection,weapplytheFUSLmethodtosolvethenonsmoothtotalvariation(TV)basedimagereconstructionproblem: minu2RN1 2kAu)]TJ /F6 11.955 Tf 11.96 0 Td[(bk22+kukTV;(2{102)whereAisagivenmatrix,uisthevectorformoftheimagetobereconstructed,brepresentstheobserveddata,andkkTVisthediscreteTVsemi-normdenedby kukTV:=NXi=1kDiuk2;(2{103) 58

PAGE 59

whereDiu2R2isthediscretegradient(nitedierencesalongthecoordinatedirections)oftheithcomponentofu,andNisthenumberofpixelsintheimage.NotethatkukTVisconvexandnonsmooth.Oneoftheapproachestosolvethisproblemistoconsidertheassociateddualorprimal-dualformulationsof( 2{103 )basedonthedualformulationoftheTVnorm: kukTV=maxp2Yhp;Dui;whereY=fp=(p1;:::;pN)2R2N:pi2R2;kpik21;1iNg:(2{104)Consequently,wecanrewrite( 2{102 )asasaddle-pointproblem: minu2RNmaxp2Y1 2kAu)]TJ /F6 11.955 Tf 11.96 0 Td[(bk22+hp;Dui:(2{105)Notethat( 2{105 )isexactlytheformweconsideredintheUSLandFUSLmethodsifweset^g(y)=0.Specically,theprox-functionv(y)onYissimplychosenasv(y)=1 2kyk2inthesesmoothingtechniques.Inourexperiments,weconsidertwotypesofinstancesdependingonhowthematrixAisgenerated.Specically,fortherstcase,theentriesofAarenormallydistributed,whileforthesecondone,theentriesareuniformlydistributed.Forbothtypesofinstances,rst,wegeneratethematrixA2Rmn,thenchoosesometrueimagextureandconvertittoavector,andnallycomputebbyb=Axtrue+,whereistheGaussiannoisewithdistribution=N(0;).Wecomparethefollowingalgorithms:theacceleratedprimaldual(APD)method[ 5 ],Nesterov'ssmoothing(NEST-S)method[ 2 42 ],andtheFUSLmethod.Forourrstexperiment,thematrixAisrandomlygeneratedofsize4;09616;384withentriesnormallydistributedaccordingtoN(0;64),theimagextrueisa128128Shepp-LoganphantomgeneratedbyMATLAB.Moreover,weset=10)]TJ /F9 7.97 Tf 6.59 0 Td[(3andthestandarddeviation=10)]TJ /F9 7.97 Tf 6.58 0 Td[(3.ThevaluesofLipschitzconstantsareprovidedforAPDandNEST-S,andtheinitiallowerboundforFUSLissetto0.Werun300iterationsforeachalgorithm,andpresenttheobjectivevaluesofproblem( 2{102 )andtherelativeerrorsdenedbykxk)]TJ /F6 11.955 Tf 12.3 0 Td[(xtruek2=kxtruek2inFigure 2-1 .Inoursecondexperiment,thematrixAisrandomlygeneratedwithentries 59

PAGE 60

uniformlydistributedin[0;1].Weusea200200brainimage[ 43 ]asthetrueimagextrue,andsetm=20;000;=10;=10)]TJ /F9 7.97 Tf 6.59 0 Td[(2.Othersetupisthesameastherstexperiment,andtheresultsareshowninFigure 2-2 Figure2-1. TV-basedreconstruction(Shepp-Loganphantom) WemakesomeobservationsabouttheresultsinFigures 2-1 and 2-2 .Fortherstexperiment,thereisalmostnodierencebetweenAPDandNEST-S,butFUSLoutperformsbothofthemafter5secondsintermsofbothobjectivevalueandrelativeerror.ThesecondexperimentclearlydemonstratestheadvantageofFUSLforsolvingCOPswithlargeLipschitzconstants.TheLipschitzconstantofmatrixAinthisinstanceisabout2108,muchlargerthantheLipschitzconstant(about5:9)intherstexperiment.FUSLstillconvergesquicklyanddecreasestherelativeerrorto0:05inlessthan100iterations,whileAPDandNEST-Sconvergeveryslowlyandmorethan1;000iterationsarerequiredduetothelargeLipschitzconstants.ItseemsthatFUSLisnotsosensitivetotheLipschitzconstantsastheothertwo 60

PAGE 61

Figure2-2. TV-basedreconstruction(brainimage) methods.ThisfeatureofFUSLmakesitmoreecientforsolvinglarge-scaleCOPswhichoftenhavebigLipschitzconstants.Insummary,fortheTV-basedimagereconstructionproblem( 2{102 ),FUSLnotonlyenjoysthecompletelyparameter-freeproperty(andhencenoneedtoestimatetheLipschitzconstant),butalsodemonstratessignicantadvantagesforitsspeedofconvergenceanditssolutionqualityintermsofrelativeerror,especiallyforlarge-scaleproblems. 2.5.3PartiallyParallelImagingInthissubsection,wecomparetheperformanceoftheFUSLmethodwithseveralrelatedalgorithmsinreconstructionofmagneticresonance(MR)imagesfrompartialparallelimaging(PPI),tofurtherconrmtheobservationsonadvantagesoftheFUSLmethod.ThedetailedbackgroundanddescriptionofPPIreconstructioncanbefoundin[ 43 ].Thisimage 61

PAGE 62

reconstructionproblemcanbemodeledasminu2CnkXj=1kMFSju)]TJ /F6 11.955 Tf 11.96 0 Td[(fjk2+NXi=1kDiuk2;wheren=2(weconsidertwodimensionalcase),uistheN-vectorformofatwo-dimensionalcomplexvaluedimagetobereconstructed,kisthenumberofcoils(considerthemassensors)inthemagneticresonance(MR)parallelimagingsystem.F2Cnnisa2DdiscreteFouriertransformmatrix,Sj2Cnnisthesensitivitymapofthej-thsensor,andM2Rnnisabinarymaskdescribesthescanningpattern.NotethatthepercentagesofnonzeroelementsinMdescribesthecompressionratioofPPIscan.Inourexperiments,thesensitivitymapsfSjgkj=1areshowninFigure 2-3 ,theimagextrueisofsize512512showninFigures 2-4 and 2-5 ,andthemeasurementsffjgaregeneratedby fj=M(FSjxtrue+rej=p 2+imj=p )]TJ /F3 11.955 Tf 9.3 0 Td[(2);j=1;:::;k;(2{106)whererej;imjarethenoisewithentriesindependentlydistributedaccordingtoN(0;).Weconducttwoexperimentsonthisdatasetwithdierentacquisitionrates,andcomparetheFUSLmethodtoNEST-Smethod,andtheacceleratedlinearizedalternatingdirectionofmultipliers(AL-ADMM)withline-searchmethod[ 12 ].Forbothexperiments,set=310)]TJ /F9 7.97 Tf 6.59 0 Td[(2;=10)]TJ /F9 7.97 Tf 6.59 0 Td[(5,andffjgkj=1aregeneratedby( 2{106 ).Intherstexperiment,weuseCartesianmaskwithacquisitionrate14%:acquireimageinonerowforeverysuccessivesevenrows,whileforthesecondone,weuseCartesianmaskwithacquisitionrate10%:acquireimageinonerowforeverysuccessivetenrows.ThetwomasksareshowninFigure 2-3 .TheresultsoftherstandsecondexperimentsareshowninFigures 2-4 and 2-5 ,respectively.TheseexperimentsagaindemonstratetheadvantagesoftheFUSLmethodoverthesestate-of-the-arttechniquesforPPIimagereconstruction, 62

PAGE 63

Figure2-3. SensitivitymapandCartesianmasks Figure2-4. PPIimagereconstruction(acquisitionrate:14%) 63

PAGE 64

Figure2-5. PPIimagereconstruction(acquisitionrate:10%) 64

PAGE 65

CHAPTER3ACCELERATEDBUNDLELEVELMETHODSWITHINEXACTORACLE 3.1BackgroundInthischapter,weconsidertheconvexoptimizationproblem(COP): f:=minx2Xf(x);(3{1)whereXRnisconvexandclosed,f:X!Risaconvexfunctionsatisfying f(y))]TJ /F6 11.955 Tf 11.95 0 Td[(f(x))-222(hf0(x);y)]TJ /F6 11.955 Tf 11.96 0 Td[(xiM 1+ky)]TJ /F6 11.955 Tf 11.96 0 Td[(xk1+;8x;y2X:(3{2)forsomeM>0and2[0;1].Moreover,weassumethesolutionsetof( 3{1 )isnonempty.Insomelarge-scaleCOPsarisingfromrealapplications,theexactfunctionvalueand(sub)gradientoftheobjectivefunctiontobeoptimizedmaynotbeableortooexpensivetocompute.Therefore,thereisaneedtodevelopecientalgorithmsthatcanhandleinexactrst-orderinformationandwithoptimaliterationcomplexitiesguaranteed.Inthischapter,weassumetheexactrst-orderinformationoff()isnotavailableandthereexistsanoraclethatcouldcomputetheapproximatedfunctionvaluef(x)and(sub)gradientf0(x)off()foranygivenx2X.DuetothefactthattheBLtypemethodshavetheoptimaliterationcomplexitiesforsolvingnonsmooth,weaklysmoothandsmoothCOPsandhavetheadvantagesofunitizinghistoricalinformationandrestricted-memorytospeeduptheconvergenceinpractice,avarietyofinexactBLtypemethods[ 18 { 21 23 { 25 27 { 32 ]havebeendevelopedtodealwithinexactrst-orderinformationoftheobjectivefunction.AndtheinexactBLtypemethodshaveshowntheiradvantagesinreducingcomputationcostforsolvingsomespecicstochasticCOPs[ 19 24 30 44 45 ],andtheCOPs,wheretheobjectivefunctionisasummationofalargenumberofsubfunctions[ 28 ]orasolutionofanotheroptimizationproblem[ 44 ].However,allthesemethodsarebasedonnon-acceleratedBLtypemethodsandlimitedtosolvenonsmooth 65

PAGE 66

COP,andwecanclassifythesemethodsintoinexactproximalbundlemethodsandinexactBLmethods.Theworkin[ 33 ]and[ 46 ]opensawaytohandleinexactnesswiththeBLmethodandtheproximalbundlemethod,respectivelyin2000and2001.TheproximalbundlemethoddiersfromtheBLmethodonlyattheupdateoftheprox-center,theBLmethodsolvesaconstrainedQPproblem,whiletheproximalbundlemethodmovesthelinearconstraintstotheobjectiveofthesubproblemandgeneratesthenewiteratebysolvinganunconstrainedproblem.Consider( 3{1 )withf()beingLipschitz-continuouswithconstantM(nonsmooth).Theinexactoracleoutputstheapproximatedrst-orderinformationfuandguforanypointu2Xasfollows: 8>><>>:fu=f(u))]TJ /F6 11.955 Tf 11.96 0 Td[(ugu2Rnsuchthatf(x)fu+hgu;x)]TJ /F6 11.955 Tf 11.95 0 Td[(ui)]TJ /F6 11.955 Tf 19.27 0 Td[(gu;gu0;8x2X:(3{3)Theoraclescorrespondingtogu=0andgu>0arecalledthelowertypeanduppertype,respectively.Theexactoraclecorrespondstogu=u=0.Itisclearfrom( 3{3 )that,u+gu0foranyu,andforlowertypeoracle,theapproximatinghyperplanep(u;x):=fu+hgu;x)]TJ /F6 11.955 Tf 12.22 0 Td[(uistaysbelowthegraphoffforallx2X.Fortheoracleofuppertype,p(u;x)maycutosomeportionsofthegraphoff().Theworkin[ 24 ]developedBLtypemethods,whichareecientinthepresenceoforacleerrors,areversatileregardingthechoiceoftheprojectionpointxk.Tosavecomputationaleortateachevaluationpointx,twoparameters,whichareadescenttargetx2R[f+1gandanerrorboundx,areintroducedtocontroltheaccuracyforthelowertypeoracles.Supposetheoracleprovidesthefunctioninformationfx=f(x))]TJ /F6 11.955 Tf 12.17 0 Td[((x)with(x)0.Theactualerror(x)isinfactunknown.Underthefollowingconditionsonf: f(x)2[fx;fx+(x)];(x)x;wheneverfxx;(3{4) 66

PAGE 67

theinaccuracyiscontrolledonlyatthosefunctionvaluesfxthatreachesthedecenttargetx,thenthecorrespondingerrorisboundedbytheerrorboundx.Otherwise,whenthealgorithmevaluatesthefunctionat"bad"points,whichdonotyieldprogressintermsofminimizingtheobjective,therst-orderinformationcanberough.Theversatilityofsuchinexactoraclecanbeseeninthefollowinginstancesstudiedintheliteratures:a).Exact:x0andx+1;b).PartlyInexact:x0;x<+1,[ 24 25 ];c).Inexact:x+1,butx>0(possibleunknown)[ 19 22 24 46 47 ];d).AsymptoticallyExact:x+1andx!0alongtheiterates,referto[ 24 28 33 45 ].e).PartlyAsymptoticallyExact:x<+1andx!0[ 24 ].Themethodsdevelopedinthisworkdrivetheinexactnessparametertozero,thusensuringthatanexactsolutionisasymptoticallyfound.Withspeciallychosenparametersfortheerrorboundxandtargetx,theoptimaliterationcomplexityfornonsmoothCOPshasbeenproveduniformlyforallveinstancesa)-e)[ 24 ].TheirapproachcoversseveralknownexactandinexactBLmethods.Whenthefeasiblesetiscompact,someoftheiron-demandaccuracymethodshavethesamerateofconvergenceofexactBLvariantsknownintheliterature.InexactproximalbundlemethodshasalsobeenappliedtoreducecomputationtimefortheCOPs,wheretheobjectivefunctionisasumofmultiplesubfunctions.Themethoddevelopedin[ 28 ]takestheadvantageoftheseparablestructurebymakingabundleiterationafterhavingevaluatedonlyasubsetofthesubfunctions,insteadofallofthem.Byassumingtheerrorboundisboundedorvanishing,theconvergencetoasolutionwithcertainaccuracyoranexactsolutionisguaranteed,butnoiterationcomplexityisprovided.Theworkin[ 44 ]presentstwoinexactbundletypealgorithmstoincorporate(alreadyavailableorcheaptocompute)uncontrolledbundleinformation.Theinexactoraclewithuncontrolledaccuracyisrequiredonlytogiveunder-linearizationsoftheobjectivefunctionwithoutboundedinexactness.Theproposedmethodsarefeaturedatconstructingnewcutting-planemodelcallingthecoarseoracleonly,testingterminationbasedontestinginexactgapandinexactupperbound,andcallingneoracletocomputenextiterate.Byassumingtherearetwooraclesavailable:aneoracle,whichisexpensivebutwithaccuracyufor 67

PAGE 68

allu,andacoarseoraclewithuncontrolledaccuracybutcheap,twoinexactBLalgorithmsareproposed,andtheconvergencetoan-solutionareproved.TheproposedmethodscanbeeectivelyappliedtotheCOPs,wheretheobjectivefunctionisthesolutiontoanotheroptimizationproblem,suchasasaddlepointproblem,andthefollowingso-calledtwo-stagestochasticLP: f=minx2Xf(x)=:cTx+E[Q(x;)];withQ(x;)=supWTuq(h)]TJ /F6 11.955 Tf 11.96 0 Td[(Tx)Tu;(3{5)where:=(q;T;h)representsrandomdata.Withnitelymanyrealizationsi(i=1;:::;N)ofwithprobabilitypi,theexpectationisexpressedbyQ(x)=:E[Q(x;)]=PNi=1Q(x;i),whereQ(x;i)=supupi(hi)]TJ /F6 11.955 Tf 11.95 0 Td[(Tix)Tu;WTuqi:Thisproblemhavealsobeenstudiedin[ 19 24 30 44 45 ].TheinexactBLmethodhasbeenappliedtosolvethisproblemin[ 30 ]withaprovidedlowertypeinexactoraclewhichrequiresxjjforsomeandallj.TheinexactBLmethoddevelopedin[ 19 ]aimedatreducingthetimespentinsolvingthesecondsubproblemin( 3{5 ).Ingeneral,theideaofstochasticgradientmethodsistoreplacethegradientoverthewholedatasetbythegradientatasinglesample(oroverasmallmini-batchofsamples),andhence,per-iterationcomplexityismuchlower.In[ 19 ],instead,thealgorithmcomputesasubsetofthesubfunctionsQ(x;j)exactlyandreplacingtheinformationoftheremainingsubfunctionbyafastproceduretogiveapproximatedrst-orderinformationforf,theproximalbundlemethodcouldbeappliedtosolvethisproblemwhiletheerrorintherst-orderinformationisbounded.Thesetofthesubfunctionstobecomputedexactlyisselectedbyacollinearityselectionstrategyasfollows.Foraxx,selecti2f1;:::;Ng,andgroupalmostcollinearvectorshj)]TJ /F6 11.955 Tf 12.48 0 Td[(Tjxtohi)]TJ /F6 11.955 Tf 12.49 0 Td[(Tixinthesamegroup.Forthisgroup,deneacommonfeasiblesetbyWTuq,whereqistheaverageofallqjinthisgroup.Then,thesolutionuitosupu(hi)]TJ /F6 11.955 Tf 12.39 0 Td[(Tix)TuwithWTuqisusedtocomputeallthesubfunctionsQ(x;j)inthisgroup. 68

PAGE 69

AlltheseaforementionedinexactBLtypemethodsareparticularusefulforsolvingverylarge-scaleCOPs,forwhichtheexactoraclesareoftentooexpensivetocomputeorevennotabletocompute.However,theexistinginexactBLtypemethodsareonlyforsolvingnonsmoothCOPs.Recently,therehasbeenseveralstudiesaboutinexactacceleratedrst-ordermethodsdealingwithinexactoracles[ 36 48 { 52 ].Contrarytothenon-acceleratedinexactrst-ordermethods,[ 36 ]provesthattheacceleratedrst-ordermethodsmustnecessarilysuerfromerroraccumulation,whichbringsanewchallengefordevelopinginexactacceleratedBLtypemethods.Inthenextsection,werstlyproposetheinexactfastacceleratedprox-level(IFAPL)methodtosolvethegeneralsmoothCOPwithinexactoracle,thenextendittosolvestronglyconvexfunctionwithbetteriterationcomplexitiesbyintroducingtheIFAPLSmethod.Inthethirdsection,weconsideraclassofSPproblemwheretheMaxsubproblemcouldnotbeexactlysolved.ByincorporatingNesterov'ssmoothingtechnique[ 2 ]intotheaforementionedIFAPLmethod,weproposetheinexactfastuniformsmoothinglevel(IFUSL)methodtosolvethisclassofSPproblemandtheIFUSLSmethodtodealwiththecasethattheobjectivefunctionisstronglyconvex.Foreachmethod,weconsidertwocases,wheretheaccuracyoftheoracleischosenbytheuserortheaccuracyoftheoracleisxedprior.Foreachcase,boththedesiredaccuracyoftheapproximatesolutionthattheproposedmethodscouldachieveanditscorrespondingiterationcomplexitiesarediscussed. 3.2FastAcceleratedProx-LevelMethodswithInexactOracleInthissection,werstlyproposetheinexactfastacceleratedprox-level(IFAPL)methodtosolvetheball-constrainedCOPs,i.e.,thefeasiblesetXisanEuclideanballinRn,thenextendtheIFAPLmethodtosolveunconstrainedCOPinRnwithstronglyconvexobjectivefunctionandintroducetheIFAPLSmethod.BoththeIFAPLandIFAPLSmethodsareabletodealwithinexactrst-orderinformationoftheobjectivefunction.Weassumeforproblem( 3{1 ),thereexistsaninexactoraclethat,foranyx2X,theoraclegivestheinexactfunction 69

PAGE 70

valueand(sub)gradientoff()satisfying 0f(y))]TJ /F3 11.955 Tf 11.95 0 Td[((f(x)+hg(x);y)]TJ /F6 11.955 Tf 11.95 0 Td[(xi)L 2ky)]TJ /F6 11.955 Tf 11.96 0 Td[(xk2+;8x;y2X:(3{6)Wecallsuchaninexactoraclean(L;)-oracleforf().WewilldiscussthedesiredaccuracytheIFAPLandIFAPLSmethodscouldachieveandtheircorrespondingiterationcomplexitieswhenischosenbytheuserorxedprior. 3.2.1IFAPLMethodforSmoothBall-ConstrainedConvexOptimizationInthissubsection,weconsiderthefollowingsmoothball-constrainedCOP: f x;R:=minx2B(x;R)f(x);(3{7)whereB(x;R):=fx2Rn:kx)]TJ /F3 11.955 Tf 13.15 0 Td[(xkRg,f()issmoothwithLipchitzconstantL,andforx2B(x;R),theinexactrst-orderinformationoff()atx,i.e.,f(x)andg(x),satises( 3{6 ).ThemainideaoftheIFAPLmethodistoextendtheFAPLmethodin[ 53 ]todealwithinexactoracle.SimilartotheFAPLmethod,theIFAPLmethodbuildsthecuttingplanemodeloftheobjectivefunctionusingtheinexactrst-orderinformation,andgeneratesasequenceofupperandlowerboundsonf x;R.Bytighteningthegapbetweentheupperandlowerboundsuntilitislessthanthedesiredaccuracy,theIFAPLmethodconvergestoanapproximatesolution.Specically,theIFAPLmethodhasthestructureofouter-innerloops.Ineachiterationoftheouterloop,ifthegapbetweencurrentupperandlowerboundsislessthanthedesiredaccuracy,terminatetheIFAPLmethodandoutputtheapproximatesolution;otherwise,aninnerloop,referredastheIFAPLgapreductionprocedureGIFAPL,iscalledtoreducethecurrentgapbyaconstantfactor.Let'sstartwiththeIFAPLgapreductionprocedureGIFAPL.InordertoguaranteetheterminationanditerationcomplexityoftheIFAPLmethod,thestepsizesfkgneedtosatisfythefollowingcondition: 1=1;0
PAGE 71

TheIFAPLgapreductionprocedure:(x+;ub+;lb+)=GIFAPL(^x;ub;lb;R; x;;;) 0: Set f0=ub,l=lb+(1)]TJ /F6 11.955 Tf 11.95 0 Td[()ub,Q0=Rn,xu0=^x,x0= xandk=1. 1: Updatethecuttingplanemodel:xlk=(1)]TJ /F6 11.955 Tf 11.96 0 Td[(k)xuk)]TJ /F9 7.97 Tf 6.58 0 Td[(1+kxk)]TJ /F9 7.97 Tf 6.59 0 Td[(1; (3{8)h(xlk;x)=f(xlk)+hg(xlk);x)]TJ /F6 11.955 Tf 11.95 0 Td[(xlki; (3{9)Q k:=fx2Qk)]TJ /F9 7.97 Tf 6.59 0 Td[(1:h(xlk;x)lg: (3{10) 2: Updatetheprox-centerandlowerbound: xk=argminx2Q kd(x):=1 2kx)]TJ ET q 0.478 w 314.14 -157.37 m 320.8 -157.37 l S Q BT /F6 11.955 Tf 314.14 -164.19 Td[(xk2:(3{11)IfQ k=;orkxk)]TJ ET q 0.478 w 116.35 -192.37 m 123 -192.37 l S Q BT /F6 11.955 Tf 116.35 -199.19 Td[(xk>R:terminatewithx+=xuk)]TJ /F9 7.97 Tf 6.58 0 Td[(1;ub+= fk)]TJ /F9 7.97 Tf 6.59 0 Td[(1;lb+=l. 3: Updatetheupperbound:~xuk=(1)]TJ /F6 11.955 Tf 11.95 0 Td[(k)xuk)]TJ /F9 7.97 Tf 6.59 0 Td[(1+kxk; (3{12)xuk=(~xuk;iff(~xuk)+< fk)]TJ /F9 7.97 Tf 6.58 0 Td[(1;xuk)]TJ /F9 7.97 Tf 6.59 0 Td[(1;otherwise; (3{13)Set fk=f(xuk)+.If fkl+( f0)]TJ /F6 11.955 Tf 11.95 0 Td[(l):terminatewithx+=xuk;ub+= fk;lb+=lb. 4: ChooseanypolyhedralsetQksatisfyingQ kQk Qk,where Qk:=fx2Rn:hxk)]TJ ET q 0.478 w 261.19 -344.22 m 267.84 -344.22 l S Q BT /F6 11.955 Tf 261.19 -351.04 Td[(x;x)]TJ /F6 11.955 Tf 11.96 0 Td[(xki0g:(3{14)Setk=k+1andgotoStep 1 forsomeC1>0;C20.Twoinstancesoftheselectionoffkghavealreadybeengivenin[ 53 ],fortheproofwereferto[ 53 ]. 1. Setk=2=(k+1),k=1;2;:::,then( 3{15 )holdswithC1=2;C2=2. 2. Iffkgisrecursivelydenedby 1=1;2k+1=(1)]TJ /F6 11.955 Tf 11.96 0 Td[(k+1)2k;8k1;(3{16)then( 3{15 )issatisedwithC1=2;C2=2.BeforetheconvergenceanalysisofGIFAPL,weneedthefollowingtwolemmas,whichareverysimilartoLemmas3.2,3.3and3.4in[ 53 ]. 71

PAGE 72

Lemma6. LetfxkgbetheiteratesgeneratedbyGIFAPLbeforeitterminates,thenforanyK1,wehave 1 2KXk=1kxk)]TJ /F6 11.955 Tf 11.95 0 Td[(xk)]TJ /F9 7.97 Tf 6.58 0 Td[(1k2d(xK)=1 2kxK)]TJ ET q 0.478 w 304.64 -52.95 m 311.3 -52.95 l S Q BT /F6 11.955 Tf 304.64 -59.77 Td[(xk21 2R2:(3{17) Proof. By( 3{10 ),( 3{11 )and( 3{14 ),itisclearthatxk2Q kQk)]TJ /F9 7.97 Tf 6.59 0 Td[(1 Qk)]TJ /F9 7.97 Tf 6.59 0 Td[(1;82kK.Observerthatx12Q0;x0=argminx2Q0d(x),wehave hrd(xk)]TJ /F9 7.97 Tf 6.59 0 Td[(1);xk)]TJ /F6 11.955 Tf 11.96 0 Td[(xk)]TJ /F9 7.97 Tf 6.59 0 Td[(1i0;81kK:(3{18)Sinced(x)isstronglyconvex,wealsohave d(xk)d(xk)]TJ /F9 7.97 Tf 6.59 0 Td[(1)+hrd(xk)]TJ /F9 7.97 Tf 6.59 0 Td[(1);xk)]TJ /F6 11.955 Tf 11.95 0 Td[(xk)]TJ /F9 7.97 Tf 6.58 0 Td[(1i+1 2kxk)]TJ /F6 11.955 Tf 11.96 0 Td[(xk)]TJ /F9 7.97 Tf 6.59 0 Td[(1k2:(3{19)Combinethetwoinequalitiesabove,weget1 2kxk)]TJ /F6 11.955 Tf 12.2 0 Td[(xk)]TJ /F9 7.97 Tf 6.58 0 Td[(1k2d(xk))]TJ /F6 11.955 Tf 12.2 0 Td[(d(xk)]TJ /F9 7.97 Tf 6.58 0 Td[(1);81kK.Summingupforkfrom1toKweconclude( 3{17 ). Lemma7. DeneEf(l):=fx2B( x;R):f(x)lg,thefollowingstatementsholdforGIFAPL. a) IfEf(l)6=;,thenEf(l)Q kQk Qkforanyk1. b) IfQ k6=;,thenthesubproblem( 3{11 )hasauniquesolution.Moreover,ifGIFAPLterminatesatStep 2 ,thenwehavelf x;R. Proof. Noticethath(xlk;x)f(x)andEf(l)fh(xlk;x)lg,thentheproofissimilartothatofLemma3.2in[ 53 ]. Proposition3. InGIFAPL,ifthestepsizesfkgarechosensuchthat( 3{15 )holds,thenforanyK1,ifGIFAPLdoesnotterminateattheKthiteration,wehave fK)]TJ /F6 11.955 Tf 11.95 0 Td[(lC21LR2 2K2+2C2K:(3{20) 72

PAGE 73

Proof. Bythedenitionof fk,( 3{6 )and( 3{12 ),wehavefork1, fk=f(xuk)+f(~xuk)+f(~xuk)+ (3{21)f(xlk)+hg(xlk);~xuk)]TJ /F6 11.955 Tf 11.95 0 Td[(xlki+L 2k~xuk)]TJ /F6 11.955 Tf 11.96 0 Td[(xlkk2+2 (3{22)=(1)]TJ /F6 11.955 Tf 11.95 0 Td[(k)f(xlk)+hg(xlk);xuk)]TJ /F9 7.97 Tf 6.59 0 Td[(1)]TJ /F6 11.955 Tf 11.96 0 Td[(xlki+kf(xlk)+hg(xlk);xk)]TJ /F6 11.955 Tf 11.96 0 Td[(xlki (3{23)+L2k 2kxk)]TJ /F6 11.955 Tf 11.95 0 Td[(xk)]TJ /F9 7.97 Tf 6.58 0 Td[(1k2+2 (3{24)(1)]TJ /F6 11.955 Tf 11.96 0 Td[(k)f(xuk)]TJ /F9 7.97 Tf 6.58 0 Td[(1)+kl+L2k 2kxk)]TJ /F6 11.955 Tf 11.95 0 Td[(xk)]TJ /F9 7.97 Tf 6.59 0 Td[(1k2+2 (3{25)(1)]TJ /F6 11.955 Tf 11.96 0 Td[(k)(f(xuk)]TJ /F9 7.97 Tf 6.58 0 Td[(1)+)+kl+L2k 2kxk)]TJ /F6 11.955 Tf 11.95 0 Td[(xk)]TJ /F9 7.97 Tf 6.58 0 Td[(1k2+2 (3{26)=(1)]TJ /F6 11.955 Tf 11.95 0 Td[(k) fk)]TJ /F9 7.97 Tf 6.58 0 Td[(1+kl+L2k 2kxk)]TJ /F6 11.955 Tf 11.96 0 Td[(xk)]TJ /F9 7.97 Tf 6.59 0 Td[(1k2+2: (3{27)Thefourthinequalityfollowsfromtheconvexityoff(),( 3{10 )and( 3{11 ).Subtractlfrombothsidesoftheaboveinequality,weget fk)]TJ /F6 11.955 Tf 11.96 0 Td[(l(1)]TJ /F6 11.955 Tf 11.96 0 Td[(k)( fk)]TJ /F9 7.97 Tf 6.58 0 Td[(1)]TJ /F6 11.955 Tf 11.96 0 Td[(l)+L2k 2kxk)]TJ /F6 11.955 Tf 11.96 0 Td[(xk)]TJ /F9 7.97 Tf 6.59 0 Td[(1k2+2:(3{28)Then,dividingbothsidesby2kandsummingupforkfrom1toK,itimplies fK)]TJ /F6 11.955 Tf 11.95 0 Td[(lL2K 2KXk=1kxk)]TJ /F6 11.955 Tf 11.95 0 Td[(xk)]TJ /F9 7.97 Tf 6.59 0 Td[(1k2+22KKXk=11 2k:(3{29)Using( 3{15 )and( 3{17 ),wehave( 3{20 ). Inviewofthepropositionabove,wealreadyhaveanupperboundfor fK)]TJ /F6 11.955 Tf 13.09 0 Td[(l,andthisupperboundwillbeusedtofurtherestimatethenumberofiterationsperformedbyeachGIFAPLbeforetheterminationcriteriainStep 3 issatised.Similartotheacceleratedgradient-descentmethodsin[ 36 ],ifL;Randarexedprior, fK)]TJ /F6 11.955 Tf 12.68 0 Td[(lcannotbereducedtozeroduetothefactthatallacceleratedrst-ordermethodsmustnecessarilysuerfromtheerroraccumulationfromtheinexactoracle,butithasaminimumwhichdependsontheaforementionedparameters.Thenextpropositionshowsthat,iftheoracleaccuracyiswithin 73

PAGE 74

certainbound,theneachcallofGIFAPLcanreducethegapbetweentheupperandlowerboundsbyaconstantfactor. Proposition4. InGIFAPL,ifthestepsizesfkgarechosensuchthat( 3{15 )holds,;;andsatisfy :=(2 3)3=2 2C1C2Rp L;(3{30)wehavethefollowingstatementshold. a) ThenumberofiterationsperformedbyeachcallofGIFAPLisboundedby N():=C1 q 2 3p LR p +1:(3{31) b) IfGIFAPLterminates,then+q,where q:=maxf;1)]TJ /F3 11.955 Tf 11.96 0 Td[((1)]TJ /F6 11.955 Tf 11.96 0 Td[()g;:=ub)]TJ /F3 11.955 Tf 11.96 0 Td[(lb;+:=ub+)]TJ /F3 11.955 Tf 11.96 0 Td[(lb+:(3{32) Proof. ForPart a ,inviewof( 3{20 )and( 3{30 ),wehave fK)]TJ /F6 11.955 Tf 11.95 0 Td[(lC21LR2 2K2+2C2K:(3{33)TherighthandsideoftheaboveinequalityachievesitsminimumatK:=C21LR2 2C21 3=C1 q 2 3p LR p ;and fK)]TJ /F6 11.955 Tf 11.96 0 Td[(l3 2(2C21C22LR22)1=3:(3{34)NoticethattheterminationcriteriainStep 3 ofGIFAPLisequivalentto fk)]TJ /F6 11.955 Tf 12.85 0 Td[(l,therefore,GIFAPLeitherterminatesatStep 2 beforetheKthiterationorterminatesinStep 3 afteratmostKiterations.Hence,thenumberofiterationsperformedbyonecallofGIFAPLisboundedbyN()=K+1.ForPart b ,bythedenitionofxukin( 3{13 ),wecanseethatf fkgismonotonicallydecreasing,andub= f0,soub+ub.SinceGIFAPLeitherterminatesatStep 2 orStep 3 74

PAGE 75

weconsiderthesetwocasesseparately.IfGIFAPLterminatesinStep 2 attheKthiteration,thenwehavelb+=l=lb+(1)]TJ /F6 11.955 Tf 11.95 0 Td[()ub,whichimpliesub+)]TJ /F3 11.955 Tf 11.95 0 Td[(lb+ub)]TJ /F6 11.955 Tf 11.95 0 Td[(lb)]TJ /F3 11.955 Tf 11.95 0 Td[((1)]TJ /F6 11.955 Tf 11.96 0 Td[()ub=(ub)]TJ /F3 11.955 Tf 11.96 0 Td[(lb):IfGIFAPLterminatesinStep 3 attheKthiteration,thenub+= fKl+(ub)]TJ /F6 11.955 Tf 12.04 0 Td[(l),togetherwithlb+=lbandl=lb+(1)]TJ /F6 11.955 Tf 11.95 0 Td[()ub,wehaveub+)]TJ /F3 11.955 Tf 11.95 0 Td[(lb+l+(ub)]TJ /F6 11.955 Tf 11.95 0 Td[(l))]TJ /F3 11.955 Tf 11.96 0 Td[(lb=[1)]TJ /F3 11.955 Tf 11.96 0 Td[((1)]TJ /F6 11.955 Tf 11.95 0 Td[()](ub)]TJ /F3 11.955 Tf 11.96 0 Td[(lb):Combinethesetwocases,wehave+q. Next,weintroducetheIFAPLmethod,alsoreferredastheouterloopoftheIFAPLmethod.Themainstepoftheouterloopistocheckifthecurrentgapislessthanthedesiredaccuray.If>,itcallstheIFAPLgapreductionprocedureGIFAPLtofurtherreducethegap,otherwise,itoutputstheapproximatesolutionandterminatestheIFAPLmethod.EachcalltoGIFAPLrequirestheinputsofthecurrentupperboundub,lowerboundlb,thebestpointfoundsofar^xandthecurrentaccuracyoftheoracle. Algorithm6. Theinexactfastacceleratedprox-level(IFAPL)method 0: GivenB( x;R)andthedesiredaccuracy>0,choosetheinitialpointp02B( x;R),parameters;2(0;1)andtheinitiateoracleaccuracy0. 1: Letp12Argminx2B( x;R)h0(p0;x),lb1=h0(p0;p1);ub1=minff0(p0);f0(p1)g+0,choose^x1aseitherp0orp1suchthatf0(^x1)+0=ub1,sets=1. 2: Ifubs)]TJ /F3 11.955 Tf 11.95 0 Td[(lbs:terminatewithoutput^xs. 3: Set(^xs+1;ubs+1;lbs+1)=GIFAPL(^xs;ubs;lbs;R; x;;;s),wheresisthecurrentaccuracyoftheoracle. 4: Sets=s+1andgotoStep 2 .WeconsiderthecasewhereischosenbytheuserandestablishthecorrespondingiterationcomplexityoftheIFAPLmethodinthenexttheorem. 75

PAGE 76

Theorem6. Foranygiven>0,ifthestepsizesfkgarechosensuchthat( 3{15 )holds,andss;8s1,wheresisdenedin( 3{30 ).ThentheIFAPLmethodconvergestoan-solutionto( 3{7 ),andthefollowingstatementsholds. a) ThenumberofcallstoGIFAPLdoesnotexceed S:=max0;log1 q2LR2+20 +1:(3{35) b) ThenumberofiterationsperformedbytheIFAPLmethodisboundedby N:=S+p 3=2 1)]TJ 11.96 7.45 Td[(p qC1p LR p :(3{36) Proof. InviewofStep 1 oftheIFAPLmethodandf0(p1)f(p1),wehave 1=ub1)]TJ /F3 11.955 Tf 11.95 0 Td[(lb1f0(p1)+0)]TJ /F6 11.955 Tf 11.96 0 Td[(h0(p0;p1)L 2kp0)]TJ /F6 11.955 Tf 11.96 0 Td[(p1k2+202LR2+20:(3{37)ByProposition 4 ,s+1qs;8s1,weget s+1qs1;8s1:(3{38)Itisclearthatfsg1s=1isdecreasinggeometrically.Therefore,wecanassumethattheIFAPLmethodachievesan-solutionto( 3{7 )afterScallstoGIFAPL,i.e., S+1
PAGE 77

ofiterationsperformedbytheIFAPLmethodbySXs=1NsN(s)S+SXs=1p 3=2C1 p p LR p s (3{41)S+SXs=1p 3=2C1p LR p qS)]TJ /F10 5.978 Tf 5.76 0 Td[(s 2S+p 3=2 1)]TJ 11.95 7.45 Td[(p qC1p LR p ; (3{42)whereNsdenotesthenumberofiterationsperformedbythesthcalltoGIFAPL. Inmanypracticalproblems,theaccuracyoftheoraclethatprovidesinexactrst-orderinformationoftheobjectivefunctionmaybexed.Theworkin[ 36 ]showsthatallacceleratedalgorithmsmustsuerfromtheerroraccumulationandtheconvergencetoan-solutioncannotbeguaranteedinthiscase.Similarly,theIFAPLmethodcouldnotdecreasethegapbetweentheupperandlowerboundstozerobykeepcallingGIFAPL.Inthiscase,weneedtoconsiderthedesiredaccuracyofthesolutionthattheIFAPLmethodcouldachieve,andterminatetheIFAPLmethodifthegapissmallerthanthedesiredaccuracy.Notethattherighthandsideof( 3{20 )achievestheminimum :=3 2(22C21C22LR22)1=3(3{43)atK=K:=C21LR2 2C21=3:ByProposition 3 ,wehave fK)]TJ /F6 11.955 Tf 11.96 0 Td[(l,andtheterminationcriteria fkl+( f0)]TJ /F6 11.955 Tf 11.96 0 Td[(l)inStep 3 ofGIFAPLisequivalentto fk)]TJ /F6 11.955 Tf 12.07 0 Td[(l.Therefore,ifisxedprior,weneedtomakethefollowingmodicationsfortheIFAPLmethod.M1a.InGIFAPL,if=ub)]TJ /F3 11.955 Tf 12.84 0 Td[(lb ,thenchangetheterminationcriteria fkl+( f0)]TJ /F6 11.955 Tf 11.95 0 Td[(l)inStep 3 to fk)]TJ /F6 11.955 Tf 11.95 0 Td[(l;M1b.IntheIFAPLmethod,changeubs)]TJ /F3 11.955 Tf 11.95 0 Td[(lbsinStep 2 toubs)]TJ /F3 11.955 Tf 11.95 0 Td[(lbs,where :=1)]TJ /F6 11.955 Tf 11.96 0 Td[(+ =1)]TJ /F6 11.955 Tf 11.96 0 Td[(+ 3 2(22C21C22LR22)1=3:(3{44) 77

PAGE 78

ThefollowingtheoremestablishesthemainconvergencepropertiesoftheIFAPLmethodforthecasethattheaccuracyoftheoracleisxed. Theorem7. IntheIFAPLmethod,ifs=;8s0,andthestepsizesfkgarechosensuchthat( 3{15 )holds,thentheIFAPLmethodconvergestoan-solutionto( 3{7 ).Moreover,thenumberofcallstoGIFAPLisboundedby S:=max0;log1 q2LR2+2 +2;(3{45)andthenumberofiterationsperformedbytheIFAPLmethodisboundedby N:=~S1+p 3=2 1)]TJ 11.96 7.45 Td[(p qC1p LR p +C1LR2 2C21=3:(3{46) Proof. When> ,itisclearthat( 3{30 )holds,byProposition 4 ,thenumberofiterationsperformedbyeachcalltoGIFAPLisboundedbyN()denedin( 3{31 ),and+q.Combinewith( 3{37 ),wehavethenumberofcallstoGIFAPLisboundedbyS1:=log1 q1 log1 q2LR2+2 +1:NoticethatS1>,wecanboundthetotalnumberofiterationsperformedbytheIFAPLmethodbyN1S1Xs=1N(s)S1+S1Xs=1p 3=2C1 p p LR p s (3{47)S1+S1Xs=1p 3=2C1p LR p qS1)]TJ /F10 5.978 Tf 5.76 0 Td[(s 2S1+p 3=2 1)]TJ 11.95 7.44 Td[(p qC1p LR p : (3{48)When ,inviewofM1aand fK)]TJ /F6 11.955 Tf 12.82 0 Td[(l,wehavethenumberofiterationsperformedbyGIFAPLdoesnotexceedK.Moreover,foranykK,ifGIFAPLterminatesdueto fk)]TJ /F6 11.955 Tf 11.96 0 Td[(lissatised,wehave +=ub+)]TJ /F3 11.955 Tf 11.95 0 Td[(lb+l+)]TJ /F3 11.955 Tf 11.95 0 Td[(lb(1)]TJ /F6 11.955 Tf 11.95 0 Td[()(ub)]TJ /F3 11.955 Tf 11.96 0 Td[(lb)+:(3{49) 78

PAGE 79

Combinewith ,weconclude+1)]TJ /F4 7.97 Tf 6.58 0 Td[(+ ,i.e.,theIFAPLmethodterminatesasitachievesthedesiredaccuracy.Hence,thetotalnumberofcallstoGIFAPLisboundedbyS:=S1+1,andthetotalnumberofiterationsperformedbytheIFAPLmethodisboundedbyN:=N1+K. 3.2.2IFAPLSMethodforUnconstrainedStronglyConvexOptimizationInthissubsection,weextendtheIFAPLmethodtosolveunconstrainedstronglyconvexoptimizationproblem.WithsomemodicationsontheIFAPLmethod,theproposedIFAPLSmethodachievesbetteriterationcomplexitiesandaccuracyofapproximatesolutionthanthoseoftheIFAPLmethod.Specically,weconsiderthefollowingunconstrainedCOP: f=minx2Rnf(x);(3{50)wheref()satises( 3{2 )andisstronglyconvexwithmodulus,i.e., 2ky)]TJ /F6 11.955 Tf 11.96 0 Td[(xk2f(y))]TJ /F6 11.955 Tf 11.96 0 Td[(f(x))-222(hf0(x);y)]TJ /F6 11.955 Tf 11.95 0 Td[(xiL 2ky)]TJ /F6 11.955 Tf 11.96 0 Td[(xk2;8x;y2Rn;(3{51)where>0.Weassumethatthereexistsan(L;;)-oracleoff(),i.e.,foranyx2Rn,theinexactfunctionvaluef(x)and(sub)gradientg(x)off()satisfy 2ky)]TJ /F6 11.955 Tf 11.96 0 Td[(xk2f(y))]TJ /F6 11.955 Tf 11.96 0 Td[(f(x))-222(hg(x);y)]TJ /F6 11.955 Tf 11.95 0 Td[(xiL 2ky)]TJ /F6 11.955 Tf 11.95 0 Td[(xk2+;8x;y2Rn:(3{52)Notethat,inthissubsection,weconsidertheunconstrainedCOPinRn,andassumethataninitialvalidlowerestimateonfisgivenaslb1.Inmostproblems,suchaninitiallowerboundisavailableandeasytoget.Underthisassumption,foranyinitialpointp0,bythestronglyconvexityoff(),wehavekp0)]TJ /F6 11.955 Tf 12.49 0 Td[(xk22[f(p0))]TJ /F3 11.955 Tf 12.49 0 Td[(lb1]=.Thisindicatesthattheuniquesolutionto( 3{50 )denotedbyxiswithintheEuclideanballB(p0;p 2[f(p0))]TJ /F3 11.955 Tf 11.95 0 Td[(lb1]=).Therefore,wecanapplytheIFAPLmethodtosolvetheseball-constrainedCOPs.Moreover,observethatthegapbetweentheupperandlowerboundsonfisreducedaftereachgapreductionprocedure,inthebeginningofeachgapreductionprocedure,wecanadjustthe 79

PAGE 80

radiusandcenteroftheEuclideanball.Consequently,thesizeoftheEuclideanballshrinkagesandwecangetbetteriterationcomplexitiesforthestronglyconvexoptimizationproblem.Next,wedescribetheIFAPLSmethod,whichismodiedfromtheIFAPLmethodproposedintheprevioussubsection.Thedetailedmodicationsaregivenasfollows. TheIFAPLSgapreductionprocedure:(x+;ub+;lb+)=GIFAPLS(^x;ub;lb;r;;;)InIFAPLgapreductionprocedure,let x=^xandreplacetheprox-functiond(x)in( 3{11 )withkx)]TJ /F3 11.955 Tf 12.68 0 Td[(^xk2=2. Algorithm7. TheIFAPLSmethodInAlgorithm 6 ,changeSteps 0 1 and 3 tothefollowingsteps. 0: Choosetheinitiallowerboundlb1f,theinitialpointp02Rn,initialoracleaccuracy0,computeinitialupperboundub1=f0(p0)+0,choosethedesiredaccuracy>0andparameters;2(0;1). 1: Let^x1=p0ands=1. 3: Set(^xs+1;ubs+1;lbs+1)=~GIFAPLS(^xs;ubs;lbs;p 2(ubs)]TJ /F3 11.955 Tf 11.95 0 Td[(lbs)=;;;s),wheresisthecurrentaccuracyoftheoracle.Inthefollowingtheorems,wediscusswhensischosenbytheuserors=;8s1,thebestaccuracyoftheapproximatesolutionto( 3{50 )theIFAPLSmethodcouldachieveanditscorrespondingiterationcomplexities.Notethat,theonlydierencebetweenGIFAPLSandGIFAPListhatthefeasiblesets,i.e.,theEuclideanballs,aredierent,GIFAPLSusesB(^x;R)andGIFAPLusesB(x;R).DierentfromtheIFAPLmethod,theinputradiusandcenteroftheEuclideanballarechangedfordierentcallstoGIFAPLS.Basedontheseobservations,wecanseeLemma 6 ,Lemma 7 andProposition 3 alsoholdforGIFAPLS,andbypluggingR=q 2 intoProposition 4 ,wecangetasimilarpropositionforGIFAPLS. Proposition5. InGIFAPLS,ifthestepsizesfkgarechosensuchthat( 3{15 )holds,andtheparameters;;andsatisfy ^:=s 33 27C21C22L;(3{53) 80

PAGE 81

whereC1;C2aredenedin( 3{15 ),thenthefollowingstatementshold. a) ThenumberofiterationsperformedbyeachcalltoGIFAPLSdoesnotexceed ^N:=p 3C1s L +1:(3{54) b) IfGIFAPLSterminates,then+q,whereq;+andaredenedin( 3{32 ). Proof. ByreplacingRwithq 2 intheproofofProposition 4 ,Part a followsimmediately.Part b isthesameasPart b ofProposition 4 ThenexttheoremprovidestheconvergencepropertiesoftheIFAPLSmethodforsolving( 3{50 )whensischosenbytheuser. Theorem8. Forany>0,ifthestepsizesfkgarechosensuchthat( 3{15 )holds,ands^sforanys1,thentheIFAPLSmethodconvergestoan-solutionto( 3{50 ),andthefollowingstatementshold. a) ThenumberofcallstoGIFAPLSdoesnotexceed ^S:=log1 qub1)]TJ /F3 11.955 Tf 11.95 0 Td[(lb1 +1:(3{55) b) ThenumberofiterationsperformedbytheIFAPLSmethodisboundedby ^N:=^S p 3C1s L +1!:(3{56) Proof. Sinceforanys1,( 3{53 )issatised,byProposition 5 ,wehaveforeachcalltoGIFAPLS,thenumberofiterationsisboundedby^Nand+q.Therefore,thenumberofcallstoGIFAPLSisboundedby^S.Consequently,wehavethetotalnumberofiterationsperformedbytheIFAPLSmethodisboundedby ^N:=^N^S=^S p 3C1s L +1!:(3{57) 81

PAGE 82

Inviewoftheabovetheorem,iftheoracleerrordecreasesalongwithgapbetweentheupperandlowerboundsonf,theIFAPLSmethodconvergestoan-solutionforanygiven>0andachievestheiterationcomplexityO)]TJ /F3 11.955 Tf 5.48 -9.68 Td[(log1 .Next,weconsiderthecasethattheoracleaccuracyisxed,i.e.,s=;8s0.Theworkin[ 48 ]showsthat,iftheoracleaccuracyisxedandboundedby,theinexactacceleratedgradient-descentmethodcanconvergetoanapproximatesolutionwithaccuracyOq L .Similarly,theIFAPLSmethodcanalsoachievethisdesiredaccuracyandhasthesameiterationcomplexityproved.Ifs=;8s0,basedonthedesiredaccuracytheIFAPLSmethodcouldachieve,weneedtomakethefollowingadjustmentsfortheIFAPLSmethod.M2a.InGIFAPLS,ifub)]TJ /F3 11.955 Tf 12.15 0 Td[(lb^ ,changetheterminationcriteria fkl+( f0)]TJ /F6 11.955 Tf 12.15 0 Td[(l)inStep3to fk)]TJ /F6 11.955 Tf 11.96 0 Td[(l^.M2b.InIFAPLSmethod,changeubs)]TJ /F3 11.955 Tf 11.96 0 Td[(lbsinStep2toubs)]TJ /F3 11.955 Tf 11.96 0 Td[(lbs^,where ^:=s 27C21C22L ;^:=1)]TJ /F6 11.955 Tf 11.95 0 Td[(+ ^=1)]TJ /F6 11.955 Tf 11.96 0 Td[(+ s 27C1C22L 33:(3{58)Afterthesemodications,wehavethefollowingconvergencepropertiesfortheIFAPLSmethod. Theorem9. IntheIFAPLSmethod,ifs=;8s1,andthestepsizesfkgarechosensuchthat( 3{15 )holds,thentheIFAPLSmethodconvergestoan^-solutionto( 3{50 ).Moreover,thenumberofcallstoGIFAPLSisboundedby ^S:=log1 qub1)]TJ /F3 11.955 Tf 11.95 0 Td[(lb1 ^+2;(3{59)andthenumberofiterationsperformedbytheIFAPLSmethodisboundedby ^N:=^S p 3C1s L +1!:(3{60) Proof. When>^ ,itisclearthat( 3{53 )holds.ByProposition 5 ,thenumberofiterationsperformedbyeachcalltoGIFAPLSisboundedby^N,where^Nisdenedin( 3{54 ),and 82

PAGE 83

+q.Therefore,when>^ ,thenumberofcallstoGIFAPLSisboundedby^S1:=log1 qub1)]TJ /F3 11.955 Tf 11.96 0 Td[(lb1 ^+1;consequently,thenumberofiterationsperformedisboundedby^N1:=^S1^N:When^ ,byProposition 3 andR=q 2s ,wehave fK)]TJ /F6 11.955 Tf 11.96 0 Td[(lC21Ls K2+2C2KC21L^ K2+2C2K:(3{61)Therighthandsideoftheaboveinequalityachievestheminimum^atK=^K:=C21L^ C21 3=p 3C1s L :Therefore, f^K)]TJ /F6 11.955 Tf 12.67 0 Td[(l^,inviewofM2a,theterminationcriteriainStep3issatisedandGIFAPLSterminates.Similarto( 3{49 ),wecanalsoget+^,andtheIFAPLSmethodterminatesasitachievesthedesiredaccuracy^.Combinethesetwocases,wehavethetotalnumberofcallstoGIFAPLSisboundedby^S:=^S1+1andthetotalnumberofiterationsperformedbytheIFAPLSmethodisboundedby^N:=^N1+^K. 3.3InexactFastUniformSmoothingLevelMethodsforSaddle-PointProblemInthissection,weincorporateNerterov'ssmoothingtechniqueintotheIFAPLandIFAPLSmethodstosolveaclassofsaddle-point(SP)problemwithinexactoracle.WeproposetwonewinexactacceleratedBLtypemethods,namely,theinexactfastuniformsmoothinglevel(IFUSL)methodandIFUSLSmethod,thelatteroneisavariantoftheIFUSLmethodforsolvingtheunconstrainedSPprobleminRnwheretheobjectivefunctionisstronglyconvex.WeassumetheMaxsubprobleminthisclassofSPproblemcannotbesolvedexactly,thustheexactrst-orderinformationoftheobjectivefunctionisnotavailable.BoththeIFUSLand 83

PAGE 84

IFUSLSmethodsaredesignedtodealwithinexactoraclesforsolvingthisclassofSPproblemswithconvergencetoanapproximatesolutionwithincertainaccuracyguaranteed. 3.3.1IFUSLMethodforBall-ConstrainedSaddle-PointProblemInthissubsection,weconsiderthefollowingclassofSPproblem: fx;R=minx2B(x;R)f(x):=^f(x)+F(x);(3{62)where^fissmoothwithLipschitzconstantL^f>0,i.e.,^f(y))]TJ /F3 11.955 Tf 14.5 3.16 Td[(^f(x))-222(hr^f(x);y)]TJ /F6 11.955 Tf 11.95 0 Td[(xiL^f 2ky)]TJ /F6 11.955 Tf 11.95 0 Td[(xk2; (3{63)and F(x):=maxy2YfhAx;yi)]TJ /F3 11.955 Tf 19.68 0 Td[(^g(y)g:(3{64)whereYRmisaclosedandconvex,^g:Y!RisasimplefunctionandA:Rn!Rmalinearoperator.Noticethat( 3{62 )isacompositeoptimizationproblemwheretheobjectivefunctionisthesumofasmoothfunctionandanonsmoothfunction,thusf()isnonsmooth.Bytakingadvantageofthespecialstructureof( 3{64 ),theworkin[ 2 ]rstintroducesthesmoothingtechniquetoapproximateF(x)uniformlybyasequenceofsmoothfunctions.Thesmoothingalgorithmin[ 2 ]improvestheiterationcomplexityforsolving( 3{62 )fromO(1 2)toO(1 ).Moreover,[ 26 ]and[ 53 ]furtherincorporatesthissmoothingtechniqueintoacceleratedBLtypemethodsandachievesthesameoptimaliterationcomplexity.Inparticular,letv:Y!Rbeastronglyconvexfunctionwithmodulusv,andcv:=argminy2Yv(y),byincorporatingthesmoothingtechniquein[ 2 ],weapproximateF(x)byasequenceofsmoothfunctions:F(x):=maxy2Y(x;y):=hAx;yi)]TJ /F3 11.955 Tf 19.67 0 Td[(^g(y))]TJ /F6 11.955 Tf 11.96 0 Td[(V(y); (3{65)where>0iscalledthesmoothingparameter,andV()istheBregmandivergenceassociatedwithv()denedby: V(y):=v(y))]TJ /F6 11.955 Tf 11.95 0 Td[(v(cv))-221(hrv(cv);y)]TJ /F6 11.955 Tf 11.95 0 Td[(cvi:(3{66) 84

PAGE 85

[ 2 ]showsthatthegradientofF()givenbyrF(x)=AyxisLipschitz-continuouswithconstant L:=kAk2=(v);(3{67)wherekAktheoperatornormofA,Aistheadjointoperator,yx2Yistheuniquesolutionto( 3{65 ).F(x)uniformlyconvergestoF(x)ifapproacheszero,andthe\closeness"ofF()toF()dependslinearlyonthesmoothingparameter,i.e., F(x)F(x)F(x)+Dv;Y;8x2B(x;R);(3{68)where Dv;Y:=maxy;z2Yfv(y))]TJ /F6 11.955 Tf 11.96 0 Td[(v(z))-222(hrv(z);y)]TJ /F6 11.955 Tf 11.95 0 Td[(zig:(3{69)Consequently,letf(x):=^f(x)+F(x); (3{70)wehavef(x)f(x)f(x)+Dv;Y: (3{71)Observethat,bysmoothingf()anddecreasingtozeroalongiterations,wecaniterativelysolveftogetanapproximatesolutionto( 3{62 ).However,inmanypracticalproblems,thesubproblem( 3{65 )cannotbesolvedexactly,whichmakestheexactrst-orderinformationofFandfarenotavailable.Therefore,itisimportanttodevelopecientalgorithmsthatcanusesuchinexactrst-orderinformationtosolve( 3{62 ).Itisclearthattheaccuracyoftherst-orderinformationofFandfdirectlydependsontheaccuracyoftheapproximatesolutionto( 3{65 ).[ 36 ]hasdiscussedthreerelatedformstodenetheaccuracyoftheapproximatesolutionto( 3{65 ).Inthissection,weuseoneofthem.Letyxbeanapproximatesolutionto( 3{65 ),theaccuracyoftheapproximatesolutionisdenedby yx:=(x;yx))]TJ /F3 11.955 Tf 11.95 0 Td[((x;yx):(3{72) 85

PAGE 86

Wemakethefollowingassumption:AssumptionC1:given>0,foranyx2B(x;R),0,anapproximatesolutionyxto( 3{65 )canbecomputedsuchthatyx.[ 36 ]furtherprovesthat,underthisassumption,forany>0,wehave F(z)+hG(z);x)]TJ /F6 11.955 Tf 9.3 0 Td[(ziF(x)=(x;yx)F(z)+hG(z);x)]TJ /F6 11.955 Tf 9.3 0 Td[(zi+Lkx)]TJ /F6 11.955 Tf 9.3 0 Td[(zk2+2;8z2B(x;R):(3{73)whereF(z):=(z;yz);G(z):=r1(z;yz).Forthesakeofsimplicity,inthissectionweassumethattheexactrst-orderinformationof^fisavailableandprovidedbyanexactoracle,andFsatisestheaboveAssumptionC1,werefertosuchaninexactoracleforf()asan(2;L)-oracle,i.e.,forany0, f(x)f(x)f(x)+;8x2B(x;R);(3{74)wheref(x)=f0(x);f(x)=f0(x),andforany>0,f(x)satises( 3{6 ),i.e., 0f(y))]TJ /F3 11.955 Tf 11.95 0 Td[((f(x)+hg(x);y)]TJ /F6 11.955 Tf 11.95 0 Td[(xi)L 2ky)]TJ /F6 11.955 Tf 11.96 0 Td[(xk2+2;8x;y2X:(3{75)where L:=L^f+2L;f(x):=^f(x)+F(x);g(x):=r^f(x)+G(x):(3{76)Next,weproposetheIFUSLmethodthatcoulddealwiththeaforementioned(2;L)-oracletosolve( 3{62 ).TheIFUSLmethodstillhasthestructureofouter-innerloops,theinnerloop,alsoreferredastheIFUSLgapreductionprocedureisgivenbelow.InordertoguaranteetheterminationofGIFUSLandtheiterationcomplexityoftheIFUSLmethod,thestepsizesfkgneedtosatisfythefollowingcondition: 1=1;0
PAGE 87

TheIFUSLgapreductionprocedure:(x+;D+;ub+;lb+)=GIFUSL(^x;D;ub;lb;R; x;;;) 0: Letk=1, f0=ub;l=lb+(1)]TJ /F6 11.955 Tf 11.96 0 Td[()ub,Q0=Rn,xu0=^x,x0= xand :=( f0)]TJ /F6 11.955 Tf 11.96 0 Td[(l) 2D:(3{77) 1: Updatethecuttingplanemodel:setxlkandQ kto( 3{8 )and( 3{10 ),respectively,h(xlk;x)=h(xlk;x):=f(xlk)+g(xlk);x)]TJ /F6 11.955 Tf 11.95 0 Td[(xlk: (3{78) 2: Updatetheprox-centerandthelowerbound:setxkto( 3{11 ).IfQ k=;orkxk)]TJ ET q 0.478 w 426.61 -140.71 m 433.26 -140.71 l S Q BT /F6 11.955 Tf 426.61 -147.53 Td[(xk>R:terminatewithoutputsx+=xuk)]TJ /F9 7.97 Tf 6.59 0 Td[(1;D+=D;ub+= fk)]TJ /F9 7.97 Tf 6.59 0 Td[(1;lb+=l. 3: UpdatetheupperboundandtheestimateofDv;Y:set~xukandxukto( 3{12 )and( 3{13 ),respectively,andlet fk=f(xuk)+.Checkthefollowingconditions 3a If fkl+( f0)]TJ /F6 11.955 Tf 10.75 0 Td[(l):terminatewithoutputsx+=xuk;D+=D;ub+= fk;lb+=lb; 3b If fk>l+( f0)]TJ /F6 11.955 Tf 11.8 0 Td[(l)andf(xuk)+2l+ 2( f0)]TJ /F6 11.955 Tf 11.81 0 Td[(l):terminatewithoutputsx+=xuk;D+=2D;ub+= fk;lb+=lb. 4: Dene QkandchooseQkasthesameasStep 4 inGIFAPL.Letk=k+1andgotoStep 2 forsomeC1;C2>0.Ifk=2=(k+1);k1,thenwehaveC1=2;C2=4.Iffkgaredenedby( 3{16 ),then( 3{79 )holdswithC1=2;C2=4.WegiveafewremarksaboutGIFUSL.Firstly,asf()isreplacedwithf,thecuttingplanemodelisbuiltonfbyusingtheinexactrst-orderinformationoff.Secondly,thesmoothingparameterisafunctiondependsonD;andthecurrentgap,whereDisthecurrentestimateonDv;Yin( 3{69 ),and f0)]TJ /F6 11.955 Tf 13.01 0 Td[(l=.Lastly,asthesameasGIFAPL,GIFUSLusesasetoflocalizersfQ k;Qk; QkgtocontrolthenumberoflinearconstraintinQk.Moreover,duetothelowdimensionofthedualproblemof( 3{11 ),( 3{11 )couldbeexactlysolved,forthedetailswereferto[ 53 ].SimilartotheUSLandFUSLmethods,inviewoftheterminationcriteriaandoutputsofGIFUSL,wehavethefollowingimportantobservations. Lemma8. InGIFUSL,thefollowingstatementshold. a) IfGIFUSLterminatesatStep 2 orStep 3 a ,then+q,whereq;and+aredenedin( 3{32 ). 87

PAGE 88

b) IfGIFUSLterminatesatStep 3 b ,thenD 2( f0)]TJ /F6 11.955 Tf 11.95 0 Td[(l) ( f0)]TJ /F6 11.955 Tf 11.95 0 Td[(l)=2D=D:(3{81)Therefore,bythedenitionofD+inStep 3 b ,wehaveD+<2Dv;Y. Noticethat,foranyGIFUSL,thesmoothingparameterischoseninthebeginningandxedduringtheinneriterations,soGIFUSLcouldbeviewedasapplyingGIFAPLonthesmoothfunctionf.SimilartoProposition 5 ,wehavethefollowingresultforGIFUSL. Proposition6. InGIFUSL,ifthestepsizesfkgarechosensuchthat( 3{79 )holds,thenforanyK1,ifGIFUSLhasnotterminatedintheKthiteration,wehave f(xuK)+2)]TJ /F6 11.955 Tf 11.96 0 Td[(lC21LR2 2K2+C2K;(3{82)whereLisdenedin( 3{76 ). Proof. NoticethatfissmoothwithLipschitzconstantL,andtheinexactrst-orderinformationoffsatisesthe(2;L)-oracle.ByProposition 5 ,wehave8k1,f(xuk)+2f(~xuk)+2f(xlk)+hg(xlk);~xuk)]TJ /F6 11.955 Tf 11.95 0 Td[(xlki+L 2k~xuk)]TJ /F6 11.955 Tf 11.96 0 Td[(xlkk+4 (3{83)(1)]TJ /F6 11.955 Tf 11.96 0 Td[(k)f(xuk)]TJ /F9 7.97 Tf 6.59 0 Td[(1)+kl+L2k 2kxk)]TJ /F6 11.955 Tf 11.96 0 Td[(xk)]TJ /F9 7.97 Tf 6.59 0 Td[(1k2+4 (3{84)(1)]TJ /F6 11.955 Tf 11.96 0 Td[(k)(f(xuk)]TJ /F9 7.97 Tf 6.59 0 Td[(1)+2)+kl+L2k 2kxk)]TJ /F6 11.955 Tf 11.96 0 Td[(xk)]TJ /F9 7.97 Tf 6.59 0 Td[(1k2+(3+k): (3{85)Therstandsecondinequalitiesfollowfrom( 3{74 )and( 3{75 ),thethirdonefollowsfromtheconvexityoff,( 3{8 )and( 3{12 ).Thelastonefollowsfrom( 3{74 ).Subtractlfrombothsidesoftheaboveinequality,thendivideby2kandsummingupforkfrom1toK,byLemma 88

PAGE 89

6 and( 3{79 ),wehave f(xuK)+2)]TJ /F6 11.955 Tf 11.96 0 Td[(lL2K 2KXk=1kxk)]TJ /F6 11.955 Tf 11.96 0 Td[(xk)]TJ /F9 7.97 Tf 6.59 0 Td[(1k2+2KKXk=13+k 2kC21LR2 2K2+C2K:(3{86) WearenowreadytopresenttheouterloopoftheIFUSLmethod.Themainstepoftheouterloopistocalltheinnerloop,thegapreductionprocedureGIFUSL,toreducethegapbetweentheupperandlowerboundsuntilthegapislessthanthedesiredaccuracy. Algorithm8. Theinexactfastuniformsmoothinglevel(IFUSL)method 0: GivenB( x;R),choosetheinitialpointp02B( x;R),stronglyconvexfunctionv()in( 3{65 )and( 3{66 ),theinitialestimateD1onDv;Yin( 3{69 ),thedesiredaccuracy>0andparameters;2(0;1). 1: Letp12Argminx2B( x;R)h0(p0;x),lb1=h0(p0;p1);ub1=minff(p0);f(p1)g,choose^x1aseitherp0orp1suchthatf(^x1)=ub1,sets=1. 2: Ifubs)]TJ /F3 11.955 Tf 11.95 0 Td[(lbs:terminateandoutput^x. 3: Set(^xs+1;Ds+1;ubs+1;lbs+1)=GIFUSL(^xs;Ds;ubs;lbs;R; x;;;s),wheresisthecurrentaccuracyoftheoracle. 4: Lets=s+1andgotoStep 2 .SincetheIFUSLmethodcouldbeviewedasavariantoftheIFAPLmethodwiththesmoothingtechniqueincorporated,theIFUSLmethodalsosuersfromtheerroraccumulation.InviewofProposition 6 ,ifisxedprior,thentherighthandsideof( 3{82 )doesnotconvergetozeroaskgrows.Therefore,similartotheIFAPLmethod,wediscussthebestdesiredaccuracyandthecorrespondingiterationcomplexitiesoftheIFUSLmethodwhentheoracleaccuracyoftheobjectivefunctionischosenbytheuserorxedprior.ThefollowingpropositionestablishesaupperboundforthenumberofiterationsperformedbyeachcalltoGIFUSL. 89

PAGE 90

Proposition7. InGIFUSL,ifthestepsizesfkgarechosensuchthat( 3{79 )holds,andtheparameters;;andsatisfy :=(1 3)3=2 C1C2Rp L:(3{87)whereLisdenedin( 3{76 ).Then,thenumberofiterationsperformedbyeachcallofGIFUSLdoesnotexceed N(;D):=p 3C1Rp L^f p +2p 3C1RkAkp D p v+1:(3{88) Proof. ByProposition 6 and( 3{87 ),wehave f(xuK)+2)]TJ /F6 11.955 Tf 11.96 0 Td[(lC21LR2 2K2+C2K;(3{89)TherighthandsideoftheaboveinequalityachievestheminimumatK=K:=C21LR2 C21 3,and f(xuK)+2)]TJ /F6 11.955 Tf 11.95 0 Td[(lC21LR2 2K2+C2K)]TJ /F3 11.955 Tf 11.95 0 Td[(23 2(C21C22LR22)1=3)]TJ /F3 11.955 Tf 11.95 0 Td[(21 2:(3{90)Since 2( f0)]TJ /F6 11.955 Tf 11.96 0 Td[(l)=1 2,theaboveinequalityimplies f(xuK)+2)]TJ /F6 11.955 Tf 11.96 0 Td[(l 2( f0)]TJ /F6 11.955 Tf 11.95 0 Td[(l):(3{91)Therefore,theterminationcriteriainStep 3 b issatisedafteratmostKiterations,thenGIFUSLterminates.By( 3{67 ),( 3{76 )and( 3{77 ),wehave L=kAk2 v=2DkAk2 v:(3{92)SothenumberofiterationsperformedbyGIFUSLdoesnotexceedK=p 3C1Rs L p 3C1R p (p L^f+p 2L)=p 3C1Rp L^f p +2p 3C1RkAkp D p v: (3{93) 90

PAGE 91

ByLemma 8 ,eachcalltoGIFUSLeitherreducesthecurrentgapbetweentheupperandlowerboundsbyaconstantq,i.e.,+q,ordoubletheestimateonDv;Y,i.e.,D+=2D.Forthesakeofsimplicity,weclassifythecallstoGIFUSLintosignicantandnon-signicantphases.IfGIFUSLterminatesatStep 2 orStep 3 a ,wecallitasignicantphase;ifGIFUSLterminatesatStep 3 b ,wecallitanon-signicantphase.Next,weconsiderthecasethattheoracleaccuracycouldbechosenbytheuser,andstudytheconditionsfortheoracleaccuracyinordertoguaranteetheIFUSLmethodconvergestoan-solution.Byestimatingthenumbersofiterationsperformedbysignicantandnon-signicantphases,wehavethefollowingtheorem. Theorem10. Foranygiven>0,ifthestepsizesfkgarechosensuchthat( 3{79 )holds,andss;8s1,whereisdenedin( 3{87 ),thentheIFUSLmethodconvergestoan-solutionto( 3{62 ),andthefollowingstatementshold. a) ThetotalnumberofcallstoGIFUSLisboundedby S:=S1+S2;(3{94)whereS1andS2aredenedin( 3{99 )and( 3{100 ),respectively. b) ThetotalnumberofiterationsperformedbytheIFUSLmethodisboundedby N:=N1+N2;(3{95)whereN1andN2aredenedin( 3{101 )and( 3{102 ),respectively. Proof. ForPart a ,weestimatethenumberofsignicantandnon-signicantphasesseparately.InviewofStep 0 andkp0)]TJ /F6 11.955 Tf 12.06 0 Td[(p1k2R,byLemma8in[ 26 ],wehavethefollowingestimateontheinitialgap:1:=ub1)]TJ /F3 11.955 Tf 11.96 0 Td[(lb1f(p1))]TJ /F6 11.955 Tf 11.95 0 Td[(h0(p0;p1) (3{96)[F(p0))]TJ /F6 11.955 Tf 11.96 0 Td[(F(p1))-222(hF0(p1);p0)]TJ /F6 11.955 Tf 11.96 0 Td[(p1i]+h^f(p0))]TJ /F3 11.955 Tf 14.51 3.16 Td[(^f(p1))]TJ /F13 11.955 Tf 11.96 13.27 Td[(D^f0(p1);p0)]TJ /F6 11.955 Tf 11.95 0 Td[(p1Ei (3{97)4p 2RkAkr Dv;Y v+2R2L^f: (3{98) 91

PAGE 92

ByLemma 8 ,foranynon-signicantphase,wehaveD+=2D,andD+<2Dv;Y,sothenumberofnon-signicantphasesisboundedby S1:=maxlog2Dv;Y D1;0+1:(3{99)Similarly,foranysignicantphase,wehave+q,whereqisdenedin( 3{32 ),whichimpliesthenumberofsignicantphasesisboundedby S2:=max0;log1 q(1 )+1max8<:0;log1 q0@4p 2RkAkq Dv;Y v+2R2L^f 1A9=;+1:(3{100)Hence,thetotalnumberofcallstoGIFUSLisboundedbyS=S1+S2.ForPart b ,similartotheproofofTheorem3.8in[ 53 ],wecanestimatethenumberofiterationsperformedbysignicantandnon-signicantphasesseparately.Letfm1;m2;:::;mS1gandfn1;n2;:::;nS2gbetheindicesofnon-signicantandsignicantphases,respectively,i.e.,ifs=mk;1kS1,thenthesthcalltoGIFUSLisanon-signicantphase;ifs=nk;1kS2,thenthesthcalltoGIFUSLisasignicantphase.Denote~D:=maxfD1;2Dv;Yg,noticethat,foranymk,thenumberofiterationsperformedbythemk-thcalldoesnotexceedN(mk;Dmk)andmk>;Dmk+1=2Dmk;foranynk,thenumberofiterationsperformedbythenk-thcalldoesnotexceedN(nk;Dnk)andnk+1qnk;Dnk<~D.Therefore,thetotalnumberofiterationsperformedbynon-signicantphasesisboundedbyN1:=S1Xk=1N(mk;Dmk)S1Xk=1N;~D=2S1)]TJ /F4 7.97 Tf 6.58 0 Td[(kS1 p 3C1Rp L^f p +1!+2p 3C1RkAkp ~D p vS1Xk=12)]TJ /F12 5.978 Tf 7.78 4.32 Td[((S1)]TJ /F10 5.978 Tf 5.75 0 Td[(k) 2S1 p 3C1Rp L^f p +1!+(4p 3+2p 6)C1RkAkp ~D p v: (3{101) 92

PAGE 93

ThetotalnumberofiterationsperformedbysignicantphasesisboundedbyN2:=S2Xk=1N(nk;Dnk)S2Xk=1N(=qS2)]TJ /F4 7.97 Tf 6.59 0 Td[(k;~D)S2+p 3C1Rp L^f p S2Xk=1qS2)]TJ /F10 5.978 Tf 5.76 0 Td[(k 2+2p 3C1RkAkp ~D p vS2Xk=1qS2)]TJ /F4 7.97 Tf 6.58 0 Td[(kS2+p 3C1Rp L^f (1)]TJ 11.96 7.45 Td[(p q)p +2p 3C1RkAkp ~D (1)]TJ /F6 11.955 Tf 11.96 0 Td[(q)p v: (3{102)Hence,thetotalnumberofiterationsperformedbytheIFUSLmethodisboundedbyN=N1+N2: Next,wediscussthedesiredaccuracytheIFUSLmethodcouldachieveifs=foranys0.( 3{71 )indicatesthattheclosenessbetweenfandfdependsonDv;Y.AndintheIFUSLmethod,inordertomaintainthenicefeaturethatnoinputoffunctionrelatedparametersisrequired,weusetheestimateDtoreplaceDv;Y.However,theestimateDapproachesDv;YdynamicallyandwecanonlyguaranteeDv;YD<2Dv;YwhentheIFUSLmethodconvergestoan-solutionwithclosetozero.Noticethat,ifL;Randarexed,therighthandsideof( 3{82 )doesnotconvergeasKincreases.Therefore,theIFUSLmethodalsosuersfromtheerroraccumulationandcannotconvergeto-solution,asaresult,thealgorithmcannotguaranteethatDisclosetoDv;Y.Basedonthesetworeasons,whenisgivenandxed,theIFUSLmethodneedstomakethefollowingadjustments:M3a.GivenDv;YastheinputoftheIFUSLmethod,i.e.,Ds=Dv;Y;8s1inStep 2 oftheIFUSLmethod.M3b.Changetheterminationcriteriaubs)]TJ /F3 11.955 Tf 11.95 0 Td[(lbsinStep 2 toubs)]TJ /F3 11.955 Tf 11.96 0 Td[(lbs,where :=1 maxn3(2C21C22R2L^f2)1=3;(63C21C22R2DkAk22=v)1=4o:(3{103)Afterthesemodications,wehavethefollowingconvergenceresult. 93

PAGE 94

Theorem11. IntheIFUSLmethod,ifs=;8s1,andthestepsizesfkgarechosensuchthat( 3{79 )holds,thentheIFUSLmethodconvergestoan-solutionto( 3{62 ),andthenumberofcallstoGIFUSLdoesnotexceed S:=max8<:0;log1 q0@4p 2RkAkq Dv;Y v+2R2L^f 1A9=;+1;(3{104)thenumberofiterationsperformedbytheIFUSLmethoddoesnotexceed N:=S+p 3C1Rp L^f (1)]TJ 11.95 7.45 Td[(p q)p +2p 3C1RkAkp Dv;Y (1)]TJ /F6 11.955 Tf 11.96 0 Td[(q)p v:(3{105) Proof. If=ub)]TJ /F3 11.955 Tf 12 0 Td[(lb<,theIFUSLmethodterminatesastheconditioninM3bissatised.Soweonlyneedtoconsiderthecasewhen.By( 3{67 ),( 3{76 )and( 3{77 ),wehaveC21C22R2L2=C21C22R22(Lf+4DkAk2 v):Inviewof( 3{103 ),wealsohaveC21C22R22Lf()3 54andC21C22R224DkAk2 v()3 54:Therefore,( 3{87 )holds,andbyProposition 7 ,thenumberofiterationsperformedbyeachcalltoGIFUSLisboundedbyN(;Dv;Y).Moreover,sinceD=Dv;Y,GIFUSLcannotterminateatStep b ,byLemma 8 ,allcallstoGIFUSLaresignicantphasesandhave+q.Combinewith( 3{96 ),wehavethenumberofcallstoGIFUSLdoesnotexceedS.Consequently,thenumberofiterationsperformedbytheIFUSLmethoddoesnotexceedSXs=1N(s;Dv;Y)SXk=1N(=qS)]TJ /F4 7.97 Tf 6.59 0 Td[(k;Dv;Y) (3{106)S+p 3C1Rp L^f p SXk=1qS)]TJ /F10 5.978 Tf 5.76 0 Td[(k 2+2p 3C1RkAkp Dv;Y p vSXk=1qS)]TJ /F4 7.97 Tf 6.58 0 Td[(k (3{107)S+p 3C1Rp L^f (1)]TJ 11.96 7.45 Td[(p q)p +2p 3C1RkAkp Dv;Y (1)]TJ /F6 11.955 Tf 11.95 0 Td[(q)p v: (3{108) 94

PAGE 95

3.3.2IFUSLSMethodforStronglyConvexSaddle-PointProblemInthissubsection,weextendtheIFUSLmethodtosolveunconstrainedstronglyconvexoptimizationprobleminRn,wheretheobjectivefunctioncontainsaSPproblem.Consider f=minx2Rnf(x):=^f(x)+F(x);(3{109)where^f(x)andF(x)aredenedin( 3{63 )and( 3{64 ),respectively.Moreover,^fissmoothandstronglyconvexwithmodulus>0,i.e., 2ky)]TJ /F6 11.955 Tf 11.96 0 Td[(xk2^f(y))]TJ /F3 11.955 Tf 14.51 3.16 Td[(^f(x))-222(h^f0(x);y)]TJ /F6 11.955 Tf 11.95 0 Td[(xiL^f 2ky)]TJ /F6 11.955 Tf 11.95 0 Td[(xk2;8x;y2Rn:(3{110)Bythestronglyconvexityof^f,itisclearthatbothfandfarestronglyconvexwithmodulustoo.AsthesameastheIFUSLmethod,weassumetheinexactoracleofFsatisesAssumptionC1,i.e.,f(x)hasan(2;L;)-oracle,forany>0, 2ky)]TJ /F6 11.955 Tf 11.95 0 Td[(xk2f(y))]TJ /F3 11.955 Tf 11.96 0 Td[((f(x)+hg(x);y)]TJ /F6 11.955 Tf 11.96 0 Td[(xi)L 2ky)]TJ /F6 11.955 Tf 11.96 0 Td[(xk2+2;8x;y2Rn;(3{111)whereL;fandgaredenedin( 3{76 ).Forthesakeofsimplicity,weassumethereexistsaninitiallowerestimateonf,givenaslb1.Foranyinitialpointp0,applyingthestronglyconvexityoff,wehavekp0)]TJ /F6 11.955 Tf 12.83 0 Td[(xk22[f(p0))]TJ /F3 11.955 Tf 13.6 0 Td[(lb1]=.ItimpliesthattheuniquesolutionxiswithintheEuclideanballB(p0;p 2[f(p0))]TJ /F3 11.955 Tf 11.95 0 Td[(lb0]=).Therefore,wecanadjustthecenterandradiusoftheEuclideanballinthebeginningofeachGIFUSLSbasedonthegapbetweencurrentupperandlowerboundsonf,whichwillleadtobetteriterationcomplexitiesforthestronglyconvexoptimizationproblem( 3{109 ).WerstgivetheIFUSLSmethod,whichismodiedfromtheIFUSLmethod.TheIFUSLSneedstomakethefollowingadjustmentsintheinnerandouterloops. TheIFUSLSgapreductionprocedure:(x+;D+;ub+;lb+)=GIFUSLS(^x;D;ub;lb;r;;;)InIFUSLgapreductionprocedure,let x=^x,andreplacetheprox-functiond(x)in( 3{11 )withkx)]TJ /F3 11.955 Tf 12.68 0 Td[(^xk2=2. 95

PAGE 96

Algorithm9. TheIFUSLSmethodInAlgorithm 8 ,changeSteps 0 1 and 3 tothefollowingsteps. 0: Giventhedesiredaccuracy>0andtheinitiallowerboundlb1f,choosethestronglyconvexfunctionv()in( 3{65 )and( 3{66 ),theinitialestimateD1onDv;Yin( 3{69 ),theinitialpointp02Rn,theinitialoracleaccuracy0andparameters;2(0;1),computetheinitialupperboundub1=f0(p0)+0. 1: Set^x1=p0,ands=1. 3: Set(^xs+1;Ds+1;ubs+1;lbs+1)=GIFUSLS(^xs;Ds;ubs;lbs;p 2(ubs)]TJ /F3 11.955 Tf 11.95 0 Td[(lbs)=;;;s),wheresisthecurrentaccuracyoftheoracle.Next,wediscussthedesiredaccuracyoftheapproximatesolutiontheIFUSLSmethodcouldachieveanditscorrespondingiterationcomplexitiesforbothcases,whereischosenbytheuserandisxedprior.NotethatGIFUSLSisalmostidenticaltoGIFUSL,themaindierencebetweenthesetwoisthattheEuclideanballischangedforeachcalltoGIFUSLS,whileitisxedforallthecallstoGIFUSL.Therefore,Lemmas 8 ,Propositions 6 and 7 alsoholdforGIFUSLS.Forthecasethatischosenbytheuser,wehavethefollowingtheorem. Theorem12. Foranygiven>0,ifthestepsizesfkgarechosensuchthat( 3{79 )hold,andforanys1, s~s:=(1 3)3=2p p 2C1C2p Ls;(3{112)then,theIFUSLSmethodconvergestoan-solutionto( 3{109 ),andthefollowingstatementshold. a) ThenumberofiterationsperformedbyeachcalltoGIFUSLSdoesnotexceed ~N(;D):=p 6C1s L^f +2p 6C1kAk s D v+1:(3{113) b) ThenumberofcallstoGIFUSLSisboundedby ~S:=max0;log2Dv;Y D1+max0;log1 q(ub1)]TJ /F3 11.955 Tf 11.95 0 Td[(lb1 )+2:(3{114) 96

PAGE 97

c) ThenumberofiterationsperformedbytheIFUSLSmethodisboundedby ~N:=~S0@p 6C1s L^f +11A+ p 2 p 2)]TJ /F3 11.955 Tf 11.96 0 Td[(1+1 1)]TJ 11.96 7.44 Td[(p q!2p 6C1kAk s D v:(3{115) Proof. ForPart a ,similartotheproofofProposition 7 ,byusingR=q 2 ands~s,wehave( 3{87 )holds.Then,thenumberofiterationsperformedbyeachGIFUSLSdoesnotexceed ~N(;D):=p 3C1Rp L^f p +2p 3C1RkAkp D p v+1=p 6C1s L^f +2p 6C1kAk s D v+1:(3{116)ForPart b ,inviewoftheproofofProposition 7 ,wehavethenumberofnon-signicantphasesisboundedbyS1,whereS1isdenedin( 3{99 ).Andthenumberofsignicantphasesisboundedby~S2:=maxn0;log1 q(ub1)]TJ /F9 7.97 Tf 6.59 0 Td[(lb1 )o+1.Therefore,Part b isproved.ForPart c ,similartotheIFUSLmethod,weestimatethenumbersofiterationsperformedbynon-signicantandsignicantphaseswhicharedenotedby~N1and~N2,respectively.ThenthetotalnumberofiterationsperformedbytheIFUSLSmethodisboundedby~N1+~N2~S0@p 6C1s L^f +11A+2p 6C1kAk s D v0@S1Xk=12)]TJ /F12 5.978 Tf 7.78 4.32 Td[((S1)]TJ /F10 5.978 Tf 5.75 0 Td[(k) 2+~S2Xk=1q~S2)]TJ /F10 5.978 Tf 5.75 0 Td[(k 21A (3{117)~S0@p 6C1s L^f +11A+ p 2 p 2)]TJ /F3 11.955 Tf 11.96 0 Td[(1+1 1)]TJ 11.96 7.45 Td[(p q!2p 6C1kAk s D v: (3{118) Fromthetheoremabove,wecanseethattheIFUSLSmethodconvergestoan-solutionto( 3{109 )withiterationcomplexityO(kAk p ).Moreover,ifDv;YisgivenasaninputoftheIFUSLSmethod,thenallcallstoGIFUSLSaresignicantphasesandtheaforementionediterationcomplexitycouldbefurtherimproved. 97

PAGE 98

Next,weconsiderthecasethattheaccuracyoftheoracleisxed,i.e.,s=;8s1.WeneedtomakethefollowingmodicationsontheIFUSLSmethod.M4a.GivenDv;Yasaninputparameter,i.e.,Ds=Dv;Y;8s1inStep 3 oftheIFUSLSmethod.M4b.InIFUSLSmethod,changeubs)]TJ /F3 11.955 Tf 11.96 0 Td[(lbsinStep2toubs)]TJ /F3 11.955 Tf 11.96 0 Td[(lbs~,where ~:=max 6p 3C1C2p L^f ()3=2;6C21C22DkAk22 44v1=3!:(3{119) Theorem13. Ifs=;8s1,thestepsizesfkgarechosensuchthat( 3{79 )holds,thentheIFUSLSmethodconvergestoan~-solutionto( 3{109 ).Moreover,thenumberofcallstoGIFUSLSisboundedby ~S:=log1 qub1)]TJ /F3 11.955 Tf 11.95 0 Td[(lb1 ~+1:(3{120)ThenumberofiterationsperformedbytheIFUSLSmethodisboundedby ~N:=~S0@p 6C1s L^f +11A+2p 6C1kAk (1)]TJ 11.96 7.45 Td[(p q)s Dv;Y v~:(3{121) Proof. When=ub)]TJ /F3 11.955 Tf 12.17 0 Td[(lb~,theIFUSLSmethodterminatesduetotheconditioninM4bissatised.Soweonlyneedtoconsiderthecase>~.By( 3{67 ),( 3{76 )and( 3{77 ),wehaveC21C222LC21C222(L^f+4DkAk2 v);Inviewof( 3{119 ),weget4C21C222L^f(1 3)32and4C21C2224DkAk2 v(1 3)32:Therefore,( 3{112 )inTheorem 12 holds,andthenumberofiterationsperformedbyeachGIFUSLSdoesnotexceed~N(;Dv;Y)denedin( 3{113 ).SinceD=Dv;Y,soallcallsaresignicantphases,andwehave+qforeachcall.Therefore,thenumberofcallstoGIFUSLSisboundedby~S. 98

PAGE 99

Consequently,thenumberofiterationsperformedbytheIFUSLSmethodisboundedby~SXs=1~N(s;Dv;Y)~SXk=1~N(~=q~S)]TJ /F4 7.97 Tf 6.59 0 Td[(k;Dv;Y) (3{122)~S0@p 6C1s L^f +11A+2p 6C1kAk s Dv;Y v~~SXk=1q(~S)]TJ /F4 7.97 Tf 6.58 0 Td[(k)]TJ /F9 7.97 Tf 6.59 0 Td[(1)=2 (3{123)~S0@p 6C1s L^f +11A+2p 6C1kAk (1)]TJ 11.96 7.45 Td[(p q)s Dv;Y v~: (3{124) 99

PAGE 100

CHAPTER4ACCELERATEDBUNDLELEVELMETHODSFORFUNCTIONALCONSTRAINEDOPTIMIZATION 4.1BackgroundInthischapter,weareinterestedinthefollowingfunctionalconstrainedoptimization(FCO)problem: f:=minx2Xf(x);s.t.g(x)0;(4{1)whereXRnisaconvexcompactset,boththeobjectivef:X!Randconstraintg:X!Rareproper,convexfunctionswhichsatisfythefollowingconditionsrespectively: f(y))]TJ /F6 11.955 Tf 11.96 0 Td[(f(x))-221(hf0(x);y)]TJ /F6 11.955 Tf 11.95 0 Td[(xiLf 1+fky)]TJ /F6 11.955 Tf 11.95 0 Td[(xk1+f;8x;y2X; (4{2) g(y))]TJ /F6 11.955 Tf 11.95 0 Td[(g(x))-222(hg0(x);y)]TJ /F6 11.955 Tf 11.95 0 Td[(xiLg 1+gky)]TJ /F6 11.955 Tf 11.95 0 Td[(xk1+g;8x;y2X; (4{3) forsomeLf;Lg>0andf;g2[0;1].Wedenotekkandh;itheEuclideannormanditsassociatedinnerproductinRn,respectively.f0()andg0()denotethe(sub)gradientoff()andg(),respectively.Thisindicatesthatboththeobjectivef()andconstraintg()canbenonsmooth(=0),smooth(=1)andweaklysmooth(0<<1).Throughoutthispaper,weassumethatthesolutionsetXof( 4{1 )isnonemptyandthereexistsanoraclethatprovidestheexactrst-orderinformation,i.e.,thefunctionalvaluesandthe(sub)gradients,forbothf()andg().Weaimtondan-solutionxto( 4{1 ),i.e.,x2Xsatisfying f(x))]TJ /F6 11.955 Tf 11.96 0 Td[(fandg(x)(4{4)foranygiven>0.TheFCOproblem( 4{1 )isconsideredtobedicult,ifg()isnotsimpleandtheprojectionontothefeasiblesetfx2X:g(x)0gisexpensivetocompute.FortheapproachesbasedonLagrangemultipliersandKarush-Kuhn-Tuckerconditionswereferto[ 54 ].Inrecentyears,sincetheBLtypemethodsnotonlyachievetheoptimaliterationcomplexitiesforCOPsbutalsoexhibitadvantagesofutilizinghistoricalinformationandrestrictedmemory 100

PAGE 101

tospeedupconvergenceinpractice,therehavebeenactiveresearchesondevelopingecientBLtypemethodsforsolvingFCOs.AnumberofconstrainedBL(CBL)methods[ 15 16 33 { 35 ]havebeendevelopedforsolving( 4{1 )withnonsmoothf()andg().ThecommonideaofthesemethodsistoreducetheFCOproblem( 4{1 )toanequivalentnon-FCOproblem,forwhichtheclassicalBLmethoddescribedinChapter 1 oritsvariantscouldbeappliedtosolveit.TheCBLmethodsin[ 15 35 ]employsomerestricted-memoryvariants[ 14 38 55 ]oftheclassicalBLmethodtosolvethefollowingproblemequivalentto( 4{1 ): minx2Xh(x):=maxff(x))]TJ /F6 11.955 Tf 11.95 0 Td[(f;g(x)g:(4{5)Clearly,ifxisan-solutionto( 4{5 ),thenf(x))]TJ /F6 11.955 Tf 12.64 0 Td[(fandg(x),itimpliesthatxisan-solutionto( 4{1 )inthesenseof( 4{4 ).However,theunknownoffintroducesanewdicultythatnooraclecancomputetherst-orderinformationofh(x).ToemploytheclassicalBLmethodin[ 16 ]tosolve( 4{5 ),theCBLmethodsin[ 15 35 ]generateanondecreasingsequencefLkgsuchthatLk"f,andreplacefwith( 4{5 )byLkateachiterationk.Consequently,h(x)isreplacedwith hk(x)=h(x;Lk):=maxff(x))]TJ /F6 11.955 Tf 11.95 0 Td[(Lk;g(x)g:(4{6)Sincehk(x)h(x)0foranyx2Xandh(x)=0,sozeroisthenaturetightlowerboundonh(x).Theupperboundonh(x)ateachiterationkisdenedby hk:=minfhk(xi)ji=1;:::;kg;(4{7)Thisisalsothegapbetweentheupperandlowerboundsatiterationksincethelowerboundiszero.Therefore,theaimoftheCBLmethodsin[ 15 35 ]istohavehk#0.Toachievethis,in[ 15 ]givenx1;:::;xk,Lkiscomputedby Lk:=minfmfk(x):mgk(x)0;x2Xg;(4{8) 101

PAGE 102

wheremfk(x)andmgk(x)arethecurrentcuttingplanemodelsoff()andg()asin( 1{6 ),respectively.Then,onhk(x),denethelevellk=(1)]TJ /F6 11.955 Tf 12.04 0 Td[()hkandthelevelsetXk:=fx2X:maxfmfk(x))]TJ /F6 11.955 Tf 12.32 0 Td[(Lk;mgk(x)glkg.Thenextiteratexk+1isgeneratedbyprojectingxktothelevelsetXk.Dierentfromthemethodin[ 15 ],theCBLmethodin[ 35 ]generatessequencesfLkganditeratesfxkgjointlywithoutsolving( 4{8 ).ItprojectsxktothelevelsetXk,ifnofeasiblesolutionxk+1,itenlargesthelevelsetbyincreasingLktoLk+lkandrepeatsthisprocedureuntilanewsearchpointxk+1isfound.TheCBLmethodsin[ 16 33 ]applytheclassicalBLmethodtosolvethefollowingnon-FCOproblemthatisequivalentto( 4{1 )duetothedualitytheory: max01minx2Xh(x;):=(f(x))]TJ /F6 11.955 Tf 11.95 0 Td[(f)+(1)]TJ /F6 11.955 Tf 11.96 0 Td[()g(x):(4{9)Ateachiterationk,theunknownfisreplacedbyLkin( 4{8 ).Consequently,h(;x)isreplacedbyhk(x;):=(f(x))]TJ /F6 11.955 Tf 11.95 0 Td[(Lk)+(1)]TJ /F6 11.955 Tf 11.96 0 Td[()g(x):Thesealgorithmsupdateandxalternatively.Givenkateachiterationk,applytheclassicalBLmethodtohk(x):=hk(x;k).AsthesameastheCBLmethodsdescribedpreviously,thelowerboundiszeroforallhk(x)andhk:=minfhk(xi)ji=1;:::;kgistheupperboundandalsothegapbetweentheupperandlowerboundsontheoptimumofhk(x).Setthelevellk=(1)]TJ /F6 11.955 Tf 9.38 0 Td[()hkandthelevelsetXk:=fx2X:k(mfk(x))]TJ /F6 11.955 Tf 9.38 0 Td[(Lk)+(1)]TJ /F6 11.955 Tf 9.38 0 Td[(k)mgk(x)lkg.Thelevelsetisbuiltonhk(x).Withsimilaridea,theCBLmethodin[ 34 ]furtherincorporatesthebundleaggregationtechniqueandalterstrategyforevaluatingcandidatepointsforsolvinghk(x).Themaindierencebetweenthisapproachandthemethodsin[ 15 35 ]isthataproperprocedureisincorporatedtoadjustineachiteration.Tondan-solutionto( 4{1 )withnomsmoothf()andg(),thealgorithmsin[ 15 16 33 ]and[ 35 ]exhibittheiterationcomplexitiesO)]TJ /F3 11.955 Tf 5.48 -9.68 Td[((1 )2log(1 )andO(1 3),respectively.Moreover,[ 33 56 ]and[ 35 ]furtherextendtheCBLmethodsin[ 16 ]and[ 15 ]todealwith 102

PAGE 103

inexactoracles,respectively.ForthenumericalexperimentsandapplicationsoftheCBLmethodswereferto[ 18 34 35 ].Thischapterisorganizedasfollows.InSection 4.2 weintroduceourrstACBLmethod.InSection 4.3 ,avariantofACBLmethod,theACPLmethodisdevelopedtoimprovetheperformanceoftheACBLmethod.Then,weextendtheACPLmethodtosolveaclassofconstrainedsaddle-pointprobleminSection 4.4 .Theconvergenceanalysisisfollowedintheendofeachsection. 4.2ACBLMethodforFunctionalConstrainedConvexOptimizationInthissection,weproposetheacceleratedconstrainedbundle-level(ACBL)methodtosolve( 4{1 )andfollowedbyitsconvergenceanalysis.Dierentfromtheexistingnon-acceleratedCBLmethods,boththeobjectiveandconstraintin( 4{1 )couldbesmooth,weaklysmoothandnonsmooth.Similarto[ 15 57 ],theACBLmethodcouldbeviewedasapplyingtheacceleratedbundle-level(ABL)method[ 26 ]tosolveh(x)in( 4{5 )whilefisreplacedbyL,thecurrentlowerestimateonf.TheABLmethodisanacceleratedversionoftheclassicalBLmethodreviewedinChapter 1 .ByincorporatingNesterov'smulti-stepaccelerationscheme[ 3 4 ]intotheclassicalBLmethod,theABLmethodgeneratesthreedierentsequencesfxlkg;fxkg;fxukgtoupdatethecuttingplanemodel,prox-centerandupperbound,respectively,whichleadstoitsacceleratediterationcomplexitiesforsmoothandweaklysmoothCPs.SimilartoalltheacceleratedBLtypemethods,theACBLmethodconsistsofouter-innerloops.Theoutlooprstlycomputes^x0,L0and0,i.e.,theinitialpoint,theinitiallowerestimateonfandtheinitialupperboundon h(x;L):=maxff(x))]TJ /F6 11.955 Tf 11.96 0 Td[(L;g(x)g;(4{10)respectively.Notethattheupperboundisalsothegapbetweentheupperandlowerboundsontheoptimumofh(x;L).Then,ineachouteriteration,ifthecurrentupperboundislessthan,terminatetheACBLmethodandoutputtheapproximatesolution;otherwise,aninnerloopreferredastheACBLgapreductionprocedureGACBLwiththeinputsofthecurrentbest 103

PAGE 104

point^xandcurrentlowerestimateL,iscalledtoeitherreducetheupperboundonh(x;L)byaconstantfactororincreasethelowerestimateonffromLtoL+.Theoutputsofx+andL+ofthecurrentGACBLwillbeusedastheinputsofthenextGACBLifnecessary.WestartwiththeACBLgapreductionprocedureGACBL. TheACBLgapreductionprocedure:(x+;L+)=GACBL(^x;L;) 0: Leth(x):=h(x;L).Set h0=h(^x),xu0=x0=^x,mf0=lf(x0;x),mg0=lg(x0;x),andsetk=1. 1: Updatethecuttingplanemodel:xlk=(1)]TJ /F6 11.955 Tf 11.96 0 Td[(k)xuk)]TJ /F9 7.97 Tf 6.58 0 Td[(1+kxk)]TJ /F9 7.97 Tf 6.59 0 Td[(1; (4{11)lf(xlk;x)=f(xlk)+f0(xlk);x)]TJ /F6 11.955 Tf 11.95 0 Td[(xlk; (4{12)lg(xlk;x)=g(xlk)+g0(xlk);x)]TJ /F6 11.955 Tf 11.96 0 Td[(xlk; (4{13)mfk(x)=maxfmfk)]TJ /F9 7.97 Tf 6.59 0 Td[(1(x);lf(xlk;x)g; (4{14)mgk(x)=maxfmgk)]TJ /F9 7.97 Tf 6.59 0 Td[(1(x);lg(xlk;x)g: (4{15) 2: Updatetheprox-centerandlowerestimateonf:setlk=(1)]TJ /F6 11.955 Tf 11.95 0 Td[() hk)]TJ /F9 7.97 Tf 6.59 0 Td[(1,dene Qk:=fx2X:mfk(x))]TJ /F6 11.955 Tf 11.95 0 Td[(Llk;mgk(x)lkg:(4{16)IfQk=;:terminatewithx+=xuk)]TJ /F9 7.97 Tf 6.59 0 Td[(1;L+=minx2Xfmfk(x):mgk(x)0g.Otherwise, xk=argminx2Qkkx)]TJ /F6 11.955 Tf 11.95 0 Td[(xk)]TJ /F9 7.97 Tf 6.58 0 Td[(1k2:(4{17) 3: Updatetheupperbound:set~xuk=(1)]TJ /F6 11.955 Tf 11.95 0 Td[(k)xuk)]TJ /F9 7.97 Tf 6.59 0 Td[(1+kxk; (4{18)xuk=(~xuk;ifh(~xuk)
PAGE 105

planemodelHk(x;L)ofh(x;L)asfollows. Hk(x;L):=maxfmfk(x))]TJ /F6 11.955 Tf 11.95 0 Td[(L;mgk(x)g;k0:(4{20)Consequently,thelevelsetQkin( 4{16 )isdenedasfx2X:Hk(x;L)lkg.Thirdly,GACBLeitherreducestheupperboundonh(x;L)byaconstantfactorinStep 3 orincreasesthelowerestimateonffromLtoL+inStep 2 onceitterminates.Fourthly,theinputLisincreasedtoL+bysolvingalinearprogrammingsubproblemwhenthelevelsetQkisempty.Lastly,thecuttingplanemodelsmfk(x)andmgk(x)aremonotonicallyincreasing,andthelevelsetsfQkgaccumulateconstraintsduringiterations,whichmakesthesubproblem( 4{17 )moreandmoreexpensivetosolve.InordertoguaranteetheterminationofGACBLandtheacceleratediterationcomplexityfortheACBLmethodtosolve( 4{1 ),theparametersfkgneedtosatisfythefollowingcondition: 00and82[0;1].Thefollowinglemmaprovidestwoexamplesfortheselectionoffkg. Lemma9. a) Ifk=2 k+2,k1,thenthecondition( 4{21 )issatisedwithc=2=. b) Iffkgisrecursivelydenedby 1=1;2k+1=(1)]TJ /F6 11.955 Tf 11.95 0 Td[(k+1)2k;8k1;(4{22)thenthecondition( 4{21 )holdswithc=2=. Proof. Part a followsimmediatelybypluggingk=2 k+2into( 4{21 ).ForPart b ,observethatfkgismonotonicallydecreasingand1=1,so0
PAGE 106

Next,weprovek<2 k;8k1.By( 4{22 )andk>k+1,wehave 1 k+1)]TJ /F3 11.955 Tf 16.51 8.09 Td[(1 k=k)]TJ /F6 11.955 Tf 11.96 0 Td[(k+1 kk+1=k k+k+1> 2;8k1:(4{23)Since2(0;1 2)inAlgorithm 10 ,1=1>,itimplies1 k)]TJ /F3 11.955 Tf 12.14 0 Td[(1>(k)]TJ /F9 7.97 Tf 6.59 0 Td[(1) 2,therefore,1 k>k 2,thenk<2 k. SimilartoLemma1in[ 26 ],wehavethefollowingobservationabouttheiteratesgeneratedbyGACBL. Lemma10. Letfxig;i=1;2;:::;k,betheiteratesgeneratedbyGACBLbeforeitterminates,thenthelevelsetsfQig;i=1;2;:::;k,haveapointincommon.Moreover,wehave kXi=1kxi)]TJ /F6 11.955 Tf 11.95 0 Td[(xi)]TJ /F9 7.97 Tf 6.58 0 Td[(1k2D2X;(4{24)where DX:=maxx;y2Xkx)]TJ /F6 11.955 Tf 11.95 0 Td[(yk:(4{25) Proof. Inviewofthedenitionof hkandtheterminationcriterioninStep 3 ,wehave h0 h1::: hk> h0:(4{26)Letu2Argminx2XHk(x;L),whereHk(x;L)isdenedin( 4{20 ).ObservethatH1(x;L)H2(x;L):::Hk(x;L),wehave Hi(u;L)Hk(u;L)lk=(1)]TJ /F6 11.955 Tf 11.95 0 Td[() hk)]TJ /F9 7.97 Tf 6.59 0 Td[(1(1)]TJ /F6 11.955 Tf 11.96 0 Td[() hi)]TJ /F9 7.97 Tf 6.59 0 Td[(1=li;8i=1;2;:::;k:(4{27)Thusu2Qi,i=1;2;:::;k.Thenbytheoptimalconditionof( 4{17 )andthefactkx)]TJ /F6 11.955 Tf 11.95 0 Td[(xk)]TJ /F9 7.97 Tf 6.58 0 Td[(1k2isstronglyconvex,wehave kxi)]TJ /F9 7.97 Tf 6.58 0 Td[(1)]TJ /F6 11.955 Tf 11.96 0 Td[(xik2kxi)]TJ /F9 7.97 Tf 6.59 0 Td[(1)]TJ /F6 11.955 Tf 11.95 0 Td[(uk2)-222(kxi)]TJ /F6 11.955 Tf 11.95 0 Td[(uk2:(4{28)Summinguptheaboveinequalityfori=1;2;:::;k,weobtain kXi=1kxi)]TJ /F6 11.955 Tf 11.96 0 Td[(xi)]TJ /F9 7.97 Tf 6.59 0 Td[(1k2kx0)]TJ /F6 11.955 Tf 11.95 0 Td[(uk2)-222(kxk)]TJ /F6 11.955 Tf 11.96 0 Td[(uk2D2X:(4{29) 106

PAGE 107

Sincethelowerboundonminx2Xh(x)iszero,theinputandoutputgapsofGACBLare :=maxff(^x))]TJ /F6 11.955 Tf 11.95 0 Td[(L;g(^x)gand+:=maxff(x+))]TJ /F6 11.955 Tf 11.96 0 Td[(L+;g(x+)g;(4{30)respectively.WenowprovideaboundonthenumberofiterationsperformedbyonecallofGACBL. Proposition8. Ifthestepsizesfkgarechosensuchthat( 4{21 )holds,thenthenumberofiterationsperformedbyprocedureGACBLdoesnotexceed N():=2c1+sLs(1+CX)D1+sX (1+s)22 1+3s+(2c=)1 1+s+1;(4{31)where s=minff;gg;Ls=maxfLf;LggandCX=Djf)]TJ /F4 7.97 Tf 6.59 0 Td[(gjX:(4{32) Proof. SupposeprocedureGACBLdoesnotterminateattheKthiterationforsomeK>0.Forany1kK,wehavef(~xuk))]TJ /F6 11.955 Tf 11.95 0 Td[(Llf(xlk;~xuk))]TJ /F6 11.955 Tf 11.96 0 Td[(L+Lf 1+fk~xuk)]TJ /F6 11.955 Tf 11.96 0 Td[(xlkk1+f (4{33)(1)]TJ /F6 11.955 Tf 11.96 0 Td[(k)lf(xlk;xuk)]TJ /F9 7.97 Tf 6.58 0 Td[(1)+klf(xlk;xk))]TJ /F6 11.955 Tf 11.96 0 Td[(L+1+fkLf 1+fkxk)]TJ /F6 11.955 Tf 11.95 0 Td[(xk)]TJ /F9 7.97 Tf 6.58 0 Td[(1k1+f (4{34)(1)]TJ /F6 11.955 Tf 11.96 0 Td[(k)(f(xuk)]TJ /F9 7.97 Tf 6.59 0 Td[(1))]TJ /F6 11.955 Tf 11.95 0 Td[(L)+k(lf(xlk;xk))]TJ /F6 11.955 Tf 11.95 0 Td[(L)+1+fkLf 1+fkxk)]TJ /F6 11.955 Tf 11.96 0 Td[(xk)]TJ /F9 7.97 Tf 6.59 0 Td[(1k1+f (4{35)(1)]TJ /F6 11.955 Tf 11.96 0 Td[(k)(f(xuk)]TJ /F9 7.97 Tf 6.59 0 Td[(1))]TJ /F6 11.955 Tf 11.95 0 Td[(L)+klk+1+fkLf 1+fkxk)]TJ /F6 11.955 Tf 11.96 0 Td[(xk)]TJ /F9 7.97 Tf 6.59 0 Td[(1k1+f: (4{36)Therstinequalityusesthesmoothnessrelationsoff()in( 4{2 ),thesecondonefollowsfromthedenitionof~xukin( 4{18 ),thethirdonefollowsfromtheconvexityoff()andthelastone 107

PAGE 108

followsfrom( 4{16 )and( 4{17 ).Similarly,forg(),wehaveg(~xuk)lg(xlk;~xuk)+Lg 1+gk~xuk)]TJ /F6 11.955 Tf 11.96 0 Td[(xlkk1+g (4{37)(1)]TJ /F6 11.955 Tf 11.96 0 Td[(k)lg(xlk;xuk)]TJ /F9 7.97 Tf 6.58 0 Td[(1)+klg(xlk;xk)+1+gkLg 1+gkxk)]TJ /F6 11.955 Tf 11.95 0 Td[(xk)]TJ /F9 7.97 Tf 6.58 0 Td[(1k1+g (4{38)(1)]TJ /F6 11.955 Tf 11.96 0 Td[(k)g(xuk)]TJ /F9 7.97 Tf 6.59 0 Td[(1)+klk+1+gkLg 1+gkxk)]TJ /F6 11.955 Tf 11.96 0 Td[(xk)]TJ /F9 7.97 Tf 6.59 0 Td[(1k1+g: (4{39)Inviewof( 4{19 ),wehavehk=maxff(xuk))]TJ /F6 11.955 Tf 12.04 0 Td[(L;g(xuk)gmaxff(~xuk))]TJ /F6 11.955 Tf 12.04 0 Td[(L;g(~xuk)g.Togetherwiththeabovetwoinequalitiesandlk=(1)]TJ /F6 11.955 Tf 11.96 0 Td[()hk)]TJ /F9 7.97 Tf 6.59 0 Td[(1,weobtainhk(1)]TJ /F6 11.955 Tf 11.95 0 Td[(k)maxff(xuk)]TJ /F9 7.97 Tf 6.58 0 Td[(1))]TJ /F6 11.955 Tf 11.95 0 Td[(L;g(xuk)]TJ /F9 7.97 Tf 6.59 0 Td[(1)g+klk (4{40)+maxf1+fkLf 1+fkxk)]TJ /F6 11.955 Tf 11.96 0 Td[(xk)]TJ /F9 7.97 Tf 6.59 0 Td[(1k1+f;1+fkLg 1+gkxk)]TJ /F6 11.955 Tf 11.95 0 Td[(xk)]TJ /F9 7.97 Tf 6.58 0 Td[(1k1+gg (4{41)(1)]TJ /F6 11.955 Tf 11.95 0 Td[(k)hk)]TJ /F9 7.97 Tf 6.58 0 Td[(1+maxf1+fkLf 1+fkxk)]TJ /F6 11.955 Tf 11.95 0 Td[(xk)]TJ /F9 7.97 Tf 6.59 0 Td[(1k1+f;1+fkLg 1+gkxk)]TJ /F6 11.955 Tf 11.96 0 Td[(xk)]TJ /F9 7.97 Tf 6.59 0 Td[(1k1+gg: (4{42)Inviewof( 4{32 ),itimplies hk(1)]TJ /F6 11.955 Tf 11.96 0 Td[(k)hk)]TJ /F9 7.97 Tf 6.59 0 Td[(1+1+skLs(1+CX) 1+skxk)]TJ /F6 11.955 Tf 11.96 0 Td[(xk)]TJ /F9 7.97 Tf 6.59 0 Td[(1k1+s:(4{43)Dividingbothsidesof( 4{43 )by1+sk,andsummingupfor1kK,by( 4{21 )andhk0,81kK,wehave hK1+sK(1)]TJ /F6 11.955 Tf 11.95 0 Td[(1)h0+1+sKLs(1+CX) 1+sKXk=1kxk)]TJ /F6 11.955 Tf 11.95 0 Td[(xk)]TJ /F9 7.97 Tf 6.58 0 Td[(1k1+s:(4{44)ApplyHolder'sinequality,anduseLemma 10 and( 4{21 ),wehave KXk=1kxk)]TJ /F6 11.955 Tf 11.96 0 Td[(xk)]TJ /F9 7.97 Tf 6.59 0 Td[(1k1+sK1)]TJ /F10 5.978 Tf 5.76 0 Td[(s 2 KXk=1kxk)]TJ /F6 11.955 Tf 11.95 0 Td[(xk)]TJ /F9 7.97 Tf 6.58 0 Td[(1k2!1+s 2K1)]TJ /F10 5.978 Tf 5.76 0 Td[(s 2D1+sX:(4{45)Sincea1+sKc1+sK)]TJ /F9 7.97 Tf 6.59 0 Td[((1+s)dueto( 4{21 )and=h0,( 4{44 )implies hKc1+s(1)]TJ /F6 11.955 Tf 11.95 0 Td[(1) K1+s+Ls(1+CX) 1+sc1+sD1+sX K(1+3s)=2:(4{46) 108

PAGE 109

InviewoftheterminationcriteriainStep 3 ofGACBL,wehavehK>,thencombiningwith( 4{46 ),weobtain K<(2c=)1 1+s+2c1+sLs(1+CX)D1+sX (1+s)22 1+3s:(4{47) TheabovepropositionshowsthatthenumberofiterationsperformedbyeachGACBLonlydependsontheinputgapandisindependentofL.TheprocedureGACBLcouldeitherterminatesinStep 2 orStep 3 ,butforbothcases,thenumberofiterationsperformedisboundedbyN()in( 4{31 ).Now,wearereadytopresenttheschemeoftheACBLmethod,alsoreferredastheouterloopoftheACBLmethod.Theouterlooprstlygeneratesaninitiallowerestimateonf,andthencallGACBLiterativelyuntilanapproximatedsolutionwithrequiredaccuracyisfound. Algorithm10. Theacceleratedconstrainedbundle-level(ACBL)method 0: Giventolerance>0andparameters2(0;1 2). 1: Choosetheinitialpointp02X,computep12Argminflf(p0;x):x2X;lg(p0;x)0g.SetL0=lf(p0;p1),^x0=p1,0=maxff(^x0))]TJ /F6 11.955 Tf 11.95 0 Td[(L0;g(^x0)g,ands=0. 2: Ifs,terminateandoutputapproximatesolution^xs. 3: Set(^xs+1;Ls+1)=GACBL(^xs;Ls;)ands+1=maxff(^xs+1))]TJ /F6 11.955 Tf 11.95 0 Td[(Ls+1;g(^xs+1)g. 4: Sets=s+1andgotoStep 2 .Fortheseekforsimplicity,wecalleachgapreductionprocedureGACBLwhichincreasesLanon-ecientphase(L+>L),otherwise,anecientphase(L+=L).InordertoanalyzetheiterationcomplexityoftheACBLmethod,weneedtoestimatethenumberofecientandnon-ecientphases.Inthefollowinglemma,weusethesameideain[ 15 ]toboundthenumberofnon-ecientphasesperformedbytheACBLmethod. 109

PAGE 110

Lemma11. Forany>0,thenumberofnon-ecientphasesperformedbytheACBLmethodtondan-solutionto( 4{1 )isboundedby T1():=1+log(2)]TJ /F9 7.97 Tf 6.58 0 Td[(2)maxnLf 1+fD1+fX;Lg 1+gD1+gX;f)]TJ /F6 11.955 Tf 11.96 0 Td[(L0o (1)]TJ /F6 11.955 Tf 11.95 0 Td[():(4{48) Proof. Supposethenumberofnon-ecientphasesperformedbytheACBLmethodism,anddenotedbyindicesfi1;i2;:::;img,thenwehaveL0=Li1lk(j)=(1)]TJ /F6 11.955 Tf 11.95 0 Td[() hk(j))]TJ /F9 7.97 Tf 6.58 0 Td[(1(1)]TJ /F6 11.955 Tf 11.95 0 Td[()^h(Lij)(1)]TJ /F6 11.955 Tf 11.95 0 Td[()j+1(Lij);(4{53) 110

PAGE 111

where^h(Lij):=minx2Xmaxff(x))]TJ /F6 11.955 Tf 11.96 0 Td[(Lij;g(x)g.Combinealltheserelations,wehavej+1 jj+1(Lij+1)(j+1(Lij))]TJ /F6 11.955 Tf 11.96 0 Td[(j+1(Lij+1))=(Lij+1)]TJ /F6 11.955 Tf 11.95 0 Td[(Lij) 2j(Lij)=(Lij+1)]TJ /F6 11.955 Tf 11.95 0 Td[(Lij) (4{54)
PAGE 112

TogetherwiththefactthatQk(m)isempty,im>andtheterminationcriteriainStep 3 ,wehavem(Lim)>lk(m)=(1)]TJ /F6 11.955 Tf 11.96 0 Td[() hk(m))]TJ /F9 7.97 Tf 6.59 0 Td[(1>(1)]TJ /F6 11.955 Tf 11.96 0 Td[()im>(1)]TJ /F6 11.955 Tf 11.95 0 Td[():Hence,wehave(1)]TJ /F6 11.955 Tf 11.96 0 Td[()<(2)]TJ /F3 11.955 Tf 11.96 0 Td[(2)1)]TJ /F4 7.97 Tf 6.58 0 Td[(mmaxfVX;f)]TJ /F6 11.955 Tf 11.95 0 Td[(L0g;thenmisboundedbyT1(). ThemainconvergencepropertiesoftheACBLmethodispresentedinthefollowingtheorem.Werstestimatethenumberofecientphases,togetherwiththeupperboundonthenumberofnon-ecientphasesintheabovelemma,wecanboundthenumberofcallstoGACBLperformedbytheACBLmethod.Moreover,bycalculatingthenumberoftotaliterationsperformedbytheecientandnon-ecientphasesseparately,wegettheresultoftotaliterationsperformedbytheACBLmethod. Theorem14. Foranygiven>0,ifthestepsizesfkginGACBLarechosensuchthat( 4{21 )holds,thenthefollowingstatementsholdfortheACBLmethodtocomputean-solutiontoproblem( 4{1 ). a) ThenumberofgapreductionproceduresGACBLperformedbytheACBLmethoddoesnotexceed S:=2+log(2)]TJ /F9 7.97 Tf 6.59 0 Td[(2)maxfVX;f)]TJ /F6 11.955 Tf 11.95 0 Td[(L0g (1)]TJ /F6 11.955 Tf 11.96 0 Td[()+log1 VX :(4{62) b) ThetotalnumberofiterationsperformedbytheACBLmethodcanbeboundedby N:=(2c=)1 1+s+1S+2c1+sLs(1+CX)D1+sX (1+s)22 1+3sT1()+1 1)]TJ /F6 11.955 Tf 11.95 0 Td[(2=(1+3s);(4{63)whereDXisdenedin( 4{25 ),T1()isdenedin( 4{48 ),VXisdenedin( 4{61 )ands;CX;Lsaredenedin( 4{32 ). Proof. Supposethatthesetsofindicesofthenon-ecientandecientphasesperformedbytheACBLmethodarefi1;i2;:::;imgandfj1;j2;:::;jng,respectively.ByLemma 11 ,wealreadyhavemT1().InviewoftheterminationcriteriainStep 3 inGACBL,wehave 112

PAGE 113

+foranyecientphase.Therefore,foralltheseecientphases,wehave jk>;81knandjk+1jk;81kn)]TJ /F3 11.955 Tf 11.95 0 Td[(1:(4{64)Itimmediatelyfollowsthatjnn)]TJ /F9 7.97 Tf 6.59 0 Td[(10,andsincejn>,weconcludethat>n)]TJ /F9 7.97 Tf 6.58 0 Td[(10,i.e.,n<1+log1 0 :Inviewof( 4{61 ),thenumberofecientphasesisboundedbyT2():=1+log1 VX :Therefore,thetotalnumberofcallstoGACBLisboundedbyT1()+T2().Forpart b ,wecalculatethenumberofiterationsperformedbynon-ecientandecientphasesseparately.LetsbethenumberofiterationsperformedbythesthGACBLprocedurewithinputgaps.Foranynon-ecientphaseik;1km,byproposition 8 andthefactik>andN()in( 4{31 )ismonotonicallydecreasingwithrespectto,wehaveikN(ik)k)]TJ /F4 7.97 Tf 6.59 0 Td[(n.Therefore,thetotalnumberofiterationsperformedbyecientphasesisboundedbyN2:=nXk=1N(jk)
PAGE 114

Hence,thetotalnumberofiterationsperformedbytheACBLmethodisboundedbyN1+N2. WecanseethattheiterationcomplexityoftheACBLmethodisoftheorderO)]TJ /F9 7.97 Tf 6.68 -4.98 Td[(1 2 1+3slog1 ,itisreasonablethattheorderofiterationcomplexityforsolving( 4{1 )isdominatedbytheobjectiveorconstraintthathasweakersmoothness.Ifbothf()andg()arenonsmooth,i.e.,s=0,theACBLmethodachievesthesamebestiterationcomplexityO)]TJ /F9 7.97 Tf 8.35 -4.97 Td[(1 2log1 astheproximallevelbundlemethodin[ 15 ]andtheconstrainedlevelmethodsin[ 16 33 ].Ifbothf()andg()aresmooth,theACBLmethodachievesO1 p log1 .Tothebestofourknowledge,thisistherstacceleratedBLtypeforsolvingsmoothFCOwithsuchiterationcomplexityproved. 4.3ACPLMethodforFunctionalConstrainedConvexOptimizationInthissection,weproposeavariantoftheACBLmethod,namely,theacceleratedconstrainedproximal-level(ACPL)method,tosolvetheFCO( 4{1 ).WecanseethattheACBLmethodneedstosolvealinearprogramming(LP)probleminordertoincreasethelowerestimateonf,andthelevelsetsaccumulatelinearconstraintswhichmakestheaforementionedLPproblemandthequadraticprogrammingproblem( 4{17 )moreandmoreexpensivetosolve.ThemaingoalofdevelopingtheACPLmethodistoimprovethepracticalperformanceoftheACBLmethodforsolvinglarge-scaleFCOs.SimilartotheideaoftheACBLmethod,westillfocusonminimizingtheobjectivefunctionh(x;L)withLmonotonicallyincreasingtoapproachf.ThemaindierencetotheACBLmethodisthatweintroduceasetoflocalizerstocontrolthenumberoflinearconstraintsinthesubproblemforupdatingtheprox-centeranduseadierentstrategytoupdateLwithoutextracostthatdoesnotrequiretosolveaLPproblem.WestartbydescribingtheschemeoftheACPLmethod.TheACPLmethodconsistsofouter-innerloops,andineachouteriteration,aninnerloop,theACPLgapreductionprocedureGACPL,iscalledtoeitherincreaseLorreducetheupperboundontheobjectivefunctionh(x;L)byaconstantfactor.Let'sbeginwiththeACPLgapreductionprocedureGACPL. 114

PAGE 115

TheACPLgapreductionprocedure:(x+;L+)=GACPL(^x;L;;) 0: Leth(x):=maxff(x))]TJ /F6 11.955 Tf 11.18 0 Td[(L;g(x)g.Set h0=h(^x),l=(1)]TJ /F6 11.955 Tf 11.18 0 Td[() h0,R0=X,xu0=x0=^x,andk=1. 1: Updatethecuttingplanemodel:computexlk;lf(xlk;x)andlg(xlk;x)by( 4{11 ),( 4{12 )and( 4{13 ),respectively.Let R k:=fx2Rk)]TJ /F9 7.97 Tf 6.59 0 Td[(1:lf(xlk;x))]TJ /F6 11.955 Tf 11.95 0 Td[(Ll;lg(xlk;x)0g:(4{69) 2: Updatetheprox-centerandlowerestimateonf:ifR k=;:terminateandoutputx+=xuk)]TJ /F9 7.97 Tf 6.59 0 Td[(1;L+=L+l.Otherwise, xk=argminx2R kd(x):=1 2kx)]TJ /F3 11.955 Tf 12.68 0 Td[(^xk2:(4{70) 3: Updatetheupperbound:set~xukandxukto( 4{18 )and( 4{19 ),respectively,let hk=h(xuk).If hk)]TJ /F6 11.955 Tf 11.95 0 Td[(l h0,thenterminateandoutputx+=xuk;L+=L. 4: ChooseanylocalizerRksatisfyingR kRk Rk,where Rk:=fx2X:hxk)]TJ /F3 11.955 Tf 12.68 0 Td[(^x;x)]TJ /F6 11.955 Tf 11.96 0 Td[(xki0g:(4{71)Setk=k+1andgotoStep 1 InordertoguaranteetheterminationofGACPLandtheoptimaliterationcomplexityoftheACPLmethod,theparametersfkgneedtosatisfythefollowingcondition: 1=1;00and82[0;1].Thefollowinglemmaprovidestwoexamplesfortheselectionoffkgalreadygivenin[ 53 ]. Lemma12. a) Ifk=2=(k+1),k1,thenthecondition( 4{72 )issatisedwithc=2. b) Iffkgisrecursivelydenedby 1=1;2k+1=(1)]TJ /F6 11.955 Tf 11.96 0 Td[(k+1)2k;8k1;(4{73)thenthecondition( 4{72 )holdswithc=2.ThebasicideaoftheACPLmethodistoapplytheFAPLmethod[ 53 ]tosolveminx2Xh(x;L)whileLisxedintheinnerloopsandupdatedintheouterloops.Similarto 115

PAGE 116

Lemma2.2andLemma2.3in[ 53 ],thefollowinglemmadescribessomeimportantobservationsregardingtheexecutionofprocedureGACPL. Lemma13. LetEl:=fx2X:f(x))]TJ /F6 11.955 Tf 12.24 0 Td[(Ll;g(x)0g,thefollowingstatementsholdforGACPL. a) IfEl6=;,thenElR kRk Rkforanyk1,and( 4{70 )hasauniquesolution. b) IfR k=;,thenwehaveEl=;,whichfurtherimpliesL+ll,weprovedL+l
PAGE 117

Proof. Inviewofthedenitionof Riin( 4{71 )andx0=^x,wehavexi=argminx2 Rid(x);80ik)]TJ /F3 11.955 Tf 11.95 0 Td[(1.Togetherwiththefactxi+12Ri Riandd(x)isstronglyconvex,wehaved(xi+1)d(xi)+hrd(xi);xi+1)]TJ /F6 11.955 Tf 11.96 0 Td[(xii+1 2kxi+1)]TJ /F6 11.955 Tf 11.96 0 Td[(xik2d(xi)+1 2kxi+1)]TJ /F6 11.955 Tf 11.96 0 Td[(xik2:Summinguptheaboveinequalityfori=1;2;:::;k)]TJ /F3 11.955 Tf 11.95 0 Td[(1,weconcludethat 1 2k)]TJ /F9 7.97 Tf 6.59 0 Td[(1Xi=0kxi+1)]TJ /F6 11.955 Tf 11.95 0 Td[(xikd(xk)=1 2kxk)]TJ /F3 11.955 Tf 12.68 0 Td[(^xk21 2D2X:(4{76) WenowprovideaboundonthenumberofiterationsperformedbyeachcallofGACPL. Proposition9. Ifthestepsizesfkgarechosensuchthat( 4{72 )holds,thenthenumberofiterationsperformedbyprocedureGACPLdoesnotexceed N():=c1+sLs(1+CX)D1+sX (1+s)2 1+3s+1;(4{77)whereDXisdenedin( 4{25 )ands;Ls;CXaredenedin( 4{32 ). Proof. SupposeGACPLdoesnotterminateattheKthiterationforsomeK>0.Asthesameas( 4{33 )and( 4{37 )intheproofofProposition 8 ,wehavehk=maxff(xuk))]TJ /F6 11.955 Tf 12.22 0 Td[(L;g(xuk)gmaxff(~xuk))]TJ /F6 11.955 Tf 11.96 0 Td[(L;g(~xuk)g,andreplacelkwithlin( 4{40 ),wehavefor1kK, hk)]TJ /F6 11.955 Tf 10.2 0 Td[(l(1)]TJ /F6 11.955 Tf 10.2 0 Td[(k)(hk)]TJ /F9 7.97 Tf 6.58 0 Td[(1)]TJ /F6 11.955 Tf 10.2 0 Td[(l)+maxf1+fkLf 1+fkxk)]TJ /F6 11.955 Tf 10.2 0 Td[(xk)]TJ /F9 7.97 Tf 6.59 0 Td[(1k1+f;1+gkLg 1+gkxk)]TJ /F6 11.955 Tf 10.2 0 Td[(xk)]TJ /F9 7.97 Tf 6.58 0 Td[(1k1+gg:(4{78)Inviewof( 4{32 ),theaboveinequalityisrewrittenas hk)]TJ /F6 11.955 Tf 11.95 0 Td[(l(1)]TJ /F6 11.955 Tf 11.96 0 Td[(k)(hk)]TJ /F9 7.97 Tf 6.59 0 Td[(1)]TJ /F6 11.955 Tf 11.95 0 Td[(l)+1+skLs(1+CX) 1+skxk)]TJ /F6 11.955 Tf 11.96 0 Td[(xk)]TJ /F9 7.97 Tf 6.59 0 Td[(1k1+s:(4{79)Dividingbothsidesof( 4{79 )by1+sk,thensummingupfor1kK,by( 4{72 ),wehave hK)]TJ /F6 11.955 Tf 11.96 0 Td[(l1+sKLs(1+CX) 1+sKXk=1kxk)]TJ /F6 11.955 Tf 11.96 0 Td[(xk)]TJ /F9 7.97 Tf 6.59 0 Td[(1k1+s:(4{80) 117

PAGE 118

UseLemma 14 ,( 4{45 )and( 4{72 ),weobtain hK)]TJ /F6 11.955 Tf 11.95 0 Td[(lLs(1+CX) 1+sc1+sD1+sX K1+s(4{81)InviewoftheterminationcriteriainStep 3 ofGACPL,wehavehk)]TJ /F6 11.955 Tf 11.13 0 Td[(l.Thencombiningwith( 4{81 ),weconclude K0andparameters;2(0;1). 1: Choosetheinitialpointp02X,computep1;L0;^x0;0asthesameasAlgorithm 10 ,sets=0. 2: Ifs,terminateandoutputapproximatesolution^xs. 3: Set(^xs+1;^Ls+1)=GACPL(^xs;Ls;;)ands+1=maxff(^xs+1))]TJ /F6 11.955 Tf 11.95 0 Td[(Ls+1;g(^xs+1)g. 4: Sets=s+1andgotoStep 2 .ThefollowingtheoremestablishesthecomplexityboundsonthenumberofcallstoprocedureGACPLandthetotalnumberofiterationsperformedbytheACPLmethod.TheproofissimilartothatofTheorem 14 .WestillclassifythecallstoGACPLintoecientandnon-ecientphasesandestimatethenumberofiterationsperformedbyecientandnon-ecientphasesseparately.ThemaindierencestotheACBLmethodarethenumberofupdatesonLandthenumberofiterationsperformedbynon-ecientphases. Theorem15. Foranygiven>0,ifthestepsizesfkginprocedureGACPLarechosensuchthat( 4{72 )holds,thenthefollowingstatementsholdfortheACPLmethodtocomputean-solutiontoproblem( 4{1 ). 118

PAGE 119

a) ThenumberofcallstoprocedureGACPLperformedbytheACPLmethoddoesnotexceed S:=2+f)]TJ /F6 11.955 Tf 11.96 0 Td[(L0 (1)]TJ /F6 11.955 Tf 11.96 0 Td[()+log1 qVX :(4{83) b) ThetotalnumberofiterationsperformedbytheACPLmethodcanbeboundedby N:=T1()+c1+sLs(1+CX)D1+sX (1+s)2 1+3sS+1 1)]TJ /F6 11.955 Tf 11.96 0 Td[(q2=(1+3s);(4{84)whereDX;q;VXaredenedin( 4{25 ),( 4{74 )and( 4{61 )respectively,ands;CX;Lsaredenedin( 4{32 ). Proof. FollowthesamenotationintheproofofTheorem 14 ,wedenotethesetsofindicesofthenon-ecientandecientphasesperformedbytheACPLmethodbyfi1;i2;:::;imgandfj1;j2;:::;jng,respectively.SimilartotheACBLmethod,fortheecientphasesoftheACPLmethod,wehave jk>;81knandjk+1qjk;81kn)]TJ /F3 11.955 Tf 11.96 0 Td[(1;(4{85)thesecondrelationfollowsfromLemma 13 Part c .Combiningwithjnqn)]TJ /F9 7.97 Tf 6.58 0 Td[(10;jn>and0VXfrom( 4{61 ),wehavethenumberofecientphasesisboundedbyT2():=1+log1 qVX :Forthenon-ecientphases,observethatL+)]TJ /F6 11.955 Tf 12.08 0 Td[(L=l=(1)]TJ /F6 11.955 Tf 12.08 0 Td[()h0=(1)]TJ /F6 11.955 Tf 12.08 0 Td[()inGACPLandik>;81km,wehaveLik+1)]TJ /F6 11.955 Tf 11.96 0 Td[(Lik>(1)]TJ /F6 11.955 Tf 11.96 0 Td[();81km.Therefore,thenumberofnon-ecientphasesisboundedby T1():=f)]TJ /F6 11.955 Tf 11.95 0 Td[(L0 (1)]TJ /F6 11.955 Tf 11.96 0 Td[()+1;(4{86)andthetotalnumberofcallstoGACPLisboundedbyT1()+T2().Next,wecalculatethenumberofiterationsperformedbynon-ecientandecientphases.LetsbethenumberofiterationsperformedbythesthcalltoGACPLwithinputgaps.SimilartotheproofofTheorem 14 ,forthenon-ecientphases,wehaveik>,and 119

PAGE 120

ikN(ik)qk)]TJ /F4 7.97 Tf 6.59 0 Td[(nandnT2(),wehavetheiterationsperformedbytheecientphasesisboundedbyN2:=nXk=1N(jk)
PAGE 121

updatesonL,whiletheACPLmethodupdatesLwithoutsolvingtheLPproblemandhencenoadditionalcomputationalcost.NotethatifthefeasiblesetisgivenasanEuclideanballinRn,i.e.,X=B(x;r),wherexandrarethecenterandradiusoftheEuclideanball,respectively,thenthesimilarideaoftheFAPLmethod[ 53 ]couldbeappliedtofurtherimprovethepracticalperformanceoftheACPLmethodforsolvinglarge-scaleFCOs.WiththefollowingmodicationsinGACPL:inStep 0 changexu0=x0=^xtoxu0=^xandx0=x;inStep 2 changetheterminationcriteriaR k=;toR k=;orkxk)]TJ /F3 11.955 Tf 12.76 0 Td[(xk>r;inStep 4 chooseanypolyhedralsetRksatisfyingR kRk Rk,where Rk:=fx2Rn:hxk)]TJ /F3 11.955 Tf 12.68 0 Td[(^x;x)]TJ /F6 11.955 Tf 11.96 0 Td[(xki0g;thenR konlycontainslinearconstraintsastheconstraintx2B(x;r)couldberemovedfromR k.Consequently,theonlysubproblem( 4{70 )couldbeexactlysolvedviaitsdualproblemandthecomputationalcostislowduetothenumberofconstraintsinR kissmall.Thedetailsaboutthisapproachcouldbefoundin[ 53 ]. 4.4ACSLMethodforConstrainedSaddle-PointProblemInthissection,weintroducetheacceleratedconstrainedsmoothing-level(ACSL)methodtosolveaclassofstructuredFCOproblem.Specically,westillsolve( 4{1 ),whereg()isconvexandsatises( 4{3 )withconstantLgandg=1,andf()isgivenintheform: f(x):=maxy2YfhAx;yi)]TJ /F6 11.955 Tf 19.26 0 Td[((y)g:(4{91)Here,YRmisacompactconvexset,:=Y!Risasimplecontinuousconvexfunction,andA:Rn!Rmisalinearoperator.Letv:Y!Rbeaprox-functionwithmodulusv,thenitsassociateBregmandivergenceisdenedby V(y):=v(y))]TJ /F6 11.955 Tf 11.95 0 Td[(v(cv))-221(hrv(cv);y)]TJ /F6 11.955 Tf 11.95 0 Td[(cvi;(4{92) 121

PAGE 122

wherecv:=argminv2Yv(y).ByincorporatingNesterov'ssmoothingtechnique,wecanapproximatef()bythesmoothfunction f(x):=maxy2YfhAx;yi)]TJ /F6 11.955 Tf 19.26 0 Td[((y))]TJ /F6 11.955 Tf 11.96 0 Td[(V(y)g;(4{93)whereiscalledthesmoothingparameter.Itwasshownin[ 2 ]thatf(x)hasLipschitz-continuousgradientwithconstant L:=kAk2=(v);(4{94)wherekAkistheoperatornormofA.Moreover,the\closeness"off()tof()dependslinearlyonthesmoothingparameter,i.e., f(x)f(x)f(x)+Dv;Y;8x2X;(4{95)where Dv;Y:=maxy;z2Yfv(y))]TJ /F6 11.955 Tf 11.96 0 Td[(v(z))-222(hrv(z);y)]TJ /F6 11.955 Tf 11.95 0 Td[(zig:(4{96)TheACSLmethodcouldbeviewedasincorporatingNesterov'ssmoothingtechniqueintotheACPLmethod,i.e.,insteadofconsideringtheobjectivefunctionmaxff(x))]TJ /F6 11.955 Tf 12.07 0 Td[(L;g(x)g,werstreplacef()withf(),thenapplytheACPLmethodon h(x;L):=maxff(x))]TJ /F6 11.955 Tf 11.96 0 Td[(L;g(x)g:(4{97)SimilartotheACPLmethod,theACSLmethodhasthestructureofouter-innerloops,andeachouteriterationcallsaninnerloop,theACSLgapreductionprocedureGACSL,toeitherincreaseL,thelowerestimateonf,orreducetheupperboundonh(x;L)byaconstantfactoruntilanapproximatesolutionwithrequiredaccuracyisfound.Let'sstartwiththeprocedureGACSL.InordertoguaranteetheterminationofGACSLandtheacceleratediterationcomplexityoftheACSLmethod,thestepsizefkgshouldbeproperlychosensuchthat( 4{72 )issatised.WenowaddafewremarksaboutprocedureGACSLdescribedabove.Firstly,thelinearapproximationsaboutf()insteadoff()areusedtodeneR k.However,thetermination 122

PAGE 123

TheACSLgapreductionprocedure:(x+;L+)=GACSL(^x;L;Dv;Y;;) 0: Leth(x):=maxff(x))]TJ /F6 11.955 Tf 12.63 0 Td[(L;g(x)g,h(x):=maxff(x))]TJ /F6 11.955 Tf 12.63 0 Td[(L;g(x)g,set h0=h(^x),l=(1)]TJ /F6 11.955 Tf 11.96 0 Td[() h0,R0=X,xu0=x0=^x,k=1and :=(h0)]TJ /F6 11.955 Tf 11.96 0 Td[(l)=(2Dv;Y):(4{98) 1: Updatethecuttingplanemodel:setxlkandlg(xlk)to( 4{11 )and( 4{13 ),respectively,andlf(xlk;x)=f(xlk)+f0(xlk);x)]TJ /F6 11.955 Tf 11.95 0 Td[(xlk; (4{99)R k:=fx2Rk)]TJ /F9 7.97 Tf 6.58 0 Td[(1:lf(xlk;x))]TJ /F6 11.955 Tf 11.96 0 Td[(Ll;lg(xlk;x)0g: (4{100) 2: Updatetheprox-centerandlowerestimateonf:ifR k=;:terminatewithx+=xuk)]TJ /F9 7.97 Tf 6.59 0 Td[(1;L+=L+l.Otherwise,solvexkby( 4{70 ). 3: Updatetheupperbound:set~xukto( 4{18 )and xuk=(~xuk;ifh(~xuk)
PAGE 124

d) Letfxig;i=1;2;:::;k,betheiteratesgeneratedbyGACSL,thenwehavePk)]TJ /F9 7.97 Tf 6.58 0 Td[(1i=0kxi+1)]TJ /F6 11.955 Tf -436.03 -14.44 Td[(xik2D2X;whereDXisdenedin( 4{25 ). Proof. ForPart a ,observethatf(x)f(x),thenforanyx2El,lf(xlk;x))]TJ /F6 11.955 Tf 9.71 0 Td[(Lf(x))]TJ /F6 11.955 Tf 9.7 0 Td[(Llandlg(xlk)g(x)0,usethesameinductioninLemma 13 ,wecanprovePart a .ThenPart b followsfromthefactf(x)f(x)too.AndPart c and d areasthesameasPart b inLemma 13 andLemma 14 ,respectively. NowweconsiderthenumberofiterationsperformedbyeachcallofGACSL. Proposition10. Ifthestepsizesfkgarechosensuchthat( 4{72 )holds,thenthenumberofiterationsperformedbyprocedureGACSLdoesnotexceed ^N():=cDX s Lg +p 2kAk r Dv;Y v!+1;(4{102)whereDXandDv;Yaredenedin( 4{25 )and( 4{96 )respectively. Proof. SupposeGACSLdoesnotterminateattheKthiterationforsomeK>0.SincetheACSLmethodcouldbeviewasapplyingtheGACSLprocedureonh(x;L),wheref()issmoothwithLipschitzconstantLandf=1,g()issmoothwithLipschitzconstantLgandg=1,similartotheargumentof( 4{78 ),wehaveforany1kK,h(xuk))]TJ /F6 11.955 Tf 11.95 0 Td[(l(1)]TJ /F6 11.955 Tf 11.95 0 Td[(k)(h(xuk)]TJ /F9 7.97 Tf 6.59 -0.01 Td[(1))]TJ /F6 11.955 Tf 11.95 0 Td[(l)+2kkxk)]TJ /F6 11.955 Tf 11.95 0 Td[(xk)]TJ /F9 7.97 Tf 6.58 0 Td[(1k2 2maxfL;Lgg (4{103)(1)]TJ /F6 11.955 Tf 11.95 0 Td[(k)(h(xuk)]TJ /F9 7.97 Tf 6.59 0 Td[(1))]TJ /F6 11.955 Tf 11.95 0 Td[(l)+2kkxk)]TJ /F6 11.955 Tf 11.95 0 Td[(xk)]TJ /F9 7.97 Tf 6.58 0 Td[(1k2 2(L+Lg): (4{104)Dividingbothsidesof( 4{103 )by2k,thensummingupfor1kK,use( 4{72 )andPart d inLemma 15 ,wehave h(xuK))]TJ /F6 11.955 Tf 11.95 0 Td[(lc2(L+Lg)D2X 2K2:(4{105)By( 4{95 )and( 4{98 ),wecanobtain hk)]TJ /F6 11.955 Tf 11.95 0 Td[(l=h(xuk))]TJ /F6 11.955 Tf 11.96 0 Td[(lh(xuk))]TJ /F6 11.955 Tf 11.96 0 Td[(l+Dv;Yc2(L+Lg)D2X 2K2+h0 2:(4{106) 124

PAGE 125

InviewoftheterminationcriteriainStep 3 ,wehavehk)]TJ /F6 11.955 Tf 11.96 0 Td[(l>,therefore, K0and;2(0;1). 1: Choosetheinitialpointp02X,computep1;L0;^x0;0asthesameasAlgorithm 10 ,sets=0. 2: Ifs,terminateandoutputapproximatesolution^xs. 3: Set(^xs+1;Ls+1)=GACPL(^xs;Ls;Dv;Y;;)ands+1=maxff(^xs+1))]TJ /F6 11.955 Tf 11.96 0 Td[(Ls+1;g(^xs+1)g. 4: Sets=s+1andgotoStep 2 .Wecontinuewiththenotationsofnon-ecientandecientphasestoclassifythecallstoGACSL.NotethattheACSLmethodusesthesamestrategytoupdateLastheACPLmethod,sothenumberofnon-ecientphasesandnumberofiterationsperformedbytheACSLmethodaresimilartothoseoftheACPLmethod.ThefollowingtheoremestablishestheiterationcomplexityoftheACSLmethodforsolving( 4{1 )-( 4{91 ). Theorem16. Foranygiven>0,ifthestepsizesfkginprocedureGACSLarechosensuchthat( 4{72 )holds,thenthefollowingstatementsholdfortheACSLmethodtocomputean-solutiontoproblem( 4{1 )-( 4{91 ). a) ThenumberofcallstoprocedureGACSLperformedbytheACSLmethoddoesnotexceed ^S:=2+f)]TJ /F6 11.955 Tf 11.96 0 Td[(L0 (1)]TJ /F6 11.955 Tf 11.96 0 Td[()+log1 q^VX ;(4{108)where^VX:=max2p 2kAkq Dv;Y v;LgD2X 2. 125

PAGE 126

b) ThetotalnumberofiterationsperformedbytheACSLmethodcanbeboundedby ^N:=f)]TJ /F6 11.955 Tf 11.95 0 Td[(L0 (1)]TJ /F6 11.955 Tf 11.96 0 Td[()+cDX s Lg +p 2kAk r Dv;Y v!1 1)]TJ 11.96 7.44 Td[(p q+1 1)]TJ /F6 11.955 Tf 11.95 0 Td[(q+^S;(4{109)whereDX;q;Dv;Yaredenedin( 4{25 ),( 4{74 )and( 4{96 )respectively. Proof. Denotethesetsofindicesofthenon-ecientandecientphasesperformedbytheACSLmethodbyfi1;i2;:::;imgandfj1;j2;:::;jng,respectively.SincetheACSLmethodusesthesamestrategytoupdateLastheACPLmethod,byTheorem 15 ,wecanboundthenumberofnon-ecientphasesbyT1()denedin( 4{86 ).Then,thenumberofiterationsperformedbynon-ecientphasesisboundedby ^N1:=T1()^N()=T1()+cDX s Lg +p 2kAk r Dv;Y v!T1():(4{110)Fortheecientphases,byLemma8in[ 26 ],wehavef(p1))]TJ /F6 11.955 Tf 11.96 0 Td[(lf(p0;p1)2p 2kAkr Dv;Y v:Togetherwith( 4{60 )andg=1,weconclude 0=h(p1;L0)^VX:(4{111)SincetheACSLmethodstillhas( 4{85 )byPart c inLemma 15 ,thenumberofecientphasesisboundedby ^T2():=1+log1 q^VX :(4{112)Moreover,thenumberofiterationsperformedbyecientphasesisboundedby^N2:=nXk=1^N(jk)
PAGE 127

Therefore,thetotalnumberofcallstoGACSLisboundedbyT1()+^T2(),andthetotalnumberofiterationsperformedbytheACSLmethodisboundedby^N1+^N2. NotethattheACSLmethodhastheiterationcomplexityOp Lg 3=2+kAk 2forsolvingtheconstrainedSPproblem( 4{1 )-( 4{91 ).Weonlyconsiderthecasethatobjectivefunctionf()isstructuredandgivenintheform( 4{91 )whilethefunctionalconstraintg()issmoothinginthissection,butwithminormodication,theACSLcouldalsoapplytoothercasesthattheobjectivefunctionf()issmoothing,weaklysmoothingornonsmoothwhilethefunctionalconstraintg()isaSPproblem. 127

PAGE 128

REFERENCES [1] A.S.NemirovskiandD.Yudin,Problemcomplexityandmethodeciencyinoptimiza-tion,ser.Wiley-InterscienceSeriesinDiscreteMathematics.JohnWiley,XV,1983. [2] Y.Nesterov,\Smoothminimizationofnon-smoothfunctions,"Mathematicalprogram-ming,vol.103,no.1,pp.127{152,2005. [3] Y.Nesterov,\Amethodforunconstrainedconvexminimizationproblemwiththerateofconvergenceo(1/k2),"vol.269,no.3,pp.543{547,1983. [4] Y.E.Nesterov,IntroductoryLecturesonConvexOptimization:abasiccourse.Massachusetts:KluwerAcademicPublishers,2004. [5] Y.Chen,G.Lan,andY.Ouyang,\Optimalprimal-dualmethodsforaclassofsaddlepointproblems,"SIAMJournalonOptimization,vol.24,no.4,pp.1779{1814,2014. [6] Y.HeandR.D.Monteiro,\Anacceleratedhpe-typealgorithmforaclassofcompositeconvex-concavesaddle-pointproblems,"SIAMJournalonOptimization,vol.26,no.1,pp.29{56,2016. [7] M.HongandZ.-Q.Luo,\Onthelinearconvergenceofthealternatingdirectionmethodofmultipliers,"MathematicalProgramming,pp.1{35,2012. [8] W.DengandW.Yin,\Ontheglobalandlinearconvergenceofthegeneralizedalternatingdirectionmethodofmultipliers,"JournalofScienticComputing,vol.66,no.3,pp.889{916,2016. [9] D.Goldfarb,S.Ma,andK.Scheinberg,\Fastalternatinglinearizationmethodsforminimizingthesumoftwoconvexfunctions,"MathematicalProgramming,vol.141,no.1-2,pp.349{382,2013. [10] R.D.MonteiroandB.F.Svaiter,\Iteration-complexityofblock-decompositionalgorithmsandthealternatingdirectionmethodofmultipliers,"SIAMJournalonOptimization,vol.23,no.1,pp.475{507,2013. [11] T.Goldstein,B.O'Donoghue,S.Setzer,andR.Baraniuk,\Fastalternatingdirectionoptimizationmethods,"SIAMJournalonImagingSciences,vol.7,no.3,pp.1588{1623,2014. [12] Y.Ouyang,Y.Chen,G.Lan,andE.PasiliaoJr,\Anacceleratedlinearizedalternatingdirectionmethodofmultipliers,"SIAMJournalonImagingSciences,vol.8,no.1,pp.644{681,2015. [13] J.Kelley,\Thecuttingplanemethodforsolvingconvexprograms,"JournaloftheSIAM,vol.8,pp.703{712,1960. [14] K.Kiwiel,\Proximitycontrolinbundlemethodsforconvexnondierentiableminimization,"MathematicalProgramming,vol.46,pp.105{122,1990. 128

PAGE 129

[15] K.C.Kiwiel,\Proximallevelbundlemethodsforconvexnondierentiableoptimization,saddle-pointproblemsandvariationalinequalities,"MathematicalProgramming,vol.69,no.1,pp.89{109,1995. [16] C.Lemarechal,A.Nemirovskii,andY.Nesterov,\Newvariantsofbundlemethods,"Mathematicalprogramming,vol.69,no.1-3,pp.111{147,1995. [17] A.Ben-TalandA.S.Nemirovski,\Non-Euclideanrestrictedmemorylevelmethodforlarge-scaleconvexoptimization,"MathematicalProgramming,vol.102,pp.407{456,2005. [18] W.vanAckooijandC.Sagastizabal,\Constrainedbundlemethodsforupperinexactoracleswithapplicationtojointchanceconstrainedenergyproblems,"SIAMJournalonOptimization,vol.24,no.2,pp.733{765,2014. [19] W.Oliveira,C.Sagastizabal,andS.Scheimberg,\Inexactbundlemethodsfortwo-stagestochasticprogramming,"SIAMJournalonOptimization,vol.21,no.2,pp.517{544,2011. [20] W.deOliveira,C.Sagastizabal,andC.Lemarechal,\Convexproximalbundlemethodsindepth:auniedanalysisforinexactoracles,"MathematicalProgramming,vol.148,no.1-2,pp.241{277,2014. [21] P.Richtarik,\Approximatelevelmethodfornonsmoothconvexminimization,"JournalofOptimizationTheoryandApplications,vol.152,no.2,pp.334{350,2012. [22] K.C.Kiwiel,\Aproximalbundlemethodwithapproximatesubgradientlinearizations,"SIAMJournalonoptimization,vol.16,no.4,pp.1007{1023,2006. [23] K.Kiwiel,\Bundlemethodsforconvexminimizationwithpartiallyinexactoracles,"Comp.Opt.Appl,vol.1702,p.1703,2012. [24] W.deOliveiraandC.Sagastizabal,\Levelbundlemethodsfororacleswithon-demandaccuracy,"OptimizationMethodsandSoftware,vol.29,no.6,pp.1180{1209,2014. [25] K.C.KiwielandC.Lemarechal,\Aninexactbundlevariantsuitedtocolumngeneration,"Mathematicalprogramming,vol.118,no.1,pp.177{206,2009. [26] G.Lan,\Bundle-leveltypemethodsuniformlyoptimalforsmoothandnonsmoothconvexoptimization,"MathematicalProgramming,vol.149,no.1-2,pp.1{45,2015. [27] D.Noll,\Bundlemethodfornon-convexminimizationwithinexactsubgradientsandfunctionvalues,"inComputationalandAnalyticalMathematics.Springer,2013,pp.555{592. [28] G.EmielandC.Sagastizabal,\Incremental-likebundlemethodswithapplicationtoenergyplanning,"ComputationalOptimizationandApplications,vol.46,no.2,pp.305{332,2010. 129

PAGE 130

[29] K.C.Kiwiel,\Aninexactbundleapproachtocutting-stockproblems,"INFORMSJournalonComputing,vol.22,no.1,pp.131{143,2010. [30] C.I.FabianandZ.Sz}oke,\Solvingtwo-stagestochasticprogrammingproblemswithleveldecomposition,"ComputationalManagementScience,vol.4,no.4,pp.313{353,2007. [31] H.Lin,\Aninexactspectralbundlemethodforconvexquadraticsemideniteprogramming,"ComputationalOptimizationandApplications,vol.53,no.1,pp.45{89,2012. [32] L.-P.PangandJ.Shen,\Anapproximatebundlemethodforsolvingvariationalinequalities,"CommunicationsinOptimizationTheory,vol.1,no.1,pp.1{18,2012. [33] C.I.Fabian,\Bundle-typemethodsforinexactdata,"CentralEuropeanJournalofOperationsResearch,vol.8,no.1,2000. [34] E.Karas,A.Ribeiro,C.Sagastizabal,andM.Solodov,\Abundle-ltermethodfornonsmoothconvexconstrainedoptimization,"MathematicalProgramming,vol.116,no.1,pp.297{320,2009. [35] W.vanAckooijandW.deOliveira,\Levelbundlemethodsforconstrainedconvexoptimizationwithvariousoracles,"ComputationalOptimizationandApplications,vol.57,no.3,pp.555{597,2014. [36] O.Devolder,F.Glineur,andY.Nesterov,\First-ordermethodsofsmoothconvexoptimizationwithinexactoracle,"MathematicalProgramming,vol.146,no.1-2,pp.37{75,2014. [37] U.Brannlund,K.C.Kiwiel,andP.O.Lindberg,\Adescentproximallevelbundlemethodforconvexnondierentiableoptimization,"OperationsResearchLetters,vol.17,no.3,pp.121{126,1995. [38] J.B.CruzandW.deOliveira,\Levelbundle-likealgorithmsforconvexoptimization,"JournalofGlobalOptimization,vol.59,no.4,pp.787{809,2014. [39] A.Astorino,A.Frangioni,M.Gaudioso,andE.Gorgone,\Piecewise-quadraticapproximationsinconvexnumericaloptimization,"SIAMJournalonOptimization,vol.21,no.4,pp.1418{1438,2011. [40] A.Ouorou,\Aproximalcuttingplanemethodusingchebychevcenterfornonsmoothconvexoptimization,"MathematicalProgramming,vol.119,no.2,pp.239{271,2009. [41] Mosek,\Themosekoptimizationtoolboxformatlabmanual.version6.0(revision93),"http://www.mosek.com. [42] S.Becker,J.Bobin,andE.J.Candes,\Nesta:afastandaccuraterst-ordermethodforsparserecovery,"SIAMJournalonImagingSciences,vol.4,no.1,pp.1{39,2011. 130

PAGE 131

[43] Y.Chen,W.Hager,F.Huang,D.Phan,X.Ye,andW.Yin,\Fastalgorithmsforimagereconstructionwithapplicationtopartiallyparallelmrimaging,"SIAMJournalonImagingSciences,vol.5,no.1,pp.90{118,2012. [44] J.Malick,W.deOliveira,andS.Zaourar,\Uncontrolledinexactinformationwithinbundlemethods,"EUROJournalonComputationalOptimization,vol.5,no.1-2,pp.5{29,2017. [45] G.Zakeri,A.B.Philpott,andD.M.Ryan,\Inexactcutsinbendersdecomposition,"SIAMJournalonOptimization,vol.10,no.3,pp.643{657,2000. [46] M.Hintermuller,\Aproximalbundlemethodbasedonapproximatesubgradients,"ComputationalOptimizationandApplications,vol.20,no.3,pp.245{266,2001. [47] M.V.Solodov,\Onapproximationswithniteprecisioninbundlemethodsfornonsmoothoptimization,"JournalofOptimizationTheoryandApplications,vol.119,no.1,pp.151{165,2003. [48] O.Devolder,F.Glineur,Y.Nesterovetal.,\First-ordermethodswithinexactoracle:thestronglyconvexcase,"UCL,Tech.Rep.,2013. [49] S.Villa,S.Salzo,L.Baldassarre,andA.Verri,\Acceleratedandinexactforward-backwardalgorithms,"SIAMJournalonOptimization,vol.23,no.3,pp.1607{1633,2013. [50] A.BeckandM.Teboulle,\Smoothingandrstordermethods:Auniedframework,"SIAMJournalonOptimization,vol.22,no.2,pp.557{580,2012. [51] M.Schmidt,N.LeRoux,andF.Bach,\Minimizingnitesumswiththestochasticaveragegradient,"MathematicalProgramming,vol.162,no.1-2,pp.83{112,2017. [52] K.Jiang,D.Sun,andK.-C.Toh,\Aninexactacceleratedproximalgradientmethodforlargescalelinearlyconstrainedconvexsdp,"SIAMJournalonOptimization,vol.22,no.3,pp.1042{1064,2012. [53] Y.Chen,G.Lan,Y.Ouyang,andW.Zhang,\Fastbundle-leveltypemethodsforunconstrainedandball-constrainedconvexoptimization,"arXivpreprintarXiv:1412.2128,2014. [54] D.P.Bertsekas,Nonlinearprogramming.AthenascienticBelmont,1999. [55] K.C.Kiwiel,Methodsofdescentfornondierentiableoptimization.Springer,2006,vol.1133. [56] C.I.Fabian,\Computationalaspectsofrisk-averseoptimizationintwo-stagestochasticmodels,"2013. [57] W.d.OliveiraandC.Sagastizabal,\Bundlemethodsinthexxistcentury:Abird's-eyeview,"PesquisaOperacional,vol.34,no.3,pp.647{670,2014. 131

PAGE 132

BIOGRAPHICALSKETCHWeiZhangwasborninHuanggang,HubeiProvince,China.In2011,hereceivedhisBachelorofScienceinmathematicsfromZhejiangUniversity,Hangzhou,China.InAugust2011,hejoinedthegraduateprogramoftheDepartmentofMathematicsinUniversityofFloridaundertheadvisorsofProf.YunmeiChenandProf.GuanghuiLan.Hisresearchinterestsareconvexoptimizationandmathematicalmodeling.HereceivedhisMasterofScienceinmathematicsin2013andMasterofScienceincomputersciencein2016,andexpectstoreceivehisPh.DfromtheDepartmentofMathematicsinAugust2017. 132