Citation
Fast First-Order Optimization Methods with Applications to Inverse Problems

Material Information

Title:
Fast First-Order Optimization Methods with Applications to Inverse Problems
Creator:
Li, Xianqi
Publisher:
University of Florida
Publication Date:
Language:
English

Thesis/Dissertation Information

Degree:
Doctorate ( Ph.D.)
Degree Grantor:
University of Florida
Degree Disciplines:
Mathematics
Committee Chair:
CHEN,YUNMEI
Committee Co-Chair:
RAO,MURALI
Committee Members:
MCCULLOUGH,SCOTT A
BOYLAND,PHILIP LEWIS
WU,DAPENG

Subjects

Subjects / Keywords:
imaging
network
optimization

Notes

General Note:
This dissertation is devoted to developing a series of fast first-order methods in optimization with theoretical guarantees leveraging Barzilai-Borwein (BB) method, non-monotone backtracking strategies and acceleration schemes by Nesterov. The research is motivated by demanding efficient algorithms for solving large-scale linear inverse problems arising from imaging science and network structure learning areas. In the first part of this work, we develop two accelerated Bregman Operator Splitting (BOS) algorithms with backtracking for the convex composite optimization problems, where the non-smooth component demonstrates complicated structures. The first algorithm improves the rate of convergence for Bregman Operator Splitting with Variable Stepsize (BOSVS) in terms of the smooth component in the objective function by incorporating multi-step acceleration scheme by Nesterov under the assumption that the feasible set is bounded. The second algorithm is capable of dealing with the case where the feasible set is unbounded. Both algorithms exhibit better practical performance than BOSVS and Accelerated Alternating Direction Method of Multipliers (AADMM) , while preserve the same accelerated rate of convergence as that for AADMM. In the second part of this dissertation, we are concerned about the modeling and fast structure learning of causality network using high-dimensional time series in presence of unobserved latent variables. We introduce a novel approach of low-rank and sparse and/or group-sparse vector autoregression (VAR) in order to accurately estimate the network of Granger causal interactions after accounting for latent effects. We argue that in presence of a few latent pervasive factors, the transition matrix of a misspecified VAR model among the observed series can be approximated as the sum of a low-rank and a sparse and/or a group-sparse component. Further, we establish non-asymptotic upper bounds on the estimation error rates of the low-rank and the sparse and/or group-sparse components. Moreover, we propose a fast accelerated proximal gradient method to estimate the network structure of interest. The rate of convergence of the proposed method is established. The last part of this work concerns the information extraction for Scanning Tunneling Potentiometry (STP) and Scanning Tunneling Microscope (STM). We develop novel methods for extracting two-dimensional (2D) conductivity profiles from large electrochemical potential datasets acquired by scanning tunneling potentiometry of a 2D conductor. The method consists of a data preprocessing procedure to reduce/eliminate noise and a numerical conductivity reconstruction. An accelerated ADMM-type algorithm is employed to speed up the conductivity reconstruction. Moreover, we propose an image postprocessing framework for Scanning Tunneling Microscope (STM) to reduce the strong spurious oscillations and scan line noise at fast scan rates and preserve the features, allowing an order of magnitude increase in the scan rate without upgrading the hardware.

Record Information

Source Institution:
UFRGP
Rights Management:
All applicable rights reserved by the source institution and holding location.
Embargo Date:
5/31/2019

Downloads

This item has the following downloads:


Full Text

PAGE 1

&(57,),&$7(RI&203/(7,21 RIWKH 6XUYH\RI(DUQHG'RFWRUDWHV 4XHVWLRQQDLUH 7KLVFHUWLILHVWKDW;LDQTL/L DVWXGHQWDW8QLYHUVLW\RI)ORULGD*UDGXDWH6FKRRO KDVFRPSOHWHGWKH6XUYH\RI(DUQHG'RFWRUDWHV RQ &RQILUPDWLRQ&RGH.+)*



PAGE 1

FASTFIRST-ORDEROPTIMIZATIONMETHODSWITHAPPLICATIONSTOINVERSEPROBLEMSByXIANQILIADISSERTATIONPRESENTEDTOTHEGRADUATESCHOOLOFTHEUNIVERSITYOFFLORIDAINPARTIALFULFILLMENTOFTHEREQUIREMENTSFORTHEDEGREEOFDOCTOROFPHILOSOPHYUNIVERSITYOFFLORIDA2018

PAGE 2

c2018XianqiLi

PAGE 3

Tomyfamily

PAGE 4

ACKNOWLEDGMENTSManypeoplehaveguidedandsupportedmethroughoutmygraduatestudy.Theirencouragement,knowledgeandperceptionscontinuouslysparkedmycuriosityregardingthosebeautifultheoriesandtheirapplications.Iwouldliketoexpressmydeepestappreciationtothem.Iwouldliketosincerelythankmyadvisor,Dr.YunmeiChen,forherfruitfuldiscussion,encouragement,criticismandadvice.IamtrulygratefultoDr.GeorgeMichailidisandDr.XiaoguangZhangfortheirsuggestionsandsupport.IamalsoverygratefultoDr.EduardoPasiliaoforhisgeneroussponsorshipsothatIhavemoretimetoconductmyresearchwork.IwouldalsoliketothankDr.PhilipBoyland,Dr.ScottMcCullough,Dr.MuraliRaoandDr.DapengWu,forservingasmydoctoralcommitteemembers.Theirguidanceandsuggestionsareinvaluabletothecompletionofthisdissertation.Then,Iwouldliketothankmyfriendsandgroupmembers,especiallyQilinZhang,HaoZhang,WeiZhang,YuyuanOuyang,fortheirdiscussionandencouragement.Finally,andmostimportantly,Iwouldliketothankmyfamily,especiallymyparents,mybrothers,mymother-in-lawandmywife,forthecontinuoussupportandendlesslovethattheyhavegivenmethroughoutmyPh.D.study.Icouldnothavecompleteditwithoutthem. 4

PAGE 5

TABLEOFCONTENTS page ACKNOWLEDGMENTS ................................... 4 LISTOFTABLES ...................................... 7 LISTOFFIGURES ..................................... 8 ABSTRACT ......................................... 10 CHAPTER 1INTRODUCTION ................................... 12 1.1First-OrderMethodsForConvexOptimization ................. 12 1.1.1AcceleratedFirst-OrderOptimizationMethods ............. 13 1.1.2BBMethodandBacktrackingStrategies ................ 16 1.2Applications ................................... 18 1.2.1TotalVariationBasedImageReconstruction .............. 18 1.2.2StructuredNetworkLearning ....................... 19 1.3OutlineandMainResultsofTheDissertation .................. 21 2ACCELERATEDBREGMANOPERATORSPLITTINGWITHBACKTRACKING .. 23 2.1OverviewofOperatorSplittingMethods .................... 23 2.2ProposedAlgorithms ............................... 28 2.3ConvergenceAnalysis .............................. 31 2.4NumericalResults ................................ 45 2.4.1Total-VariationBasedImageReconstruction .............. 45 2.4.2PartiallyParallelImaging ......................... 48 2.5ConclusionsofThisChapter ........................... 50 3MODELLINGANDFASTSTRUCTURELEARNINGOFCAUSALITYNETWORK 53 3.1OverviewofStructuredNetworkLearning .................... 53 3.2ModelFormulationandEstimationProcedure ................. 56 3.3ComputationalAlgorithmsandConvergenceAnalysis .............. 59 3.4ErrorBoundAnalysis .............................. 65 3.5PerformanceEvaluations ............................. 74 3.5.1PerformanceMetricsandExperimentalSettings ............. 74 3.5.2Large-ScaleSparseNetworkLearning .................. 75 3.5.3NetworkLearningWithLow-RankTransitionMatrices ......... 76 3.5.4SparsePlusGroup-SparseNetworkLearningProblem .......... 79 3.5.5SparsePlusLow-RankNetworkLearning ................ 81 3.5.6SparsePlusGroup-SparsePlusLow-RankNetworkLearning ...... 84 3.6ConclusionsofThisChapter ........................... 85 5

PAGE 6

4INFORMATIONEXTRACTIONFORSCANNINGTUNNELINGPOTENTIOMETRYANDMICROSCOPEIMAGING ............................ 88 4.1OverviewofScanningTunnelingPotentiometry/Microscope .......... 88 4.2ProposedMethods ................................ 95 4.2.1ExtractionofConductivity ........................ 95 4.2.1.1GeneralEquations ....................... 95 4.2.1.2TotalVariation(TV)Method ................. 96 4.2.2Preprocessing ............................... 99 4.2.2.1ImageRegistration ....................... 100 4.2.2.2ImageRestoration ....................... 102 4.2.3LinebyLineBackgroundRemoval .................... 104 4.2.4ImagePostprocessing ........................... 105 4.2.5ConstructionofTheRankingMap .................... 109 4.3ValidationwithExperimentalData ....................... 110 4.4ConclusionsofThisChapter ........................... 113 REFERENCES ........................................ 115 BIOGRAPHICALSKETCH ................................. 124 6

PAGE 7

LISTOFTABLES Table page 2-1ParametersettingsforABOSVS-IandABOSVS-IIin2.4.1.Notethatandareonlyusedintherstinstance ............................. 48 2-2ParametersettingsforABOSVS-IIin2.4.2 ...................... 50 2-3Comparisonofobjectivefunctionvalue,relativeerror,andCPUtimeinsecondsusingdata1 .......................................... 50 2-4Comparisonofobjectivefunctionvalue,relativeerror,andCPUtimeinsecondsusingdata2 .......................................... 52 3-1Parametersettingsintheproposedalgorithmsforalltheexperiments. ........ 75 3-2PerformancecomparisonofFNSLwithFISTAonlarge-scalesparsenetworkstructurelearningproblem. .................................... 76 3-3PerformancecomparisonofFISTAandFNSLonestimationoflow-ranktransitionmatricesproblems. .................................. 78 3-4PerformancecomparisonofS+GwithLassoandSGLonsparseplusgroup-sparsenetworkidenticationproblem. ............................ 79 3-5TruepositiverateandfalsealarmrateoftheL+SmodelonidentifyingthesparsecomponentSwithdierent. ............................. 81 3-6PerformancecomparisonofL+SwithOLSandLasso. ................ 82 3-7PerformancecomparisonofL+S+GwithS+GandL+S. .............. 84 7

PAGE 8

LISTOFFIGURES Figure page 2-1Theobjectivefunctionvaluesandrelativeerrorvs.CPUtimeforrstinstance ... 46 2-2Theobjectivefunctionvaluesandrelativeerrorvs.CPUtimeforsecondinstance .. 47 2-3Sensitivitymapsofdata1 ............................... 48 2-4TrueimageandCartesianmaskfordata1 ....................... 49 2-5ComparisonofimagereconstructionandtheirdierenceswithtrueimagebyBOSVS,TVAL3,ALADMML,andABOSVS-IIusingdata1. .................. 51 2-6Sensitivitymapsofdata2 ............................... 51 2-7TrueimageandCartesianmaskfordata2 ....................... 51 2-8ComparisonofimagereconstructionandtheirdierenceswithtrueimagebyBOSVS,TVAL3,ALADMML,andABOSVS-IIusingdata2 .................. 52 3-1Theobjectivefunctionvaluesvs.CPUtimeforsparsenetworklearningproblemwithp=1000andN=2000. ............................ 77 3-2Theobjectivefunctionvaluesvs.CPUtimeforlow-ranktransitionmatrixestimationproblemwithwithp=400andN=2000. ...................... 78 3-3TruenetworkstructureofS+G,SandGwithp=50andN=200. ........ 80 3-4Networkstructureidentiedby^S+^G,LassoandSGL. ............... 80 3-5EstimatedGrangercausalnetworksusinglassoandlow-rankplussparseVARestimates. ............................................. 83 3-6EstimatedGrangercausalnetworksusinglow-rankplussparseplusgroup-sparseVARestimates. .................................... 86 3-7EstimatedGrangercausalnetworksusinglow-rankplussparseandgroup-sparseVARestimates. ....................................... 86 4-1OriginalSTPdataofagraphenegrainboundaryonaSiO2/Sisubstrate ...... 89 4-2STMdataofasinglecrystalCu(111)surfaceobtainedwithafastscanrate(0.1msecperpixel) ..................................... 92 4-3ComparisonofprocessedtopographyimagesoftheCusampleinFig.4-2 ...... 93 4-4ProcessedatomicresolutionimageoftheCusamplefromthefastscandatainFig.4-2 ........................................... 94 4-5AverageoftheforwardandbackwardscansinFig.4-2. ............... 94 8

PAGE 9

4-6DierenceoftheforwardandbackwardscansinFig.4-2. .............. 95 4-7Comparisonoftheaveragedpotentialc=(f+b)=2. .............. 99 4-8Comparisonoftheabsolutedierencesofjf)]TJ /F3 11.955 Tf 11.96 0 Td[(bj. ................. 100 4-9Comparisonoftheobjectivefunctionvalue(kAk2)versustheCPUtime. ..... 103 4-10Illustrationoftherubberbandmethodusingsimulateddata. ............. 107 4-11Thealignedforwardandbackwarddatafromthesamelineofscans. ........ 107 4-12Comparisonofextractedxusingtheproposedmethod. ............... 111 4-13Comparisonofextractedyusingtheproposedmethod. ............... 112 4-14Distributionofthevaluesofun-normalizedx. .................... 112 9

PAGE 10

AbstractofDissertationPresentedtotheGraduateSchooloftheUniversityofFloridainPartialFulllmentoftheRequirementsfortheDegreeofDoctorofPhilosophyFASTFIRST-ORDEROPTIMIZATIONMETHODSWITHAPPLICATIONSTOINVERSEPROBLEMSByXianqiLiMay2018Chair:YunmeiChenMajor:MathematicsThisdissertationisdevotedtodevelopingaseriesoffastrst-ordermethodsinoptimizationwiththeoreticalguaranteesleveragingBarzilai-Borwein(BB)method,non-monotonebacktrackingstrategiesandNesterov'saccelerationschemes.Theresearchismotivatedbydemandingecientalgorithmsforsolvinglarge-scalelinearinverseproblemsarisingfromimagingscienceandnetworkstructurelearningareas.Intherstpartofthiswork,wedeveloptwoacceleratedBregmanOperatorSplitting(BOS)algorithmswithbacktrackingfortheconvexcompositeoptimizationproblems,wherethenon-smoothcomponentdemonstratescomplicatedstructures.TherstalgorithmimprovestherateofconvergenceforBregmanOperatorSplittingwithVariableStepsize(BOSVS)intermsofthesmoothcomponentintheobjectivefunctionbyincorporatingNesterov'smulti-stepaccelerationschemeundertheassumptionthatthefeasiblesetisbounded.Thesecondalgorithmiscapableofdealingwiththecasewherethefeasiblesetisunbounded.BothalgorithmsexhibitbetterpracticalperformancethanBOSVSandAcceleratedAlternatingDirectionMethodofMultipliers(AADMM),whilepreservethesameacceleratedrateofconvergenceasthatforAADMM.Inthesecondpartofthisdissertation,weareconcernedaboutthemodelingandfaststructurelearningofcausalitynetworkusinghigh-dimensionaltimeseriesinpresenceofunobservedlatentvariables.Weintroduceanovelapproachoflow-rankandsparseand/orgroup-sparsevectorautoregression(VAR)inordertoaccuratelyestimatethenetworkof 10

PAGE 11

Grangercausalinteractionsafteraccountingforlatenteects.Wearguethatinpresenceofafewlatentpervasivefactors,thetransitionmatrixofamisspeciedVARmodelamongtheobservedseriescanbeapproximatedasthesumofalow-rankandasparseand/oragroup-sparsecomponent.Further,weestablishnon-asymptoticupperboundsontheestimationerrorratesofthelow-rankandthesparseand/orgroup-sparsecomponents.Moreover,weproposeafastacceleratedproximalgradientmethodtoestimatethenetworkstructureofinterest.Therateofconvergenceoftheproposedmethodisestablished.ThelastpartofthisworkconcernstheinformationextractionforScanningTunnelingPotentiometry(STP)andScanningTunnelingMicroscope(STM).Wedevelopnovelmethodsforextractingtwo-dimensional(2D)conductivityprolesfromlargeelectrochemicalpotentialdatasetsacquiredbyscanningtunnelingpotentiometryofa2Dconductor.Themethodconsistsofadatapreprocessingproceduretoreduce/eliminatenoiseandanumericalconductivityreconstruction.AnacceleratedADMM-typealgorithmisemployedtospeeduptheconductivityreconstruction.Moreover,weproposeanimagepostprocessingframeworkforScanningTunnelingMicroscope(STM)toreducethestrongspuriousoscillationsandscanlinenoiseatfastscanratesandpreservethefeatures,allowinganorderofmagnitudeincreaseinthescanratewithoutupgradingthehardware. 11

PAGE 12

CHAPTER1INTRODUCTIONInthischapter,werstintroducesomebackgroundofrst-ordermethodsinconvexoptimization.Inparticular,wereviewtwotypesofeort,whicharecloselyrelatedtoourwork,forimprovingtheoreticalandpracticalperformanceofrst-ordermethods.Oneisonimprovingiterationcomplexityoracceleratingconvergencerateofrst-ordermethodsbyincorporatingNesterov'saccelerationschemes.Theotheroneisaboutdevelopingbacktrackingstrategiestosearchforlargerstepsize.Thenwereviewsomelinearinverseproblemsofinterestarisingfromimagingscienceandnetworkstructurelearningareas,anddiscusstheassociatedoptimizationproblems.Thelastpartconcludesourmainresultsforthisdissertation. 1.1First-OrderMethodsForConvexOptimizationInthissection,weintroducebasicknowledgeofrst-ordermethodsinconvexoptimizationanddiscusssomeclassicrst-orderalgorithms.Considerthefollowingoptimizationproblem minx2XF(x),(1{1)whereXisaconvexsetandFisaLipschitzcontinuousconvexfunctionoverX.Basically,rst-ordermethodsonlyexploitinformationontheobjectivefunctionvalueF(x)andits(sub)gradientF0(x)2@F(x)toiterativelyndanapproximationsolution^xof( 1{1 ).Hence,theyareveryattractivetolarge-scaleoptimizationproblemswithlowaccuracyrequirement,suchasmachinelearningandstatisticalinference,duetothecheapcostofeachiteration.Manyrst-ordermethodshavebeenproposedfor( 1{1 )underdierentassumptionsinthepastfewdecades,suchasgradientdescent/subgradientmethod[ 16 91 ],mirrordescentmethod[ 6 75 ],conditionalgradientmethod[ 36 ],acceleratedgradientmethods[ 76 80 ],justtonameafew.Nextwediscusssomeoftheexistingmethodsandtechniques,whichisabletoenhancetheperformanceofrst-ordermethods,fromtwoaspects.Oneisonimprovingiterationcomplexityfor( 1{1 )toreachfasterconvergenceintermsoftheobjectivefunction 12

PAGE 13

values.TheotheroneisaboutpairingBarzilai-Borwein(BB)methodwithbacktrackingstrategies.Firstwediscusstheapproachestoimprovingiterationcomplexity. 1.1.1AcceleratedFirst-OrderOptimizationMethodsInthissubsection,weintroducetheworktoenhancetheperformanceofrst-ordermethodsfromtheaspectofimprovingiterationcomplexityorreducingthenumberoffunctionanditsgradientevaluationsforndinganapproximatesolution.WemainlyfocusontheaccelerationschemesproposedbyNesterov.AssumingF(x)iscontinuouslydierentiablewithLipschitzcontinuousgradientLoverX: krF(x))-222(rF(y)kLkx)]TJ /F4 11.955 Tf 11.95 0 Td[(yk(1{2)foreveryx,y2X,wherekkisthestandardEuclideannorm,L>0isLipschitzconstantofrF,Xisanite-dimensionalvectorspace,andxisaminimizerof( 1{1 ),thenforanyapproximatesolution^xof( 1{1 )inX,^xisan)]TJ /F1 11.955 Tf 13.2 0 Td[(solutionofproblem( 1{1 )if F(^x))]TJ /F4 11.955 Tf 11.96 0 Td[(F(x),(1{3)Toreachthe)]TJ /F1 11.955 Tf 13.2 0 Td[(solutionof( 1{1 ),traditionalrst-ordermethodsusuallyneedO(Lkx0)]TJ /F7 7.97 Tf 6.59 0 Td[(xk )iterations.ThebreakthroughoccurredwhenNesterovproposedamethod[ 76 ]in1983,whichisgivenasAlgorithm 1.1 .Itisabletoobtainan)]TJ /F1 11.955 Tf 13.2 0 Td[(solutionofproblem( 1{1 )withO(q Lkx0)]TJ /F7 7.97 Tf 6.59 0 Td[(xk )iterationsatmost.Intermsoftherateofconvergence,itcanbeexpressedas F(xk))]TJ /F4 11.955 Tf 11.95 0 Td[(F(x)O(Lkx0)]TJ /F4 11.955 Tf 11.95 0 Td[(xk k2),(1{4)wherekisiterationcounter.Indeed,Nesterov'smethodhasbeenproventobeoptimalintermsofrateofconvergencebyshowingthatO(1 k2)isunbeatableforanyrst-ordermethod[ 80 ],i.e.foranykwith1k1 2(n)]TJ /F5 11.955 Tf 12.22 0 Td[(1)andanyL,thereexistsafunctionfwithgradientLipschitzconstantLsuchthatforanyrst-ordermethod,thek-thiteratexkalwayssatisesf(xk))]TJ /F4 11.955 Tf 11.95 0 Td[(f(x)3Lkx0)]TJ /F4 11.955 Tf 11.95 0 Td[(xk 32(k+1)2, 13

PAGE 14

wherexisaminimizeroff. Algorithm1.1. Nesterov'sAcceleratedMethod-SchemeI Takey1=x02
PAGE 15

ThemainconvergenceresultestablishedbyNesterovregardingAlgorithm 1.1 and 1.2 issummarizedinthefollowingtheorems. Theorem1.1. Letthesequencesfxkg,fykgbegeneratedviaAlgorithm 1.1 .Thenforanyk1 F(xk))]TJ /F4 11.955 Tf 11.96 0 Td[(F(x)2Lkx0)]TJ /F4 11.955 Tf 11.96 0 Td[(xk2 (k+1)2.(1{5)wherex2XisaminimizerofF. Theorem1.2. LetthesequencesfxagkgbegeneratedviaAlgorithm 1.2 .Thenforanyk1 F(xagk))]TJ /F4 11.955 Tf 11.95 0 Td[(F(x)2Lkx1)]TJ /F4 11.955 Tf 11.95 0 Td[(xk2 (k+1)2.(1{6)Inmanyapplications,inordertoincorporatethepriorinformationfromtheunknowntruetarget,suchassparsityorgroupsparsity,aregularizationtermisusuallyaddedto( 1{1 ),thentheoptimizationproblemscanbecastintothefollowingcompositeform: minx2Xnl(x):=F(x)+G(Bx)o,(1{7)whereG(Bx)isapossiblynon-smoothconvexfunction,BisaboundedlinearoperatorandF(x)satises( 1{2 ).G(Bx)isusuallychosenbasedonthepriorinformationofx.Forexample,wesetG(Bx)=kxk1ifxitselfissparseandG(Bx)=kxkTV:=Pni=1k(rx)ikifxissparseingradientdomain.Numerousalgorithmshavebeenproposedtosolve( 1{7 )basedoncomplexityofG(Bx),suchasproximalgradientmethods[ 7 27 35 78 ],smoothingmethod[ 8 79 ],primal-dualmethods[ 17 21 44 50 116 118 ]andoperatorsplittingmethods[ 2 41 42 54 72 81 ].WhenG(Bx)isasimplenon-smoothconvexfunction,i.e.theproximalmapcanbecomputedeasily,itisknowntousthattheoptimalrateofconvergencefortheclassofconvexoptimizationproblem( 1{7 )isO(L k2).Otherwise,in[ 79 ],NesterovshowedthattheoptimalrateofconvergenceforthisoptimizationproblemisO(L k2+LB k),whereLBisanupperboundoftheinducednormofthelinearoperatorB.Indeed,thisresultsignicantlyimprovesthepreviousboundO(1 p k).Forthiscase,insteadofusingsmoothingtechniqueasthatin 15

PAGE 16

[ 79 ],ourworkismorecloselyrelatedwithprimal-dualmethodsandoperatorsplittingmethods,bothofwhicharewell-suitedforsolvingproblem( 1{7 )duetothat( 1{7 )canberespectivelyformulatedas minx2Xmaxy2YF(x)+hBx,yi)]TJ /F4 11.955 Tf 19.26 0 Td[(G(y),(1{8)whereXandYareclosedconvexsetsandGistheconvexconjugateofG,and minx2X,w2WF(x)+G(w)s.t.w=Bx,(1{9)whereWisanitedimensionalvectorspace.Further,( 1{9 )isequivalenttothefollowingoptimizationproblem: minx2X,w2Wmax2YF(x)+G(w))-222(h,w)]TJ /F4 11.955 Tf 11.96 0 Td[(Bxi,(1{10)whereYisanitedimensionalvectorspace.Infact,bothof( 1{8 )and( 1{10 )aresaddlepointsproblemsandhavebeenstudiedbymanyresearchers.Particularly,thealgorithmsproposedbyChenetc.[ 21 ]andOuyangetc.[ 81 ]for( 1{8 )and( 1{10 )respectivelyconvergewithrateofO(L k2+LB k),whichisoptimal,basedonNesterov'swork[ 79 ].Inaddition,ifeitherForGisstronglyconvex,therateO(1 k2)isobtainedrespectivelyfor( 1{8 )in[ 17 ]and( 1{9 )in[ 106 ].Inthisdissertation,weproposenewmethodsleveragingNesterov'saccelerationframeworkforthosestructuredoptimizationproblems.NextwereviewBBmethodandsomeclassicbacktrackingstrategies.Thosetechniquesplayasignicantroleonimprovingpracticalperformanceof(accelerated)rst-ordermethods. 1.1.2BBMethodandBacktrackingStrategiesBBmethodisinitiallyproposedbyBarzilaiandBorweinin[ 3 ]in1988.Ithasbeenprovenveryhelpfultoimprovethepracticalperformanceofsomerst-ordermethods.TheessentialideabehindBBmethodismotivatedbyNewton'smethodbutdoesnotneedtocomputeHessian.SupposingF(x)issmoothin( 1{1 ),atypicalformofNewton'smethodforsolving 16

PAGE 17

( 1{1 )is xk+1=xk)]TJ /F5 11.955 Tf 11.95 0 Td[([r2F(xk)])]TJ /F6 7.97 Tf 6.59 0 Td[(1rF(xk),(1{11)However,computingHessian[r2F(xk)])]TJ /F6 7.97 Tf 6.59 0 Td[(1couldbeveryexpensive,especiallywhenthedimensionoftheproblemislarge.Tohandlethisdiculty,researchersndaway,theso-calledBBmethod,tocomputeHessianapproximately.Morespecically,thismethodsndsascalarmatrixkIsothatkI[r2F(xk)])]TJ /F6 7.97 Tf 6.59 0 Td[(1rF(xk)byminkkxk)]TJ /F5 11.955 Tf 11.95 0 Td[(gkk2,wherexk=xk)]TJ /F4 11.955 Tf 11.96 0 Td[(xk)]TJ /F6 7.97 Tf 6.58 0 Td[(1andgk=rF(xk))-222(rF(xk)]TJ /F6 7.97 Tf 6.58 0 Td[(1),whichleadsto k=(gk)Txk kxkk2.(1{12)WhenF(x)isquadratic,thismethodhasR-linearconvergence[ 26 ].However,thereisnoconvergenceguaranteeforthegeneralsmoothconvexoptimizationproblems.Indeed,thismethodisoftenemployedalongwithnon-monotonebacktrackingmethodsasaconvergencesafeguardfornon-quadraticproblems.ForgeneralizationsandvariantsoftheBBmethod,interestedreadersarereferredto[ 83 84 ].Innonlinearoptimizationeld,backtrackingisalinesearchmethod,whichdeterminesthemaximumamounttomovealongagivensearchdirection.Specically,givenastartingpointxandasearchdirectionp,theaimofbacktrackingistondastepsizeaslargeaspossibletodecreaseorincreasetheobjectivefunctionvalueofF(x).Foraminimizationproblem,thedecreasingoftheobjectivefunctionvaluecouldbeinamonotoneornon-monotonemanneraslongastheoverallfunctionvalueseventuallydecrease.Inthisdissertation,wefocusonnon-monotonebacktrackingstrategies.Firstwereviewsomeclassicnon-monotonebacktrackingtechniques.Anearlynon-monotonebacktrackingtechniqueisdevelopedbyGrippoetc.[ 46 ]in1986forNewton'smethod.Thekeyideaforthismethodliesinthatthestepsizeisselectedbasedonthefollowingrule:thefunctionvalueofeachnewiteratessatisfyanArmijo'sconditionwithrespecttothemaximumfunctionvalueofaprexednumberof 17

PAGE 18

perviousiteratesmk,i.e.F(xk+kpk)max0jmk[F(xk)]TJ /F7 7.97 Tf 6.59 0 Td[(j)]+k(rF(xk))Tpk,where2(0,1),kandpkarethestepsizeandsearchdirectionatkthiterate.Anotherclassicnon-monotonebacktrackingtechniqueisproposedbyZhangandHager[ 114 ].Insteadofrequiringthatamaximumofrecentfunctionvaluesdecreasesin[ 46 ],theschemebyZhangandHagerdemandsthatanaverageofthesuccessivefunctionvaluesdecreases.ThestepsizeisselectedtosatisfyF(xk+kpk)Ck+k(rF(xk))Tpk,whereCk=(Qk)]TJ /F6 7.97 Tf 6.59 0 Td[(1Ck)]TJ /F6 7.97 Tf 6.58 0 Td[(1+F(xk))=Qk,Qk=Qk)]TJ /F6 7.97 Tf 6.59 0 Td[(1+1,Q0=1,2(0,1)andC0=F(x0).Inthisway,theirmethodusesfewerfunctionandgradientevaluationsonaverage.Formanyapplications,BBmethod,pairingwithnon-monotonebacktrackingstrategiesusuallyshowssignicantimprovementonpracticalperformance,whichinspiresustoincorporatethosetechniquesintoourdevelopedmethods. 1.2ApplicationsInverseproblemshavemanyapplicationsinsignal/imageprocessing,statisticalinference,andmachinelearning,tonamejustafew.Theirinterdisciplinarynatureisevidentbyavastliteraturewhichincludesalargevolumeofmathematicalandalgorithmicdevelopments[ 32 ].Inthissectionwediscusssomelinearinverseproblemsofinterest. 1.2.1TotalVariationBasedImageReconstructionIngeneral,imagereconstructionproblemcanbemodeledasalinearinverseproblemoftheform f=Au+w,(1{13)whereA2
PAGE 19

bystackingthecolumnsofitscorrespondingtwo-dimensionalimages,andthematrixAdescribestheFourieroperator.Wecanndanapproximatesolutionof( 1{13 )usingtheLeastSquare(LS)method: min1 2kAu)]TJ /F4 11.955 Tf 11.95 0 Td[(fk2.(1{14)However,inmanyapplications,itisoftenthecasethatAisill-conditioned[ 49 ].Tohandlethisdiculty,regularizationmethodsarerequiredtostabilizethesolution.Oneofthepopularregularizationtechniquesisthetotalvariationbasedregularizationinwhichatotalvariationpenaltytermisaddedtotheobjectivefunction( 1{14 ): min(1 2kAu)]TJ /F4 11.955 Tf 11.95 0 Td[(fk2+nXi=1k(ru)ik),(1{15)where(ru)iisadiscretegradient(nitedierencesalongthecoordinatedirections)ofuatthei-thvoxel,andisaparameter,whichbalancesthedelitytermandtheregularizationterm.Indeed,thersttermoftheproblem( 1{15 )isdierentiablewithgradientLipschitzconstantL:=max(ATA),i.e.thelargestsingularvalueofATA,whilethesecondtermishighlynon-smooth.Hence,( 1{15 )belongstothecategoryofstructuredoptimizationproblem( 1{7 ). 1.2.2StructuredNetworkLearningTheproblemoflearningthenetworkstructurefromalargesetoftimeseriesarisesinmanyeconomic,nanceandbiomedicalapplications.Examplesincludemacroeconomicpolicymakingandforecasting[ 64 ],assessingconnectivityamongstnancialrms[ 4 ],reconstructinggeneregulatoryinteractionsfromtime-coursegenomicdata[ 71 ]andunderstandingconnectivitybetweenbrainregionsfromfMRImeasurements[ 99 ].Toaccuratelylearnthenetworkstructures,researchersconcentrateondevelopingappropriatemathematicalmodelsandecientalgorithms.Inpractice,discreteobservationofthenetworkcanbeacquiredalongtemporalaxis.Therefore,wecanrepresentthoseobservationsbymultivariatetimeseriesofwhichwecanusethemodelofvectorautoregressive 19

PAGE 20

process(VAR)tocharacterizetheirrelationship.Morespecically,thecurrentobservationofanodeatatimepointisalinearcombinationofthepreviousmeasurementsofitselfandthenodesregulatingitplusaninnovationnoise Xt=B0Xt)]TJ /F6 7.97 Tf 6.58 0 Td[(1+t,t=1,,T,(1{16)whereBisapqtransitionmatrixspecifyingthelead-lagcrossdependenciesamongsttheptimeseriesandazeromeanerrorprocess.Thelearningofthenetworkstructureisbasedonthefollowingregressionformulation.Givensamplevectorsfort=1,,TtimepointsfX0,,XTg,weformthelinearmodelasfollows: 266664(XT)0...(X1)0377775| {z }Y=266664(XT)]TJ /F6 7.97 Tf 6.59 0 Td[(1)0...(X0)377775| {z }XB+266664(T)0...(1)0377775| {z }E, (1{17) i.e.Y=XB+E.Similarly,thenetworkstructure,i.e.thetransitionmatrixB,canbeestimatedviathefollowingregularizedmethod: min1 2kXB)-222(Yk2+g(B),(1{18)whereg(B)istheregularizationtermdependingonthepriorinformationfromB.Usuallyithasasimplynon-smoothstructure,whichcanbecharaterizedbyl1normand/orl2,1norm.Themainworkofthisdissertationdevelopsecientalgorithmsforthosestructuredoptimizationproblems( 1{15 )and( 1{18 )arisingfromimagingandnetworklearningareasleveragingthoseoptimizationtechniquesaforementioned. 20

PAGE 21

1.3OutlineandMainResultsofTheDissertationThisdissertationisorganizedasfollows.InChapter 2 ,weproposetwoacceleratedBregmanOperatorSplitting(BOS)algorithmswithbacktrackingforaconvexcompositeoptimizationproblemwherethenon-smoothcomponentdemonstratesacomplicatedstructure.TherstalgorithmimprovestherateofconvergenceforBregmanOperatorSplittingwithVariableStepsize(BOSVS)undertheassumptionthatthefeasiblesetisbounded.Thesecondalgorithmiscapableofdealingwiththecasewherethefeasiblesetisunbounded.BothalgorithmsexhibitbetterpracticalperformancethanBOSVSandAcceleratedAlternatingDirectionMethodofMultipliers(AADMM),whilepreservethesameacceleratedrateofconvergenceasthatforAADMM.InChapter 3 ,weconsiderthemodelingandfaststructurelearningofcausalitynetworkusinghigh-dimensionaltimeseriesinpresenceofunobservedlatentvariables.InordertoaccuratelyestimateanetworkofGrangercausalinteractionsafteraccountingforlatenteects,weintroduceanovelapproachoflow-rankandstructuredsparsevectorautoregressivemodels(VAR)models.Weintroducearegularizedframeworkinvolvingacombinationofnuclearnormandstructuredsparse(lasso/grouplasso).Further,weestablishnon-asymptoticupperboundsontheestimationerrorratesofthelow-rankandthestructuredsparsecomponents.Moreover,weproposeafastacceleratedproximalgradientalgorithmforestimatingthenetworkstructureofinterest.InChapter 4 ,wedevelopnovelmethodsforextractingtwo-dimensional(2D)conductivityprolesfromlargeelectrochemicalpotentialdatasets.Themethodconsistsofadatapreprocessingproceduretoreduce/eliminatenoiseandanumericalconductivityreconstruction.AnacceleratedADMM-typealgorithmisdesignedtospeeduptheconductivityreconstruction.Themethodisdemonstratedonameasurementofthegrainboundaryofamonolayergraphene,yieldinganearly10:1ratioforthegrainboundaryresistivityoverbulkresistivity.Moreover,weproposeanalgorithmthatcangreatlyreducethenoiseanderrorofSTMimages 21

PAGE 22

bycombiningforwardandbackwardscandatainalinebylinemanner.ThisallowsustopushthescanrateforaconventionalSTMsetuptobeyonditsnormallimit,upto10timesfaster.. 22

PAGE 23

CHAPTER2ACCELERATEDBREGMANOPERATORSPLITTINGWITHBACKTRACKING 2.1OverviewofOperatorSplittingMethodsThemainpurposeofthispaperistodevelopacceleratedBregmanOperatorSplittingalgorithmswithbacktrackingforsolvingthefollowingconvexcompositeoptimizationproblem minu2U1 2kAu)]TJ /F4 11.955 Tf 11.96 0 Td[(fk2+ (Bu),(2{1)whereU2Cnisaclosedconvexset,A2Cmnisamatrix,f2Cmisavector, :Cdn!
PAGE 24

constrainedoptimizationproblem: minu2U,w2Cdn1 2kAu)]TJ /F4 11.955 Tf 11.95 0 Td[(fk2+ (w)subjecttow=Bu.(2{3)TheaugmentedLagrangianfunctionassociatedwith( 2{3 )is L(u,w,)=1 2kAu)]TJ /F4 11.955 Tf 11.96 0 Td[(fk2+ (w)+h,Bu)]TJ /F4 11.955 Tf 11.96 0 Td[(wi+ 2kBu)]TJ /F4 11.955 Tf 11.95 0 Td[(wk2.(2{4)TheADMMminimizes( 2{4 )byiterativelyupdating(u,w,)asfollows: 8>>>>>>>><>>>>>>>>:uk+1=argminu2U(1 2kAu)]TJ /F4 11.955 Tf 11.96 0 Td[(fk2+ 2Bu)]TJ /F4 11.955 Tf 11.96 0 Td[(wk+k 2)wk+1=argminw2Cdn( (w)+ 2Buk+1)]TJ /F4 11.955 Tf 11.96 0 Td[(w+k 2)k+1=k)]TJ /F3 11.955 Tf 11.95 0 Td[((wk+1)]TJ /F4 11.955 Tf 11.96 0 Td[(Buk+1).(2{5)Inmanyapplications,thedimensionofthematrixAcouldbeverylargeandAmightbealsodenseandill-conditioned.Inthosecases,thehighcomputationalcostforsolvingtheu-subproblemhinderedtheapplicabilityofADMM.Recently,therehavebeenactiveresearchesonimprovingthetheoreticalandpracticalperformanceofADMM.Onetypeofeortshasbeenfocusedondevelopingbacktrackingstrategiestosearchforlargerstepsizewithconvergenceguaranteed[ 19 20 47 ].TheothereortisonimprovingtheiterationcomplexityoftheADMMalgorithmbyincorporatingNesterov'smulti-stepaccelerationschemes[ 28 42 45 55 72 79 81 ].Tohavemoreinsightfromthosetwotypesofeorts,belowwewilldiscussafewcloselyrelatedworks.In[ 115 116 ],theu)]TJ /F1 11.955 Tf 9.3 0 Td[(subproblemof( 2{5 )issolvedbyBregmanoperatorsplitting(BOS)technique,i.e.linearizedADMM.Itupdatesuk+1of( 2{5 )bylinearizing1 2kAu)]TJ /F4 11.955 Tf 11.72 0 Td[(fk2atukandaddingaproximalterm,i.e. uk+1=argminu2U(hAT(Auk)]TJ /F4 11.955 Tf 11.96 0 Td[(f),ui+ 2ku)]TJ /F4 11.955 Tf 11.95 0 Td[(ukk2+ 2Bu)]TJ /F4 11.955 Tf 11.95 0 Td[(wk+k 2).(2{6) 24

PAGE 25

Ithasbeenshownin[ 116 ]thattheBOSalgorithmconvergestoanoptimalsolutionof( 2{1 )withtheconstantstepsize1=1=kATAk.In[ 19 ],thisstepsizeisreplacedbytheBarzilai-Borwein(BB)stepsize,i.e.=BBk,where BBk=kA(uk)]TJ /F4 11.955 Tf 11.95 0 Td[(uk)]TJ /F6 7.97 Tf 6.59 0 Td[(1)k2 kuk)]TJ /F4 11.955 Tf 11.96 0 Td[(uk)]TJ /F6 7.97 Tf 6.59 0 Td[(1k2.(2{7)Theirexperimentalresultsshowedthattheobjectivefunctionvaluedecreasedmuchfasterbytaking=BBkthan=kATAk2.However,theBBstepsize1=BBk1=kATAk2violatestheconvergenceconditionoftheBOSalgorithm.Thus,theconvergenceofthealgorithmisnotguaranteed.Theimprovementwasmadein[ 20 ],whereaschemeofBOSwithvariablestepsize(BOSVS)wasdevelopedtoimprovetheperformanceofADMMforsolving( 2{1 )withguaranteedconvergence.Inthisworkalinesearchstrategywaspresentedforsearchingforabetterstepsize.ThestepsizerulestartswithasafeguardedBBstepsizeandgraduallyincreasesthenominalstepsizekuntiltheterminationconditionissatised.Withagoodchoiceofparametersinthelinesearchconditions,moreaggressivestepsizeisallowed,especiallyattheearlyiterates.TheglobalconvergenceoftheiteratesoftheBOSVSwasestablishedin[ 20 ].Lateron,itwasshownin[ 48 ]thattheobjectivefunctionattheaverageoftheBOSVSiteratesconvergetoanoptimalvaluewiththerateofO(1=k),wherekisthenumberofiterations.ToimprovetheiterationcomplexityofADMM,in[ 81 ]anacceleratedADMM(AADMM)wasdevelopedbyincorporatingNesterov'sfastgradientscheme[ 79 ].TheAADMMsolvesaclassofconvexcompositeoptimizationproblemswithlinearequalityconstraints,whichincludestheproblem( 2{3 )asaspecialcase.AADMMimprovestherateofconvergenceofADMM(orBOS)intermsofthesmoothcomponentintheobjectivefunctionfromO(1=k)toO(1=k2).Theacceleratedrateofconvergenceisachievedbytheaggregatediteratesratherthantheaverageoftheiteratesin[ 48 ].ToimprovethepracticalperformanceofAADMM,asimplebacktrackingstrategyisincorporatedinthealgorithm.TheideaofthebacktrackingtechniqueistosearchforanunderestimatedLipschitzconstantsLkattheiterationkinconsiderationofthedependenceofthestepsizeonLk.Thebacktrackingprocedurein[ 81 ] 25

PAGE 26

startswitharelativelysmallerLk,andproperlyselectsinvolvedparameterstosolveuk+1.Ifanemployedlinesearchconditionisviolated,thenLkisdoubledandusedasthenewLktosolveuk+1.Thisprocedureisrepeateduntiltheutilizedlinesearchconditionholds.Motivatedbytheaforementionedwork,weproposetwoacceleratedBregmanOperatorSplitting(BOS)schemeswithbacktrackingforsolvingtheproblem( 2{1 ).TherstproposedalgorithmimprovestheconvergencerateofBOSVSintermsofthesmoothcomponentintheobjectivefunctionbyincorporatingNesterov'smulti-stepaccelerationscheme[ 79 ]undertheassumptionthatthefeasiblesetisbounded.Thesecondonecandealwiththesituationwhenthefeasiblesetisunbounded.Byjointlycomputingtheaccelerationparameterandstepsize,themonotonicityconditiononthenominalstepsizerequiredinBOSVS[ 20 ]andtherstproposedalgorithmcanberemovedinthesecondalgorithm.Combiningwiththegoodchoicesofthepenaltyparametersforupdatingu,wandin( 2{5 )(possiblydierent'sinu,w,subproblems),moreaggressivestepsizeisallowedinthesecondproposedalgorithm.InsteadofsearchingthelocalLipschitzconstanttosatisfytheconservativelinesearchconditioninAADMM[ 81 ],theproposedalgorithmsutilizetheproductoftheaccelerationparameterandasafeguardedBarzilai-Borwen(BB)choiceastheinitialstepsize,thengraduallyincreaseituntilamorerelaxedlinesearchconditionsthanthatin[ 81 ]issatised.Theproposedalgorithmsarecapableofhuntingformoreaggressivestepsizeviaconductingfewernumberoflinesearches.Meanwhile,theproposedalgorithmspreservethesameacceleratedrateofconvergenceasthatforAADMM.In[ 81 ],acenterpieceofthetheoreticalanalysisisfortheconvergencerateofAADMMintermsofthedependenceontheLipschitzconstantofthesmoothobjective.Asimplebacktrackingschemeisproposedin[ 81 ]todemonstratethatpriorknowledgeoftheLipschitzconstantisnotadenitiverequirement.Inthisworkmostoftheconvergenceanalysesisdevotedtomaintainingtheacceleratedconvergenceresultsunderaggressivestepsizestrategy,whichwasnottouchedineither[ 20 ]or[ 81 ].Theproblem( 2{1 )isaspecialcaseoftheproblemofinterestin[ 81 ]whichiscalledtheunconstrainedcompositeoptimization(UCO) 26

PAGE 27

problemthere.Thebacktrackingtechniquein[ 81 ]isdesignedtosolveUCOwithboundedprimalanddualfeasiblesets.Inthismanuscript,theproposedABOSVSIIalgorithmisabletosolve( 2{1 )withanunboundedprimalfeasiblesetwithoutanyfeasibilityresidue,whichwasnotdiscussedin[ 81 ].Itseemsthatthetechnicaldetailsforstudying( 2{1 )withunboundedprimalfeasiblesetisnontrivialcomparingwiththeanalysisin[ 81 ].Moreover,inthisworkweuseadierentterminationcriterionfromtheoneforAADMMin[ 81 ]toobtainthesameacceleratedrateofconvergence.In[ 81 ],theconvergenceanalysisisbasedontheestimationofthedualitygapfunctionforitscorrespondingsaddlepointproblem.Inthiswork,theterminationcriterionisbasedontheerrorbetweentheobjectivefunctionvaluesattheaggregatediteratesandtheoptimalsolution.Consequently,weareabletoconductadierentproofoftheacceleratedrateofconvergencewhenthefeasiblesetisunbounedbyobservingtherelationshipbetweentheLipschitzcontinuityofthefunction in( 2{1 )andtheboundednessofitssubgradients.Webelievethatthestrategyintheproofofthecasewithunboundedfeasiblesetsprovidesarelativelysimpleralternativeoftheproofofacceleratedconvergenceresultsin[ 81 ].Ourexperimentalresultsshowthattheproposedalgorithmsoutperformseveralstate-of-the-artalgorithmsontotalvariationbasedimagereconstructionproblems.1.1OutlineofthepaperOurpaperisorganizedasfollows.Section2presentstheproposedalgorithms,namelyAcceleratedBOSVS-I(ABOSVS-I)andAcceleratedBOSVS-II(ABSOVS-II),forsolvingthetypeofproblems( 2{1 ).Section3studiestheconvergenceanalysisfortheproposedalgorithms.Section4isdevotedtonumericalexperimentsandcomparisonswithstate-of-the-artalgorithmsontotal-variationbasedimagereconstructionproblems.Thelastsectiondrawstheconclusionforthispaper.1.2NotationandterminologiesTheEuclideaninnerproductoftwocolumnvectorsx,y2Cnisdenotedbyhx,yi=xTy,wherethesuperscriptTdenotestheconjugatetranspose.Assumeuisanoptimalsolution 27

PAGE 28

of( 2{1 ).DeneDu:=ku1)]TJ /F4 11.955 Tf 12.48 0 Td[(uk,Du,B:=kB(u1)]TJ /F4 11.955 Tf 12.48 0 Td[(u)k,D:=k1)]TJ /F3 11.955 Tf 12.47 0 Td[(k,DU,B:=supu1,u22UkBu1)]TJ /F4 11.955 Tf 12.53 0 Td[(Bu2k,DV:=supv1,v22Vkv1)]TJ /F4 11.955 Tf 12.54 0 Td[(v2kforanycompactsetsUandV.ForaconvexfunctionF:Cn!<,@FrepresentsthesubdierentialofF. 2.2ProposedAlgorithmsInthissection,wepresenttheframeworksoftheABOSVS-IandABOSVS-II.TherstalgorithmincorporatesNesterov'smulti-stepaccelerationschemetoimprovetherateofconvergenceofBOSVSundertheassumptionthatthefeasiblesetisbounded.Whilethesecondoneiscapableofdealingwiththecasewherethefeasiblesetisunbounded.Inthesecondalgorithmtheaccelerationparameterandthestepsizeareupdatedjointlyandthepenaltyparametersin( 2{5 )arechosendierentlyforupdatingu)]TJ /F1 11.955 Tf 9.29 0 Td[(,w)]TJ /F1 11.955 Tf 13.2 0 Td[(and.Forconvenience,Weuseui,wi,anditoreplaceinthosethreesubproblems,respectively.ForABOSVS-I,withassumptionofboundedfeasibleset,canbechosentobethesameconstantin( 2{5 )withoutaectingtheconvergencerate,i.e. ui=wi=i=>0.(2{8)Intheproposedalgorithms,theinitialchoiceofthenominalstepiisasafeguardedBBchoice: 0,1minand0,i=maxmin,BBifori>1.(2{9)whereBBiisdenedin( 2{7 ).NowwepresenttheschemeofABOSVS-IinAlgorithm 2.1 undertheassumptionthatthefeasiblesetUisbounded. Algorithm2.1. AcceleratedBOSVSI(ABOSVS-I) ChooseC0,>1,>0,>1,0,1min>0,andu12U.Set1=1,uag1=u1,wag1=w1=Bu1,ag1=1=0,andQ1=0. Fori=1,2,...,k, 28

PAGE 29

//Backtracking 1. Seti=i0,i,where0,iisfrom( 2{9 ).Computeumdi=(1)]TJ /F3 11.955 Tf 11.95 0 Td[(i)uagi+iui,ui+1=argminu2U(hAT(Aumdi)]TJ /F4 11.955 Tf 11.96 0 Td[(f),ui+i 2ku)]TJ /F4 11.955 Tf 11.96 0 Td[(uik2+ui 2Bu)]TJ /F4 11.955 Tf 11.95 0 Td[(wi+i ui2),)]TJ /F7 7.97 Tf 6.78 -1.79 Td[(i=i ikui+1)]TJ /F4 11.955 Tf 11.96 0 Td[(uik2+ui ikBui+1)]TJ /F4 11.955 Tf 11.95 0 Td[(wik2)-222(kA(ui+1)]TJ /F4 11.955 Tf 11.95 0 Td[(ui)k2,Qi+1=iQi+)]TJ /F7 7.97 Tf 18.73 -1.79 Td[(i,where0i(1)]TJ /F5 11.955 Tf 13.15 8.09 Td[(1 i)2. 2. IfQi+1<)]TJ /F4 11.955 Tf 9.3 0 Td[(C=i2,thenreplace0,iby0,iandreturntostep1. 3. Updatethesafeguardthresholdminbyminifi=i>i)]TJ /F6 7.97 Tf 6.59 0 Td[(1=i)]TJ /F6 7.97 Tf 6.59 0 Td[(1,fori>1. //Updatingiterates 4. Computeuagi+1=(1)]TJ /F3 11.955 Tf 11.96 0 Td[(i)uagi+iui+1,wi+1=argminw2Cdn( (w)+wi 2w)]TJ /F4 11.955 Tf 11.96 0 Td[(Bui+1)]TJ /F3 11.955 Tf 14.8 8.09 Td[(i wi2),wagi+1=(1)]TJ /F3 11.955 Tf 11.96 0 Td[(i)wagi+iwi+1,i+1=i)]TJ /F3 11.955 Tf 11.95 0 Td[(i(wi+1)]TJ /F4 11.955 Tf 11.95 0 Td[(Bui+1),agi+1=(1)]TJ /F3 11.955 Tf 11.96 0 Td[(i)agi+ii+1,i+1=i)]TJ /F3 11.955 Tf 9.3 0 Td[(i+p 2i+4 2. EndFor Output(uagk+1,wagk+1). FromABOSVS-I,wecanseethatifi1,thenumdi=uiandtheaggregatepointsuagi+1,wagi+1,agi+1areexactlytheiteratesui+1,wi+1,andi+1,respectively.Inthiscase,ABOSVS-IbecomesBOSVSwithaminormodication.ForABSOVS-I,thederivationoftheacceleratedconvergenceraterepliesontheasymptoticmonotonicityofi=ifori=1,2,.... 29

PAGE 30

Wheneveri=iisnotmonotonedecreasing,minisincreasedbyafactor>1instep3fori=1,2,....Hence,ifthemonotonicityofi=iviolatescontinuously,theni=iwillapproachaconstant,whichisusuallysmallerthankATAk2,afteranitenumberofiterations.Moreover,itshouldbenotethatfromQi+1=iQi+)]TJ /F7 7.97 Tf 18.73 -1.79 Td[(i,wehave kXi=2(i)]TJ /F5 11.955 Tf 11.96 0 Td[(1)Qi)]TJ /F4 11.955 Tf 11.96 0 Td[(Qk+1=)]TJ /F7 7.97 Tf 17.51 14.95 Td[(kXi=1)]TJ /F7 7.97 Tf 6.78 -1.79 Td[(i.(2{10)whereiandtheconditiononQicanbechosenjointlyaslongasthat)]TJ /F7 7.97 Tf 15.73 11.35 Td[(kPi=1)]TJ /F7 7.97 Tf 6.78 -1.8 Td[(iisanitepositivenumber.ABOSVS-IIcandealwiththecaseswherethefeasiblesetiseitherboundedorunbounded.Whenthefeasiblesetisunbounded,thepenaltyparametersui,wi,andihavetobechosendierentlyfromthoseforboundedfeasibleset.Moreprecisely,ifthefeasiblesetUisunbounded,wechoose ui=wi=Ki minandi=min Ki,(2{11)whereKisthetotalnumberofiterations.IfthefeasiblesetUisbounded,wechoose ui=wi=ii minandi=min ii,(2{12)inABOSVS-IIbyjointlyupdatingtheaccelerationparameterandthestepsize.TheschemeofABOSVS-IIispresentedinAlgorithm 2.2 Algorithm2.2. AcceleratedBOSVSII(ABOSVS-II) ChooseC0,>1,0,1min,u1andw1suchthatw1=Bu1.Set1=1,uag1=u1,wag1=w1,ag1=1=0,andi=1. Fori=1,2,...,k, //Backtracking 30

PAGE 31

1. Seti=i0,i,where0,iisfrom( 2{9 ).Solveifrom1 i)]TJ /F6 5.978 Tf 5.76 0 Td[(1i)]TJ /F6 5.978 Tf 5.76 0 Td[(1=1)]TJ /F11 7.97 Tf 6.58 0 Td[(i iifori>1.Updateui,wiandiby( 2{11 )or( 2{12 ).Computeumdi=(1)]TJ /F3 11.955 Tf 11.96 0 Td[(i)uagi+iui.ui+1=argminu2UnhAT(Aumdi)]TJ /F4 10.909 Tf 10.91 0 Td[(f),ui+i 2ku)]TJ /F4 10.909 Tf 10.91 0 Td[(uik22+ui 2kBu)]TJ /F4 10.909 Tf 10.9 0 Td[(wi+i uik2o. 2. IfiPj=1)]TJ /F7 7.97 Tf 6.77 -1.8 Td[(j<)]TJ /F4 11.955 Tf 9.3 0 Td[(C,where)]TJ /F7 7.97 Tf 6.77 -1.79 Td[(j:=kuj+1)]TJ /F4 11.955 Tf 11.96 0 Td[(ujk2+uj jkBuj+1)]TJ /F4 11.955 Tf 11.96 0 Td[(wjk2)]TJ /F11 7.97 Tf 13.15 6.11 Td[(j jkA(uj+1)]TJ /F4 11.955 Tf 11.95 0 Td[(uj)k2,then0,i:=0,i,returntostep1. //Updatingiterates 3. Computetheiteratesuagi+1,wi+1,wagi+1,i+1,agi+1fromstep4inAlgorithm 2.1 EndFor Output(uagk+1,wagk+1). Tworemarksareinplace.First,step3fromAlgorithm 2.1 isnotrequiredanymoreinAlgorithm 2.2 .Second,inAlgorithm 2.2 ,step2actuallycanbewrittenas IfQi+1<)]TJ /F4 11.955 Tf 9.29 0 Td[(C,then0,i:=0,i,returntostep1(2{13)bysettingi=1fori>1.Obviously,italsosatisestheconditionweimposeon)]TJ /F7 7.97 Tf 15.72 11.36 Td[(kPi=1)]TJ /F7 7.97 Tf 6.77 -1.79 Td[(ifrom( 2{10 ).Additionally,in( 2{13 ),wecanalsosetisuchthat(1)]TJ /F5 11.955 Tf 11.95 0 Td[(1=i2)i1. 2.3ConvergenceAnalysisInthissection,wefocusonprovingtheconvergencepropertiesoftheproposedalgorithms.WestartwithtwolemmasdescribingpropertiesregardingthelinesearchschemeinAlgorithms 2.1 and 2.2 .Lemma1belowshowsthatthesafeguardstepsizethresholdmininstep3ofABOSVS-Iwillstopincreasingafteranitenumberofiteration. Lemma1. Thereplacementofminbymininstep3ofABOSVS-Icanoccurinatmostanitenumberofiterations,denotedbyN0. Proof. SeeLemma3.2(II)of[ 20 ]. 31

PAGE 32

Lemma2. Insteps2and3ofAlgorithm 2.1 andstep2ofAlgorithm 2.2 ,thenumberoflinesearch,denotedbyl,islessthanorequaltodlog(jjATAjj min)e,wheredxeisthesmallestintegergreaterthanorequaltoxforanyx2<. Proof. TheproofforAlgorithm 2.1 canbeobtainedbyaasimilarargument,weonlygivetheproofforAlgorithm 2.2 .Bystep2ofAlgorithm 2.2 (seealsotheremarkon( 2{13 )),thelinesearchstopswhenQi+1)]TJ /F4 11.955 Tf 22.18 0 Td[(C.Noting( 2{10 )(withi=1fori>1)andthedenitionof)]TJ /F7 7.97 Tf 6.78 -1.8 Td[(jinAlgorithm 2.2 ,theconditionQi+1)]TJ /F4 11.955 Tf 21.92 0 Td[(CisequivalenttoQi+kui+1)]TJ /F4 11.955 Tf 11.96 0 Td[(uik2+ui ikBui+1)]TJ /F4 11.955 Tf 11.95 0 Td[(wi+1k2)]TJ /F3 11.955 Tf 13.15 8.09 Td[(i ikA(ui+1)]TJ /F4 11.955 Tf 11.95 0 Td[(ui)k2)]TJ /F4 11.955 Tf 21.91 0 Td[(C.Afterrearrangingterms,theaboverelationcanbereformulatedtoi(C+Qi)+ikui+1)]TJ /F4 11.955 Tf 11.95 0 Td[(uik2+uikBui+1)]TJ /F4 11.955 Tf 11.96 0 Td[(wi+1k2)]TJ /F3 11.955 Tf 11.95 0 Td[(ikA(ui+1)]TJ /F4 11.955 Tf 11.96 0 Td[(ui)k20,ortheequivalentformiikA(ui+1)]TJ /F4 11.955 Tf 11.95 0 Td[(ui)k2 kui+1)]TJ /F4 11.955 Tf 11.96 0 Td[(uik2)]TJ /F3 11.955 Tf 13.15 8.08 Td[(uikBui+1)]TJ /F4 11.955 Tf 11.95 0 Td[(wi+1k2 kui+1)]TJ /F4 11.955 Tf 11.95 0 Td[(uik2)]TJ /F3 11.955 Tf 16.11 8.08 Td[(i(C+Qi) kui+1)]TJ /F4 11.955 Tf 11.95 0 Td[(uik2.Notingthetwonegativetermsattherighthandsideoftheabovelinesearchstoppingcriterion,weobservethatitissatisedwheniikA(ui+1)]TJ /F7 7.97 Tf 6.59 0 Td[(ui)k2 kui+1)]TJ /F7 7.97 Tf 6.59 0 Td[(uik2.Bythedenitionofi,suchconditionissatisedafterlroundsoflinesearch,aslongaslkA(ui+1)]TJ /F4 11.955 Tf 11.96 0 Td[(ui)k2 0,ikui+1)]TJ /F4 11.955 Tf 11.96 0 Td[(uik2.NotingthatkA(ui+1)]TJ /F4 11.955 Tf 12.87 0 Td[(ui)k2kATAkkui+1)]TJ /F4 11.955 Tf 12.88 0 Td[(uik2andthat0,imin,wehaveldlog(kATAk min)e. ThefollowinglemmaplaysaprimaryrolefortheconvergenceanalysisoftheproposedAlgorithms.Sincewehavewi=uiforbothproposedalgorithms,inthefollowingproofwiisreplacedbyuiforthepurposeofauniedanalysis.Throughoutthissection,weusenotationsuei=ui)]TJ /F4 11.955 Tf 11.96 0 Td[(uandwei=wi)]TJ /F4 11.955 Tf 11.95 0 Td[(wfori1.Forconvenience,wedenote1 2kAu)]TJ /F4 11.955 Tf 11.96 0 Td[(fk2byH(u). 32

PAGE 33

Lemma3. Forallu2Uandallw2Cdn,theiteratesf(uagi,wagi)gi1generatedbyAlgorithms 2.1 and 2.2 satisfy [H(uagi+1)+ (wagi+1))]TJ /F4 11.955 Tf 11.96 0 Td[(H(u))]TJ /F3 11.955 Tf 11.96 0 Td[( (w)])]TJ /F5 11.955 Tf 11.96 0 Td[((1)]TJ /F3 11.955 Tf 11.96 0 Td[(i)[(H(uagi)+ (wagi))]TJ /F4 11.955 Tf 11.96 0 Td[(H(u))]TJ /F3 11.955 Tf 11.96 0 Td[( (w))]i(i 2(kueik2)-222(kuei+1k2))]TJ /F5 11.955 Tf 13.15 8.09 Td[(1 2(uikwi)]TJ /F4 11.955 Tf 11.95 0 Td[(Bui+1k2+ikui+1)]TJ /F4 11.955 Tf 11.96 0 Td[(uik2)]TJ /F3 11.955 Tf 11.95 0 Td[(ikA(ui+1)]TJ /F4 11.955 Tf 11.96 0 Td[(ui)k2)+ui 2(kweik2)-221(kwei+1k2)+1 2i(kik2)-222(ki+1k2))]TJ /F3 11.955 Tf 13.15 8.08 Td[(ui)]TJ /F3 11.955 Tf 11.95 0 Td[(i 2kwi+1)]TJ /F4 11.955 Tf 11.96 0 Td[(Bui+1k2+uihwi)]TJ /F4 11.955 Tf 11.96 0 Td[(Bui+1,w)]TJ /F4 11.955 Tf 11.95 0 Td[(Bui)-222(hi,w)]TJ /F4 11.955 Tf 11.95 0 Td[(Bui).(2{14) Proof. SinceHisdierentiable,wehave H(uagi+1)=H(umdi)+Z10d.(2{15)ApplyingthedenitionofH(u)anduagi+1totheaboveequation,andobservingtherelationshipuagi+1)]TJ /F4 11.955 Tf 11.95 0 Td[(umdi=i(ui+1)]TJ /F4 11.955 Tf 11.96 0 Td[(ui),wehave H(uagi+1)=H(umdi)+Z10hATA(umdi+(uagi+1)]TJ /F4 11.955 Tf 11.95 0 Td[(umdi)))]TJ /F4 11.955 Tf 11.95 0 Td[(ATf,uagi+1)]TJ /F4 11.955 Tf 11.96 0 Td[(umdiid=H(umdi)+Z10hAT(Aumdi)]TJ /F4 11.955 Tf 11.96 0 Td[(f),uagi+1)]TJ /F4 11.955 Tf 11.96 0 Td[(umdiid+Z10kA(uagi+1)]TJ /F4 11.955 Tf 11.96 0 Td[(umdi)k2d=H(umdi)+hrH(umdi),uagi+1)]TJ /F4 11.955 Tf 11.95 0 Td[(umdii+1 2kA(uagi+1)]TJ /F4 11.955 Tf 11.96 0 Td[(umdi)k2=H(umdi)+(1)]TJ /F3 11.955 Tf 11.95 0 Td[(i)hrH(umdi),uagi)]TJ /F4 11.955 Tf 11.95 0 Td[(umdii+ihrH(umdi),ui+1)]TJ /F4 11.955 Tf 11.96 0 Td[(umdii+2i 2kA(ui+1)]TJ /F4 11.955 Tf 11.95 0 Td[(ui)k2=(1)]TJ /F3 11.955 Tf 11.96 0 Td[(i)(H(umdi)+hrH(umdi),uagi)]TJ /F4 11.955 Tf 11.96 0 Td[(umdii)+i(H(umdi)+hrH(umdi),ui+1)]TJ /F4 11.955 Tf 11.95 0 Td[(umdii)+2i 2kA(ui+1)]TJ /F4 11.955 Tf 11.95 0 Td[(ui)k2.(2{16) 33

PAGE 34

HerebytheconvexityofH(u)and( 2{16 ),wehave H(uagi+1)=(1)]TJ /F3 11.955 Tf 11.95 0 Td[(i)(H(umdi)+hrH(umdi),uagi)]TJ /F4 11.955 Tf 11.95 0 Td[(umdii)+i(H(umdi)+hrH(umdi),u)]TJ /F4 11.955 Tf 11.95 0 Td[(umdii)+ihrH(umdi),ui+1)]TJ /F4 11.955 Tf 11.96 0 Td[(ui+2i 2kA(ui+1)]TJ /F4 11.955 Tf 11.96 0 Td[(ui)k2(1)]TJ /F3 11.955 Tf 11.96 0 Td[(i)H(uagi)+iH(u)+ihrH(umdi),ui+1)]TJ /F4 11.955 Tf 11.96 0 Td[(ui+2i 2kA(ui+1)]TJ /F4 11.955 Tf 11.96 0 Td[(ui)k2,8u2U.(2{17)By( 2{17 )andtheconvexityof ,wecancalculatethefollowingdierence. [H(uagi+1)+ (wagi+1))]TJ /F4 11.955 Tf 11.95 0 Td[(H(u))]TJ /F3 11.955 Tf 11.95 0 Td[( (w)])]TJ /F5 11.955 Tf 11.95 0 Td[((1)]TJ /F3 11.955 Tf 11.95 0 Td[(i)[(H(uagi)+ (wagi))]TJ /F4 11.955 Tf 11.95 0 Td[(H(u))]TJ /F3 11.955 Tf 11.95 0 Td[( (w))]=(H(uagi+1)+ (wagi+1))]TJ /F4 11.955 Tf 11.95 0 Td[(H(u))]TJ /F3 11.955 Tf 11.95 0 Td[( (w)))]TJ /F5 11.955 Tf 11.95 0 Td[((1)]TJ /F3 11.955 Tf 11.95 0 Td[(i)(H(uagi)+ (wagi))]TJ /F4 11.955 Tf 11.95 0 Td[(H(u))]TJ /F3 11.955 Tf 11.95 0 Td[( (w))=(H(uagi+1))]TJ /F5 11.955 Tf 11.95 0 Td[((1)]TJ /F3 11.955 Tf 11.95 0 Td[(i)H(uagi))]TJ /F3 11.955 Tf 11.96 0 Td[(iH(u))+( (wagi+1))]TJ /F5 11.955 Tf 11.95 0 Td[((1)]TJ /F3 11.955 Tf 11.95 0 Td[(i) (wagi))]TJ /F3 11.955 Tf 11.95 0 Td[(i (w))inhrH(umdi),uei+1i+i 2kA(ui+1)]TJ /F4 11.955 Tf 11.96 0 Td[(ui)k2+( (wi+1))]TJ /F3 11.955 Tf 11.96 0 Td[( (w))oinhrH(umdi),uei+1i+i 2kA(ui+1)]TJ /F4 11.955 Tf 11.96 0 Td[(ui)k2+hsi+1,wei+1io.(2{18)wheresi+12@ (wi+1).Ontheotherhand,bytherst-orderoptimalityconditionsforthesequence(ui+1,wi+1,i+1)generatedbyAlgorithm 2.1 and 2.2 ,forallu2Uandw2Cdn,wehave 8>>>>>>>><>>>>>>>>:hrH(umdi),uei+1i+ihui+1)]TJ /F4 11.955 Tf 11.96 0 Td[(ui,uei+1i+uihBui+1)]TJ /F4 11.955 Tf 11.96 0 Td[(wi+(ui))]TJ /F6 7.97 Tf 6.59 0 Td[(1i,Buei+1i0hsi+1,wei+1i)-222(hi,wei+1i+uihwi+1)]TJ /F4 11.955 Tf 11.96 0 Td[(Bui+1,wei+1i0i+1=i)]TJ /F3 11.955 Tf 11.95 0 Td[(i(wi+1)]TJ /F4 11.955 Tf 11.96 0 Td[(Bui+1),(2{19) 34

PAGE 35

Usingtherelationship2ha)]TJ /F4 11.955 Tf 12.05 0 Td[(b,a)]TJ /F4 11.955 Tf 12.04 0 Td[(ci=kb)]TJ /F4 11.955 Tf 12.05 0 Td[(ck2+ka)]TJ /F4 11.955 Tf 12.04 0 Td[(ck2+ka)]TJ /F4 11.955 Tf 12.04 0 Td[(bk2andthedenitionofuei+1,wei+1,theaboveequationcanberewrittenas 8>>>>>>>>>>>><>>>>>>>>>>>>:i 2(kui+1)]TJ /F4 11.955 Tf 11.96 0 Td[(uik2+kuei+1k2)-222(kueik2)+hrH(umdi),uei+1i+uihBuei+1)]TJ /F4 11.955 Tf 11.96 0 Td[(wei,Buei+1i)]TJ /F3 11.955 Tf 19.27 0 Td[(uihw)]TJ /F4 11.955 Tf 11.95 0 Td[(Bu,Buei+1i+hi,Buei+1i0hsi+1,wei+1i+uihwei+1)]TJ /F4 11.955 Tf 11.95 0 Td[(Buei+1,wei+1i)-223(hi,wei+1i+uihw)]TJ /F4 11.955 Tf 11.95 0 Td[(Bu,wei+1i0i+1=i)]TJ /F3 11.955 Tf 11.95 0 Td[(i(wi+1)]TJ /F4 11.955 Tf 11.96 0 Td[(Bui+1).(2{20)Substituting( 2{20 )to( 2{18 ),wehave [H(uagi+1)+ (wagi+1))]TJ /F4 11.955 Tf 11.95 0 Td[(H(u))]TJ /F3 11.955 Tf 11.95 0 Td[( (w)])]TJ /F5 11.955 Tf 11.96 0 Td[((1)]TJ /F3 11.955 Tf 11.96 0 Td[(i)[(H(uagi)+ (wagi))]TJ /F4 11.955 Tf 11.96 0 Td[(H(u))]TJ /F3 11.955 Tf 11.96 0 Td[( (w))]i(i 2(kueik2)-222(kuei+1k2))]TJ /F5 11.955 Tf 13.15 8.09 Td[(1 2(ikui+1)]TJ /F4 11.955 Tf 11.96 0 Td[(uik2)]TJ /F3 11.955 Tf 11.96 0 Td[(ikA(ui+1)]TJ /F4 11.955 Tf 11.95 0 Td[(ui)k2)+uihwei)]TJ /F4 11.955 Tf 11.95 0 Td[(Buei+1,Buei+1i| {z }(I)+hwei+1)]TJ /F4 11.955 Tf 11.95 0 Td[(Buei+1,ii| {z }(II))]TJ /F3 11.955 Tf 9.3 0 Td[(uihwei+1)]TJ /F4 11.955 Tf 11.95 0 Td[(Buei+1,wei+1i| {z }(III)+uihw)]TJ /F4 11.955 Tf 11.95 0 Td[(Bu,Buei+1)]TJ /F4 11.955 Tf 11.96 0 Td[(wei+1i| {z }(IV)).(2{21)Togiveafurtherestimationof( 2{21 ),nextwefocusonestimatingterms(I)-(IV).(I)=uihBuei+1,weii)]TJ /F3 11.955 Tf 19.26 0 Td[(uikBuei+1k2=)]TJ /F3 11.955 Tf 10.49 8.09 Td[(ui 2kwei)]TJ /F4 11.955 Tf 11.95 0 Td[(Buei+1k2+ui 2kweik2)]TJ /F3 11.955 Tf 13.15 8.09 Td[(ui 2kBuei+1k2=)]TJ /F3 11.955 Tf 10.49 8.09 Td[(ui 2kwi)]TJ /F4 11.955 Tf 11.95 0 Td[(Bui+1k2+ui 2kweik2)]TJ /F3 11.955 Tf 13.15 8.09 Td[(ui 2kBuei+1k2+uihwi)]TJ /F4 11.955 Tf 11.95 0 Td[(Bui+1,w)]TJ /F4 11.955 Tf 11.96 0 Td[(Bui)]TJ /F3 11.955 Tf 20.46 8.09 Td[(ui 2kw)]TJ /F4 11.955 Tf 11.95 0 Td[(Buk2, 35

PAGE 36

(II)=hi,wi+1)]TJ /F4 11.955 Tf 11.95 0 Td[(Bui+1i)-222(hi,w)]TJ /F4 11.955 Tf 11.95 0 Td[(Bui=1 ihi,i)]TJ /F3 11.955 Tf 11.96 0 Td[(i+1i)-222(hi,w)]TJ /F4 11.955 Tf 11.95 0 Td[(Bui=1 2i(ki)]TJ /F3 11.955 Tf 11.95 0 Td[(i+1k2)-222(ki+1k2+kik2))-221(hi,w)]TJ /F4 11.955 Tf 11.96 0 Td[(Bui=i 2kwi+1)]TJ /F4 11.955 Tf 11.95 0 Td[(Bui+1k2+1 2i(kik2)-221(ki+1k2))-222(hi,w)]TJ /F4 11.955 Tf 11.95 0 Td[(Bui,(III)=)]TJ /F3 11.955 Tf 9.29 0 Td[(uikwei+1k2+uihBuei+1,wei+1i=)]TJ /F3 11.955 Tf 10.49 8.08 Td[(ui 2kwei+1)]TJ /F4 11.955 Tf 11.96 0 Td[(Buei+1k2)]TJ /F3 11.955 Tf 13.15 8.08 Td[(ui 2kwei+1k2+ui 2kBuei+1k2=)]TJ /F3 11.955 Tf 10.49 8.09 Td[(ui 2kwi+1)]TJ /F4 11.955 Tf 11.96 0 Td[(Bui+1k2)]TJ /F3 11.955 Tf 13.15 8.09 Td[(ui 2kwei+1k2+ui 2kBuei+1k2+uihwi+1)]TJ /F4 11.955 Tf 11.96 0 Td[(Bui+1,w)]TJ /F4 11.955 Tf 11.96 0 Td[(Bui)]TJ /F3 11.955 Tf 20.45 8.09 Td[(ui 2kw)]TJ /F4 11.955 Tf 11.95 0 Td[(Buk2,and(IV)=uihBui+1)]TJ /F4 11.955 Tf 11.95 0 Td[(wi+1,w)]TJ /F4 11.955 Tf 11.95 0 Td[(Bui+uikw)]TJ /F4 11.955 Tf 11.96 0 Td[(Buk2.Combiningthoseaboveterms,wehave(I)+(II)+(III)+(IV)=)]TJ /F3 11.955 Tf 10.5 8.09 Td[(ui 2kwi)]TJ /F4 11.955 Tf 11.96 0 Td[(Bui+1k2+ui 2(kweik2)-222(kwei+1k2)+1 2i(kik2)-222(ki+1k2))]TJ /F3 11.955 Tf 13.15 8.09 Td[(ui)]TJ /F3 11.955 Tf 11.96 0 Td[(i 2kwi+1)]TJ /F4 11.955 Tf 11.95 0 Td[(Bui+1k2+uihwi)]TJ /F4 11.955 Tf 11.95 0 Td[(Bui+1,w)]TJ /F4 11.955 Tf 11.96 0 Td[(Bui)-222(hi,w)]TJ /F4 11.955 Tf 11.96 0 Td[(Bui.Applyingtheabovesummationto( 2{21 ),weobtain( 2{14 ). Thefollowinglemmapresentsanimportantpropertyoftheout(uagk+1,wagk+1)generatedbyAlgorithm 2.1 Lemma4. Supposethattheparametersui,wi,andiinAlgorithm 2.1 satisfy( 2{8 ),theoutput(uagk+1,wagk+1)generatedbyAlgorithm 2.1 satises 36

PAGE 37

1 2k[H(uagk+1)+ (wagk+1))]TJ /F4 11.955 Tf 11.96 0 Td[(H(u))]TJ /F3 11.955 Tf 11.95 0 Td[( (w)]kXi=1i 2i(kui)]TJ /F4 11.955 Tf 11.95 0 Td[(uk2)-222(kui+1)]TJ /F4 11.955 Tf 11.96 0 Td[(uk2)+kXi=1 2i(kwi)]TJ /F4 11.955 Tf 11.95 0 Td[(wk2)-222(kwi+1)]TJ /F4 11.955 Tf 11.95 0 Td[(wk2)+kXi=11 2i(kik2)-221(ki+1k2)+kXi=2(1)]TJ /F3 11.955 Tf 11.95 0 Td[(i)C 2(i)]TJ /F5 11.955 Tf 11.95 0 Td[(1)2+C 2k2.(2{22) Proof. Dividingbothsidesof( 2{14 )by2i,using( 2{8 )andstep1inAlgorithm 2.1 ,wehave 1 2i[H(uagi+1)+ (wagi+1))]TJ /F4 11.955 Tf 11.95 0 Td[(H(u))]TJ /F3 11.955 Tf 11.95 0 Td[( (w)])]TJ /F5 11.955 Tf 13.15 8.08 Td[((1)]TJ /F3 11.955 Tf 11.96 0 Td[(i) 2i[(H(uagi)+ (wagi))]TJ /F4 11.955 Tf 11.95 0 Td[(H(u))]TJ /F3 11.955 Tf 11.95 0 Td[( (w))]+(iQi+)]TJ /F7 7.97 Tf 18.73 -1.79 Td[(i) 2i 2i(kueik2)-221(kuei+1k2)+ 2i(kweik2)-222(kwei+1k2)+1 2ii(kik2)-221(ki+1k2)+ ihwi)]TJ /F4 11.955 Tf 11.96 0 Td[(Bui+1,w)]TJ /F4 11.955 Tf 11.95 0 Td[(Bui)]TJ /F5 11.955 Tf 22.62 8.09 Td[(1 ihi,w)]TJ /F4 11.955 Tf 11.96 0 Td[(Bui+(k)]TJ /F5 11.955 Tf 11.95 0 Td[(1)Qk 2+Qk 2.(2{23)SinceQi+1=iQi+)]TJ /F7 7.97 Tf 18.73 -1.79 Td[(i,0i(1)]TJ /F6 7.97 Tf 13.15 4.71 Td[(1 i)2,andQi)]TJ /F7 7.97 Tf 32.02 4.71 Td[(C (i)]TJ /F6 7.97 Tf 6.59 0 Td[(1)2,weobtain 1 2i[H(uagi+1)+ (wagi+1))]TJ /F4 11.955 Tf 11.95 0 Td[(H(u))]TJ /F3 11.955 Tf 11.95 0 Td[( (w)])]TJ -196.24 -32.57 Td[()]TJ /F5 11.955 Tf 13.15 8.09 Td[((1)]TJ /F3 11.955 Tf 11.96 0 Td[(i) 2i[(H(uagi)+ (wagi))]TJ /F4 11.955 Tf 11.95 0 Td[(H(u))]TJ /F3 11.955 Tf 11.95 0 Td[( (w))]+Qi+1 2i 2i(kueik2)-221(kuei+1k2)+ 2i(kweik2)-222(kwei+1k2)+1 2ii(kik2)-221(ki+1k2)+ ihwi)]TJ /F4 11.955 Tf 11.96 0 Td[(Bui+1,w)]TJ /F4 11.955 Tf 11.95 0 Td[(Bui)]TJ /F5 11.955 Tf 22.62 8.09 Td[(1 ihi,w)]TJ /F4 11.955 Tf 11.96 0 Td[(Bui+(1)]TJ /F3 11.955 Tf 11.95 0 Td[(i)C 2(i)]TJ /F5 11.955 Tf 11.96 0 Td[(1)2+Qi 2.(2{24)Settingu=uandw=w,bytherelationship1 2i=1)]TJ /F11 7.97 Tf 6.59 0 Td[(i+1 2i+1,Qi+1)]TJ /F7 7.97 Tf 24.12 4.71 Td[(C i2,and1=1,wehave( 2{22 )aftersumming( 2{24 )fromi=1tok. NowwearereadytoprovetheacceleratedconvergencerateoftheABOSVS-IalgorithmwhenthefeasiblesetsUandV:=dom arecompact,where istheconvexconjugateof (). 37

PAGE 38

Theorem2.1. Theoutput(uagk+1,wagk+1)generatedbyAlgorithm 2.1 satises H(uagk+1)+ (Buagk+1))]TJ /F4 11.955 Tf 11.96 0 Td[(H(u))]TJ /F3 11.955 Tf 11.96 0 Td[( (Bu)2kATAkD2U+2C+2C1 (k+1)2+8D2V (k+1)+4D2U,B k+1,(2{25)whereCandC1,whichareindependentofk,arenitenonnegativenumbers. Proof. IfminkATAk,bythedenitionof( 2{9 ),0,imin,whichyields)]TJ /F7 7.97 Tf 6.77 -1.79 Td[(i>0.Thus,thereisnolinesearchneededbasedonthebacktrackingstrategyinsteps1and2.Wehavetoexcludethiscasebysetting0,1
PAGE 39

handsideof( 2{22 )canbeestimatedby kXi=1i 2i(kui)]TJ /F4 11.955 Tf 11.96 0 Td[(uk2)-222(kui+1)]TJ /F4 11.955 Tf 11.96 0 Td[(uk2)=1 21kue1k2)]TJ /F7 7.97 Tf 12.66 14.95 Td[(k)]TJ /F6 7.97 Tf 6.59 0 Td[(1Xi=1(i 2i)]TJ /F3 11.955 Tf 17.12 8.09 Td[(i+1 2i+1)kui+1)]TJ /F4 11.955 Tf 11.95 0 Td[(uk2)]TJ /F3 11.955 Tf 17.13 8.09 Td[(k 2kkuk+1)]TJ /F4 11.955 Tf 11.95 0 Td[(uk2=1 21ku1)]TJ /F4 11.955 Tf 11.96 0 Td[(uk2)]TJ /F5 11.955 Tf 11.95 0 Td[((N0Xi=1+k)]TJ /F6 7.97 Tf 6.58 0 Td[(1Xi=N0+1)(i 2i)]TJ /F3 11.955 Tf 17.12 8.09 Td[(i+1 2i+1)kui+1)]TJ /F4 11.955 Tf 11.95 0 Td[(uk2)]TJ /F3 11.955 Tf 17.12 8.09 Td[(k 2kkuk+1)]TJ /F4 11.955 Tf 11.96 0 Td[(uk21 2ku1)]TJ /F4 11.955 Tf 11.96 0 Td[(uk2+C 2kATAk 2D2U+C 2,(2{29)whereC=8><>:0:ifPN0i=1(i i)]TJ /F11 7.97 Tf 13.74 5.65 Td[(i+1 i+1)kui+1)]TJ /F4 11.955 Tf 11.95 0 Td[(uk20C0:ifPN0i=1(i i)]TJ /F11 7.97 Tf 13.75 5.65 Td[(i+1 i+1)kui+1)]TJ /F4 11.955 Tf 11.95 0 Td[(uk2<0,sinceUisacompactset,clearlyCisanitenonnegativenumber.Bytheoptimalityconditionofu-subprobleminstep4,wehavei)]TJ /F3 11.955 Tf 12.16 0 Td[(ui(wi+1)]TJ /F4 11.955 Tf 12.16 0 Td[(Bui+1)2@ (wi+1)dom =V,whichindicatesthat kwi+1)]TJ /F4 11.955 Tf 11.95 0 Td[(wk22kwi+1)]TJ /F4 11.955 Tf 11.96 0 Td[(Bui+1k2+2kBui+1)]TJ /F4 11.955 Tf 11.96 0 Td[(wk22D2V (ui)2+2kBui+1)]TJ /F4 11.955 Tf 11.96 0 Td[(Buk22D2V 2+2D2U,B.(2{30)Bythesecondinequalityof( 2{28 ),1=1,and( 2{30 ),wehave kXi=1 2i(kwi)]TJ /F4 11.955 Tf 11.96 0 Td[(wk2)-222(kwi+1)]TJ /F4 11.955 Tf 11.96 0 Td[(wk2)= 21kw1)]TJ /F4 11.955 Tf 11.96 0 Td[(wk2+ 2k)]TJ /F6 7.97 Tf 6.59 0 Td[(1Xi=1(1 i+1)]TJ /F5 11.955 Tf 15.32 8.09 Td[(1 i)kwi+1)]TJ /F4 11.955 Tf 11.95 0 Td[(wk2)]TJ /F3 11.955 Tf 19.66 8.09 Td[( 2kkwk+1)]TJ /F4 11.955 Tf 11.95 0 Td[(wk2 2kXi=1kwi)]TJ /F4 11.955 Tf 11.96 0 Td[(wk2k 2(2D2V 2+2D2U,B).(2{31) 39

PAGE 40

Also, kXi=11 2i(kik2)-222(ki+1k2)=1 21jj1jj2+1 2k)]TJ /F6 7.97 Tf 6.59 0 Td[(1Xi=1(1 i+1)]TJ /F5 11.955 Tf 15.32 8.09 Td[(1 i)jji+1jj2)]TJ /F5 11.955 Tf 22.56 8.09 Td[(1 2kjjk+1jj21 2kXi=1kik2k 2D2V.(2{32)By( 2{29 ),( 2{31 ),and( 2{32 ),wehave H(uagk+1)+ (wagk+1))]TJ /F4 11.955 Tf 11.96 0 Td[(H(u))]TJ /F3 11.955 Tf 11.95 0 Td[( (w)2k(kATAk 2D2U+C 2+k 2(2D2V 2+2D2U,B)+k 2D2V+C1 2).(2{33)whereC1=Pki=2(1)]TJ /F11 7.97 Tf 6.59 0 Td[(i)C (i)]TJ /F6 7.97 Tf 6.59 0 Td[(1)2+C k2,whichisanitenonnegativenumber.Toobtain( 2{25 ),wealsoneedthefollowingestimation: )]TJ /F5 11.955 Tf 16.4 8.09 Td[(1 2kh,wagk+1)]TJ /F4 11.955 Tf 11.96 0 Td[(Buagk+1i=h,kXi=11 i(wi+1)]TJ /F4 11.955 Tf 11.96 0 Td[(Bui+1)i=)]TJ /F7 7.97 Tf 18.17 14.95 Td[(kXi=11 ih,wi+1)]TJ /F4 11.955 Tf 11.96 0 Td[(Bui+1i=)]TJ /F7 7.97 Tf 17.51 14.95 Td[(kXi=11 ih,i)]TJ /F3 11.955 Tf 11.96 0 Td[(i+1ikXi=11 2i(ki)]TJ /F3 11.955 Tf 11.96 0 Td[(k2)-222(ki+1)]TJ /F3 11.955 Tf 11.96 0 Td[(k2)k 2D2V,82V,(2{34)wheretherstequalitywasobtainedbysummingthesequencesoffuagi+1)]TJ /F4 11.955 Tf 12.31 0 Td[(Buagi+1gfori1andusingthesametechniqueinLemma 4 andthelastinequalitywasderivedbyusingthesameprocessasthatin( 2{32 ).Bytheconvexityof (),for82V,wehave H(uagk+1)+ (Buagk+1))]TJ /F4 11.955 Tf 11.96 0 Td[(H(u))]TJ /F3 11.955 Tf 11.96 0 Td[( (Bu)H(uagk+1)+ (Buagk+1))]TJ /F4 11.955 Tf 11.96 0 Td[(H(u))]TJ /F3 11.955 Tf 11.96 0 Td[( (Bu)+sup2V[ (wagk+1))]TJ /F3 11.955 Tf 11.96 0 Td[( (Buagk+1))-222(h,wagk+1)]TJ /F4 11.955 Tf 11.96 0 Td[(Buagk+1i]=H(uagk+1)+ (wagk+1))]TJ /F4 11.955 Tf 11.95 0 Td[(H(u))]TJ /F3 11.955 Tf 11.95 0 Td[( (Bu)+sup2Vh,wagk+1)]TJ /F4 11.955 Tf 11.95 0 Td[(Buagk+1i(2{35) 40

PAGE 41

Bytherstinequalityof( 2{28 ),combining( 2{33 ),( 2{34 )and( 2{35 ),weobtain( 2{25 ). NextwefocusonanalyzingtheacceleratedconvergencerateofABOSVS-IIalgorithm.Firstweneedtoestablishanimportantlemmasimilarlyaslemma 4 beforegivingtheacceleratedconvergencerateofABOSVS-II. Lemma5. Supposethattheparametersui,wi,andiinAlgorithm 2.2 satisfy( 2{11 )or( 2{12 ),theoutput(uagk+1,wagk+1)generatedbyAlgorithm 2.2 satisfy 1 kk[H(uagk+1)+ (wagk+1))]TJ /F4 11.955 Tf 11.95 0 Td[(H(u))]TJ /F3 11.955 Tf 11.96 0 Td[( (w)]kXi=11 2(kui)]TJ /F4 11.955 Tf 11.95 0 Td[(uk2)-222(kui+1)]TJ /F4 11.955 Tf 11.95 0 Td[(uk2)+kXi=1ui 2i(kwi)]TJ /F4 11.955 Tf 11.95 0 Td[(wk2)-221(kwi+1)]TJ /F4 11.955 Tf 11.96 0 Td[(wk2)+kXi=11 2ii(kik2)-222(ki+1k2))]TJ /F7 7.97 Tf 18.17 14.94 Td[(kXi=1(ui)]TJ /F3 11.955 Tf 11.95 0 Td[(i) 2(i)2iki+1)]TJ /F3 11.955 Tf 11.96 0 Td[(ik2)]TJ /F7 7.97 Tf 18.17 14.94 Td[(kXi=1)]TJ /F7 7.97 Tf 6.77 -1.8 Td[(i 2.(2{36) Proof. Dividingbykkfrombothsidesof( 2{14 ),wehave 1 ii[H(uagi+1)+ (wagi+1))]TJ /F4 11.955 Tf 11.95 0 Td[(H(u))]TJ /F3 11.955 Tf 11.95 0 Td[( (w)])]TJ /F5 11.955 Tf 13.15 8.09 Td[((1)]TJ /F3 11.955 Tf 11.96 0 Td[(i) ii[(H(uagi)+ (wagi))]TJ /F4 11.955 Tf 11.96 0 Td[(H(u))]TJ /F3 11.955 Tf 11.96 0 Td[( (w))]1 2(kueik2)-222(kuei+1k2)+1 2ii(kik2)-222(ki+1k2)+ui 2i(kweik2)-222(kwei+1k2))]TJ /F3 11.955 Tf 15.16 8.09 Td[(ui)]TJ /F3 11.955 Tf 11.96 0 Td[(i 2(i)2iki)]TJ /F3 11.955 Tf 11.95 0 Td[(i+1k2+ ihwi)]TJ /F4 11.955 Tf 11.96 0 Td[(Bui+1,w)]TJ /F4 11.955 Tf 11.95 0 Td[(Bui)]TJ /F5 11.955 Tf 21.79 8.08 Td[(1 ihi,w)]TJ /F4 11.955 Tf 11.96 0 Td[(Bui)]TJ /F5 11.955 Tf 20.45 8.08 Td[()]TJ /F7 7.97 Tf 6.78 -1.79 Td[(i 2.(2{37)Settingu=uandw=w,bytherelationship1 ii=1)]TJ /F11 7.97 Tf 6.58 0 Td[(i+1 i+1i+1,and1=1,weget( 2{36 )aftersumming( 2{37 )fromi=1tok. NowweanalyzetheacceleratedconvergencerateoftheABSOVS-IIalgorithmwhenthefeasiblesetsUisunbounded.Inthiscase,thetotalnumberofiterationshastobexed 41

PAGE 42

inadvanceandchosenbasedonaworst-casecomplexityanalysis.Additionally,since isLipschitzcontinuous,forallw2Cdn,wehavekk,for82@ (w). Theorem2.2. Supposethattheparametersui,wi,andiinAlgorithm 2.2 satisfy( 2{11 ),thentheoutput(uagk+1,wagk+1)generatedbyAlgorithm 2.2 satises kwagK)]TJ /F4 11.955 Tf 11.96 0 Td[(BuagKk4kATAk minK,(2{38) H(uagK)+ (BuagK))]TJ /F4 11.955 Tf 11.95 0 Td[(H(u))]TJ /F3 11.955 Tf 11.96 0 Td[( (Bu)2kATAk(D2u+C) K2+2kATAkD2u,B minK+4kATAk minK2.(2{39) Proof. Sincemin0,1,wehaveminkATAk.Then,bydenitionof0,i,weget min0,ikATAk.(2{40)Denotel0,iby0i,wherelisthenumberoflinesearchesinstep3ofABOSVS-II.By1 ii=1)]TJ /F11 7.97 Tf 6.59 0 Td[(i+1 i+1i+1andthedenitionofi,wehave 1 ip 0i=p 1)]TJ /F3 11.955 Tf 11.95 0 Td[(i+1 i+1q 0i+11)]TJ /F6 7.97 Tf 13.15 4.7 Td[(1 2i+1 i+1q 0i+11 i+1q 0i+1)]TJ /F5 11.955 Tf 23.14 8.09 Td[(1 20i+1fori1.(2{41)Then,byinductionwecanget,with1=1,(1 p 01+1 2kXi=21 p 0k)21 2k0k,whichimplies kk1 (1 p 01+1 2Pki=21 p 0k)24kATAk (k+1)2fork1,(2{42)whereweused( 2{40 )andthedenitionof0i.Thisestimateiscrucialforobtainingtheacceleratedrateofconvergence.Also,since 1 ip 0i=p 1)]TJ /F3 11.955 Tf 11.96 0 Td[(i+1 i+1q 0i+11)]TJ /F3 11.955 Tf 11.95 0 Td[(i+1 i+1q 0i+1=1 i+1q 0i+1)]TJ /F5 11.955 Tf 19.99 8.08 Td[(1 0i+1,(2{43) 42

PAGE 43

byinductionagain,weobtain1 kp 0kk p min,i.e.kkmin k2,whichimplies 1 kk minfork1.(2{44)Nextweestimatethetermsontherighthandsideof( 2{36 ).Bythedenitionofuiandi,and( 2{44 ),itisclearthatuiifori1.Thus, kXi=1(ui)]TJ /F3 11.955 Tf 11.96 0 Td[(i) 2(i)2iki+1)]TJ /F3 11.955 Tf 11.96 0 Td[(ik20.(2{45)Also,usingthedenitionofuiandiandthenon-increasingpropertyofthesequencesfui 2igandf1 2iigfori=1,2,...,k,wehave kXi=11 2(kui)]TJ /F4 11.955 Tf 11.95 0 Td[(uk2)-222(kui+1)]TJ /F4 11.955 Tf 11.96 0 Td[(uk2)=1 2ku1)]TJ /F4 11.955 Tf 11.95 0 Td[(uk2)]TJ /F5 11.955 Tf 13.15 8.09 Td[(1 2kuk+1)]TJ /F4 11.955 Tf 11.96 0 Td[(uk21 2D2u,(2{46) kXi=1ui 2i(kwi)]TJ /F4 11.955 Tf 11.95 0 Td[(wk2)-221(kwi+1)]TJ /F4 11.955 Tf 11.95 0 Td[(wk2)=u1 21kw1)]TJ /F4 11.955 Tf 11.95 0 Td[(wk2)]TJ /F7 7.97 Tf 12.67 14.94 Td[(k)]TJ /F6 7.97 Tf 6.58 0 Td[(1Xi=1(ui 2i)]TJ /F3 11.955 Tf 16.16 8.49 Td[(ui+1 2i+1)kwi+1)]TJ /F4 11.955 Tf 11.95 0 Td[(wk2)]TJ /F3 11.955 Tf 16.16 8.09 Td[(uk 2kkwk+1)]TJ /F4 11.955 Tf 11.95 0 Td[(wk2K 2minkw1)]TJ /F4 11.955 Tf 11.95 0 Td[(wk2=K 2minkBu1)]TJ /F4 11.955 Tf 11.96 0 Td[(Buk2K 2minD2u,B,(2{47) andkXi=11 2ii(kik2)-222(ki+1k2)=1 211k1k2)]TJ /F7 7.97 Tf 12.67 14.95 Td[(k)]TJ /F6 7.97 Tf 6.59 0 Td[(1Xi=1(1 2ii)]TJ /F5 11.955 Tf 33.25 8.09 Td[(1 2i+1i+1)ki+1k2)]TJ /F5 11.955 Tf 24.44 8.09 Td[(1 2kkkk+1k2=K 2min(k1k2)-222(kk+1k2)0,(2{48)By( 2{45 ),( 2{46 ),( 2{47 ),( 2{48 ),andthefactthat)]TJ /F7 7.97 Tf 15.73 11.36 Td[(kPi=1)]TJ /F7 7.97 Tf 6.78 -1.8 Td[(iC,wehave 1 kk[H(uagk+1)+ (wagk+1))]TJ /F4 11.955 Tf 11.95 0 Td[(H(u))]TJ /F3 11.955 Tf 11.95 0 Td[( (w)]D2u+C 2+KD2u,B 2min.(2{49) 43

PAGE 44

Nextwefocusonestimatingkwagk+1)]TJ /F4 11.955 Tf 11.96 0 Td[(Buagk+1kinordertoobtain( 2{39 ).Similarlyas( 2{34 ),for82Cdn,wecanobtain 1 kk(wagk+1)]TJ /F4 11.955 Tf 11.96 0 Td[(Buagk+1)=kXi=11 i(wi+1)]TJ /F4 11.955 Tf 11.95 0 Td[(Bui+1)=kXi=11 i(wi+1)]TJ /F4 11.955 Tf 11.96 0 Td[(Bui+1)=kXi=11 ii(i)]TJ /F3 11.955 Tf 11.96 0 Td[(i+1).(2{50)Thus,by1 kk(wagk+1)]TJ /F4 11.955 Tf 11.96 0 Td[(Buagk+1)=Pki=11 ii(i)]TJ /F3 11.955 Tf 11.96 0 Td[(i+1),wehave wagk+1)]TJ /F4 11.955 Tf 11.96 0 Td[(Buagk+1=Kkk min(1)]TJ /F3 11.955 Tf 11.95 0 Td[(k+1).(2{51)Bysettingk=K)]TJ /F5 11.955 Tf 11.96 0 Td[(1,wehave kwagk+1)]TJ /F4 11.955 Tf 11.95 0 Td[(Buagk+1k=4kATAk minKkk+1k.(2{52)Theonlyleftworkistoboundthetermkk+1k.Bytheoptimalityconditionofw-subprobleminstep3ofAlgorithm 2.2 ,wehave02@ (wi+1))]TJ /F3 11.955 Tf 11.95 0 Td[(i+wi(wi+1)]TJ /F4 11.955 Tf 11.96 0 Td[(Bui+1)8i1,ori+1=i)]TJ /F3 11.955 Tf 11.96 0 Td[(wi(wi+1)]TJ /F4 11.955 Tf 11.96 0 Td[(Bui+1)2@ (wi+1)8i1.Thereforewehaveki+1k,whichleadsto kwagk+1)]TJ /F4 11.955 Tf 11.95 0 Td[(Buagk+1k=4kATAk minK.(2{53)Bytheconvexityof (),wehave H(uagK)+ (BuagK))]TJ /F4 11.955 Tf 11.95 0 Td[(H(u))]TJ /F3 11.955 Tf 11.96 0 Td[( (w)H(uagK)+ (wagK)+h,BuagK)]TJ /F4 11.955 Tf 11.95 0 Td[(wagKi)]TJ /F4 11.955 Tf 19.26 0 Td[(H(u))]TJ /F3 11.955 Tf 11.96 0 Td[( (w)H(uagK)+ (wagK)kkkBuagK)]TJ /F4 11.955 Tf 11.95 0 Td[(wagKk)]TJ /F4 11.955 Tf 20.59 0 Td[(H(u))]TJ /F3 11.955 Tf 11.96 0 Td[( (w),(2{54)where2@ (BuagK).Then,by( 2{49 ),( 2{53 ),( 2{54 ),wehave( 2{39 ). 44

PAGE 45

ThefollowingtheoremgivestheacceleratedconvergencerateoftheABOSVS-IIalgorithmwhenthefeasiblesetsUiscompact.SinceitsproofissimilarwiththatforTheorem 2.2 ,wejustpresenttheconvergencerateresultwithoutproof. Theorem2.3. Supposethattheparametersui,wi,andiinAlgorithm 2.2 satisfy( 2{12 ),thentheoutput(uagk+1,wagk+1)generatedbyAlgorithm 2.2 satises H(uagk+1)+ (Buagk+1))]TJ /F4 11.955 Tf 11.95 0 Td[(H(u))]TJ /F3 11.955 Tf 11.95 0 Td[( (Bu)2kATAk(D2U+C) (k+1)2+10kATAkD2V min(k+1)+4kATAkD2U,B min(k+1).(2{55) 2.4NumericalResultsInthissection,weconductseveralexperimentsonsyntheticdataandthedatafrompartiallyparallelimaging(PPI)toexaminetheperformanceoftheproposedalgorithms.Wealsocomparethemwithseveralstate-of-the-artalgorithms.AllthealgorithmsareimplementedinMATLAB,R2015aonacomputerwitha2.6GHzInteli5processor. 2.4.1Total-VariationBasedImageReconstructionInthissubsection,wepresentthenumericalresultsonsolvingthefollowingTVbasedimagereconstructionproblem: minu2U1 2kAu)]TJ /F4 11.955 Tf 11.95 0 Td[(fk22+kukTV(2{56)whereU:=fu2
PAGE 46

Figure2-1. Theobjectivefunctionvaluesandrelativeerrorvs.CPUtimeforrstinstance f=Autrue+",whereutrueisthetrueimage,"istheGaussiannoisewithdistributionN(0,).WeapplyABSOVS-I,ABOSVS-II,BOSVS1,TVAL32andALADMML3tosolve( 2{56 ).Weconsidertwoinstancesofthisproblem.Intherstinstance,wesetU:=fu2
PAGE 47

Figure2-2. Theobjectivefunctionvaluesandrelativeerrorvs.CPUtimeforsecondinstance [0,1].Moreover,wesetthestandardderivation=10)]TJ /F6 7.97 Tf 6.59 0 Td[(2.Werun100iterationstocomparetheperformanceofABOSVS-I,ABOSVS-II,BOSVS,TVAL3andALADMML.Inthesecondinstance,wehaveU:=
PAGE 48

Table2-1. ParametersettingsforABOSVS-IandABOSVS-IIin 2.4.1 .Notethatandareonlyusedintherstinstance ParametersminCk Values102kAk2=101001/k12 thesecondinstance,ABOSVS-IIandALADMMLachievebetterperformancethanBOSVS,anditisalsoevidentthatABOSVS-IIoutperformsTVAL3andALADMML. 2.4.2PartiallyParallelImaging Figure2-3. Sensitivitymapsofdata1 Inthissection,weapplytheproposedalgorithmtotwoPPIdatasetsdenoteddata1anddata2andcomparetheperformanceofABOSVS-IIwithALADMMandTVAL3,andBOSVS.DetailsofPPIreconstructionproblemscanbefoundin[ 19 ].Theunderlyingimagecanbereconstructedbysolvingthefollowingoptimizationproblem: minu2Cn(1 2LXl=1kFp(slu))]TJ /F4 11.955 Tf 11.96 0 Td[(flk22+kukTV)(2{57)whereFpistheundersampledFouriertransformdenedbyFp:=PF,andFistheFouriertransform,Pisabinarymatrixrepresentingtheundersamplingpattern,slisthesensitivitymapforthel-thchannel,andflisthemeasurement.ThesymbolistheHadamardproduct 48

PAGE 49

betweentwovectors.Fornotationsimplicity,letA=[FpS1;FpS2;...;FpSL],f=[f1;f2;...;fL],whereSl:=diag(sl)2Cnnisthediagonalmatrixwithsl2Cnonthediagonal,l=1,2,...,L.Thentheaboveoptimizationproblemcanberewrittenas minu2Cn1 2kAu)]TJ /F4 11.955 Tf 11.96 0 Td[(fk22+kukTV(2{58)whichclearlycanbesolvedbytheproposedalgorithm.Figure 2-3 and 2-6 showthesensitivity Figure2-4. TrueimageandCartesianmaskfordata1 mapsofdata1anddata2,respectively.Figure 2-4 and 2-7 showthetrueimagesandsamplingpatterncorrespondingtodata1anddata2,respectively.TheCartesianmaskonlysamples18%oftheFouriercoecients.Inthisexperiment,themeasurementsfflgaregeneratedbyfl=P(FSlutrue+"rel=p 2+"iml=p )]TJ /F5 11.955 Tf 9.3 0 Td[(2),forl=1,...,L,where"reland"imlarethenoisewithentriesindependentlygeneratedfromdistributionN(0,10)]TJ /F6 7.97 Tf 6.59 0 Td[(4p nIn).Thesizeofdata1isn=256256andthesizeofdata2isn=512512.TheparametersettingsforABOSVS-IIaregiveninTable 2-2 .Fordata1anddata2,theperformanceofBOSVS,TVAl3,ALADMML,andABOSVS-IIisshowninTable 2-3 and 2-4 intermsoftheobjectivefunctionvalue,relativeerror,andCPU 49

PAGE 50

Table2-2. ParametersettingsforABOSVS-IIin 2.4.2 ParametersminC Values10)]TJ /F6 7.97 Tf 6.59 0 Td[(10n2kAk2=5100 Table2-3. Comparisonofobjectivefunctionvalue,relativeerror,andCPUtimeinsecondsusingdata1 AlgorithmsObjectivevalueRelativeerrorCPU BOSVS15.468410.05962528.0TVAL356.72630.050228.6ALADMML15.468100.049966.5ABOSVS-II15.445350.047949.7 timeinseconds.FromTable 2-3 and 2-4 ,wecanseethatABOSVS-II,ALADMMLconvergemuchfasterthanBOSVS,andtheperformanceofTVAL3isbetweenALADMMLandBOSVS.ABOSVS-IIhasthebestperformanceamongthem.ThereconstructedimagesbythosethreealgorithmsandthedierencesbetweenthereconstructedimagesandthetrueimagesareshowninFigure 2-5 and 2-8 forcomparison.Clearly,theimagesrecoveredbyABOSVS-II,ALADMML,andTVAL3havemuchsharperresolutionthanthatbyBOSVS,andtheimagerecoveredbyABOSVS-IIisslightlybetterthanthatbyALADMMLandbetterthanthatbyTVAL3.BothalgorithmsmaintainthesameacceleratedrateofconvergenceasthatforAADMM.Experimentalresultsshowthattheproposedalgorithmsarepromisingforlarge-scaleimagereconstructionproblems. 2.5ConclusionsofThisChapterInthispaper,weproposetwoacceleratedBregmanOperatorSplittingschemeswithbacktrackingforsolvingregularizedlarge-scalelinearinverseproblems.Theproposedschemes,namedABOSVS-IandABOSVS-II,arewellsuitedforsolvingtotal-variationbasedlarge-scalelinearinverseproblems,especiallywhenthematrixinthedelitytermislarge,denseandill-conditioned.ABOSVS-I,whichimprovestheconvergencerateofBOSVSintermsofthesmoothcomponentintheobjectivefunction,canhandlethecasewherethefeasiblesetis 50

PAGE 51

Figure2-5. ComparisonofimagereconstructionandtheirdierenceswithtrueimagebyBOSVS,TVAL3,ALADMML,andABOSVS-IIusingdata1. Figure2-6. Sensitivitymapsofdata2 Figure2-7. TrueimageandCartesianmaskfordata2 51

PAGE 52

Table2-4. Comparisonofobjectivefunctionvalue,relativeerror,andCPUtimeinsecondsusingdata2 AlgorithmsObjectivevalueRelativeerrorCPU BOSVS5.007e+20.06467448.4TVAL31.704e+30.050681.2ALADMML5.006e+20.0499180.0ABOSVS-II4.970e+20.0497144.7 Figure2-8. ComparisonofimagereconstructionandtheirdierenceswithtrueimagebyBOSVS,TVAL3,ALADMML,andABOSVS-IIusingdata2 bounded.ABOSVS-IIcandealwiththecasewherethefeasiblesetisunbounded.Forbothcasesofboundedandunboundedfeasiblesets,weemployanerrorbetweentheobjectivefunctionvaluesattheaggregatediteratesandatasolutionofproblem( 2{3 )toestimatetherateofconvergenceinsteadofusingthedualitygaptechniqueinAADMM. 52

PAGE 53

CHAPTER3MODELLINGANDFASTSTRUCTURELEARNINGOFCAUSALITYNETWORK 3.1OverviewofStructuredNetworkLearningLearningnetworkstructurefromalargesetoftimeseriesarisesinmanyeconomic,nanceandbiomedicalapplications.Examplesincludemacroeconomicpolicymakingandforecasting[ 64 ],assessingconnectivityamongstnancialrms[ 4 ],reconstructinggeneregulatoryinteractionsfromtime-coursegnomicdata[ 71 ]andunderstandingconnectivitybetweenbrainregionsfromfMRImeasurements[ 99 ].Vectorautoregressive(VAR)modelsprovideaprincipledframeworkforthesetasks.Formally,aVARmodelforptimesseriesisdenedinitssimplestpossibleforminvolvingasingletime-lagas Xt=B0Xt)]TJ /F6 7.97 Tf 6.58 0 Td[(1+t,t=1,,T,(3{1)whereBisapotransitionmatrixspecifyingthelead-lagcrossdependenciesamongsttheptimeseriesandazeromeanerrorprocess.VARmodelsforsmallnumberoftimeseries(low-dimensional)havebeenthoroughlystudiedintheliterature[ 69 ].However,theabovementionedapplications,wheredozenstohundredsoftimeseriesareinvolved,createdtheneedforthestudyofVARmodelsunderhighdimensionalscalingandtheassumptionthattheirinteractionsaresparsetocompensateforthepossiblelackofadequatenumberoftimepoints(samples;see[ 5 ]andreferencestherein).Nevertheless,thereareoccasionswherethesparsityassumptionmaynotbesucient.Forexample,duringnancialcrisisperiods,returnsonassetsmovetogetherinamoreconcertedmanner[ 4 9 ],whiletranscriptionfactorsregulatealargenumberofgenesthatmayleadtohub-nodenetworkstructures[ 93 ].Similarly,inbrainconnectivitynetworks,particulartasksactivateanumberofregionsthatcrosstalkinacollaborativemanner[ 89 ].Hence,itisofinteresttostudyVARmodelsunderhighdimensionalscalingwherethetransitionmatrixgoverningthetemporaldynamicsexhibitsamorecomplexstructure;e.g.itislowrankand/orgroupsparse. 53

PAGE 54

Therehasbeenlimitedworkintheliteratureaddressingthisproblem.Inalow-dimensionalregime,wherethenumberoftimepointsscalestoinnity,butthenumberoftimeseriesunderstudyremainsxed,[ 101 ]examinedVARmodelswheretheparametersexhibitreducedrankstructureandalsodiscussedconnectionswithcanonicalcorrelationanalysisofsuchmodelspresentedin[ 12 ].Specically,thetransitionmatrixBin( 3{2 )canbewrittenastheproductoftworank-kmatrices,i.e.B=0,sothatintheresultingmodelspecicationtheoriginalptimeseriesareexpressedaslinearcombinationsZt=Xtoftheoriginalones,andspeciesthedependencebetweenXtandZt;namelyXt=0Zt)]TJ /F6 7.97 Tf 6.59 0 Td[(1+t.Hence,toobtainand[ 101 ]suggesttoestimatetheparametersoftheoriginalmodelin( 3{1 )undertheconstraintthatB=andthatrank(B)=k.Otherwork,includeslowrankapproximationsofHankelmatricesthatrepresenttheinput-outputstructureofalineartimeinvariantsystemswerestudiedin[ 18 34 ].Finally,abriefmentiontothepossibilitythattheVARtransitionmatrixmayexhibitsuchastructureappearedasamotivatingexamplein[ 1 ].Ontheotherhand,thereisamatureliteratureonimposinglowrankplussparse,orpuregroupsparsestructureformanylearningtasksforindependentandidenticaldistributeddata.Examplesincludegroupsparsityinregression[ 112 ],lowrankandsparsematrixapproximationsfordimensionreduction[ 18 ],sparsecoupledwithgroupsparsestructuresinregressionanalysisandgraphicalmodeling[ 107 ],justtonameafew.However,asshownin[ 5 ],thepresenceoftemporaldependenceacrossobservationsinducesintricatedependenciesbetweenbothrowsandcolumnsofthedesignmatrixofthecorrespondingleastsquaresestimationproblem,aswellasbetweenthedesignmatrixandtheerrorterm,thatrequirecarefulhandlingtoestablishconsistencypropertiesforthemodelparametersundersparsityandhighdimensionalscaling.Theseissuesarefurthercompoundedwhenmorecomplexregularizingnormsareinvolvedasdiscussedin[ 70 ].Inthischapter,theauthorsmodelgroupingstructureswithineachcolumnofB,butdonotconsideralow-rankcomponent.Incontrast,wefocusongroupspotentiallyspanningacrossdierentcolumnsandalsoallowalow-rankcomponentinB. 54

PAGE 55

Further,toestimatethepositedmodelin( 3{1 )withBbeinglow-rankandstructuredsparse(henceforthindicatingthatitcouldbeeitherpuresparseorgroupsparseorboth),wealsointroduceafastacceleratedproximalgradientalgorithm,inspiredby[ 22 98 ],forthecorrespondingoptimizationproblems.ThekeyideaisthatinsteadofsearchingforthelocalLipschitzconstantofthegradientofthesmoothcomponentoftheobjectivefunction,theproposedalgorithmutilizesasafeguardedBarzilai-Bowen(BB)initialstepsizeandemploysrelaxedlinesearchconditionstoachievebetterperformanceinpractice.Thelatterenablestheselectionofmore\aggressive"stepsizes,thusresultinginfewerlinesearches,whilepreservingtheacceleratedconvergencerateofO(1 k2),wherekdenotesthenumberofiterationsrequireduntilconvergence.Finally,theperformanceofthemodelparametersunderdierentstructurestogetherwiththeassociatedestimationprocedurebasedontheacceleratedproximalgradientalgorithmarecalibratedonsyntheticdata.Notation:Throughoutthepaper,weemploythefollowingnotation:k.k,k.k2andk.kFdenotethe`2-normofavector,thespectralnormandtheFrobeniusnormofamatrix,respectively.ForappmatrixB,thesymbolkBkisusedtodenotethenuclearnorm,i.e.Ppj=1j(B),thesumofthesingularvaluesofamatrix,whileBydenotestheconjugatetransposeofamatrixB.ForanymatrixB,weusekBk0todenotecard(vec(B)),kBk1forkvec(B)k1andkBkmaxtodenotekvec(B)k1.Further,iffG1,G2,...,GKgdenoteapartitionoff1,2,...,p2gintoKnon-overlappinggroups,thenweusekBk2,1todenotePKk=1k(B)GkkF,kBk2,maxformaxk=1,2,...Kk(B)GkkFwhilekBk2,0denotesthenumberofnonzerogroupsinB.Here,withalittleabuseofnotation,weuseBGktodenotevec(B)Gk.Inaddition,max(.),min(.)denotethemaximumandminimumeigenvaluesofasymmetricorHermitianmatrix.Foranyintegerp1,weuseSp)]TJ /F6 7.97 Tf 6.59 0 Td[(1todenotetheunitballfv2Rp:kvk=1g.Weusefe1,e2,...ggenericallytodenoteunitvectorsinRp,whenpisclearfromthecontext.Finally,wewriteB%Aifthereexistsanabsolutepositiveconstantc,independentofthemodelparameters,suchthatBcA. 55

PAGE 56

3.2ModelFormulationandEstimationProcedureConsideraVAR(1)modelwherethetransitionmatrixBislow-rankplusstructuredsparsegivenby Xt=B0Xt)]TJ /F6 7.97 Tf 6.58 0 Td[(1+t,ti.i.d.N(0,), (3{2) B=L+R,rank(L)=r, (3{3) whereLcorrespondstothelow-rankcomponentandRrepresentseitherasparseS,orgroup-sparsecomponentG.Itisfurtherassumedthatthenumberofnon-zeroelementsinthesparsecaseiskSk0=s,whileinthegroupsparsecasethenon-zerogroupsarekGk2,0=gandthefollowingrelationshold:rp,sp2andgp2,withrbeingtherankofL.ThematrixLcapturespersistencestructureacrossallthetimeseriesandenablesthemodeltobeapplicableinsettingswheretherearestrongcross-autocorrelations,afeaturethatthestandardsparseVARmodelisnotdesignedfor,whilethesparseorgroupsparsecomponentcapturesadditionalcross-sectionalautocorrelationstructureamongstthetimeseries.Finally,itisassumedthattheerrortermsareseriallyandcrossuncorrelated,sincethestructureofthemodelisrichenoughandunlikelythatadditionalcontemporaneousdependenceispresent.TheobjectiveistoestimateLandRaccuratelybasedonarelativesmallnumberoftimepoints(samples).StabilityoftheVARprocess.Toestablishtheresults,weassumethatthepositedVARmodelin( 3{2 )isstable;i.e.itsspectraldensity fX()=1 2)]TJ /F2 11.955 Tf 5.48 -9.68 Td[(B)]TJ /F6 7.97 Tf 6.59 0 Td[(1)]TJ /F4 11.955 Tf 5.48 -9.68 Td[(e)]TJ /F7 7.97 Tf 6.59 0 Td[(i)]TJ /F2 11.955 Tf 5.48 -9.68 Td[(B)]TJ /F6 7.97 Tf 6.59 0 Td[(1)]TJ /F4 11.955 Tf 5.48 -9.68 Td[(e)]TJ /F7 7.97 Tf 6.59 0 Td[(i,(3{4)where2[)]TJ /F3 11.955 Tf 9.3 0 Td[(,],B(z):=Ip)]TJ /F4 11.955 Tf 12.04 0 Td[(B0zsatisesdet(B(z))6=0ontheunitcircleofthecomplexplanefz2C:jzj=1g.Itwasshownin[ 5 ]thatthisconditionissucienttoestablishtheoreticalpropertiesoftheestimatesofthetransitionmatrix.Further,thefollowingquantities 56

PAGE 57

playacentralroleintheerrorboundsestablishedoftheregularizedestimates: M(fX)=esssup2[)]TJ /F11 7.97 Tf 6.59 0 Td[(,]max(fX()),m(fX)=esssup2[)]TJ /F11 7.97 Tf 6.58 0 Td[(,]min(fX()),max(B)=maxjzj=1max(B(z)B(z)),min(B)=minjzj=1min(B(z)B(z)), (3{5) wheremax(.)andmin(.)denotethemaximumandminimumeigenvaluesofasymmetricorHermitianmatrix,respectively.NotethatM(fX)andm(fX)togethercapturethenarrownessofthespectraldensityoftheunderlyingstochasticprocess.Hence,processeswithstrongertemporalandcross-sectionaldependencehavenarrowerspectrathatinturnleadtoslowerconvergenceratesfortheregularizedestimates.ForVARmodels,M(fX)andm(fX)arerelatedtomax(B)andmin(B)asfollows: m(fX)1 2min() max(B),M(fX)1 2max() min(B)(3{6)Proposition2.2in[ 5 ]providesalowerboundonmin(B).Forthespecialstructureofthemodelsconsideredhere,wecangetanimprovedupperboundonmax(B),asshowninthefollowinglemma: Lemma6. ForastableVAR(1)modeloftheclass( 3{2 ),wehave max(B)[1+l+(vin+vout)=2]2(3{7)wherelisthelargestsingularvalueofL,vin=max1jpjRijjandvout=max1ipjRijj. Proof. kB(z)k=kI)]TJ /F5 11.955 Tf 12.22 0 Td[((L+R)zkkIk+kLk+kRkforanyz2Cwithjzj=1.Theresultfollowsfromthefactthatmax(B)=maxjzj=1kB(z)k2. TheestimationofVARmodelparametersisbasedonthefollowingregressionformulation(see[ 69 ]).Givensamplevectorsfort=0,,TtimepointsfX0,,XTg,weformthe 57

PAGE 58

linearmodelasfollows: 266664(XT)0...(X1)0377775| {z }Y=266664(XT)]TJ /F6 7.97 Tf 6.59 0 Td[(1)0...(X0)377775| {z }XB+266664(T)0...(1)0377775| {z }E, (3{8) whichisastandardregressionproblemwithNTsamplesandq=p2variables.ThegoalistoestimateL,RwithhighaccuracywhenNp2.ThereisaninherentidentiabilityissueintheestimationoftheLandRcomponents.Supposethelow-rankcomponentLitselfiss-sparseorg-groupsparseandthesparseorgroup-sparsecomponentRisofrankr.Inthatscenario,wecannothopeforanymethodtoestimateLandRseparately,withoutimposinganyfurtherconstraints.So,aminimalconditionforlow-rankandstructuredsparserecoveryisthatthelowrankcomponentshouldnotbetoosparseandthestructuredsparseoneshouldnotbelow-rank.Thisissuehasbeenaddressedintheliteraturebyseveralauthors[ 15 18 ]forindependentandidenticallydistributeddataandresolvedbyimposinganincoherencecondition.Suchaconditionissucientforexactrecoveryofthelowrankandthestructuredsparsecomponentbysolvinganassociatedconvexprogram.Further,[ 1 ]showedthatinanoisysetting,whereexactrecoveryofthetwocomponentsisimpossible,itisstillpossibletoachievegoodapproximationofthemodelparametersunderacomparativelymildassumption.Inparticular,ageneralmeasurefortheradiusofnon-identiabilityoftheproblemunderconsiderationisintroducedandsubsequentlyanon-asymptoticupperboundontheapproximationerrork^L)]TJ /F4 11.955 Tf 11.78 0 Td[(Lk2F+k^R)]TJ /F4 11.955 Tf 11.79 0 Td[(Rk2Fisestablishedthatdependsonthisradius.Thekeyideaistoallowforstructuredsparseandlow-rankmatricesinthemodel,butcontrollingfortheerrorintroduced.Wereferthereaderstotheabovepaperforamoredetaileddiscussiononthisnotionofnon-identiability.Inthiswork,thelow-rankplusstructuredsparsedecompositionproblem 58

PAGE 59

underrestrictionsontheradiusofnon-identiabilitytakestheform (^L,^R)=argminL,R2RppL2l(L,R):=1 2kY)-222(X(L+R)k2F+NkLk+NkRk,(3{9)where=nL2Rpp:kLkmax poornL2Rpp:kLk2,max p Ko,kkrepresentskk1orkk2,1,bothdependingonthestructuredsparsitythatRexhibits,andNandNarenon-negativetuningparameterscontrollingtheregularizationofthelow-rankandstructuredsparseparts,respectively.Theparametersandcontrolthedegreeofnon-identiablematricesallowedinthemodelclass.Inpractice,andarechosenfrom[1,p]and[1,K],respectivelyandsmallerandwillleadtobetteridentiability.TheissueofselectingthemrobustlyinpracticeisdiscussedinSection 3.5 .Remark:Oncertainoccasions,itmaybeusefultohavebothsparseandgroup-sparsestructuresinthemodel,inadditiontothelowrankone.WethenhaveR=S+Gin( 3{9 )withkRk=kSk1+N NkGk2,1.However,toguaranteethesimultaneousidentiabilityofthesparseandgroup-sparsecomponentsfromthelow-rankcomponent,strongerconditionsneedtobeimposedonL;namely,=nL2Rpp:kLkmax p&kLk2,max p Ko. 3.3ComputationalAlgorithmsandConvergenceAnalysisNext,weintroduceafastalgorithmforestimatingthetransitionmatrixBfromdata.Foreaseofpresentationandtoconveythekeyideasclearly,werstpresentthealgorithmforBrepresentingasinglestructure(e.g.onlylowrank,oronlygroupsparse,oronlysparse),andinadditionestablishitsconvergenceproperties.Subsequently,wemodifythealgorithmtohandlethecompositestructuresassumedinthispaperandalsoestablishitsconvergence.Thefastnetworkstructurelearning(FNSL)Algorithm 3.1 isdescribednext.AsafeguardedBBinitialvalueisselected,astheinitialchoiceofthenominalstepi,i.e. 0,i=maxmin,kX(Bi)]TJ /F4 11.955 Tf 11.95 0 Td[(Bi)]TJ /F6 7.97 Tf 6.59 0 Td[(1)k2F kBi)]TJ /F4 11.955 Tf 11.95 0 Td[(Bi)]TJ /F6 7.97 Tf 6.59 0 Td[(1k2Ffori>1.(3{10)Fornotationalconvenience,thepenaltytermforestimatingthetransitionmatrixBisdenotedbyPB(B,),where>0representsthetuningparameter. 59

PAGE 60

Algorithm3.1. Fastnetworkstructurelearning(FNSL)method ChooseC0,>1,0,1min.Set1=1,Bag1=B1,andQ1=0. Fori=1,2,...,k, 1. Seti=i0,i,where0,iisfrom( 3{10 ).Solveifrom1 i)]TJ /F6 5.978 Tf 5.76 0 Td[(1i)]TJ /F6 5.978 Tf 5.76 0 Td[(1=1)]TJ /F11 7.97 Tf 6.59 0 Td[(i iifori>1.ComputeBmdi=(1)]TJ /F3 11.955 Tf 11.96 0 Td[(i)Bagi+iBi,Bi+1=argminBnhrl(Bmdi),Bi+i 2kB)]TJ /F4 11.955 Tf 11.96 0 Td[(Bik2F+PB(B,)o,)]TJ /F7 7.97 Tf 6.77 -1.8 Td[(i=kBi+1)]TJ /F4 11.955 Tf 11.95 0 Td[(Bik2)]TJ /F3 11.955 Tf 13.16 8.09 Td[(i ikX(Bi+1)]TJ /F4 11.955 Tf 11.96 0 Td[(Bi)k2F,Qi+1=iQi+)]TJ /F7 7.97 Tf 18.73 -1.79 Td[(i,where0i(1)]TJ /F5 11.955 Tf 13.15 8.09 Td[(1 i)2. 2. IfQi+1<)]TJ /F4 11.955 Tf 9.3 0 Td[(C=i2,thenreplace0,iby0,iandreturntostep1. 3. ComputeBagi+1=(1)]TJ /F3 11.955 Tf 11.96 0 Td[(i)Bagi+iBi+1. EndFor OutputBagk+1. ThespecicBi+1updatedependsontheemployedpenaltyterm;foran`1penaltyinducingsparsity,itcorrespondstosoft-thresholding[ 27 ],foragroupsparsepenaltytogroupsoft-thresholding[ 112 ],whileforanuclearnormpenaltytosingularvaluethresholding[ 14 ].ItcanalsobeenseeninAlgorithm 3.1 ,thatfori1for8i1,thenBmdi=BiandBagi+1=Bi+1,whichleadstothetraditionalgradientdescentalgorithm.Indeed,Algorithm 3.1 isobtainedbyincorporatinganecientbacktrackingstrategyintotheacceleratedmulti-stepschemeby[ 77 98 ].Itprovidesadierentwaytolookforforalargerstepsizebyemployingarelaxedlinesearchcondition,insteadofsearchingforthegradientLipschitzconstantofthedatadelityterm.Infact,theparameterCinstep2playsanimportantrole,i.e.thenumberofthetrialstepscanbereducedsignicantlywhenarelativelylargerCischosen.However, 60

PAGE 61

thisCcannotbechosentoolargereither,sinceitmightimpairtheconvergencerateintermsoftheobjectivefunctionvalue.Theconvergencerateoftheproposedalgorithm 3.1 isestablishednext. Proposition3.1. LetfBagk+1gbegeneratedbyAlgorithm 3.1 .Then,foranyk1 l(Bagk+1))]TJ /F4 11.955 Tf 11.96 0 Td[(l(^B)2kXk22kB0)]TJ /F6 7.97 Tf 7.6 1.78 Td[(^Bk2F+~C (k+1)2(3{11)where~Cisanitepositivenumberindependentofk.Inthefollowingproof,wedenote1 2kY)-222(XBk2FandtheregularizationtermbyH(B)andPB(B,),respectively. ProofofTheorem 3.1 BythedierentiabilityofH,wehave H(Bagi+1)=H(Bmdi)+Z10d,(3{12) l(Bagi+1)=H(Bagi+1)+PB(Bagi+1,)=H(Bmdi)+Z10hXTX(Bmdi+(Bagi+1)]TJ /F4 11.955 Tf 11.96 0 Td[(Bmdi)))-221(XTY,Bagi+1)]TJ /F4 11.955 Tf 11.96 0 Td[(Bmdiid+PB(Bagi+1,)=H(Bmdi)+Z10hXT(XBmdi)-222(Y),Bagi+1)]TJ /F4 11.955 Tf 11.95 0 Td[(Bmdiid+Z10kX(Bagi+1)]TJ /F4 11.955 Tf 11.96 0 Td[(Bmdi)k2d+PB(Bagi+1,)=H(Bmdi)+hrH(Bmdi),Bagi+1)]TJ /F4 11.955 Tf 11.96 0 Td[(Bmdii+1 2kX(Bagi+1)]TJ /F4 11.955 Tf 11.96 0 Td[(Bmdi)k2+PB(Bagi+1,)=H(Bmdi)+(1)]TJ /F3 11.955 Tf 11.96 0 Td[(i)hrH(Bmdi),Bagi)]TJ /F4 11.955 Tf 11.96 0 Td[(Bmdii+ihrH(Bmdi),Bi+1)]TJ /F4 11.955 Tf 11.95 0 Td[(Bmdii+2i 2kX(Bi+1)]TJ /F4 11.955 Tf 11.96 0 Td[(Bi)k2+(1)]TJ /F3 11.955 Tf 11.95 0 Td[(i)PB(Bagi,)+iPB(Bi+1,)=(1)]TJ /F3 11.955 Tf 11.96 0 Td[(i)(H(Bmdi)+hrH(Bmdi),Bagi)]TJ /F4 11.955 Tf 11.95 0 Td[(Bmdii+PB(Bagi,))+i(H(Bmdi)+hrH(Bmdi),Bi+1)]TJ /F4 11.955 Tf 11.96 0 Td[(Bmdii)+2i 2kX(Bi+1)]TJ /F4 11.955 Tf 11.96 0 Td[(Bi)k2+iPB(Bi+1,).(3{13) 61

PAGE 62

whereweusethedenitionofH(B)andBagi+1,andtherelationshipBagi+1)]TJ /F4 11.955 Tf 12.15 0 Td[(Bmdi=i(Bi+1)]TJ /F4 11.955 Tf -450.77 -23.9 Td[(Bi).BytheconvexityofH(B)and( 3{13 ),wehave l(Bagi+1)=(1)]TJ /F3 11.955 Tf 11.95 0 Td[(i)(H(Bmdi)+hrH(Bmdi),Bagi)]TJ /F4 11.955 Tf 11.95 0 Td[(Bmdii+PB(Bagi))+i(H(Bmdi)+hrH(Bmdi),B)]TJ /F4 11.955 Tf 11.96 0 Td[(Bmdii)+ihrH(Bmdi),Bi+1)]TJ /F4 11.955 Tf 11.96 0 Td[(Bi+2i 2kX(Bi+1)]TJ /F4 11.955 Tf 11.95 0 Td[(Bi)k2+iPB(Bi+1,)(1)]TJ /F3 11.955 Tf 11.96 0 Td[(i)L(Bagi)+iL(B)+ihrH(Bmdi),Bi+1)]TJ /F4 11.955 Tf 11.95 0 Td[(Bi+2i 2kX(Bi+1)]TJ /F4 11.955 Tf 11.95 0 Td[(Bi)k2+iPB(Bi+1,))]TJ /F3 11.955 Tf 11.95 0 Td[(iPB(B,).(3{14)Subtractingl(B)frombothsidesof( 3{14 )andrearrangingsometerms,wehave [l(Bagi+1))]TJ /F4 11.955 Tf 11.95 0 Td[(l(B)])]TJ /F5 11.955 Tf 11.95 0 Td[((1)]TJ /F3 11.955 Tf 11.95 0 Td[(i)[l(Bagi))]TJ /F4 11.955 Tf 11.96 0 Td[(l(B)]ihrH(Bmdi),Bi+1)]TJ /F4 11.955 Tf 11.96 0 Td[(Bi+2i 2kX(Bi+1)]TJ /F4 11.955 Tf 11.95 0 Td[(Bi)k2+ih,Bi+1)]TJ /F4 11.955 Tf 11.96 0 Td[(Bi,(3{15)where2@PB(Bi+1,).Ontheotherhand,bytherst-orderoptimalityconditionsforthesequenceBi+1inAlgorithm 3.1 ,wehave hrH(Bmdi),Bei+1i+ihBi+1)]TJ /F4 11.955 Tf 11.96 0 Td[(Bi,Bei+1i+h@PB(Bi+1,),Bi+1)]TJ /F4 11.955 Tf 11.96 0 Td[(Bi0.(3{16)Combining( 3{15 )and( 3{16 ),weobtain [l(Bagi+1))]TJ /F4 11.955 Tf 11.96 0 Td[(l(B)])]TJ /F5 11.955 Tf 11.96 0 Td[((1)]TJ /F3 11.955 Tf 11.96 0 Td[(i)[l(Bagi))]TJ /F4 11.955 Tf 11.96 0 Td[(l(B)]inihBi)]TJ /F4 11.955 Tf 11.96 0 Td[(Bi+1,Bei+1i+i 2kX(Bi+1)]TJ /F4 11.955 Tf 11.95 0 Td[(Bi)k2oini 2(kBeik2)-222(kBei+1k2)-221(kBi+1)]TJ /F4 11.955 Tf 11.96 0 Td[(Bik2)+i 2kX(Bi+1)]TJ /F4 11.955 Tf 11.96 0 Td[(Bi)k2o,(3{17)whereweusedtherelationship2ha)]TJ /F4 11.955 Tf 12.14 0 Td[(b,a)]TJ /F4 11.955 Tf 12.14 0 Td[(ci=kb)]TJ /F4 11.955 Tf 12.14 0 Td[(ck2+ka)]TJ /F4 11.955 Tf 12.14 0 Td[(ck2+ka)]TJ /F4 11.955 Tf 12.14 0 Td[(bk2andthedenitionofBei+1. 62

PAGE 63

Dividingbothsidesof( 3{17 )byii,wehave 1 ii[l(Bagi+1))]TJ /F4 11.955 Tf 11.95 0 Td[(l(B)])]TJ /F5 11.955 Tf 13.15 8.09 Td[((1)]TJ /F3 11.955 Tf 11.96 0 Td[(i) ii[L(Bagi))]TJ /F4 11.955 Tf 11.96 0 Td[(L(B)]1 2(kBeik2)-222(kBei+1k2))]TJ /F5 11.955 Tf 13.15 8.09 Td[(1 2(kBi+1)]TJ /F4 11.955 Tf 11.95 0 Td[(Bik2)]TJ /F3 11.955 Tf 13.15 8.09 Td[(i ikX(Bi+1)]TJ /F4 11.955 Tf 11.95 0 Td[(Bi)k2)1 2(kBeik2)-222(kBei+1k2))]TJ /F5 11.955 Tf 13.15 8.09 Td[(1 2)]TJ /F7 7.97 Tf 6.78 -1.79 Td[(i.(3{18)Adding(iQi+)]TJ /F7 5.978 Tf 11.4 -1.41 Td[(i) 2tobothsidesof( 3{18 ),wehave 1 ii[l(Bagi+1))]TJ /F4 11.955 Tf 11.96 0 Td[(l(B)])]TJ /F5 11.955 Tf 13.15 8.09 Td[((1)]TJ /F3 11.955 Tf 11.96 0 Td[(i) ii[l(Bagi))]TJ /F4 11.955 Tf 11.96 0 Td[(l(B)]+(iQi+)]TJ /F7 7.97 Tf 18.72 -1.79 Td[(i) 21 2(kBeik2)-222(kBei+1k2)+(i)]TJ /F5 11.955 Tf 11.95 0 Td[(1)Qi 2+Qi 2.(3{19)SinceQi+1=iQi+)]TJ /F7 7.97 Tf 18.73 -1.8 Td[(i,0i(1)]TJ /F6 7.97 Tf 13.15 4.7 Td[(1 i)2,andQi)]TJ /F7 7.97 Tf 32.02 4.7 Td[(C (i)]TJ /F6 7.97 Tf 6.59 0 Td[(1)2,weobtain 1 ii[l(Bagi+1))]TJ /F4 11.955 Tf 11.96 0 Td[(l(B)])]TJ /F5 11.955 Tf 13.15 8.08 Td[((1)]TJ /F3 11.955 Tf 11.95 0 Td[(i) iil(Bagi))]TJ /F4 11.955 Tf 11.95 0 Td[(l(B)]+Qi+1 21 2(kBeik2)-222(kBei+1k2)+(1)]TJ /F3 11.955 Tf 11.96 0 Td[(i)C 2(i)]TJ /F5 11.955 Tf 11.95 0 Td[(1)2+Qi 2.(3{20)SettingB=^B,bytherelationship1 ii=1)]TJ /F11 7.97 Tf 6.59 0 Td[(i+1 i+1i+1,and1=1,weobtain 1 ii[l(Bagi+1))]TJ /F4 11.955 Tf 11.95 0 Td[(l(B)]1 2kB0)]TJ /F5 11.955 Tf 13.42 2.66 Td[(^Bk2+kXi=2(1)]TJ /F3 11.955 Tf 11.95 0 Td[(i)C (i)]TJ /F5 11.955 Tf 11.95 0 Td[(1)2+C k2.(3{21)aftersumming( 3{20 )fromi=1tok.Nextweshowtheupperboundofkk.Sincemin0,1,wehaveminkXTXk2.Then,bydenitionof0,i,weget min0,ijjXTXjj2.(3{22)Denotel0,iby0i,wherelisthenumberoflinesearchinstep3ofAlgorithm 3.1 .By1 ii=1)]TJ /F11 7.97 Tf 6.59 0 Td[(i+1 i+1i+1andthedenitionofi,wehave 1 ip 0i=p 1)]TJ /F3 11.955 Tf 11.96 0 Td[(i+1 i+1q 0i+11 i+1q 0i+1)]TJ /F5 11.955 Tf 23.14 8.09 Td[(1 20i+1fori1,(3{23) 63

PAGE 64

Then,byinductionwecanget,with1=1,(1 p 01+1 2kXi=21 p 0k)21 2k0k,whichimplies kk1 (1 p 01+1 2Pki=21 p 0k)24jjXTXjj2 (k+1)2fork1,(3{24)whereweused( 3{23 )andthedenitionof0i.Combining( 3{21 )and( 3{24 ),wehave( 3{11 ). Next,weenhancethealgorithmforsolving( 3{9 )inthegeneralcase.TheacceleratedconvergenceratecanbeobtainedbyfollowingtheproofforProposition 3.1 .SimilarlytothecaseinAlgorithm 3.1 ,theinitialtrialstepofiisasafeguardedBBchoice 0,i=maxmin,kX(Li+Ri)]TJ /F4 11.955 Tf 11.96 0 Td[(Li)]TJ /F6 7.97 Tf 6.58 0 Td[(1)]TJ /F4 11.955 Tf 11.96 0 Td[(Ri)]TJ /F6 7.97 Tf 6.59 0 Td[(1)k2F kLi+Ri)]TJ /F4 11.955 Tf 11.96 0 Td[(Li)]TJ /F6 7.97 Tf 6.59 0 Td[(1)]TJ /F4 11.955 Tf 11.96 0 Td[(Ri)]TJ /F6 7.97 Tf 6.58 0 Td[(1k2F(3{25) Algorithm3.2. AdaptiveFastnetworkstructurelearning(AFNSL)method ChooseC0,>1,0,1min.Set1=1,Lag1=L1,Rag1=R1,andQ1=0. Fori=1,2,...,k, 1. Seti=i0,i,where0,iisfrom( 3{25 ).Solveifrom1 i)]TJ /F6 5.978 Tf 5.76 0 Td[(1i)]TJ /F6 5.978 Tf 5.76 0 Td[(1=1)]TJ /F11 7.97 Tf 6.59 0 Td[(i iifori>1.ComputeLmdi=(1)]TJ /F3 11.955 Tf 11.96 0 Td[(i)Lagi+iLi,Rmdi=(1)]TJ /F3 11.955 Tf 11.96 0 Td[(i)Ragi+iRi,Li+1=argminL2nhrl(Lmdi,Rmdi),Li+i 2kL)]TJ /F4 11.955 Tf 11.96 0 Td[(Lik2F+NkLko,Ri+1=argminRnhrl(Lmdi,Rmdi),Ri+i 2kR)]TJ /F4 11.955 Tf 11.95 0 Td[(Rik2F+NkRko,)]TJ /F7 7.97 Tf 6.77 -1.79 Td[(i=kLi+1+Ri+1)]TJ /F4 11.955 Tf 11.96 0 Td[(Li)]TJ /F4 11.955 Tf 11.96 0 Td[(Rik2)]TJ /F3 11.955 Tf 13.15 8.09 Td[(i i(kX(Li+1+Ri+1)]TJ /F4 11.955 Tf 11.96 0 Td[(Li)]TJ /F4 11.955 Tf 11.96 0 Td[(Rik2F),Qi+1=iQi+)]TJ /F7 7.97 Tf 18.73 -1.8 Td[(i,where0i(1)]TJ /F5 11.955 Tf 13.15 8.08 Td[(1 i)2. 2. IfQi+1<)]TJ /F4 11.955 Tf 9.3 0 Td[(C=i2,thenreplace0,iby0,iandreturntostep1. 64

PAGE 65

3. ComputeLagi+1=(1)]TJ /F3 11.955 Tf 11.95 0 Td[(i)Lagi+iLi+1,Ragi+1=(1)]TJ /F3 11.955 Tf 11.95 0 Td[(i)Ragi+iRi+1. EndFor Output(Lagk+1,Ragk+1). TheupdateoftheLcomponentisbasedonsingularvaluethresholding,whilethatoftheRcomponenton(group)soft-thresholding.Notethatthemostexpensivecomputationaloperationcorrespondstothesingularvaluedecomposition(SVD)whenupdatingLi+1.Asmentionedearlier,theproposedalgorithmisabletolookforlargermagnitudestepsizesbyconductingfewernumberoflinesearches,duetoemployingmorerelaxedlinesearchconditions.Actually,thisisanimportantimprovementconsideringthecomputationalcostofSVD.Indeed,theeciencyoftheproposedalgorithmcanbeenhancedfurtherifweemploythetruncatedSVD[ 65 ]insteadofthefullSVD.Theconvergencerateoftheproposedalgorithm 3.2 isestablishednext. Proposition3.2. Let(Lagk+1,Ragk+1)beasequenceofupdatesgeneratedbyAlgorithm 3.2 .Then,foranyk1 l(Lagk+1,Ragk+1))]TJ /F4 11.955 Tf 11.95 0 Td[(l(^L,^R)2kXk22)]TJ /F9 7.97 Tf 5.48 -9.68 Td[(kL0)]TJ /F6 7.97 Tf 6.78 1.77 Td[(^Lk2+kR0)]TJ /F6 7.97 Tf 7.53 1.77 Td[(^Rk2+~C (k+1)2(3{26)where~Cisanitepositivenumberindependentofk.Proposition 3.2 isadirectextensionofProposition 3.1 andcanbeprovedbyfollowingtheroadmapoftheprooffortheformerresult. 3.4ErrorBoundAnalysisNext,wederivenon-asymptoticupperboundsontheestimationerrorsofthelow-rankplusstructuredsparsecomponentsofthetransitionmatrixB.ThemainresultshowsthatconsistentestimationispossiblewithasamplesizeoftheorderNpM2(fX)=m2(fX),as 65

PAGE 66

longastheprocessfXtgisstable,stationaryandtheradiusofnon-identiability,asmeasuredbykLkmaxand/orkLk2,maxissmallinanappropriatesensedetailednext.Toestablishtheresults,werstconsiderxedrealizationsofXandEandimposingthefollowingassumptions:1)restrictedstrongconvexity(RSC),i.e.1 2kXk2F 2kk2F)]TJ /F3 11.955 Tf 11.96 0 Td[(N2(),forall2
PAGE 67

(b)SupposethatthematrixLhasrankatmostr,whilethematrixGhasatmostgnon-zerogroups.Then,foranyN4kX0Ek2andN4kX0Ek2,max+4 p K,anysolutions(^L,^G)of( 3{9 )satisfyk^L)]TJ /F4 11.955 Tf 11.95 0 Td[(Lk2F+k^G)]TJ /F4 11.955 Tf 11.95 0 Td[(Gk2F4 2(9 22Nr+42Ng)Remark.ItshouldbenotedthatifeachgroupinGhasonlyoneelement,thenwehaveK=p2andgnon-zeroentries.Forsuchcases,part(b)ofProposition 3.3 becomesidenticaltopart(a).Asabyproduct,wealsogivetheestimationerrorboundofthetransitionmatrixwhichcanbecharacterizedbythesparseplusgroup-sparseandthelow-rankplussparseandgroup-sparsecomponents,respectively,undertheassumptionthatthestrengthoftheconnectionsinthegroup-sparsecomponentGisweak;i.e.G2with=nG2Rpp:kGkmax po,where2[1,p]. Corollary1. (a)SupposethatthematrixShasatmostsnonzeroentries,whilethematrixGhasatmostgnon-zerogroups.Then,foranyN4kX0Ekmax+4 pandN4kX0Ek2,max,anysolutions(^S,^G)of( 3{9 )satisfyk^S+^G)]TJ /F4 11.955 Tf 11.96 0 Td[(S)]TJ /F4 11.955 Tf 11.95 0 Td[(Gk2F4 2(82Ns+92Ng)(b)SupposethatthematrixLhasrankatmostr,whilethematrixShasatmostsnonzeroentriesandthematrixGhasatmostgnon-zerogroups.Then,foranyN4kX0Ek2,N4kX0Ekmax+4 p+4 p,andN4kX0Ek2,max+4 p K,anysolutions(^L,^S,^G)of( 3{9 )satisfyk^L)]TJ /F4 11.955 Tf 11.96 0 Td[(Lk2F+k^S+^G)]TJ /F4 11.955 Tf 11.96 0 Td[(S)]TJ /F4 11.955 Tf 11.96 0 Td[(Gk2F4 2(92Nr+25 22Ns+82Ng)NotethattheobjectiveinCorollary 1 isnottheaccuraterecoveryoftheSandGcomponentsseparately.Thelattercanbeinprincipleachieved,ifonesetstoaverysmallvalue.Inordertoobtainmeaningfulresultsinthecontextofourproblem,weneedupperboundsonkX0Ek2,kX0EkmaxandkX0Ek2,maxandalowerboundonmin(X0X)thatholdwithhigh 67

PAGE 68

probability.InthecontextoftimeserieswherealltheentriesofthematrixXaredependentoneachother,itisanon-trivialtasktoestablishsuchdeviationbounds.Akeytechnicalcontributionofthisworkistoderivethesedeviationbounds,whichleadtomeaningfulanalysisinthecontextofVARmodeling.Theresultsrelyonthemeasureofstabilitydenedin( 3{5 )andananalysisofthejointspectrumoffXt)]TJ /F6 7.97 Tf 6.59 0 Td[(1gandftg. Proposition3.4. ConsiderarandomrealizationoffX0,...,XTggeneratedaccordingtoastableVAR(1)process( 3{2 )andformtheautoregressivedesign( 3{8 ).Dene(B,)=max()1+1+max(B) min(B)Then,thereexistuniversalpositiveconstantsci>0suchthat 1. forN%p,PkX0E Nk2>c0(B,)r p Nc1exp[)]TJ /F4 11.955 Tf 9.3 0 Td[(c2logp]andforanyN%logp,P"kX0E Nkmax>c0(B,)r logp N#c1exp[)]TJ /F4 11.955 Tf 9.3 0 Td[(c2logp]andforN%mlogK,PkX0E Nk2,max>c0(B,)p mlogp p Nc1exp[)]TJ /F6 7.97 Tf 9.3 -1.8 Td[(2logp] 2. forN%pM2(fX)=m2(fX),Pmin(X0X N)>min() 2max(B)c1exp[)]TJ /F4 11.955 Tf 9.3 0 Td[(c2logp]TheproofofProposition 3.3 andpart(a)ofCorollary 1 canbeeasilyobtainedbythefollowingproofforpart(b)ofCorollary 1 68

PAGE 69

Proofofpart(b)ofCorollary 1 Bytheoptimalityof(^L,^S,^G)andthefeasibilityof(L,S,G),wehave 1 2kY)-223(X(^L+^S+^G)k2F+Nk^Lk+Nk^Sk1+Nk^Gk2,11 2kY)-222(X(L+S+G)k2F+1kLk+NkSk1+NkGk2,1.(3{27)Bysetting^L=^L)]TJ /F4 11.955 Tf 12.87 0 Td[(L,^S=^S)]TJ /F4 11.955 Tf 12.87 0 Td[(S,and^G=^G)]TJ /F4 11.955 Tf 12.86 0 Td[(GandcombiningwithY=X(L+S+G)+E,wehave1 2kX(^L+^S+^G)k2Fh^L+^S+^G,X0Ei+NkLk+NkSk1+NkGk2,1)]TJ /F3 11.955 Tf 11.95 0 Td[(NkL+^Lk)]TJ /F3 11.955 Tf 11.96 0 Td[(NkS+^Sk1)]TJ /F3 11.955 Tf 11.95 0 Td[(NkG+^Gk2,1.ByLemma1inAgarwaletal.[ 1 ]andlemma2.3inRechtetal.[ 85 ],weobtain 1 2kX(^L+^S+^G)k2Fh^L+^S+^G,X0Ei+N(k^LAk)-222(k^LBk)+2NPdj=r+1j(L)+N(k^SMk1)-222(k^SM?k1)+2NkSM?k1+N(k^GNk2,1)-222(k^GN?k2,1)+2NkGN?k2,1.(3{28)wherethematrices(A,B)2(A,B):AB0=0&A0B=0,(M,M?)and(N,N?)denoteanarbitrarysubspacepairforwhichkSk1andkGk2,1aredecomposable,respectively.Since h^L+^S+^G,X0Eik^LkkX0Ek2+k^Sk1kX0Ekmax+k^Gk2,1kX0Ek2,max(k^LAk+k^LBk)kX0Ek2(k^SMk1+k^SM?k1)kX0Ekmax+(k^GNk2,1+k^GN?k2,1)kX0Ek2,max.(3{29)Substituting( 3{29 )into( 3{28 )andrecallingconditionsforN,NandN,wehave 1 2kX(^L+^S+^G)k2F3 2Nk^LAk+3 2Nk^SMk1++3 2Nk^GNk2,1+2NPdj=r+1j(L)+2NkSM?k1+2NkGN?k2,1.(3{30) 69

PAGE 70

BytheRSCcondition,theconstraintsonLandGin( 3{9 ),andthedenitionofNandN,wehave1 2kX(^L+^S+^G)k2F 2k^L+^S+^Gk2F 2k^Lk2F+ 2k^Sk2F+ 2k^Gk2F)]TJ /F3 11.955 Tf 11.96 0 Td[(jh^L,^Sij)]TJ /F3 11.955 Tf 22.58 0 Td[(jh^L,^Gij)]TJ /F3 11.955 Tf 22.58 0 Td[(jh^G,^Sij 2k^Lk2F+ 2k^Sk2F+ 2k^Gk2F)]TJ /F3 11.955 Tf 11.96 0 Td[(k^Lkmaxk^Sk1)]TJ /F3 11.955 Tf 11.95 0 Td[(k^Lk2,maxk^Gk2,1)]TJ /F3 11.955 Tf 9.3 0 Td[(k^Gkmaxk^Sk1 2k^Lk2F+ 2k^Sk2F+ 2k^Gk2F)]TJ /F11 7.97 Tf 13.15 5.25 Td[(N 2k^Sk1)]TJ /F11 7.97 Tf 13.15 5.11 Td[(N 2k^Gk2,1)]TJ /F11 7.97 Tf 13.15 5.25 Td[(N 2k^Sk1.Insertingtheaboveinequalityinto( 3{30 ),wehave 2(k^Lk2F+k^Sk2F+k^Gk2F)3 2Nk^LAk+3 2Nk^SMk1+3 2Nk^GNk2,1+Nk^Sk1+N 2k^Gk2,1+2NPdj=r+1j(L)+2NkSM?k1+2NkGN?k2,1.BythecompatibilityconstantinAgarwaletal.[ 1 ],wehave 2(k^Lk2F+k^Sk2F+k^Gk2F)(3 2Np 2r)k^LkF+(5 2N)p sk^SkF+2Np gk^GkF+2NPdj=r+1j(L)+2NkSM?k1+2NkGN?k2,1Byourassumptions,wehave 4(k^Lk2F+k^Sk2F+k^Gk2F)q (3 21p 2r)2+(5 22p s)2+(23p g)2q k^Lk2F+k^Sk2F+k^Gk2F.Combiningwiththeinequalityk^Lk2F+k^S+^Gk2F2(k^Lk2F+k^Sk2F+k^Gk2F),weconcludethepart(b)ofCorollary 1 ProofofProposition 3.4 WewanttondupperboundsonkX0E=Nkmax,kX0E=NkandkX0E=Nk2,maxthatholdwithhighprobability.NotethatsuchanupperboundforkX0E=Nkmaxhasalreadybeenderivedin[ 5 ].Hereweadoptadierenttechniquethattakesauniedapproachtoprovideupperboundsonbothquantities.Tothisend,notethatthetwo 70

PAGE 71

normshavethefollowingrepresentations1 NkX0Ek=supu,v2Sp)]TJ /F6 5.978 Tf 5.75 0 Td[(11 Nu0X0Ev,1 NkX0Ekmax=supu,v2fe1,...,epg1 Nu0X0EvForanygivenu,v2Sp)]TJ /F6 7.97 Tf 6.59 0 Td[(1,werstprovideaboundonu0(X0E=N)v.UsingProposition2.3of[ 5 ],weobtain P[ju0(X0E=N)vj>2(A,)]6exp)]TJ /F4 11.955 Tf 9.3 0 Td[(cNminf,2g(3{31)foranyu,v2Sp)]TJ /F6 7.97 Tf 6.59 0 Td[(1andany>0.ToderivethedeviationboundonkX0E=Nkmax,wesimplytakeaunionboundoverthep2possiblechoicesofu,v2fe1,e2,...,epg.ThisleadstoP[kX0E=Nkmax>2(A,)]6exp)]TJ /F4 11.955 Tf 9.3 0 Td[(cNminf,2g+2logpSinceN%p,wecanset=p (2+c1)logp=cNsothat<1(i.e.,2<)willbesatisedforlargeenoughN.ThisimpliesthatP[kX0E=Nkmax>c0(A,)]c1exp[)]TJ /F4 11.955 Tf 9.3 0 Td[(c2logp]forsomeuniversalconstantsci>0.Toderivethedeviationboundonthespectralnorm,wediscretizetheunitballSp)]TJ /F6 7.97 Tf 6.59 0 Td[(1usingan-netNofcardinalityatmost(1+2=)p.AnargumentalongthelineofSuppelementaryLemmaF.2in[ 5 ]thenshowsthatforasmallenough>0,supu,v2Sp)]TJ /F6 5.978 Tf 5.76 0 Td[(1ju0(X0E=N)vjKsupu,v2Nju0(X0E=N)vjforsomeconstantK>1,possiblydependenton.Asbefore,takingaunionboundoverthe(1+2=)2pchoicesofuandv,wegetP[kX0E=Nk>2K(A,)]6exp)]TJ /F4 11.955 Tf 9.3 0 Td[(cNminf,2g+2plog(1+2=) 71

PAGE 72

SinceN%p,choosing=p (c1+2log(1+2=))p=cNensures<1forlargeenoughN.Settingasaboveconcludestheproof.Similarly,foranygroupGiofsizemi,wehave Pkvec(X0rEs=N,(r,s)2Gi)k>2p mi(B,)6exp)]TJ /F4 11.955 Tf 9.29 0 Td[(cNminf,2g+logmi.(3{32)TakingaunionboundoverKnon-overlappinggroupsGileadsto PkX0E=Nk2,max>2p m(B,)6exp)]TJ /F4 11.955 Tf 9.29 0 Td[(cNminf,2g+logp,(3{33)wherem=maxi=1,...,Kmi.Asbefore,setting=p logp=Nimplies PhkX0E=Nk2,max>2p mlogp=N(B,)ic1exp[)]TJ /F4 11.955 Tf 9.3 0 Td[(c2logp](3{34)forsomeci>0.WewanttoobtainalowerboundontheminimumeigenvalueofX0X=Nthatholdswithhighprobability.Sincemin(X0X=N)=infv2Sp)]TJ /F6 5.978 Tf 5.75 0 Td[(1v0(X0X=N)v,westartwiththesingledeviationboundofProposition2.3in[ 5 ],Phjv0(X0X=N)]TJ /F5 11.955 Tf 11.96 0 Td[()]TJ /F7 7.97 Tf 6.77 -1.79 Td[(X(0))vj>2M(fX)i2exp)]TJ /F4 11.955 Tf 9.3 0 Td[(cNminf,2gforanyv2Sp)]TJ /F6 7.97 Tf 6.59 0 Td[(1and>0.ThenextstepistoextendthissingledeviationbounduniformlyonthesetSp)]TJ /F6 7.97 Tf 6.58 0 Td[(1.Asintheproofofpart1,weconstructa-netofcardinalityatmost(1+2=)pandapproximatethequadraticformusingitsvaluesonthenet.ThisyieldsthefollowingdeviationboundPhsupv2Sp)]TJ /F6 5.978 Tf 5.76 0 Td[(1v0X0X N)]TJ /F5 11.955 Tf 11.96 0 Td[()]TJ /F7 7.97 Tf 6.77 -1.79 Td[(X(0)v>2KM(fX)i2exp)]TJ /F4 11.955 Tf 9.3 0 Td[(cNminf,2g+plog1+2 72

PAGE 73

forsomeconstantK>1.Seting=m(fX)=4KM(fX)<1andnotingthatN%M2(fX)=m2(fX)p,weconcludePhsupv2Sp)]TJ /F6 5.978 Tf 5.75 0 Td[(1jv0(X0X=N)]TJ /F5 11.955 Tf 11.96 0 Td[()]TJ /F7 7.97 Tf 6.77 -1.79 Td[(X(0))vj>m(fX)=2ic0exp[)]TJ /F4 11.955 Tf 9.3 0 Td[(c1logp]Theresultfollowsfromthelowerboundonm(fX)presentedin( 3{6 )andthefactthatv0)]TJ /F7 7.97 Tf 6.77 -1.79 Td[(X(0)vm(fX)forallv2Sp)]TJ /F6 7.97 Tf 6.58 0 Td[(1. Usingtheabovedeviationboundsinthenon-asymptoticerrorsofProposition 3.3 ,respectively,weobtainthenalresultforapproximaterecoveryofthelow-rankandthestructuredsparsecomponentsusingnuclearand`1=l2,1normrelaxations,asshownnext. Proposition3.5. ConsiderthesetupofProposition 3.4 .Thereexistuniversalpositiveconstantsci>0suchthatforN%pM2(fX)=m2(fX),foranyS0withkLkmax=p,anysolution(^L,^S)oftheprogram( 3{9 )satises,withprobabilityatleast1)]TJ /F4 11.955 Tf 11.96 0 Td[(c1exp[)]TJ /F4 11.955 Tf 9.3 0 Td[(c2logp], k^S)]TJ /F4 11.955 Tf 11.96 0 Td[(Sk2F+k^L)]TJ /F4 11.955 Tf 11.95 0 Td[(Lk2Fc02(B,)2max(B) 2min()(rp+slogp) N+322min() 2max(B)s2 p2(3{35)Remark:Theerrorboundpresentedintheabovepropositionconsistsoftwokeyterms.Thersttermistheerrorofestimationemanatingfromrandomnessinthedataandlimitedsamplecapacity.Foragivenmodel,thiserrorgoestozeroasthesamplesizeincreases.Thesecondtermrepresentstheerrorduetotheunidentiabilityoftheproblem.Thisismorefundamentaltothestructureofthetruelow-rankandstructuredsparsecomponents,anddependsonlyonthemodelparametersanddoesnotchangewithsamplesize.Further,theestimationerrorcomprisesoftwoterms-thesecondterm(rp+slogp)=Ninvolvesthedimensionalityparametersandmatchestheparametricconvergencerateforindependentobservations.Theeectofdependenceinthedataiscapturedthroughtherstpartoftheterm:c02(B,)2max(B) 2min().Asdiscussedin[ 5 ],thistermislargerwhenthespectraldensityismorespiky,indicatingastrongertemporalandcross-sectionaldependenceinthedata. 73

PAGE 74

Proposition3.6. ConsiderthesetupofProposition 3.4 .Thereexistuniversalpositiveconstantsci>0suchthatforN%pM2(fX)=m2(fX),foranyG0withkLk2,max=p K,anysolution(^L,^G)oftheprogram( 3{9 )satises,withprobabilityatleast1)]TJ /F4 11.955 Tf -420.04 -23.91 Td[(c1exp[)]TJ /F4 11.955 Tf 9.3 0 Td[(c2logp], k^G)]TJ /F4 11.955 Tf 11.96 0 Td[(Gk2F+k^L)]TJ /F4 11.955 Tf 11.96 0 Td[(Lk2Fc02(B,)2max(B) 2min()(rp+g(mlogp)) N+322min() 2max(B)g2 K(3{36)Remark.BasedonProposition 3.6 ,similarconclusionscanbeobtainedasthatforthelowrankplussparsecase. 3.5PerformanceEvaluationsNext,wepresentexperimentalresultsonbothsyntheticandrealdata.Specically,thersttwoexperimentsfocusonlarge-scalenetworklearningwithsinglepenaltytermtoshowtheeciencyandeectivenessoftheproposedalgorithms,whiletheremainingonesassesstheaccuracyofrecoveringlowrankplusstructuredsparsetransitionmatricesB. 3.5.1PerformanceMetricsandExperimentalSettingsWeintroducetheperformancemetricesusedinthenumericalwork.Fornetworkestimation,weusethetruepositiverate(TPR)andfalsealarmrate(FAR)denedas: TPR:=]f^bij6=0andbij6=0g ]fbij6=0g FAR:=]fbij=0and^bij6=0g ]fbij=0gwherebijand^bijarethecorrespndingelemnetsinBand^B,respectively.Theestimationerror(EE)andout-of-samplepredictionerror(PE)aredenedas EE:=k^B)]TJ /F7 7.97 Tf 6.59 0 Td[(BkF kBkF PE:=k^Y)-222(Yk2F=kYk2FToselecttheoptimalvalueofthetuningparameters,wecombinethethree(ortwoorone,respectively)-dimensionalgridsearchmethodwiththeAIC/BIC/forwardcross-validationcriterions.Wewillspecifythecriterioncasebycaseforthefollowingexperiments.In 74

PAGE 75

Parameters min C k Values 2 kX0Xk2=10 100 1/k Table3-1. Parametersettingsintheproposedalgorithmsforalltheexperiments. examplesBandC,thetuningparameterisselectedbytheAICcriterion.Agridof100valuesintheinterval[0,kX0Ykmax]isusedfor.InexamplesD,EandF,weutilizeatwo/three-dimensionalgridsearchtoselecttheoptimalvaluesofN,Nand/orNasthatfor.Fortheexperimentsemployingsyntheticdata,thetuningparametersareselectedbyassumingtherankofthetruelow-ranktransitionmatrixand/orthenon-zerogroup-sparsecomponentsofthetruegroup-sparsetransitionmatrixareknown.Wewillspecifytheforwardcross-validationprocedurefortherealdatacaseinexampleG.Foralltheexperiments,theparametersusedintheproposedalgorithmsaredepictedintable 3-1 .Alsoweset=2Iand2=1.WerescaletheentriesofBtoensurestabilityoftheprocess(thespectralradius2(0.45,0.95).Finally,allalgorithmsarerunintheMATLABR2015aenvironmentonaPCequippedwith12GBmemory. 3.5.2Large-ScaleSparseNetworkLearningWestartbycomparingtheperformanceoftheproposedalgorithm 3.1 withFISTAwithlinesearch[ 7 ]tosolveproblem( 3{9 )withthesparsitypenaltytermforlearninglarge-scalesparsenetworkstructure.WeconsiderthreedierentVAR(1)modelswithp=800,900and1000variables.Foreachofthesemodels,wegenerateN=1000,1500,and2000observationsfromaGaussianVAR(1)process( 3{2 ).ThepptransitionmatrixBwithsparsityisgeneratedinthefollowingway.First,thetopologyisgeneratedfromadirectedrandomgraphG(p,),wheretheedgefromonenodetoanothernodeoccursindependentlywithprobability=10=p.Then,thestrengthoftheedgesisgeneratedindependentlyfromaGaussiandistribution.ThisprocessisrepeateduntilweobtainatransitionmatrixBwithadesiredspectralradius.WecompareTPR,FAR,EE,andcomputationaltime(denotedbyT). 75

PAGE 76

p N method (TPR,FAR)(%) EE T 800 1000 FISTA (88.5,14.4) 0.22 35.4 FNSL (88.6,13.9) 0.22 10.7 1500 FISTA (90.8,12.3) 0.17 43.0 FNSL (90.7,12.0) 0.17 12.5 2000 FISTA (91.3,11.0) 0.14 57.3 FNSL (91.3,11.3) 0.14 18.5 900 1000 FISTA (87.8,15.0) 0.24 41.3 FNSL (87.8,14.7) 0.24 13.7 1500 FISTA (89.1,10.7) 0.19 51.3 FNSL (89.1,10.5) 0.19 17.7 2000 FISTA (90.9,12.0) 0.16 72.3 FNSL (90.9,11.8) 0.16 23.0 1000 1000 FISTA (88.5,15.2) 0.25 53.2 FNSL (88.4,14.8) 0.25 20.4 1500 FISTA (90.3,14.1) 0.20 75.8 FNSL (90.2,13.8) 0.20 24.7 2000 FISTA (91.2,12.1) 0.16 93.8 FNSL (91.2,12.4) 0.16 32.3 Table3-2. PerformancecomparisonofFNSLwithFISTAonlarge-scalesparsenetworkstructurelearningproblem. Table 3-2 showstheexperimentalresultsforsparsenetworkstructurewithdierentnetworksizepandsamplesizeN.ItcanbeseenthattheproposedalgorithmperformssimilarlytoFISTAintermsofTPR,FAR,andestimationerror.Toshowtheeciencyoftheproposedalgorithm,wealsocomparethecomputationaltimeinsecondsintermsoftheconvergenceoftheobjectivefunctionvalue.Clearly,theproposedalgorithmoutperformsFISTAineciency,especiallywhenthenetworkandsamplesizebecomelarger.Thisismainlyduetosearchingforlargerstepsizewithrelativelyfewernumberoflinesearches,aspreviouslydiscussed.Tofurthersupportourclaim,wealsoshowthegraphsofthedecreasingobjectivefunctionvaluevs.CPU,seeFigure 3-1 whenp=1000andN=2000. 3.5.3NetworkLearningWithLow-RankTransitionMatricesTofurthershowtheeciencyoftheproposedalgorithms,wecomparetheperformanceofavariantofFISTA[ 58 ]andFNSLonestimatinglow-rankthetransitionmatrices. 76

PAGE 77

Figure3-1. Theobjectivefunctionvaluesvs.CPUtimeforsparsenetworklearningproblemwithp=1000andN=2000. WeconsiderthreedierentVAR(1)modelswithp=200,300and400variables.Foreachofthesemodels,wegenerateN=400,1200,and2000observationsfromaGaussianVAR(1)process( 3{2 ).Thepplow-ranktransitionmatrixBisgeneratedwithrankbp=25c+1.Subsequently,werescaletheentriesofBtoensurethespectralradiusliesin(0.45,0.95).Wecomparetherankoftheestimatedtransitionmatrix,denotedby^r,EE,andcomputationaltimeT.Table 3-3 showstheexperimentalresultsforlow-ranknetworkstructurewithdierentnetworkandsamplesize.BothFISTAandtheproposedalgorithmachievegoodrecoveryofthetransitionmatrixBwiththecorrectrankandtheyhavesimilarperformancesintermsofestimationerror.Clearly,theproposedalgorithmoutperformsFISTAineciencyforthiscaseaswell.Thegraphsofthedecreasingobjectivefunctionvaluevs.CPUaredepictedinFigure 3-2 .Thersttwoexperimentsfocusedoncomputationaleciencyoftheproposedalgorithms,whileretaininggoodnetworkestimationproperties.Next,wedemonstratetheiraccuracyforlearningstructuredsparsenetworks. 77

PAGE 78

p N method ^r EE T 200 400 FISTA 8 0.80 11.5 FNSL 8 0.80 5.6 1200 FISTA 8 0.63 19.6 FNSL 8 0.63 12.7 2000 FISTA 8 0.57 31.8 FNSL 8 0.57 19.3 300 400 FISTA 12 0.84 17.1 FNSL 12 0.84 11.6 1200 FISTA 12 0.72 36.1 FNSL 12 0.72 23.3 2000 FISTA 12 0.68 41.7 FNSL 12 0.68 26.0 400 400 FISTA 16 0.87 40.3 FNSL 16 0.87 20.2 1200 FISTA 16 0.82 45.3 FNSL 16 0.82 23.1 2000 FISTA 16 0.75 77.4 FNSL 16 0.75 51.3 Table3-3. PerformancecomparisonofFISTAandFNSLonestimationoflow-ranktransitionmatricesproblems. Figure3-2. Theobjectivefunctionvaluesvs.CPUtimeforlow-ranktransitionmatrixestimationproblemwithwithp=400andN=2000. 78

PAGE 79

p N model (TPR,FAR)(%) EE 50 100 Lasso (67.9,16.0) 0.40 SGL (71.1,18.4) 0.41 S+G (86.7,16.5) 0.39 200 Lasso (69.3,10.4) 0.32 SGL (71.1,11.6) 0.33 S+G (90.4,10.4) 0.28 100 100 Lasso (70.2,21.2) 0.44 SGL (71.6,23.5) 0.46 S+G (84.7,21.2) 0.43 200 Lasso (87.8,17.4) 0.34 SGL (77.7,19.7) 0.35 S+G (87.8,17.4) 0.32 200 100 Lasso (75.2,43.4) 0.76 SGL (76.7,43.3) 0.76 S+G (84.3,43.7) 0.74 200 Lasso (70.1,23.1) 0.55 SGL (71.4,23.5) 0.55 S+G (78.5,23.5) 0.54 Table3-4. PerformancecomparisonofS+GwithLassoandSGLonsparseplusgroup-sparsenetworkidenticationproblem. 3.5.4SparsePlusGroup-SparseNetworkLearningProblemWestartwithanexperimentforasparseplusgroup-sparsetransitionmatrixandcomparetheperformancewithmethodsthateitherassumepuresparsity(lasso)orpuregroupsparsity(grouplasso,SGL).WeconsiderthreedierentVAR(1)modelswithp=50,100and200variables.Foreachofthesemodels,wegenerateN=100and200observationsfromaGaussianVAR(1)process( 3{2 )whereBcanbedecomposedintoasparsematrixSwith5%non-zeroentriesandagroup-sparsematrixGwitheachcolumncorrespondingtoadierentgroup(hencewehavepgroupsinG).Werandomlyselecttwocolumns(groups)consistingoftwosuperhubs,inwhichthestrengthoftheedgesisgeneratedindependentlyfromaGaussiandistribution.Tobemoreconsistentwithourerrorboundanalysis,wesettop=2in( 3{9 ).ThenetworktopologyofSisgeneratedthesamewayasthatinsubsection 3.5.2 exceptthattheoccurringprobabilityoftheedgefromonenodetoanothernodeissettobe0.05.Subsequently,theentriesof 79

PAGE 80

Figure3-3. TruenetworkstructureofS+G,SandGwithp=50andN=200. Figure3-4. Networkstructureidentiedby^S+^G,LassoandSGL. thecorrespondingtwocolumnsinGaresettobezero.Finally,werescaletheentriesofBsothatadesiredspectralradiusisreached.WeemploytheTPR,FAR,andEEmetricsinthecomparisons.Table 3-4 showstheexperimentalresultsfordierentnetworksizeandnumberofsamples.ItcanbeseenthatutilizinganS+GmodelenablesustoidentifyalargerportionofcorrectnonzeronumbersinB,whileachievingalmostthesamefalsealarmratecomparedtolassoandSGL.Particularly,theS+GmodelcanrecoverthegroupinformationperfectlywhilelassoandSGLmissanumberofedges,asexpectedsincetheycorrespondtomisspeciedstructures. 80

PAGE 81

(TPR,FAR)(%) p/8 (84.5,17.4) p/4 (82.4,17.2) p/2 (80.4,17.5) p (73.2,17.1) 2p (54.7,17.0) 4p (40.2,17.8) 8p (22.7,17.4) Table3-5. TruepositiverateandfalsealarmrateoftheL+SmodelonidentifyingthesparsecomponentSwithdierent. Further,theS+Gmodelexhibitsthelowestestimationerroramongstthem.ItshouldbenotedthattheadvantageoftheS+Gmodelwillbemoreevidentifthestrengthoftheedgeswithinthegroupsisweaker.Figure 3-3 showsthetruenetworkstructureS+G,SandGwithp=50andN=200.TherecoverednetworkstructuresbyS+GmodelaregiveninFigure 3-4 (top).Clearly,thegroup-sparsecomponentsarerecoveredperfectly.WealsocomparetherecoverednetworkstructuresbyS+G,SGLandlassomodels,seeFigure 3-4 (bottom),fromwhichwecanseethatS+Gperformsbest. 3.5.5SparsePlusLow-RankNetworkLearningNext,weinvestigateestimationofsparsepluslow-ranktransitionmatricesandcompareittoordinaryleastsquare(OLS)andlassoestimates.WeconsiderthreedierentVAR(1)modelswithp=50,75and100variables.Foreachofthesemodels,wegenerateN=100and200observationsfromaGaussianVAR(1)process( 3{2 )whereBcanbedecomposedintoalow-rankmatrixLofrankbp=25c+1andasparsematrixSwith2)]TJ /F5 11.955 Tf 12.23 0 Td[(4%non-zeroentries.WerescaletheentriesofBtoensurestabilityoftheprocess(thespectralradiusissetto(B)=0.7).Wecomparetheestimationandout-of-samplepredictionerrors.Thenumberofoutofsamplesissetto10.First,westudytheinuenceofin( 3{9 )onthislearningproblemwithp=50andN=200.FromTable 3-5 ,thatasmallerparameterleadstomarkedlyimproved 81

PAGE 82

p N model (TPR,FAR)(%) EE PE 50 100 OLS (-,-) 0.84 0.72 Lasso (73.2,30.0) 0.69 0.53 L+S (76.3,18.9) 0.48 0.47 200 OLS (-,-) 0.52 0.41 Lasso (77.3,35.0) 0.57 0.45 L+S (80.4,17.5) 0.31 0.36 75 100 OLS (-,-) 0.75 0.37 Lasso (71.0,24.7) 0.75 0.37 L+S (79.0,18.0) 0.51 0.29 200 OLS (-,-) 0.53 0.18 Lasso (77.0,28.6) 0.67 0.22 L+S (83.8,18.3) 0.36 0.16 100 100 OLS (-,-) 3.7 4.0 Lasso (57.3,29.0) 1.06 1.05 L+S (52.3,20.1) 0.92 1.0 200 OLS (-,-) 2.07 1.73 Lasso (59.4,25.5) 0.86 0.95 L+S (60.4,20.5) 0.72 0.90 Table3-6. PerformancecomparisonofL+SwithOLSandLasso. identicationofallthetruenonzeroentriesinthesparsecomponent,whichconsequentlyleadstobetterseparatingthesparsecomponentSfromthelow-rankcomponentL.ThecorrespondingestimationerrorsarereportedinTable 3-6 .Inallthethreesettings,wendthatthelow-rankplussparseVARestimatesoutperformtheestimatesusingordinaryleast-squaresandlasso,asexpected.Weobservethat,astheratioofN=pincreases,OLSmayproducelowerestimationerrorthanlasso,eventhoughOLSmodelisnotinterpretableforthiscase.Also,weobservethattheestimationerrorsofallthreemethodsdecreasewithincreasingsamplesizesasexpectedandpredictedbytheory.Inadditiontoitsimprovedestimationandpredictionperformance,thelow-rankplussparsemodelingstrategyaidsinrecoveringtheunderlyingGrangercausalnetworkafteraccountingforthelatentstructure.InFigures 3-5 ,wedemonstratethisusingaVAR(1)modelwithp=50andn=200.ThetoppaneloftheFigures 3-5 displaysthetruetransitionmatrixB,itslow-rankcomponentLandthestructureofitssparsecomponentS.ThebottompaneloftheFigures 3-5 displaysthestructureoftheGrangercausalnetworksestimatedbythe 82

PAGE 83

Figure3-5. EstimatedGrangercausalnetworksusinglassoandlow-rankplussparseVARestimates. methodofLassoandthelow-rankplussparsemodelingstrategy.Aspredictedbythetheory,itcanbethatthelassoestimateoftheGrangercausalnetworkselectsmanyfalsepositivesduetoitsfailuretoaccountforthelatentstructure.Ontheotherhand,thesparsecomponentSprovidesanestimateexhibitingsignicantlyfewerfalsepositivesentries.Itisinterestingtonotethattheestimationperformanceoftheregularizedestimatesinlow-rankplussparseVARmodelsisworsethantheperformanceoflassoinsparseVARmodelsofsimilardimension[ 5 ],evenforthesamesamplesizes.ThisisinlinewiththeerrorboundspresentedinProposition 3.5 .Theestimationerrorinlow-rankplussparsemodelsisoftheorderofO(rp+slogp)=N,whiletheerroroflassoinsparseVARmodelsscalesatafasterrateofO(slogp=N).Furthernotethatas-sparseVARrequiresestimatingsparametersinS,whilethepresenceofrfactorsintroducesanadditionalrpparametersintheloadingmatrix. 83

PAGE 84

p N method (TPR,FAR)(%) EE PE 50 200 S+G (85.5,34.1) 0.46 0.53 L+S (82.6,26.4) 0.41 0.51 L+S+G (83.3,26.9) 0.41 0.51 300 S+G (91.7,47.0) 0.37 0.58 L+S (88.6,24.4) 0.31 0.56 L+S+G (90.6,24.9) 0.30 0.56 100 200 S+G (92.3,49.9) 0.48 0.73 L+S (84.3,28.4) 0.44 0.72 L+S+G (85.3,27.3) 0.44 0.72 300 S+G (94.8,49.0) 0.44 0.72 L+S (89.6,25.4) 0.37 0.70 L+S+G (90.0,25.1) 0.36 0.70 150 200 S+G (92.0,50.5) 0.64 0.73 L+S (83.3,28.8) 0.57 0.71 L+S+G (84.0,28.1) 0.55 0.70 300 S+G (93.6,50.2) 0.55 0.71 L+S (85.6,27.4) 0.46 0.68 L+S+G (86.4,27.6) 0.46 0.68 Table3-7. PerformancecomparisonofL+S+GwithS+GandL+S. 3.5.6SparsePlusGroup-SparsePlusLow-RankNetworkLearningFinally,weconductnumericalexperimentstoassesstheperformanceoflow-rankplussparseplusgroup-sparsemodelinginVARanalysisandcompareittotheperformanceofsparseplusgroup-sparseandlow-rankplussparseestimates.WeconsiderthreedierentVAR(1)modelswithp=50,100and150variables.Foreachofthesemodels,wegenerateN=200and300observationsfromaGaussianVAR(1)process( 3{2 ),whereBcanbedecomposedintoalow-rankmatrixLofrankbp=25c+1,asparsematrixSwith2)]TJ /F5 11.955 Tf 12.09 0 Td[(4%non-zeroentries,andagroup-sparsematrixGwitheachcolumncorrespondingtoadierentgroupforatotalofpgroups.WerescaletheentriesofBtoensurestabilityoftheprocess(thespectralradiusissetto(B)=0.7)andcomparetheestimationandout-of-samplepredictionerrors,withthenumberofout-samplessetto10.ThecorrespondingestimationerrorsarereportedinTable 3-7 .Inthosethreesettings,wendthatthelow-rankplussparseplusgroup-sparseVARestimatesperformsonlyslightlybetterthanlow-rankplussparseVARestimates.Oneofthereasonslieinthattheability 84

PAGE 85

oftheidenticationwilldegradeasmorestructuresareinvolved.Theotheroneisthatmultiple-timesshrinkageforthemultiplestructuresleadtoseverebiasestimation.Eventhoughthegroupstructurescanberecoveredcompletely,somenon-zeroelementsinsparsecomponentvanished.Anad-hocwaytoimprovetheperformanceforthiscaseistocombinethistwomethodstogether.However,bothmethodsoutperformtheestimatesusingsparseplusgroup-sparseVAR.Wealsoobservethat,astheratioofN=pincreases,theestimationerrorsofallthreemethodsdecreasewithincreasingsamplesizesasexpectedandpredictedbytheory.Figures 3-7 showstheestimatedGrangercausalnetworkusinglow-rankplussparseplusgroup-sparseVARestimatesusingaVAR(1)modelwithp=50andn=300.ThetoppaneloftheFigures 3-5 displaysthetruethestructureofsparseplusgroup-sparsecomponentsS+G,thestructureofgroup-sparsecomponentG,andthestructureoflow-rankcomponentL.ThebottompaneloftheFigures 3-5 displaysthestructureoftheGrangercausalnetworksestimatedbythemethodofL+SandS+Gmodelingstrategy.ItcanbeseenthattheS+Gestimateselectsmanyfalsepositivesduetoitsfailuretoaccountforthelatentstructure.Ontheotherhand,theL+SmethodprovidesanestimateexhibitingsignicantlyfewerfalsepositivesentriesasthatbyL+S+G. 3.6ConclusionsofThisChapterNetworkmodelingofhigh-dimensionaltimeseriesdatarepresentsakeylearningtaskduetoitsuseinanumberofapplicationareas,includingmacroeconomic,nanceandneuroscience.Whiletheproblemofsparsemodelingbasedonvectorautoregressivemodels(VAR)hasbeeninvestigatedintheliterature,moreinvolvednetworkstructuresthatinvolvelowrankandgroupsparsecomponentsremainunexploredatlarge,despitetheirpresenceindata.Failuretoaccountforsuchstructuresresultsinspuriousconnectivityamongsttheobservedtimeseries,whichmayleadpractitionerstodrawthewrongconclusionsaboutpertinentscienticorpolicyquestions.InordertoaccuratelyestimateanetworkofGrangercausalinteractionsafteraccountingforlatenteects,weintroduceanovelapproachoflow-rankandstructuredsparseVARmodels.Weintroducearegularizedframeworkinvolvinga 85

PAGE 86

Figure3-6. EstimatedGrangercausalnetworksusinglow-rankplussparseplusgroup-sparseVARestimates. Figure3-7. EstimatedGrangercausalnetworksusinglow-rankplussparseandgroup-sparseVARestimates. 86

PAGE 87

combinationofnuclearnormandstructuredsparse(lasso/grouplasso).Further,weestablishnon-asymptoticupperboundsontheestimationerrorratesofthelow-rankandthestructuredsparsecomponents,basedonfastestimationalgorithmsanddemonstratetheperformanceoftheproposedmodelsoverordinaryandsparseVARestimatesthroughnumericalexperimentsonsyntheticandrealdata. 87

PAGE 88

CHAPTER4INFORMATIONEXTRACTIONFORSCANNINGTUNNELINGPOTENTIOMETRYANDMICROSCOPEIMAGING 4.1OverviewofScanningTunnelingPotentiometry/MicroscopeProgressinmaterialsscienceandelectronicdevicesisdrivenbyscalingtosmallerandlowerdimensions.Whenthesizeofasampleissmallerthantheelectronmean-free-path,forexample,itsconductivitycanbedominatedbythepresenceandcharacterofinterfacessuchasgrainboundaries.Interfaceorgrainboundaryscatteringcanbeevenmoreimportantinreduceddimensions.Grainboundariesinpolycrystallinegraphenehavebeenshowntostronglyimpacttheelectronicpropertiesofthematerial[ 25 94 97 111 ].InmonolayerMoS2,localconductivityatagrainboundaryexhibitsanisotropyalongandacrosstheboundary,withdierentanisotropiesfordierenttypesofgrainboundaries[ 39 100 ].Conventionaltransportmeasurementtechniquemeasuresthemacroscopiccurrent-voltagerelationshipforasampleandextractstheconductivity.Theestimateofthegrainboundaryconductivityreliesontheestimateofthegrainboundarydensity(grainsize)andthemodeloftheresistornetworkformedbythegrainboundaries.Otherextraneousresistancesourcessuchasphononandimpurityscatteringmustalsobecarefullyexcluded.Eveninthebestexperiments,thismethodcanonlyyieldanaverageestimateofthegrainboundaryconductivity[ 67 110 117 ].ApplicationofSTMtomaterialsresearchhasbeenmostfruitfulwhenitiscombinedwithmeasurementsoftransportproperties[ 60 ].Itisaparticularlyusefultoolforstudying2Dmaterialssuchasgraphene[ 24 25 60 ]andtopologicalinsulators[ 31 ].IncontrasttotheatomicresolutionimagesontopographymeasurementsusingSTM,atomicresolutionfortransportmeasurementsisafargreaterchallenge.Yetmanyofunresolvedproblemsintransportpropertiesofgrapheneandother2Delectronicmaterialsarerelatedtospatialinhomogeneityinthesematerialsonthenanometerscale.Therefore,acapabilityofreal-space,atomicresolutiontransportmeasurementswouldprovideapowerfultoolforovercomingthesescienticchallenges. 88

PAGE 89

Figure4-1. OriginalSTPdataofagraphenegrainboundaryonaSiO2/Sisubstrate Scanningtunnelingpotentiometry(STP)[ 73 ]hasbeenrecentlyemployedtoyieldtwo-dimensionalmapsoftheelectrochemicalpotentialonthesurfaceofamaterialwhileanelectriccurrentisowingalongthesurface[ 53 57 102 ].Theobviousapproachusingthistechniqueforconductivitymeasurementofagrainboundaryistoperformascanalongalineperpendiculartotheboundary,andextractthelocalconductivityusingthepotentialprolealongtheline[ 94 ].Thismethodproducesexcellentresultifthelocalcurrentdirectionisexactlyperpendiculartothegrainboundary.Onewaytoensurethatistomakethemeasurementonananowireetchedfromalargersamplecontaininggrainboundaries[ 59 ].Ingeneral,however,thisisnotafeasibleapproach.HerewepresentanalgorithmforextractingaconductivityprolefromlargeamountofdataproducedbySTP.TypicalexperimentaldataislikethetwopanelsshowninFIG. 4-1 forasampleofagrainboundaryofgrapheneonaSiO2/Sisubstrate,withtheforwardscanin(a)andbackwardscanin(b).Thepreparationofthesampleandthemeasurementdetailsarethesameasdescribedinapreviouswork[ 25 ].Thenoiseinthepotentiometrymapisclearlyvisible.Wesolveasetofunderdeterminedpartialdierentialconductivityequationsusingasinputthenoisy2Dpotentialproleobtainedfromsuchmaps.Thelinearsystemcontainsgradientsthusisunusuallysensitivetothenoiseinthedata.Thereforewedevisearobustandreliablenoiseremovalprocedurebeforeconductivityisextracted.Themethodpresented 89

PAGE 90

herecoversboththenoiseremovalprocedureandthenumericalconductivityreconstruction.Itexploitsthedataredundancyintheforwardandbackwardscansofthesamesampletoreduce/eliminatethenoise.Thisworkrepresentsasignicantadvancefromapreviousworkthatextractedthegrainboundaryresistanceofgraphene[ 25 ].Comparedtothepreviouswork,thecurrentmethodismorerobust,ensuresconvergence,andgreatlyreducestheeectofnoise.Weintroduceouralgorithmsforextractingconductivityproleanddatapreprocessinginsection 4.2 anddemonstratethemethodonasetofgraphenegrainboundarydatainsection 4.3 .Theinventionofthescanningtunnelingmicroscopy(STM)revolutionizedthestudyofnanoscaleandatomicscalesurfacestructuresandproperties[ 10 95 ].However,STMhasrarelybeenconsideredareal-timemethodbecauseofitsslowscanningratecomparedtomostdynamicprocessesonasurface[ 63 ].Thishasseverelylimiteditsapplicationtothestudyofmostdynamicprocessesonsurfacessuchassurfacediusion,phasetransitions,self-assemblyphenomena,lmgrowthandetching,chemicalreactions,conformationalchangesofmolecules.Raisingthescanrateofscanningprobeshasbeentheobjectiveofintenseresearcheortsinthepastdecades[ 33 63 86 ],withmostoftheeortsfocusedonhardwareimprovements.Ontheotherhand,researchershavealsoappliedothertechniquestoutilizeconventional,slow-scanSTMtostudydynamicprocesses.Forexample,low-frequencydynamicbehaviorofaexiblefree-standinggraphenesheethasbeenstudiedusingcleverpost-processing[ 105 ].ThecommonpracticeforSTMmeasurementisbybringingthetipclosetothesamplesurfaceandapplyingavoltagebiastogenerateatunnelcurrentbetweenthetipandsample.Thetipismovedacrossthesampleparalleltothesurface(inthexyplane).Changesinthesurfaceheightzorinthedensityofstatescauseachangeinthetunnelingcurrent.Thechangeincurrentwithrespecttopositioncanbemeasureditself,oralternativelytheheightofthetipcorrespondingtoaconstantcurrentcanbemeasured.Thesetwomodesarecalledtheconstantheightmodeandtheconstantcurrentmode,respectively.Intheconstantcurrentmode,feedbackelectronicsadjusttheheightbyavoltagetothepiezoelectricheightcontrol 90

PAGE 91

mechanism.Intheconstantheightmode,thevoltageandheightarebothheldconstantwhilethecurrentchangestokeepthejunctionvoltagefromchanging.TheconstantcurrentmodeisusuallyusedinSTMbecausesurfacefeaturescaneasilyexceedapre-denedtip-sampleseparation(typically4-7A)andcancrashthetipinaconstantheightmode.Buttheconstantcurrentmodeisslow,duetomoretimerequiredbythepiezoelectricmovementstoregistertheheightchange.Thetimetocompleteameasurementforeachpixelpositionisabout2msecforatypicalequipment,andapproaches0.1msecforatop-of-linesetup[ 33 63 86 ].ConventionalSTMhasalimitedscanningspeedbecauseoftheresponsetimeoftheelectriccircuitandthepiezoelectriccomponentusedtocontrolthemovementoftheprobes.Thecontrolofthemotionofthetiprequiresanelectricfeedbackcircuitwhichcangeneratearesonanceatthefrequencyoftheorderof10kHz.Ifthescanningspeedistoofastlargenoiseappearsintheimageintheformofaspatialoscillation(Fig. 4-2 ),withtopographyrawdatafromforwardscan(Fo),scanarea8080nm2,oreectivescanningspeed1.56=secin(a),topographyrawdatafrombackwardscan(Bo)in(b),atomicresolutionrawdatafromforwardscan,scanarea1010nm2,oreectivescanningspeed195nm=secin(c),atomicresolutionrawdatafrombackwardscanin(d).Brightstreaksareindicationsofpossibletipcrashes..However,belowacriticalscanratedeterminedbytheresonantfrequency,suchnoiseusuallyappearsdierentlyintheforwardscansandthebackwardones.Thereforeonecandeviseanalgorithmthatexploitsthedierencebetweentheforwardandbackwardscanstoeliminatetheoscillatorynoiseintheimage,thusallowingfasterscanstobeperformed.Foratomicresolutionimages,theerrorduetoafastscanrateappeartobelargeintensityuctuationsbetweenscanlines,asshowninFig. 4-2 (c)and(d).Theselargeuctuationsarelikelycausedby\soft"crashesofthetip,anunavoidableproblemwhenthescanrateisdrivenfasterthantheresponserateofthefeedbackcircuit.However,aftereachtipcrash,theatomicfeaturesremainbutwithadierentbackgroundintensity.Thusaneectivebackgroundremovelprocedureisneededbeforetheforwardandbackwardscanimagescanbecombinedtoobtainanoptimalresult. 91

PAGE 92

Figure4-2. STMdataofasinglecrystalCu(111)surfaceobtainedwithafastscanrate(0.1msecperpixel) TheeectivenessofthemethodisdemonstratedontopographydatafromasinglecrystalCu(111)surface.CusurfaceisaprototypicalplatformforatomicresolutionSTMstudiesonelectronscattering[ 51 ],nanostructuresynthesis[ 92 96 ],andcatalysisbehaviors[ 38 ].ThefasterscanningwillallowrevealingdynamicprocessesonCusurface,suchasmolecularself-assembly[ 62 ],surfacediusion[ 30 ],andchemicalreaction[ 52 ].Wewillshowthatafterprocessingusingthealgorithmpresentedinthispaper,theobtainedtopographyimagefromafastscanrateof0.1msecperpixel(eectivescanningspeed1.56=secforscanarea8080 92

PAGE 93

Figure4-3. ComparisonofprocessedtopographyimagesoftheCusampleinFig. 4-2 nm2)nearlymatchesthequalityoftheimageofthesamesamplefromaslowscanrateof2msecperpixel,andalsoeliminatesthescandriftthatisapparentintheslowscanimage(Fig. 4-3 )with(a)slowscanrate(2msecperpixel)dataaveragedoverforwardandbackwardscansand(b)processeddatausingthealgorithmpresentedinthispaperfromthefastscanrate(0.1msecperpixel)datashowninFig. 4-2 .Scandriftisclearlyvisiblein(a),appearingasagradualchangeintheheight(representedbythecolor)ofaplateau.Thereisnovisiblescandriftin(b).Foratomicresolutiondata,ourmethodisabletorecoveratomicimages(Fig. 4-4 (a)Resultafterbackgroundremoval,registration,andimagerestorationsteps;(b)Therankingmapofthepost-processedimageasthenalatomicresolutionimage)fromafastscandatawithascanratealsoat0.1msecperpixel(eectivescanningspeed195nm=secforscanarea1010nm2)wheretherawimageisoverwhelmedbynoiseandtipcrashstreaks(seeFig. 4-2 ).Weintroduceourmethodsofbackgroundremovalandsignalregistrationinsection 4.2.3 andsignalregistrationinsection 4.2.4 ,followedbytheproposedsignalrestorationmethodandatomicimageextractionmethodinsection 4.2.5 .ThedescriptionoftheSTMexperimentalsetup,samplepreparation,andresultsofourmethodappliedonadatasetwithfastscan 93

PAGE 94

Figure4-4. ProcessedatomicresolutionimageoftheCusamplefromthefastscandatainFig. 4-2 Figure4-5. AverageoftheforwardandbackwardscansinFig. 4-2 speedandcomparisontodatawithslowscanspeedarepresentedinsection 4.3 .Concludingremarksaregiveninsection 4.4 94

PAGE 95

Figure4-6. DierenceoftheforwardandbackwardscansinFig. 4-2 4.2ProposedMethods 4.2.1ExtractionofConductivity 4.2.1.1GeneralEquationsInSTP,theconductivitycanbesolvedfromtheelectrochemicalpotentialmapbysolvingaPoisson-likeequation,whichisderivedfromthecurrentcontinuityconditionrJ=0awayfromthecurrentsource.UsingOhm'slawforthecurrentdensity,J=)]TJ /F3 11.955 Tf 9.3 0 Td[(rwhere=(x,y)istheconductivitytensoratlocation(x,y)andistheelectrochemicalpotential,wend, r(r)=0.(4{1)inwhichrprovidestheknowncoecients.Becauseisatensor,theabovepartialdierentialequationprovidesonlylessthanhalfnumberofequationsneededtodetermine.Anadditionalsetofequationsisprovidedbytherequirementthattherearenocurrentloopsinthesystem,rJ=0,whichtranslatesto, r(r)=0.(4{2) 95

PAGE 96

Writingthetensorintoavectorform,=(1(x1,y1),1(x2,y2),...,2(x1,y1),2(x2,y2),...),wecancombinethetwosetsofequationsas, A=0.(4{3)Theboundaryconditionsarenotusuallyknownbecausethecurrentprobesareplacedfarawayfromthescannedregiontoensurearelativelyuniformcurrentdensityinthescannedregion.Duetothepresenceofinhomogeneity,thelocalcurrentdirectionisunknown,thereforetheboundaryconditionscannotbedetermined.Asaresult,thesystemofequationsisunderdetermined.Astraightforwardwaytosolveproblem( 4{3 )approximatelywouldbeusingthegradientdescentmethodtosolvetheassociatedminimizationproblem minkk2=1kAk2(4{4)withthefollowingscheme, (k+1)=(k)+pk(ATA)-222(kAk2),(4{5)wherethestepsizepkischosensuchthattheglobalerrorofthelinearsystemisminimized.However,thegradientdescentmethodhaslowrateofconvergenceO(1=N)intermsoftheobjectivefunctionvalue.Furthermore,thisproblem( 4{4 )isill-posedsoweproposetointroduceregularizationtermsintothemodelwithappropriateassumptions. 4.2.1.2TotalVariation(TV)MethodToimprovetheaccuracyofthesolutionandtoreducethecomputationeort,weapplythefollowingtotalvariation(TV)regularizedlinearinversionmodel, minkk=1Zjr(r))j2+jr(r)j2+jrjdxdy,(4{6)wherethersttwotermsenforcetobeapproximatesolutionsofEq.( 4{1 )and( 4{2 ),andthelasttermisaTVregularization,whichisanedge(ordiscontinuity)preservingsmoothing. 96

PAGE 97

Toavoidthetrivialsolution=0,theconstraintkk=1isimposedtothemodelsuchthatthesolutionoftheconstraintproblemalwaysstaysontheunitsphere.Thediscreteformreadsas, minkk=1kAk2+(kD1k2,1+kD2k2,1).(4{7)Here=[12]TandkDk2,1denotesthediscreteformoftheTVseminormofudenedasPip (@xui)2+(@yui)2where@xand@yrepresentnitedierenceoperatoralongxandydirectionofthe2Dgridatpixelirespectively.Theminimizationproblem( 4{7 )canbesolvedthroughtheAlternatingDirectionMethodofMultiplier(ADMM)byintroducingauxiliaryvariablesw1=D1andw2=D2.Thentheoriginalproblemistransformedtothefollowingconstraintminimizationproblem: minkk=1,w1,w2kAk2+(kw1k2,1+kw2k2,1)s.t.w1=D1,w2=D2.(4{8)ThentheaugmentedLagrangianfor( 4{8 )is: L(,y1,w1,y2,w2)=kAk2+Xi=1,2(kwik2,1)-222(hyi,wi)]TJ /F4 11.955 Tf 11.95 0 Td[(Dii+ 2kwi)]TJ /F4 11.955 Tf 11.96 0 Td[(Dik2).(4{9)TheADMMsolves(,w1,w2,y1,y2)iterativelyandalternativelybyminimizing(forandw)ormaximizing(fory)theLagrangianwithrespecttoonevariablewhilexingtheothers.ToacceleratetheconvergenceofADMM,anacceleratedlinearizedandpreconditionedADMM(ALP-ADMM)wasproposed[ 81 ].DierentfromtheclassicalADMM,ALP-ADMMintroducesequencesmdandagfortheaccelerationpurposeinspiredbytheNesterovacceleratingscheme.Inaddition,whensolvingsub-problemandwsub-problem,theALP-ADMMlinearizeskAkand 2kwi)]TJ /F4 11.955 Tf 12.12 0 Td[(Dik2att-thiteratemdttosimplifythecalculationofthesub-problem.Itisproven[ 81 ]thatALP-ADMMhastherateofconvergenceofO(1=N2)withregardtothesmoothterm(therstterminthiscase)whilethestandardADMMonlyoerstherateofO(1=N).ThedetailsofthealgorithmisgivenbyAlgorithm 4.1 97

PAGE 98

Thesub-problem( 4{14 )canbesolvedexactlyandthenmapsontotheunitspherebynormalization:^=t)]TJ /F5 11.955 Tf 15.05 8.09 Td[(1 t(ATAmdt+[^W1t^W2t]T) (4{10)t+1=^=k^k (4{11)where^Wit=DT((yi)t+(wi)t)]TJ /F4 11.955 Tf 11.96 0 Td[(D(i)t)fori=1,2.Thewsub-problem( 4{16 )canbesolvedexactlyusingshrinkageoperator: (wi)t+1=Swi)]TJ /F4 11.955 Tf 11.95 0 Td[(D(i)t+1)]TJ /F5 11.955 Tf 13.15 8.09 Td[((yi)t t, t(4{12)whereS(x,):=sign(x)fmaxjxj)]TJ /F3 11.955 Tf 17.94 0 Td[(,0g. Algorithm4.1. TheALP-ADMMalgorithmforsolvingproblem( 4{7 ) Choose1withk1k2=1andsetw1=D1,ag1=1,wag1=w1andyag1=0. Fort=1,,N)]TJ /F5 11.955 Tf 11.95 0 Td[(1,do Updatevariables:mdt=(1)]TJ /F3 11.955 Tf 11.95 0 Td[(t)agt+tt (4{13)t+1=argminkk2=1hATAmdt,i+t 2k)]TJ /F3 11.955 Tf 11.95 0 Td[(mdtk2+h(y1)t,D1i+h(w1)t)]TJ /F4 11.955 Tf 11.96 0 Td[(D(1)mdt,D1i+h(y2)t,D2i+h(w2)t)]TJ /F4 11.955 Tf 11.96 0 Td[(D(2)mdt,D2i (4{14)agt+1=(1)]TJ /F3 11.955 Tf 11.95 0 Td[(t)agt+tt+1 (4{15) Updateauxiliaryvariableswifori=1,2:(wi)t+1=argminwikwik2,1)-222(h(yi)t,wii+t 2kwi)]TJ /F4 11.955 Tf 11.96 0 Td[(D(i)t+1k2 (4{16) 98

PAGE 99

Figure4-7. Comparisonoftheaveragedpotentialc=(f+b)=2. Updatemultiplieryifori=1,2:(yi)t+1=(yi)t)]TJ /F3 11.955 Tf 11.95 0 Td[(t((wi)t+1)]TJ /F4 11.955 Tf 11.96 0 Td[(D(i)t+1) (4{17) EndFor Output=agN. 4.2.2PreprocessingPreprocessingisanessentialpartinourproposedmethodofextractingconductivityprole.Firstly,toobtainareliablewiththeforwardandbackwardscansofthesameline,weneedtoperformimageregistrationonthemastheytendtonotalignedwellinpractice.Secondly,themodel( 4{8 )involvestherstandsecondorderderivativesof,hencethesmoothnessofshouldbeensuredinadvanceoftheimagerestorationprocess.Thepreprocessingofthedataisdividedintotwokeysteps:1.imageregistrationofforward(f)andbackwardscan(b);2.restorethecombineddatafromtheprocessedforwardandbackwardscans(c)byTVbaseddenoising. 99

PAGE 100

Figure4-8. Comparisonoftheabsolutedierencesofjf)]TJ /F3 11.955 Tf 11.96 0 Td[(bj. 4.2.2.1ImageRegistrationThepurposeofimageregistrationistoutilizetheforwardandbackwardscantorestoreareliable.Ourapproachfortheimageregistrationfollowsa2-stepprocess:rst,theimagesfromforwardscanandbackwardscan,representedbyf2Rmnandb2Rmn,arealignedthroughglobalshiftings;next,adeformableinverseconsistentimageregistrationisperformedtoobtainlocaltransformationsunderwhichfandbwouldhavegreatercorrelation.Sincethedata,althoughhastwodimensions,isacquiredthroughalinebylinemanner,theimageregistrationmethodsdescribedaboveareappliedtoeachpairoflinesfromforwardandbackwardscans.Fortherststep,considerthei-thlinesoffandbwhicharerepresentedbyf2R1nandb2R1nrespectively.Thentheobjectiveistondaconstantc,suchthatf(x+c)issimilartob(x)]TJ /F4 11.955 Tf 11.95 0 Td[(c)whichisgivenbythefollowingmodel: minc2Zkf(x+c))]TJ /F4 11.955 Tf 11.96 0 Td[(b(x)]TJ /F4 11.955 Tf 11.96 0 Td[(c)k2(4{18)Duetothesmallsizeofthesearchingspaceforcinpractice,wesolvethisminimizationproblemthroughheuristicsearch. 100

PAGE 101

Aftertherststep,twosignalsfandbareroughlyaligned.Tofurtherincreasethecorrelationbetweenthem,weadoptainverseconsistentdeformableimageregistration(ICDIR)[ 23 ]whichallowslocalizedtransformation.Inthisproblem,weaimstonddeformationsu,v:R1n7!R1nsuchthatthedeformedsignalsf(x+u)andg(x+v)areclosetoeachother.ThemathematicalmodelforICDIRconsiststhreepartsandreadsas minu,~u,v,~v;F(u,v,)+(I(u,~u)+I(v,~v))+R(u,~u,v,~v).(4{19)ThersttermF(u,v,)representsthedatadelityandisdenedby F(u,v,):=kf(x+u(x)))]TJ /F4 11.955 Tf 11.95 0 Td[(b(x+v(x))k2=22+jjlog.(4{20)whereRisthedomainthatfandgdenedon.Itisthenegativelog-likelihoodofthetheresidueimagef(x+u(x)))]TJ /F4 11.955 Tf 11.65 0 Td[(b(x+v(x))undertheassumptionthatthepixelintensitiesoftheresidueimageareindependentsamplesdrawnfromaGaussiandistributionwithzero-meanandvariance2.MinimizingF(u,v,)willnotonlyenforcesf(x+u(x))closetob(x+v(x))butalsotunestheweightingparameter1=22automatically.Thesecondtermaimsatenforcinginverseconsistencyforthetransformationsuandv.Weuse~uand~vtoapproximatetheinversedeformationeldsu)]TJ /F6 7.97 Tf 6.59 0 Td[(1andv)]TJ /F6 7.97 Tf 6.59 0 Td[(1withrespecttouandvastheinverseishardtoobtaindirectly.Bythepropertyofinverse,weobtainthefollowingequations x=(x+u(x))+~u(x+u(x))x=(x+~u(x))+u(x+~u(x))x=(x+v(x))+~v(x+v(x))x=(x+~v(x))+v(x+~v(x))(4{21)Thus,inordertoensuretheinverseconsistenceofuandv,weminimizeI(u,~u)andI(v,~v)denedas I(u,~u):=ku(x)+~u(x+u(x))k2+k~u(x)+u(x+~u(x))k2(4{22) 101

PAGE 102

I(v,~v):=kv(x)+~v(x+v(x))k2+k~v(x)+v(x+~v(x))k2.(4{23)Thelasttermintheobjectivefunctionisdenedas R(u,~u,v,~v)=kDuk2+kD~uk2+kDvk2+kD~vk2(4{24)Itensuresthesmoothnessofthetransformationelds. 4.2.2.2ImageRestorationOncetheforwardandbackwardscansarewellalignedthroughtheproposedimageregsitrationmethod,wecombinethemtoobtainmorereliablecbytakingtheaverageofthetwopre-processedpotentials:c=(f+b)=2.However,ifwedirectlyconstructoperatorAinEq.( 4{7 )basedonthiscombinedpotentialc,itisimpossibletoextractadesiredconductivitymapduetothefactthatAinvolvestherstandsecondorderderivativesofc.Anoisyccansignicantlydegradethequalityofthereconstructionresult.Toreducethenoise,asecondstepofdatapreprocessingisperformedtorestoredasmoothedpotentialfromthecombineddatac. Algorithm4.2. TheADMMalgorithmforsolvingproblem( 4{28 ) Choose1=0,w1=0,y1=1.SpatiallyResolvedMappingofElectricalConductivityacrossIndividualDomain(Grain)BoundariesinGraphene. Fort=1,,N)]TJ /F5 11.955 Tf 11.95 0 Td[(1,do Updatevariable: t+1=argmin1 2k)]TJ /F3 11.955 Tf 11.95 0 Td[(ck2+hyt,Di+ 2kw)]TJ /F4 11.955 Tf 11.96 0 Td[(Dk2(4{25) Updatew wt+1=argminwkwk2,1)-222(hyt,wi+ 2kw)]TJ /F4 11.955 Tf 11.96 0 Td[(Dt+1k2(4{26) Updatemultipliery: yt+1=yt)]TJ /F3 11.955 Tf 11.96 0 Td[((wt+1)]TJ /F4 11.955 Tf 11.96 0 Td[(Dt+1)(4{27) 102

PAGE 103

EndFor Output=agN. IthasbeenknownthatTVbasedsmoothingpreservesedgeswhilesmoothesimages.HenceweemployaTVbasedimagerestorationmodel,knownasROFmodel[ 87 ]torestorethenoisyimagesinStep4.Letcandbetheobservednoisyimageandrestoredimagerespectively.TheROFmodelreadsas min1 2Zk)]TJ /F3 11.955 Tf 11.96 0 Td[(ck2+Zjrj,(4{28)where>0isaparameter.Therstterminthemodelservesasthedatadelitytermwhilethesecondtermisthetotalvariationofimage.Inpractice,imagewithredundantdetails(e.g.noisyimage)hashightotalvariation.Thus,byminimizingthetotalvariationofaimagewhilerestrictingitnotdiversetoofarfromitsoriginalform,theundesirabledetails(e.g.noise)arethenremoved.Numerousalgorithmswereintroducedtosolvethisproblem.TokeepaconsistentapproachinthisworkweadopttheADMMalgorithmforthistask.ItissucientforourpurposebecausetherstterminEq.( 4{28 )isstronglyconvexandsmoothsobothacceleratedandclassicADMMhavethesameconvergencerate. Figure4-9. Comparisonoftheobjectivefunctionvalue(kAk2)versustheCPUtime. 103

PAGE 104

4.2.3LinebyLineBackgroundRemovalTheintensityuctuationintheatomicresolutionimagesseriouslydegradestheimagequalityandimpedestheregistrationandimagerestorationmethodthatwewillapplylater.Becausetheimageintensityvariesstronglyfromonescanlinetothenext,butstaysnearlyconstantoverlongsegmentswithineachline,thisuctuationcanbetreatedasabackgroundforeachscanline.Therefore,therststepinprocessingtheatomicresolutionimagesistoremovethisbackgroundinaline-by-linemanner.TheforwardandthebackwardscansarerepresentedbyFa2RmnandBa2Rmn,respectively.Consideroneoftheimages,sayFa,whosei-thlineisrepresentedbyf2R1n.Thebackgroundisconsideredaslowvaryingfunctionalongtheline.Wewishtoremovethebackgroundandkeepthefastvaryingfeatures.Toachievethis,weestimatethevalueofthebackgroundatpointx,gxbyaweightedleastmean-squarelinearttoasegmentofthelinecenteredatxwithlengthSmax, minax,gxx+Smax 2Xx0=x)]TJ /F7 5.978 Tf 7.79 3.52 Td[(Smax 2jf(x0))]TJ /F4 11.955 Tf 11.95 0 Td[(ax(x0)]TJ /F4 11.955 Tf 11.96 0 Td[(x))]TJ /F4 11.955 Tf 11.96 0 Td[(gxj2w(x0)]TJ /F4 11.955 Tf 11.96 0 Td[(x)(4{29)where w(x0)]TJ /F4 11.955 Tf 11.96 0 Td[(x)=S2max S2max+4(x0)]TJ /F4 11.955 Tf 11.96 0 Td[(x)2.(4{30)Theweightw(x0)]TJ /F4 11.955 Tf 12.1 0 Td[(x)istoreducetheimpactofnoiseinthedata.Itisanumericallyecientalternativetorobustleastmean-squarets.Thecorrectedimageisgivenby f(x)=f(x))]TJ /F4 11.955 Tf 11.95 0 Td[(gx,(4{31)foreachx.WefoundnumericallyPxf(x)Pxjf(x)j.Thismeansthatonaveragethecorrectedimageintensityisclosetozero.Thisremovesmostofthebackgroundandbringsalllinestoaboutthesameintensity(closetozero).Anadditionalminorimprovementtothebackgroundremovalprocessistoremovethesmallremnantslopeatthetwoedgesbyttingasmallpartofthelineattwoendstoalinear 104

PAGE 105

backgroundwithslopesofsamemagnitudebutoppositesign(sothatthetwolinesmeetatthesameheightinthemiddle)andremovingthisbackgroundfromthecorrespondinghalvesoftheline.Thisstepmakestheintensitysomewhatmoreuniformacrossthewholeline.ThevalueofSmaxisdeterminedbysearchingforthemaximumcorrelationcoecientafterimageregistration,aprocessthatwewilldescribebelow. 4.2.4ImagePostprocessingOurgoalistocombinedatafromforwardandbackwardscanstoobtainamoreaccurateimage.Becausethereisalwaysasmallregistrationdierencebetweentheforwardandbackwardscans,asimplecombinationoftheunprocesseddata(e.g.,asimpleaverage)willyieldablurredimage(Fig. 4-5 (a)).Indeed,thedierencebetweentheforwardandbackwardscansislargenearthefeaturesintheimage(Fig. 4-6 (a)).Thereforeitisnecessarytorstmatchthetwoscansviaaprocessofimageregistrationbeforecombiningthem.ThemethodforimageregistrationisimprovedsignicantlyfromtheoneusedinRef.[ 113 ].ThemaindierencesarethatwenowuseanewglobalregistrationmethoddierentthantheoneusedinRef.[ 113 ],andthatwenolongeruseadeformationbasedlocalregistrationmethod.Thelattertendstolockontothenoisethusmagnifyingtheeectsofnoise.Althoughtheimagesaretwodimensional,thedataisacquiredinalinebylinemanner.Thereforetheimageregistrationmethodisappliedseparatelytoeachlineoftheforwardandthebackwardscans,FaandBa.Theregistrationprocedureconsistsofalineshift,inwhicheachpairofcorrespondinglinesfromFaandBaareadjustedbyalineshifttominimizetheirdierence,followedbyadeformableinverseconsistantimageregistration,whichisperformedtoobtainlocaltransformationsunderwhichFaandBahavethegreatestcorrelation.Thelineshiftiscalculatedforthei-thlinesofFaandBa,whicharerepresentedbyf2R1nandb2R1nrespectively,asaminimizationproblem.Theobjectiveistondaconstantc,suchthatf(x+c)isasclosetob(x)]TJ /F4 11.955 Tf 12 0 Td[(c)aspossible.InRef.[ 113 ],thecondition 105

PAGE 106

isexpressedasthefollowingmodel, minc2Zkf(x+c))]TJ /F4 11.955 Tf 11.96 0 Td[(b(x)]TJ /F4 11.955 Tf 11.96 0 Td[(c)k2(4{32)Thesmallsearchingspaceforcinpracticeallowsonetosolvethisminimizationthroughaheuristicsearch.However,thismethodissusceptibletonoise,especiallyforatomicresolutiondatainwhichpresenceofnoisesignalmayhavethesamemagnitudeasimagesofatoms.Thereforeweneedamorerobustapproachtoimageregistration.TheimprovedapproachistotreatthepairofdatasetsFaandBaaslinearlycorrelateddata,andndtheconstantcsuchthattheregressioncoecientismaximized, maxc2ZPx[f(x+c))]TJ /F4 11.955 Tf 11.96 0 Td[(favg][b(x)]TJ /F4 11.955 Tf 11.95 0 Td[(c))]TJ /F4 11.955 Tf 11.96 0 Td[(bavg] p kf(x+c))]TJ /F4 11.955 Tf 11.96 0 Td[(favgk2kb(x)]TJ /F4 11.955 Tf 11.95 0 Td[(c))]TJ /F4 11.955 Tf 11.96 0 Td[(bavgk2,(4{33)wherefavg=(1=N)Pxf(x+c)andbavg=(1=N)Pxb(x)]TJ /F4 11.955 Tf 12.53 0 Td[(c)andNisthenumberofpointsineachline.ThebenetofimageregistrationisclearlyvisibleinFig. 4-5 (b),whichshowsthesimpleaverageofthealigneddataFaandBa.TheaverageimageismuchsharpercomparedtoFig. 4-5 (a).Thedierencebetweenthealignedimages,jFa)]TJ /F4 11.955 Tf 12.66 0 Td[(BajshowninFig. 4-6 (a),isalsogreatlyreducedcomparedtothatoftheunprocesseddata,Fig. 4-6 (b).However,evenwiththealigneddata,thespuriousoscillationsstillexist.Infact,inthebestcasescenario,asimpleaveragewouldonlyreducethespuriousoscillationsbyabouthalf.Abetteralgorithmisneededtoeliminatetheseoscillations,aswewillpresentnext.UsingthealignedforwardscanFaandbackwardscanBa,wewishtondanapproximateimagethatisasclosetothetrueimageSaspossible.Ourmaingoalistoeliminateanynoiseandspurioussignalsintheimage,whilekeepingallfeaturesintheimage.Weconsideranyfeaturethatexistsonlyinone(forwardorbackward)scanbutisnotmatchedintheotherscanasspuriousandmustberemoved.Toremovesuchspurioussignals,weproposea\rubberband"model,inwhicha\rubberband"isinsertedbetweenthecurvesofthetwoscanedsignalsandispulledtight,asillustratedinFig. 4-10 .Theredandgreencurvesaresimulated 106

PAGE 107

Figure4-10. Illustrationoftherubberbandmethodusingsimulateddata. Figure4-11. Thealignedforwardandbackwarddatafromthesamelineofscans. 107

PAGE 108

forwardandbackwardscansignals,respectively.Arubberband(black)isinsertedbetweenthetwocurvesandpulledtight,yieldingthenalapproximationofthetrueimage.The\rubberband"curveisthenalimagethatclosestrepresentsthetruesignal.Mathematically,letsdenotesonerowofSwhilefandbrepresentthecorrespondingrowsofFaandBa.Theassociatedmodelforthispairoflinesisgivenby: minsnXp=2js(p))]TJ /F4 11.955 Tf 11.96 0 Td[(s(p)]TJ /F5 11.955 Tf 11.95 0 Td[(1)j2s.t.s(p)maxff(p),b(p)g,s(p)minff(p),b(p)g,wherep=1,,n.(4{34)ThismodelcanbesolvedinanecientmannerbyrepeatedlysmoothingacandidatesignalinsertedbetweenthetwocurvesFaandBaviaaconstrainedsignallteringthatmaintainsthelteredvaluewithintheboundssetbyFaandBauntilconvergence.Theexistenceofthesolutionisguaranteedastheaverageoftheforwardandbackwardscanssatisestheconstraintandweuseitastheinitialcandidatesignal.Forthelteringprocess,thevalueofeachpointonthelineisreplacedbytheaveragevalueofitstwonearestneighbors,subjecttotheconstraintthatitisboundedbyFaandBa.ThealgorithmisoutlinedinALG. 4.3 Algorithm4.3. TheConstrainedAdaptiveandIterativeFilteringalgorithmforapproximatelysolvingproblem( 4{34 ) 1: Input:AlignedforwardandbackwardscanFaandBawithdimensionmn,thetolerancetolandthemaximumiterationnumberN. 2: Forr=1,,m,do 1. Setf=Fa(:,r),b=Ba(:,r)ands0=1 2(f+b). 2. Calculatesmax(p)=maxff(p),b(p)gandsmin(p)=minff(p),b(p)g. 3. Fori=1,,Ndo: 108

PAGE 109

(a) Setsi=si)]TJ /F6 7.97 Tf 6.58 0 Td[(1. (b) Forp=1,,n,do: i. Computey(p)=0.5[si(p)]TJ /F5 11.955 Tf 11.95 0 Td[(1)+si(p+1)]. ii. Setsi(p)=max(min(y(p),smax(p)),smin(p)). (c) Computed=ksi)]TJ /F4 11.955 Tf 11.96 0 Td[(si)]TJ /F6 7.97 Tf 6.58 0 Td[(1k2,terminateloop 3 ifd
PAGE 110

Clearly,wehave R(x,y)=8>>><>>>:0,ifS(x,y)=min(x0,y0)2x,yS(x0,y0),n2)]TJ /F5 11.955 Tf 11.95 0 Td[(1,ifS(x,y)=max(x0,y0)2x,yS(x0,y0),(4{35)Thisensuresthattheentiremapisrangedbetween0andn2)]TJ /F5 11.955 Tf 12.24 0 Td[(1.Thebestvaluefornisthenumberofpixelsthatcoversmorethan1-2atomicdistances.Therankingmapobtainedthiswayisaraggedimage.Weapplya2Dmedianlter[ 56 ]toobtainasmoothedimage.Theideaofusingtherankingmapforfeatureenhancementcanbeviewedasthereverseofrankingbasedsmoothingtechniquessuchasmedianltering.Howevertherankingmapalgorithmpresentedhereisuniqueinthatitdoesnotmerelyusetherankingtohelpdeterminetheimageintensityasinthemedianlteringmethod.Intherankingmapmethod,thenalimageintensityistherankingitself,andtheoriginalimageisdiscardedoncetherankingmapisconstructed. 4.3ValidationwithExperimentalDataTheproposedalgorithmsarevalidatedonasetofSTPdataofgraphenegrainboundaryonSiO2/SisubstrateasshowninFIG. 4-1 .TheimprovementofpreprocessingovertherawdataisdemonstratedinFIGs. 4-7 and 4-8 .InFIG. 4-7 ,wecomparetheaveragepotentialobtainedfromtheoriginalforwardandbackwardscandata(panela)againstthatobtainedfromthepreprocessedforwardandbackwarddata(panelb).Theformerclearlyshowsmorenoiseandhasafuzzyedgeatthegrainboundary.Thepreprocesseddatayieldsasmootherpotentialwithasharpergrainboundary.FIG. 4-8 showsthedierencebetweentheforwardandbackwardscansfortherawdata(panela)andthepreprocesseddata(panelb).Thedierenceissignicantlyreducedbythepreprocessingprocedure.Thedecreaseoftheobjectivefunctionvalueoftheproposedreconstructionalgorithm(Algorithm. 4.1 )iscomparedtothegradientdescentalgorithmonthesamepreprocesseddatainFIG. 4-9 .BothALP-ADMMandgradientdescentmethodsareperformedbasedonthe 110

PAGE 111

Figure4-12. Comparisonofextractedxusingtheproposedmethod. preprocessedgeneratedbytheproposedmethod.Sincethegradientdescentalgorithmdoesnotsolvemodel( 4{8 )directlybutsolvesmodel( 4{4 ),forcomparisonweonlyusethettingtermkAk2.Theproposedalgorithmclearlyout-performsthegradientdescentalgorithmwithabetterasymptoticrateofconvergence.TheextractedconductivitymapsforxandyareshowninFIGs. 4-12 with(a)originaldataforwardscanf;(b)originaldatabackwardscanb;(c)preprocesseddatac)and 4-13 with(a)originaldataforwardscanf;(b)originaldatabackwardscanb;(c)preprocesseddatac,repsectively.Wecompareresultsfromtheproposedreconstructionalgorithmontheoriginalforward(panela),backwarddata(panelb),andthepreprocesseddata(panelc).Inallcasestherawdatawithoutpreprocessingfailedtoyieldanymeaningfulresult.ThisisnotsurprisingconsideringthattheoriginaldataareverynoisyandthematrixAinvolvesuptothesecondderivativesofthedata.Withpreprocessingapplied,bothxandyconvergedtophysicallymeaningfulresults,withhighconductivityontheplateauandlowconductivityatthegrainboundary.InFIG. 4-14 weshowahistogramplotofthedistributionofthevaluesoftheun-normalizedx.Thehighpeakatabout0.7representsthehighconductivitywithinthegraphenesheet.Thereisasecondsmallpeakatabout0.08.Thiscorrespondstothegrainboundaryconductivity.Theratiobetweenthetwovalues,0.12,isingoodagreementwithpreviouswork[ 25 ].Toestablishtheeectivenessofthealgorithm,itistestedonSTMtopographyscansofacleancopper(111)singlecrystalsurfaceatroomtemperature(299K)usingatungstentip 111

PAGE 112

Figure4-13. Comparisonofextractedyusingtheproposedmethod. Figure4-14. Distributionofthevaluesofun-normalizedx. underconventionalscanspeed(about2msecperdatapoint)aswellasfastscanspeedof0.1msecperdatapointwithmultiplescansforthesameregiononthesamplesurface.TheSTMscansareacquiredbyusingavariabletemperatureSTM(Omicron)withNanonis(SPECS)controllerinconstantcurrentmode.CleanCu(111)singlecrystalwaspreparedbycyclesofAr+sputteringandpost-annealing(550C)inultra-highvacuumcondition(810)]TJ /F6 7.97 Tf 6.58 0 Td[(11Torr).Electrochemicallyetchedtungstentipwascleanedbyin-situelectronbombardmentheating.Asalreadydiscussedinprevioussections,theoriginalunprocessedtopographydatafromafastscanwithascanrateof0.1msecperpixelareshowninFIG. 4-2 (a)and(b).Wecanseethatbothoriginalforwardscanandbackwardscansuerfromsignicantnoisebutthepatternsofnoisearedierent,makingitpossiblefornoiseeliminationthroughcombiningthetwoimages.Theaveragesandthedierencesbetweentheorginalforwardandbackwardscans 112

PAGE 113

andthealignedonesafterimageregistrationareshownInFIG. 4-5 with(a)averageoftheunprocesseddata,1 2(Fo+Bo)and(b)averageofthealigneddata,1 2(Fa+Ba)andFIG. 4-6 with(a)dierencebetweentheunprocesseddata(jFo)]TJ /F4 11.955 Tf 12.3 0 Td[(Boj)and(b)dierencebetweenthealigneddata(jFa)]TJ /F4 11.955 Tf 12.46 0 Td[(Baj).TheeectoftheimagerestorationalgorithmontheexperimentalimageisshowninFIG. 4-11 with(a)Row46;(b)Row80;(c)Row100forthreeselectedrowsofthedata.ThenalcorrectedimageiscomparedtoasetofSTMscanwithslowscanrateonthesamesurfaceinFIG. 4-3 .Fortheatomicresolutionfastscandata,therawimageinFig. 4-2 (c)and(c)isoverwhelmedbynoiseandhasnoobservableatomicstructure.Thenalprocessedimage,Fig. 4-4 (b),however,displaysclearlythesurfacelatticestructure.Onecanevenseethesignicantlatticestrainanddisorderduetotheproximityofalargedefect. 4.4ConclusionsofThisChapterWepresentanovelmethodforextractingtwo-dimensional(2D)conductivityprolesfromlargeelectrochemicalpotentialdatasetsacquiredbyScanningTunnelingPotentiometry(STP)ofa2Dconductor.Themethodconsistsofadatapreprocessingproceduretoreduce/eliminatenoiseandanumericalconductivityreconstruction.Thepreprocessingprocedureemploysaninverseconsistentimageregistrationmethodtoaligntheforwardandbackwardscansofthesamelineforeachimagelinefollowedbyatotalvariation(TV)basedimagerestorationmethodtoobtaina(nearly)noise-freepotentialfromthealignedscans.Thepreprocessedpotentialisthenusedfornumericalconductivityreconstruction,basedonaTVmodelsolvedbyacceleratedAlternatingDirectionMethodofMultiplier(A-ADMM).Themethodisdemonstratedonameasurementofthegrainboundaryofamonolayergraphene,yieldinganearly10to1ratioforthegrainboundaryresistivityoverbulkresistivity.WehavealsodemonstratedanalgorithmthatcangreatlyreducethenoiseanderrorofSTMimagesbycombiningforwardandbackwardscandatainalinebylinemanner.ThisallowsustopushthescanrateforaconventionalSTMsetuptobeyonditsnormallimit,upto10timesfaster.Anorderofmagnitudeincreaseinthescanratewillgreatlyreduce 113

PAGE 114

systematicerrorssuchasthescandriftandenvironmentalnoise,andwillalsoimproveresearchproductivity.Furthermore,thisincreaserepresentsarststeptowardsthegoalofrealtimeobservationofdynamicprocessesonsurfacebySTM.Usingtherankingmapastheatomicresolutionimage,asdescribedinthiswork,isageneralalgorithmforpost-processingSTMimagesbeyondfastscans.Anyimagesobstructedbysignicantsurfacetopologyvariationscanbetreatedbythisalgorithmtoyieldsharp,atomicresolutionresults.Thisalgorithmisalsousefulasanalternativewaytoenhanceimagefeaturesandcanbeusedasageneraltoolinimageprocessingtechnology. 114

PAGE 115

REFERENCES [1] Agarwal,Alekh,Negahban,Sahand,andWainwright,MartinJ.\Noisymatrixdecompositionviaconvexrelaxation:Optimalratesinhighdimensions."TheAn-nalsofStatistics(2012):1171{1197. [2] Attouch,HedyandSoueycatt,Mohamed.\AugmentedLagrangianandproximalalternatingdirectionmethodsofmultipliersinHilbertspaces.Applicationstogames,PDE'sandcontrol."PacicJournalofOptimization5(2008).1:17{37. [3] Barzilai,JonathanandBorwein,JonathanM.\Two-pointstepsizegradientmethods."IMAjournalofnumericalanalysis8(1988).1:141{148. [4] Basu,Sumanta,Das,Sreyoshi,Michailidis,George,andPurnanandam,AmiyatoshK.\ASystem-WideApproachtoMeasureConnectivityintheFinancialSector."(2016). [5] Basu,SumantaandMichailidis,George.\Regularizedestimationinsparsehigh-dimensionaltimeseriesmodels."TheAnnalsofStatistics43(2015).4:1535{1567. [6] Beck,AmirandTeboulle,Marc.\Mirrordescentandnonlinearprojectedsubgradientmethodsforconvexoptimization."OperationsResearchLetters31(2003).3:167{175. [7] |||.\Afastiterativeshrinkage-thresholdingalgorithmforlinearinverseproblems."SIAMjournalonimagingsciences2(2009).1:183{202. [8] Becker,S.,Bobin,J.,andCandes,E.J.\NESTA:afastandaccuraterst-ordermethodforsparserecovery."SIAMJournalonImagingSciences4(2011).1:1{39. [9] Billio,Monica,Getmansky,Mila,Lo,AndrewW,andPelizzon,Loriana.\Econometricmeasuresofconnectednessandsystemicriskinthenanceandinsurancesectors."JournalofFinancialEconomics104(2012).3:535{559. [10] Binnig,GerdandRohrer,Heinrich.\Scanningtunnelingmicroscopy."Surfacescience126(1983).1-3:236{244. [11] Block,KaiTobias,Uecker,Martin,andFrahm,Jens.\UndersampledradialMRIwithmultiplecoils.Iterativeimagereconstructionusingatotalvariationconstraint."Magneticresonanceinmedicine57(2007).6:1086{1098. [12] Box,GeorgeEPandTiao,GeorgeC.\Acanonicalanalysisofmultipletimeseries."Biometrika64(1977).2:355{365. [13] Boyd,Stephen,Parikh,Neal,Chu,Eric,Peleato,Borja,andEckstein,Jonathan.\Distributedoptimizationandstatisticallearningviathealternatingdirectionmethodofmultipliers."FoundationsandTrendsRinMachineLearning3(2011).1:1{122. [14] Cai,Jian-Feng,Candes,EmmanuelJ,andShen,Zuowei.\Asingularvaluethresholdingalgorithmformatrixcompletion."SIAMJournalonOptimization20(1956).4:2010. 115

PAGE 116

[15] Candes,EmmanuelJ,Li,Xiaodong,Ma,Yi,andWright,John.\Robustprincipalcomponentanalysis?"JournaloftheACM(JACM)58(2011).3:11. [16] Cauchy,Augustin.\Methodegeneralepourlaresolutiondessystemesdequationssimultanees."Comp.Rend.Sci.Paris25(1847).1847:536{538. [17] Chambolle,A.andPock,T.\Arst-orderprimal-dualalgorithmforconvexproblemswithapplicationstoimaging."JournalofMathematicalImagingandVision40(2011).1:120{145. [18] Chandrasekaran,Venkat,Sanghavi,Sujay,Parrilo,PabloA,andWillsky,AlanS.\Rank-sparsityincoherenceformatrixdecomposition."SIAMJournalonOptimization21(2011).2:572{596. [19] Chen,Y.,Hager,W.,Huang,F.,Phan,D.,Ye,X.,andYin,W.\FastalgorithmsforimagereconstructionwithapplicationtopartiallyparallelMRimaging."SIAMJournalonImagingSciences5(2012).1:90{118. [20] Chen,Yunmei,Hager,WilliamW,Yashtini,Maryam,Ye,Xiaojing,andZhang,Hongchao.\Bregmanoperatorsplittingwithvariablestepsizefortotalvariationimagereconstruction."ComputationalOptimizationandApplications54(2013).2:317{342. [21] Chen,Yunmei,Lan,Guanghui,andOuyang,Yuyuan.\Optimalprimal-dualmethodsforaclassofsaddlepointproblems."SIAMJournalonOptimization(inpress). [22] Chen,Yunmei,Li,Xianqi,Ouyang,Yuyuan,andPasiliao,Eduardo.\ACCELERATEDBREGMANOPERATORSPLITTINGWITHBACKTRACKING."InverseProblems&Imaging11(2017).6. [23] Chen,YunmeiandYe,Xiaojing.\Inverseconsistentdeformableimageregistration."TheLegacyofAlladiRamakrishnanintheMathematicalSciences.Springer,2010.419{440. [24] Clark,KendalW,Zhang,X-G,Gu,Gong,Park,Jewook,He,Guowei,Feenstra,RandallM,andLi,An-Ping.\EnergygapinducedbyFriedeloscillationsmanifestedastransportasymmetryatmonolayer-bilayergrapheneboundaries."PhysicalReviewX4(2014).1:011021. [25] Clark,KendalW,Zhang,X-G,Vlassiouk,IvanV,He,Guowei,Feenstra,RandallM,andLi,An-Ping.\Spatiallyresolvedmappingofelectricalconductivityacrossindividualdomain(grain)boundariesingraphene."ACSnano7(2013).9:7956{7966. [26] Dai,Yu-HongandLiao,Li-Zhi.\R-linearconvergenceoftheBarzilaiandBorweingradientmethod."IMAJournalofNumericalAnalysis22(2002).1:1{10. [27] Daubechies,Ingrid,Defrise,Michel,andDeMol,Christine.\Aniterativethresholdingalgorithmforlinearinverseproblemswithasparsityconstraint."Communicationsonpureandappliedmathematics57(2004).11:1413{1457. 116

PAGE 117

[28] Deng,WeiandYin,Wotao.\Ontheglobalandlinearconvergenceofthegeneralizedalternatingdirectionmethodofmultipliers."JournalofScienticComputing(2012):1{28. [29] Donoho,DavidL.\Compressedsensing."InformationTheory,IEEETransactionson52(2006).4:1289{1306. [30] Dulot,F,Eugene,J,Kierren,B,andMalterre,D.\STM-TIPinducedsurfacediusionofcopperoncopper(100)."Appliedsurfacescience162(2000):86{93. [31] Durand,Corentin,Zhang,X-G,Hus,SabanM,Ma,Chuanxu,McGuire,MichaelA,Xu,Yang,Cao,Helin,Miotkowski,Ireneusz,Chen,YongP,andLi,An-Ping.\Dierentiationofsurfaceandbulkconductivitiesintopologicalinsulatorsviafour-probespectroscopy."Nanoletters16(2016).4:2213{2220. [32] Engl,HeinzWerner,Hanke,Martin,andNeubauer,Andreas.Regularizationofinverseproblems,vol.375.SpringerScience&BusinessMedia,1996. [33] Esch,Friedrich,Dri,Carlo,Spessot,Alessio,Africh,Cristina,Cautero,Giuseppe,Giuressi,Dario,Sergo,Rudi,Tommasini,Riccardo,andComelli,Giovanni.\TheFASTmodule:Anadd-onunitfordrivingcommercialscanningprobemicroscopesatvideorateandbeyond."ReviewofScienticInstruments82(2011).5:053702. [34] Fazel,Maryam,Hindi,Haitham,andBoyd,StephenP.\Log-detheuristicformatrixrankminimizationwithapplicationstoHankelandEuclideandistancematrices."AmericanControlConference,2003.Proceedingsofthe2003.vol.3.IEEE,2003,2156{2162. [35] Figueiredo,MarioATandNowak,RobertD.\AnEMalgorithmforwavelet-basedimagerestoration."IEEETransactionsonImageProcessing12(2003).8:906{916. [36] Frank,MargueriteandWolfe,Philip.\Analgorithmforquadraticprogramming."NavalResearchLogistics(NRL)3(1956).1-2:95{110. [37] Gabay,DanielandMercier,Bertrand.\Adualalgorithmforthesolutionofnonlinearvariationalproblemsvianiteelementapproximation."Computers&MathematicswithApplications2(1976).1:17{40. [38] Gawande,ManojB,Goswami,Anandarup,Felpin,Francois-Xavier,Asefa,Tewodros,Huang,Xiaoxi,Silva,Rafael,Zou,Xiaoxin,Zboril,Radek,andVarma,RajenderS.\CuandCu-basednanoparticles:synthesisandapplicationsincatalysis."Chemicalreviews116(2016).6:3722{3811. [39] Ghorbani-Asl,Mahdi,Enyashin,AndreyN,Kuc,Agnieszka,Seifert,Gotthard,andHeine,Thomas.\Defect-inducedconductivityanisotropyinMoS2monolayers."PhysicalReviewB88(2013).24:245440. [40] Glowinski,RolandandMarroco,A.\Surl'approximation,parelementsnisd'ordreun,etlaresolution,parpenalisation-dualited'uneclassedeproblemesdeDirichlet 117

PAGE 118

nonlineaires."ESAIM:MathematicalModellingandNumericalAnalysis-ModelisationMathematiqueetAnalyseNumerique9(1975).R2:41{76. [41] Goldfarb,Donald,Ma,Shiqian,andScheinberg,Katya.\Fastalternatinglinearizationmethodsforminimizingthesumoftwoconvexfunctions."MathematicalProgramming141(2013).1-2:349{382.URL http://dx.doi.org/10.1007/s10107-012-0530-2 [42] |||.\Fastalternatinglinearizationmethodsforminimizingthesumoftwoconvexfunctions."MathematicalProgramming141(2013).1-2:349{382. [43] Goldstein,T.andOsher,S.\TheSplitBregmanMethodforL1-RegularizedProblems."SIAMJournalonImagingSciences2(2009).2:323{343. [44] Goldstein,Tom,Esser,Ernie,andBaraniuk,Richard.\Adaptiveprimal-dualhybridgradientmethodsforsaddle-pointproblems."arXivpreprintarXiv:1305.0546(2013). [45] Goldstein,Tom,O'Donoghue,Brendan,Setzer,Simon,andBaraniuk,Richard.\Fastalternatingdirectionoptimizationmethods."SIAMJournalonImagingSciences7(2014).3:1588{1623. [46] Grippo,Luigi,Lampariello,Francesco,andLucidi,Stephano.\AnonmonotonelinesearchtechniqueforNewtonsmethod."SIAMJournalonNumericalAnalysis23(1986).4:707{716. [47] Hager,William,Ngo,Cuong,Yashtini,Maryam,andZhang,Hong-Chao.\AnalternatingdirectionapproximateNewtonalgorithmforill-conditionedinverseproblemswithapplicationtoparallelMRI."JournaloftheOperationsResearchSocietyofChina3(2015).2:139{162. [48] Hager,WILLIAMW,Yashtini,MARYAM,andZhang,HONGCHAO.\AnO(1/K)convergenceratefortheBOSVSalgorithmintotalvariationregularizedleastsquaresproblems."Tech.rep.,TechnicalReport,GeorgiaInstituteofTechnology,Schoolofmathematics,2015. [49] Hansen,PerChristian,Nagy,JamesG,andO'leary,DianneP.Deblurringimages:matrices,spectra,andltering,vol.3.Siam,2006. [50] He,BingshengandYuan,Xiaoming.\Convergenceanalysisofprimal-dualalgorithmsforasaddle-pointproblem:Fromcontractionperspective."SIAMJournalonImagingSciences5(2012).1:119{149. [51] Heller,EJ,Crommie,MF,Lutz,CP,andEigler,DM.\Scatteringandabsorptionofsurfaceelectronwavesinquantumcorrals."Nature369(1994).6480:464. [52] Hla,Saw-Wai,Meyer,Gerhard,andRieder,Karl-Heinz.\Inducingsingle-moleculechemicalreactionswithaUHV-STM:Anewdimensionfornano-scienceandtechnology."ChemPhysChem2(2001).6:361{366. 118

PAGE 119

[53] Homoth,J,Wenderoth,M,Druga,T,Winking,L,Ulbrich,RG,Bobisch,CA,Weyers,B,Bannani,A,Zubkov,E,Bernhart,AM,etal.\Electronictransportonthenanoscale:ballistictransmissionandOhmslaw."Nanoletters9(2009).4:1588{1592. [54] Hong,MingyiandLuo,Zhi-Quan.\Onthelinearconvergenceofthealternatingdirectionmethodofmultipliers."arXivpreprintarXiv:1208.3922(2012). [55] |||.\Onthelinearconvergenceofthealternatingdirectionmethodofmultipliers."arXivpreprintarXiv:1208.3922(2012). [56] Huang,T,Yang,GJTGY,andTang,G.\Afasttwo-dimensionalmedianlteringalgorithm."IEEETransactionsonAcoustics,Speech,andSignalProcessing27(1979).1:13{18. [57] Ji,Shuai-Hua,Hannon,JB,Tromp,RM,Perebeinos,V,Terso,J,andRoss,FM.\Atomic-scaletransportinepitaxialgraphene."Naturematerials11(2012).2:114. [58] Ji,ShuiwangandYe,Jieping.\Anacceleratedgradientmethodfortracenormminimization."Proceedingsofthe26thannualinternationalconferenceonmachinelearning.ACM,2009,457{464. [59] Kim,Tae-Hwan,Zhang,X-G,Nicholson,DonM,Evans,BoydM,Kulkarni,NagrajS,Radhakrishnan,B,Kenik,EdwardA,andLi,An-Ping.\Largediscreteresistancejumpatgrainboundaryincoppernanowire."Nanoletters10(2010).8:3096{3100. [60] Li,An-Ping,Clark,KendalW,Zhang,X-G,andBaddorf,ArthurP.\ElectronTransportattheNanometer-ScaleSpatiallyRevealedbyFour-ProbeScanningTunnelingMicroscopy."AdvancedFunctionalMaterials23(2013).20:2509{2524. [61] Li,Chengbo,Yin,Wotao,Jiang,Hong,andZhang,Yin.\AnecientaugmentedLagrangianmethodwithapplicationstototalvariationminimization."ComputationalOptimizationandApplications56(2013).3:507{530. [62] Li,Qing,Owens,JonathanR,Han,Chengbo,Sumpter,BobbyG,Lu,Wenchang,Bernholc,Jerzy,Meunier,V,Maksymovych,Peter,Fuentes-Cabrera,Miguel,andPan,Minghu.\Self-organizedandCu-coordinatedsurfacelinearpolymerization."Scienticreports3(2013):2102. [63] Li,QuanfengandLu,Qingyou.\Atomicresolutionultrafastscanningtunnelingmicroscopewithscanratebreakingtheresonantfrequencyofaquartztuningforkresonator."ReviewofScienticInstruments82(2011).5:053705. [64] Lin,JiaheandMichailidis,George.\RegularizedEstimationandTestingforHigh-DimensionalMulti-BlockVector-AutoregressiveModels."arXivpreprintarXiv:1708.05879(2017). [65] Lin,Zhouchen.\SomesoftwarepackagesforpartialSVDcomputation."arXivpreprintarXiv:1108.1548(2011). 119

PAGE 120

[66] Liu,Qiegen,Luo,Jianhua,Wang,Shanshan,Xiao,Moyan,andYe,Meng.\AnaugmentedLagrangianmulti-scaledictionarylearningalgorithm."EURASIPJournalonAdvancesinSignalProcessing2011(2011).1:1{16. [67] Lu,Lei,Shen,Yongfeng,Chen,Xianhua,Qian,Lihua,andLu,Ke.\Ultrahighstrengthandhighelectricalconductivityincopper."Science304(2004).5669:422{426. [68] Lustig,Michael,Donoho,David,andPauly,JohnM.\SparseMRI:TheapplicationofcompressedsensingforrapidMRimaging."Magneticresonanceinmedicine58(2007).6:1182{1195. [69] Lutkepohl,H.Newintroductiontomultipletimeseriesanalysis.Springer,2005. [70] Melnyk,IgorandBanerjee,Arindam.\Estimatingstructuredvectorautoregressivemodels."InternationalConferenceonMachineLearning.2016,830{839. [71] Michailidis,GeorgeanddAlcheBuc,Florence.\Autoregressivemodelsforgeneregulatorynetworkinference:Sparsity,stabilityandcausalityissues."Mathematicalbiosciences246(2013).2:326{334. [72] Monteiro,RenatoDCandSvaiter,BenarF.\Iteration-complexityofblock-decompositionalgorithmsandthealternatingdirectionmethodofmultipliers."SIAMJournalonOptimization23(2013).1:475{507. [73] Muralt,PandPohl,DW.\Scanningtunnelingpotentiometry."AppliedPhysicsLetters48(1986).8:514{516. [74] Negahban,Sahand,Yu,Bin,Wainwright,MartinJ,andRavikumar,PradeepK.\Auniedframeworkforhigh-dimensionalanalysisofM-estimatorswithdecomposableregularizers."AdvancesinNeuralInformationProcessingSystems.2009,1348{1356. [75] Nemirovski,A.andYudin,D.Problemcomplexityandmethodeciencyinoptimization.Wiley-InterscienceSeriesinDiscreteMathematics.JohnWiley,XV,1983. [76] Nesterov,Y.E.\AmethodforunconstrainedconvexminimizationproblemwiththerateofconvergenceO(1=k2)."DokladyANSSSR269(1983):543{547.TranslatedasSovietMath.Docl. [77] |||.IntroductoryLecturesonConvexOptimization:ABasicCourse.Massachusetts:KluwerAcademicPublishers,2004. [78] Nesterov,Yu.\Gradientmethodsforminimizingcompositefunctions."MathematicalProgramming(2012):1{37. [79] Nesterov,Yurii.\Smoothminimizationofnon-smoothfunctions."Mathematicalprogramming103(2005).1:127{152. [80] |||.Introductorylecturesonconvexoptimization:Abasiccourse,vol.87.SpringerScience&BusinessMedia,2013. 120

PAGE 121

[81] Ouyang,Yuyuan,Chen,Yunmei,Lan,Guanghui,andPasiliaoJr,Eduardo.\Anacceleratedlinearizedalternatingdirectionmethodofmultipliers."SIAMJournalonImagingSciences8(2015).1:644{681. [82] Pruessmann,KlaasP,Weiger,Markus,Bornert,Peter,andBoesiger,Peter.\Advancesinsensitivityencodingwitharbitraryk-spacetrajectories."MagneticResonanceinMedicine46(2001).4:638{651. [83] Raydan,Marcos.\OntheBarzilaiandBorweinchoiceofsteplengthforthegradientmethod."IMAJournalofNumericalAnalysis13(1993).3:321{326. [84] |||.\TheBarzilaiandBorweingradientmethodforthelargescaleunconstrainedminimizationproblem."SIAMJournalonOptimization7(1997).1:26{33. [85] Recht,Benjamin,Fazel,Maryam,andParrilo,PabloA.\Guaranteedminimum-ranksolutionsoflinearmatrixequationsvianuclearnormminimization."SIAMreview52(2010).3:471{501. [86] Rost,MJ,Crama,L,Schakel,P,VanTol,E,vanVelzen-Williams,GBEM,Overgauw,CF,TerHorst,H,Dekker,H,Okhuijsen,B,Seynen,M,etal.\Scanningprobemicroscopesgovideorateandbeyond."ReviewofScienticInstruments76(2005).5:053710. [87] Rudin,LeonidI,Osher,Stanley,andFatemi,Emad.\Nonlineartotalvariationbasednoiseremovalalgorithms."PhysicaD:nonlinearphenomena60(1992).1-4:259{268. [88] Scheinberg,Katya,Goldfarb,Donald,andBai,Xi.\Fastrst-ordermethodsforcompositeconvexoptimizationwithbacktracking."FoundationsofComputationalMathematics14(2014).3:389{417. [89] Sharaev,MaksimG,Zavyalova,ViktoriaV,Ushakov,VadimL,Kartashov,SergeyI,andVelichkovsky,BorisM.\Eectiveconnectivitywithinthedefaultmodenetwork:dynamiccausalmodelingofresting-statefMRIdata."Frontiersinhumanneuroscience10(2016). [90] Shepp,LawrenceAandLogan,BenjaminF.\TheFourierreconstructionofaheadsection."NuclearScience,IEEETransactionson21(1974).3:21{43. [91] Shor,NaumZuselevich.Minimizationmethodsfornon-dierentiablefunctions,vol.3.SpringerScience&BusinessMedia,2012. [92] Stroscio,JosephAandCelotta,RobertJ.\Controllingthedynamicsofasingleatominlateralatommanipulation."Science306(2004).5694:242{247. [93] Tan,KeanMing,London,Palma,Mohan,Karthik,Lee,Su-In,Fazel,Maryam,andWitten,DanielaM.\Learninggraphicalmodelswithhubs."JournalofMachineLearningResearch15(2014).1:3297{3331. 121

PAGE 122

[94] Tapaszto,Levente,Nemes-Incze,Peter,Dobrik,Gergely,JaeYoo,Kwon,Hwang,Chanyong,andBiro,LaszloP.\Mappingtheelectronicpropertiesofindividualgraphenegrainboundaries."AppliedPhysicsLetters100(2012).5:053114. [95] Terso,JandHamann,DR.\Theoryandapplicationforthescanningtunnelingmicroscope."Physicalreviewletters50(1983).25:1998. [96] Torija,MA,Li,An-Ping,Guan,XC,Plummer,EWard,andShen,Jian.\LiveSurfaceFerromagnetisminFeNanodots/CuMultilayersonCu(111)."Physicalreviewletters95(2005).25:257203. [97] Tsen,AdamW,Brown,Lola,Levendorf,MarkP,Ghahari,Fereshte,Huang,PinshaneY,Havener,RobinW,Ruiz-Vargas,CarlosS,Muller,DavidA,Kim,Philip,andPark,Jiwoong.\Tailoringelectricaltransportacrossgrainboundariesinpolycrystallinegraphene."Science336(2012).6085:1143{1146. [98] Tseng,P.\Onacceleratedproximalgradientmethodsforconvex-concaveoptimization."submittedtoSIAMJournalonOptimization(2008). [99] VanDenHeuvel,MartijnPandPol,HillekeEHulsho.\Exploringthebrainnetwork:areviewonresting-statefMRIfunctionalconnectivity."Europeanneuropsychopharmacol-ogy20(2010).8:519{534. [100] VanDerZande,ArendM,Huang,PinshaneY,Chenet,DanielA,Berkelbach,TimothyC,You,YuMeng,Lee,Gwan-Hyoung,Heinz,TonyF,Reichman,DavidR,Muller,DavidA,andHone,JamesC.\Grainsandgrainboundariesinhighlycrystallinemonolayermolybdenumdisulphide."Naturematerials12(2013).6:554. [101] Velu,RajaP,Reinsel,GregoryC,andWichern,DeanW.\Reducedrankmodelsformultipletimeseries."Biometrika73(1986).1:105{118. [102] Wang,Weigang,Munakata,Ko,Rozler,Michael,Beasley,MalcolmR,etal.\Localtransportmeasurementsatmesoscopiclengthscalesusingscanningtunnelingpotentiometry."Physicalreviewletters110(2013).23:236802. [103] Wang,Yilun,Yang,Junfeng,Yin,Wotao,andZhang,Yin.\Anewalternatingminimizationalgorithmfortotalvariationimagereconstruction."SIAMJournalonImagingSciences1(2008).3:248{272. [104] Wohlberg,Brendt.\Ecientalgorithmsforconvolutionalsparserepresentations."IEEETransactionsonImageProcessing25(2016).1:301{315. [105] Xu,P,Neek-Amal,M,Barber,SD,Schoelz,JK,Ackerman,ML,Thibado,PM,Sadeghi,A,andPeeters,FM.\Unusualultra-low-frequencyuctuationsinfreestandinggraphene."Naturecommunications5(2014):3720. [106] Xu,Yangyang.\Acceleratedrst-orderprimal-dualproximalmethodsforlinearlyconstrainedcompositeconvexprogramming."SIAMJournalonOptimization27(2017).3:1459{1484. 122

PAGE 123

[107] Yang,EunhoandLozano,AurelieC.\Sparse+Group-SparseDirtyModels:StatisticalGuaranteeswithoutUnreasonableConditionsandaCaseforNon-Convexity."Interna-tionalConferenceonMachineLearning.2017,3911{3920. [108] Yang,Junfeng,Zhang,Yin,andYin,Wotao.\AfastalternatingdirectionmethodforTVL1-L2signalreconstructionfrompartialFourierdata."IEEEJournalofSelectedTopicsinSignalProcessing4(2010).2:288{297. [109] Ye,X.,Chen,Y.,andHuang,F.\ComputationalaccelerationforMRimagereconstructioninpartiallyparallelimaging."MedicalImaging,IEEETransactionson30(2011).5:1055{1063. [110] Yoshinaga,H.\MeasurementsoftheanisotropyofthedislocationresistivityinAu,Ag,andCu."physicastatussolidi(b)18(1966).2:625{636. [111] Yu,Qingkai,Jauregui,LuisA,Wu,Wei,Colby,Robert,Tian,Jifa,Su,Zhihua,Cao,Helin,Liu,Zhihong,Pandey,Deepak,Wei,Dongguang,etal.\Controlandcharacterizationofindividualgrainsandgrainboundariesingraphenegrownbychemicalvapourdeposition."Naturematerials10(2011).6:443. [112] Yuan,MingandLin,Yi.\Modelselectionandestimationinregressionwithgroupedvariables."JournaloftheRoyalStatisticalSociety:SeriesB(StatisticalMethodology)68(2006).1:49{67. [113] Zhang,Hao,Li,Xianqi,Chen,Yunmei,Durand,Corentin,Li,An-Ping,andZhang,X-G.\Conductivitymapfromscanningtunnelingpotentiometry."ReviewofScienticInstruments87(2016).8:083702. [114] Zhang,HongchaoandHager,WilliamW.\Anonmonotonelinesearchtechniqueanditsapplicationtounconstrainedoptimization."SIAMjournalonOptimization14(2004).4:1043{1056. [115] Zhang,Xiaoqun,Burger,Martin,Bresson,Xavier,andOsher,Stanley.\Bregmanizednonlocalregularizationfordeconvolutionandsparsereconstruction."SIAMJournalonImagingSciences3(2010).3:253{276. [116] Zhang,Xiaoqun,Burger,Martin,andOsher,Stanley.\Auniedprimal-dualalgorithmframeworkbasedonBregmaniteration."JournalofScienticComputing46(2011).1:20{46. [117] Zhang,Xin,Song,Xiaohui,Zhang,Xiao-Guang,andZhang,Dianlin.\GrainboundaryresistivitiesofpolycrystallineAulms."EPL(EurophysicsLetters)96(2011).1:17010. [118] Zhu,M.andChan,T.\Anecientprimal-dualhybridgradientalgorithmfortotalvariationimagerestoration."UCLACAMReport(2008):08{34. [119] Zhu,Mingqiang,Wright,StephenJ,andChan,TonyF.\Duality-basedalgorithmsfortotal-variation-regularizedimagerestoration."ComputationalOptimizationandApplications47(2010).3:377{400. 123

PAGE 124

BIOGRAPHICALSKETCHXianqiLiwasborninLiaocheng,ShandongProvince,China.HereceivedhisBachelorofSciencedegreeininformationandcomputationalmathematicsfromLiaoningUniversity,China,inJuly2007.ThenheenrolledatTheUniversityofTexas-PanAmerican(nownamedasTheUniversityofRioGrandeValley)andreceivedhisMasterofSciencedegreeinmathematicsinAugust2009.Afterthat,hewasadmittedintothePh.D.programinDepartmentofMathematicsandStatisticsatUniversityofSouthFloridaandstudiedthereforoneyearandahalf,thenenrolledatUniversityofFloridaandreceivedhisMasterofSciencedegreeinelectricalandcomputerengineeringandDoctorofPhilosophyinmathematicsinMay2015andMay2018,respectively. 124