Multiple Surrogates for Prediction and Optimization

MISSING IMAGE

Material Information

Title:
Multiple Surrogates for Prediction and Optimization
Physical Description:
1 online resource (155 p.)
Language:
english
Creator:
Viana,Felipe Antonio C
Publisher:
University of Florida
Place of Publication:
Gainesville, Fla.
Publication Date:

Thesis/Dissertation Information

Degree:
Doctorate ( Ph.D.)
Degree Grantor:
University of Florida
Degree Disciplines:
Aerospace Engineering, Mechanical and Aerospace Engineering
Committee Chair:
Haftka, Raphael T
Committee Members:
Fregly, Benjamin J
Kim, Nam Ho
Pardalos, Panagote M

Subjects

Subjects / Keywords:
design -- kriging -- optimization -- surrogates
Mechanical and Aerospace Engineering -- Dissertations, Academic -- UF
Genre:
Aerospace Engineering thesis, Ph.D.
bibliography   ( marcgt )
theses   ( marcgt )
government publication (state, provincial, terriorial, dependent)   ( marcgt )
born-digital   ( sobekcm )
Electronic Thesis or Dissertation

Notes

Abstract:
Statistical modeling of computer experiments embraces the set of methodologies for generating a surrogate model (also known as metamodel or response surface approximation) used to replace an expensive simulation code. The aim of surrogate modeling is to construct an approximation of a response of interest based on a limited number of expensive simulations. Nevertheless, after years of intensive research on the field, surrogate-based analysis and optimization is still a struggle to achieve maximum accuracy for a given number of simulations. In this dissertation, we have taken advantage of multiple surrogates to address the issues that we face when we (i) want to build an accurate surrogate model under limited computational budget, (ii) use the surrogate for constrained optimization and the exact analysis shows that the solution is infeasible, and (iii) use the surrogate for global optimization and do not know where to place a set of points in which we are most likely to have improvement. In terms of prediction accuracy, we have found that multiple surrogates work as insurance against poorly fitted models. Additionally, we propose the use of safety margins to conservatively compensate for fitting errors associated with surrogates. We were able to estimate the safety margin for a specific conservativeness level, and we found that it is possible to select a surrogate with the best compromise between conservativeness and loss of accuracy. In terms of optimization, we proposed two strategies for enabling surrogate-based global optimization with parallel function evaluations. The first one is based on the simultaneous use of multiple surrogates (a set of surrogates collaboratively provide multiple points). The second strategy uses a single surrogate and one cheap to evaluate criterion (probability of improvement) for multiple point selection approximation. In both cases, we found that we could successfully speed up the optimization convergence without clear penalties as far as number of function evaluations.
General Note:
In the series University of Florida Digital Collections.
General Note:
Includes vita.
Bibliography:
Includes bibliographical references.
Source of Description:
Description based on online resource; title from PDF title page.
Source of Description:
This bibliographic record is available under the Creative Commons CC0 public domain dedication. The University of Florida Libraries, as creator of this bibliographic record, has waived all rights to it worldwide under copyright law, including all related and neighboring rights, to the extent allowed by law.
Statement of Responsibility:
by Felipe Antonio C Viana.
Thesis:
Thesis (Ph.D.)--University of Florida, 2011.
Local:
Adviser: Haftka, Raphael T.

Record Information

Source Institution:
UFRGP
Rights Management:
Applicable rights reserved.
Classification:
lcc - LD1780 2011
System ID:
UFE0043101:00001


This item is only available as the following downloads:


Full Text

PAGE 1

MULTIPLESURROGATESFORPREDICTIONANDOPTIMIZATIONByFELIPEA.C.VIANAADISSERTATIONPRESENTEDTOTHEGRADUATESCHOOLOFTHEUNIVERSITYOFFLORIDAINPARTIALFULFILLMENTOFTHEREQUIREMENTSFORTHEDEGREEOFDOCTOROFPHILOSOPHYINAEROSPACEENGINEERINGUNIVERSITYOFFLORIDA2011

PAGE 2

c2011FelipeA.C.Viana 2

PAGE 3

TomywifeNadiaandmydaughterBruna,whoseunconditionalloveandencouragementhavegivenmewingstoyhigherthanIeverimagined. 3

PAGE 4

ACKNOWLEDGMENTS Firstofall,IwouldliketothankwhomademewhoIam.Iwouldliketoexpressmygratitudetomyparentsandsiblings.Thegoodeducationtheygaveme,theirlove,support,andincentivewereandwillalwaysbefundamentalinmylife.IamalsoimmenselythankfultoNadiaandBrunafordreamingmydreamswithmeandforthelovetheyhavedevotedtomealltheseyears.IamthankfultomyacademicadvisorDr.RaphaelHaftkaforhisguidance,patience,andencouragement.Iwouldalsoliketothankthemembersofmyadvisorycommittee,Dr.Nam-HoKim,Dr.BenjaminFregly,andDr.PanosPardalos.Iamgratefulfortheirwillingnesstoserveonmycommittee,andforconstructivecriticismtheyhaveprovided.Iamalsogratefultotheonesthathavecontributedscienticallytothisdissertation:Dr.BenjaminFregly,Dr.ChristianGogu,Dr.GerhardVenter,Dr.LayneWatson,Mr.MatthewPais,Dr.NestorQueipo,Dr.SankethBhat,Dr.TimothySimpson,Dr.TusharGoel,Dr.VassiliToropov,Dr.VictorPicheny,Dr.VijayJagdale,Dr.VladimirBalabanov,andDr.WeiShyy.Theyprovidedmewiththoughtfulideasandadvices,andencyclopedicknowledgeofmathematics,surrogatemodeling,anddesignoptimization.Iwishtothanktoallmycolleaguesfortheirfriendshipandsupport.ThanksAlex,Anirban,Anurag,Ben,Bharani,Bryan,Christian,Diane,Jinuk,Jungeun,Kyle,Matt,Palani,Park,Pat,Saad,Sanketh,Sriram,Sunil,Taiki,Tushar,Victor,Xiangchun,andYoung-Chang.Financialsupports,providedbytheNationalScienceFoundation(grantsCMMI-0423280andCMMI-0856431),theNASAConstellationUniversityInstituteProgram(CUIP),andtheAirForceOfceofScienticResearch(grantFA9550-09-1-0153)aregratefullyacknowledged. 4

PAGE 5

TABLEOFCONTENTS page ACKNOWLEDGMENTS .................................. 4 LISTOFTABLES ...................................... 8 LISTOFFIGURES ..................................... 10 ABSTRACT ......................................... 13 CHAPTER 1INTRODUCTION ................................... 15 1.1Motivation .................................... 15 1.2OutlineofDissertation ............................. 19 1.2.1Objectives ................................ 19 1.2.2PublicationsandSoftware ....................... 19 1.2.3OrganizationoftheText ........................ 20 2BACKGROUNDANDLITERATUREREVIEW ................... 21 2.1MultipleSurrogates ............................... 21 2.1.1HowtoGenerateDifferentSurrogates ................ 21 2.1.2MultipleSurrogatesinAction ..................... 22 2.2SequentialSamplingandOptimization .................... 25 2.2.1SequentialSampling .......................... 25 2.2.2Optimization-DrivenSequentialSampling .............. 27 2.3BeingSafeunderLimitedNumberofSimulations .............. 29 2.3.1ConservativeSurrogates ........................ 29 2.3.2AccurateApproximationofConstraintsNeartheBoundarybetweenFeasibleandUnfeasibleDomains ................... 31 2.4Summary .................................... 32 3MULTIPLESURROGATESFORPREDICTION .................. 33 3.1Background ................................... 33 3.1.1RootMeanSquareErroreRMS ..................... 33 3.1.2CrossValidationandPRESSRMS .................... 34 3.2EnsembleofSurrogates ............................ 35 3.2.1SelectionBasedonPRESSRMS .................... 35 3.2.2WeightedAverageSurrogate ..................... 36 3.2.3HeuristicComputationoftheWeights ................. 37 3.2.4ComputationoftheWeightsforMinimumeRMS ............ 37 3.2.5ShouldWeUseAllSurrogates? .................... 39 3.2.6CombiningAccuracyandDiversity .................. 39 3.3NumericalExperiments ............................ 40 5

PAGE 6

3.3.1BasicandDerivedSurrogates ..................... 40 3.3.2PerformanceMeasures ........................ 40 3.3.3TestProblems .............................. 42 3.4ResultsandDiscussion ............................ 45 3.5Summary .................................... 52 4USINGCROSSVALIDATIONTODESIGNCONSERVATIVESURROGATES 58 4.1Background ................................... 58 4.1.1ConservativeSurrogates ........................ 58 4.1.2ConservativenessLevelandRelativeErrorGrowth ......... 59 4.2DesignofSafetyMarginUsingCrossValidationErrors ........... 61 4.3NumericalExperiments ............................ 64 4.3.1BasicSurrogates ............................ 64 4.3.2TestProblems .............................. 64 4.4ResultsandDiscussion ............................ 66 4.5Summary .................................... 73 5EFFICIENTGLOBALOPTIMIZATIONALGORITHMASSISTEDBYMULTIPLESURROGATES .................................... 74 5.1Background:EfcientGlobalOptimizationAlgorithm ............ 74 5.2ImportingErrorEstimatesfromAnotherSurrogate ............. 78 5.3EfcientGlobalOptimizationAlgorithmwithMultipleSurrogates ...... 80 5.4NumericalExperiments ............................ 84 5.4.1SetofSurrogates ............................ 84 5.4.2AnalyticExamples ........................... 84 5.4.3EngineeringExample:TorqueArmOptimization ........... 87 5.4.4OptimizingtheExpectedImprovement ................ 89 5.5ResultsandDiscussion ............................ 91 5.5.1AnalyticExamples ........................... 91 5.5.2EngineeringExample:TorqueArmOptimization ........... 98 5.6Summary .................................... 103 6SURROGATE-BASEDOPTIMIZATIONWITHPARALLELSIMULATIONSUSINGTHEPROBABILITYOFIMPROVEMENT ..................... 106 6.1Background:SinglePointProbabilityofImprovement ............ 106 6.2OptimizingtheApproximatedMultipointProbabilityofImprovement .... 107 6.3NumericalExperiments ............................ 110 6.3.1KrigingModel .............................. 110 6.3.2AnalyticExamples ........................... 110 6.3.3OptimizingtheProbabilityofImprovement .............. 112 6.4ResultsandDiscussion ............................ 114 6.5Summary .................................... 123 6

PAGE 7

7CONCLUSIONSANDFUTUREWORK ...................... 125 7.1SummaryandLearnings ............................ 125 7.2Perspectives .................................. 127 APPENDIX AOVERVIEWOFTHESURROGATETECHNIQUESUSEDINTHISWORK .. 129 A.1ResponseSurface ............................... 129 A.2Kriging ...................................... 130 A.3RadialBasisNeuralNetworks ......................... 132 A.4LinearShepard ................................. 133 A.5SupportVectorRegression .......................... 135 BBOXPLOTS ..................................... 138 CCONSERVATIVEPREDICTORSANDMULTIPLESURROGATES ....... 139 DDERIVATIONOFTHEEXPECTEDIMPROVEMENT ............... 143 REFERENCES ....................................... 144 BIOGRAPHICALSKETCH ................................ 155 7

PAGE 8

LISTOFTABLES Table page 1-1Motivationforreviewpapersondesignandanalysisofcomputerexperiments. 16 1-2Numberofpublicationsinengineeringperyearusingfoursurrogatetechniques. 18 2-1Methodsforcreatingconservativesurrogates. ................... 30 3-1Informationaboutthesetof24basicsurrogates. ................. 41 3-2Selectionofsurrogatesusingdifferentcriteria. .................. 41 3-3ParametersusedintheHartmanfunction. ..................... 43 3-4SpecicationsforthedesignofexperimentsandtestpointsforthebenchmarkfunctionsofChapter 3 ................................ 44 3-5Frequency,innumberofDOEs(outof100),ofbesteRMSandPRESSRMSforeachbasicsurrogate. ................................ 49 3-6%dierenceintheeRMSofthebest3basicsurrogatesandBestPRESSforeachtestproblem. ..................................... 54 3-7%dierenceintheeRMSoftheWASschemesandBestPRESSforeachtestproblemwhenusingall24surrogates. ....................... 56 4-1Informationaboutthetwogeneratedsurrogatesusedinthestudyofconservativesurrogates. ...................................... 64 4-2ExperimentaldesignspecicationsforbenchmarkfunctionsofChapter 4 ... 66 5-1SetofsurrogatesusedinthestudyofEGOassistedbymultiplesurrogates. .. 85 5-2ParametersusedintheHartmanfunction. ..................... 86 5-3EGOsetupfortheanalytictestproblems. ..................... 87 5-4Rangeofthedesignvariables(meters). ...................... 88 5-5Optimizationsetupfortheengineeringexample. ................. 90 5-6RankingofthesurrogatesaccordingtomedianvaluesofPRESSRMSandeRMS. 94 5-7PRESSRMSanalysisforthetorquearmexample. ................. 101 5-8Identityofsurrogatethatsuggesteddesignswithreductioninmass. ...... 102 6-1Krigingmodelusedintheoptimizationwithprobabilityofimprovementstudy. 111 6-2ParametersusedintheHartmanfunction. ..................... 111 8

PAGE 9

6-3EGOsetupfortheanalytictestproblems. ..................... 113 6-4Probabilityofimprovementofpointspointsselectedbykriging. ......... 123 A-1Exampleofkernelfunctions. ............................ 135 C-1Surrogatesusedinthestudyofconservativepredictorsandmultiplesurrogates. 140 9

PAGE 10

LISTOFFIGURES Figure page 1-1Evolutioninengineeringdesignoptimization. ................... 17 1-2Numberofpublicationsinengineeringperyearusingfoursurrogatetechniques. 18 1-3Minimizationofy(x)=)]TJ /F5 11.955 Tf 5.48 -9.69 Td[(10cos(2x)+15)]TJ /F5 11.955 Tf 11.95 0 Td[(5x+x2=50withmultiplesurrogates. 19 2-1Surrogatesttedtovedatapointsofthey=(6x)]TJ /F5 11.955 Tf 11.96 0 Td[(2)2sin(2(6x)]TJ /F5 11.955 Tf 11.95 0 Td[(2))function. 22 2-2WeightedaveragesurrogatebasedonthemodelsofFigure 2-1 ........ 23 2-3Basicsequentialsampling. ............................. 26 2-4CycleoftheEfcientGlobalOptimization(EGO)algorithm. ........... 28 2-5Conservativesurrogatesviasafetymarginanderrordistribution. ........ 31 3-1Crossvalidationerroratthesecondsampledpoint,eXV2,exempliedbyttingakrigingmodel(KRG)top=6datapoints. .................... 35 3-2Plotofthetwodimensionaltestfunctions. ..................... 43 3-3CorrelationcoefcientbetweenthevectorsofPRESSRMSandeRMSforthelowdimensionalproblems. ................................ 46 3-4PRESSRMS=eRMSratioforPRS(degree=2)forthehighdimensionalproblems. 47 3-5CorrelationbetweenPRESSRMSandeRMS. ..................... 50 3-6SuccessinselectingBestRMSE(outof100experiments). ............ 51 3-7%dierenceintheeRMSwhenusingweightedaveragesurrogates. ........ 55 3-8%dierenceintheeRMSfortheoverallbestsixsurrogatesandBestPRESS. ... 57 4-1Illustrationofunbiasedsurrogatemodeling. .................... 59 4-2CumulativedistributionfunctionofthepredictionerrorFE(e). .......... 60 4-3DesignofsafetymarginusingFE(e). ........................ 61 4-4Conservativekrigingmodel(%c=90%). ..................... 62 4-5DesignofsafetymarginusingFEXV(eXV). ..................... 63 4-6Estimationofactualconservativenessusingpolynomialresponsesurfaceandcrossvalidationerrors. ................................ 68 4-7Estimationofactualconservativenessusingkrigingandcrossvalidationerrors. 69 10

PAGE 11

4-8AnalysisofkrigingcrossvalidationforBranin-Hoo. ................ 70 4-9Estimationoftheerrorintheactualconservativeness. .............. 71 4-10Spreadoftherelativeerrorgrowth. ......................... 72 5-1Cycleoftheefcientglobaloptimization(EGO)algorithm. ............ 76 5-2Errorestimationofthekrigingmodel. ........................ 79 5-3Importinguncertaintyestimates. .......................... 81 5-4GlobalsearchwithMSEGOandtwosurrogates. ................. 82 5-5LocalsearchwithMSEGOandtwosurrogates. .................. 83 5-6IllustrationoftheSasenafunction. ......................... 86 5-7Baselinedesignofthetorquearm. ......................... 88 5-8Boxplotsofthecorrelationbetweenerroranduncertaintyestimates. ...... 91 5-9BoxplotsofPRESSRMSandeRMSofsurrogatesforthetestproblems. ...... 93 5-10Medianoftheoptimizationresultsasafunctionofthenumberofcycles. .... 95 5-11Boxplotoftheoptimizationresultsasafunctionofthenumberofcycles. ... 96 5-12Medianoftheoptimizationresultswithrespecttothenumberofevaluations. 97 5-13Contourplotsofforexperimentaldesign#8ofSasenafunction. ........ 98 5-14Boxplotsofintersitedistancemeasures. ...................... 99 5-15Initialdatasetforthetorquearmexample. ..................... 100 5-16ComparisonbetweenEGOandMSEGOforthetorquearmproblem. ...... 102 5-17Optimizationhistoryformasswithrespecttothefunctionevaluationsforthetorquearmdesign. .................................. 103 5-18Torquearmdesignssuggestedbyoptimization. .................. 104 6-1Cycleoftheefcientglobaloptimization(EGO)algorithmusingtheprobabilityofimprovement. ................................... 107 6-2Cycleoftheefcientglobaloptimization(EGO)algorithmusingthemultipointprobabilityofimprovement. ............................. 110 6-3ComparisonofestimatesofprobabilityofimprovementforasingleexperimentaldesignoftheHartman3function. .......................... 115 6-4Boxplotsoftheinitialbestsample,target,andtestpoints. ............ 116 11

PAGE 12

6-5Medianoftheglobalandestimatedprobabilityofimprovement. ......... 118 6-6CumulativedistributionfunctionforkrigingpredictioninoneexperimentaldesignofHartman6ttedwith56points. .......................... 119 6-7Optimizationhistoryforsinglepointexpectedimprovementandprobabilityofimprovement. ..................................... 120 6-8Optimizationhistorywithrespecttothenumberofpointspercycle. ....... 121 6-9Optimizationhistorywithrespecttothenumberoffunctionevaluations. .... 122 A-1Quadraticpolynomialresponsesurface. ...................... 130 A-2Krigingmodel. .................................... 132 A-3Radialbasisneuralnetworkarchitecture. ..................... 133 A-4Radialbasisneuralnetworkmodel. ......................... 133 A-5LinearShepardmodel. ................................ 134 A-6Lossfunctionsusedinsupportvectorregression. ................. 136 A-7Supportvectorregressionmodel. .......................... 137 B-1Exampleofboxplot. ................................. 138 C-1eRMSanalysisforHartman6(110points). ...................... 141 C-2RelativeerrorgrowthversustargetconservativenessfortheHartman6with110points. ...................................... 142 12

PAGE 13

AbstractofDissertationPresentedtotheGraduateSchooloftheUniversityofFloridainPartialFulllmentoftheRequirementsfortheDegreeofDoctorofPhilosophyinAerospaceEngineeringMULTIPLESURROGATESFORPREDICTIONANDOPTIMIZATIONByFelipeA.C.VianaAugust2011Chair:RaphaelT.HaftkaMajor:AerospaceEngineeringStatisticalmodelingofcomputerexperimentsembracesthesetofmethodologiesforgeneratingasurrogatemodel(alsoknownasmetamodelorresponsesurfaceapproximation)usedtoreplaceanexpensivesimulationcode.Theaimofsurrogatemodelingistoconstructanapproximationofaresponseofinterestbasedonalimitednumberofexpensivesimulations.Nevertheless,afteryearsofintensiveresearchontheeld,surrogate-basedanalysisandoptimizationisstillastruggletoachievemaximumaccuracyforagivennumberofsimulations.Inthisdissertation,wehavetakenadvantageofmultiplesurrogatestoaddresstheissuesthatwefacewhenwe(i)wanttobuildanaccuratesurrogatemodelunderlimitedcomputationalbudget,(ii)usethesurrogateforconstrainedoptimizationandtheexactanalysisshowsthatthesolutionisinfeasible,and(iii)usethesurrogateforglobaloptimizationanddonotknowwheretoplaceasetofpointsinwhichwearemostlikelytohaveimprovement.Intermsofpredictionaccuracy,wehavefoundthatmultiplesurrogatesworkasinsuranceagainstpoorlyttedmodels.Additionally,weproposetheuseofsafetymarginstoconservativelycompensateforttingerrorsassociatedwithsurrogates.Wewereabletoestimatethesafetymarginforaspecicconservativenesslevel,andwefoundthatitispossibletoselectasurrogatewiththebestcompromisebetweenconservativenessandlossofaccuracy. 13

PAGE 14

Intermsofoptimization,weproposedtwostrategiesforenablingsurrogate-basedglobaloptimizationwithparallelfunctionevaluations.Therstoneisbasedonthesimultaneoususeofmultiplesurrogates(asetofsurrogatescollaborativelyprovidemultiplepoints).Thesecondstrategyusesasinglesurrogateandonecheaptoevaluatecriterion(probabilityofimprovement)formultiplepointselectionapproximation.Inbothcases,wefoundthatwecouldsuccessfullyspeeduptheoptimizationconvergencewithoutclearpenaltiesasfarasnumberoffunctionevaluations. 14

PAGE 15

CHAPTER1INTRODUCTIONTheobjectiveofthischapteristopresentthemotivationandreviewoftheliteraturethatservedasbasisofthisresearch.Thereisalsoanoutlineofthisdissertationattheendofthechapter. 1.1MotivationDespiteadvancesincomputerthroughput,thecomputationalcostofcomplexhigh-delityengineeringsimulationsoftenmakesitimpracticaltorelyexclusivelyonsimulationfordesignoptimization[ 1 ].Designandanalysisofcomputerexperiments(DACE)isthetoolofchoicetoreducethecomputationalcost[ 2 7 ].DACEembracesthesetofmethodologiesforgeneratingsurrogatemodels(alsoknownasmetamodels)usedtoreplacethegenerallyexpensivecomputercodes1.Typicalapplicationsmayinclude,butarenotlimitedto,designoptimization,sensitivityanalysisanduncertaintyquantication.ToidentifythemotivationforDACE-relatedresearch,considerTable 1-1 ,whichpresentquotesfromfrequently-citedreviewpaperspublishedinthepasttwodecades.Thecommonthemeinallofthesepapersisthehighcostofcomputersimulations:despitegrowingincomputingpower,surrogatemodelsarestillcheaperalternativestoactualsimulationmodelsinengineeringdesign.Advancesincomputationalthroughputhavehelpedthedevelopmentofnumericaloptimization;buttheyseemtofavortheincreaseincomplexityofthestate-of-the-artsimulation[ 8 ],asillustratedinFigure 1-1 .DACEtechniqueshavealsousedthegrowingcomputationalresources.Forexample,somesurrogates(suchaskrigingmodelsandresponsesurface)alsoofferinformationabouttheerrorinpredictionatagivenpoint.Thisinformationhasbeenused 1Intheliterature[ 1 7 ],thetermsmetamodel,approximationandsurrogateareusedinterchangeably.Inthisdissertation,wewillusesurrogateunlesswhenwequotesomeotherwork. 15

PAGE 16

Table1-1. Motivationforreviewpapersondesignandanalysisofcomputerexperiments. PaperYearMotivation Sacksetal.[ 2 ]1989Abstract:ManyscienticphenomenaarenowinvestigatedbycomplexcomputermodelsorcodesOften,thecodesarecomputationallyexpensivetorun,andcommonobjectiveofanexperimentistotacheaperpredictoroftheoutputtothedata.BarthelemyandHaftka[ 9 ]1993Introduction:...applicationsofnonlinearprogrammingmethodstolargestructuraldesignproblemscouldprovecosteffective,providedthatsuitableapproximationconceptswereintroduced.Sobieszczanski-SobieskiandHaftka[ 10 ]1997Abstract:TheprimarychallengesinMDOarecomputationalexpenseandorganizationalcomplexity.Simpsonetal.[ 11 ]2001Abstract:Theuseofstatisticaltechniquestobuildapproximationsofexpensivecomputeranalysiscodespervadesmuchoftodaysengineeringdesign.Simpsonetal.[ 12 ]2004Introduction:Computer-basedsimulationandanalysisisusedextensivelyinengineeringforavarietyoftasks.Despitethesteadyandcontinuinggrowthofcomputingpowerandspeed,thecomputationalcostofcomplexhigh-delityengineeringanalysesandsimulationsmaintainspace...Consequently,approximationmethodssuchasdesignofexperimentscombinedwithresponsesurfacemodelsarecommonlyusedinengineeringdesigntominimizethecomputationalexpenseofrunningsuchanalysesandsimulations.WangandShan[ 13 ]2007Abstract:Computation-intensivedesignproblemsarebecomingincreasinglycommoninmanufacturingindustries.Thecomputationburdenisoftencausedbyexpensiveanalysisandsimulationprocessesinordertoreachacomparablelevelofaccuracyasphysicaltestingdata.Toaddresssuchachallenge,approximationormeta-modelingtechniquesareoftenused. 16

PAGE 17

Table 1-1 .Continued PaperYearMotivation ForresterandKeane[ 6 ]2009Abstract:Theevaluationofaerospacedesignsissynonymouswiththeuseoflongrunningandcomputationallyintensivesimulations.Thisfuelsthedesiretoharnesstheefciencyofsurrogate-basedmethodsinaerospacedesignoptimization. Figure1-1. Evolutioninengineeringdesignoptimization(adaptedfrom[ 8 ]). toguidetheselectionofpointsinoptimization(e.g.,theefcientglobaloptimization(EGO)[ 14 ]andenhancedsequentialoptimization(ESO)[ 15 ]algorithms).ThebottomlineisthattherepertoireofdesigntoolshassubstantiallygrownovertheyearsandDACEmethodshelptailoringproblem-orientedapproachesduringthedesignprocess.Withadvancesincomputerthroughput,thecostofttingagivensurrogatedropsinrelationtothecostofsimulations.Consequentlymoresophisticatedandmoreexpensivesurrogateshavebecomepopular.Surrogatessuchasradialbasisneuralnetworks[ 16 18 ],krigingmodels[ 19 21 ],supportvectorregression[ 22 24 ]thatrequireoptimizationinthettingprocess,increasinglyreplacethetraditionalpolynomialresponsesurfaces[ 25 27 ]thatonlyrequirethesolutionofalinearsystemequations.TherelativepopularityofdifferentmethodsisshowninFigure 1-2 ,whichillustratesthenumberofpublicationsreportingtheuseofsomeofthemajorsurrogatetechniques. 17

PAGE 18

ThedatawasobtainedusingtheISIWebofKnowledge( www.isiknowledge.com ,accessedonMarch11,2011)settosearcharticlesinengineering(searchsetupshowninTable 1-2 ).Figure 1-2 showsasteadygrowthofpublicationsforallsurrogates.Thelessonisthatdifferentsurrogatesappeartobecompetitiveandreceivinggrowingattentionfromthescienticcommunity. Figure1-2. Numberofpublicationsinengineeringperyearusingfoursurrogatetechniques(datacollectedonMarch17,2011). Table1-2. Numberofpublicationsinengineeringperyearusingfoursurrogatetechniques(datacollectedonMarch17,2011)a. SurrogatetechniqueTopic(s) ResponsesurfaceresponsesurfaceKrigingkrigingORGaussianprocessSupportvectormachinesupportvectorregressionORsupportvectormachineNeuralnetworksarticialneuralnetworkORradialbasisneuralnetwork aResultsonlyforarticles(documenttype)inengineering(subjectarea).ResultsmayvaryastheISIWebofKnowledgedatabaseisupdated. Recently,therehasbeeninterestinthesimultaneoususeofmultiplesurrogatesratherthanasingleone[ 28 30 ].Thismakessensebecause(i)nosinglesurrogateworkswellforallproblems,(ii)thecostofconstructingmultiplesurrogatesisoftensmallcomparedtothecostofsimulations,and(iii)useofmultiplesurrogatesreducestheriskassociatedwithpoorlyttedmodels.Figure 1-3 exempliestheadvantagesofmultiplesurrogatesinoptimization.Twosurrogates(krigingandasecondorderpolynomialresponsesurface)arettedtovedatapoints.Itisclearthatthekrigingmodelismoreaccuratethanthesecondorder 18

PAGE 19

polynomialresponsesurface.However,theoptimization(minimization)basedonthesecondorderpolynomialresponsesurfaceleadstoasolutionclosertotheactualone. Figure1-3. Minimizationofy(x)=)]TJ /F5 11.955 Tf 5.48 -9.69 Td[(10cos(2x)+15)]TJ /F5 11.955 Tf 11.96 0 Td[(5x+x2=50withmultiplesurrogates.Functionwassampledwithvedatapoints.Theoptimumpointedbytheapparentlylessaccuratesurrogateisclosertotheactualoptimumthantheonepointedbythemostaccuratesurrogate. 1.2OutlineofDissertation 1.2.1ObjectivesTheobjectiveofthisresearchistoaddresstheissueswefacein: Surrogateaccuracy:wewanttobuildapropersurrogatemodel,butitrequiresmoresimulationsthanwecanafford.Theobjectiveistoshowtheadvantagesofusingmultiplesurrogates. Constrainedoptimizationandreliability-basedoptimization:weusethesurrogateforoptimization,andwhenwedoanexactanalysiswendthatthesolutionisinfeasible.Here,weaimatusingconservativeconstraintssothattheoptimizationispushedtothefeasibleregionandmultiplesurrogateswouldminimizethelossinaccuracy. Globaloptimization:wedonotknowhowtosimultaneouslyobtaingoodaccuracynearallpossibleoptimalsolutions.Here,theobjectiveisintelligentlyselectmultiplepointsperoptimizationcycle(samplingonregionsofhighpotentialforasingleormultipleoptima).Wewillstudytwoapproaches(i)ttingasequenceofsurrogateswitheachsurrogatedeningthepointsthatneedtobesampledforthenextsurrogates,and(ii)usingonlykrigingandexploringanaffordablecriterionforselectionofmultiplepoints. 1.2.2PublicationsandSoftwareTheresearchhasproducedthefollowingcontributionswithrespectto: 19

PAGE 20

Surrogateaccuracy:wecanidentifyaccuratesurrogateswell,especiallyasthenumberofpointsinthedatasetincreases.Thispartoftheworkwasalreadypublishedandcanbefoundin[ 31 34 ]. Constrainedoptimizationandreliability-basedoptimization:wecandesignconservativesurrogatesandselectoneforminimallossinaccuracy.Thisworkwasalreadypublishedandcanbefoundin[ 35 37 ]. Globaloptimization:wehaveresultsdemonstratingbenetsofmultiplesurrogatesin(i)runningtheefcientglobaloptimization(EGO)algorithm,and(ii)assessthevalueofanothercycle(howmuchcanweimprovegiventheselectedpoints).Thisworkwaspublishedandcanbefoundin[ 38 42 ].Althoughthisresearchisnotfocusedonsoftwaredevelopment,wemadethedevelopedcodeavailableintheformofatoolboxtobeusedintheMATLABRenvironment.WecalledittheSURROGATEStoolbox[ 43 ]anditisavailableat http://sites.google.com/site/felipeacviana/surrogatestoolbox 1.2.3OrganizationoftheTextTheorganizationofthisdissertationisasfollows.Chapter 2 presentsaliteraturereviewandsituatesthisresearchinthecontextofmultiplesurrogates.Chapter 3 discussesmultiplesurrogatesforpredictionandtheuseofcrossvalidationforselectionandcombinationofsurrogates.Chapter 4 presentstheusecrossvalidationtodesignconservativesurrogates.Chapter 5 discussestheefcientglobaloptimization(EGO)algorithmandhowwecanparallelizeitwithmultiplesurrogates.Chapter 6 presentsalternativesforparallelizationofEGOwithasinglesurrogate.Chapter 7 closesthedissertationwithasummaryofthemainlearningsandperspectiveoffuturework.Therearealsofourappendices.Appendix A givesanoverviewofthesurrogatetechniquesusedinthisdissertation.Appendix B explainswhatboxplotsare.Appendix C discussthebenetsofmultiplesurrogateswhenusingconservativesurrogates.Finally,Appendix D detailsthederivationoftheexpectedimprovement. 20

PAGE 21

CHAPTER2BACKGROUNDANDLITERATUREREVIEWInthischapterwesituateourworkwithrespecttothecurrentresearcheffortsinsurrogatebasedoptimization.Wepresenthowusingmultiplesurrogateswasreportedintheliteratureandwhataretheperspectivesinoptimization.ThischapterispartofareviewpaperpublishedintheASME2010InternationalDesignEngineeringTechnicalConferencesincollaborationwithDr.ChristianGogu(Vianaetal.[ 44 ]). 2.1MultipleSurrogatesThesimultaneoususeofmultiplesurrogatesaddressestwooftheproblemsmentionedinChapter 1 :(i)accurateapproximationrequiresmoresimulationsthanwecanafford,and(ii)surrogatemodelsforglobaloptimization. 2.1.1HowtoGenerateDifferentSurrogatesMostpractitionersintheoptimizationcommunityarefamiliaratleastwiththetraditionalpolynomialresponsesurface[ 25 27 ];somewithmoresophisticatedmodelssuchaskriging[ 19 21 ],neuralnetworks[ 16 18 ],orsupportvectorregression[ 22 24 ],andfewwiththeuseofweightedaveragesurrogates[ 30 45 47 ](Appendix A givesanoverviewofthesurrogatetechniquesusedinthiswork.).Thediversityofsurrogatemodelsmightbeexplainedbythreebasiccomponents[ 31 ].Therstdifferenceisthestatisticalmodel.Forexample,whileresponsesurfacetechniquesusuallyassumethatthedataisnoisyandtheobtainedmodelisexact,krigingusuallyassumesthatthedataisexactandtheresponseisarealizationofaGaussianprocess.Surrogatesalsodifferintheirbasisfunctions.Thatis,whileresponsesurfacesfrequentlyusemonomials,supportvectorregressionspeciesthebasisintermsofakernel(manydifferentfunctionsareused).Finally,surrogatesdifferintheirlossfunction.Theminimizationofthemeansquareerroristhemostpopularcriterionforttingthesurrogate,buttherearealternativemeasuressuchastheaverageabsoluteerror(i.e.,theL1norm). 21

PAGE 22

Itispossibletocreatedifferentinstancesofsurrogatesusingthesamesurrogatetechnique.Withpolynomialresponsesurface,onecouldchoosedifferentsetsofmonomials.Withkriging,onecangeneratedifferentmodelsbychangingthecorrelationfunction[ 48 ].Withsupportvectorregression,onecouldchangethekernelandlossfunctions[ 49 ].Figure 2-1 illustratesthisidea(Section 3.1.1 detailsthecomputationoftherootmeansquareerroreRMS). A BFigure2-1. Differentsurrogatesttedtovedatapointsofthey=(6x)]TJ /F5 11.955 Tf 11.95 0 Td[(2)2sin(2(6x)]TJ /F5 11.955 Tf 11.95 0 Td[(2))function(adaptedfrom[ 7 ]).A)Krigingsurrogateswithdifferentcorrelationfunctions.B)Supportvectorregressionmodelswithdifferentkernelfunctions. Differentsurrogatescangreatlydifferintermsofaccuracy(seetheireRMSvalues).Thechoiceofthesurrogateisadifcultproblembecauseitmightbehardtopointthebestonejustbasedonthedataset. 2.1.2MultipleSurrogatesinActionAsseenintheliterature[ 34 ],theuseofmultiplesurrogatesactslikeaninsurancepolicyagainstpoorlyttedmodels.Ifonlyonepredictorisdesired,onecouldapplyeitherselectionorcombinationofsurrogates.Selectionisusuallybasedonaperformanceindexthatappliestoallsurrogatesoftheset.Becausetestpointsmightbeprohibitivelyexpensive,weusethedatasettoestimatetheaccuracyofthesurrogatemodels.Onepopularwayofdoingthatisviacrossvalidation[ 50 51 ].Crossvalidationstartsbydividingdatapointsintosubsets. 22

PAGE 23

Thesurrogateisttoallsubsetsexceptone,anderrorischeckedinthesubsetthatwasleftout.Thisprocessisrepeatedforallsubsets.TheerrorsarecombinedinameasurecalledPRESS,whichisanestimatoroftherootmeansquareerror(asdetailedinSection 3.1.2 ).OnemaythenselectthesurrogatewiththelowestPRESS[ 52 54 ].Combiningsurrogatesisbasedonthehopeofcancelingerrorsinpredictionthroughproperaveragingofthemodels.ThisisshowninFigure 2-2 ,inwhichtheweightedaveragesurrogatecreatedusingthefoursurrogatesofFigure 2-1 hassmallereRMSthananyofthebasicsurrogates. Figure2-2. WeightedaveragesurrogatebasedonthemodelsofFigure 2-1 Crossvalidationerrorscanbeusedtoobtaintheweightsviaminimizationoftheintegratedsquareerror[ 34 46 ].Alternatively,theweightcomputationmightalsoinvolvetheuseoflocalestimatoroftheerrorinprediction.Forexample,Zerpaetal.[ 55 ]presentedaweightingschemethatusesthepredictionvarianceofthesurrogatemodels(availableinkrigingandresponsesurfaceforexample).Acar[ 56 ]investigatedusinglocalerrorestimatesforcomputingweightsinlinearcombinationofsurrogatesandalsopresentedapointwisecrossvalidationerrorasanalternativetothepredictionvariance.Nevertheless,asdiscussedbyYang[ 57 ],theadvantagesofcombinationoverselectionhaveneverbeenclaried.Accordingtothisauthor,selectioncanbebetterwhentheerrorsinpredictionaresmallandcombinationworksbetterwhentheerrorsarelarge.Vianaetal.[ 34 ]showedwhileintheorythesurrogatewithbesteRMScanbebeaten(viaweightedaveragesurrogate),inpractice,thequalityofinformationgiven 23

PAGE 24

bythecrossvalidationerrorsmakesitverydifcult;andontopofthat,potentialgainsdiminishsubstantiallyinhighdimensions.Ontheotherhand,therearecases(e.g.,designoptimization)whereitmightbebetterusingthesetsimultaneously.Thismightbethesimplestattempttosolvetheproblemofglobaloptimizationbasedonsurrogatemodels.Afterall,onesurrogatemaysuggestapointinoneregionofthedesignspace,whileadifferentsurrogatepointtoanotherregion.Forinstance,Macketal.[ 28 ]employedpolynomialresponsesurfacesandradialbasisneuralnetworkstoperformglobalsensitivityanalysisandshapeoptimizationofbluffbodydevicestofacilitatemixingwhileminimizingthetotalpressureloss.Theyshowedthatduetosmallislandsinthedesignspacewheremixingisveryeffectivecomparedtotherestofthedesignspace,itisdifculttouseasinglesurrogatemodeltocapturesuchlocalbutcriticalfeatures.Samadetal.[ 29 ]usedpolynomialresponsesurface,kriging,radialbasisneuralnetwork,andweightedaveragesurrogateinacompressorbladeshapeoptimizationoftheNASArotor37.Itwasfoundthatthemostaccuratesurrogatedidnotalwaysleadtothebestdesign.Glazetal.[ 58 ]usedpolynomialresponsesurfaces,kriging,radialbasisneuralnetworks,andweightedaveragesurrogateforhelicopterrotorbladevibrationreduction.Theirresultsindicatedthatmultiplesurrogatescanbeusedtolocatelowvibrationdesignswhichwouldbeoverlookedifonlyasingleapproximationmethodwasemployed.Saijaletal.[ 59 ]usedpolynomialresponsesurfaceandneuralnetworksforoptimallocationsofdualtrailing-edgeapsandbladestiffnesstoachieveminimumhubvibrationlevels(improveddesignsresultinginabout27%reductioninhubvibrationandabout45%reductioninappower).Thisdemonstratedthatusingmultiplesurrogatescanimprovetherobustnessoftheoptimizationataminimalcomputationalcost. 24

PAGE 25

Uptothispoint,wedidnotconsidersequentialoptimization(thatis,pointsindicatedbythesurrogateareusedtoupdatethesurrogateuntilconvergenceoftheoptimization).Multiplesurrogatesarealsousefulinthiscontext,aswewilldiscussinthenextsection. 2.2SequentialSamplingandOptimizationSequentialsamplingtsasequenceofsurrogateswitheachsurrogatedeningthepointsthatneedtobesampledforthenextsurrogate.Thiscanimprovetheaccuracyforagivennumberofpoints,becausepointsmaybeassignedtoregionswherethesurrogateshowssignsofpooraccuracy.Alternativelyoradditionally,thisapproachmayfocusthesamplingonregionsofhighpotentialfortheglobaloptimum. 2.2.1SequentialSamplingIntheliterature[ 60 62 ],thewordsequentialissometimessubstitutedbyadaptive,objective-driven,orapplication-driven,andthewordsamplingissometimesreplacedbyexperimental,designofexperiment,orsimplydesign.Weusetheuncertaintymodelassociatedwithmanysurrogatestoselectnewsimulationssothatwereducethepredictionerror.Theuncertaintystructureisreadilyavailableinsurrogatessuchaspolynomialresponsesurfaceandkriging.Wewillshowanexamplewithkriging(Appendix A givesmoreinformationaboutkriging).Thebasicsequentialsamplingapproachsamplesnextthepointinthedesignspacethatmaximizesthekrigingpredictionstandarddeviation,sKRG(x)(here,wewillusethesquarerootofthekrigingpredictionvariance).Figure 2-3 illustratestherstcycleofthealgorithm.Figure 2-3A showstheinitialkrigingmodelandthecorrespondingpredictionerror.Themaximizationofthepredictionerrorsuggestsaddingx=0.21tothedataset.TheupdatedkrigingmodelisshowninFigure 2-3B .Thereisasubstantialdecreaseintherootmeansquareerror,from4.7to1.7.Jinetal.[ 60 ]reviewedvarioussequentialsamplingapproaches(maximizationofthepredictionerror,minimizationoftheintegratedsquareerror,maximizationoftheminimumdistance,andcrossvalidation)andcomparedthemwithaonestageapproach 25

PAGE 26

A BFigure2-3. Basicsequentialsampling.A)Maximizationofkrigingpredictionstandarddeviation,sKRG(x),suggestsaddingx=0.21tothedataset.B)Updatedkriging(KRG)modelafterx=0.21isaddedtothedataset. (simplyllingoftheoriginaldesign).Theyfoundthattheperformanceofthesequentialsamplingmethodsdependedonthequalityoftheinitialsurrogate(i.e.,thereisnoguaranteethatsequentialsamplingwilldobetterthantheonestageapproach).KleijnenandVanBeers[ 61 ]proposedanalgorithmthatmixesspacellingwithsequentialsampling.Theideaisthataftertherstmodelistted,thealgorithmiteratesbyplacingasetofpointsllingasmuchofthedesignspaceaspossible(spacellingsampling)andthenchoosingtheonethatmaximizesthevarianceofthepredictedoutput(varianceoftheresponsestakenfromcrossvalidationoftheoriginaldataset).Inafollowup,VanBeersandKleijnen[ 62 ]improvedtheirapproachtoaccountfornoisyresponses.IntheworksofKleijnenandVanBeers,animprovedkrigingvarianceestimate[ 63 ]isusedandthatmightbeareasonforbetterresults.Recentdevelopmentsinsequentialsamplingareexempliedby[ 64 68 ].Rai[ 64 ]introducedaqualitativeandquantitativesequentialsamplingtechnique.Themethodcombinesinformationfrommultiplesources(includingcomputermodelsandthe 26

PAGE 27

designer'squalitativeintuitions)throughacriterioncalledcondencefunction.Thecapabilitiesoftheapproachweredemonstratedusingvariousexamplesincludingthedesignofabi-stablemicroelectromechanicalsystem.Turneretal.[ 65 ]proposedaheuristicschemethatsamplesmultiplepointsatatimebasedonnonuniformrationalB-splines(NURBs).Thecandidatesitesaregeneratedbysolvingamulti-objectiveoptimizationproblem.Theyusedfourobjectives:(i)proximitytoexistingdata,(ii)condenceincontrolpointlocations(inNURBs,thegreaterthedistancebetweenacontrolpointanditsnearestdatapoints,thelesscondencethereisinthelocationofthecontrolpoint),(iii)slopemagnitude(authorsarguethatrapidchangecouldbeduetothepresenceofmultiplelocaloptimalandthismightbeofinteresttoadesigner),and(iv)model(usedtosearchforminimaormaxima).Theeffectivenessofthealgorithmwasdemonstratedforvetrialproblemsofengineeringinterest.Gorissenetal.[ 66 ]broughtmultiplesurrogatestoadaptivesampling.Theobjectiveisselectingthebestsurrogatemodelbyaddingpointsiteratively.Theytailoredageneticalgorithmthatcombinesautomaticmodeltypeselection,automaticmodelparameteroptimization,andsequentialdesignexploration.Theyusedasetofanalyticalfunctionsandengineeringexamplestoillustratethemethodology.Rennenetal.[ 67 ]proposednesteddesigns.Theideaisthatthelowaccuracyofamodelobtainedmightjustifytheneedofanextrasetoffunctionevaluations.Theyproposedanalgorithmthatexpandsanexperimentaldesignaimingmaximizationofspacellingandnon-collapsingpoints.Finally,Loeppkyetal.[ 68 ]useddistancebasedmetricstoaugmentaninitialdesigninabatchsequentialmanner.TheyalsoproposedasequentialupdatingstrategytoanorthogonalarraybasedLatinhypercubesample. 2.2.2Optimization-DrivenSequentialSamplingSurrogate-basedoptimizationhasbeenastandardtechnologyforlongtime[ 9 ].Traditionally,thesurrogatereplacestheexpensivesimulationsinthecomputationoftheobjectivefunction(anditsgradient,ifthatisthecase).Yet,whenJonesetal.[ 14 ] 27

PAGE 28

usedkrigingtoprovidepredictionvaluesandmeasureofpredictionuncertaintyoftheobjectiveateverypoint,theyaddedanewtwistbyusingtheuncertaintymodelofthesurrogatetodirecttheoptimization.Theyintroducedtheefcientglobaloptimization(EGO)algorithm[ 14 69 70 ],whichwewillbrieydescribehere.EGOstartsbyttingakrigingmodelfortheinitialsetofdatapoints(Appendix A givesmoredetailskriging).Afterttingthekrigingmodel,thealgorithmiterativelyaddspointstothedatasetinanefforttoimproveuponthepresentbestsample.Ineachcycle,thenextpointtobesampledistheonethatmaximizestheexpectedimprovement,E[I(x)].E[I(x)]isameasureofhowmuchimprovement(uponthepresentbestsample)weexpecttoachieveifweadda(setof)point(s)(moredetailsaboutEGOcanbefoundinChapter 5 ).Unlikemethodsthatonlylookfortheoptimumpredictedbythesurrogate,EGOwillalsofavorpointswheresurrogatepredictionshavehighuncertainty.Afteraddingthenewpointtotheexistingdataset,thekrigingmodelisupdated.Figure 2-4 illustratestherstcycleoftheEGOalgorithm.Figure 2-4A showstheinitialkrigingmodelandthecorrespondingexpectedimprovement.MaximizationofE[I(x)]addsx=0.21tothedataset.Inthenextcycle,EGOusestheupdatedkrigingmodelshowninFigure 2-4B A BFigure2-4. CycleoftheEfcientGlobalOptimization(EGO)algorithm.A)Maximizationoftheexpectedimprovement,E[I(x)],suggestsaddingx=0.21tothedataset.B)Updatedkriging(KRG)modelafterx=0.21isaddedtothedataset. 28

PAGE 29

SincetheworkofJonesetal.[ 14 ],EGO-likealgorithmshaveattractedattentionofthescienticcommunity(e.g.,[ 41 69 73 ]).Inthefollowupof[ 14 ],Jones[ 70 ]providedadetailedstudyonthewaysthatsurrogatemodelscanbeusedinglobaloptimization(fromthesimpleuseofthepredictiontotheelaboratedEGOalgorithm).ForresterandKeane[ 69 ]providedanextendedandmodernreview,whichalsoincludestopicssuchasconstrainedandmulti-objectiveoptimization.Ginsbourgeretal.[ 74 ],Villemonteixetal.[ 71 ],andQueipoetal.[ 72 ]sharedthecommonpointofproposingalternativestotheexpectedimprovementforselectionofpoints.Villemonteixetal.[ 71 ]introducedanewcriterionthattheycalledconditionalminimizersentropywiththeadvantageofbeingapplicabletonoisyapplications.Queipoetal.[ 72 ]focusedontheassessmentoftheprobabilityofbeingbelowatargetvaluegiventhatmultiplepointscanbeaddedineachoptimizationcycle.Ginsbourgeretal.[ 73 ]extendedtheexpectedimprovementassamplingcriteriaallowingformultiplepointsineachadditionalcycle.However,theyalsomentionthehighcomputationalcostsassociatedwiththisstrategy.Finally,VianaandHaftka[ 41 ]proposedusingmultiplesurrogatestogeneratemultiplecandidatepointsofmaximumexpectedimprovement. 2.3BeingSafeunderLimitedNumberofSimulationsInconstrainedoptimization(constraintsbeingsurrogatemodels)orinreliability-baseddesignoptimization(limitstatecomposedbysurrogatemodels),itcanhappenthatafterrunningtheoptimizationthesolutionturnsouttobeinfeasibleduetosurrogateerrorsThissectionisdevotedtoapproachesthateither(i)useconservativeconstraintssothattheoptimizationispushedtothefeasibleregion;or(ii)maketheconstraintsmoreaccurateneartheboundarybetweenfeasibleandunfeasibledomains. 2.3.1ConservativeSurrogatesUsually,surrogatemodelsarettobeunbiased(i.e.,theerrorexpectationiszero).However,incertainapplications,itmightbeimportanttosafelyestimatetheresponse(e.g.,instructuralanalysis,themaximumstressmustnotbeunderestimatedinorder 29

PAGE 30

toavoidfailure).Oneofthemostwidelyusedmethodsforconservativeestimationistobiasthepredictionbyadditiveormultiplicativeconstants(termedsafetymarginandsafetyfactors,respectively)[ 75 77 ].Thechoiceoftheconstantisoftenbasedonpreviousknowledgeoftheproblem.However,thepracticeisfairlyrecentforsurrogate-basedanalysis.Onewayofimprovingtheconservativenessofasurrogateistorequireittoconservativelytthedatapoints([ 78 80 ]).However,thisapproachdoesnotallowtuninginthelevelofdesiredconservativeness.Previousworkofourresearchgroup([ 81 82 ]),exploredandcompareddifferentapproachestodesignconservativesurrogatemodels.TheyaresummarizedinTable 2-1 .Pichenyetal.[ 81 ]foundthattherewasnoclearadvantageofthealternativesoverthesimpleuseofsafetymargintobiasthesurrogatemodel.However,thesafetymarginapproachlackedabasisforselectingitsmagnitude.Vianaetal.[ 37 ]proposedamethodforselectingthesafetymarginbasedoncrossvalidationerrors. Table2-1. Methodsforcreatingconservativesurrogates.Adaptedfrom[ 81 ]. MethodPrinciple BiasedttingThesurrogateisconstrainedtobeabovethetrainingpoints.ConstantsafetyfactorThesurrogateresponseismultipliedbyaconstant.ConstantsafetymarginAconstantisaddedtothesurrogateresponse.ErrordistributionThepredictionerror(uncertaintyestimate)ofsurrogatessuchaskrigingandresponsesurfaceisusedtobuildcondenceintervals(conservativenesslevel). Figure 2-5 illustratestwoofthetechniquesshowninTable 2-1 .Consideraconservativeprediction,onethatoverestimatestheactualresponse.Figure 2-5A showsthattheoriginalkrigingmodel(ttobeconservative50%ofthetime)wouldpresentarootmeansquareerrorof1.5.Figure 2-5B and 2-5C areconservativesurrogatesdesignedfor90%conservativeness.Figure 2-5B showsthatbyusingsafetymargin,weshiftthesurrogateupandthatwouldleadtoarootmeansquareerrorof2.5(i.e., 30

PAGE 31

lossinaccuracyof67%).Figure 2-5C illustrateswhathappenswiththeerrordistributionapproach.Here,therootmeansquareerrorwouldbe2.9(i.e.,lossinaccuracyof93%). A B CFigure2-5. Conservativesurrogatesviasafetymarginanderrordistribution(overestimationmeansbeingconservative).A)Originalkrigingmodel(ttobeconservative50%ofthetime).B)Conservativekrigingviasafetymarginfor90%conservativeness.C)Conservativekrigingviaerrordistributionfor90%conservativeness.Conservativenesscomeswithlossinaccuracy. 2.3.2AccurateApproximationofConstraintsNeartheBoundarybetweenFeasibleandUnfeasibleDomainsOnealternativetotheuseofconservativesurrogatesistheimprovementofthesurrogatemodelneartheboundarybetweenthefeasibleandinfeasibledomains(i.e.,improvedaccuracyfortargetvaluesoftheactualfunction).Recentdevelopmentsinthisdirectionemploysequentialsampling.Audetetal.[ 83 ]andPichenyetal.[ 84 ]lookedattheissueofbettercharacterizingthefunctionofinterestaroundtargetvalues(ofthefunction).Audetetal.[ 83 ]usedtheexpectedviolationtoimprovetheaccuracyofthesurrogatefortheconstraintfunctionalongtheboundariesofthefeasible/unfeasibleregion.Pichenyetal.[ 84 ]proposedamodiedversionoftheclassicalintegratedmeansquareerrorcriterionbyweightingthepredictionvariancewiththeexpectedproximitytothetargetlevelofresponse.Themethodshowedsubstantialreductionoferrorinthetargetregions,withreasonablelossofglobalaccuracy.Bichonetal.[ 85 ]discussedhowtoformallyapplytheideasbehindEGOtothereliability-basedoptimization(RBDO)problem.Theypresentdetailsabouttheeffcientglobalreliabilityanalysis(EGRA) 31

PAGE 32

methodincludingtheexpectedviolationandfeasibilityfunctionsandhowEGRAdealswithdifferentformulationsoftheRBDOproblem. 2.4SummaryInthischapter,wereviewedbackgroundforthisproposal.Wediscussed(i)simultaneoususeofmultiplesurrogates,(ii)sequentialsamplingandoptimization,and(iii)conservativesurrogates.Thelearningsthathighlyinuencedthisresearchare:(i)usingmultiplesurrogatesisattractive(becausenosinglesurrogateworkswellforallproblemsandthecostofconstructingmultiplesurrogatesisoftensmallcomparedtothecostofsimulations),and(ii)sequentialsamplingandoptimizationisanefcientwayofmakinguseoflimitedcomputationalbudget(techniquesmakeuseofboththepredictionandtheuncertaintyestimatesofthesurrogatemodelstointelligentlysamplethedesignspace). 32

PAGE 33

CHAPTER3MULTIPLESURROGATESFORPREDICTIONPreviousworkfromKohavi[ 51 ]andRonchettietal.[ 86 ]hasshownthatttingmultiplesurrogatesandpickingonebasedoncrossvalidationerrors(PRESSRMSinparticular)isagoodstrategy,andthatcrossvalidationerrorsmayalsobeusedtocreateaweightedsurrogate(linearcombinationofsurrogatemodels).Inthischapter,wediscusshowPRESSRMSisemployedtoestimatetherootmeansquareerrorandtheminimizationoftheintegratedsquareerrorasawaytocomputetheweightsoftheweightedaveragesurrogate.WefoundthatPRESSRMSisgoodforlteringoutinaccuratesurrogates(withenoughpointsPRESSRMSmayidentifythebestsurrogateoftheset).Wealsofoundthatweightedsurrogatesbecomemoreattractiveinhighdimensions(whenalargenumberofpointsisnaturallyrequired).However,itappearsthatthepotentialgainsfromusingweightedsurrogatesdiminishsubstantiallyinhighdimensions.Finally,wealsoexaminedtheutilityofusingallthesurrogatesforformingtheweightedsurrogatesversususingasubsetofthemostaccurateones.Thisdecisionisshowntodependontheweightingscheme.Wedisseminatedthepreliminaryresultsofthisinvestigationatsomeconferences[ 31 33 ],andthenalresultswerealreadypublishedandcanbefoundinVianaetal.[ 34 ]. 3.1Background 3.1.1RootMeanSquareErroreRMSThereareseveralmeasuresthatcanbeusedtoassesstheaccuracyofthesurrogatemodel(rootmeansquareerror,maximumabsoluteerror,averageerror,etc.).Inthischapter,weusetherootmeansquareerror,though.Wechooseitbecauseitisastandardmeasureofglobalaccuracythatcanbeappliedtoanysurrogatemodel.Considery(x)and^y(x)theactualsimulationandthesurrogatepredictionatthepointx=[x1,...,xd]T,respectively.TherootmeansquareerroreRMSisgivenby: 33

PAGE 34

eRMS=vuut 1 VZD(^y(x))]TJ /F3 11.955 Tf 11.96 0 Td[(y(x))2dx,(3) whereVisthevolumeofthedesigndomainD.WecomputeeRMSbyMonte-Carlointegrationatptesttestpoints1: eRMS=vuut 1 ptestptestXi=1(^yi)]TJ /F3 11.955 Tf 11.96 0 Td[(yi)2,(3) where^yiandyiarevaluesofthesurrogatepredictionandactualsimulationatthei-thtestpoint,respectively. 3.1.2CrossValidationandPRESSRMSAlthoughtestpointsallowmuchmoreaccurateassessmentofaccuracy,inengineeringdesign,theircostmightbeprohibitive.Becauseofthat,crossvalidation(assessmentbasedondatapoints)isbecomingpopular[ 51 54 86 89 ].Crossvalidationisattractivebecauseitdoesnotdependonthestatisticalassumptionsofaparticularsurrogatetechniqueanditdoesnotrequireextraexpensivesimulations(testpoints).Nevertheless,crossvalidationshouldbeusedwithcaution,sincetheliteraturehasreportedproblemssuchasbiasinerrorestimation[ 50 90 ].Acrossvalidationerroristheerroratadatapointwhenthesurrogateisttedtoasubsetofthepdatapointsthatdoesnotincludethispoint.Whenthesurrogateisttedtoalltheotherp)]TJ /F5 11.955 Tf 12.08 0 Td[(1points,theprocesshastoberepeatedptimes(leave-one-outstrategy)toobtainthevectorofcrossvalidationerrors,eXV.Figure 3-1 illustratescomputationofthecrossvalidationerrorsforakrigingsurrogate.Whenleave-one-outbecomesexpensive,thek-foldstrategycanalsobeusedforcomputationoftheeXVvector.Theclassicalk-foldstrategy[ 51 ]divides 1Inthisdissertation,wegeneratetestpointsusinglargeLatinhypercubedesigns[ 87 88 ].Wewilldetailthisprocesswhenevernecessary. 34

PAGE 35

theavailabledata(ppoints)intop=kclusters,eachfoldisconstructedusingapointrandomlyselected(withoutreplacement)fromeachoftheclusters.Ofthekfolds,asinglefoldisretainedasthevalidationdatafortestingthemodel,andtheremainingk)]TJ /F5 11.955 Tf 12.48 0 Td[(1foldsareusedastrainingdata.Thecrossvalidationprocessisthenrepeatedktimeswitheachofthekfoldsusedexactlyonceasvalidationdata2. Figure3-1. Crossvalidationerroratthesecondsampledpoint,eXV2,exempliedbyttingakrigingmodel(KRG)top=6datapoints. ThesquarerootofthePRESSvalue3istheestimatoroftheeRMS: PRESSRMS=r 1 peXVTeXV.(3) 3.2EnsembleofSurrogates 3.2.1SelectionBasedonPRESSRMSSincePRESSRMSisanestimatoroftherootmeansquareerror,eRMS,onepossiblewayofusingmultiplesurrogatesistoselectthemodelwithbest(i.e.,smallest)PRESSRMSvalue(wecallthistheBestPRESSsurrogate).Becausethequalityoftdependsonthedatapoints,theBestPRESSsurrogatemayvaryfromonedesignof 2Weimplementedthek-foldstrategyby:(1)extractingp=kpointsofthesetusingamaximincriterion(maximizationoftheminimuminter-distance),(2)removingthesepointsfromtheset,and(3)repeatingstep(1)withtheremainingpoints.Eachsetofextractedpointsisusedforvalidationandtheremainingfortting.3PRESSstandsformpredictionsumofsquares[ 27 ,pg.170].PRESSRMSisthesquarerootofPRESS 35

PAGE 36

experiment(DOE)toanother4.Thisstrategymayincludeselectionofsurrogatesbasedonthesamemethodology,suchasdifferentinstancesofkriging(e.g.,krigingmodelswithdifferentregressionand/orcorrelationfunctions).Themainbenetfromadiverseandlargesetofsurrogatesistheincreasingchanceofavoiding(i)poorlyttedsurrogatesand(ii)DOEdependenceoftheperformanceofindividualsurrogates.Obviously,thesuccesswhenusingBestPRESSreliesonthediversityofthesetofsurrogatesandonthequalityofthePRESSRMSestimator. 3.2.2WeightedAverageSurrogateAlternatively,aweightedaveragesurrogate(WAS)intendstotakeadvantageofnsurrogatesinthehopeofcancelingerrorsinpredictionthroughproperweightingselectioninthelinearcombinationofthemodels: ^yWAS(x)=nPi=1wi(x)^yi(x)=(w(x))T^y(x),nPi=1wi(x)=1Tw(x)=1,(3) where^yWAS(x)isthepredictedresponsebytheWASmodel,wiistheweightassociatedwiththei-thsurrogateatthepointx=[x1,...,xd]T,^yi(x)isthepredictedresponsebythei-thsurrogate,and1isan1vectorof1.Furthermore,nPi=1wi(x)=1meansthatifallsurrogatesprovidethesameprediction,sowould^yWAS(x).Theweightsandthepredictedresponsescanbewritteninthevectorformasw(x)andy(x).Inthisdissertation,westudyweightselectionbasedonglobalmeasuresoferror,whichleadstow(x)=w,8x. 4Intheliterature[ 91 92 ],designofexperimentreferstothedatasampledtotthesurrogatemodel.Theexpressionissometimesreplacedbyexperimentaldesign,dataset,sample,orsimplydata. 36

PAGE 37

Inaspecicdesignofexperiment,whenconsideringasetofsurrogates,ifnotallareused,weassumesurrogatesareaddedtotheensembleoneatatimebasedontherankgivenbythePRESSRMS.Then,therstonetobepickedistheBestPRESS. 3.2.3HeuristicComputationoftheWeightsGoeletal.[ 45 ]proposedaheuristicschemeforcalculationoftheweights,namelythePRESSweightedaveragesurrogate,PWS.InPWS,theweightsarecomputedas: wi=wi nPj=1wj,wi=(Ei+Eavg),Eavg=1 nnPi=1Ei,<0,<1,(3) whereEiisthePRESSRMSofthei-thsurrogate.Thetwoparametersand,controltheimportanceofaveragingandimportanceofindividualPRESSRMS,respectively.Goeletal.[ 45 ]suggested=0.05and=)]TJ /F5 11.955 Tf 9.3 0 Td[(1. 3.2.4ComputationoftheWeightsforMinimumeRMSUsinganensembleofneuralnetworks,Bishop[ 93 ,pg.364]proposedaweightedaveragesurrogateobtainedbyapproximatingthecovariancebetweensurrogatesfromresidualsattrainingortestpoints.Here,asinAcarandRais-Rohani[ 46 ],weoptinsteadforbasingBishop'sapproachonminimizingthemeansquareerror(eWASMS): eWASMS=1 VZDe2WAS(x)dx=wTCw,(3) whereVisthevolumeofthedesigndomainD,andeWAS(x)=y(x))]TJ /F5 11.955 Tf 12.24 0 Td[(^yWAS(x)istheerrorassociatedwiththepredictionoftheWASmodel.Theintegral,takenoverthedomainofinterest,permitsthecalculationoftheelementsofCas: cij=1 VZDei(x)ej(x)dx,(3) 37

PAGE 38

whereei(x)andej(x)aretheerrorsassociatedwiththepredictiongivenbythesurrogatemodeliandj,respectively.CplaysthesameroleasthecovariancematrixinBishop'sformulation.However,weapproximateCbyusingthevectorsofcrossvalidationerrors,eXV, cij1 peXVTieXVj,(3) wherepisthenumberofdatapointsandthesub-indexesiandjindicatedifferentsurrogatemodels.GiventheCmatrix,theoptimalweightedsurrogate(OWS)isobtainedfromminimizationoftheeWASMSas: minweWASMS=wTCw,(3) subjectto: 1Tw=1.(3)ThesolutionisobtainedusingLagrangemultipliers,as: w=C)]TJ /F7 7.97 Tf 6.59 0 Td[(11 1TC)]TJ /F7 7.97 Tf 6.59 0 Td[(11.(3)Thesolutionmayincludenegativeweightsaswellasweightslargerthanone.AllowingthisfreedomwasfoundtoamplifyerrorscomingfromtheapproximationofCmatrix(Eq. 3 ).OnewaytoenforcepositivityistosolveEq. 3 usingonlythediagonalelementsofC,whicharemoreaccuratelyapproximatedthantheoff-diagonalterms.WedenotethisapproachOWSdiag.Itisworthobservingthatwhen=0and=)]TJ /F5 11.955 Tf 9.3 0 Td[(2,PWSgivesthesameweightsofOWSdiag.Wealsostudiedthepossibilityofaddingtheconstraintwi0totheoptimizationproblem;howeveritwasnotsufcienttoovercometheeffectofpoorapproximationsoftheCmatrix.OWSisrecommendedfor 38

PAGE 39

thecaseswhenanaccurateapproximationofCisavailable(i.e.,largenumberofdatapoints)andOWSdiagforalessaccurateone.WhenemployingeitherselectionbasedonPRESSRMS(i.e.,BestPRESS)oroneoftheabovementionedWASschemes,thecomputationalcostofusinganensembleofsurrogatesdependsonthecalculationofthecrossvalidationerrors.Althoughthepreviouslymentionedk-foldstrategy(Section 3.1.2 givesmoredetailsaboutcrossvalidation)canalleviatetheburden,ingeneralwecansaythat,thelargerthenumberofpointsinthedesignofexperimentthehigherthecost. 3.2.5ShouldWeUseAllSurrogates?Whenformingaweightedsurrogate,wemayuseallsurrogateswehavecreated;oralternatively,wemayusejustasubsetofthebestones.WithanexactCmatrix,thereisnoreasonnottousethemall.However,withtheapproximationgivenbyEq. 3 ,itispossiblethataddinginaccuratesurrogateswillleadtolossofaccuracy.Whenweuseasubsetofthesurrogates,wewilladdthesurrogatesaccordingtotherankgivenbyPRESSRMS.BestPRESS(surrogatewithlowestPRESSRMS)istherstonetobepicked.Itisexpectedthatthesetofsurrogatesintheensemblechangewiththedataset,sincetheperformanceofindividualsurrogatesmayvaryfromonedesignofexperimenttoanother. 3.2.6CombiningAccuracyandDiversityForbothselectionandcombination,thebestcasescenariowouldbetohaveasetofsurrogatesthataredifferentintermsofpredictionvalues(^y(x))butsimilarintermsofpredictionaccuracy(eRMS).ThiswouldincreasethechancethatWASallowserrorcancellation.Inthisdissertation,eventhoughwegeneratedasubstantialnumberofsurrogateswithdifferentstatisticalmodels,lossfunctions,andshapefunctions,weusuallymissedthisgoal.Thatis,comparablyaccuratesurrogateswereoftenhighlycorrelated.Futureresearchmayincludetheefcientgenerationofthesetofbasicsurrogates. 39

PAGE 40

3.3NumericalExperiments 3.3.1BasicandDerivedSurrogatesTable 3-1 givesdetailsaboutthe24differentbasicsurrogatesusedduringtheinvestigation(Appendix A givesashorttheoreticalreviewaboutsurrogates).TheDACEtoolboxofLophavenetal.[ 48 ],thenativeneuralnetworksMATLABRtoolbox[ 94 ],andthecodedevelopedbyGunn[ 49 ]wereusedforkriging,theradialbasisneuralnetwork,andsupportvectorregressionalgorithms,respectively.TheSURROGATEStoolboxofViana[ 43 ]wasusedtoexecutethePRSandWASalgorithmsandalsoforeasymanipulationofthesurrogates5.Table 3-2 summarizesthesurrogatesthatcanbeselectedfromthesetusingdifferentcriteria.Theseincludethebestchoiceofsurrogatethatwouldbeselectedifwehadperfectknowledgeofthefunction(BestRMSE),thesurrogateselectedbasedonthelowestPRESSRMS(BestPRESS),aswellasthevariousweightedsurrogates.Ofthese,theidealweightedsurrogatehasweightsbasedonperfectknowledge,anditprovidesaboundtothegainspossiblebyusingWAS. 3.3.2PerformanceMeasuresThenumericalexperimentsareintendedto(i)measurehowefcientisPRESSRMSasanestimatoroftheeRMS(andconsequently,howgoodisPRESSRMSforidentifyingthesurrogatewiththesmallesteRMS),and(ii)explorehowmuchtheeRMScanbefurtherreducedbytheWAS.TherstobjectiveisquantiedbycomparingthecorrelationbetweenPRESSRMSandeRMSacrossthe24surrogates.Forbothobjectives,wecompareeachbasicsurrogate,BestPRESS,andWASmodelwiththebestsurrogateof 5Theinterestedreadercancertainlyndotherpackages(e.g.,thoseavailableat http://www.kernel-machines.org http://www.support-vectormachines.org http://www.sumo.intec.ugent.be ,andthefreecompanioncodeof[ 69 ])[retrievedApril,102010]. 40

PAGE 41

Table3-1. Informationaboutthesetof24basicsurrogates(Appendix A givesandoverviewofthesurrogatetechniques). SurrogatesDetails (1)krg-poly0-exp(2)krg-poly0-gauss(3)krg-poly1-exp(4)krg-poly1-gauss(5)krg-poly2-exp(6)krg-Poly2-gaussKrigingmodel:Poly0,Poly1,andPoly2indicatezero,rst,andsecondorderpolynomialregressionmodel,re-spectively.ExpandGaussindicategeneralexponentialandGaussiancorrelationmodel,respectively.Inallcases,0i=10,and0i200,i=1,2,...,dwereused.Wechose6differentKrigingsurrogatesbyvaryingtheregressionandcorrelationmodels.(7)prs2Polynomialresponsesurface:Fullmodelofdegree2.(8)rbnnRadialbasisneuralnetwork:goal=(0.5y)2andspread=1=3.(9)svr-anova-e-full(10)svr-anova-e-short01(11)svr-anova-e-short02(12)svr-anova-q(13)svr-erbf-e-full(14)svr-erbf-e-short01(15)svr-erbf-e-short02(16)svr-erbf-q(17)svr-grbf-e-full(18)svr-grbf-e-short01(19)svr-grbf-e-short02(20)svr-grbf-q(21)svr-spline-e-full(22)svr-spline-e-short01(23)svr-spline-e-short02(24)svr-Spline-qSupportvectorregression:anova,erbf,grbfandsplineindicatethekernelfunction(erbfandgrbfkernelfunctionsweresetwith=0.5).eandqindicatethelossfunctions(eforinsensitiveandqforquadratic).fullandshortrefertodifferentvaluesfortheregularizationparameter,C,andfortheinsensitivity,.FulladoptsC=1and=110)]TJ /F7 7.97 Tf 6.58 0 Td[(4,whileshort01andshort02usestheselectionofval-uesaccordingtoCherkasskyandMa[ 95 ].Forshort01=y=p p,andforshort02=3yp lnp=p;andforbothC=100maxjy+3yj,jy)]TJ /F24 10.909 Tf 10.91 0 Td[(3yj,whereyandyarethemeanvalueandthestandarddeviationofthefunctionvaluesatthedesigndata,respectively.Wechose16differentSVRsurrogatesbyvaryingtheker-nelfunction,thelossfunction(insensitiveorquadratic)andtheSVRparameters(Cand)denethesesurrogates. Table3-2. Selectionofsurrogatesusingdifferentcriteria.BestRMSEandOWSidealaredenedbasedontestingpoints;allothersareobtainedusingdatapoints. SurrogatesDetails BestRMSEMostaccuratesurrogateforagivenDOE(basisofcomparisonforallothersurrogates).BestRMSESurrogatewithlowestPRESSRMSforagivenDOE.PWSWeightedsurrogatewithheuristiccomputationofweights.OWSidealMostaccurateweightedsurrogatebasedonthetrueCmatrix.Dependingonthesurrogateselection,OWSidealmaybelessaccuratethanBestRMSE.OWSWeightedsurrogatebasedonanapproximationoftheCmatrixusingcrossvalidationerrors.OWSdiagWeightedsurrogatebasedonthemaindiagonalelementsoftheapproximatedCmatrix(usingcrossvalidationerrors). 41

PAGE 42

thesetinaspecicdesignofexperiment(BestRMSE).Wedene%dierencesuchasthepercentgainbychoosingaspecicmodeloverBestRMSE: %dierence=100eBestRMSERMS)]TJ /F3 11.955 Tf 11.95 0 Td[(esrgtRMS eBestRMSERMS,(3) whereeBestRMSERMSistheeRMSofthebestsurrogateofthatspecicdesignofexperiment(i.e.,BestRMSE)andesrgtRMSistheeRMSofthesurrogateweareinterestedin(itcanbeeitherasingleoraWASmodel).When%dierence>0thereisagaininusingthespecicsurrogate,andwhen%dierence<0thereisaloss.ForeachbasicsurrogateandalsoforBestPRESS,itisexpectedthat%dierence0,whichmeansthattheremaybelossesandthebestcasescenarioiswhenoneofthebasicsurrogates(hopefullyBestPRESS)coincideswithBestRMSE.WhenconsideringBestPRESS,thesmallerthelossthebetteristheabilityofPRESSRMStoselectthebestsurrogateoftheset.FortheWAS,inaparticulardesignofexperiment,weaddsurrogatesaccordingtotherankgivenbythePRESSRMSvalue(i.e.,wealwaysstartfromBestPRESS).Thus,the%dierencemaystartnegative,andasweincreasethenumberofsurrogatesintheensemble,itisexpectedthat%dierenceturnstopositive,whichexpressthepotentialofWAStobebetterthanthebestsurrogateoftheset. 3.3.3TestProblemsTotesttheeffectivenessofthevariousapproaches,weemployasetofanalyticalfunctionswidelyusedasbenchmarkproblemsinoptimization[ 96 97 ].Theseare: Branin-Hoofunction(2variables) y(x)=x2)]TJ /F7 7.97 Tf 13.15 6.25 Td[(5.1x21 42+5x1 )]TJ /F5 11.955 Tf 11.95 0 Td[(62+10)]TJ /F5 11.955 Tf 5.48 -9.68 Td[(1)]TJ /F7 7.97 Tf 15.72 4.7 Td[(1 8cos(x1)+10,)]TJ /F5 11.955 Tf 9.3 0 Td[(5x110,0x215.(3) Camelbackfunction(2variables) y(x)=x41 3)]TJ /F5 11.955 Tf 11.96 0 Td[(2.1x21+4x21+x1x2+)]TJ /F5 11.955 Tf 5.48 -9.69 Td[(4x22)]TJ /F5 11.955 Tf 11.96 0 Td[(4x22,)]TJ /F5 11.955 Tf 9.29 0 Td[(3x13,)]TJ /F5 11.955 Tf 11.96 0 Td[(2x22.(3) 42

PAGE 43

Figure 3-2 illustratesthecomplexityofthetwodimensionalcases.Plotsrevealthepresenceofhighgradients. A BFigure3-2. Plotofthetwodimensionaltestfunctions.A)Branin-Hoofunction.B)Camelbackfunction. Hartmanfunctions(3and6variables) y(x)=)]TJ /F4 7.97 Tf 15.77 11.62 Td[(qPi=1aiexp )]TJ /F4 7.97 Tf 14.75 11.35 Td[(mPj=1bij(xj)]TJ /F3 11.955 Tf 11.96 0 Td[(dij)2!,0xj1,j=1,2,...,m.(3)Weusetwoinstances:Hartman3,with3variablesandHartman6with6variables.Forbothq=4anda=1.01.23.03.2.OtherparametersaregiveninTable 3-3 Table3-3. ParametersusedintheHartmanfunction. FunctionParameters Hartman3B=26643.010.030.00.110.035.03.010.030.00.110.035.03775D=26640.36890.11700.26730.46990.43870.74700.10910.87320.55470.038150.57430.88283775Hartman6B=266410.03.017.03.51.78.00.0510.017.00.18.014.03.03.51.710.017.08.017.08.00.0510.00.114.03775D=26640.13120.16960.55690.01240.82830.58860.23290.41350.83070.37360.10040.99910.23480.14510.35220.28830.30470.66500.40470.88280.87320.57430.10910.03813775 43

PAGE 44

ExtendedRosenbrockfunction(9variables) y(x)=m)]TJ /F7 7.97 Tf 6.59 0 Td[(1Pi=1h(1)]TJ /F3 11.955 Tf 11.95 0 Td[(xi)2+100)]TJ /F3 11.955 Tf 5.48 -9.68 Td[(xi+1)]TJ /F3 11.955 Tf 11.96 0 Td[(x2i2i,)]TJ /F5 11.955 Tf 9.3 0 Td[(5xi10,i=1,2,...,m=9.(3) Dixon-Pricefunction(12variables) y(x)=(x1)]TJ /F5 11.955 Tf 11.96 0 Td[(1)2+mPi=2i2x2i)]TJ /F3 11.955 Tf 11.95 0 Td[(xi)]TJ /F7 7.97 Tf 6.59 0 Td[(12,)]TJ /F5 11.955 Tf 9.29 0 Td[(10xi10,i=1,2,...,m=12.(3)Theaccuracyoftdependsonthetrainingdata(functionvaluesatDOE).Asaconsequence,performancemeasuresmayvaryfromoneexperimentaldesigntoanother.Thus,foralltestproblems,asetof100differentdesignofexperimentswereusedasawayofaveragingouttheDOEdependenceoftheresults.TheywerecreatedbytheMATLABRLatinhypercubefunctionlhsdesign[ 94 ],setwiththemaxminoptionwith1000iterations.Table 3-4 showsdetailsaboutthedatasetgeneratedforeachtestfunction. Table3-4. SpecicationsforthedesignofexperimentsandtestpointsforthebenchmarkfunctionsofChapter 3 TestproblemDesignvariablesFittingpointsTestpoints Branin-Hoo212,20,and4210,000Camelback21210,000Hartman332010,000Hartman665610,000ExtendedRosenbrock9110and22012,500Dixon-Price1218220,000 Naturally,thenumberofpointsusedtotsurrogatesincreasewithdimensionality.WealsousetheBranin-HooandtheExtendedRosenbrockfunctionstoinvestigatewhathappensinlowandhighdimensions,respectively,ifwecanaffordmorepoints.Forthecomputationofthecrossvalidationerrors,inmostcasesweusetheleave-one-outstrategy(k=pinthek-foldstrategydetailedinSection 3.1.2 ).However,fortheDixon-Pricefunction,duetothehighcostoftheleave-one-outstrategyforthe24surrogatesforall100DOEs;weadoptedthek-foldstrategywithk=14,instead.This 44

PAGE 45

meansthatthesurrogateistted14times,eachtimewith13pointsleftout(thatistotheremaining169points).Forallproblems,weusealargeLatinhypercubedesignforevaluatingtheaccuracyofthesurrogatebyMonteCarlointegrationoftheeRMS.TheseDOEsarealsocreatedbytheMATLABRLatinhypercubefunctionlhsdesign,butsetwiththemaxminoptionwithteniterations.TheeRMSistakenasthemeanofthevaluesfortheveDOEs. 3.4ResultsandDiscussionWebeginwithashortstudyontheuseofthek-foldstrategy.Whenthenumberoffoldsissmall,theaccuracyofthesurrogatetbyomittingthisfoldcanbemuchpoorerthanthatoftheoriginalsurrogate,sothatPRESSRMSislikelytobemuchlargerthaneRMS.Sincewemostlyusedanumberofpointswhichistwicethatofthepolynomialcoefcients,twofoldswouldtapolynomialwiththesamenumberofpointsasthenumberofcoefcients,whichislikelytobemuchlessaccuratethanusingallthepoints.Sointhisstudy,wevariedthevaluesofk,startingfromthesmallestvalueofkthatdividestheppointsintomorethantwofolds.Forexample,ifp=12,thisvalueisk=3(whichgeneratesfoldsof4pointseach).Thenweuseallpossiblevaluesofkuptop(whenthek-foldturnsintotheleave-one-outstrategy).Duetothecomputationalcostofthecrossvalidationerrors,wedividedthetestproblemsintotwosets: 1. Lowdimensionalproblems(Branin-Hoo,CamelbackandHartman3).WecomputethecorrelationsbetweeneRMSandPRESSRMSfordifferentkvalues.ForagivenDOE,thecorrelationiscomputedbetweenthevectorsofeRMSandPRESSRMSvaluesforthedifferentsurrogates.ThecorrelationmeasurestheabilityofPRESStosubstitutefortheexacteRMSforchoosingthebestsurrogate(thecloserthecorrelationisto1thebetter).ThisisrepeatedforallDOEs. 2. Highdimensionalproblems(Hartman6,ExtendedRosenbrock,andDixon-Price).ThecostofperformingPRESSforseveralkvaluesishigh.Tokeeplowcomputationalcost,wedothestudyonlyfortheleastexpensivesurrogate,i.e.,thePRS(degree=2)andwecalculatetheratiobetweenPRESSRMSandeRMSforeachDOE(theclosertheratioisto1thebetter). 45

PAGE 46

Figure 3-3 showsthatinlowdimensionstheuseofthek-folddoesnotdrasticallyaffectthecorrelationbetweeneRMSandPRESSRMS.Itmeansthatthepoorcorrelationhastodowiththenumberofpointsusedtotthesetofsurrogates.ThisisclearlyseenintheBranin-Hooexample,Figure 3-3A to 3-3C ,wheremorepointsimprovethecorrelation. A B C D EFigure3-3. BoxplotofthecorrelationcoefcientbetweenthevectorsofPRESSRMSandeRMS(24surrogateseach)forthelowdimensionalproblems(thiscoefcientiscomputedforall100DOEs).A)Branin-Hoo,12points.B)Branin-Hoo,20points.C)Branin-Hoo,42points.D)Camelback,12points.E)Hartman3,20points.Inparenthesis,itisthemedian,mean,andstandarddeviationofthecorrelationcoefcient,respectively.PRESSRMSiscalculatedviak-foldstrategy(whichreducestotheleave-one-outstrategywhenkisequaltothenumberofpointsusedfortting).Exceptwhenweusethesmallestvalueofk,thereisnosignicantdisadvantagecomparingthek-foldandtheleave-one-outstrategies.Appendix B detailsboxplots. 46

PAGE 47

Figure 3-4 illustratesthatinhighdimensions,anincreasingofthevalueofkimprovesthequalityoftheinformationgivenbythecrossvalidationerrors.Thebestscenarioiswhenk=p.However,theratioseemstobeacceptablewhentheeachfoldhasaround10%oftheppoints(i.e.,k=8whenp=56;k=11whenp=110;andk=14whenp=182).ThisagreeswithMeckesheimeretal.[ 89 ]. A B C DFigure3-4. BoxplotofthePRESSRMS=eRMSratioforPRS(degree=2)forthehighdimensionalproblems(thisratioiscomputedforall100DOEs).A)Hartman6,56points.B)ExtendedRosenbrock,110points.C)ExtendedRosenbrock,220points.D)Dixon-Price,182points.PRESSRMSiscalculatedviak-foldstrategy(whichequalstotheleave-one-outstrategywhenkisequaltothenumberofpointsusedfortting).Inparenthesis,itisthemedian,mean,andstandarddeviationoftheratio,respectively.Asexpected,theratiobecomesbetteraskapproachesthenumberofpoints.Inthesecases,itdoesnotpaymuchtoperformmorethan30tsforthecrossvalidations(i.e.k>30).Appendix B detailsboxplots. Backtothediscussionaboutasacriterionforsurrogateselection,Table 3-5 showsthefrequencyofbesteRMSandthePRESSRMSforeachofthesurrogatesinalltestproblems.Itcanbeobservedthatthebestsurrogatedepends(i)ontheproblem,i.e.nosinglesurrogateorevenmodelingtechniqueisalwaysthebest;and(ii)onthedesignofexperiment(DOE),i.e.forthesameproblem,thesurrogatethatperformsthebest 47

PAGE 48

canvaryfromDOEtoDOE.Inaddition,asthenumberofpointsincreases,thereisabetteragreementbetweeneRMSandPRESSRMS.Particularly,thetopthreesurrogates(boldfaceinTable 3-5 )areidentiedbetter.However,fortheBranin-Hooproblem,wenotedeteriorationinidentifyingthebestsurrogatewhengoingfrom20to42points.Thisisbecauseforhighdensityofpoints,thetrendforkrigingbecomeslessimportant,sosurrogates2,4,and6haveverysimilarperformance.Inaddition,forkrigingsurrogatesinhighdimensions,thecorrelationmodelislessimportantsincepointsaresosparse,henceseveralalmostidenticalsurrogates.Figure 3-5 showshistogramsofthecorrelationsbetweeneRMSandPRESSRMS.Figure 3-6 complementsFigure 3-5 andshowsthatasthenumberofpointsincreases,thereisabetteragreementbetweentheselectionsgivenbyeRMSandPRESSRMS.Particularly,thetopthreesurrogatesareidentiedbetter.FortheExtended-Rosenbrockwith220points,thenumberofsurrogatesthatarealmostequallyaccurateislargerthan3.ThisexplainswhythenumberoftimesthatBestRMSEiswithinthebest3PRESSRMS-rankedsurrogatesdropswhencomparedtothecasewith110points.Altogether,whentherearefewpoints(lowdimensionalproblems,i.e.,2and3variables),PRESSRMSisgoodforlteringoutbadsurrogates;whentherearemorepoints(highdimensionalproblems,i.e.,6,9and12variables)andPRESSRMScanalsoidentifythesub-setofthebestsurrogates.Table 3-6 providesthemean,medianandstandarddeviationofthe%dierenceintheeRMSforthebest3surrogatesandBestPRESS(Eq. 3 ).Sinceasinglexedsurrogatecannotbeatthebestsurrogateoftheset(whichinfactvariesfromDOEtoDOE),therecannotbeanygain.Asthelossapproacheszero,BestPRESSbecomesmorereliableinselectingthebestsurrogateoftheset,insteadofbeingahedgeagainstselectinganinaccuratesurrogate.Itcanbeobservedthat:(i)inlowdimensions,thepoorqualityofinformationgivenbyPRESSRMSmakessomesurrogatesoutperformBestPRESS(i.e.,%dierenceclosertozero,orsmallerloss),and(ii)in 48

PAGE 49

Table3-5. Frequency,innumberofDOEs(outof100),ofbesteRMSandPRESSRMSforeachbasicsurrogatea. SurrogateBranin-Hoo(12points)Branin-Hoo(20points)Branin-Hoo(42points)Camelback(12points)Hartman3(20points)Hartman6(56points)ExtendedRosen-brock(110points)ExtendedRosen-brock(220points)Dixon-Price(182points) eRMS PRESSRMS eRMS PRESSRMS eRMS PRESSRMS eRMS PRESSRMS eRMS PRESSRMS eRMS PRESSRMS eRMS PRESSRMS eRMS PRESSRMS eRMS PRESSRMS (1)0000005154213001000(2)115879163116381172000000(4)301494628005200001000(5)0100001630000183234391021(6)2223437413061100774135267440(7)0100000180000423011639(8)1300001233631176000000(9)421905107110400000000(10)030000060200000000(11)010000000000000000(12)0500002110101000000(13)000000006813000000(14)000000000501000000(16)000000000200000000(17)1132000003541418000000(18)010000033135053000000(20)161801000529109000000(21)24852002610010002211200(22)0100000003040161400(23)000000000000002800(24)040000100100110000top3surro-gates826995939910072448055818099969079100100 aThenumbersindicatetheidentityasinTable 3-1 .Itcanbeseenthat(i)forbotheRMSandPRESSRMS,thebestsurrogatedependsontheproblemandalsoontheDOE,and(ii)especiallyinthecaseofthehighdimensionalproblems,thebest3surrogatesaccordingtobotheRMSandPRESSRMStendtobethesame(boldface).Asthenumberofpointsincreasesmostsurrogatesfallbehind,i.e.theadvantageofthetopsurrogatesismorepronounced. highdimension,thequalityofinformationgivenbyPRESSRMSisbetterandasaconsequence,BestPRESSbecomeshardtobeat.ForDixon-Priceforexample,thebest3surrogatespracticallycoincidewithBestPRESS,sothe%dierenceisclosetozeroforthem. 49

PAGE 50

A B C D E F G H IFigure3-5. CorrelationbetweenPRESSRMSandeRMS.A)Branin-Hoo,12points.B)Branin-Hoo,20points.C)Branin-Hoo,42points.D)Camelback,12points.E)Hartman3,20points.F)Hartman6,56points.G)ExtendedRosenbrock,110points.H)ExtendedRosenbrock,220points.I)Dixon-Price,182points.InagivenDOE,thecorrelationiscomputedbetweenthesetsofPRESSRMSvaluesandeRMS.Correlationappearstoimprovewiththenumberofpoints. Next,westudyifwecandoanybetterbyusingaweightedaveragesurrogateratherthanBestPRESS.ForagivenDOE,surrogatesareaddedaccordingtorankbasedonPRESSRMS.WestartthestudyconsideringbestpossibleperformanceofOWS,i.e.,basedonexactcomputationoftheCmatrix(whichmaynotbepossibleinrealworldapplications,butitiseasytocomputeinoursetofanalyticalfunctions).Figure 3-7A and 3-7B exemplifywhatideallyhappenswiththe%dierenceintheeRMSasweaddsurrogatestotheWASfortheBranin-HooandExtendedRosenbrockfunctionsttedwith12and110points,respectively.Itisseenthatwecanpotentially 50

PAGE 51

Figure3-6. SuccessinselectingBestRMSE(outof100experiments).SuccessofusingPRESSRMSforsurrogatesselectionincreaseswiththenumberofpoints. gainbyaddingmoresurrogates,buteventheidealpotentialgainlevelsoffafterawhile.Disappointingly,Figure7 3-7C showsthatthemaximumpossiblegaindecreaseswithdimensionality.KeepingtheBranin-HooandExtendedRosenbrockfunctions,Figure7 3-7D and 3-7E comparetheidealgainwiththegainobtainedwithinformationbasedoncrossvalidationerrors.Itcanbeobservedthatwhileintheory(thatiswithOWSideal),BestPRESSaswellasthesurrogatewithbesteRMS(BestRMSE)canbebeaten,inpracticenoneoftheWASschemesisabletosubstantiallyimprovetheresultsofBestPRESS,i.e.,morethan10%gain.Inbothlowandhighdimensions,thebestscenarioisgivenbyOWSdiag,whichappearstotoleratewelltheuseofalargenumberofsurrogates.PWSisnotabletohandletheadditionofpoorlyttedsurrogates.Forthisreason,theremainderofthepaperdoesnotincludePWS.OWSisunstableinlowdimensionswhilepresentingsmallgainsandtheriskoflossesinhighdimensions.Table 3-7 summarizestheinformationaboutthe%dierenceintheeRMSforalltestproblems.ItisthenclearthatinlowdimensionverylittlecanbedonetoimproveBestPRESS(seemeanandmedian).Forthehighdimensionalproblems,OWSdiagseemstohandletheuncertaintyonthecrossvalidationerrorsbetterthanOWS.However,thegainsarelimitedtobetween10%and20%.Asnotedearlier,thebehaviorfortheBranin-Hoofunctionwhengoingfrom12to42pointsisanomalousbecause 51

PAGE 52

surrogates2,4and6becomeverysimilaranddependentontheDOE.WhilePRESSRMSislessreliableinidentifyingthemostaccuratesurrogate,thisislessimportant,becauseascanbeseenfromFigure 3-8 ,thedifferencesareminisculeforthesesurrogates.TheresultsofBestPRESSfortheExtendedRosenbrockfunctionshowninTable 3-7 arealsocounter-intuitive.Figure 3-8 showsthatunlikethecaseof220points,for110pointstherstthreesurrogatesareequallymuchbetterthantheremainingsurrogates.Thismakesselectionlessriskyfor110points. 3.5SummaryInthischapter,wehaveexploredtheuseofmultiplesurrogatesfortheminimumrootmeansquareerror(eRMS)inmeta-modeling.Weexplored(i)thegenerationofalargesetofsurrogatesandtheuseofcrossvalidationerrorsasacriterionforsurrogateselection,and(ii)aweightedaveragesurrogatebasedontheminimizationoftheintegratedsquareerror(incontrasttoheuristicschemes).Thestudyallowstheconclusionthatthebenetsofbothstrategiesdependondimensionalityandnumberofpoints: 1. Withsufcientnumberofpoints,PRESSRMSbecomesverygoodforrankingthesurrogatesaccordingtopredictionaccuracy.Ingeneral,PRESSRMSisgoodforlteringoutinaccuratesurrogates,andasthenumberofpointsincreasesPRESSRMScanalsoidentifythebestsurrogateoftheset(oranotherequallyaccuratesurrogate). 2. Asthedimensionincreasesthepossiblegainsfromaweightedsurrogatediminisheventhoughourabilitytoapproximatetheidealweightsimproves.Inthetwodimensionalproblems,OWSidealpresentedamediangainof60%;whileinpractice,therewasalossof25%andnoimprovementoverBestPRESS.FortheExtended-Rosenbrockfunction(ninevariables),OWSidealpresentedamediangainof20%;whileinpractice,therewasagainofjust6%(forbothOWSdiagandOWS;however,theformerpresentsbetterperformancewhenconsideringthemeanandstandarddeviation).Therefore,wecansaythatusingmultiplesurrogatesandPRESSRMSforidentifyingthebestsurrogateisagoodstrategythatbecomesevermoreusefulwithincreasingnumberofpoints.Ontheotherhand,theuseofweightedaveragesurrogatedoesnot 52

PAGE 53

seemtohavethepotentialofsubstantialerrorreductionsevenwithlargenumberofpointsthatimproveourabilitytoapproximatetheidealweightedsurrogate.Additionally,wecanpointoutthefollowingndings: 1. Forlargenumberofpoints,PRESSRMSasobtainedthroughthek-foldstrategysuccessfullyestimatestheeRMS.Inthesetoftestproblems,thereisverylittleimprovementwhenperformingmorethan30tsforthecrossvalidations(i.e.k>30). 2. Athighpointdensity(possibleinlowdimensions)thechoiceofthetrendfunction(orregressionfunctionasitisalsosometimescalled)forkrigingisnotimportant.Atverylowpointdensity(characteristicofhighdimensions)thechoiceofcorrelationfunctionisnotimportant.Bothsituationsleadtotheemergenceofmultiplealmostidenticalkrigingsurrogates. 3. WhenusingaWAS,OWSdiagseemstobethebestchoice,unlessthelargenumberofpointsallowstheuseofOWS.Inthecasesthatwehavestudied,OWSdiagwasabletostabilizetheadditionoflousysurrogatesuptothepointofusingallcreatedsurrogates. 4. Whilewehavebeenabletogeneratealargenumberofdiversesurrogates,wewerelesssuccessfulingeneratingasubstantialnumberofdiversesurrogateswithcomparablehighaccuracy.Thischallengeisleftforfutureresearch. 53

PAGE 54

Table3-6. %dierenceintheeRMS,denedinEq. 3 ,ofthebest3basicsurrogates(accordingtohowoftentheyhavethebestPRESSRMS,asshowninTable 3-5 )andBestPRESSforeachtestproblema. ProblemSurrogateFreq.ofbestPRESSRMSMedianMeanStdDev Branin-Hoo,12points(9)19)]TJ /F5 11.955 Tf 9.3 0 Td[(3)]TJ /F5 11.955 Tf 9.3 0 Td[(2136(17)32)]TJ /F5 11.955 Tf 9.3 0 Td[(18)]TJ /F5 11.955 Tf 9.3 0 Td[(2733(20)18)]TJ /F5 11.955 Tf 9.3 0 Td[(17)]TJ /F5 11.955 Tf 9.3 0 Td[(1918BestPRESS)]TJ /F5 11.955 Tf 9.3 0 Td[(26)]TJ /F5 11.955 Tf 9.3 0 Td[(4355Branin-Hoo,20points(2)790)]TJ /F5 11.955 Tf 9.3 0 Td[(1331(4)9)]TJ /F5 11.955 Tf 9.3 0 Td[(14)]TJ /F5 11.955 Tf 9.3 0 Td[(2955(9)5)]TJ /F5 11.955 Tf 9.3 0 Td[(152)]TJ /F5 11.955 Tf 9.3 0 Td[(189148BestPRESS)]TJ /F5 11.955 Tf 9.3 0 Td[(3)]TJ /F5 11.955 Tf 9.3 0 Td[(3161Branin-Hoo,42points(2)31)]TJ /F5 11.955 Tf 9.3 0 Td[(12)]TJ /F5 11.955 Tf 9.3 0 Td[(2657(4)28)]TJ /F5 11.955 Tf 9.3 0 Td[(3)]TJ /F5 11.955 Tf 9.3 0 Td[(2156(6)41)]TJ /F5 11.955 Tf 9.3 0 Td[(21)]TJ /F5 11.955 Tf 9.3 0 Td[(4255BestPRESS)]TJ /F5 11.955 Tf 9.3 0 Td[(18)]TJ /F5 11.955 Tf 9.3 0 Td[(3751Camelback,12points(1)15)]TJ /F5 11.955 Tf 9.3 0 Td[(40)]TJ /F5 11.955 Tf 9.3 0 Td[(4230(7)18)]TJ /F5 11.955 Tf 9.3 0 Td[(8)]TJ /F5 11.955 Tf 9.3 0 Td[(1213(9)11)]TJ /F5 11.955 Tf 9.3 0 Td[(24)]TJ /F5 11.955 Tf 9.3 0 Td[(3125BestPRESS)]TJ /F5 11.955 Tf 9.3 0 Td[(29)]TJ /F5 11.955 Tf 9.3 0 Td[(3529Hartman3,20points(2)11)]TJ /F5 11.955 Tf 9.3 0 Td[(12)]TJ /F5 11.955 Tf 9.3 0 Td[(2226(8)31)]TJ /F5 11.955 Tf 9.3 0 Td[(5)]TJ /F5 11.955 Tf 9.3 0 Td[(33206(18)13)]TJ /F5 11.955 Tf 9.3 0 Td[(23)]TJ /F5 11.955 Tf 9.3 0 Td[(2617BestPRESS)]TJ /F5 11.955 Tf 9.3 0 Td[(21)]TJ /F5 11.955 Tf 9.3 0 Td[(2831Hartman6,56points(17)18)]TJ /F5 11.955 Tf 9.3 0 Td[(7.9)]TJ /F5 11.955 Tf 9.29 0 Td[(10.29.7(18)53)]TJ /F5 11.955 Tf 9.3 0 Td[(0.1)]TJ /F5 11.955 Tf 9.29 0 Td[(1.72.6(20)9)]TJ /F5 11.955 Tf 9.3 0 Td[(3.5)]TJ /F5 11.955 Tf 9.29 0 Td[(4.03.4BestPRESS)]TJ /F5 11.955 Tf 9.3 0 Td[(1.9)]TJ /F5 11.955 Tf 9.29 0 Td[(5.28.2ExtendedRosenbrock,110points(5)32)]TJ /F5 11.955 Tf 9.3 0 Td[(0.09)]TJ /F5 11.955 Tf 9.29 0 Td[(0.470.85(6)410.00)]TJ /F5 11.955 Tf 9.29 0 Td[(0.191.02(7)23)]TJ /F5 11.955 Tf 9.3 0 Td[(0.13)]TJ /F5 11.955 Tf 9.29 0 Td[(0.520.94BestPRESS)]TJ /F5 11.955 Tf 9.3 0 Td[(0.02)]TJ /F5 11.955 Tf 9.29 0 Td[(1.034.15ExtendedRosenbrock,220points(5)39)]TJ /F5 11.955 Tf 9.3 0 Td[(1.28)]TJ /F5 11.955 Tf 9.29 0 Td[(3.557.85(6)26)]TJ /F5 11.955 Tf 9.3 0 Td[(2.10)]TJ /F5 11.955 Tf 9.29 0 Td[(6.5515.11(22)14)]TJ /F5 11.955 Tf 9.3 0 Td[(3.75)]TJ /F5 11.955 Tf 9.29 0 Td[(9.1917.54BestPRESS)]TJ /F5 11.955 Tf 9.3 0 Td[(2.80)]TJ /F5 11.955 Tf 9.29 0 Td[(7.1716.49Dixon-Price,182points(5)210.000.000.00(6)400.00)]TJ /F5 11.955 Tf 9.29 0 Td[(0.010.05(7)390.000.000.00BestPRESS0.000.000.03 aForthebasicsurrogates,thenumbersindicatetheidentityasinTable 3-1 .Thenegativesignof%dierenceindicatesalossintermsofeRMSforthespecicsurrogatecomparedwiththebestsurrogate.Thelossdecreaseswithincreasingnumberofpoints.Atlownumberofpoints,BestPRESSmaynotbeevenasgoodasthethirdbestsurrogate.Inhighdimension,BestPRESSisasgoodasthebestofthethree. 54

PAGE 55

A B C D EFigure3-7. %dierenceintheeRMS,denedinEq. 3 ,whenusingweightedaveragesurrogates.A)IdealOWSforBranin-Hoo,12points.B)IdealOWSforExtendedRosenbrock,110points.C)IdealOWSforalltestproblems(usingall24surrogates).D)Medianofthe%differenceofWASforBranin-Hoo,12points.E)Medianofthe%differenceofWASforExtendedRosenbrock,110points.Figure7 3-7A and 3-7B illustratestheeffectofaddingsurrogatestotheensemble(pickedoneatatimeaccordingtothePRESSRMSranking)fortheBranin-HooandExtendedRosenbrockfunctions(bestcasesinlowandhighdimensions,respectively).Figure7 3-7C showsthatthegaininusingOWSidealdecreasestoaround10%forproblemsinhighdimension(BH,CB,H3,H6,ER,andDParetheinitialsofthetestproblemandinparenthesisitisthenumberofpointsontheDOEs).ForFigure7 3-7D and 3-7E ,inparenthesis,itisthemedianandstandarddeviationwhenusingvesurrogates(afterwhichthereisnoimprovementinpractice).WhileintheorythesurrogatewithbesteRMS(BestRMSE)canbebeaten(OWSideal),inpractice,thequalityofinformationgivenbythecrossvalidationerrorsmakesitverydifcultinlowdimensions.Inhighdimensionthequalitypermitsgain,butitisverylimited.Appendix B detailsboxplots. 55

PAGE 56

Table3-7. %dierenceintheeRMSoftheWASschemesandBestPRESSforeachtestproblemwhenusingall24surrogates. ProblemaStatisticsBranin-Hoo,12pointsBranin-Hoo,20pointsBranin-Hoo,42pointsCamelback,12pointsHartman3,20points OWSidealMedian6269665434Mean6168665534StdDev10101297BestPRESSMedian)]TJ /F5 11.955 Tf 9.3 0 Td[(26)]TJ /F5 11.955 Tf 9.29 0 Td[(3)]TJ /F5 11.955 Tf 9.3 0 Td[(18)]TJ /F5 11.955 Tf 9.29 0 Td[(29)]TJ /F5 11.955 Tf 9.3 0 Td[(21Mean)]TJ /F5 11.955 Tf 9.3 0 Td[(43)]TJ /F5 11.955 Tf 9.29 0 Td[(31)]TJ /F5 11.955 Tf 9.3 0 Td[(37)]TJ /F5 11.955 Tf 9.29 0 Td[(35)]TJ /F5 11.955 Tf 9.3 0 Td[(28StdDev5561512931OWSMedian)]TJ /F5 11.955 Tf 9.3 0 Td[(35)]TJ /F5 11.955 Tf 9.29 0 Td[(24)]TJ /F5 11.955 Tf 9.3 0 Td[(34)]TJ /F5 11.955 Tf 9.29 0 Td[(23)]TJ /F5 11.955 Tf 9.3 0 Td[(18Mean)]TJ /F5 11.955 Tf 9.3 0 Td[(43)]TJ /F5 11.955 Tf 9.29 0 Td[(41)]TJ /F5 11.955 Tf 9.3 0 Td[(51)]TJ /F5 11.955 Tf 9.29 0 Td[(42)]TJ /F5 11.955 Tf 9.3 0 Td[(24StdDev39556113126OWSdiagMedian)]TJ /F5 11.955 Tf 9.3 0 Td[(119)]TJ /F5 11.955 Tf 9.29 0 Td[(249)]TJ /F5 11.955 Tf 9.3 0 Td[(61)]TJ /F5 11.955 Tf 9.29 0 Td[(84)]TJ /F5 11.955 Tf 9.3 0 Td[(182Mean)]TJ /F5 11.955 Tf 9.3 0 Td[(144)]TJ /F5 11.955 Tf 9.29 0 Td[(356)]TJ /F5 11.955 Tf 9.3 0 Td[(104)]TJ /F5 11.955 Tf 9.29 0 Td[(181)]TJ /F5 11.955 Tf 9.3 0 Td[(205StdDev99374151769120ProblemStatisticsHartman6,56pointsExtendedRosenbrock,110pointsExtendedRosenbrock,220pointsDixon-Price,24surrogates,182pointsDixon-Price,21surrogates,182pointsOWSidealMedian12.320.825.89.230.5Mean12.721.124.99.730.7StdDev3.34.63.92.42.5BestPRESSMedian)]TJ /F5 11.955 Tf 9.3 0 Td[(1.90.0)]TJ /F5 11.955 Tf 9.3 0 Td[(3.20.00.0Mean)]TJ /F5 11.955 Tf 9.3 0 Td[(5.2)]TJ /F5 11.955 Tf 9.29 0 Td[(1.0)]TJ /F5 11.955 Tf 9.3 0 Td[(7.60.0)]TJ /F5 11.955 Tf 9.3 0 Td[(2.6StdDev8.24.216.60.035.2OWSMedian)]TJ /F5 11.955 Tf 9.3 0 Td[(2.76.111.4)]TJ /F5 11.955 Tf 9.29 0 Td[(7.04.4Mean)]TJ /F5 11.955 Tf 9.3 0 Td[(3.95.39.1)]TJ /F5 11.955 Tf 9.29 0 Td[(7.34.1StdDev5.24.211.84.94.2OWSdiagMedian)]TJ /F5 11.955 Tf 9.3 0 Td[(31615.6)]TJ /F5 11.955 Tf 9.29 0 Td[(917.2Mean)]TJ /F5 11.955 Tf 9.3 0 Td[(37)]TJ /F5 11.955 Tf 9.3 0 Td[(1313.6)]TJ /F5 11.955 Tf 9.29 0 Td[(15315.5StdDev2418213.5139610.7 aFortheBranin-Hoofunction,thenumberofpointsusedforttingthesurrogatesisvaried.FortheDixon-Pricefunction,boththefullsetofsurrogatesandapartialsetobtainedwhentherstthreebestsurrogateswereleftoutareconsideredandused.Thesignof%dierenceindicatesthatthereisloss()]TJ /F1 11.955 Tf 9.3 0 Td[()orgain(+)inusingthespecicWASschemewhencomparedwiththebestsurrogateofthesetintermsofeRMS. 56

PAGE 57

A B C D EFigure3-8. %dierenceintheeRMS,denedinEq. 3 ,fortheoverallbestsixsurrogatesandBestPRESS.A)IdealOWSforBranin-Hoo,12points.B)IdealOWSforExtendedRosenbrock,110points.C)IdealOWSforalltestproblems(usingall24surrogates).D)Medianofthe%differenceofWASforBranin-Hoo,12points.E)Medianofthe%differenceofWASforExtendedRosenbrock,110points.Inparenthesis,itisthemedian,mean,andstandarddeviation.Valuesclosertozeroindicatebetterts.NumbersontheabscissaindicatethesurrogatenumberinTable 3-1 .BestPRESShasperformancecomparablewiththebest3surrogatesoftheset 57

PAGE 58

CHAPTER4USINGCROSSVALIDATIONTODESIGNCONSERVATIVESURROGATESUsually,surrogatesarettobeunbiased(i.e.,theerrorexpectationiszero).However,incertainapplications,itmightbeimportanttosafelyestimatetheresponse(e.g.,instructuralanalysis,themaximumstressmustnotbeunderestimatedinordertoavoidfailure).Inthischapterweusesafetymarginstoconservativelycompensateforttingerrorsassociatedwithsurrogates.ThiswasacollaborativeeffortwithDr.VictorPicheny.Weproposetheuseofcrossvalidationforestimatingtherequiredsafetymarginforadesiredlevelofconservativeness(percentageofsafepredictions).Theapproachwastestedonthreealgebraicexamplesfortwobasicsurrogates,namelykrigingandpolynomialresponsesurface.Fortheseexampleswefoundthatcrossvalidationiseffectiveforselectingthesafetymargin.Wedisseminatedthepreliminaryresultsofthisinvestigationatsomeconferences[ 35 36 ],andthenalresultswerepublishedinajournal[ 37 ]. 4.1Background 4.1.1ConservativeSurrogatesWhentheresponsey(x)isexpensivetoevaluate,weapproximateitbyalessexpensivemodel^y(x)(surrogatemodel),basedon1)assumptionsonthenatureofy(x)and2)ontheobservedvaluesofy(x)atasetofpoints,calledtheexperimentaldesign[ 92 98 ](alsoknownasdesignofexperiment).Figure 4-1A showsakrigingmodelttedtosevenequallyspaceddatapointsofthey=(6x)]TJ /F5 11.955 Tf 11.95 0 Td[(2)2sin(2(6x)]TJ /F5 11.955 Tf 11.95 0 Td[(2))function.Inthisexample,thepredictionerrorsareconservativeinhalfofthedesignspace,asillustratedbyFigure 4-1B .Here,whenestimatesarehigherthanthetrueresponsewecallthemconservative.Hence,conservativeestimationstendtooverestimatetheactualresponse.Aconservativesurrogate^yC(x)obtainedbyaddingasafetymarginstoanunbiasedsurrogatemodel^y(x)isanempiricalestimatorofthetype 58

PAGE 59

A BFigure4-1. Illustrationofunbiasedsurrogatemodeling.A)krigingmodel(KRG)fory=(6x)]TJ /F5 11.955 Tf 11.95 0 Td[(2)2sin(2(6x)]TJ /F5 11.955 Tf 11.95 0 Td[(2))ttedwithsevendatapoints.B)Predictionerrorassociatedwiththissurrogate. ^yC(x)=^y(x)+s.(4)Whenwechecktheaccuracyofaconservativesurrogate,wecalculatetherootmeansquareerror eRMS=vuut 1 VZD(^yC(x))]TJ /F3 11.955 Tf 11.95 0 Td[(y(x))2dx,(4) whereVisthevolumeofthedesigndomainD.TheeRMSiscomputedbyMonte-Carlointegrationatalargenumberofptesttestpoints eRMS=vuut 1 ptestptestXi=1e2Ci,(4) eCi=^yCi)]TJ /F3 11.955 Tf 11.96 0 Td[(yi=^yi)]TJ /F3 11.955 Tf 11.96 0 Td[(yi+s,(4) where^yCiandyiarevaluesoftheconservativepredictionandactualsimulationatthei)]TJ /F3 11.955 Tf 11.95 0 Td[(thtestpoint,respectively. 4.1.2ConservativenessLevelandRelativeErrorGrowthAlthoughtherearedifferentmeasuresoftheconservativenessofanapproximation(e.g.,theaverageerrororthemaximumnon-conservativeerror);forconvenience,weusethepercentageofconservative(i.e.,positive)errors.Withtheactualprediction 59

PAGE 60

errorse(x)=^y(x))]TJ /F3 11.955 Tf 12.16 0 Td[(y(x)(atanypointofthedesignspace),wecanusethecumulativedistributionfunctionofthepredictionerrorFE(e)tondthefractionoftheerrorsthatareconservative.Infact,FE(e)givestheproportionoftheerrorsthatliesbellowanyvalue.Therefore,thepercentageofconservativeerrors%cisobtainedby %c=100(1)]TJ /F3 11.955 Tf 11.95 0 Td[(FE(0)).(4)AsillustratedinFigure 4-2 ,withFE(e)wecaneasilyobtaintheconservativenesslevelofthesurrogate.Itmightbeworthnotingthatthedistributionoftheerrorsisnotalwayssymmetricwithrespecttoe=0.BothFigure 4-1 and 4-2 showthatalthough50%oftheerrorsareconservative,thenon-conservativeerrorscanhavegreatermagnitudethantheconservativeones. Figure4-2. CumulativedistributionfunctionofthepredictionerrorFE(e)ofthesurrogateshowninFigure 4-1 .50%oftheerrorsareconservative(positive). Thesafetymarginisselectedtoturnadesiredpercentageoftheerrorsconservative.Thatis,foragivenconservativeness%c,thesafetymargincanbeexpressedintermsofFE(e)as s=)]TJ /F3 11.955 Tf 9.29 0 Td[(F)]TJ /F7 7.97 Tf 6.58 0 Td[(1E1)]TJ /F5 11.955 Tf 14.02 8.09 Td[(%c 100.(4)Asstatedbefore,conservativeestimatorstendtooverestimatethetruevalues.Asaconsequence,theaccuracyofthesurrogateisdegraded.Wedenetherelativeerrorgrowth(lossinaccuracy)intermsoftheeRMSas 60

PAGE 61

REG=eRMS eRMS)]TJ /F5 11.955 Tf 11.96 0 Td[(1,(4) whereeRMSistakenatagiventargetconservativeness,andeRMSistheeRMSvalueofthesurrogatewithoutaddinganysafetymargin.Figure 4-3 illustrateshowweapplyEq. 4 toobtainthesafetymarginsthatwouldleadtoaconservativenessof%c=90%ofthesurrogateshowninFigure 4-1 .Figure 4-4 illustratestheeffectofusingasafetymargintoachieve%c=90%whenttingthedatausedinFigure 4-1 .Figure 4-4A showstheoriginalkrigingmodelandtheconservativecounterpart.Addingthesafetymargins=2.4makestheeRMStoincreasefrom1.4to2.6.Thatmeansthattheerrorincreasedbyapproximately86%.Figure 4-4B illustratestheshiftinginthepredictionerrors.Afteraddingthesafetymargin,90%ofthedomain(0x1)becomesconservative. A BFigure4-3. DesignofsafetymarginusingFE(e).A)%c=90%achievedwithasafetymarginofs=2.4.B)90%oftheerrorsaregreaterthanzerofortheconservativesurrogate. 4.2DesignofSafetyMarginUsingCrossValidationErrorsInpractice,FE(e)isunknown,sothesafetymarginthatensuresagivenlevelofconservativenesscannotbedeterminedexactly.Weproposetodesignthesafetymarginusingcrossvalidation.Thatis,wereplaceFE(e)bythecumulativedistributionfunctionofthecrossvalidationerrorsFEXV(eXV).ThevectorofcrossvalidationerrorseXVCassociatedwiththeconservativesurrogateissimply 61

PAGE 62

A BFigure4-4. Illustrationofconservativekrigingmodel(%c=90%)forthey=(6x)]TJ /F5 11.955 Tf 11.95 0 Td[(2)2sin(2(6x)]TJ /F5 11.955 Tf 11.95 0 Td[(2))functionttedwithsevendatapoints.A)Effectofsafetymarginonprediction.B)Effectofsafetymarginonpredictionerror.Conservativenesscomesatthepriceoflosingaccuracy. eXVC=eXV+s.(4)Thatmeansthatwhenweaddasafetymargintoapredictor,wedonotneedtorepeatthecostlyprocessofcrossvalidation.ThecomputationofPRESSRMSemploysEq. 3 witheXVCreplacingeXV.Thisway,theestimatedsafetymarginforagiventargetconservativenessis: ^s=)]TJ /F3 11.955 Tf 9.3 0 Td[(F)]TJ /F7 7.97 Tf 6.58 0 Td[(1EXV1)]TJ /F5 11.955 Tf 14.02 8.08 Td[(%c 100.(4)Figure 4-5 illustrateshowwewoulddesignthesafetymarginfor%c=90%usingcrossvalidationintheexampleofFigure 4-1 .UsingFEXV(eXV),thesafetymarginfor%c=90%iss=2.7,whichmeansthatwewouldgetanslightlymoreconservativesurrogate(actualconservativenesswouldbe%c=91%).NotethattherearetwoerrorsourcesassociatedwithusingEq. 4 forestimatings(ascomparedtotheexactvaluedenedbyEq. 4 ).Therstisduetonitesamplingandthesecondisduetotheuseofcrossvalidationerrorsinsteadofactualerrors.WecanestimatetheuncertaintyduetonitesamplingbyassumingthatI(eXVi)(whichequals1ifeXVi>0and0otherwise)areindependentandidenticallydistributedrandom 62

PAGE 63

Figure4-5. DesignofsafetymarginusingFEXV(eXV).Crossvalidationerrorswouldoverestimatethesafetymarginsuggestings=2.7asopposedtos=2.4suggestedbyactualerrors(Figure 4-3 ). variablesfollowingaBernoullidistributionwithaprobabilityofsuccessc.Althoughthisisastrongassumption(sincethepredictionerrorsarespatiallycorrelated),itmightbeacceptableifthepointsarefarfromeachother(asinthecaseofthedatapointsandcrossvalidationerrors).Withthat,thestandardintervalestimationofthe(1)]TJ /F8 11.955 Tf 12.59 0 Td[()condenceinterval,CIS,fortheconservatives,^c,isusuallytakenas[ 99 ]: CIS=^cp)]TJ /F7 7.97 Tf 6.58 0 Td[(1=2(^c^q)1=2,^q=1)]TJ /F5 11.955 Tf 12.14 0 Td[(^c,(4) z=2=)]TJ /F7 7.97 Tf 6.59 0 Td[(1(1)]TJ /F8 11.955 Tf 11.95 0 Td[(=2),(4) wherepisthenumberofdatapoints,(z)isthestandardnormaldistributionfunction,andissomepre-speciedvaluebetween0and1.CISisalsoknownastheWaldinterval.However,aspointedbytheliterature[ 100 ],Eq. 4 failsspeciallyforsmallpandhighvaluesofc(whicharetheonesofinterestmanytimes).Anexampleofwell-acceptedalternativetothestandardintervalisthesocalledWilsoninterval[ 101 ]: CIW=(^cp)+2=2 p+2p1=2 p+2^c^q+2 4p1=2.(4) 63

PAGE 64

Inadditiontodesigningthesafetymargin,wealsousecrossvalidationtoestimatetherelativeerrorgrowth(i.e.,toestimateoflossinaccuracy)usingPRESSRMSinEq. 4 ,insteadofeRMS REGXV=PRESSRMS PRESSRMS)]TJ /F5 11.955 Tf 11.96 0 Td[(1,(4) wherePRESSRMSisthePRESSRMSvalueofthesurrogatewithoutaddinganysafetymargin.Althoughthefocusofthischapteristheselectionofpropersafetymarginforagivenconservativenesslevel,wealsocheckedimmediatebenetsofmultiplesurrogates.Appendix C showsbyusingcrossvalidationwecanminimizethelossesinaccuracybyselectingbetweenalternatesurrogates. 4.3NumericalExperiments 4.3.1BasicSurrogatesTable 4-1 detailsthetwosurrogatesusedduringtheinvestigation(thestrengthofthemethodisthatitisnottiedtoaparticularsurrogate).TheDACEtoolboxofLophavenetal.[ 48 ],andtheSURROGATEStoolboxofViana[ 43 ]wereusedtoexecutethekrigingandthepolynomialresponsesurfacealgorithms,respectively.TheSURROGATEStoolboxwasalsousedforeasymanipulationofthesurrogates. Table4-1. Informationaboutthetwogeneratedsurrogatesusedinthestudyofconservativesurrogates. SurrogatesDetails prsPolynomialresponsesurface:fullsecondordermodel.krgKrigingmodel:Setwithzeroorderpolynomialregressionmodel(constanttrendfunction)andGaussiancorrelation.0i=10,and0i200,i=1,2,...,dwereused. 4.3.2TestProblemsAstestproblems,weemployedthethreefollowingwidelyusedanalyticalbenchmarkproblems[ 96 ]: 64

PAGE 65

Branin-Hoofunction(2variables) y(x)=x2)]TJ /F7 7.97 Tf 13.15 6.25 Td[(5.1x21 42+5x1 )]TJ /F5 11.955 Tf 11.95 0 Td[(62+10)]TJ /F5 11.955 Tf 5.48 -9.68 Td[(1)]TJ /F7 7.97 Tf 15.72 4.7 Td[(1 8cos(x1)+10,)]TJ /F5 11.955 Tf 9.3 0 Td[(5x110,0x215.(4) Hartmanfunction(6variables) y(x)=)]TJ /F7 7.97 Tf 15.92 11.36 Td[(4Pi=1aiexp )]TJ /F4 7.97 Tf 14.02 11.36 Td[(nvPj=1bij(xj)]TJ /F3 11.955 Tf 11.96 0 Td[(dij)2!,0xj1,j=1,2,...,nv,nv=6,a=1.01.23.03.2,B=266410.03.017.03.51.78.00.0510.017.00.18.014.03.03.51.710.017.08.017.08.00.0510.00.114.03775,D=26640.13120.16960.55690.01240.82830.58860.23290.41350.83070.37360.10040.99910.23480.14510.35220.28830.30470.66500.40470.88280.87320.57430.10910.03813775,(4) ExtendedRosenbrockfunction(9variables) y(x)=m)]TJ /F7 7.97 Tf 6.59 0 Td[(1Pi=1h(1)]TJ /F3 11.955 Tf 11.95 0 Td[(xi)2+100)]TJ /F3 11.955 Tf 5.48 -9.68 Td[(xi+1)]TJ /F3 11.955 Tf 11.96 0 Td[(x2i2i,)]TJ /F5 11.955 Tf 9.3 0 Td[(5xi10,i=1,2,...,m=9.(4)Thequalityoft,andthustheperformance,dependsontheexperimentaldesign.Hence,foralltestproblems,asetofdifferentLatinhypercubedesigns[ 87 88 ]wereusedasawayofaveragingouttheexperimentaldesigndependenceoftheresults.WeusedtheMATLABRLatinhypercubefunctionlhsdesign[ 94 ],setwiththemaxminoptionwith1,000iterationstogeneratetheexperimentaldesignsfortting.FortheBranin-HooandHartman6weused1,000experimentaldesignsandforRosenbrockweused100(becauseofthehighcostofcrossvalidation).Table 4-2 detailsthedatasetgeneratedforeachtestfunction.FortheBranin-Hoofunction,wechose17pointsbecausetohavereasonableaccuracyforthetwosurrogates(34isjustthedouble).ForHartman6,wechose56pointsbecause56istwicethenumberoftermsinthefullquadraticpolynomialinsixdimensions(110isroughlythedouble).ForRosenbrock, 65

PAGE 66

wechose110pointsbecause110istwicethenumberoftermsinthefullquadraticpolynomialinninedimensions.WealsousetheBranin-HooandtheHartman6functionstoinvestigatetoinvestigatetheeffectofthepointdensity.Forthecomputationofthecrossvalidationerrors,inmostcasesweusetheleave-one-outstrategy(k=pinthek-foldstrategy,asdescribedinSection 3.1.2 ).However,fortheRosenbrockfunction,duetothehighcostoftheleave-one-outstrategy;weusedthek-foldstrategywithk=22,instead.Forallproblems,weusealargeLatinhypercubedesignforevaluatingtheactualconservativeness,Eq. 4 ,andtherelativeerrorgrowthofthesurrogates,Eq. 4 .TheseexperimentaldesignsarealsocreatedbytheMATLABRLatinhypercubefunctionlhsdesign,butsetwiththemaxminoptionwithteniterations. Table4-2. ExperimentaldesignspecicationsforbenchmarkfunctionsofChapter 4 TestproblemaDesignvariablesFittingpointsTestpoints Branin-Hoo217and3410,000Hartman6656and11010,000ExtendedRosenbrock911012,500 aFortheBraninHoofunction,wechose17pointstohavereasonableaccuracyforthetwosurrogates(34isjustthedouble).ForHartman,wechose56points,because56istwicethenumberoftermsinthefullquadraticpolynomialinsixdimensions(110isroughlythedouble).ForRosenbrock,wechose110points,because110istwicethenumberoftermsinthefullquadraticpolynomialinninedimensions. 4.4ResultsandDiscussionFirst,wecheckedwhethercrossvalidationisreliableforselectingthesafetymargin.Figure 4-6 illustratestheperformanceofthepolynomialresponsesurfacemodelinestimatingtheconservativenesslevel.WedesignedthesafetymarginforagiventargetconservativenessusingEq. 4 (crossvalidationerrors).WecheckedtheactualconservativenessusingEq. 4 (largesetoftestpoints).Forsmallnumberofpoints,forexampleFigure 4-6A and 4-6B ,wecanincurlargeerrorsintheconservativenesslevel(duetowrongselectionofsafetymargin).However,increasingthenumberof 66

PAGE 67

pointsallowsbetteraccuracyinselectionofsafetymargin,asseeninFigure 4-6C to 4-6E (althoughitisdifculttoknowpreciselytherequirednumberofpointsforagivenapplication).Notethatsparsenessdoesnotseemtobeimportant.Figure 4-6C to 4-6E showsthattheHartman6andRosenbrockfunctionswith56and110points,respectively,dowelleventhoughthereislessthanonepointperorthant.Additionally,wecanmaketwoobservations: Figure 4-6A showsthatweunder-estimatetheactualconservativeness.Itmeanswhenweaskfor50%conservativeness,thedistributionofcrossvalidationerrorsactuallysaysthatweshouldaddanegativesafetymargin.Although,thisisnotasafepractice,weobservedthatithappensmorefrequentlyinthelowlevelsofconservativeness.Inmostapplications,weareinterestedinhighlevelsofconservativeness. Figure 4-6E showswecouldadequatelyestimatethesafetymarginevenwhenweusethek-foldstrategyforcrossvalidation.Figure 4-7 isthecounterpartofFigure 4-6 forthekrigingmodel.Again,thesafetymarginisdesignedusingcrossvalidationerrorsandactualconservativenessischeckedwithtestpoints.Here,krigingstronglyoverestimatesthesafetymarginfortheBranin-Hoo.Webelievethatthereasonforsuchbehavioristhevoidcreatedwhenweleftoutonepointduringcrossvalidation.Figure 4-8A illustratesthevolumefractionofthelargestemptycirclethatcantinbetweenpointswhenonepointisremovedforcrossvalidation.Onaverage,thisvoidoccupiesavolumeaslargeas30%and16%ofthedesignspacewhen17and34pointsareused,respectively.Figure 4-8B showstheratioofthevolumefractionofthevoidsofcrossvalidationdataandtheoriginalexperimentaldesign.Thevolumeofthevoidsincrossvalidationisabout50%timesbiggerthanthevoidsintheoriginaldataset.Surrogatesthatdointerpolationofthedata,suchaskriging,areverysensitivetothis.Asaresult,crossvalidationtendstooverestimatetheerrors,asseeninFigure 4-8C and 4-8D .Becauseofthat,weclosethediscussionsonkrigingforBranin-Hoofunctionknowingthatweoverestimatethesafetymargins,whichleadstoover-conservative(saferthandesired)surrogates.From 67

PAGE 68

A B C D EFigure4-6. Estimationofactualconservativenessusingpolynomialresponsesurface(prs)andcrossvalidationerrors.A)Branin-Hoo,17points.B)Branin-Hoo,34points.C)Hartman6,56points.D)Hartman6,110points.E)Rosenbrock,110points.Solidlinerepresentsthemedianoftheactualconservativenessovertheexperimentaldesigns(1,000ofthemforallfunctionsbutRosenbrock,whichhasonly100).Grayarearepresentsthe[1090]percentiles.Dashedlinesrepresentperfectestimation(target%c=actual%c).Increasingthenumberofpoints(inspiteofsparsity)allowsbetteraccuracy. thispointon,forBranin-Hoo,weonlyshowtheresultsforthepolynomialresponsesurface.Forallotherfunctions,bothkrigingandpolynomialresponsesurfacebenetsfromincreasednumberofpointsandthecrossvalidationtendstoestimatetheerrorsefciently.Next,weestimatetheerrorsduetolimitedsamplingintheactualconservativenessusingtheWilsoninterval.Figure 4-9 showsthecomparisonsbetweenthecondencelimitsoftheactualconservativenessovertheexperimentaldesignsanditsestimationbasedonEq. 4 .Withenoughpoints,theWilsonintervalgivesareasonableestimate 68

PAGE 69

A B C D EFigure4-7. Estimationofactualconservativenessusingkriging(krg)andcrossvalidationerrors.A)Branin-Hoo,17points.B)Branin-Hoo,34points.C)Hartman6,56points.D)Hartman6,110points.E)Rosenbrock,110points.Solidlinerepresentsthemedianoftheactualconservativenessovertheexperimentaldesigns(1,000ofthemforallfunctionsbutRosenbrock,whichhasonly100).Grayarearepresentsthe[1090]percentiles.Dashedlinesrepresentperfectestimation(target%c=actual%c).Increasingthenumberofpoints(inspiteofsparsity)allowsbetteraccuracy. ofthecondenceintervalsforhigherlevelsofconservativeness(asshowntheresultsforHartman6).ForRosenbrock,theresultsarenotasgoodinthelowerlevelsofconservativeness(wehavetorememberthatinpracticeweareinterestedontheoppositerange,thatishigherlevelsofconservativeness).Wesuspectthatseveralfactorscontributeforthatsuchastheuseofk-foldcrossvalidation(sincealargenumberofpointsisleftout,itishardertoestimatethesmallererrorsneededinthelowerconservativenessrange)andthesmallnumberofexperimentaldesigns(weusedonly100,andweexpectthatbythetimethatwegetto1,000experimentaldesigns, 69

PAGE 70

A B C DFigure4-8. AnalysisofkrigingcrossvalidationforBranin-Hoo.A)Boxplotsofthevolumefractionofthelargestemptycirclethatcantinbetweenpointswhenonepointisremovedforcrossvalidation(outofthe1,000experimentaldesigns).B)BoxplotsoftheratiobetweenthevolumefractionofthelargestemptycirclesthatcantinbetweenpointswhenonepointisremovedforcrossvalidationVXVfandfortheoriginaldatasetVf(outofthe1,000experimentaldesigns).C)ErrorratioforBranin-Hoottedwith17points.D)ErrorratioforBranin-Hoottedwith34points.Figure 4-8C and 4-8D showtheboxplotsoftheratiobetweenPRESSRMSandeRMSoutofthe1,000experimentaldesigns.Appendix B describesboxplots.Estimationofaccuracywithcrossvalidationmightnotalwaysbenetfromthedatadensitybecauseofthehighimpactofthevoidsinsurrogatesthatinterpolatesdatasuchaskriging. thegurewouldbelessnoisy).Webelievethattheuseofthek-foldstrategyisaninterestingtopicoffutureresearch.Finally,wecheckwhethercrossvalidationallowsestimationofrelativeerrorgrowth.Figure 4-10 comparestheactualandtheestimatedrelativeerrorgrowthasafunctionofthetargetconservativeness.TheactualrelativeerrorgrowthischeckedwithEq. 4 (largesetoftestpoints)andusingtheeRMSoftheunbiasedsurrogatesasreference. 70

PAGE 71

A B C D E F G HFigure4-9. Estimationoftheerrorintheactualconservativenessversustargetconservativeness:coverageprobabilityofthe[1090]percentinterval(resultsfor1,000experimentaldesignsforallfunctionsbutRosenbrock,whichhasonly100).A)Branin-Hoo,17points:prs.B)Branin-Hoo,34points:prs.C)Hartman6,56points:prs.D)Hartman6,110points:prs.E)Rosenbrock,110points:prs.F)Hartman6,56points:krg.G)Hartman6,110points:krg.H)Rosenbrock,110points:krg. TheestimatedrelativeerrorgrowthisobtainedwithEq. 4 andusesthePRESSRMSoftheunbiasedsurrogatesasreference.Onceagain,theestimatesarepoorforsmall 71

PAGE 72

numberofpoints,butincreasingthenumberofpointspermitsbetterestimation.ThistrendisobservedfromFigure 4-10A to 4-10E A B C D E F G HFigure4-10. Spreadswiththe[105090]percentilesoftherelativeerrorgrowth(resultsfor1,000experimentaldesignsforallfunctionsbutRosenbrock,whichhasonly100).A)Branin-Hoo,17points:prs.B)Branin-Hoo,34points:prs.C)Hartman6,56points:prs.D)Hartman6,110points:prs.E)Rosenbrock,110points:prs.F)Hartman6,56points:krg.G)Hartman6,110points:krg.H)Rosenbrock,110points:krg.Morepointsallowgoodestimationofrelativeerrorgrowth. 72

PAGE 73

4.5SummaryWeproposedusingcrossvalidationfordesigningconservativeestimatorsandmultiplesurrogatesforimprovedaccuracy.Theapproachwastestedonthreealgebraicexamplesfortenbasicsurrogatesincludingdifferentinstancesofkriging,polynomialresponsesurface,radialbasisneuralnetworksandsupportvectorregressionsurrogates.Fortheseexampleswefoundthat: 1. Thebestsurrogatechangeswithsamplingpoints(densityandlocation)andwithtargetconservativeness. 2. Crossvalidationappearstobeusefulforbothestimationofsafetymarginandselectionofsurrogate.However,itmaynotbeaccurateenoughwhenthenumberofdatapointsissmall. 3. TheuncertaintyintheestimationoftheactualconservativenesscanbeestimatedusingtheWilsoninterval. 73

PAGE 74

CHAPTER5EFFICIENTGLOBALOPTIMIZATIONALGORITHMASSISTEDBYMULTIPLESURROGATESSurrogate-basedoptimizationforcomplexengineeringsystemsproceedsincycles.Eachcycleconsistsofanalyzinganumberofdesigns,ttingasurrogate,performingoptimizationbasedonthesurrogateforacandidateoptimum,andnallyanalyzingthatcandidate.Algorithmsthatusesurrogateuncertaintyestimatortoguidetheselectionofnextsamplingcandidatearereadilyavailable,e.g.,theefcientglobaloptimization(EGO)algorithm.However,addingonepointatatimemaynotbeidealwhenonecanrunsimulationsinparallelandthemainconcerniswall-clocktimeratherthanthetotalnumberofsimulations.Moreover,theneedforuncertaintyestimateslimitsadaptivesamplingtosurrogatessuchaskrigingandpolynomialresponsesurface(normallyimplementedwithuncertaintyestimates).Weproposethemultiplesurrogateefcientglobaloptimization(MSEGO)algorithm,whichaddsseveralpointsperoptimizationcyclewiththehelpofmultiplesurrogates.Weimportuncertaintyestimatesfromonesurrogatetoanothertoallowuseofsurrogatesthatdonotprovidethem.Theapproachistestedonthreeanalyticandoneengineeringexamplesfortenbasicsurrogatesincludingkriging,radialbasisneuralnetworks,radialbasisfunction,linearShepard,andsixdifferentinstancesofsupportvectorregression.Wefoundthat(i)ourMSEGOworkswellwiththeimporteduncertaintyestimates,and(ii)MSEGOdeliveredbetterresultsinafractionoftheoptimizationcyclesneededbyEGO.Wedisseminatedthepreliminaryresultsofthisinvestigationattwoconferences[ 39 41 ],andthenalresultsweresubmittedtoajournal[ 102 ]. 5.1Background:EfcientGlobalOptimizationAlgorithmInthissection,wegiveanoverviewoftheefcientglobaloptimization(EGO)algorithmbyJonesetal.[ 14 ].EGOstartsbyconstructingakrigingsurrogate(Appendix 74

PAGE 75

A detailskriging)interpolatingtheinitialsetofdatapoints1.Afterconstructingthekrigingmodel,thealgorithmiterativelyaddspointstothedatasetinanefforttoimproveuponthepresentbestsampleyPBS.Theimprovementatapointxis I(x)=max(yPBS)]TJ /F3 11.955 Tf 11.96 0 Td[(Y(x),0),(5) whichisarandomvariablebecauseYisarandomvariable(recollectthatkrigingmodelstheresponsey(x)asarealizationofaGaussianprocessY(x)).Thus,theexpectedimprovement2is EI(x)=E[I(x)]=s(x)[u(u)+(u)],u=[yPBS)]TJ /F5 11.955 Tf 12.25 0 Td[(^y(x)]=s(x),(5) where(.)and(.)arethecumulativedensityfunction(CDF)andprobabilitydensityfunction(PDF)ofanormaldistribution,respectively,yPBSisthepresentbestsample,^y(x)isthekrigingprediction,ands(x)isthepredictionstandarddeviation(hereestimatedasthesquarerootofthekrigingpredictionvariance3). 1Inpractice,thereisnoparticularreasontorestricttheefcientglobaloptimizationalgorithmtokriging.Theonlyrequirementisasurrogatewithanuncertaintymodel.Krigingisapopularchoiceinthedesignofcomputerexperimentscommunity.Ifinterpolationisdesired,someimplementationsofradialbasisfunctions[ 103 ]couldalsobeused.Ifthereinterpolationisnotalimitation(suchasinexperimentaloptimization),eventhetraditionalpolynomialresponsesurfacecouldbeused(inwhichcase,onehastobeawarethatthesamepointmightbesampledmorethanonce).2AlgebranecessarytogetEq. 5 fromEq. 5 canbefoundinAppendix D andalsoin[ 69 73 ].3Kriginguncertaintyhasbeenknowntounderestimatetheactualerrorofthefunctionduetothemaximumlikelihoodestimateofregressioncoefcientsandcorrelationfunctionparameters[ 63 ].TheinterestedreadercanlearnfromKleijnenetal.[ 104 ]howthequalityoftheuncertaintyestimatesinuencestheefcientglobaloptimizationalgorithm. 75

PAGE 76

Figure 5-1 illustratestwocyclesoftheefcientglobaloptimization(EGO)algorithm.Figure 5-1A showstheinitialkrigingmodelandthecorrespondingexpectedimprovementEI(x).ThemaximizationofEI(x)addsx=0.19tothedataset.Afteraddingthenewpointtotheexistingdataset,thekrigingmodelisupdated(usuallywithoutthecostlyupdateofthecorrelationparameters).Inthiscycletheexpectedimprovementismaximalwheretheuncertaintyishigh,leadingtoexplorationofthedesignspace.Inthenextcycle,EGOusestheupdatedkrigingmodelshowninFigure 5-1B .Thistime,themaximizationofEI(x)addsx=0.74tothedataset.Thatis,theexpectedimprovementismaximalnearthepresentbestsolutionindicatingarenementoranexploitationcycle. A BFigure5-1. Cycleoftheefcientglobaloptimization(EGO)algorithm.A)Maximizationoftheexpectedimprovement,EI(x),suggestsaddingx=0.19tothedataset.B)Updatedkriging(KRG)modelafterx=0.19isaddedtothedataset. EGOiteratesuntilastoppingcriterionismet.Duetohighcomputationalcostofactualsimulations,itiscommontousethemaximumnumberoffunctionevaluationsasthestoppingcriterionorthevisualobservationoftheconvergencerateoftheobjectivefunction.Anotheralternativeistosetatargetvaluefortheexpectedimprovement, 76

PAGE 77

meaningthenextcycleisonlycarriediftheexpectedimprovementisaboveacertainvalue[ 69 ].TraditionalimplementationsofEGO-likealgorithmsaddasinglesimulationpointpercycle.However,opportunitiesforparallelcomputingandthehumaneffortassociatedwithsettingupsimulationsdrivecomplexapplicationstowardsrunningmultiplesimulationspercycle.Thefragilityofsomesimulations(i.e.,theymayabort)alsoencouragesalargenumberofsimulationspercycle,whichmakestheapproachlesssensitivetoafewfailedsimulations.Inaddition,inmanyengineeringapplications,itmaytakeweekstocompletesimulations,andonlyaveryfewcyclesareundertaken.Insuchcasesitmaypaytorunmultiplesimulationsatonceevenifthiswillincreasethetotalnumberofsimulationsneededtoreachtheoptimumascomparedtodoingthemoneatatime.Onepossiblestrategyforprovidingmultiplepointsperoptimizationcyclewouldbesearchingformultiplelocaloptimaoftheexpectedimprovement.However,thisdoesnotgivecontrolofthenumberofpointspercycle(numberofpointsisnotadesignerchoicebutitistightenwiththenumberoflocaloptima).Alternatively,researchintohowtoaddmultiplepointspercyclerstextendedtheexpectedimprovementofEq. 5 toaversionthatratherthanasinglepointxreturnstheexpectedimprovementwhenmultiplennewpointsD=[x1...xnnew]Tareaddedtothedataset[ 105 106 ].However,theoptimizationofthedesignofexperimentsDtomaximizethemultiplepointexpectedimprovementiscomputationallychallenging[ 73 ].Weproposethemultiplesurrogateefcientglobaloptimization(MSEGO)algorithm.MSEGOusesasetofsurrogatestoprovidemultiplepointspercycle.Onecouldimplementitwithdifferentinstancesofkriging(createdbychangingthecorrelationfunctionforexample).However,wewillillustratethealgorithmwithasetofsurrogatescreatedbasedondifferenttechniques(e.g.,radialbasisneuralnetworks,linearShepard,andsupportvectorregression).ThisversionofMSEGOrequiresrunning 77

PAGE 78

EGOwithsurrogatesthatcommonlydonotfurnishuncertaintyestimates(suchassupportvectorregressionmodels).Thoseestimatescancertainlybeprovidedbysophisticatedandmathematicallysoundapproaches(suchastheBayesianapproachofSeoketal.[ 107 ]).Here,weproposeimportingerrorestimatesfromkrigingandusingthemwithallothersurrogates.Althoughlessattractivefromthetheoreticalpointofview,thisheuristicapproachavoidstheoverheadassociatedwithestimatingtheuncertaintyforeachsurrogate.WehopethatMSEGOimprovestheglobalandlocalsearchcapabilitiesofthetraditionalEGO(withonlyonepointprovidedbykriging)byusingseveralsimulationspercycle.Withthat,weexpecttoreducethenumberofcyclesforconvergence.Whenkrigingisanaccuratesurrogate,itmaybebettertousemultiplelocaloptimaofthekrigingexpectedimprovement.However,MSEGOisexpectedtobemorerobusttothecasesinwhichkrigingpoorlytsthedata. 5.2ImportingErrorEstimatesfromAnotherSurrogateBeforeweexploreimportinguncertaintyestimates,wecheckwhethertheestimatesareaccurateforsurrogatesthatprovidethem.Theremightexistmanypossiblemeasuresofthequalityofthepredictionvariance.Wefollowourpreviouswork[ 38 ]andusethecorrelationbetweentheabsoluteerrorsje(x)jandthesquarerootofthepredictionvariances(x)atalargenumberptestoftestpoints =corr(je(x)j,s(x)),e(x)=^y(x))]TJ /F3 11.955 Tf 11.96 0 Td[(y(x).(5)isnotanindicatorofaccuracyofthesurrogatemodel,butanindicatorofhowwellthepredictionvariancecapturesthespatialvariationoftheerror.Figure 5-2A providesvisualinformationonhowwelltheerrormagnitudeagreeswiththeuncertaintyestimates.Wecanseethatkriginguncertaintymightnotalwaysbeaccurate.Figure 78

PAGE 79

5-2B illustrateshowwelltheuncertaintyestimatecapturesthevariationsoftheerrormagnitude.Forthiscase,wehave=0.91.Ideally,pointswouldlieonthedashedline.approachingonedoesnotmeanthatwehavegoodagreementbetweenthemagnitudes.isoneifthevariationofs(x)islinearwithj^yKRG(x))]TJ /F3 11.955 Tf 11.95 0 Td[(y(x)j(nomattertheslope).Figure 5-2B alsoshowsacuriousbranchingoftheuncertainty.Weexplainitbylookingattheinterval0x0.5.Figure 5-2A showsthats(x)followsj^yKRG(x))]TJ /F3 11.955 Tf 11.96 0 Td[(y(x)jverycloselyonlyfromx=0tox=0.235.Asaconsequence,thosepointswillappearmuchclosertothe45deglineofFigure 5-2B .Figure 5-2C showsthesameplotconsideringonly0x0.5. A B CFigure5-2. ErrorestimationofthekrigingmodelofFigure 5-1A (200pointsevenlyspreadbetween0and1).A)Kriginguncertaintyestimateandpredictionerror.B)Kriginguncertaintyestimateversuspredictionerror.C)Kriginguncertaintyestimateversuspredictionerrorwith0x0.5.s(x)isthesquarerootofthekrigingpredictionvariance.Inthiscase,wehave=0.91. Weproposeimportinguncertaintyestimatesfromsurrogatesthatsharesomecommonaspects(e.g.,interpolationofdatapoints).Weillustratethemechanismwithasimpleexample.Supposethatinstancesofkrigingandsupportvectorregressionarebuiltfromfourpoints,asillustratedinFigure 5-3A .ThekrigingmodelhasaGaussiancorrelationfunctionandaconstantforthelowfrequencycomponent(trendfunction).ThesupportvectorregressionmodelusesaGaussiankernelfunctionwith=0forthe-insensitivelossfunction.Asaresult,bothsurrogatesareinterpolators.Weproposecombiningthepredictionofamodel,forexamplesupportvectorregression,withthe 79

PAGE 80

uncertaintyestimateofanothermodel,forexamplekriging,asshowninFigure 5-3B .ComparingFigure 5-3C and 5-3A ,wecanseethatthekriginguncertaintyperformsjustalittleworseinestimatingtheerrorsinsupportvectorregressionthaninkrigingitself.Wechecknextthecorrelationbetweentheabsoluteerrorsofsurrogateaandthesquarerootofthepredictionvarianceimportedfromsurrogateb(sameprincipleasinEq. 5 ) ab=corr(jea(x)j,sb(x)).(5)Figure 5-3D illustrateswhatwewouldgetforthesupportvectorregression(counterpartofFigure 5-3B ).Whilethekrigingsurrogatepresents=0.91,thesupportvectorregressionmodelhas=0.83.Thatis,thekriginguncertaintystructurecapturesverywellthevariationsofthesupportvectorregressionpredictionerrors. 5.3EfcientGlobalOptimizationAlgorithmwithMultipleSurrogatesWeassumethatmultiplesurrogatesareavailable(withnativeorimportederrorestimates).Weproposerunningtheefcientglobaloptimization(EGO)algorithmwithmultiplesurrogatessimultaneously.Ineachoptimizationcycle,wemaximizetheexpectedimprovement,Eq. 5 ,ofeachsurrogateoftheset.Wecallitthemultiplesurrogateefcientglobaloptimization(MSEGO)algorithm.WehopethatthemultiplesurrogatespotentiallysuggestmultiplepointsthatwillreducethenumberofcyclesEGOneedsforconvergence.Intermsofwallclocktime,MSEGOisadvantageousonlyifrunningoneormultiplesimulationstakesapproximatelythesametime(trueforparallelcomputationofthesimulations).Forsimplicity,weillustratetheapproachusingonlytwosurrogates,krigingandsupportvectorregression(thatimportsthepredictionvariancefromthekrigingmodel).Themultiplesurrogateefcientglobaloptimization(MSEGO)algorithmiterativelyaddstwopointstothedatasetcomingfromtheindividualmaximizationoftheexpectedimprovementofbothsurrogates.Afteraddingthenewpointstotheexistingdata 80

PAGE 81

A B C DFigure5-3. Importinguncertaintyestimates(samedatapointsasinFigure2 5-1A ).A)Kriging(KRG)andsupportvectorregression(SVR)modelsttedtovedatapoints.B)SVRandthesquarerootofpredictionvarianceimportedfromKRG(ingray).C)SquarerootofkrigingpredictionvarianceandabsoluteerrorofSVR.D)KRGuncertaintyestimateversusSVRpredictionerror.AlthoughnotbasedontheSVRstatisticalassumptions,theKRGuncertaintyestimateoffersareasonablereplacement.Herewehave=0.83. set,bothsurrogatesareupdated.Figure 5-4 illustratesonecycleoftheMSEGOalgorithmrunningwiththesetwosurrogates.Figure 5-4A showstheinitialkrigingmodelandthecorrespondingexpectedimprovement.Maximizationofthekrigingexpectedimprovementsuggestsaddingx=0.19tothedataset.Figure 5-4B showstheinitialsupportvectorregressionmodelandthecorrespondingexpectedimprovement.Here,theexpectedimprovementsuggestsaddingx=0.77.Weaddbothpointsandinthenextcycle,ouralgorithmusestheupdatedmodelsshowninFigure 5-4C .Heretheuseoftwosurrogatesfavorsexplorationofthedesignspacesincethesuggestedpointsarefarapart.Ifthetwopointsareclose,theymayinsteadacceleratethelocalsearchasillustratedinFigure 5-5 .Figure 5-5A showsthatbymaximizingthekrigingexpected 81

PAGE 82

A B CFigure5-4. GlobalsearchwithMSEGOandtwosurrogates.Thefunctiony=(6x)]TJ /F5 11.955 Tf 11.95 0 Td[(2)2sin(2(6x)]TJ /F5 11.955 Tf 11.95 0 Td[(2))isinitiallysampledatx=00.50.681T.A)Maximizingtheexpectedimprovementofthekrigingmodel.B)Maximizingtheexpectedimprovementofthesupportvectorregressionmodel.C)Updatedmodelsafterx=0.19andx=0.77areaddedtotheset. improvement,wewouldaddx=0.69tothedataset.Figure 5-5B showsthatthesupportvectorregressionsuggestsaddingx=0.81tothedataset.AfterincludingbothpointsweobtaintheupdatedmodelsshowninFigure 5-5C .InbothFigures 5-4 and 5-5 ,weobservedthatsometimesthepointsuggestedbyasupportvectorregression(oranyothersurrogateinamoregeneralsense)happenstobeveryclosetothesecondorthirdbestpointthatkrigingwouldsuggest(localoptimaofthekrigingexpectedimprovement).Wedonotbelievethatthiswouldbea 82

PAGE 83

A B CFigure5-5. LocalsearchwithMSEGOandtwosurrogates.Thefunctiony=(6x)]TJ /F5 11.955 Tf 11.95 0 Td[(2)2sin(2(6x)]TJ /F5 11.955 Tf 11.95 0 Td[(2))isinitiallysampledatx=00.50.681T.A)Maximizingtheexpectedimprovementofthekrigingmodel.B)Maximizingtheexpectedimprovementofthesupportvectorregressionmodel.C)Updatedmodelsafterx=0.69andx=0.81areaddedtotheset. generalobservation(particularlyinhighdimensionalproblems,wheresurrogatestendtodisagreewitheachother).Nevertheless,wendthistobepositiveinMSEGO.Thatis,atagivenoptimizationcycle,thereisachancethatanothersurrogatemightsuggestapointclosetowherekrigingwouldsuggestinthenextcycles(potentiallyreducingtheexplorationefforts).WhenusingMSEGO,wearechangingtheshapeoftheexpectedimprovementbychangingthesurrogateprediction^y(x)thatweuseinEq. 5 (consideringthesame 83

PAGE 84

uncertaintys(x)andpresentbestsolutionyPBS).FromFigures 5-4 and 5-5 ,welearnedthatevensmalldifferencesinthesurrogatepredictioncanleadtodifferentpointsofmaximumexpectedimprovement.Inthecaseofmultiplesurrogateefcientglobaloptimization(MSEGO)algorithm,wehaveaheuristicapproachforintroducingdiversityofsurrogatesbyusingdifferenttechniques(krigingplussupportvectorregression,radialbasisneuralnetwork,etc).Unfortunately,thisheuristicdoesnotcontrolthelevelofdiversity(thischallengeisleftforfutureresearch).InMSEGO,thenumberofpointsisthesameasthenumberofsurrogates.Intermsofimplementation: Wecouldvirtuallycreateasmanysurrogatesaswewant(easilymorethanthenumberofpointsrequestedperoptimizationcycle).Weadvisethatthecreatedsurrogatescomefromdifferenttechniques(asawaytogeneratediversity)andthatasubsetusedforoptimizationbeselectedaccordingtoaccuracy(hereweusePRESSRMS).Webelievethatthechoiceofsurrogatesmightimpactthenalresult.However,wethinkthataccuracyisanaturalpolicyforselection.Afterall,wedonotexpectpoorsurrogatestoperformwell(althoughtheymight). Wemightwanttoavoidpointsthatareclosertogetherthanagiventhresholdforsurrogatessuchaskrigingthatsufferfromillconditioninginthatcase. 5.4NumericalExperiments 5.4.1SetofSurrogatesTable 5-1 detailsthedifferentsurrogatesusedduringthisinvestigation.TheDACEtoolboxofLophavenetal.[ 48 ],thenativeneuralnetworksMATLABRtoolbox[ 94 ],thecodewrittenbyJekabsons[ 108 ],andthetoolboxdevelopedbyGunn[ 49 ]wereusedforkriging,theradialbasisneuralnetwork,andsupportvectorregressionalgorithms,respectively.TheSURROGATEStoolboxofViana[ 43 ]wasusedtoruntheShepard(adaptedfromSHEPPACK[ 109 ])andtheMSEGOalgorithms. 5.4.2AnalyticExamplesWeemployedthefollowingtwoanalyticbenchmarkproblemstocompareEGOandMSEGOwithouthavingtoworryaboutthecomputationalcostoftheobjectivefunction: 84

PAGE 85

Table5-1. SetofsurrogatesusedinthestudyofEGOassistedbymultiplesurrogates.(Appendix A reviewsthesesurrogatetechniques). SurrogatesDetailsProvideuncertaintyestimate? (1)krgKrigingmodel:constanttrendfunctionandGaus-siancorrelation.0i=p1=nv,and10)]TJ /F7 7.97 Tf 6.59 0 Td[(3i20,i=1,2,...,nvwereused.YES(2)rbnnRadialbasisneuralnetwork:goal=(0.5y)2andspread=1=3.NO(3)rbfRadialbasisfunction:Multiquadricbasisfunctionandspread=2.NO(4)shepLinearShepardmodel:SubroutineLSHEPfromSHEP-PACK[ 109 ]NO(5)svr-grbf-e-full(6)svr-grbf-e-short(7)svr-grbf-q(8)svr-poly-e-full(9)svr-poly-e-short(10)svr-poly-qSupportvectorregression:grbfandpolyindicatethekernelfunction(Gaussianandsecondorderpolyno-mial,respectively).eandqindicatethelossfunctions(eforinsensitiveandqforquadratic).`fullandshortrefertodifferentvaluesfortheregular-izationparameterCandfortheinsensitivity.FulladoptsC=1and=110)]TJ /F7 7.97 Tf 6.59 0 Td[(4,whileshortuses=y=p pandC=100maxjy+3yj,jy)]TJ /F24 10.909 Tf 10.91 0 Td[(3yj,whereyandyarethemeanvalueandthestandarddeviationofthefunctionvaluesatthedesigndata,respectively[ 95 ]. Sasenafunction(2variables,illustratedinFigure 5-6 )[ 110 ] y(x)=2+0.01)]TJ /F3 11.955 Tf 5.48 -9.69 Td[(x2)]TJ /F3 11.955 Tf 11.95 0 Td[(x212+(1)]TJ /F3 11.955 Tf 11.96 0 Td[(x1)2+2(2)]TJ /F3 11.955 Tf 11.95 0 Td[(x2)2+7sin(0.5x1)sin(0.7x1x2),0x15,0x25.(5) Hartmanfunctions(3and6variables)[ 96 ] y(x)=)]TJ /F4 7.97 Tf 15.76 11.63 Td[(qPi=1aiexp )]TJ /F4 7.97 Tf 14.74 11.36 Td[(mPj=1bij(xj)]TJ /F3 11.955 Tf 11.95 0 Td[(dij)2!,0xj1,j=1,2,...,m.(5)Weusetwoinstances:Hartman3,with3variablesandHartman6with6variables.Forbothq=4anda=1.01.23.03.2.OtherparametersaregiveninTable 5-2 .Toaverageouttheinuenceoftheinitialdataset,werepeattheexperimentswith100differentLatinhypercubedesigns[ 87 88 ].Theexperimentaldesignsarecreatedby 85

PAGE 86

Figure5-6. IllustrationoftheSasenafunction. Table5-2. ParametersusedintheHartmanfunction. FunctionParameters Hartman3B=26643.010.030.00.110.035.03.010.030.00.110.035.03775D=26640.36890.11700.26730.46990.43870.74700.10910.87320.55470.038150.57430.88283775Hartman6B=266410.03.017.03.51.78.00.0510.017.00.18.014.03.03.51.710.017.08.017.08.00.0510.00.114.03775D=26640.13120.16960.55690.01240.82830.58860.23290.41350.83070.37360.10040.99910.23480.14510.35220.28830.30470.66500.40470.88280.87320.57430.10910.03813775 theMATLABRLatinhypercubefunctionlhsdesign[ 94 ],setwiththemaxminoptionwith1,000iterations.WestartsamplingSasena,Hartman3,andHartman6with12,20,and56points,respectively.Then,weletEGOrunforsix,ten,andfourteencyclesforSasena,Hartman3,andHartman6functions,respectively.WerunMSEGOalgorithmwithveandtensurrogates(withoneofthembeingkriging).Giventheexperimentaldesign,weselectthesurrogatesthatwillassistkrigingbasedonPRESSRMS.Wepairkriging 86

PAGE 87

withthesurrogateswithsmallestPRESSRMSintheset.Ineachcycle,weaddatmostthesamenumberofpointsasthenumberofsurrogates(thatisve,ortenpointspercyclebutavoidingrepeatedpoints)untilthemaximumnumberofoptimizationcyclesisreached(e.g.,sixandtenfortheSasenaandtheHartman3functions,respectively).FulldetailsaregiveninTable 5-3 Table5-3. EGOsetupfortheanalytictestproblems.SurrogateswithsmallPRESSRMSarechosentopairwithkriging. SetupSasenaHartman3Hartman6 Pointsintheinitialexperimentaldesign122056Maximumnumberofoptimizationcycles61014NumberoffunctionevaluationsrunningEGOwithkriging(1pointpercycle)61014MaximumnumberoffunctionevaluationsrunningEGOwith5surrogates(5pointspercycle)305070MaximumnumberoffunctionevaluationsrunningEGOwith10surrogates(10pointspercycle)60100140 5.4.3EngineeringExample:TorqueArmOptimizationThisexamplewasoriginallypresentedbyBennettandBotkin[ 111 ].Itconsistsofthedesignofapiececalledtorquearm.Themodel,illustratedinFigure 5-7A ,isunderahorizontalandverticalload(Fx=)]TJ /F5 11.955 Tf 9.3 0 Td[(3000NandFy=5000N,respectively)transmittedfromashaftattherighthole,whiletheleftholeisxed.ThetorquearmconsistsofamaterialwithYoung'smodulusE=200GPaandPoisson'sratio=0.29.ThegoalisminimizingthemassmofthetorquearmsuchthatthemaximumvonMissesstressdoesnotexceedYield=250MPa.ThelinearstressanalysisisdonebyusingANSYS[ 112 ].Sixdesignvariablesaredenedtomodifytheinitialshape.Figure 5-7B andTable 5-4 showthedesignvariablesandtheirlowerandupperbounds,respectively.Theoptimizationproblemisformallydenedas 87

PAGE 88

A BFigure5-7. Baselinedesignofthetorquearm.A)Fixeddimensions,boundaryconditionsandloading(thetorquearmthicknessisxedat0.01m).B)Designvariablesandinitialvalues.Themassofthebaselinedesignismbaseline=2.28KgandthemaximumvonMissesstressisbaselinemax=167MPa. Table5-4. Rangeofthedesignvariables(meters). Designvariablex1x2x3x4x5x6 Lowerbound0.050.040.100.010.200.01Upperbound0.060.050.140.040.300.03 minimizem(x)=m(x) mbaseline,suchthatG(x)=max(x))]TJ /F8 11.955 Tf 11.96 0 Td[(Yield baselinemax,Yield=250MPa,xLixixUi,i=1,2,...,6,(5) wherethelowerandupperboundsforeachdesignvariable,xLiandxUi,respectively,aredenedinTable 5-4 .Wehavenormalizedtheconstrainedbyusingthebaselinedesignasreferencesothatwereducetheimpactoftheorderofmagnitudeoftheresponsesintheoptimizationtask(butonecouldcertainlyuseYield). 88

PAGE 89

Thisisaconstrainedoptimizationproblem;butbothEGOandMSEGOwereproposedtohandleunconstrainedoptimization.Theinterestedreadermightndintheliterature[ 69 83 84 113 ]sequentialsamplingalgorithmsthathandleconstrainedoptimization(whenobjectivefunctionand/orconstraintsareexpensive).Here,wejustwanttoillustratetheuseofthemultiplesurrogateefcientglobaloptimization(MSEGO)algorithm,soweoptedforincorporatingtheconstraintviapenalty.Thus,theoptimizationproblemsolvedbyEGOandMSEGOisthefollowing minimizeJ(x)=m(x)+fG(x),where8>><>>:f>0,ifmax(x)>Yield,f=0,otherwise,xLixixUi,i=1,2,...,6.(5)Westartbysamplingthedesignspacewith56datapoints.Wetakethebaselinedesignandaddother55pointsusingaLatinhypercubecreatedbytheMATLABRLatinhypercubefunctionlhsdesign[ 94 ],setwiththemaxminoptionwith1,000iterations.WecreatesurrogatesforthefunctionalJ(x)andthatiswhatisusedbyEGOandMSEGOintheoptimizationtask.WecompareEGOrunningwithkrigingversusMSEGOrunningwiththetensurrogatesfromTable 5-1 after14optimizationcycles.Table 5-5 summarizesthesetupusedintheengineeringexample. 5.4.4OptimizingtheExpectedImprovementTheefcientglobaloptimization(EGO)algorithmissequentialinnatureandinvolvesoptimizingEq. 5 ateachcycle.WehavechosenusingthepopulationbasedglobaloptimizationalgorithmcalleddifferentialevolutiontomaximizeEq. 5 .DifferentialEvolution(DE)[ 114 115 ]wasproposedtosolveunconstrainedglobaloptimizationproblemsofcontinuousvariables.Likemostpopulationbasedalgorithms,inDE,candidatesolutionsaretakenfromthepopulationofindividualsand 89

PAGE 90

Table5-5. Optimizationsetupfortheengineeringexample. DetailsEGOMSEGO Surrogates(Table 5-1 givesfurtherdetails)krgAllsurrogatesfromTable 5-1 (allusingkrguncertainty)Numberofpointsintheinitialdataset5656Numberofoptimizationcycles1414Numberofniteelementanalysispercycle110Maximumnumberofniteelementanalysis14140 recombinedtoevolvethepopulationtothenextDEiteration.AteachDEiteration,newcandidatesolutionsaregeneratedbythecombinationofindividualschosenfromthecurrentpopulation(thisisalsoreferredtoasmutationintheDEliterature).Thenewcandidatesolutionsaremixedwithapredeterminedtargetvector(operationcalledrecombination).Finally,eachoutcomeoftherecombinationisacceptedforthenextDEiterationifandonlyifityieldsareductionintheobjectivefunction(thislastoperatorisreferredtoasselection).Thefreecompanioncodeof[ 115 ]wasusedtoexecutetheDEalgorithmwiththefollowingspecications: Initialpopulation:apopulationof10nv(wherenvisthenumberofdesignvariables)candidatesolutionsisrandomlyspreadoverthedesignspace. Numberofiterations:50. DE-stepsize:0.8. Crossoverprobability:0.8. Strategy:DE/rand/1/bin,whichmeansthatindividualsrandomlychosenfromthepopulationwillundergocrossoverusingtheindependentbinomialexperimentsscheme[ 115 ].ToincreasethechanceofsuccessintheDEoptimization,werunitfourtimesandtakethebestsolution(intermsofexpectedimprovement)(successofmultiple 90

PAGE 91

independentoptimizationwasstudiedbySchutteetal.[ 116 ]).ThispointisthenusedtoevaluatetheobjectivefunctionEGOistryingtooptimizeandlateronupdatekriging. 5.5ResultsandDiscussion 5.5.1AnalyticExamplesWerstdiscussourstrategyforimportinguncertaintyestimates.Figure 5-8 showsboxplots(Appendix B describesboxplots)ofthecorrelationbetweenthesquarerootofkrigingpredictionvarianceandtheabsoluteerrorofallthesurrogates(forall100experimentaldesigns).Therearetwoimportantobservations.First,thecorrelationbetweenthesquarerootofthekrigingpredictionvarianceandthekrigingabsoluteerrorismuchsmallerthansuggestedbytheearlierexamples(Figures 5-2 and 5-3 ).ForSasenaandHartman3,thevaluesaremostlybetween0.25and0.5.Inhigherdimensions,suchasforHartman6,thecorrelationsareevensmaller.Theotherobservationisthatthecorrelationsfortheimportedcasesareoverallcomparablewithwhatweseewithkrigingforatleastthreeothersurrogatesoftheset. A B CFigure5-8. Boxplotsofthecorrelationbetweenerroranduncertaintyestimates=corr(je(x)j,s(x))(for100experimentaldesigns).A)Sasena,12points.B)Hartman3,20points.C)Hartman6,56points.Allsurrogatesusethekrigingerrorestimate.Appendix B explainsboxplots. 91

PAGE 92

Wealsocheckedhowwellcrossvalidationselectsthesurrogatestobepairedwithkriging.Figure 5-9 givesboxplotsofbothPRESSRMSandeRMSforallsurrogatesforalltestproblems.ForalltestfunctionsthereisatleastonesurrogatethatisasgoodaskrigingintermsofeRMS.FortheSasenafunction,Figure 5-9A showsthatthesupportvectorregressionmodelswiththepolynomialkernel(variationsofsvr-poly)mayoutperformkrigingintermsoftheeRMS.ForHartman3,Figure 5-9B illustratesthatkrigingiscomparabletotheradialbasisneuralnetwork(rbnn).ForHartman6,Figure 5-9C showsthatmostofthesurrogatesareequallygood,exceptforsvr-grbf-e-fullandshepthatarejustslightlylessaccuratethantheothersurrogates.Tosomeextent,theselectionofthesurrogatesbasedonPRESSRMSisalmostthesameasthatbasedoneRMS.Table 5-6 showshowthesurrogatesrankaccordingtooverallperformance.TheagreementbetweentheranksgivenbyeRMS(errorestimatebasedontestpoints)andPRESSRMS(errorestimatebasedondataset)forHartman3andHartman6isbetterthanforSasena.TherankaccordingtoPRESSRMSandeRMSisthesameforHartman6.Overall,mostofthetimethesurrogateschosentoassistkrigingwillbethemostaccurateones.Figure 5-10 showsthemedianoftheoptimizationresultsoutofthe100experimentaldesignsforthetraditionalefcientglobaloptimization(EGO)algorithm(runningonlywithkriging)andMSEGO(EGOassistedbymultiplesurrogates).Usingmultiplesimulationspercycleacceleratesconvergence.Infact,thereissubstantialimprovementparticularlyfortheHartman6function.There,usingvesurrogatesfortwocyclesprovidesthesameimprovementasusingkrigingalonefor14cycles.Forallcases,themorepointsaddedpercycle(i.e.,moresurrogates),thefastertheconvergence.Unfortunately,theconvergenceratedoesnotscaleupwiththenumberofpoints(orsurrogates,forthatmatter).Thatis,theresultsusing5and10surrogatesareverycomparable.Thisindicatesthatamongthesurrogates,probablyonlyfewarecontributingineachcycle.BecauseitishardtodetectaprioriwhichsurrogatesareimportantforMSEGO(after 92

PAGE 93

A B CFigure5-9. BoxplotsofPRESSRMSandeRMSofsurrogatesforthetestproblems(over100experimentaldesigns).A)Sasena,12points.B)Hartman3,20points.C)Hartman6,56points.Dataobtainedwiththeinitialexperimentaldesigns(12,20,and56pointsforSasena,Hartman3,andHartman6,respectively).ForbothSasenaandHartman3,eRMSrangesfrom8%to52%oftheresponserange.ForHartman6,thevariationisfrom8.5%to70%.Intermsofprediction,othersurrogatesmightbejustasgoodaskriging.Appendix B explainsboxplots. all,thischoiceisveryproblemdependent),weadviselimitingthenumberofpointspercycleaccordingtothecomputationalcapabilities.Figure 5-11 showsboxplotsoftheoptimizationresultsoutofthe100experimentaldesignsforthetraditionalEGO(efcientglobaloptimizationalgorithmrunningonlywithkriging)andMSEGO(EGOassistedbyninesurrogates)iterationbyiteration. 93

PAGE 94

Table5-6. Rankingofthesurrogatesaccordingtomedianvalues(over100experimentaldesigns)ofPRESSRMSandeRMS.PRESSRMSsatisfactorilyranksthesurrogates. TestproblemRankPRESSRMSeRMS Sasena,12points1svr-poly-qsvr-poly-e-full2svr-poly-e-fullsvr-poly-q3svr-poly-e-shortrbnn4svr-grbf-qsvr-poly-e-short5svr-grbf-e-shortsvr-grbf-e-short6krgshep7shepkrg8rbnnsvr-grbf-q9rbfsvr-grbf-e-full10svr-grbf-e-fullrbfHartman3,20points1rbnnrbnn2krgkrg3svr-poly-e-shortsvr-poly-q4svr-grbf-qsvr-poly-e-short5svr-poly-qsvr-grbf-q6svr-grbf-e-shortshep7shepsvr-poly-e-full8svr-poly-e-fullsvr-grbf-e-short9rbfrbf10svr-grbf-e-fullsvr-grbf-e-fullHartman6,56points1rbnnrbnn2svr-poly-qsvr-poly-q3svr-poly-e-shortsvr-poly-e-short4krgkrg5svr-grbf-qsvr-grbf-q6svr-grbf-e-shortsvr-grbf-e-short7svr-poly-e-fullsvr-poly-e-full8rbfrbf9shepshep10svr-grbf-e-fullsvr-grbf-e-full Cyclenumber0(initialdataset)locatestheinitialsamplewithrespecttotherangeofvaluesoftheobjectivefunction(whichactuallyreectshowmuchroomforimprovementthereis).ForSasena,thatmeansanaverageof6%(withmaximumofabout17%).ForHartman6,theaveragerelativevalueoftheinitialbestsampleis40%(with78%maximum).Abenetofusingofmultiplesurrogatesisthereduceddispersionofthe 94

PAGE 95

A B CFigure5-10. Median(over100experimentaldesigns)oftheoptimizationresultsasafunctionofthenumberofcycles.A)Sasena,12points.B)Hartman3,20points.C)Hartman6,56points.Morepointspercycle(moresurrogates)speedsupperformance. results.Duetoparallelcomputation,intermsofwallclocktime,MSEGOtakesonlyafractionofthetimeatraditionalEGOimplementationwouldneed.Usingmultiplepointspercycleismostlyattractivewhenthemainconcernisthenumberofcyclesratherthanthetotalnumberofsimulations.However,itisinterestingtolookatwhatpenaltyispaidforacceleratedtimeconvergencebyhavingtorunmoresimulations.Forthatpurpose,wecomparethealgorithmsforaxednumberoftotalsimulationsratherthanaxednumberofcycles.Figure 5-12 showsthemedianover100experimentaldesignsoftheoptimizationresultswithrespecttonumberoffunctionevaluationsforEGOandMSEGO.ForSasena,thereisasubstantialpenaltyinthenumberoffunctionevaluationsandforHartman3thereisasignicantpenaltyearlyon,butbetternalconvergence.ForHartman6,themultiplesurrogateapproachhasadistinctadvantage.Ingeneral,wedidnotseeaclearadvantageofanyofthemethods 95

PAGE 96

A B C D E FFigure5-11. Boxplot(over100experimentaldesigns)oftheoptimizationresultsasafunctionofthenumberofcycles.A)Sasena:EGO.B)Sasena:MSEGO.C)Hartman3,EGO.D)Hartman3:MSEGO.E)Hartman6:EGO.F)Hartman6:MSEGO.ResultsofMSEGOareobtainedwith10surrogates.Again,MSEGOachievesbetterresultsthanthetraditionalEGO.Appendix B explainsboxplots. abovetheother.Perhapsinahighdimensionalspace,wheremoreexplorationisneeded,thediversityofthesetofsurrogatesmightpresentsomeadvantage.WebelievethatthedifferenceinperformancebetweentraditionalEGOandMSEGOisduetoacombinationoffactors.OnefactoristhatsincethetraditionalEGOusesonlykriging,itissensitivetoexperimentaldesignsforwhichthekrigingmodelisapoorsurrogate.Incontrast,MSEGOwouldhaveothersurrogatestobalancethat. 96

PAGE 97

A B CFigure5-12. Median(over100experimentaldesigns)oftheoptimizationresultswithrespecttothenumberofevaluations.A)Sasena,12points.B)Hartman3,20points.C)Hartman6,56points.Itisnotclearifmultiplepointspercycleimplyinmorefunctionevaluationsforagivenimprovement. Figure 5-13 illustratesthiseffectwiththeexperimentaldesignforwhichEGOhadthepoorestperformance(experimentaldesign#8outof100).ThePRESSRMSfortheninesurrogatesrangedfrom8%to17%,withkrigingbeingtheleastaccuratesurrogate(threesurrogateshadPRESSRMSbelow10%).AnotherreasonforthesuperiorityofMSEGOisthatdiversityofsurrogatestendstoimprovebothexplorationandexploitation.Figure 5-14 showstwointersitedistancemeasuresofthenaldataset(wecomputedthedistancebetweenallpossiblepairs).Themedianintersitedistance(left-handplots)illustrateshowmuchexplorationthereis.Inalltestproblems,MSEGOtendstospreadthepointsoverthedesignspacemorethankrigingalone.Ontheotherhand,MSEGOdoesnotpreventpointsfromclustering.Thismakestheminimumintersitedistance(right-handplotsofFigure 5-14 forSasenaandHartman3)smallercomparedtotraditionalEGO(althoughthistendencyisreversed 97

PAGE 98

forHartman6).Sinceweareaimingtoreducethenumberofcyclesneededtoachieveacertainnormalizeddistancetotheglobaloptimum,weseethatthebenetsoffasterexplorationcounterbalancetheeffectsofclustering.Ontopofthat,Figure 5-5 illustratesthathavingpointsclosetoeachothermaynotnecessarilybebad. A BFigure5-13. Contourplotsofforexperimentaldesign#8ofSasenafunction.A)Krigingresults:6extrapoints.B)MSEGOresults:54extrapoints.Forthisexperimentaldesign,eRMSrangedfrom8%to17%oftheresponserange.Kriging,theleastaccuratesurrogate,returnsy=4.33after6cycles.MSEGOofferedaninsuranceagainstit,returningy=)]TJ /F5 11.955 Tf 9.3 0 Td[(1.45(globaloptimumisy=)]TJ /F5 11.955 Tf 9.3 0 Td[(1.46). 5.5.2EngineeringExample:TorqueArmOptimizationFigure 5-15 illustratesthedatausedtobuildthesurrogatemodels.Figure 5-15A showsthemaximumvonMissesstressversusmassplotforthe56trainingpoints.Onlytwelveoutofthe56pointsarefeasible(maximumvonMissesbellowallowablevalue).Thebestinitialsample(whichisthelightestfeasibleone)is6%lighterthanthebaselinedesign.Figure 5-15B illustrateshowwedeterminedthepenaltyfactorbylookingattheinclinationofthelinethatpassesthroughthebestinitialsampleandtherstpointtotherightoftheconstraintviolation.Thatlinehasinclination)]TJ /F5 11.955 Tf 9.3 0 Td[(0.45.Weconservativelychosethepenaltyfactorf=0.5.TheobjectivefunctionJ(x)islinearlypenalizedwith 98

PAGE 99

A B CFigure5-14. Boxplotsofintersitedistancemeasures(over100experimentaldesigns).A)Sasena(60pointsadded).B)Hartman3(100pointsadded).C)Hartman6(140pointsadded).Appendix B explainsboxplots. theconstraintviolationasshowninFigure 5-15C (recollectthattheconstraintisviolatedwhenG(x)>0).Next,webrieyexaminethesurrogatescreatedbasedontheinitial56points.Table 5-7 presentsthePRESSRMSvaluesandtheorderinwhichthesurrogatesrankaccordingtoPRESSRMS.Foursurrogates(svr-poly,svr-poly-e-full,svr-grbf-q,andsvr-grbf-e-full)canbeconsideredequallygoodatthetop(withPRESSRMSbellow10%oftheresponserange),otherve(krg,rbf,shep,svr-poly-e-short,andsvr-grbf-e-short)arereasonablygood(withPRESSRMSabout15%oftheresponserange),andrbnnseemspoorlytted(withPRESSRMSontheorderof365%oftheresponserange).Nevertheless,weran 99

PAGE 100

A B CFigure5-15. Initialdatasetforthetorquearmexample.A)56datapoints(initialbestsamplehasmassof2.15KgandmaximumvonMissesstressof234MPa).B)Normalizationoftheinitial56datapointsandzoomedregionusedtocalculatethepenaltyfactor.C)ObjectivefunctionJ(x),Eq. 5 ,usedforttingsurrogates.Initialbestsample(lightestfeasible)is6%lighterthanthebaselinedesign.Thelinebetweentheinitialbestsampleandtherstpointtotherightoftheconstraintviolationhasinclination)]TJ /F5 11.955 Tf 9.3 0 Td[(0.45,asshowninFigure 5-15B .Weconservativelypenalizedtheobjectivefunction,Eq. 5 ,withconstraintviolationbyafactoroff=0.5. MSEGObothwithandwithoutrbnntoseewhethertherbnnoverallaccuracyrendersitinvalidforoptimization.Figure 5-16 showstheoptimizationresultsobtainedbybothEGO(efcientglobaloptimizationalgorithmrunningonlywithkriging)andMSEGO(EGOrunningwithnineandalltensurrogates).Figure 5-16A conrmswhatwelearnedfromprevioussection.Thatis,indeedMSEGOspeedsuptheconvergencewhencomparedtotraditionalEGO.Italsoshowsthatrbnnwasvitalfortheoverallperformance,eventhoughmuch 100

PAGE 101

Table5-7. PRESSRMSanalysisforthetorquearmexample.InparenthesisthepercentPRESSRMSrelativetothesampledobjectivefunction(whichrangesfrom0.9to4.33).Fortherank,thesmallerthePRESSRMSvaluethebetter. SurrogatePRESSRMSRank krg0.597(12%)5rbf0.663(13%)6rbnn18.631(365%)10shep0.756(15%)7svr-grbf-e-full0.406(8%)4svr-grbf-e-short0.857(17%)9svr-grbf-q0.406(8%)3svr-poly-e-full0.404(8%)2svr-poly-e-short0.838(16%)8svr-poly-q0.404(8%)1 lessaccuratethantheothersurrogatesusedinMSEGO.Surprisingly,rbnnplayedanimportantsupportrole.Table 5-8 showstheidentityofthesurrogatesthatbroughtmassreductionalongtheoptimization.Innoneofthecases,themassreductionwasobtainedatapointsuggestedbyrbnn.Thismeansthatrbnnwasessentiallyusedinexplorationofthedesignspace.Figure 5-16B showsthat,attheendoffourteenoptimizationcyclesthestressconstrainwasnotactive.ThismeansthatbothEGOandMSEGO(bothversions)didnotndtheglobalsolutionoftheoptimizationproblemstatedinEq. 5 .TheoptimaldesigncongurationisexpectedtohavethemaximumvonMissesstressequaltheallowablevalue.Figure 5-16B indicatesthatthechoiceofthepenaltyfactorasf=0.5wasratherconservative.Basedonthat,wedecidedtorunMSEGOforvemorecyclesbutnowwithf=0.25.Unfortunately,wedidnotndanyimprovement.Wesurmisedthatthisisanindicationthatpossibleimprovementsarerestrictedtoasmallareaofthedesignspace.Then,weswitchedtheoptimizationalgorithmtotheNelder-MeadSimplexofferedbytheMATLABRfunctionfminsearchandusedittosolveEq. 5 withf=0.25.Figure 5-16C illustratesallveresultingdesignsfromthisexercise.Wecanobservethe6%improvementintheinitialsamplecomparedtothebaseline.Then,weseethe7%and11%improvementovertheinitialbestsample 101

PAGE 102

obtainedbyEGOandMSEGO,respectively.Finally,wehaveanextra7%improvementoverMSEGOobtainedwithfminsearch. A B CFigure5-16. ComparisonbetweenEGOandMSEGOforthetorquearmproblem.A)Optimizationhistoryformass.B)OptimizationhistoryformaximumvonMissesstress.C)Comparisonintheresponsespace.AlthoughMSEGOoutperformsEGO,bothfailedinreachingtheglobalsolutionofEq. 5 .Thisisnotasurpriseconsideringthatbothareglobaloptimizationmethods.Nevertheless,EGOwith14extrafunctionevaluations(14cycles)isnotasgoodasMSEGOwith20extrafunctionevaluations(2cycles).ThenativeMATLABRfunctionfminsearchisabletonishtheoptimizationlocally. Table5-8. Identityofsurrogatethatsuggesteddesignswithreductioninmass. OptimizationcycleMSEGOwithoutrbnnMSEGOwithrbnn 2svr-grbf-e-shortsvr-grbf-e-full3krgsvr-grbf-e-full10svr-grbf-e-short Figure 5-16A showsthat,asearlyasinthesecondcycle,MSEGOoutperformedthedesignfoundbytraditionalEGOoverthefourteencycles.Intermsofnumberoffunctionevaluations(meaningniteelementanalysis),MSEGOrequiredeighteenextrapoints. 102

PAGE 103

WealsocheckedwhathappensifEGOrunsfor140cycles.Figure 5-17 illustratestheresults.WecanseethatEGOwouldoutperformMSEGOwith19functionevaluations.Nevertheless,wecanonlysaythatforthisparticularproblem,withthisparticularinitialsetofpointsandoptimizationsetup(changesonanyofthesewouldcertainlyimpactbothEGOandMSEGOandresultscouldbecompletelydifferent). Figure5-17. Optimizationhistoryformasswithrespecttothefunctionevaluationsforthetorquearmdesign.Ittakes19cyclesforEGOtooutperformMSEGO. Finally,Figure 5-18 illustratesdifferentdesigncongurationsofthetorquearmobtainedinthisexercise.Apparently,mostofthemassistakenawayreducingtheouterradiusofbothgrips(designvariablesx1andx2).Then,thegeometryoftheinsidecavitycontrolsthemaximumvonMissesstress. 5.6SummaryWeproposedamultiplesurrogateefcientglobaloptimization(MSEGO)algorithm,whichusesmultiplesurrogatestoaddmultipledesignsitesateachcycleoftheefcientglobaloptimization(EGO)algorithm.MSEGOisacheapalternativetoaddingmultiplepointsineachEGOcyclebasedonsinglekrigingmodel(computationallyverydifcult).AlthoughMSEGOisfeasiblewithmultipleinstancesofkriging,wechosetoillustrateitwithadiversesetofsurrogates.Becausesomeofthesesurrogatesdonotprovideuncertaintystructure,weproposeenablingthemtoestimatetheexpectedimprovementbyimportinguncertaintyestimatesfromkriging.Threealgebraic,oneengineeringexamples,andtensurrogateswereusedtostudyhowwellimportinguncertaintyestimatesworksandtocomparethetraditional 103

PAGE 104

A B C D EFigure5-18. Torquearmdesigns(geometry,massandmaximumvonMissesstress).A)Baselinedesign:massof2.28KgandmaximumvonMissesstressof167MPa.B)Bestinitialdesign:massof2.15KgandmaximumvonMissesstressof234MPa.C)DesignobtainedwithEGO:massof1.99KgandmaximumvonMissesstressof239MPa.D)DesignobtainedwithMSEGO:massof1.91KgandmaximumvonMissesstressof215MPa.E)Designobtainedwithfminsearch(renementofMSEGOdesign):massof1.77KgandmaximumvonMissesstressof250MPa. implementationofEGO(runningwithkrigingalone)withMSEGO(EGOassistedbyasetofsurrogates).Wefoundthat 1. theimporteduncertaintystructureperformedreasonablywellandallowednon-krigingsurrogatestobeusedintheEGOalgorithm; 2. MSEGOreducedsubstantiallythenumberofcyclesrequiredforconvergence(MSEGObenetsfromthediversesetofsurrogates,whichreducestherisksassociatedwithpoorlyttedsurrogates); 3. thepenaltyintermsoftotalnumberoffunctionevaluationswassubstantialforoneofthethreeanalyticexamples,butforanotherMSEGOprovidedasubstantialreductioninthenumberoffunctionevaluationsandaverylargereductioninthenumberofcycles; 4. theengineeringexampleshowedthatthereisnoreasontodiscardlessaccuratesurrogates(providedthecomputationalresourcestorunacertainnumberofadditionalsimulationspercycle).Althoughaccuracyiscertaintydesirable,diversityalsoplaysanimportantroleinglobaloptimization.Here,thereisnopenaltyintermsoftotalnumberoffunctionevaluationsforshortcycleoptimization(uptotwo 104

PAGE 105

optimizationcycles,i.e.,20functionevaluations).However,thereissomepenaltyonthelongrun(consideringthe140functionevaluations). 105

PAGE 106

CHAPTER6SURROGATE-BASEDOPTIMIZATIONWITHPARALLELSIMULATIONSUSINGTHEPROBABILITYOFIMPROVEMENTInChapter 5 weshowedthatoptimization-basedadaptivesamplingalgorithmscantakeadvangateofmultiplesurrogatesforgeneratingmultiplecandidatesolutionsperoptimizationcycle.Nevertheless,wehavefoundthattheapproachhasthedrawbackoflinkingthenumberofpointswiththenumberofsurrogates(makingitlessconvinientasthenumberofpointsincrease).Inthischapter,wepursuethealternativeofselectingthemultiplepointsusingasinglesurrogate.Wechosedoingsobymaximizingtheprobabilityofimprovement(thatis,theprobabilityofbeingbelowatargetvalue).Weproposeacheaptoevaluateapproximationoftheprobabilityofimprovement(basedontheassumptionthatthecorrelationbetweenpointscanbeneglected).Theapproachistestedonthreeanalyticexamples.Fortheseexampleswecompareourapproachwithtraditionalsequentialoptimizationbasedonkriging.Wefoundthatindeedourapproachwasabletodeliverbetterresultsinafractionoftheoptimizationcyclesneededbythetraditionalkrigingimplementation.Theresultsweredivulgedatthe13thAIAA/ISSMOMultidisciplinaryAnalysisandOptimizationConference(VianaandHaftka[ 40 ]). 6.1Background:SinglePointProbabilityofImprovementAfterconstructingthekrigingsurrogatemodel,theefcientglobaloptimization(EGO)algorithm[ 14 ]canalsoaddpointstothedatasetbymaximizingtheprobabilityofimprovinguponatargetvalueyT(whichisadesigner'schoice).Consideringthesurrogatemean^y(x)andthevariances2(x)atagivenpointxofthedesignspace(whichfullycharacterizeanormaldistribution),theprobabilityofimprovementis PI(x)=Pr(Y(x)
PAGE 107

Afteraddingthenewpointtotheexistingdataset,thekrigingmodelisupdated(usuallywithoutthecostlyupdateofthecorrelationparameters).Figure 6-1 illustratescyclesofEGOwhentheprobabilityofimprovementissetwithdifferenttargets.Figure 6-1A illustratestheinitialkrigingmodelandthecorrespondingprobabilityofimprovementconsideringatargetyT=)]TJ /F5 11.955 Tf 9.3 0 Td[(4.6(25%belowthepresentbestsample).MaximizationofPI(x)suggestsincludingx=0.63tothedataset.Thiscycledrivesoptimizationtowardsexploitationofsurrogate.However,iftheismoreambitioustargetyT=)]TJ /F5 11.955 Tf 9.29 0 Td[(5.5wasused,maximizationofPI(x)suggestsx=0.21(Figure 6-1B ).Thisisanexplorationcycle1. A BFigure6-1. Cycleoftheefcientglobaloptimization(EGO)algorithmusingtheprobabilityofimprovement.A)MaximizationoftheprobabilityofimprovementwithyT=)]TJ /F5 11.955 Tf 9.3 0 Td[(4.6.B)MaximizationoftheprobabilityofimprovementwithyT=)]TJ /F5 11.955 Tf 9.3 0 Td[(5.5.ThepointsuggestedbythemaximizationoftheprobabilityofimprovementdependsonthetargetyT.ForyT=)]TJ /F5 11.955 Tf 9.3 0 Td[(4.6andyT=)]TJ /F5 11.955 Tf 9.3 0 Td[(5.5,thesuggestedpointsarex=0.63andx=0.21,respectively. 6.2OptimizingtheApproximatedMultipointProbabilityofImprovementWhennnewmultiplesitesXnewcanbeadded,weareinterestedintheprobabilityofhavingatleastoneofthepredictionsitesbelowthetargetyT[ 72 ] 1Thechoiceofthetargethasaclearimpactontheselectedpoint.Furtherreadingonsettingthetargetforoptimizationcanbefoundin[ 70 72 117 ] 107

PAGE 108

PI(X)=Pr(9Ynewi2Ynew:YnewiyT), (6) whereXannndvmatrixthatdenesthenpoints,Ynewisann1vectorofthevaluesofthefunctionevaluatedatthenewpredictionsitesandTisann1vectorwithallelementsequaltoyT.WecancomputetherighthandsideofEq. 6 usingalgorithmsforevaluatingGaussiancumulativeprobabilitydistributionsinhighdimensions[ 118 ]2.Nevertheless,ourexperienceisthatevaluatingEq. 6 becomesexpensiveenoughtoinhibititsuseinoptimization(particularlyforlargenumberofnewpredictionsites).Ontheotherhand,ifweassumethatthemultiplesitesXnewarenothighlycorrelated(inpractice,theoptimizationhastobesuchthatpointsarefarenough),thecorrelationbetweenpointscanbeneglected.Then,wecanapproximateEq. 6 by PI(X)=Pr(9Ynewi2Ynew:Ynewi
PAGE 109

matricesneededin 6 [ 72 ],and(ii)theintegralrequiredinEq. 6 isreadilyavailableinaclosedform(thatis,itdoesnotinvolveexpensivenumericalevaluation,suchasviaMonteCarlointegrationforexample).Whenweselectpointsbymaximizing(justonce)Eq. 6 ,wesolveanoptimizationproblemwiththesamenumberofvariablesoftheoriginalsurrogate(thatis,thetotalnumberofvariablesinthemaximizationofEq. 6 isndv).Ontheotherhand,maximizationof 6 increasesthenumberofvariablesoftheoptimizationproblembythenumberofpointspercycle(thatis,thetotalnumberofvariablesisnndv).Obviously,thatmakesitaharderproblemtosolve.Onewaytoovercomethislimitationistoconstraintthesearchsuchthatpointsareapartfromatleasta.Thus,thelocationofthenewdatapointsisobtainedbysolvingthefollowingoptimizationproblem maximizePI(X)=1)]TJ /F6 11.955 Tf 11.96 8.97 Td[(Qni=1(1)]TJ /F5 11.955 Tf 11.96 0 Td[(Pr(Ynewi,d(xi,xj)=q Pndvk=1(xik)]TJ /F3 11.955 Tf 11.96 0 Td[(xjk)2,(6)Fortunately,theoptimizationcanbefurthersimplied.SolvingEq. 6 isequivalenttosequentiallymaximizeEq. 6 suchthatpointsareapartfromatleasta3.Thatis,werepeatthemaximizationofEq. 6 ntimes.Everytimewestorethecandidatesolutionandimposeaconstraintsuchthatthenextpointsareatleastfromallpreviouslyfoundpoints.Figure 6-2 illustratestheideawiththemaximizationofthemultipointprobabilityusingthesurrogateofFigure 6-1 .SequentialmaximizationofEq. 6 suggestsvenewcandidatesolutions.BycomparingFigures 6-2A and 6-2B ,wenoticethatoptimizingthe 3Thechoiceofisbyandlargeadesigner'schoice.Herewehaveused10%ofthediagonalthatcutstheoriginandthediametricallyopposedcornerofthedesignspace 109

PAGE 110

multipointprobabilityofimprovementusingbothyT=)]TJ /F5 11.955 Tf 9.3 0 Td[(4.60andyT=)]TJ /F5 11.955 Tf 9.3 0 Td[(5.50populatestheregionsaroundx=0.2andx=0.6(whenyT=)]TJ /F5 11.955 Tf 9.29 0 Td[(4.60thereisalsoapointatx=0.74).Thisindicatesthatthemultipointprobabilityofimprovementmightbeslightlylesssensitivetothetargetthanthesinglepointprobabilityofimprovement. A BFigure6-2. Cycleoftheefcientglobaloptimization(EGO)algorithmusingthemultipointprobabilityofimprovement.A)MaximizationofthemultipointprobabilityofimprovementwithyT=)]TJ /F5 11.955 Tf 9.3 0 Td[(4.6.B)MaximizationofthemultipointprobabilityofimprovementwithyT=)]TJ /F5 11.955 Tf 9.3 0 Td[(5.5.FivenewcandidatesolutionsaresuggestingbysequentiallymaximizeEq. 6 suchthatpointsareapartfromatleasta=0.05(equivalenttosolvingEq. 6 ). 6.3NumericalExperiments 6.3.1KrigingModelTable 6-1 detailsthesetupofthekrigingmodelusedduringthisinvestigation.TheDACEtoolboxofLophavenetal.[ 48 ]wasusedtorunthekrigingalgorithm.TheSURROGATEStoolboxofViana[ 43 ]wasusedforeasymanipulationofthesurrogateandforrunningtheefcientglobaloptimization(EGO)algorithm(usingsinglepointprobabilityofimprovementandtheapproximationoftheprobabilityofimprovement). 6.3.2AnalyticExamplesAstestproblems,weemployedthetwofollowinganalyticalbenchmarkproblems: Sasenafunction(2variables,illustratedFigure 5-6 )[ 110 ] 110

PAGE 111

Table6-1. Krigingmodelusedintheoptimizationwithprobabilityofimprovementstudy. KrigingcomponentDetails Regressionmodel(trend)Zeroorderpolynomialregressionmodel(constanttrendfunction).CorrelationmodelGaussiancorrelationwith0i=10and0i200,i=1,2,...,dforoptimizatinghyper-parameters. y(x)=2+0.01)]TJ /F3 11.955 Tf 5.48 -9.68 Td[(x2)]TJ /F3 11.955 Tf 11.95 0 Td[(x212+(1)]TJ /F3 11.955 Tf 11.96 0 Td[(x1)2+2(2)]TJ /F3 11.955 Tf 11.95 0 Td[(x2)2+7sin(0.5x1)sin(0.7x1x2),0x15,0x25.(6) Hartmanfunctions(3and6variables)[ 96 ] y(x)=)]TJ /F4 7.97 Tf 15.76 11.63 Td[(qPi=1aiexp )]TJ /F4 7.97 Tf 14.74 11.36 Td[(mPj=1bij(xj)]TJ /F3 11.955 Tf 11.95 0 Td[(dij)2!,0xj1,j=1,2,...,m.(6)Weusetwoinstances:Hartman3,with3variablesandHartman6with6variables.Forbothq=4anda=1.01.23.03.2.OtherparametersaregiveninTable 6-2 Table6-2. ParametersusedintheHartmanfunction. FunctionParameters Hartman3B=26643.010.030.00.110.035.03.010.030.00.110.035.03775D=26640.36890.11700.26730.46990.43870.74700.10910.87320.55470.038150.57430.88283775Hartman6B=266410.03.017.03.51.78.00.0510.017.00.18.014.03.03.51.710.017.08.017.08.00.0510.00.114.03775D=26640.13120.16960.55690.01240.82830.58860.23290.41350.83070.37360.10040.99910.23480.14510.35220.28830.30470.66500.40470.88280.87320.57430.10910.03813775 First,westudytheestimatesoftheprobabilityofimprovementusingonetwenty-pointexperimentaldesignoftheHartman3function.Wecompareaccuracyandcomputationtimeswhennnew=5,10,and20pointspercycleareaddedtothedataset.Wegenerate1,000setsofnnewpoints(randomlyselectedoverthedesignspacewithoutduplicating 111

PAGE 112

alreadysampledpoints).TheglobalprobabilityofimprovementistakenastheratioofthenumberofpointsbelowyTandthetotalnumberoftestedpoints(1,000nnew).Foreachset,wealsocomputethemultipointprobabilityofimprovementusingEqs. 6 and 6 andweusetheirmean(overthe1,000sets)astheestimatoroftheglobalprobabilityofimprovement.Next,wecheckhowgoodtheapproximation(Eq. 6 )isasanestimateoftheprobabilityofimprovement.WestartsamplingSasena,Hartman3,andHartman6with12,20,and56points,respectively.Thenwebuildanestimateoftheglobalprobabilityofimprovementoverthedesignspace.WeuseMonteCarlosimulationwith50,000setsofnnewpoints(nnew=1,5,and10,randomlyselectedoverthedesignspacewithoutduplicatingalreadysampledpoints).TheglobalprobabilityofimprovementistakenastheratioofthenumberofpointsbelowyTandthetotalnumberoftestedpoints(50,000nnew).ForeachentryofthesetwealsocomputetheprobabilityofimprovementusingEq. 6 andweuseitsmean(overthe50,000sets)astheestimatoroftheprobabilityofimprovement.Finally,weusetheprobabilitiesforoptimizationdrivensequentialsampling.WestartsamplingSasena,Hartman3,andHartman6with12,20,and56points,respectively.Inthesequence,weletEGOrunforsix,ten,andfourteencyclesforSasena,Hartman3,andHartman6functions,respectively.Toaverageouttheinuenceoftheinitialdataset,werepeattheexperimentswith100differentLatinhypercubedesigns[ 87 88 ].TheexperimentaldesignsarecreatedbytheMATLABRLatinhypercubefunctionlhsdesign[ 94 ],setwiththemaxminoptionwith1,000iterations.FulldetailsofthesequentialsamplingstudyaregiveninTable 6-3 6.3.3OptimizingtheProbabilityofImprovementTheefcientglobaloptimization(EGO)algorithmissequentialinnatureandinvolvesoptimizingEq. 6 (asmanytimesastherequestednumberofpointspercycle)ateachcycle.Wehavechosenusingthepopulationbasedglobaloptimization 112

PAGE 113

Table6-3. EGOsetupfortheanalytictestproblems.Whenmultiplepointspercycleareadded,wemaximizetheapproximatedmultipointprobabilityofimprovement. SetupSasenaHartman3Hartman6 Pointsintheinitialexperimentaldesign122056Maximumnumberofoptimizationcycles61014NumberoffunctionevaluationsrunningEGOwith1pointpercycle61014NumberoffunctionevaluationsrunningEGOwith5pointspercycle305070NumberoffunctionevaluationsrunningEGOwith10pointspercycle60100140 algorithmcalleddifferentialevolutiontomaximizeEq. 6 .DifferentialEvolution(DE)[ 114 115 ]wasproposedtosolveunconstrainedglobaloptimizationproblemsofcontinuousvariables.Likemostpopulationbasedalgorithms,inDE,candidatesolutionsaretakenfromthepopulationofindividualsandrecombinedtoevolvethepopulationtothenextDEiteration.AteachDEiteration,newcandidatesolutionsaregeneratedbythecombinationofindividualsrandomlychosenfromthecurrentpopulation(thisisalsoreferredtoasmutationintheDEliterature).Thenewcandidatesolutionsaremixedwithapredeterminedtargetvector(operationcalledrecombination).Finally,eachoutcomeoftherecombinationisacceptedforthenextDEiterationifandonlyifityieldsareductionintheobjectivefunction(thislastoperatorisreferredtoasselection).Thefreecompanioncodeof[ 115 ]wasusedtoexecutetheDEalgorithmwiththefollowingspecications: Initialpopulation:apopulationof10nv(wherenvisthenumberofdesignvariables)candidatesolutionsisrandomlyspreadoverthedesignspace. Numberofiterations:50. DE-stepsize:0.8. Crossoverprobability:0.8. 113

PAGE 114

Strategy:DE/rand/1/bin.ToincreasethechancesofsuccessintheDEoptimization,werunitfourtimesandtakethebestsolution(intermsoftheexpectedimprovement)(successofmultipleindependentoptimizationwasstudiedbySchutteetal.[ 116 ]).ThispointisthenusedtoevaluatetheobjectivefunctionEGOistryingtooptimizeandlateronupdatekriging. 6.4ResultsandDiscussionFirst,westudythedifferentestimatesoftheprobabilityofimprovementusingtheHartman3function.Figure 6-3A showsthatoverallthekrigingoverestimatestheglobalprobabilities.However,thedifferencesbetweenEqs. 6 and 6 intermsofaccuracyareminute.Figure 6-3B and 6-3C showtheresultsforthecomputationaltime.WithEq. 6 ,evaluatingtheprobabilityofimprovementofasinglesetofpointsrangedfromafractiontoalmosttenseconds,asseeninFigure 6-3B .Thesenumbersmakes 6 unattractiveforoptimization.Ontheotherhand,Figure 6-3C showsthatthenumbersaremuchbetterfortheapproximationgivenby 6 ,astheyallstayinthefractionofamillisecondrange.Figure 6-3D illustratestheratiobetweenthecomputationaltimesofthesetwoapproaches.Theapproximationrunsbetween100and100,000fasterthanthecomputationthattakesthecorrelationbetweenpointsintoaccount.Next,wecheckhowdifcultitmightbetoachievethetargetforthethreetestproblems.Figure 6-4 showsdeboxplotsoftheinitialbestsampleyIBSandthetargetvaluesyT(forboth10%and25%improvement)andcontraststhemwithtestpointsytest(createdwiththeMATLABRLatinhypercubefunctionlhsdesign,usingthemaxminoptionwith100iterations).FortheSasenafunction(Figure 6-4A ),thetargetvalueslieinthelowertailofthedistributionofytestanddonotgobelowtheglobaloptimumy?.ThisispartlyduetothefactthatthefunctionchangessignnearthemedianofyIBS,sothatthetargetisoftenverymodest.Thiswouldmeanthatitshouldbereasonablyeasytoachievethetarget.ForHartman3(Figure 6-4B ),sometargetvaluesfor10%improvementandmostofthemfor25%improvementareactuallybelowy?,sothat 114

PAGE 115

A B C DFigure6-3. Comparisonofestimatesofprobabilityofimprovementforasingletwenty-pointexperimentaldesignoftheHartman3function.A)Globalandestimatedprobabilityofimprovementwhennewpointsllthedesignspacewithmaximizationofthedistance.B)ComputationaltimeusingEq. 6 .C)ComputationaltimeusingEq. 6 .D)Ratioofcomputationaltimes.SimulationsconductedonanIntelRCore2QuadCPUQ6600at2.40GHz,with3GBofRAMrunningMATLABR7.7R2008bunderWindowsXPR. thetargetvalueswouldbedifcultorimpossibletoachieve.ForHartman6(Figure 6-4C ),sometargetvaluesarebelowy?,butheretheinterestingobservationisthatbothyIBSandyTlieintheoutliersofthedistributionofytest.Thiswouldmeanthatrandomexperimentaldesignswouldhavealowprobabilityofachievingthetarget,andthesearchforthemaximumprobabilityofachievingthetargetwouldbemorearduous.Inaddition,wecanseethedifcultiesinsettingagoodtargetvaluebothbecausewedonotknowwhetheritwillfallbelowtheglobaloptimum,andbecausethedistributionofthefunctioncanmakeitdifculttoreach. 115

PAGE 116

A B CFigure6-4. BoxplotsoftheinitialbestsampleyIBS,targetyT,andtestpointsytest(yIBSandyTaretakenover100experimentaldesigns).A)Sasena,12initialpoints.B)Hartman3,20initialpoints.C)Hartman6,56initialpoints.Theglobaloptimumy?isalsoshown.Appendix B detailsboxplots. WealsocheckedhowwellEq. 6 estimatesoftheprobabilityofimprovement.Figure 6-5 showsthemedian(over100experimentaldesigns)oftheglobalandestimatedprobabilityofimprovementfortwodifferenttargetsettingsandfordifferentnumberofpoints.AsshowninFigure 6-5A ,theSasenafunctionshowshighprobabilitiesofimprovements.Alsotheagreementbetweentheestimateandtheglobalprobabilityofimprovementisreasonable.ForHartman3(Figure 6-5B ),thereisunderestimationwhenthetargetis10%improvement.Thismaybeduetotheapproximationoftreatingtheprobabilitiesasindependent,orduetotheinaccuracyofthekriginguncertaintymodel.Whenthetargetis25%improvement,sinceitismostofthetimebelowtheglobaloptimum,theglobalprobabilityofimprovementisusuallyzero.However,sincetheestimatedprobabilityofimprovementisneverzerowestillgetsomesmallvalues.ForHartman6,Figure 6-4C showedthattypicallythereisroomforimprovement(exceptthecasesinwhichthetargetfallsbelowthepresentbestsample),butFigure 6-5C 116

PAGE 117

showsthatkriginggrosslyunderestimatestheglobalprobabilities.Twofactorscontributetothis.First,theglobalminimumisfairlydeep(aswesawinFigure 6-4C ).Second(andasaconsequence),thekrigingpredictionoftenistrappedinthehighervaluesoftheobjectivefunction.Theresultisverysmallprobabilityofmeetingthetarget.Figure 6-6 illustratesuchcaseshowingboththecumulativedistributionfunctionofthekrigingpredictionforone56pointexperimentaldesignandtheHartman6function(basedon10,000testpoints).Thekrigingpredictionissubstantiallymoredistributedtowardstheupper15%oftheresponse(thatis,between)]TJ /F5 11.955 Tf 9.3 0 Td[(0.5and0).ThismakestheprobabilityofbeingbelowthetargetyT=)]TJ /F5 11.955 Tf 9.3 0 Td[(1.7verysmall.Nevertheless,anoverallconclusionfromFigure 6-5A maybethatprobabilityofimprovementapproximatedbyEq. 6 maynotbeverygoodwhentheglobalprobabilitiesaresmall.WehaveseenthatEqs. 6 and 6 areonlypoorestimatesoftheglobalprobabilityofimprovement.Nevertheless,webelievetheymaystillbeusefulforoptimization.Sonextwecomparetheprobabilityofimprovement(Eq. 6 )withtheexpectedimprovement(Eq. 5 )ascriteriaforpointselectiontobeusedbyEGO.Figure 6-7 showsthemedian(over100experimentaldesigns)oftheoptimizationhistorywhenrunningEGOwiththesetwocriteria.ExceptfortheSasenafunction,theprobabilityofimprovementexhibitsperformancecomparabletotheexpectedimprovementmostofthetime(withverylittlesensitivitywithrespecttothetargetvalue).Next,weinvestigatehowgoodthemultipointprobabilityofimprovementapproximatedbyEq. 6 isasaninllcriterion.Figure 6-8 showsthemedian(over100experimentaldesigns)oftheoptimizationhistorywhentheprobabilityofimprovementisusedwithincreasingnumberofpointspercycle.Wecanseethatalthoughthereisnodramaticdifferenceduetodifferenttargetsettings,thereisclearbenetofaddingmorepointspercycle.Forallcases,themorepointsaddedpercycle,thefastertheconvergence.Unfortunately,theinsignicantdifferencebetween5and10pointspercycleshowsthatthegainsdonotscaleverywellwiththenumberofpointspercycle. 117

PAGE 118

A B CFigure6-5. Medianoftheglobalandestimatedprobabilityofimprovement(over100experimentaldesigns).A)Sasena,12initialpoints.B)Hartman3,20initialpoints.C)Hartman6,56initialpoints.Krigingstrugglesinestimatingtheprobabilityofimprovement. 118

PAGE 119

Figure6-6. CumulativedistributionfunctionforkrigingpredictioninoneexperimentaldesignofHartman6ttedwith56points.Mostofthetimekrigingpredictionissubstantiallyabovethetarget.Thatmakestheprobabilityofmeetingthetargetverysmall. Usingmultiplepointspercycleismostlyattractivewhenthemainconcernisthenumberofcyclesratherthanthetotalnumberofsimulations.However,itisinterestingtolookatwhatpenaltyispaidforacceleratedtimeconvergencebyhavingtorunmoresimulations.Forthatpurpose,wecomparethealgorithmsforaxednumberoftotalsimulationsratherthanaxednumberofcycles.Figure 6-9 showsthemedianover100experimentaldesignsoftheoptimizationhistorywithrespecttonumberoffunctionevaluations.AsChapter 5 ,wedidnotseeaclearadvantageorpenaltyofanyofthestrategies(asfarasnumberofpointspercycle)abovetheother.Thereisasmalladvantageinsettingthetargetto25%though.WefoundthatthesuccessinusingtheprobabilityofimprovementforoptimizationseemedorthogonaltotherstlearningthattheEqs. 6 6 and 6 arepoorestimatesoftheglobalprobabilityofimprovement.WesawthattheprobabilityandexpectedimprovementarepossiblyequallyusefulforEGO(Figure 6-7 ).Butwestillneedtounderstandwhyoptimizingpoorprobabilityestimatesiseffective.OnepossiblewaytolookatitistochecktheprobabilitiesofpointsthatwereselectedbyEGO.Differentlyfromanyotherpointsinthedesignspace,thesepointspresentthehighestprobabilityvaluesandpotentiallymightleadtoactualimprovement.Table 6-4 showthemedian(outof100experimentaldesigns)oftheprobabilityofimprovementof 119

PAGE 120

A B CFigure6-7. Median(over100experimentaldesigns)oftheoptimizationhistoryforsinglepointexpectedimprovementEI(x)(Eq. 5 )andprobabilityofimprovementPI(x)(Eq. 6 ).A)Sasena,12initialpoints.B)Hartman3,20initialpoints.C)Hartman6,56initialpoints.Probabilityofimprovementmaybeassuitableforoptimizationastheexpectedimprovement. pointsselectedintherstoptimizationcycle(bymaximizingEqs. 6 and 6 ).ForSasena,wedonotseesignicantdifferencesbetweenpointsthatmetanddidnotmetthetarget.Thisisprobablybecauseofthepointdensity(12pointsintwodimensionalspace)doesnotimpactthekrigingestimates.Ontheotherhand,bothHartman3andHartman6showthatforreasonabletarget(i.e.,10%)pointsthatmetthetargethaveingeneralhigherprobabilityofimprovementthanpointsthatdidnot.Thistendencyisalsoobservedwhentargeting25%improvementinHartman3.Overall,weunderstand 120

PAGE 121

A B CFigure6-8. Median(over100experimentaldesigns)oftheoptimizationhistorywithrespecttothenumberofpointspercycle.A)Sasena,12initialpoints.B)Hartman3,20initialpoints.C)Hartman6,56initialpoints.WeoptimizeEqs. 6 and 6 toobtainthesinglepointandmultipointpercycle,respectively.Asexpected,morepointspercyclespeedsupperformance. 121

PAGE 122

A B CFigure6-9. Median(over100experimentaldesigns)oftheoptimizationhistorywithrespecttothenumberoffunctionevaluations.A)Sasena,12initialpoints.B)Hartman3,20initialpoints.C)Hartman6,56initialpoints.WeoptimizeEqs. 6 and 6 toobtainthesinglepointandmultipointpercycle,respectively.Surprisingly,multiplepointspercyclemightnotnecessarilyimplyinmorefunctionevaluationsforagivenimprovement. 122

PAGE 123

thatalthoughresultsmightdependontheselectedtarget,thekrigingprobabilityofimprovementrelatesreasonablewellwithactualimprovementinthepromisingregionsofthedesignspace. Table6-4. Median(outof100experimentaldesigns)oftheprobabilityofimprovementofpointspointsselectedbykriging. FunctionTargetPointspercyclePointsthatmetthetargetPointsthatdidnotmeetthetarget Sasena10%10.070.0750.300.29100.300.3025%10.070.0650.300.27100.290.29Hartman310%10.400.2750.910.81100.890.8125%10.080.0350.320.13100.230.15Hartman610%10.140.0550.390.26100.370.2225%10.010.00150.010.01100.020.01 6.5SummaryWeproposedrunningtheefcientglobaloptimization(EGO)algorithmwithanapproximatedmultipointprobabilityofimprovement.Theapproachreliesonusinganestimateoftheprobabilityofimprovementthatneglectsthecorrelationbetweendatapoints.Inthenumericalexperiments,weusedthreealgebraicexamplesthatallowedustondthat 1. thecheap-to-evaluatemultipointapproximationoftheprobabilityofimprovementisgoodenoughforpointselection(wefoundittobeacheapalternativetothemultipointexpectedimprovement), 123

PAGE 124

2. fromthepointsselectedbymaximizingtheprobabilityofimprovement,thosewhichmetthetargetpresenthigherprobabilitiesofimprovementthanthosethatdidnotmeetthetarget.Thisisanindicationthatthekrigingprobabilityofimprovementmightcorrelatewithactualimprovementonlylocally,and 3. thenumberofcyclesforconvergenceisreducedasweincreasethenumberofpointspercycle. 124

PAGE 125

CHAPTER7CONCLUSIONSANDFUTUREWORK 7.1SummaryandLearningsDesignanalysisandoptimizationbasedonhigh-delitycomputerexperimentsiscommonlyexpensive.Surrogatemodelingisoftenthetoolofchoiceforreducingthecomputationalburden.However,evenafteryearsofintensiveresearch,surrogatemodelingstillinvolvesastruggletoachievemaximumaccuracywithinlimitedresources.Althoughitispossibletoimprovethesurrogateaccuracybyusingmoresimulations,limitedcomputationalresourcesoftenmakesusfaceatleastoneofthefollowingproblems: Desiredaccuracyofasurrogaterequiresmoresimulationsthanwecanafford. Weusethesurrogateforoptimization,butwendthatthesolutionisinfeasiblewhenwerunthehigh-delityanalysis. Weusethesurrogateforglobaloptimizationandwedonotknowhowtosimultaneouslyobtainmultiplepossibleoptimalsolutions.Thisworkdiscussedandproposedsophisticatedandyetstraightforwardtechniquestoaddressthesethreeissues.Wefocusedon(i)useofmultiplesurrogates(ii)safeestimatorsunderlimitedbudget,and(iii)sequentialsamplinganddrivenbyoptimization.First,wediscussedmultiplesurrogatesandcrossvalidationforestimatingpredictionerror.Wediscussedpotentialadvantagesofselectionandcombinationofsurrogates.Wefoundthat: 1. Usingmultiplesurrogatesisattractivebecausenosinglesurrogateworkswellforallproblemsandthecostofconstructingmultiplesurrogatesisoftensmallcomparedtothecostofsimulations. 2. Crossvalidationiseffectiveinlteringoutinaccuratesurrogates.Withsufcientnumberofpoints,crossvalidationmayevenidentifythebestsurrogateoftheset. 3. Crossvalidationforeitherselectionorcombinationbecomesmoreattractiveinhighdimensions(whenalargenumberofpointsisnaturallyrequired).However,itappearsthatthepotentialgainsfromcombinationdiminishessubstantiallyinhighdimensions. 125

PAGE 126

Wealsoexaminedtheutilityofusingaverylargesetofsurrogatesversusaselectedsubsetforcombination.Thisdecisionisshowntodependontheweightingscheme.Next,weapproachedsurrogatesthatsafelypredicttheactualresponse(particularlyusefulinconstrainedoptimizationorinreliability-baseddesignoptimization).Onewidelyusedmethodforsafeestimationistobiasthepredictionresponsebyadditiveormultiplicativeconstants(termedsafetymarginandsafetyfactors,respectively).Weproposedastrategyfordesigningconservativesurrogatesviaestimationofsafetymarginswithcrossvaliation.Wefoundthat: 1. Crossvalidationappearstobeusefulforbothestimationofsafetymarginandselectionofsurrogate.However,itmaynotbeaccurateenoughwhenthenumberofdatapointsissmall. 2. TheuncertaintyintheestimationoftheactualconservativenesscanbeestimatedusingtheWilsoninterval.Finally,westudiedmodernalgorithmsforsurrogate-basedglobaloptimizationwithparallelfunctionevaluation.Wefocusedonapproachesthatusethesurrogateuncertaintyestimatortoguidetheselectionofthenextsamplingcandidatee.g.,theefcientglobaloptimization(EGO)algorithm.Weproposedtwostrategiesforenablingoptimizationwithparallelfunctionevaluations.First,weproposedusingmultiplesurrogateswecalleditmultiplesurrogateefcientglobaloptimization(MSEGO)algorithm.Thestrategyisnotlimitedtoonlysurrogatesthatprovideuncertaintyestimates(suchaskriging).Insuchcase,weproposedimportationofuncertaintyestimatesfromothermodels.Wefoundthat: 1. Theimporteduncertaintystructureenablenon-krigingsurrogatestoruntheEGOalgorithm. 2. MSEGOspeedsuptheoptimizationconvergence.Substantialreductioninthenumberofrequiredcyclesisachievedbyusingseveralsimulationspercycle.Thereisalsoabenetfromthediversityofthesetofsurrogates.Wefoundthatthisreducedtherisksassociatedwithpoorlyttedsurrogates. 126

PAGE 127

3. Itisnotclearwhetherthereispenaltyintermsoftotalnumberoffunctionevaluations.Outofthreeanalyticexamples,therewassubstantialpenaltyforonlyone.However,foranother,MSEGOprovidedreductioninthenumberoffunctionevaluations.Thesecondstrategyusesacheap-to-evaluateapproximationoftheprobabilityofimprovementtoprovidemultiplepointspercycle.Theapproximationisbasedontheassumptionthatcorrelationamongdifferentpointscanbeneglectedprovidedthattheyaresufcientlydistantfromoneanother.Wefoundthat: 1. Estimatesoftheprobabilityofimprovement(eitherwhenconsideringorneglectingcorrelation)mightbeunsatisfactorywhentheactualprobabilitiesarelow.Whilethatfacthaspreventedsuchestimatestobeusedforstoppingcriterion,wefoundthatitdoesnothinderoptimization. 2. Theproposedcheap-to-evaluatemultipointapproximationoftheprobabilityofimprovementwassuccessfullyappliedforpointselection. 3. AswithMSEGO,multiplepointspercyclespeededupoptimizationconvergence. 7.2PerspectivesTherearefewdirectionsthatwouldbeworthpursuinginmultiplesurrogatesforpredictionandoptimization.Withrespecttomultiplesurrogates,wepointthefollowingtopicsforfutureresearch: 1. Variabledelityframework:multiplesurrogatescanpotentiallyidentifyregionsofhighuncertainty.Thatinformationmightbeusedforproperlydesignatingthedelitylevelofextrasimulations. 2. Reliability-baseddesginoptimization:multiplesurrogatesmightbeusedrstasaninsuranceagainstpoorlyrepresentedlimitstate,orinsequentialsampling(forfurtheraccuracyofthelimitstateapproximation). 3. Visualizationanddesignspaceexploration:thisisverypromisingsincedifferentsurrogatesmightbemoreaccurateindifferentregionsofthedesignspace.Inaddition,recentdevelopmentsincomputerhardware(e.g.,graphicsprocessingunits)mightmakemultiplesurrogatesvisualizationmoreaffordable.Withrespecttosafeestimatorsunderlimitedbudget,wesuggestthecombineduseofconservativepredictionsandsequentialsamplingstrategies.Thiscouldpotentially 127

PAGE 128

increasesurrogateaccuracyintheregionsneartheoptimum(reducingchancesofoverdesignorfailure).Withrespecttosequentialsamplingdrivenbyoptimization,weseepotentialin: 1. Estimationoftheexpectedimprovementandprobabilityofimprovement:currentmetricsareusefulforpointselection.Webelievethatresearchinbetterestimatorswouldimprovebothsamplingandinformationusedtostopoptimization. 2. Combineduseofspace-llingandadaptivesampling:thismightincreaserobustnessinshortcycleoptimization. 3. Inuenceofthesurrogateaccuracy:whensimulationsareextremelyexpensive,thepoorqualityofthesurrogatesmightbeanissueevenwhenusingmultiplesurrogates.Finally,complexityandinsomecasesthelackofcommercialsoftwaremayhinderthesetechniquesfrompopularityinthenearterm.So,webelievethatinvestmentsinpackagesandlearningtoolstogetherwithongoingscienticinvestigationareverybenecial. 128

PAGE 129

APPENDIXAOVERVIEWOFTHESURROGATETECHNIQUESUSEDINTHISWORK A.1ResponseSurfaceResponsesurfaceassumesthatwhiledataiscontaminatedwithnoise(randomwithstandardnormaldistribution)andthattheactualmodelisknown(sumofbasisfunctionsfi(xj)).Anobservationatapointxjofthedesignspace(functionevaluationorexperimentation)ismodeledby: y(x)=ej+nbXi=1bifi(xj),(A)Theestimatedresponse^y(x)atanypointxisobtainedbywiththeestimatorsiofthecoefcientsbi ^y(x)=nXi=1ifi(xj),(A) wheren=nb.Accordingtotheresponsesurfacetheory[ 25 27 ],thecoefcientsicanestimatedbyleastsquaresprovidedthesetof(enough)datapointsX=[x1,...,xp]Tandrespectiveobservationsy: =)]TJ /F17 11.955 Tf 5.47 -9.69 Td[(FTF)]TJ /F7 7.97 Tf 6.58 0 Td[(1FTy,(A) whereFistheGramianmatrix(matrixoflinearequationsconstructedusingthebasisfunctionsfi(x)[ 25 27 ])ofthedesignmatrixX.Responsesurfacemodelsalsooffersapoint-wisemeasureofuncertainty,theso-calledpredictionvarianceatapointxcanestimatedby s2PRS(x="(y)]TJ /F17 11.955 Tf 11.75 0 Td[(^y)T(y)]TJ /F17 11.955 Tf 11.75 0 Td[(^y) p)]TJ /F3 11.955 Tf 11.95 0 Td[(n#fT0)]TJ /F17 11.955 Tf 5.48 -9.68 Td[(FTF)]TJ /F7 7.97 Tf 6.59 0 Td[(1f0,(A) 129

PAGE 130

whereyand^yarethevaluesoftheactualandestimatedresponsesatthepsampledpoints,nisthenumberofcoefcientsoftheresponsesurface,andf0andFaretheGramianmatricesofxandthedesignmatrixX,respectively.Figure A-1 depictstheconceptspresentedsofarshowingboththepredictionandtheerrorestimatesoftheresponsesurface. FigureA-1. Quadraticpolynomialresponsesurface^yPRS(x)ofanarbitrarysetofvepointsofthefunctiony(x)=)]TJ /F5 11.955 Tf 5.48 -9.68 Td[(10cos(2x)+15)]TJ /F5 11.955 Tf 11.95 0 Td[(5x+x2=50.Uncertainty(whichamplitudeissPRS(x))associatedwith^yPRS(x)isshowningray. Withintheresponsesurfaceframework,theobtainedmodelisexpectedtolterthenoiseoftheobserveddata(thismightexplainthepopularityinindustrialapplicationsandexperimentalsciences[ 123 ]).Itisalsopossibletoidentifythesignicanceofdifferentdesignfactorsdirectlyfromthecoefcientsinthenormalizedregressionmodel(inpractice,usingt-statistics).Inspiteoftheadvantages,thereisalwaysadrawbackwhenapplyingresponsesurfacetomodelhighlynonlinearfunctions(asinFigure A-1 ).Eventhoughhigher-orderpolynomialscanbeusedforexample,itmaybedifculttotakesufcientsampledatatoestimatewellallcoefcients.Inthiswork,theSURROGATEStoolbox[ 43 ]wasusedtoexecutethe(polynomial)responsesurface. A.2KrigingKrigingestimatesthevalueofafunctionasacombinationofknownfunctionsfi(x)(e.g.,alinearmodelsuchasapolynomialtrend)anddepartures(representinglowandhighfrequencyvariationcomponents,respectively)oftheform 130

PAGE 131

^y(x)=mXi=1ifi(x)+z(x),(A) wherez(x)isassumedtobearealizationofastochasticprocessZ(x)withzeromean,processvariance2,andspatialcovariancefunctiongivenby cov(Z(xi),Z(xj))=2R(xi,xj),(A) 2=1 p(y)]TJ /F17 11.955 Tf 11.96 0 Td[(Xb)TR)]TJ /F7 7.97 Tf 6.59 0 Td[(1(y)]TJ /F17 11.955 Tf 11.95 0 Td[(Xb),(A) whereR(xi,xj)isthecorrelationbetweenxiandxj,yisvalueoftheactualresponsesatthesampledpoints,XistheGramiandesignmatrixconstructedusingbasisfunctionsinthetrendmodelatthedesignpoints,Risthematrixofcorrelationsamongdesignpoints,bisanapproximationofthevectorofcoefcientsiofEq. A .Wecanestimatetheuncertaintyin^y(x)usingthekrigingpredictionvariances2KRG(x)(alsoknownasmeansquarederrorofthepredictor): s2KRG(x)=21+uT)]TJ /F17 11.955 Tf 5.48 -9.68 Td[(XTR)]TJ /F7 7.97 Tf 6.59 0 Td[(1X)]TJ /F7 7.97 Tf 6.59 0 Td[(1u)]TJ /F17 11.955 Tf 11.95 0 Td[(rTX)]TJ /F7 7.97 Tf 6.59 0 Td[(1r,(A) whereu=XTR)]TJ /F7 7.97 Tf 6.59 0 Td[(1r)]TJ /F17 11.955 Tf 11.96 0 Td[(f,risthevectorofcorrelationsbetweenthepointxandthedesignpoints,fisthevectorofbasisfunctionsinthetrendmodelatpointx.Figure A-2 depictstheconceptspresentedsofarshowingboththepredictionandtheerrorestimatesofkriging.Wecanseetheimplicationsofthekrigingstatisticalassumptions;asaninterpolator,theerrorvanishesatdatapoints.Althoughconventionalkrigingmodelsinterpolatetrainingdata,theextensiontonoisydataisalreadyavailable[ 124 ].Krigingisaexibletechniquesincedifferentinstancescanbecreatedbychoosingdifferentpairsoftrendandcorrelationfunctions.See[ 19 21 ]formoredetailsaboutkriging.Inthisresearch,thetoolboxofLophavenetal.[ 48 ]wasusedtoexecutethekriginginterpolator. 131

PAGE 132

FigureA-2. Krigingmodel^yKRG(x)ofanarbitrarysetofvepointsofthefunctiony(x)=)]TJ /F5 11.955 Tf 5.48 -9.68 Td[(10cos(2x)+15)]TJ /F5 11.955 Tf 11.96 0 Td[(5x+x2=50.Uncertainty(whichamplitudeissKRG(x))associatedwith^yKRG(x)isshowningray. A.3RadialBasisNeuralNetworksAradialbasisneuralnetworkisanarticialneuralnetworkwhichusesradialbasisfunctionsastransferfunctions.Aradialbasisneuralnetworkconsistsoftwolayers:ahiddenradialbasislayerandanoutputlinearlayer,asshowninFigure A-3 .Radialbasisneuralnetworkestimatesthevalueofafunctionas: ^y(x)=nHXi=1ai(x,ci),(A) wherenHisthenumberofneuronsinthehiddenlayer,aiaretheweightsofthelinearoutputneuron,ciisthecentervectorforneuroni,and(x,ci)istheradialbasisfunction.Atypicalbasisfunctionisthefollowing (x,ci)=exp)]TJ /F8 11.955 Tf 9.3 0 Td[(kx)]TJ /F17 11.955 Tf 11.95 0 Td[(cik2.(A)Figure A-4 illustratesaradialbasisneuralnetworkttedtothesamedatapointsofFigure A-1 .Aradialbasisneuralnetworkmayrequiremoreneuronsthanstandardfeedforwardandbackpropagationnetworks,butoftentheycanbedesignedinafractionofthetimeittakestotrainstandardfeedforwardnetworks.Acrucialdrawbackforsomeapplications 132

PAGE 133

FigureA-3. Radialbasisneuralnetworkarchitecture. FigureA-4. Radialbasisneuralnetworkmodel^yRBNN(x)ofanarbitrarysetofvepointsofthefunctiony(x)=)]TJ /F5 11.955 Tf 5.48 -9.68 Td[(10cos(2x)+15)]TJ /F5 11.955 Tf 11.95 0 Td[(5x+x2=50.Currentimplementationdoesnotofferuncertaintyestimates. istheneedoftoomanytrainingpoints.Radialbasisneuralnetworkiscomprehensivelypresentedin[ 17 18 ].Here,weusethenativeneuralnetworksMATLABRtoolbox[ 94 ]toexecutetheradialbasisneuralnetworkalgorithm. A.4LinearShepardShepardisaninterpolationmethodbasedonweightedaverageofvaluesatthedatapoints.Itisaninstanceofmovingleastsquarethatestimatesthevalueofafunctionas ^y(x)=nPi=1Wi(x)Pi(x) nPi=1Wi(x),(A) 133

PAGE 134

wherePi(x)isalocalapproximationfunctioncenteredatx(k)andWi(x)areweightfunctions.WeusethelinearShepard,whichhasweightfunctionsoftheform[ 125 ] Wk(x)=264R(k)w)]TJ /F3 11.955 Tf 11.96 0 Td[(dk(x)+ R(k)wdk(x)3752,(A) wheredk(x)istheEuclideandistancebetweenthepointsxandx(k),R(k)w=q NwD2 4n>0istheradiusofinuenceaboutthepointx(k)chosenlargeenoughtoincludeNwpoints,andDisthemaximumdistancebetweentwodatapoints.ThelinearSheparduses Pi(x)=yk+mXj=1a(k)j(xj)]TJ /F3 11.955 Tf 11.95 0 Td[(x(k)j),(A) wherea(k)jaretheminimumnormsolutionofthelinearleastsquareproblem mina2EmkAa)]TJ /F3 11.955 Tf 11.96 0 Td[(bk2,Aj=p wijk)]TJ /F3 11.955 Tf 5.48 -9.68 Td[(x(ij))]TJ /F3 11.955 Tf 11.95 0 Td[(x(k)t,bj=p wijk)]TJ /F3 11.955 Tf 5.48 -9.69 Td[(yij)]TJ /F3 11.955 Tf 11.96 0 Td[(yk.(A)Figure A-7 illustratesalinearShepardmodelttedtothesamedatapointsofFigure A-1 FigureA-5. LinearShepardmodel^ySHEP(x)ofanarbitrarysetofvepointsofthefunctiony(x)=)]TJ /F5 11.955 Tf 5.48 -9.68 Td[(10cos(2x)+15)]TJ /F5 11.955 Tf 11.96 0 Td[(5x+x2=50.Currentimplementationdoesnotofferuncertaintyestimates. 134

PAGE 135

AnopenissuethelinearShepardisthechoiceofNw.TolearnmoreaboutShepardalgorithmsee[ 126 127 ].Inthisdissertation,thecodedevelopedbyMr.DavidEasterlingandMr.NickRadcliffe(basedonSHEPPACK[ 127 ])toexecutethelinearShepardalgorithm. A.5SupportVectorRegressionSupportvectorregressionisaparticularimplementationofsupportvectormachines.Insupportvectorregression,theaimistond^y(x)thathasatmostadeviationofmagnitudefromeachofthetrainingdata.Supportvectorregressionestimatesthevalueofafunctionas ^y(x)=pXi=1(ai)]TJ /F3 11.955 Tf 11.95 0 Td[(ai)K(xi,x)+b,(A) whereK((x)i,(x))istheso-calledkernelfunction,(x)iaredifferentpointsoftheoriginaldatasetand(x)isthepointofthedesignspaceinwhichthesurrogateisevaluated.Parametersai,ai,andbareobtainedduringthettingprocess. TableA-1. Exampleofkernelfunctions. NameFormula Gaussianradialbasisfunction(GRBF)K(x,x0)=exp)]TJ /F15 7.97 Tf 10.49 5.7 Td[(kx)]TJ /F7 7.97 Tf 6.59 0 Td[(x0k2 22Exponentialradialbasisfunction(ERBF)K(x,x0)=exp)]TJ /F15 7.97 Tf 10.49 5.7 Td[(kx)]TJ /F7 7.97 Tf 6.59 0 Td[(x0k 22SplineK(x,x0)=1+hx,x0i+1 2hx,x0imin(x,x0))]TJ /F7 7.97 Tf -259.51 -11.29 Td[(1 6(min(x,x0))3ANOVAK(x,x0)=QiKi(xi,x0i)Ki(xi,x0i)=1+xix0i+(xix0i)2+(xix0i)2min(xi,x0i))]TJ /F3 11.955 Tf 13.9 0 Td[(xix0i(xi+x0i)(min(xi,x0i))2+1 3)]TJ /F3 11.955 Tf 5.48 -9.68 Td[(xi2+4xix0i+x0i2(min(xi,x0i))3)]TJ /F7 7.97 Tf -259.51 -11.35 Td[(1 2(xi+x0i)(min(xi,x0i))4+1 5(min(xi,x0i))5 Duringthettingprocess,supportvectorregressionminimizesanupperboundontheexpectedriskunlikeempiricalriskminimizationtechniques,whichminimizetheerroronthetrainingdata.Thisisdonebyusingalternativelossfunctions.Figure A-6 135

PAGE 136

showstwoofthemostcommonpossiblelossfunctions.Figure A-6A correspondstotheconventionalleastsquareserrorcriterionandFigure A-6B illustratesthe-insensitivelossfunction,whichisgivenbythefollowingequation: l(x)=8><>:",ifjy(x))]TJ /F5 11.955 Tf 12.25 0 Td[(^y(x)j"jy(x))]TJ /F5 11.955 Tf 12.25 0 Td[(^y(x)j,otherwise.(A) A BFigureA-6. Lossfunctionsusedinsupportvectorregression.A)Quadratic.B)-insensitive. Theimplicationisthatinsupportvectorregressionminimizesthegoalistondafunctionthathasatmostdeviationfromthetrainingdata.Inotherwords,theerrorsareconsideredzeroaslongastheyarelessthan.Besides,thettingthesupportvectorregressionminimizesmodelhasaregularizationparameterC.Cdeterminesthecompromisebetweenthecomplexityandthedegreetowhichdeviationslargerthanaretoleratedintheoptimizationformulation.IfCistoolarge(innity),thetoleranceissmallandthetendencyistohavethemostpossiblecomplexsupportvectorregressionmodel.Figure A-7 illustratesasupportvectorregressionmodelttedtothesamedatapointsofFigure A-1 .Anopenissueinsupportvectorregressionminimizesisthechoiceofthevaluesofparametersforbothkernelandlossfunctions.Tolearnmoreaboutsupportvectorregressionsee[ 22 24 ]. 136

PAGE 137

FigureA-7. Supportvectorregressionmodel^ySVR(x)ofanarbitrarysetofvepointsofthefunctiony(x)=)]TJ /F5 11.955 Tf 5.48 -9.68 Td[(10cos(2x)+15)]TJ /F5 11.955 Tf 11.96 0 Td[(5x+x2=50.Currentimplementationdoesnotofferuncertaintyestimates. Inthisthesis,thecodedevelopedbyGunn[ 49 ]wasusedtoexecutethesupportvectorregressionalgorithm. 137

PAGE 138

APPENDIXBBOXPLOTSInaboxplot,theboxisdenedbylinesatthelowerquartile(25%),median(50%),andupperquartile(75%)values.Linesextendfromeachendoftheboxtoshowthecoverageoftherestofthedata(i.e.,theyareplottedatadistanceof1.5timestheinter-quartilerangeineachdirectionorthelimitofthedata,ifthelimitofthedatafallswithin1.5timestheinter-quartilerange).Outliersaredatawithvaluesbeyondtheendsofthelinesbyplacinga+signforeachpoint.TheexamplegivenintheMATLABRtutorial[ 94 ](seeFigure B-1 )showstheboxplotofcarmileagegroupedbycountries. FigureB-1. Exampleofboxplot. 138

PAGE 139

APPENDIXCCONSERVATIVEPREDICTORSANDMULTIPLESURROGATESInthisappendix,weshowthebenetsofmultiplesurrogatesintermsoftherelativeerrorgrowth.Wedonotexpecttomaketheselectthesurrogatewithbestestimationofthesafetymargin(seetheissuesfacedintheBranin-HooexamplewhilediscussingFigures 4-7 and 4-6 ).Instead,weexpecttoreducetherelativeerrorgrowthduetoinadequatechoiceofthesurrogate.Givenasetofsurrogates,were-denetherelativeerrorgrowthby REG=eRMS eRMS)]TJ /F5 11.955 Tf 11.96 0 Td[(1,(C) whereeRMSistakenatagiventargetconservativeness,andeRMSistheeRMSofthesetofsurrogateswithoutaddinganysafetymargin.Eq. C impliesthatevenfortheunbiasedsurrogates,theremightberelativeerrorgrowth.Onlythemostaccuratesurrogate(i.e.,surrogatewithsmallesteRMS,whichwecallBestRMSE)hasREG=0.Asweaskforconservativeestimation,twothingshappen:(i)becauseofEq. 4 ,theidentityofBestRMSEmightchange,and(ii)evenBestRMSEloosessomeaccuracy(butthelossisminimal).Thus,ifthegoalistoreducethelosses,wesuggesttoselectthesurrogatethathassmallestestimatedrelativeerrorgrowth.Thatis,foreverytargetconservativeness,wepickthesurrogatewithsmallestPRESSRMS,whichwecallBestPRESS(weupdatethePRESSRMSvaluesusingEqs. 4 and 3 ).WeemployedtheHartman6functionsampledwith110pointsandttedtothesurrogatesshowninTable C-1 toillustrateourstrategyforsurrogateselection.Again,weusedtheDACEtoolboxofLophavenetal.[ 48 ]andSURROGATEStoolboxofViana[ 43 ]toexecutethekrigingandpolynomialresponsesurfacemethods.Additionally,wealsousedthenativeneuralnetworksMATLABRtoolbox[ 94 ]andthecodedevelopedbyGunn[ 49 ]torunradialbasisneuralnetworkandsupportvectorregressionalgorithms, 139

PAGE 140

respectively.WeusemultipleinstancesofdifferentsurrogatesinthesamefashionofVianaetal.[ 34 ]andSanchezetal.[ 30 ].Thisispossiblebecausekrigingallowsdifferentinstancesbychangingparameterssuchasbasisandcorrelationfunctions.Aswesaid,althoughtheremightbebetterimplementations,forthepurposeofthispaperitisadvantageoustohavesomepoorsurrogatesinthemix,becauseitiswhensurrogatesareleastaccuratethatcompensatingfortheirerrorsbyconservativenessismostimportant.Weuse1,000experimentaldesignstoaverageoutthedependencyofthedatapoints. TableC-1. Surrogatesusedinthestudyofconservativepredictorsandmultiplesurrogates. SurrogatesDetails krg0krg1krg2Krigingmodel:krg0,krg1,andkrg2indicatezero,rst,andsecondorderpolynomialregressionmodel,respectively.Inallcases,aGaussiancorrelationand0i=10,and0i200,i=1,2,...,dwereused.Wechose3differentkrigingsurrogatesbyvaryingtheregressionmodel.rbnnRadialbasisneuralnetwork:goal=(0.5y)2andspread=1=3.svr-grbf-fullsvr-grbf-shortsvr-poly-fullsvr-poly-shortSupportvectorregression:grbfandpolyindicatethekernelfunctions(Gaussianandsecondorderpolynomialrespectively).Alluseinsensitivelossfunctions.fullandshortrefertodifferentvaluesfortheregularizationparameter,C,andfortheinsensitivity,.FulladoptsC=1and=110)]TJ /F7 7.97 Tf 6.59 0 Td[(4,whileshort01andshort02usestheselectionofvaluesaccordingtoCherkasskyandMa[ 95 ].Forshort01=y=p p,andforshort02=3yp lnp=p;andforbothC=100maxjy+3yj,jy)]TJ /F5 11.955 Tf 11.96 0 Td[(3yj,whereyandyarethemeanvalueandthestandarddeviationofthefunctionvaluesatthedesigndata,respectively.Wechose4differentsurrogatesbyvaryingthekernelfunctionandtheparametersCand.prs2prs3Polynomialresponsesurface:Fullmodelsofdegree2and3. Figure C-1 showsinblack,thefrequencyinwhichdifferentunbiasedsurrogatesappearasmostaccurateofthesetoutof1,000experimentaldesigns.Thegraybars 140

PAGE 141

inFigure C-1 showthefrequencyinwhichthesurrogatesaremostaccuratefortheexperimentaldesign#50whenthetargetconservativenessvariesfrom50%to100%.Wecanextendwhatisknownintheliteraturebysayingthattheaccuracywilldependnotonlyontheproblemandexperimentaldesignbutalsoontheconservativenesslevel.WehopethatbyselectingthesurrogateusingPRESSRMSwecanavoidfurtherlossesofapoorlyttedmodel. FigureC-1. eRMSanalysisforHartman6(110points):1,000experimentaldesignand51target%c(from50%to100%).Mostaccuratesurrogatechangesnotonlywiththedesignofexperimentbutalsowiththetarget%c. Figure C-2 givesthemedianover1,000experimentaldesignsoftherelativeerrorgrowthfortheHartman6functionttedwith110points.TheactualrelativeerrorgrowthischeckedwithEq. C (largesetoftestpoints)usingtheeRMSofthemostaccurateunbiasedsurrogateofthesetasreference.Thisgureshowsfoursurrogates: krg0:mostaccuratesurrogatefortargetconservativenesscloseto50%(unbiasedcase). rbnn:mostaccuratesurrogateforhighvaluesoftargetconservativeness. BestPRESS:surrogatewiththesmallestPRESSRMSateachconservativenesslevel. BestRMSE:surrogatewiththesmallesteRMSateachconservativenesslevel.Figure C-2A reinforcestheideathatthelossinaccuracydependsontheexperimentaldesignandontheconservativenesslevel.Itshowsthatchangingthetargetconservativenessmaychangethesurrogatethatofferstheminimumrelativeerrorgrowth.Forexample, 141

PAGE 142

attargetconservativenesscloseto50%,thechoiceofkrg0leadstoamedianof0%relativeerrorgrowth,whilethechoiceofrbnnleadstoarelativeerrorgrowthof10%.However,forhighlevelsofconservativeness,forexample95%,thedifferencebetweenthesurrogatesfavorsrbnnby18%.So,justbycomparingattheunbiasedsurrogateswewouldneverknowthattheperformanceoftheconservativesurrogatesmaybedifferent.Wefurtherseethatcrossvalidationsuccessfullyselectsthesurrogateforminimumlossofaccuracy.Figure C-2A showsthatBestPRESS(selectionbasedondatapoints)performsalmostaswellasBestRMSE(selectionbasedontestpoints,notpractical). A BFigureC-2. MedianoftheactualrelativeerrorgrowthversustargetconservativenessfortheHartman6with110points.A)Singlesurrogatesversusthebestoftheset.B)BestPRESSversusthebestsurrogateoftheset.Wecanseethat(i)mostaccuratesurrogatechangeswithtarget%cand(ii)crossvalidationsuccessfullyselectsthebestchoice. 142

PAGE 143

APPENDIXDDERIVATIONOFTHEEXPECTEDIMPROVEMENTJonesetal.[ 14 ]denedtheimprovementatapointxas I(x)=max(yPBS)]TJ /F3 11.955 Tf 11.96 0 Td[(Y(x),0),(D) whichisarandomvariablebecauseY(x)isarandomvariable(recollectthatkrigingmodelstheresponsey(x)asarealizationofaGaussianprocessY(x)).TheexpectedimprovementiscalculatedastheexpectationofI(x).Consideringw=(y)]TJ /F5 11.955 Tf 12.24 0 Td[(^y(x))=s(x),wehave: EI(x)=E[I(x)]=RyPBS(yPBS)]TJ /F3 11.955 Tf 11.95 0 Td[(y(x))fY(y)dy,EI(x)=RyPBS(yPBS)]TJ /F5 11.955 Tf 12.24 0 Td[(^y(x)+^y(x))]TJ /F3 11.955 Tf 11.96 0 Td[(y(x))fY(y)dy,EI(x)=RyPBS(yPBS)]TJ /F5 11.955 Tf 12.25 0 Td[(^y(x))fY(y)dy+RyPBS(^y(x))]TJ /F3 11.955 Tf 11.95 0 Td[(y(x))fY(y)dy,EI(x)=(yPBS)]TJ /F5 11.955 Tf 12.25 0 Td[(^y(x))R(yPBS)]TJ /F7 7.97 Tf 6.77 0 Td[(^y(x))=s(x)fW(w)dw)]TJ /F3 11.955 Tf 11.95 0 Td[(s(x)R(yPBS)]TJ /F7 7.97 Tf 6.77 0 Td[(^y(x))=s(x)wfW(w)dw,EI(x)=(yPBS)]TJ /F5 11.955 Tf 12.24 0 Td[(^y(x))FWyPBS)]TJ /F7 7.97 Tf 6.77 0 Td[(^y(x) s(x))]TJ /F3 11.955 Tf 11.96 0 Td[(s(x)R(yPBS)]TJ /F7 7.97 Tf 6.77 0 Td[(^y(x))=s(x)wfW(w)dw.(D)ForGaussiandistribution: (A)=)]TJ /F6 11.955 Tf 11.29 9.63 Td[(RAz(z)dz.(D)SubstitutinginEquation D EI(x)=(yPBS)]TJ /F5 11.955 Tf 12.24 0 Td[(^y(x))yPBS)]TJ /F7 7.97 Tf 6.77 0 Td[(^y(x) s(x)+s(x)yPBS)]TJ /F7 7.97 Tf 6.77 0 Td[(^y(x) s(x),EI(x)=s(x)[u(u)+(u)],u=[yPBS)]TJ /F5 11.955 Tf 12.24 0 Td[(^y(x)]=s(x).(D) 143

PAGE 144

REFERENCES [1] Jin,R.,Chen,W.,andSimpson,T.W.,Comparativestudiesofmetamodellingtechniquesundermultiplemodellingcriteria,StructuralandMultidisciplinaryOptimization,Vol.23,No.1,2001,pp.1. [2] Sacks,J.,Welch,W.J.,Mitchell,T.J.,andWynn,H.P.,Designandanalysisofcomputerexperiments,StatisticalScience,Vol.4,No.4,1989,pp.409. [3] Kleijnen,J.P.C.,Sanchez,S.M.,Lucas,T.W.,andCioppa,T.M.,Auser'sguidetothebravenewworldofdesigningsimulationexperiments,INFORMSJournalonComputing,Vol.17,No.3,2005,pp.263. [4] Queipo,N.V.,Haftka,R.T.,Shyy,W.,Goel,T.,Vaidyanathan,R.,andTucker,P.K.,Surrogate-basedanalysisandoptimization,ProgressinAerospaceSciences,Vol.41,No.1,2005,pp.1. [5] Chen,V.C.P.,Tsui,K.L.,Barton,R.R.,andMeckesheimer,M.,Areviewondesign,modelingandapplicationsofcomputerexperiments,IIETransactions,Vol.38,No.4,2006,pp.273. [6] Forrester,A.I.J.andKeane,A.J.,Recentadvancesinsurrogate-basedoptimization,ProgressinAerospaceSciences,Vol.45,No.1-3,2009,pp.50. [7] Simpson,T.W.,Toropov,V.,Balabanov,V.,andViana,F.A.C.,Metamodelinginstructuralandmultidisciplinaryoptimization:howfarwehavecomeornot,StructuralandMultidisciplinaryOptimization,Vol.underreview,2011. [8] Venkataraman,S.andHaftka,R.T.,Structuraloptimizationcomplexity:whathasMooreslawdoneforus?StructuralandMultidisciplinaryOptimization,Vol.28,No.6,2004,pp.375. [9] Barthelemy,J.F.M.andHaftka,R.T.,Approximationconceptsforoptimumstructuraldesignareview,StructuralandMultidisciplinaryOptimization,Vol.5,No.3,1993,pp.129. [10] Sobieszczanski-Sobieski,J.andHaftka,R.T.,Multidisciplinaryaerospacedesignoptimization:surveyofrecentdevelopments,StructuralandMultidisciplinaryOptimization,Vol.14,No.1,1997,pp.1. [11] Simpson,T.W.,Poplinski,J.,Koch,P.N.,andAllen,J.K.,Metamodelsforcomputer-basedengineeringdesign:surveyandrecommendations,EngineeringwithComputers,Vol.17,No.2,2001,pp.129. [12] Simpson,T.W.,Booker,A.J.,Ghosh,D.,Giunta,A.A.,Koch,P.N.,andYang,R.J.,Approximationmethodsinmultidisciplinaryanalysisandoptimization:apaneldiscussion,StructuralandMultidisciplinaryOptimization,Vol.27,No.5,2004,pp.302. 144

PAGE 145

[13] Wang,G.G.andShan,S.,Reviewofmetamodelingtechniquesinsupportofengineeringdesignoptimization,JournalofMechanicalDesign,Vol.129,No.4,2007,pp.370. [14] Jones,D.R.,Schonlau,M.,andWelch,W.J.,Efcientglobaloptimizationofexpensiveblack-boxfunctions,JournalofGlobalOptimization,Vol.13,No.4,1998,pp.455. [15] Huang,D.,Allen,T.T.,Notz,W.I.,andZeng,N.,Globaloptimizationofstochasticblack-boxsystemsviasequentialkrigingmeta-models,JournalofGlobalOpti-mization,Vol.34,No.3,2006,pp.441. [16] Park,J.andSandberg,I.W.,Universalapproximationusingradial-basis-functionnetworks,NeuralComputation,Vol.3,No.2,1991,pp.246. [17] Smith,M.,NeuralNetworksforStatisticalModeling,VonNostrandReinhold,1993. [18] Cheng,B.andTitterington,D.M.,Neuralnetworks:areviewfromastatisticalperspective,StatisticalScience,Vol.9,No.1,1994,pp.2. [19] Stein,M.L.,InterpolationofSpatialData:sometheoryforkriging,SpringerVerlag,1999. [20] Simpson,T.W.,Mauery,T.M.,Korte,J.J.,andMistree,F.,Krigingmodelsforglobalapproximationinsimulation-basedmultidisciplinarydesignoptimization,AIAAJournal,Vol.39,No.12,2001,pp.2233. [21] Kleijnen,J.P.C.,Krigingmetamodelinginsimulation:Areview,EuropeanJournalofOperationalResearch,Vol.192,No.3,2009,pp.707. [22] Muller,K.R.,Mika,S.,Ratsch,G.,Tsuda,K.,andScholkopf,B.,Anintroductiontokernel-basedlearningalgorithms,IEEETransactionsonNeuralNetworks,Vol.12,No.2,2001,pp.181. [23] Scholkopf,B.andSmola,A.J.,LearningwithKernels,TheMITPress,2002. [24] Smola,A.J.andScholkopf,B.,Atutorialonsupportvectorregression,StatisticsandComputing,Vol.14,No.3,2004,pp.199. [25] Box,G.E.P.,Hunter,J.S.,andHunter,W.G.,StatisticsforExperimenters:anintroductiontodesign,dataanalysis,andmodelbuilding,JohnWiley&Sons,1978. [26] Myers,R.H.andMontgomery,D.C.,ResponseSurfaceMethodology:processandproductoptimizationusingdesignedexperiments,JohnWiley&Sons,1995. [27] Myers,R.H.,ClassicalandModernRegressionwithApplications,DuxburyPress,2000. 145

PAGE 146

[28] Mack,Y.,Goel,T.,Shyy,W.,Haftka,R.T.,andQueipo,N.V.,Multiplesurrogatesfortheshapeoptimizationofbluffbody-facilitatedmixing,43rdAIAAaerospacesciencesmeetingandexhibit,AIAA,Reno,NV,USA,Jan2005,pp.AIAA. [29] Samad,A.,Kim,K.Y.,Goel,T.,Haftka,R.T.,andShyy,W.,Multiplesurrogatemodelingforaxialcompressorbladeshapeoptimization,JournalofPropulsionandPower,Vol.24,No.2,2008,pp.302. [30] Sanchez,E.,Pintos,S.,andQueipo,N.V.,Towardanoptimalensembleofkernel-basedapproximationswithengineeringapplications,StructuralandMultidisciplinaryOptimization,Vol.36,No.3,2008,pp.247. [31] Viana,F.A.C.andHaftka,R.T.,Usingmultiplesurrogatesformetamodeling,7thASMO-UKISSMOInternationalConferenceonEngineeringDesignOptimization,ISSMO,Bath,UK,Jul2008,pp.1. [32] Viana,F.A.C.andHaftka,R.T.,UsingmultiplesurrogatesforminimizationoftheRMSerrorinmeta-modeling,ASME2008InternationalDesignEngineeringTechnicalConferences&ComputersandInformationinEngineeringConference,ASME,NewYork,NY,USA,Aug2008,pp.DETC2008. [33] Viana,F.A.C.,Haftka,R.T.,SteffenJr,V.,Butkewitsch,S.,andLeal,M.F.,Ensembleofsurrogates:aframeworkbasedonminimizationofthemeanintegratedsquareerror,49thAIAA/ASME/ASCE/AHS/ASCStructures,StructuralDynamics,andMaterialsConference,AIAA,Schaumburg,IL,USA,Apr2008,pp.AIAA. [34] Viana,F.A.C.,Haftka,R.T.,andSteffenJr,V.,Multiplesurrogates:howcross-validationerrorscanhelpustoobtainthebestpredictor,StructuralandMultidisciplinaryOptimization,Vol.39,No.4,2009,pp.439. [35] Viana,F.A.C.,Picheny,V.,andHaftka,R.T.,Safetymarginsforconservativesurrogates,8thWorldCongressonStructuralandMultidisciplinaryOptimization,ISSMO,Lisbon,Portugal,Jun2009,pp.1. [36] Viana,F.A.C.,Picheny,V.,andHaftka,R.T.,Conservativepredictionviasafetymargin:designthroughcrossvalidationandbenetsofmultiplesurrogates,ASME2009InternationalDesignEngineeringTechnicalConferences&Comput-ersandInformationinEngineeringConference,ASME,SanDiego,CA,USA,Aug2009,pp.DETC2009. [37] Viana,F.A.C.,Picheny,V.,andHaftka,R.T.,Usingcrossvalidationtodesignconservativesurrogates,AIAAJournal,Vol.48,No.10,2010,pp.2286. [38] Viana,F.A.C.andHaftka,R.T.,Crossvalidationcanestimatehowwellpredictionvariancecorrelateswitherror,AIAAJournal,Vol.47,No.9,2009,pp.2266. 146

PAGE 147

[39] Viana,F.A.C.andHaftka,R.T.,Importinguncertaintyestimatesfromonesurrogatetoanother,50thAIAA/ASME/ASCE/AHS/ASCStructures,StructuralDynamics,andMaterialsConference,AIAA,PalmSprings,CA,USA,May2009,pp.AIAA. [40] Viana,F.A.C.andHaftka,R.T.,Surrogate-basedoptimizationwithparallelsimulationsusingtheprobabilityofimprovement,13thAIAA/ISSMOMultidisci-plinaryAnalysisandOptimizationConference,AIAA,FortWorth,TX,USA,Sep2010,pp.AIAA. [41] Viana,F.A.C.,Haftka,R.T.,andWatson,L.T.,Whynotruntheefcientglobaloptimizationalgorithmwithmultiplesurrogates?51thAIAA/ASME/ASCE/AHS/ASCStructures,StructuralDynamics,andMaterialsConference,AIAA,Orlando,FL,USA,Apr2010,pp.AIAA. [42] Haftka,R.T.andViana,F.A.C.,Surrogate-basedglobaloptimizationwithparallelfunctionevaluation,2011NSFEngineeringResearchandInnovationConference,NSF,Atlanta,GA,USA,Jan2011,pp.1. [43] Viana,F.A.C.,SURROGATESToolboxUser'sGuide,Gainesville,FL,USA,version2.1ed.,2010,availableat http://sites.google.com/site/felipeacviana/surrogatestoolbox [44] Viana,F.A.C.,Gogu,C.,andHaftka,R.T.,Makingthemostoutofsurrogatemodels:tricksofthetrade,ASME2010InternationalDesignEngineeringTechni-calConferences&ComputersandInformationinEngineeringConference,ASME,Montreal,QC,Canada,Aug2009,pp.DETC2010. [45] Goel,T.,Haftka,R.T.,Shyy,W.,andQueipo,N.V.,Ensembleofsurrogates,StructuralandMultidisciplinaryOptimization,Vol.32,No.3,2007,pp.199. [46] Acar,E.andRais-Rohani,M.,Ensembleofmetamodelswithoptimizedweightfactors,StructuralandMultidisciplinaryOptimization,Vol.37,No.3,2009,pp.279. [47] Zhou,T.,Weightingmethodforalinearmixedmodel,CommunicationsinStatistics,PartA:TheoryandMethods,Vol.39,No.2,2010,pp.214. [48] Lophaven,S.N.,Nielsen,H.B.,andSndergaard,J.,DACEaMATLABkrigingtoolbox,Tech.Rep.IMMTR,InformaticsandMathematicalModelling,TechnicalUniversityofDenmark,Denmark,Aug2002,availableat http://www2.imm.dtu.dk/~hbn/dace/ [49] Gunn,S.R.,Supportvectormachinesforclassicationandregression,Tech.rep.,ImageSpeechandIntelligentSystemsResearchGroup,UniversityofSouthampton,UK,1997,availableat http://www.isis.ecs.soton.ac.uk/resources/svminfo/ 147

PAGE 148

[50] Picard,R.R.andCook,R.D.,Cross-validationofregressionmodels,JournaloftheAmericanStatisticalAssociation,Vol.79,No.387,1984,pp.575. [51] Kohavi,R.,Astudyofcross-validationandbootstrapforaccuracyestimationandmodelselection,FourteenthInternationalJointConferenceonArticialIntelligence,IJCAII,Montreal,QC,CA,Aug1995,pp.1137. [52] Roecker,E.B.,Predictionerroranditsestimationforsubset-selectedmodels,Technometrics,Vol.33,No.4,1991,pp.459. [53] Utans,J.andMoody,J.,Selectingneuralnetworkarchitecturesviathepredictionrisk:applicationtocorporatebondratingprediction,IEEE1stInternationalConferenceonAIApplicationsonWallStreet,IEEE,NewYork,NY,USA,Oct1991,pp.35. [54] Zhang,P.,Modelselectionviamultifoldcrossvalidation,TheAnnalsofStatistics,Vol.21,No.1,1993,pp.299. [55] Zerpa,L.E.,Queipo,N.V.,Pintos,S.,andSalager,J.L.,Anoptimizationmethodologyofalkaline-surfactant-polymeroodingprocessesusingeldscalenumericalsimulationandmultiplesurrogates,JournalofPetroleumScienceandEngineering,Vol.7,No.3,2005,pp.197. [56] Acar,E.,Variousapproachesforconstructinganensembleofmetamodelsusinglocalmeasures,StructuralandMultidisciplinaryOptimization,Vol.42,No.6,2011,pp.879. [57] Yang,Y.,Regressionwithmultiplecandidatemodels:selectingormixing?StatisticaSinica,Vol.13,No.3,2003,pp.783. [58] Glaz,B.,Goel,T.,Liu,L.,Friedmann,P.,andHaftka,R.T.,Amultiplesurrogateapproachtohelicopterrotorbladevibrationreduction,AIAAJournal,Vol.47,No.1,2009,pp.271. [59] Saijal,K.K.,Ganguli,R.,andViswamurthy,S.R.,Optimizationofhelicopterrotorusingpolynomialandneuralnetworkmetamodels,JournalofAircraft,Vol.48,No.2,2011,pp.553. [60] Jin,R.,Chen,W.,andSudjianto,A.,Onsequentialsamplingforglobalmetamodelingforinengineeringdesign,ASME2002InternationalDesignEngi-neeringTechnicalConferences&ComputersandInformationinEngineeringCon-ference,ASME,Montreal,QC,Canada,Sep2002,pp.DETC2002/DAC. [61] Kleijnen,J.P.C.andvanBeers,W.C.M.,Application-drivensequentialdesignsforsimulationexperiments:krigingmetamodelling,JournaloftheOperationalResearchSociety,Vol.55,No.8,2004,pp.876. 148

PAGE 149

[62] vanBeers,W.C.M.andKleijnen,J.P.C.,Customizedsequentialdesignsforrandomsimulationexperiments:krigingmetamodelingandbootstrapping,EuropeanJournalofOperationalResearch,Vol.186,No.3,2008,pp.1099. [63] denHertog,D.,Kleijnen,J.P.C.,andSiem,A.Y.D.,Thecorrectkrigingvarianceestimatedbybootstrapping,JournaloftheOperationalResearchSociety,Vol.57,No.4,2006,pp.400. [64] Rai,R.,QualitativeandQuantitativeSequentialSampling,Ph.D.thesis,UniversityofTexas,Austin,TX,USA,2006. [65] Turner,C.J.,Crawford,R.H.,andCampbell,M.I.,MultidimensionalsequentialsamplingforNURBs-basedmetamodeldevelopment,EngineeringwithComput-ers,Vol.23,No.3,2007,pp.155. [66] Gorissen,D.,Dhaene,T.,anddeTurck,F.,Evolutionarymodeltypeselectionforglobalsurrogatemodeling,JournalofMachineLearningResearch,Vol.10,No.1,2009,pp.2039. [67] Rennen,G.,Husslage,B.,vanDam,E.R.,anddenHertog,D.,NestedmaximinLatinhypercubedesigns,StructuralandMultidisciplinaryOptimization,Vol.41,No.3,2010,pp.371. [68] Loeppky,J.L.,Moore,L.M.,andWilliams,B.J.,Batchsequentialdesignsforcomputerexperiments,JournalofStatisticalPlanningandInference,Vol.140,No.6,2010,pp.1452. [69] Forrester,A.I.J.,Sobester,A.,andKeane,A.J.,EngineeringDesignviaSurro-gateModelling:apracticalguide,JohnWiley&Sons,2008. [70] Jones,D.R.,Ataxonomyofglobaloptimizationmethodsbasedonresponsesurfaces,JournalofGlobalOptimization,Vol.21,No.4,2001,pp.345. [71] Villemonteix,J.,Vazquez,E.,andWalter,E.,Aninformationalapproachtotheglobaloptimizationofexpensive-to-evaluatefunctions,JournalofGlobalOptimization,Vol.44,No.4,2009,pp.509. [72] Queipo,N.V.,Verde,A.,Pintos,S.,andHaftka,R.T.,AssessingthevalueofanothercycleinGaussianprocesssurrogate-basedoptimization,StructuralandMultidisciplinaryOptimization,Vol.39,No.5,2009,pp.1. [73] Ginsbourger,D.,LeRiche,R.,andCarraro,L.,Krigingiswell-suitedtoparallelizeoptimization,ComputationalIntelligenceinExpensiveOptimizationProblems,editedbyY.TenneandC.Goh,Vol.2,SpringerBerlin/Heidelberg,2010,pp.131. [74] Ginsbourger,D.,LeRiche,R.,andCarraro,L.,Multi-pointscriterionfordeterministicparallelglobaloptimizationbasedonkriging,InternationalCon-ferenceonNonconvexProgramming,NIAS,Rouen,France,Dec2007,pp.1. 149

PAGE 150

[75] Wu,Y.,Shin,Y.,Sues,R.,andCesare,M.,Safety-factorbasedapproachforprobability-baseddesignoptimization,42ndAIAA/ASME/ASCE/AHS/ASCStructures,StructuralDynamics,andMaterialsConference,AIAA,Seattle,WA,USA,Apr2001,pp.AIAA. [76] Elishakoff,I.,SafetyFactorsandReliability:friendsorfoes?,KluwerAcademic,2004. [77] Acar,E.,Kale,A.,andHaftka,R.T.,Comparingeffectivenessofmeasuresthatimproveaircraftstructuralsafety,JournalofAerospaceEngineering,Vol.20,No.3,2007,pp.186. [78] Lee,J.,Jeong,H.,Choi,D.H.,Volovoi,V.,andMavris,D.,AnenhancementofconstraintfeasibilityinBPNbasedapproximateoptimization,ComputerMethodsinAppliedMechanicsandEngineering,Vol.196,No.17,2007,pp.2147. [79] Lee,J.,Jeong,H.,andKang,S.,DerivativeandGA-basedmethodsinmetamodelingofback-propagationneuralnetworksforconstrainedapproximateoptimization,StructuralandMultidisciplinaryOptimization,Vol.35,No.1,2008,pp.29. [80] Lee,J.andShin,K.H.,Aconservativemethodofwaveletneuralnetworkbasedmeta-modelinginconstrainedapproximateoptimization,Computers&Structures,Vol.89,No.1,2011,pp.109. [81] Picheny,V.,Kim,N.,Haftka,R.T.,andQueipo,N.V.,Conservativepredictionsusingsurrogatemodeling,49thAIAA/ASME/ASCE/AHS/ASCStructures,Struc-turalDynamics,andMaterialsConference,Schaumburg,AIAA,Schaumburg,IL,USA,Apr2008,pp.AIAA. [82] Picheny,V.,ImprovingAccuracyandCompensatingforUncertaintyinSurrogateModeling,Ph.D.thesis,DepartmentofMechanicalandAerospaceEngineering,UniversityofFlorida,Gainesville,FL,USA,2009. [83] Audet,C.,Dennis,J.E.,Moore,D.W.,Booker,A.,andFrank,P.D.,Asurrogate-model-basedmethodforconstrainedoptimization,8thAIAA/NASA/USAF/ISSMOSymposiumonMultidisciplinaryAnalysisandOpti-mization,AIAA,LongBeach,CA,USA,Sep2000,pp.AIAA. [84] Picheny,V.,Ginsbourger,D.,Roustant,O.,Haftka,R.T.,andKim,N.,Adaptivedesignofexperimentsforaccurateapproximationofatargetregion,JournalofMechanicalDesign,Vol.132,No.7,2010,pp.071008(9pages). [85] Bichon,B.J.,Mahadevan,S.,andEldred,M.S.,Reliability-baseddesignoptimizationusingefcientglobalreliabilityanalysis,50thAIAA/ASME/ASCE/AHS/ASCStructures,StructuralDynamics,andMaterialsConference,AIAA,PalmSprings,CA,USA,May2009,pp.AIAA. 150

PAGE 151

[86] Ronchetti,E.,Field,C.,andBlanchard,W.,Robustlinearmodelselectionbycross-validation,JournaloftheAmericanStatisticalAssociation,Vol.92,No.439,1997,pp.1017. [87] McKay,M.D.,Beckman,R.J.,andConover,W.J.,Acomparisonofthreemethodsforselectingvaluesofinputvariablesfromacomputercode,Technomet-rics,Vol.21,No.2,1979,pp.239. [88] Iman,R.L.andConover,W.J.,Smallsamplesensitivityanalysistechniquesforcomputermodels,withanapplicationtoriskassessment,CommunicationsinStatistics,PartA:TheoryandMethods,Vol.9,No.17,1980,pp.1749. [89] Meckesheimer,M.,Booker,A.J.,Barton,R.R.,andSimpson,T.W.,Computationallyinexpensivemetamodelassessmentstrategies,AIAAJour-nal,Vol.40,No.10,2002,pp.2053. [90] Varma,S.andSimon,R.,Biasinerrorestimationwhenusingcross-validationformodelselection,BMCBioinformatics,Vol.7,No.91,2006,pp.1290. [91] Montgomery,D.C.,DesignandAnalysisofExperiments,JohnWiley&Sons,6thed.,2004. [92] Simpson,T.W.,Lin,D.K.J.,andChen,W.,Samplingstrategiesforcomputerexperiments:designandanalysis,InternationalJournalofReliabilityandApplica-tions,Vol.2,No.3,2001,pp.209. [93] Bishop,C.M.,NeuralNetworksforPatternRecognition,OxfordUniversityPress,1995. [94] MathWorkscontributors,MATLABThelanguageoftechnicalcomputing,TheMathWorks,Inc,Natick,MA,USA,version7.0release14ed.,2004. [95] Cherkassky,V.andMa,Y.,PracticalselectionofSVMparametersandnoiseestimationforSVMregression,NeuralNetworks,Vol.17,No.1,2004,pp.113. [96] Dixon,L.C.W.andSzego,G.P.,TowardsGlobalOptimization2,NorthHolland,1978. [97] Lee,J.,Anovelthree-phasetrajectoryinformedsearchmethodologyforglobaloptimization,JournalofGlobalOptimization,Vol.38,No.1,2007,pp.61. [98] Giunta,A.A.,Wojtkiewicz,S.F.,andEldred,M.S.,Overviewofmoderndesignofexperimentsmethodsforcomputationalsimulations,41stAIAAAerospaceSciencesMeetingandExhibit,AIAA,Reno,NV,USA,Jan2003,pp.AIAA. [99] Stone,C.J.,ACourseinProbabilityandStatistics,Duxbury,1995. 151

PAGE 152

[100] Brown,L.D.,Cai,T.T.,andDasGupta,A.,Intervalestimationforabinomialproportion,StatisticalScience,Vol.16,No.2,2001,pp.101. [101] Wilson,E.B.,Probableinference,thelawofsuccession,andstatisticalinference,JournaloftheAmericanStatisticalAssociation,Vol.22,No.158,1927,pp.209. [102] Viana,F.A.C.,Haftka,R.T.,andWatson,L.T.,Efcientglobaloptimizationalgorithmassistedbymultiplesurrogates,EngineeringOptimization,Vol.submitted,2011. [103] Gutmann,H.M.,Aradialbasisfunctionmethodforglobaloptimization,JournalofGlobalOptimization,Vol.19,No.3,2001,pp.201. [104] Kleijnen,J.P.,vanBeers,W.,andvanNieuwenhuyse,I.,Expectedimprovementinefcientglobaloptimizationthroughbootstrappedkriging,JournalofGlobalOptimization,Vol.acceptedforpublication,2011. [105] Henkenjohann,N.andKunert,J.,Anefcientsequentialoptimizationapproachbasedonthemultivariateexpectedimprovementcriterion,QualityEngineering,Vol.19,No.4,2007,pp.267. [106] Ponweiser,W.,Wagner,T.,andVincze,M.,Clusteredmultiplegeneralizedexpectedimprovement:anovelinllsamplingcriterionforsurrogatemodels,IEEECongressonEvolutionaryComputation,IEEE,HongKong,Jun2008,pp.3514. [107] Seok,K.,Hwang,C.,andCho,D.,Predictionintervalsforsupportvectormachineregression,CommunicationsinStatistics,PartA:TheoryandMethods,Vol.31,No.10,2002,pp.1887. [108] Jekabsons,G.,RBF:RadialBasisFunctioninterpolationforMATLAB/OCTAVE,RigaTechnicalUniversity,Latvia,version1.1ed.,2009,availableat http://www.cs.rtu.lv/jekabsons/regression.html [109] Thacker,W.I.,Zhang,J.,Watson,L.T.,Birch,J.B.,Iyer,M.A.,andBerry,M.W.,Algorithm905:SHEPPACK:modiedShepardalgorithmforinterpolationofscatteredmultivariatedata,ACMTransactionsonMathematicalSoftware,Vol.37,No.3,2010,pp.1. [110] Sasena,M.J.,OptimizationofComputerSimulationsviaSmoothingSplinesandKrigingMetamodels,Master'sthesis,UniversityofMichigan,AnnArbor,MI,USA,1998. [111] Bennett,J.A.andBotkin,M.E.,TheOptimumShape:automatedstructuraldesign,Springer,1986. [112] ANSYScontributors,ANSYSMechanical,ANSYSInc,version7.0release14ed.,2010. 152

PAGE 153

[113] Bichon,B.J.,Eldred,M.S.,Swiler,L.P.,Mahadevan,S.,andMcFarland,J.,Efcientglobalreliabilityanalysisfornonlinearimplicitperformancefunctions,AIAAJournal,Vol.46,No.10,2008,pp.2459. [114] Storn,R.andPrice,K.,Differentialevolutionasimpleandefcientheuristicforglobaloptimizationovercontinuousspaces,JournalofGlobalOptimization,Vol.11,No.4,1997,pp.341. [115] Price,K.V.,Storn,R.M.,andLampinen,J.A.,DifferentialEvolution:apracticalapproachtoglobaloptimization,SpringerVerlag,2005. [116] Schutte,J.F.,Haftka,R.T.,andFregly,B.J.,Improvedglobalconvergenceprobabilityusingmultipleindependentoptimizations,InternationalJournalforNumericalMethodsinEngineering,Vol.71,No.6,2007,pp.678. [117] Queipo,N.,Pintos,S.,Verde,A.,andNava,E.,Settingtargetsforsurrogate-basedoptimization,51thAIAA/ASME/ASCE/AHS/ASCStructures,StructuralDynamics,andMaterialsConference,AIAA,Orlando,FL,USA,Apr2010,pp.AIAA. [118] Genz,A.,Comparisonofmethodsforthecomputationofmultivariatenormalprobabilities,25thSymposiumontheInterface:ComputingScienceandStatis-tics,SASInstitute,SanDiego,CA,USA,Apr1993,pp.400. [119] Drezner,Z.andWesolowsky,G.O.,Onthecomputationofthebivariatenormalintegral,JournalofStatisticalComputationandSimulation,Vol.35,No.1-2,1990,pp.101. [120] Genz,A.,Numericalcomputationofrectangularbivariateandtrivariatenormalandtprobabilities,StatisticsandComputing,Vol.14,No.3,2004,pp.251. [121] Genz,A.andBretz,F.,Numericalcomputationofmultivariatetprobabilitieswithapplicationtopowercalculationofmultiplecontrasts,JournalofStatisticalComputationandSimulation,Vol.63,No.4,1999,pp.361. [122] Genz,A.andBretz,F.,Comparisonofmethodsforthecomputationofmultivariatetprobabilities,JournalofComputationalandGraphicalStatistics,Vol.11,No.4,2002,pp.950. [123] Biles,W.E.andSwain,J.J.,OptimizationandIndustrialExperimentation,JohnWiley&Sons,1980. [124] Rasmussen,C.E.andWilliams,C.K.I.,GaussianProcessesforMachineLearning,TheMITPress,2006. [125] Franke,R.andNielson,G.,Smoothinterpolationoflargesetsofscattereddata,InternationalJournalforNumericalMethodsinEngineering,Vol.15,No.11,1980,pp.1691. 153

PAGE 154

[126] Shepard,D.,Atwo-dimensionalinterpolationfunctionforirregularlyspaceddata,23rdACMNationalConference,ACM,NewYork,NY,USA,1968,pp.517. [127] Thacker,W.I.,Zhang,J.,Watson,L.T.,Birch,J.B.,Iyer,M.A.,andBerry,M.W.,AlgorithmXXX:SHEPPACK:modiedShepardalgorithmforinterpolationofscatteredmultivariatedata,Tech.Rep.TR,ComputerScience,VirginiaPolytechnicInstitute&StateUniversity,USA,2009. 154

PAGE 155

BIOGRAPHICALSKETCH FelipeA.C.VianawasborninPatosdeMinas,Brazilin1980.HespenthischildhoodintwoothercitiesofBrazil(MontesClarosandUberlandia).Hisinterestforsciencewastriggedduringphysicsandchemistrylabsinmiddleschool.Bythattime,Felipewasdividedbetweenpursuingengineeringormedicine.AfrustratingvisittotheanatomylaboftheUniversidadeFederaldeUberlandia(Brazil)inthesenioryearofhighschoolputanendtotheindecision.In1998,FelipeenrolledintheSchoolofElectricalEngineeringoftheUniversidadeFederaldeUberlandia.Duringtheveyearsofundergraduatestudies(inBrazil,gettingabachelorinengineeringtakesveyears),hewasalwaysinvolvedwithextra-curricularactivities.Heworkedasteachingassistantsincehissecondsemester.Inhisfourthsemester,hereceivedhisrstundergraduatefellowshipandstarteddoingresearchontribology.In2003,heenrolledatthegraduateschooloftheUniversidadeFederaldeUberlandia.In2005,heearnedhisMasterofScienceinmechanicalengineeringdegreewiththethesisVibrationDampingbyusingPiezoelectricPatchesandResonantShuntCircuits.In2008,heearnedhisDoctorofPhilosophyinmechanicalengineeringdegreewiththedissertationSurrogateModelingTechniquesandHeuristicOptimizationMethodsappliedtoDesignandIdenticationProblems.DuringtheSummerof2006,heworkedasinvitedresearcherfortheVanderplaatsResearchandDevelopmentInc.(USA).HeimplementedoftheoptimalLatinhypercubedesignofexperiments.ThismoduleisnowintegratedtoVisualDOC(oneoftheVR&Dproducts).During2007,heworkedasresearchscholarintheStructuralandMultidisciplinaryOptimizationGroupofUniversityofFlorida.In2008heenrolledinthegraduateschooloftheUniversityofFloridawhereheworkedonaseconddoctorate(inaerospaceengineering)undersupervisionofDr.RaphaelHaftka.HereceivedhisPh.D.fromtheUniversityofFloridainthesummerof2011.Hisresearchinterestsincludeoptimizationmethods,reliabilityanalysisandprobabilisticdesign,surrogatemodeling,anddesignofexperiments. 155