<%BANNER%>

Autonomic Application and Resource Management in Virtualized Distributed Computing Systems

Permanent Link: http://ufdc.ufl.edu/UFE0042294/00001

Material Information

Title: Autonomic Application and Resource Management in Virtualized Distributed Computing Systems
Physical Description: 1 online resource (177 p.)
Language: english
Creator: XU,JING
Publisher: University of Florida
Place of Publication: Gainesville, Fla.
Publication Date: 2011

Subjects

Subjects / Keywords: AUTONOMIC -- CONTROL -- FUZZY -- GRIDS -- MULTIOBJECTIVE -- OPTIMIZATION -- VIRTUALIZATION
Electrical and Computer Engineering -- Dissertations, Academic -- UF
Genre: Electrical and Computer Engineering thesis, Ph.D.
bibliography   ( marcgt )
theses   ( marcgt )
government publication (state, provincial, terriorial, dependent)   ( marcgt )
born-digital   ( sobekcm )
Electronic Thesis or Dissertation

Notes

Abstract: Large-scale distributed computing systems, such as computational grids and enterprise data centers, present complex management challenges. Such systems experience inherent dynamism due to unpredictable resource availability and usage, or/and highly dynamic workloads. By introducing a layer of abstraction, virtualization technology provides ways of provisioning and customizing resource environments as needed, and migrating workloads to adapt to dynamic changes. However, the scale of such computing systems makes it extremely hard to control them manually by one or more human operators. Our solution is to incorporate autonomic capabilities into the management of applications and resources in grid and data center environments to reduce direct human intervention. Such capabilities are accomplished through a two-level feedback-control framework in which local controllers at the application level have detailed information about the applications and allow independent adaptation and optimization. The global controller at the resource level collects resource information and optimizes the system behavior from a global perspective. It also acts as a coordinator when conflicts occur at different local controllers. For grid environments, the proposed two-level control system is studied in the context of In-VIGO, a grid-computing system that provides application services on-demand using dynamically instantiated virtual machines, networks, data and applications. Local controllers utilize application-specific information for tracking and predicting the performance of jobs executing on grid resources, which is then used to guide the scheduling/rescheduling decisions. Its effectiveness has been evaluated for CPU-intensive jobs with relatively short execution times (ranging from tens of seconds to less than an hour) on resources with highly variable loads. The results show that In-VIGO jobs managed by the two-level controllers consistently meet their execution deadlines under varying load conditions and gracefully recover from unexpected failures. Under the most dynamic and heavy loading environment created by the experiments, the average job runtime of the proposed approach is 10% and 20% shorter than two other competing scheduling strategies, one using round-robin and the other using the same scheduling as the proposed approach but without rescheduling actions. The percentage of jobs meeting their predefined deadlines is improved by 40% and 5\%, respectively. In a virtualized data center, the two-level control system is designed to deliver performance guarantees while optimizing resource usage, and also other important aspects of data centers such as power and cooling costs. At the application level, two fuzzy-logic-based methods - fuzzy modeling and fuzzy prediction - are proposed to estimate the resource demands for dynamic workloads. The global controller at the resource level tries to find the optimal resource allocation and virtual machine (VM) placement/replacement, with multiple objectives including the elimination of thermal hotspots, the minimization of total power consumption, and the efficient use of resources. The problem is posed as a multi-objective combinatorial optimization problem and an improved genetic algorithm with fuzzy multi-objective evaluation is proposed for efficiently searching the large solution space and conveniently combining possibly conflicting objectives. An online local search algorithm using multi-objective optimization and stabilization techniques is designed for dynamically changing virtual machine placement to quickly adapt to changes in system conditions or workloads. The proposed approaches are implemented and evaluated on a virtualized testbed built upon an IBM BladeCenter. Under both synthetic and real-world Web workloads the local controller is validated to accurately estimate resource needs (the difference is within 5%) using fuzzy modeling and fuzzy prediction approaches. The global controller for determining virtual machine placement is tested with simulation-based experiments over a wide range of problem sizes and the results show that the multi-objective optimization using genetic algorithm achieve good balance among different objectives, resulting in relatively low values for power consumption, peak temperature, and resource wastage. For the dynamic virtual machine migration problem, experimental evaluations are conducted using a mix of types of workloads to emulate the variety and dynamics of data center workloads. The results indicate that the proposed multi-objective optimization with stabilization significantly reduces unnecessary VM migration and unstable host selection by up to 80% and also improves the application performance by up to 30% and the efficiencies of power usage by up to 20%. The rapid growth of computing systems raises new challenges for centralized management at the global-control level in the proposed two-level architecture. A network of cooperative controllers is proposed in this work, each managing a subset of resources and collectively collaborating to manage the entire system. The proposed network model is validated on a testbed for In-VIGO and the results show that the decentralized and cooperative nature of the system yields a number of desirable properties, including efficiency, robustness, and scalability under a highly dynamic environment.
General Note: In the series University of Florida Digital Collections.
General Note: Includes vita.
Bibliography: Includes bibliographical references.
Source of Description: Description based on online resource; title from PDF title page.
Source of Description: This bibliographic record is available under the Creative Commons CC0 public domain dedication. The University of Florida Libraries, as creator of this bibliographic record, has waived all rights to it worldwide under copyright law, including all related and neighboring rights, to the extent allowed by law.
Statement of Responsibility: by JING XU.
Thesis: Thesis (Ph.D.)--University of Florida, 2011.
Local: Adviser: Fortes, Jose A.

Record Information

Source Institution: UFRGP
Rights Management: Applicable rights reserved.
Classification: lcc - LD1780 2011
System ID: UFE0042294:00001

Permanent Link: http://ufdc.ufl.edu/UFE0042294/00001

Material Information

Title: Autonomic Application and Resource Management in Virtualized Distributed Computing Systems
Physical Description: 1 online resource (177 p.)
Language: english
Creator: XU,JING
Publisher: University of Florida
Place of Publication: Gainesville, Fla.
Publication Date: 2011

Subjects

Subjects / Keywords: AUTONOMIC -- CONTROL -- FUZZY -- GRIDS -- MULTIOBJECTIVE -- OPTIMIZATION -- VIRTUALIZATION
Electrical and Computer Engineering -- Dissertations, Academic -- UF
Genre: Electrical and Computer Engineering thesis, Ph.D.
bibliography   ( marcgt )
theses   ( marcgt )
government publication (state, provincial, terriorial, dependent)   ( marcgt )
born-digital   ( sobekcm )
Electronic Thesis or Dissertation

Notes

Abstract: Large-scale distributed computing systems, such as computational grids and enterprise data centers, present complex management challenges. Such systems experience inherent dynamism due to unpredictable resource availability and usage, or/and highly dynamic workloads. By introducing a layer of abstraction, virtualization technology provides ways of provisioning and customizing resource environments as needed, and migrating workloads to adapt to dynamic changes. However, the scale of such computing systems makes it extremely hard to control them manually by one or more human operators. Our solution is to incorporate autonomic capabilities into the management of applications and resources in grid and data center environments to reduce direct human intervention. Such capabilities are accomplished through a two-level feedback-control framework in which local controllers at the application level have detailed information about the applications and allow independent adaptation and optimization. The global controller at the resource level collects resource information and optimizes the system behavior from a global perspective. It also acts as a coordinator when conflicts occur at different local controllers. For grid environments, the proposed two-level control system is studied in the context of In-VIGO, a grid-computing system that provides application services on-demand using dynamically instantiated virtual machines, networks, data and applications. Local controllers utilize application-specific information for tracking and predicting the performance of jobs executing on grid resources, which is then used to guide the scheduling/rescheduling decisions. Its effectiveness has been evaluated for CPU-intensive jobs with relatively short execution times (ranging from tens of seconds to less than an hour) on resources with highly variable loads. The results show that In-VIGO jobs managed by the two-level controllers consistently meet their execution deadlines under varying load conditions and gracefully recover from unexpected failures. Under the most dynamic and heavy loading environment created by the experiments, the average job runtime of the proposed approach is 10% and 20% shorter than two other competing scheduling strategies, one using round-robin and the other using the same scheduling as the proposed approach but without rescheduling actions. The percentage of jobs meeting their predefined deadlines is improved by 40% and 5\%, respectively. In a virtualized data center, the two-level control system is designed to deliver performance guarantees while optimizing resource usage, and also other important aspects of data centers such as power and cooling costs. At the application level, two fuzzy-logic-based methods - fuzzy modeling and fuzzy prediction - are proposed to estimate the resource demands for dynamic workloads. The global controller at the resource level tries to find the optimal resource allocation and virtual machine (VM) placement/replacement, with multiple objectives including the elimination of thermal hotspots, the minimization of total power consumption, and the efficient use of resources. The problem is posed as a multi-objective combinatorial optimization problem and an improved genetic algorithm with fuzzy multi-objective evaluation is proposed for efficiently searching the large solution space and conveniently combining possibly conflicting objectives. An online local search algorithm using multi-objective optimization and stabilization techniques is designed for dynamically changing virtual machine placement to quickly adapt to changes in system conditions or workloads. The proposed approaches are implemented and evaluated on a virtualized testbed built upon an IBM BladeCenter. Under both synthetic and real-world Web workloads the local controller is validated to accurately estimate resource needs (the difference is within 5%) using fuzzy modeling and fuzzy prediction approaches. The global controller for determining virtual machine placement is tested with simulation-based experiments over a wide range of problem sizes and the results show that the multi-objective optimization using genetic algorithm achieve good balance among different objectives, resulting in relatively low values for power consumption, peak temperature, and resource wastage. For the dynamic virtual machine migration problem, experimental evaluations are conducted using a mix of types of workloads to emulate the variety and dynamics of data center workloads. The results indicate that the proposed multi-objective optimization with stabilization significantly reduces unnecessary VM migration and unstable host selection by up to 80% and also improves the application performance by up to 30% and the efficiencies of power usage by up to 20%. The rapid growth of computing systems raises new challenges for centralized management at the global-control level in the proposed two-level architecture. A network of cooperative controllers is proposed in this work, each managing a subset of resources and collectively collaborating to manage the entire system. The proposed network model is validated on a testbed for In-VIGO and the results show that the decentralized and cooperative nature of the system yields a number of desirable properties, including efficiency, robustness, and scalability under a highly dynamic environment.
General Note: In the series University of Florida Digital Collections.
General Note: Includes vita.
Bibliography: Includes bibliographical references.
Source of Description: Description based on online resource; title from PDF title page.
Source of Description: This bibliographic record is available under the Creative Commons CC0 public domain dedication. The University of Florida Libraries, as creator of this bibliographic record, has waived all rights to it worldwide under copyright law, including all related and neighboring rights, to the extent allowed by law.
Statement of Responsibility: by JING XU.
Thesis: Thesis (Ph.D.)--University of Florida, 2011.
Local: Adviser: Fortes, Jose A.

Record Information

Source Institution: UFRGP
Rights Management: Applicable rights reserved.
Classification: lcc - LD1780 2011
System ID: UFE0042294:00001


This item has the following downloads:


Full Text

PAGE 1

AUTONOMICAPPLICATIONANDRESOURCEMANAGEMENTINVIRTUALIZED DISTRIBUTEDCOMPUTINGSYSTEMS By JINGXU ADISSERTATIONPRESENTEDTOTHEGRADUATESCHOOL OFTHEUNIVERSITYOFFLORIDAINPARTIALFULFILLMENT OFTHEREQUIREMENTSFORTHEDEGREEOF DOCTOROFPHILOSOPHY UNIVERSITYOFFLORIDA 2011

PAGE 2

Tomyparents,myhusband,andmywonderfulbabies 2

PAGE 3

ACKNOWLEDGMENTS Firstandforemost,Iwouldliketoexpressmysinceregratitudetomyadvisor, Prof.JoseFortesforhisendlesspatience,greatmotivation,excellentguidanceand atremendousamountofsupportthroughoutmyPhDstudyatACISlab.Iamgreatly indebtedtohimforconstantlysupportingmeandpatientlygivingmetimeandhelping megrow. Iamverygratefultotheothermembersofmysupervisorycommittee,Prof.Renato Figueiredo,Prof.PramodKhargoneka,andProf.ZhigangChenfortakingtimeoutof theirbusyschedulestoreviewmywork.Theirinsightfulsuggestionsareveryhelpfulfor improvingmywork. MyheartfeltthanksisextendedtomycurrentandformerfellowACISlabmembers. Thelabprovidesmeapleasantplacetoconductmyresearch. AspecialnoteofgratitudeisduetoProf.Zhen'anLiu,myformaladvisoratthe UniversityofScienceandTechnologyofChina,forencouragingandassistingmeto pursuemyPh.D.overseas. Finally,andmostimportantly,Iwouldliketothankmyparents.Theirunconditional lovearealwayswithme.IoweeverythingthatIhaveachievedtothem. Therewerehappytimes,thereweretoughtimes,therewereexcitingtimes,and therewereunbearabletimes.Allthroughthis,myhusband,Ming,hasbeenwithme, encouraging,supporting,andloving. ThisdissertationworkissupportedinpartbyNationalScienceFoundation grantsNo.OCI-0721867,CNS-0540304,CNS-0821622,CNS-0855123,IIP-0758596, IIP-0932023,theBellSouthFoundation,anIntelgrantITRCouncil,andequipment awardsfromDURIPandIBM. 3

PAGE 4

TABLEOFCONTENTS page ACKNOWLEDGMENTS..................................3 LISTOFTABLES......................................8 LISTOFFIGURES.....................................9 ABSTRACT.........................................12 CHAPTER 1INTRODUCTIONANDBACKGROUND......................15 1.1Grids......................................15 1.2DataCenters..................................17 1.3AutonomicComputing.............................18 1.4ProposedApproach..............................19 1.4.1Two-LevelControlSystem.......................19 1.4.2Optimization...............................20 1.4.3DecentralizedManagementNetwork.................22 1.5Roadmap....................................23 2RELATEDWORK..................................24 2.1WorkloadManagementandJobSchedulinginClusters..........24 2.2ApplicationandResourceManagementinGrids..............25 2.3VirtualizedSystems..............................27 2.3.1VirtualizationinGridComputing....................27 2.3.2VirtualizationinDataCenters.....................28 2.4Optimization..................................30 3AUTONOMICAPPLICATIONMANAGEMENTINGRIDSYSTEMS.......32 3.1ProblemDescription..............................32 3.1.1ChallengesofResourceManagementinGrids...........32 3.1.2WorkloadsofInterest..........................32 3.2In-VIGOIntroduction..............................33 3.2.1VirtualizationinIn-VIGO........................33 3.2.2ArchitectureofIn-VIGO........................34 3.3AutonomicVirtualApplicationManagement.................35 3.3.1Two-LevelControlArchitecture....................35 3.3.2AutonomicManager..........................37 3.3.3ResourceUsageandPerformancePrediction............38 3.3.4ResourceSelection...........................40 3.3.5JobControl...............................41 3.4Evaluation....................................44 4

PAGE 5

3.4.1ExperimentalSetup..........................44 3.4.1.1Modelingnon-dedicatedresourcesingridenvironments.45 3.4.1.2Workloads..........................46 3.4.1.3Schedulingstrategies....................47 3.4.2ExperimentalResults..........................48 3.4.2.1Performanceimprovement.................48 3.4.2.2Sensitivitytoloadvariances................51 3.5RelatedWork..................................53 4AUTONOMICRESOURCEMANAGEMENTINVIRTUALIZEDDATACENTERS56 4.1ProblemDescription..............................56 4.1.1VirtualizedDataCenters........................56 4.1.2ChallengesofResourceManagementinVirtualizedDataCenters57 4.2Two-LevelResourceControl..........................58 4.2.1ApplicationServiceLevelAgreementsSLAandResourceSLA.59 4.2.2BenetsofTwo-LevelControl.....................59 4.3LocalController.................................60 4.3.1Input-OutputModelofVirtualContainer...............60 4.3.2BasicsofFuzzyLogic.........................61 4.3.3FuzzyModeling.............................62 4.3.4FuzzyPrediction............................66 4.4GlobalController................................70 4.5ExperimentalEvaluation............................72 4.5.1ExperimentalSetup..........................72 4.5.2Fuzzy-ModelingApproach.......................74 4.5.2.1StaticWebrequests.....................74 4.5.2.2DynamicWebrequests...................76 4.5.2.3Trace-basedworkload....................78 4.5.3Fuzzy-PredictionApproach......................79 4.5.4GlobalControllerValidation......................81 4.6RelatedWork..................................83 4.7Conclusions...................................85 5MULTI-OBJECTIVEOPTIMIZATIONINVIRTUALMACHINEMANAGEMENT.87 5.1ProblemDescription..............................87 5.1.1Multi-ObjectiveOptimization......................87 5.1.2VirtualMachineVMPlacementandMigration...........88 5.2InitialVirtualMachinePlacement.......................89 5.2.1Multi-ObjectiveVMPlacementDecision...............90 5.2.2BasicsofGeneticAlgorithms.....................94 5.2.3GroupingGeneticAlgorithm......................95 5.2.4FuzzyMulti-ObjectiveEvaluation...................97 5.2.5GGAwithFuzzyMulti-ObjectiveEvaluation.............99 5.3DynamicVirtualMachineMigration......................100 5

PAGE 6

5.3.1Cross-layerProling,ModelingandControlling...........101 5.3.2ConditionsforDynamicVMPlacement................102 5.3.3ControllerFunctionality.........................104 5.3.4StabilizationConsiderationsandComplexityAnalysis........109 5.4ExperimentalEvaluation............................112 5.4.1ExperimentalSetup..........................112 5.4.2ProlingandModeling.........................112 5.4.3EvaluationofMulti-ObjectiveInitialVMPlacement.........114 5.4.4EvaluationofDynamicVMMigration.................120 5.4.4.1Prototypeimplementation..................120 5.4.4.2Workloadgeneration.....................121 5.4.4.3Competingapproaches...................122 5.4.4.4Evaluationresultsandanalysis...............123 5.5RelatedWork..................................127 5.6Conclusions...................................130 6COOPERATIVEAUTONOMICMANAGEMENTINDYNAMICDISTRIBUTED SYSTEMS......................................136 6.1ProblemDescription..............................136 6.2DecentralizedAutonomicManagementArchitecture............137 6.2.1GenericAutonomicElementModel..................137 6.2.2DistributedDomainRegistry......................139 6.2.3AutonomicManagerAMNetworkBuilding.............139 6.2.4AMNetworkDynamism........................141 6.3AnalyticalEvaluation..............................142 6.3.1NetworkModel.............................142 6.3.2NodeJoiningandNeighborSelection.................142 6.3.3NodeLeavingandNeighborhoodRebuilding............143 6.3.4LocalLoadAdjustment.........................144 6.3.5CommunicationCost..........................144 6.4CaseStudy...................................145 6.4.1Background...............................145 6.4.2CooperativeAM.............................145 6.4.3Controller................................146 6.4.4MonitorandCommunicator......................148 6.4.5InformationFiltering..........................149 6.5ExperimentalEvaluation............................150 6.5.1ExperimentalSetup..........................150 6.5.2ExperimentalEvaluationofEfciency.................152 6.5.3ExperimentalEvaluationofScalability................153 6.5.4ExperimentalEvaluationofRobustness...............154 6.6Discussion...................................156 6.7RelatedWork..................................156 6.8Conclusions...................................157 6

PAGE 7

7CONCLUSIONSANDFUTUREWORK......................159 REFERENCES.......................................164 BIOGRAPHICALSKETCH................................177 7

PAGE 8

LISTOFTABLES Table page 3-1Theprobabilityofmachineidleinfourloadingscenarios.............46 5-1SymbolsusedinVMplacementproblemformulation...............93 5-2Thelistofutilityfunctions..............................133 5-3Thresholdsandwindowsizesforconditiondetection...............134 5-4Parametersetupforthreesetsofexperiments...................134 5-5ThetimingandresourceoverheadfordynamicVMmigration..........134 5-6Relatedworkondynamicvirtualmachineplacement...............135 8

PAGE 9

LISTOFFIGURES Figure page 1-1Controlloopinanautonomicmanager.......................18 3-1High-levelblockdiagramofIn-VIGO........................34 3-2High-levelviewofAutonomicVirtualApplicationManagerAVAMinIn-VIGO.36 3-3Autonomicmanagercomponentsandtheirinteractions..............38 3-4Fourscenariosofresourceload...........................45 3-5CPUtimepredictionerrorforNN-algorithmonTunProb..............46 3-6Theaverageexecutiontimeswithxedinputsinthefourloadingscenarios..49 3-7Thepercentageofjobsmeetingtheirdeadlineinthefourloadingscenarios..49 3-8Theaverageexecutiontimeswithvaryinginputsinthefourloadingscenarios.50 3-9Thepercentageofjobswithvaryinginputsmeetingtheirdeadline.......50 3-10Executiontimeandmachineload..........................52 3-11Executiontimeandloadintroductiontime.....................53 4-1Two-levelcontrolinvirtualizeddatacenters....................58 4-2Inputsandoutputsofavirtualcontainer......................60 4-3Fuzzymodelingandinferencefunctionsinalocalresourcecontroller......62 4-4Fuzzypredictionfunctionsinalocalcontroller...................67 4-5Three-inputtwo-outputdatapairexamples....................67 4-6Fuzzydomainsandtheirmembershipfunctions..................68 4-7Thefuzzy-ruleupdatingprocedure.........................69 4-8FuzzymodellearnedfromtheworkloadwithstaticWebrequests........75 4-9Comparisonofthethroughputforthedynamicworkload.............76 4-10ThefuzzymodellearnedfromtheworkloadwithdynamicWebrequests....76 4-11Comparisonofthethroughputforthedynamicworkload.............77 4-12ThefuzzymodellearnedfromtheworkloadwithdynamicWebrequests....78 4-13Comparisonofthethroughputforthetrace-basedworkload...........79 9

PAGE 10

4-14Comparisonofthethroughputbydynamicallocationandtheidealthroughput.80 4-15TheCPUallocatedtothevirtualcontainer.....................80 4-16CPUrequestsfromtwovirtualcontainers.....................81 4-17CPUallocationthatfavorsVC2...........................82 4-18CPUallocationthatfavorsVC1...........................82 5-1Two-levelcontrolarchitectureformanagingvirtualmachineplacement.....90 5-2AnexampleofresourceallocationofVMs.....................91 5-3AnexampleofVMplacementanditscorrespondingchromosome.......95 5-4Improvedgroupinggeneticalgorithmforvirtualmachineplacement.......99 5-5Cross-layercontrolarchitectureandinformationowfordynamicVMmanagement101 5-6Controlowofthecontroller.............................103 5-7PowerconsumptionwithvaryingCPUutilization.................113 5-8PowerconsumptionwithvaryingCPUutilization.................113 5-9Performancecomparisonsofsevenplacementalgorithms............115 5-10Solutionpointsobtainedfromsevenplacementalgorithmsfortwentyrandomly generated128-machine250-VMinputs......................116 5-11FitnessvalueofMGGAfordifferentvaluesofSandG..............117 5-12FitnessvalueofMGGAfordifferentvaluesofcrossoverrate...........118 5-13FitnessvalueofMGGAfordifferentvaluesofSandG..............119 5-14FitnessvalueofMGGAfordifferentvaluesofcrossoverrate...........119 5-15Monitoringdatafromthebladenodesduringanexperimentrun.........124 5-16Comparisonoftotalthermalandresourceusageviolationperiodbetween MOSapproachandnocontrol...........................125 5-17ComparisonoftheperformanceofaVMrunningSysbencholtpmodeonserver 1undernocontrol,controlusingMOSandidealcase..............125 5-18Comparisonofmulti-objectiveoptimizationwithstabilizationconsideration withfourothercompetingsolutions.........................126 6-1Anoverviewofadecentralizedautonomicsystem................138 6-2CooperativeAutonomicVirtualApplicationManagerfunctions..........146 10

PAGE 11

6-3Thecomparisonofjobruntimewithdifferentnumberofneighbors.......151 6-4Thecomparisonofjobthroughputwithdifferentnumberofneighbors......152 6-5ThecomparisonofthejobsexecutiontimewithDAVAMandcentralizedapproaches153 6-6ThecomparisonofthejobsthroughputwithDAVAMandcentralizedapproaches154 6-7TunProbjobsaverageexecutiontimebeforeandafterAVAMleaving......155 6-8TunProbjobs'throughput..............................155 11

PAGE 12

AbstractofDissertationPresentedtotheGraduateSchool oftheUniversityofFloridainPartialFulllmentofthe RequirementsfortheDegreeofDoctorofPhilosophy AUTONOMICAPPLICATIONANDRESOURCEMANAGEMENTINVIRTUALIZED DISTRIBUTEDCOMPUTINGSYSTEMS By JingXu May2011 Chair:JoseFortes Major:ElectricalandComputerEngineering Large-scaledistributedcomputingsystems,suchascomputationalgridsand enterprisedatacenters,presentcomplexmanagementchallenges.Suchsystems experienceinherentdynamismduetounpredictableresourceavailabilityandusage, or/andhighlydynamicworkloads.Byintroducingalayerofabstraction,virtualization technologyprovideswaysofprovisioningandcustomizingresourceenvironmentsas needed,andmigratingworkloadstoadapttodynamicchanges.However,thescaleof suchcomputingsystemsmakesitextremelyhardtocontrolthemmanuallybyoneor morehumanoperators. Oursolutionistoincorporateautonomiccapabilitiesintothemanagementof applicationsandresourcesingridanddatacenterenvironmentstoreducedirecthuman intervention.Suchcapabilitiesareaccomplishedthroughatwo-levelfeedback-control frameworkinwhichlocalcontrollersattheapplicationlevelhavedetailedinformation abouttheapplicationsandallowindependentadaptationandoptimization.Theglobal controllerattheresourcelevelcollectsresourceinformationandoptimizesthesystem behaviorfromaglobalperspective.Italsoactsasacoordinatorwhenconictsoccurat differentlocalcontrollers. Forgridenvironments,theproposedtwo-levelcontrolsystemisstudiedinthe contextofIn-VIGO,agrid-computingsystemthatprovidesapplicationservices on-demandusingdynamicallyinstantiatedvirtualmachines,networks,dataand 12

PAGE 13

applications.Localcontrollersutilizeapplication-specicinformationfortrackingand predictingtheperformanceofjobsexecutingongridresources,whichisthenusedto guidethescheduling/reschedulingdecisions.Itseffectivenesshasbeenevaluatedfor CPU-intensivejobswithrelativelyshortexecutiontimesrangingfromtensofseconds tolessthananhouronresourceswithhighlyvariableloads.Theresultsshowthat In-VIGOjobsmanagedbythetwo-levelcontrollersconsistentlymeettheirexecution deadlinesundervaryingloadconditionsandgracefullyrecoverfromunexpectedfailures. Underthemostdynamicandheavyloadingenvironmentcreatedbytheexperiments, theaveragejobruntimeoftheproposedapproachis10%and20%shorterthantwo othercompetingschedulingstrategies,oneusinground-robinandtheotherusingthe sameschedulingastheproposedapproachbutwithoutreschedulingactions.The percentageofjobsmeetingtheirpredeneddeadlinesisimprovedby40%and50%, respectively. Inavirtualizeddatacenter,thetwo-levelcontrolsystemisdesignedtodeliver performanceguaranteeswhileoptimizingresourceusage,andalsootherimportant aspectsofdatacenterssuchaspowerandcoolingcosts.Attheapplicationlevel, twofuzzy-logic-basedmethods-fuzzymodelingandfuzzyprediction-areproposed toestimatetheresourcedemandsfordynamicworkloads.Theglobalcontrollerat theresourceleveltriestondtheoptimalresourceallocationandvirtualmachine VMplacement/replacement,withmultipleobjectivesincludingtheeliminationof thermalhotspots,theminimizationoftotalpowerconsumption,andtheefcientuseof resources.Theproblemisposedasamulti-objectivecombinatorialoptimizationproblem andanimprovedgeneticalgorithmwithfuzzymulti-objectiveevaluationisproposed forefcientlysearchingthelargesolutionspaceandconvenientlycombiningpossibly conictingobjectives.Anonlinelocalsearchalgorithmusingmulti-objectiveoptimization andstabilizationtechniquesisdesignedfordynamicallychangingvirtualmachine placementtoquicklyadapttochangesinsystemconditionsorworkloads.Theproposed 13

PAGE 14

approachesareimplementedandevaluatedonavirtualizedtestbedbuiltuponanIBM BladeCenter.Underbothsyntheticandreal-worldWebworkloadsthelocalcontroller isvalidatedtoaccuratelyestimateresourceneedsthedifferenceiswithin5%using fuzzymodelingandfuzzypredictionapproaches.Theglobalcontrollerfordetermining virtualmachineplacementistestedwithsimulation-basedexperimentsoverawide rangeofproblemsizesandtheresultsshowthatthemulti-objectiveoptimizationusing geneticalgorithmachievegoodbalanceamongdifferentobjectives,resultinginrelatively lowvaluesforpowerconsumption,peaktemperature,andresourcewastage.Forthe dynamicvirtualmachinemigrationproblem,experimentalevaluationsareconducted usingamixoftypesofworkloadstoemulatethevarietyanddynamicsofdatacenter workloads.Theresultsindicatethattheproposedmulti-objectiveoptimizationwith stabilizationsignicantlyreducesunnecessaryVMmigrationandunstablehostselection byupto80%andalsoimprovestheapplicationperformancebyupto30%andthe efcienciesofpowerusagebyupto20%. Therapidgrowthofcomputingsystemsraisesnewchallengesforcentralized managementattheglobal-controllevelintheproposedtwo-levelarchitecture.Anetwork ofcooperativecontrollersisproposedinthiswork,eachmanagingasubsetofresources andcollectivelycollaboratingtomanagetheentiresystem.Theproposednetwork modelisvalidatedonatestbedforIn-VIGOandtheresultsshowthatthedecentralized andcooperativenatureofthesystemyieldsanumberofdesirableproperties,including efciency,robustness,andscalabilityunderahighlydynamicenvironment. 14

PAGE 15

CHAPTER1 INTRODUCTIONANDBACKGROUND Today'scomputinginfrastructuresarebecomingincreasinglylargescale.Examples includescienticgridswhichharnessresourcesfromdifferentdomainsfordistributed andcooperativecomputing,andenterprisedatacenterswhichcanpotentiallyhouse thousandsofphysicalservers.Commoninthesesystemsistheresourcesharing amongavarietyofapplicationswithdifferentresourcerequirementsandpossibly dynamicworkloads.Virtualizationoffersanewapproachtosharingresources,by allowingtheprovisioningandcustomizingofcomputingenvironmentsasneeded, andmigratingworkloadstoadapttochanges.However,thegrowingmanagement complexityduetotheincreasedsystemsizeandtheinherentdynamismexperiencedby bothsystemsposegreatchallengesinapplicationandresourcemanagementofsuch systems. Thereisagrowinginterestinintegratingautonomiccapabilitiesintocomputer systems,aimingtodevelopself-managementcapabilitytoovercomethegrowing complexityofsystemmanagement.Theworkpresentedinthisdissertationaimsto addressthechallengesofautonomicmanagementinlarge-scaledistributedcomputing systems,targetinggridanddatacenterenvironments.Inthischapter,werstintroduce thebackgroundofthesetwotypesofcomputingsystems,andtheirmanagement challengesarealsodiscussed.Thesecondpartbrieyintroducesthebasicideaof autonomiccomputingandourproposedapproachtoachieveautonomicapplicationand resourcemanagementingridanddatacenterenvironments.Thelastpartshowsthe roadmapofthewholedissertation. 1.1Grids Agridisatypeofdistributedsystemthatenablesthesharing,selection,and aggregationofresourcesdistributedacrossmultipleadministrativedomainsbasedon theiravailability,capacity,performance,costandusers'quality-of-servicerequirements 15

PAGE 16

[1].Duetothelackofcentralcontrolovertheresources,theavailabilityofgridresources cannotbeguaranteedandresourceutilizationishardtopredict,whichpreventtheuse ofgridsystemsasonelargecohesivesetofresourcesfortheusersandtheprovision ofperformanceguarantees.Thefollowingsummarizesthekeycharacteristicsofgrid environments. Heterogeneity: Ingridsystems,resourcesfromdifferentdomainstendtobe heterogeneous,intermsoftheirplatform,capacitiesandperformance.Theapplications alsohavediversecharacteristicsandresourceneeds. Dynamism: Thecomputingenvironmentingridsiscontinuouslychangingduring thelifetimeofanapplication,includingtheavailabilityandthestatesofresources.Due tothelargedecentralizedandasynchronousnatureofgridenvironments,itishardto obtainacompleteknowledgeofglobalsystemstate. Uncertainty: Thedynamicnatureofgridresourcescausesunpredictableand changingbehaviorsthatcanonlybedetectedandmanagedatruntime.Furthermore,as thescaleofsystemandapplicationincreases,theprobabilityandfrequencyoffailures increasedramatically. Manyapplicationsrunningingridshavestringentend-to-endperformance requirementsacrossmultiplecomputationalresourcesthatarepossiblygeographically separated.Forexample,educationalusagescenariosofgridsystemslikePUNCH[2] andIn-VIGO[3],offermanyexamplesoflargenumbersofusersrunningmanyrelatively smalljobsoverconcentratedperiodsoftimese.g.beforehomeworkdeadlines. Unexpectedlongrunningtimesforsupposedlyquicktasksdodiscouragefurtheruse ofagrid-computingsystem,andgenerateuserdiscontentandcustomerlosses.Inthe contextofgridsystems,theproblemaddressedinthisdissertationishowtoenablesuch workloadstoadapttohighlydynamicandfault-pronegridenvironments,forobtaining consistentperformanceandsatisfyingapplicationrequirementsautomatically. 16

PAGE 17

1.2DataCenters Adatacenterisdenedasanenvironmentallycontrolledcentralizedfacility providingbusinessservicesbysecurelydeliveringapplicationsanddataacrossa networktoremoteusers[4].Inatraditionaldatacenterenvironment[5],applications aredeployedatdifferentserverstoprovidenecessarysecurityandperformance isolation.Asmoreapplicationsaredeployed,thenumberofserversalsogrowsrapidly. Thisleadstowhatisreferredtoasserversprawl,i.e.,alargenumberofunderutilized, andheterogeneousservers. Applicationshostedindatacenterareusuallybusiness-criticalapplicationswith quality-of-serviceQoSrequirements.Suchapplicationstypicallyhavetime-varying workloadswithhighpeak-to-averageratio,resultingindynamicallychangingresource demands.Traditionalover-provisioningapproachesusedformeetingpeakdemand usuallyleadtolowresourceutilization.Inaddition,thepowerconsumptionandcooling costs[6][7][8][9]becomegreatconcernsinrecentyears.Accordingtoareportin [10],theamountofenergyusedtopowertheworld'sdatacenterserversdoubledina ve-yearspanduemainlytoanincreaseindemandforInternetservices,suchasmusic andvideodownloads. Allthesedifcultiesindatacenterresourcemanagementpromotedtheusageof virtualizationtechnologyaimingtoproducemorecost-efcientdatacenters,hereon referredasvirtualizeddatacenters.Servervirtualization[11][12]entailsthepossibility ofonephysicalserverhostingmultipleindependentvirtualmachines,andtheability oftransparentlymovingworkloadsfromonephysicalservertoanotherthroughvirtual machinemigration.Ontheotherhand,thesecapabilitiescreategreatdemandson systemmanagement,especiallyforlarge-scaleenterprisedatacentersthatcontain thousandsofserversandevenmorevirtualmachines.Thisdissertationaddressesthe questionsonhowtodynamicallyallocateresourcesamongvirtualmachinesandtheir applications,andhowtomap/remapthosevirtualmachinestophysicalservers,inorder 17

PAGE 18

Figure1-1.MAPEcontrolloopinanautonomicmanager. todeliverperformanceguaranteeswhilesimultaneouslyoptimizingresourceusage,and otherimportantfeaturesofdatacenterssuchaspowerconsumptionandcoolingcosts. 1.3AutonomicComputing Asdiscussedinprecedingsections,applicationandresourcemanagementinboth datacentersandgridsbringsgreatchallengesduetotheincreasingscaleandinherited dynamism.Usingad-hocmanualtuningperformedbyhumanoperatorsisimpractical. AutonomicComputing,asdenedbyIBMin2001[13],aimstobuildcomputersystems thatmanagethemselvesmuchinthesamewayourautonomicnervoussystem[14] regulatesandprotectsourbodies.Byintegratingmonitoring,decisionprocessingand actuationintosystemcomponents,acontrolloopprovidesthebasicbackbonestructure foranautonomiccomputingsystem. IBMrepresentsthiscontrolloopastheMAPEMonitor-Analyze-Plan-Executeloop [15].Figure1-1depictsthecomponentsandkeyinteractionsofMAPEarchitecture.The managedresourcerepresentswhatisbeingmanagedanditcouldbeanysoftwareor hardwareresource.Sensorsareusedtomeasureusageandperformanceofmanaged resources.Forexample,forawebserver,thatcouldincluderesponsetimeofclient requests,andutilizationofaserver'sCPUandmemory.Effectorsprovideaway 18

PAGE 19

tochangethebehaviorofthemanagedresources.Anautonomicmanageristhe componentthatimplementsanintelligentcontrolloop.Themonitoreddataisrstltered andcorrelatedbythemonitoringcomponent,andthereneddataisprocessedbythe analysiscomponentforthepurposessuchasforecastingandproblemdetermination. Planningcomponentconstructsanorderofactionstoaccomplishhigh-levelgoals. Theexecutecomponentcontrolstheexecutionoftheactions.Knowledgeaboutthe managedresourcesisaccessibletoallthefourcomponents. Thefollowingsectionintroducestheproposedtwo-levelcontrolsystemtoaddress thechallengesoftheapplicationandresourcemanagementinlarge-scalecomputing system.Theautonomicmanagersa.k.acontrollersimplementedinthesystemutilize methodsandtechniquesfrommachinelearning,optimization,andcontroltheoryto optimizeapplicationperformance,improveresourceuseandreducerelatedcosts. 1.4ProposedApproach Inthescenariosconsideredinourwork,themanagementofcomputingsystems typicallyinvolvestwoparties:applicationsandcorrespondingusersandresources andcorrespondingresourceowners.Themanagementgoalsforthesetwopartiesare typicallydifferent.Fromapplications'pointofview,theperformancerequirementsshould besatised.Forexample,theresponsetimeofawebrequestshouldnotexceed5 seconds.Fromtheviewpointofresourcesorresourceowners,theresourcepoolis sharedamongapplicationsandtheuseofresourcesandreturnedinvestmentshould bemaximized.Thisdissertationproposesatwo-levelcontrolsystemtoaddressthe managementobjectivesatboththeapplicationlevelandtheresourcelevel,whichallows exibleandindependentoptimizationsforbothparties. 1.4.1Two-LevelControlSystem Inthetwo-levelcontrolsystem,alocalcontrollerattheapplicationlevelhasdetailed informationabouttheapplication,whichcanbeusedtooptimizeitsownperformance, makeresourcerequestsandadapttothedynamicallychangingenvironment.Theglobal 19

PAGE 20

controllerattheresourcelevelcollectsresourceinformationandrespondstherequests fromlocalcontroller.Italsoactsasacoordinatorwhenconictsoccuratdifferent localcontrollers.Localmanagementallowsapplicationstocontroltheirbehaviorsand optimizetheirperformanceindependentlywhileglobalmanagementoptimizesthe systembehaviorfromaglobalperspective.Thistwo-levelresourcecontrolsystemis preferredoverthemoreobviouscentralizedapproachinwhichallthecontrolfunctions areimplementedatonecentralizedlocation.Theinternalcomplexitiesofcontrol functionsarecompressedbylocalcontrollersintostraightforwardresourcerequests usinglocallyavailableinformation.Moreover,itiseasytoadd,changeorremovelocal controllerswithoutaffectingtheglobalcontroller. Theproposedtwo-levelcontrolarchitectureisdevelopedinbothgridanddata centerenvironments.Chapter3explainshowthelocalcontrollerscooperatewiththe globalcontrollertooptimizeapplicationperformanceandrecoverfromperformance faultcausedbydynamicallychangingresourceusageinagridenvironment.Chapter4 and5discussestheproposedtwo-levelautonomicresourcemanagementdevelopedin avirtualizeddatacenter.Alocalcontrollerimplementedateachvirtualmachineuses adaptiveonlinelearningapproachestopredictresourcedemandsforitsapplicationand sendsresourcerequeststotheglobalcontrollertomeetitsapplicationperformance requirementsandreduceresourcecost.Theglobalcontrollerdeterminestheoptimal virtualmachineplacementandresourceallocationbytakingseveralfactorsinto considerationincludingresourceusage,powerconsumptionandcoolingcosts. 1.4.2Optimization Optimizationofmanagementrelatedmeasuresisoneofthemostimportant goalsoftheproposedtwo-levelcontrolsystem.Forexample,invirtualizeddatacenter environment,themaintaskofthelocalcontrolleristooptimizethesetofresources neededbyanapplicationrunninginthevirtualmachineinordertominimizethe resourcecostofeachindividualapplication.Theglobalcontrollertriestomaximize 20

PAGE 21

thetotalprotfromrunningusers'applicationsandtominimizethecostsfrom powerconsumption,cooling,andinefcientuseofresources.Thefollowingliststhe optimizationproblemsaddressedineachchapterfollowedbyabriefdiscussionofthe mainoptimizationapproachesusedinthetwo-levelcontrolsystem. 1.Optimizeapplicationperformanceandimprovefaulttoleranceunderadynamically changingresourceenvironment,asexempliedbyautonomicapplication managementintheIn-VIGOgridsystemChapter3:AutonomicApplication ManagementinGridSystems. 2.Optimizeresourceallocationinavirtualizeddatacentersoastomaximizethe obtainedprotwhilemeetingapplicationSLAunderdynamicallychanging workloadsChapter4:AutonomicResourceManagementinVirtualizedData Centers. 3.Optimizevirtualmachineplacementandmigrationconsideringmultipleobjectives inavirtualizedcomputingenvironmentChapter5:Multi-objectiveOptimizationin VirtualMachineManagement. Fuzzy-logic-basedOptimization Toenableautomaticandadaptiveresourceprovisioningunderdynamically chancingworkloads,twofuzzy-logic-basedmethods-fuzzymodelingandfuzzy prediction-areproposedtoguideresourceallocationbasedononlinemeasurements. Bothapproacheshaveanadaptive-learningabilitytocapturetransientorunexpected workloadchanges.Specically,thefuzzymodelingapproachcharacterizesthe relationshipbetweentheworkloadsandthecorrespondingresourcedemands,while thefuzzypredictionbuildsamappingfromrecentresourceusagetofutureresource needs.Theknowledgeobtainedbythefuzzy-logic-basedsystemcanbeeasilyupdated whennewinformationisavailabletoadapttothesystemchangesandreectthemost recentsystemconditions.Theseapproachesmakenounderlyingassumptionofthe workloadcharacteristics,andcanlearnanytypeofrelationshipveryfast.Especially, thefuzzy-logic-basedsystemcanefcientlymodelanonlinearsystemwithdynamically changingoperatingconditions[16][17][18]. 21

PAGE 22

Multi-objectiveCombinatorialOptimization Theglobalcontrollerattheresourcelevelofvirtualizeddatacentersmanagesall thevirtualmachinesanddetermineswheretoplacethem,whenandwheretomigrate themasneeded.Toaddressthecomplextradeoffsbetweensystemperformance, reliability,andcostofdatacenters,theglobalcontrollerutilizesamulti-objective optimizationapproachtosimultaneouslyoptimizemultipleobjectivesincludingefciently usingmultidimensionalresources,avoidingthermalhotspots,andreducingenergy consumption. Theproblemofvirtualmachineplacementcanbereducedtomulti-dimensional bin-packing,whichisNP-hardcombinatorialoptimizationproblem.Toefcientlysearch potentiallylargesolutionspaceforlarge-scaledatacenters,animprovedgenetic algorithmformappingvirtualmachinestophysicalhostsisproposedinthisdissertation. ThegeneticalgorithmGAisasearchheuristicwhichgeneratesolutionstooptimization problemsusingtechniquesinspiredbynaturalevolution,suchasinheritance,mutation, selection,andcrossover.Solutionsinsearchspacearerepresentedasstringsusingan encodingschema.Inourapproach,encodingandgeneticoperatorsforcrossoverand mutationaremodiedtosuitthestructureofvirtualmachineplacementproblemand improvethesearchingperformance.Anothernoveltyofourproposedapproachisthe useofafuzzymulti-objectiveapproachforevaluatingsolutionsobtainedbythegenetic algorithm.Fuzzylogicallowsthemappingofvaluesofdifferentobjectivesintolinguistic valuescharacterizinglevelsofsatisfactionandprovidesaconvenientwayofcombining conictingobjectiveswithoutspecifyingweightsorpreferenceamongdifferentobjectives [19]. 1.4.3DecentralizedManagementNetwork Theever-increasingscaleofcomputingsystemraiseschallengesforthedesign ofglobalmanagementintheproposedtwo-levelarchitecture.Thecentralizedglobal managerintroducesasinglepointoffailureandcanbecomeabottleneckinhandlingall 22

PAGE 23

informationandmanagementtasksinlarge-scalesystems.Thisdissertationadopteda solutionthatusesanetworkofcooperativelocalautonomicmanagers,eachmanaging asubsetofresourcesandapplicationsandcollectivelycollaboratingtomanagethe entiresystem.Aself-organizingmanagementnetworkisbuiltthroughthebuildingand rebuildingofadynamiclocalneighborhood,toprovidescalable,robustandefcient informationsharingforlocalmanagerstocontrol,monitorandoptimizeperformanceof thesystem. 1.5Roadmap Therestofthisproposalisorganizedasfollows: Chapter2: Thischapterreviewstheresearchworkcloselyrelatedtothis dissertation. Chapter3: Thischapterpresentsoursolutionsforautonomicapplicationmanagement inthecontextoftheIn-VIGOgridsystemtoenableself-healingandself-optimization. Chapter4: Thischapterpresentsautonomicresourcemanagementinvirtualized datacentersusingatwo-levelcontrolsystem.Thefuzzy-logic-basedapproaches areusedtoadaptivelymodelresourcedemandsfortime-varyingworkloadssoasto optimizeresourceusage. Chapter5: Thischapterpresentsoursolutionforoptimizingvirtualmachine placementandmigrationconsideringmultipleobjectives.Animprovedgeneticalgorithm withfuzzy-logic-basedevaluationapproachisproposedtondoptimalmappingof virtualmachines.Across-layercontrolapproachisusedtomanagethedynamic migrationtocopewithdynamicchangesofworkloadsandsystemconditions. Chapter6: Thischapterdiscussesdecentralizedautonomicsystemsbyutilizing cooperativelocalautonomicmanagersandself-organizingmanagementoverlays,as exempliedbyadecentralizedautonomicapplicationmanagementintheIn-VIGOgrid system. Chapter7: Thischapterconcludesthedissertationanddiscussesthefuturework. 23

PAGE 24

CHAPTER2 RELATEDWORK Theresearchworkrelatedtothisdissertationspansseveralareasincludingjob schedulingandworkloadmanagementincomputingclusters,applicationandresource managementingrids,andautonomicmanagementofresources,workloads,and power/thermalinvirtualizeddatacenters.Thefollowingbrieyreviewstheworkinthese areas.Morerelatedworkwillbediscussedindetailinthefollowingchaptersasfollows. 1.Chapter3discussestheapproachesonresource-usageprediction,resource discoveryandallocation,andjobreschedulingproposedinrelatedwork. 2.Chapter4discussestheresourcemanagementinvirtualizeddatacentersbased onrule-basedsystems,controltheory,mathematicalmodels,andreinforcement learning. 3.Chapter5presentstherelatedworkonworkload/applicationplacementonshared resourcesanddynamicresourceallocationandvirtualmachinemigrationin virtualizeddatacenters. 4.Chapter6presentstherelatedworkondecentralizedresourcemanagementbased onagent-basedsystemsandgossipalgorithms. 2.1WorkloadManagementandJobSchedulinginClusters Incomputingclusters,theworkloadmanagementsystemtypicallyinvolvesa centralizedschedulerwhichassignscomputingresourcestocomputationaltasks initiatedbyendusers[20].Byhavingfullcontroloveralltheapplicationsandresources, theschedulertakestheresponsibilityofcollectinginformationofapplicationsand resources,allocatingresources,andcontrollingjobexecution.Thejobssubmittedto thesystemmayhavevariousresourcerequirementssuchasdifferentnumberofnodes, differentprocessortypesandmemorysize.Theymayalsohaveotherconstraintssuch asjobresponsetimesorjobdeadlines.Theworkloadmanagementsoftwaremaximizes theuseofresourcestojobs,givingcompetingusers'requirementsandenforcinglocal policies.EtsionandTsafrir[21]giveashortsurveyofsomecommoncommercial workloadmanagementsoftwareforcomputingclustersincludingMaui[22],LoadLeveler 24

PAGE 25

[23],PBS[24]andLSF[25].Themostcommonlyusedschedulingalgorithmsinclude FCFSrstcomerstserved,fair-share,andbacklling.Thereareanumberof researcheffortsanalyzingjobschedulingandimpactofjobschedulingonsystem performanceforlarge-scalecomputingsystemssuchaspaper[26][27][28][29].Someof themuseuser-providedestimationofjobruntimetoimproveoverallresourceutilization [30][31].However,theseapproachesarehighlydependentontheaccuracyofuser estimatesofjobruntimes,whichhavebeenrepeatedlydemonstratedtobehighly inaccurate[32][33][34].Recently,increasingattentionhasbeenpaidtofault-aware schedulinginhigh-performancecomputing.In[35],Zhangetal.suggestutilizing temporalandspatialcorrelationsamongfailureeventsforbetterscheduling.Oliner etal.[36]presentafault-awarejobschedulingalgorithmforBlueGene/Lsystemsby exploitingnodefailureprobabilities.In[37],theproposedschedulingsystemdynamically adjustingtheplacementofactivejobsi.e.,runningjobstoavoidimminentfailures discoveredbyon-linefailurepredictors. 2.2ApplicationandResourceManagementinGrids Thetraditionalworkloadmanagementincomputingclustersismostlysystem-centric anddoesnotconsiderindividualapplicationneeds.Inaddition,itdoesnotaddressthe issueofheterogeneityandunpredictabilityinlarge-scalecomputingresourcessuch asgrids.Somerecentprojectsexploretheeffectofdifferentapplicationrequirements andapplyadaptiveschedulingmechanismsbasedonruntimeresourceavailability. Forexample,Application-LevelSchedulingProjectAppLes[38]developedan approachthatincorporatesstaticanddynamicresourceinformation,applicationanduser-specicinformationandperformancepredictionsintojobscheduling.Each applicationisttedwithacustomizedschedulingagentwhichperformsresource discoveryandselection,schedulegenerationandselection,andapplicationexecution. AgentsalsousetheservicesofferedbytheNWSNetworkWeatherService[39] tomonitorthevaryingperformanceofavailableresources.Theirapproachrequires 25

PAGE 26

acertainamountcustomworkoneachindividualapplicationandtheirperformance modelsareapplication-specicandnoteasytore-applytonewapplications.Condor [40]isahigh-throughputcomputingenvironmentthatcanmanagealargecollectionof distributivelyownedcomputingresources.TheCondorsystemsimpliesjobsubmission byactingasmatchmakerbetweenjobsandresources.Bothjobrequirementsand resourcepropertiesaredescribedusingClassAdlanguage.TheCondor'sscheduling systemcombinesdedicatedandopportunisticschedulingtoallowprogramstouse unusedcyclesonidleresources.Opportunisticschedulingassumesthatjobsmay beinterruptedatanytime,andiscapableofmigratingajobtoadifferenthost.A majorlimitationofCondoristhatthecondorcheckpointingcodemustbelinkedinwith theuser'ssourcecodewhichsometimesmaynotbeavailable.Inaddition,Condor cannotmigrateprocesseswhichexecuteforkorexec,orcommunicatewithother processes[41].GrADS[42]providesaclose-loopexecutionenvironment,withrealtime monitorsprovidingfeedbackaboutsystemperformance.Applicationsareencapsulated ascongurableobjects,whichcanbeoptimizedforexecutionongridresources.It employsanalyticalmodelsthatareconstructedsemi-automaticallyfromempirical modelshistoricaldata/sampleexecutiondata,inordertoestimatetheperformanceof aworkowcomponentonasinglegridresource.GrADSutilizesAutopilot[43]toassess application'sprogressusingperformancecontractsbetweentheapplicationdemands andresourcecapabilities.Oncethecontractisviolated,therescheduler[44]ofthe GrADStakescorrectiveactions,suchasreschedulingtheapplicationviathestop/restart approachorprocessswapping.KlausKrauteretal.[45]developedanabstractmodel andacomprehensivetaxonomyfordescribingresourcemanagementarchitectures, andsurveyeddifferentresourcemanagementapproachesinfteenrepresentativeGrid system. Toautomateapplicationmanagementingridenvironment,thisdissertation proposedtousemachinelearningapproachestoestimateresourcedemandsand 26

PAGE 27

predictapplicationperformanceusingbothhistoricaldataandonlineinformation, whichinturnguidesthefuturecontrolactionssuchasrescheduling.Itdoesnotrequire detailedapplicationinformationoruser-providedestimation.Theresourceestimation andselection,performancepredictionandjobreschedulingareperformedlocally withthefacilityofglobalscheduler,whichsignicantlyreducesthecomplexityofthe centralizedmanagement.Ourproposedworkaimstoenableautonomiccapabilities includingself-healingandself-optimizingintoageneralapplication-centricmanagement framework.Thecontextofinterestisvirtualizedgridsystemsanddatacenters,whichis discussedinthenextsection. 2.3VirtualizedSystems Virtualizationisgenerallyamethodofmakingaphysicalentityactasmultiple, independentlogicalentities.Thegoalofvirtualizationistoprovidelogicalviewand controlofphysicalinfrastructureinordertoenablegreateroptimization,utilizationand simplicationofmanagement.Thefollowingdiscusseshowvirtualizationbenetthe managementofgridsystemsanddatacenters. 2.3.1VirtualizationinGridComputing Virtualizationcangreatlysimplifygridcomputingbydecouplingthearchitecture anduser-perceivedbehaviorofresourcesfromtheirphysicalimplementations.The capabilitiesandfunctionsenabledbyvirtualizationhelphidetheheterogeneityand dealwithcomplexityinheritedingridcomputingsystems.Severalprojectshavestudied theintegrationofvirtualizationingridsystems.ThemainideaoftheIn-VIGO[3] systemistoconstructvirtualgridsoutofphysicalresourcesbyaddingvirtualization layerstothetraditionalgridmiddleware,inordertoenableon-demand,dynamicvirtual resourcesforapplication-specic,user-specicgridcomputing.TheVirtualWorkspace Service[46]exposesthefunctionalityneededtomanageworkspaces-abstraction ofexecutionenvironmentsimplementedthroughvirtualmachinesVMs.Santhanam etal.[47]consideredintegratingVMsintogridsystemsforsandboxingpurposesand 27

PAGE 28

exploreddifferentdesignofVM-enabledsandboxes.VirtualComputingLaboratory VCL[48]isanopensourceimplementationofasecureproduction-levelon-demand utilitycomputingandservicesorientedtechnologyforaccesstosolutionsbasedon virtualizedresources.Ithasawebportalwhereusersmayrequestavirtualmachine thatVCLcongureson-demand.VCLprovidesanapproachtomappingandmanaging userapplicationandserviceneedstoavailablelocalordistributedsoftwareimages andhardwareresources.PlanetLab[49]isalarge-scale,distributedplatformfornew networkservices.PlanetLabusesvirtualizationcontainers,calledslidetomanage resourceallocationandtoachieveisolationbetweenapotentiallylargenumberof long-lived,independentservices.However,itdoesnotprovideassuranceoverthe underlyingresourcesboundtoeachvirtualmachineandnonoticationswhenaslice's resourceallotmentchangesduetocontentionorfailure. 2.3.2VirtualizationinDataCenters Byintroducingalayerofabstraction,virtualizationenablesmultiplevirtualmachines torunonthesamephysicalserverwhileguaranteeingthenecessarysecurityand performanceisolation.Virtualmachinemigrationmakesitpossibletotransparently moveworkloadsfromonephysicalservertoanotherwithnoimpacttoendusers. Withne-graineddynamicresourceallocationtovirtualmachinesbeingenabled, virtualizationprovidesexibleresourcemanagement. ResourceManagementinvirtualizeddatacentershavegainedalotofattentionin bothacademicresearch[50][51][52][53]andcommercialproducts[12][54].Givenserver consolidation,exiblemanagement,andeasydeploymentprovidedbyvirtualization, manyresearcheffortshavefocusedondatacenteroptimizationsuchasmaximizing prot[52][51],andminimizingpowerandcoolingcosts[6].Virtualmachinemigration capabilityisoftenusedtoadapttodynamicworkloadchangesandimproveefciency ofresourceusage[55][56][57][58][59].However,thehighresourceoverheadand performancelossincurredbymigrationprohibitsunlimitedusageofthismechanism. 28

PAGE 29

Howtominimizetheimpactofmigrationaswellassatisfyotherobjectivesand constraintsneedstobeaddressedindatacenteroptimization.Thefollowingsection broadlyreviewstheongoingresearchworkonresourcemanagementoptimization, withfocusondatacenterenvironments.Amorecomprehensivesurveyontheuse ofoptimizationtechniquesinresourcemanagementofdatacentersispresentedin AppendixI. Controltheoryhasalsobeenappliedextensivelytocomputersystemsforresource managementandperformancecontrol.Asurveyoffeedbackperformancecontrolin variouscomputingsystemsispresentedin[60].Tomodelthedynamicsofthecontrolled systeme.g.,applicationworkloadsandtheirhostedVMs,mostpriorworkemploys anempiricalandblackboxapproachtosystemmodelingbyvaryingtheinputsin theoperatingregionandobservingthecorrespondingoutputs.However,thechoice ofmodelstructuresandmodelsizesandordersarehardtodetermine.Linearmodels areoftenappliedbecauseoftheirsimplicity,butnonlinearbehaviorsaretypically showninsuchdynamicsystems[61][18].Anadaptiveintegralcontrollerisdeveloped in[62]fordynamicsizingofavirtualmachinebasedonitsconsumptionsuchthatthe relativeutilizationoftheVMcanbemaintainedinspiteofthechangingdemand.In[63], QoS-drivenworkloadmanagementwaspresentedusinganestedintegralfeedback controller,wheretheinnerloopregulatestheCPUutilizationofavirtualcontainerand theouterloopmaintainstheapplication-levelresponsetimeatitstarget. Predictivecontrolandoptimalcontrolrecentlydrawmuchattentioninresource andpowermanagementofdatacenters.In[64],apredictivecontrollerwasdeveloped toallocateCPUresourcetoavirtualcontainerproactivelybyexploitingrepeatable patternsinanapplication'sresourcedemands.Thepredictivecontrollersusethree differentpredictionalgorithmsbasedonastandardautoregressiveARmodel,a combinedANOVA-ARmodel,andamulti-pulseMPmodel.Someworksuchas [52][65][66]formulatetheperformanceandpowermanagementindatacentersas 29

PAGE 30

dynamicoptimizationproblemwhicheitherminimizesassociatedcostsormaximizes totalprots.LinearQuadraticRegulatorLQR,ModelPredictiveControlMPC,and limitedlookaheadcontrolLLCareusedtosolvetheoptimizationproblem. Inthisdissertation,fuzzy-logic-basedmodelingandpredictionapproachesareused toestimatedynamicapplications'resourceneedsinordertooptimizeresourceusage ofindividualvirtualmachineseeChapter4.Comparedtothecontrol-basedapproach, thefuzzycontrolapproachcanefcientlydescribedynamicnonlinearsystembehavior withoutinvolvingheavycalculationsormakinganyassumptionsandsimplication. 2.4Optimization Decision-makingproblemsoftenappearinsystemmanagementresearchincluding resourceprovisioning,resourceallocationandapplicationplacement.Theseproblems arecharacterizedbyalargespaceofpotentialsolutions,withcomplextradeoffs betweensystemperformance,reliability,manageabilityandcost.Someresearch e.g.,[55][56][57][58]addressesresourceandworkloadmanagementasoptimizationor constraintsatisfactionproblems,allowingtheuseofoperationsresearchORsolution techniques,suchasmathematicalprogrammingandmeta-heuristics. Dynamicresourceallocationtoapplicationsinordertoadapttoworkloadvariations hasbeenstudiedinmanyresearcheffortsincluding[67][68][69][70][71][50].Tohandle resourcecompetitionamongmultipleapplications,utilityfunctionsareusedin[72][7][50] torepresentthevalueofresourcesfordifferentapplications.Thegoalistomaximize theglobalutilitybyallocatingresourcesamongcompetingapplications.Similarly,some worksuchas[52][6]translatesallocatedresourcesintobusinessvaluesuchasrevenue orpenaltyformeetingorviolatingpredenedServiceLevelAgreementSLA.The optimizationproblemistomaximizethetotalprotwhichisequaltotherevenueminus theassociatedcostsandpenalties.Zhangetal.[67]andZhuetal.[68]formulated resourceallocationandassignmentascombinatorialoptimizationproblemsandshowed thattheyareNP-hardproblems.Intelligentsearchsuchasgeneticalgorithms,simulated 30

PAGE 31

annealing,tabusearchandotherproblem-specicheuristicalgorithms[19]areapplied tosolvethistypeofproblems. Everincreasingpowerdensityanddissipationindatacentershasturnedenergy managementintoakeyconcern.Manyresearcheffortshaveattemptedtocontrol poweratdifferentlevels.Atthenestgranularity,CPUvoltageandfrequencycanbe manipulatedtoconservepower.Thissocalleddynamicvoltage/frequencyscaling DVFStechniquehasbeenwidelyusedinbothembeddedandgeneral-purpose systems.In[6],aDVFSpolicybasedonfeedbackcontrolandqueueingtheoryis presentedforminimizingoperationalcostswhilemeetingtheSLAs.In[73][74], performancemanagementandpowermanagementarecoordinatedtoachieve speciedpowerandperformanceobjectives.Alotofeffortsalsohavebeenmadeto addresspowerefciencyatthelevelofserverclustersordatacenters.In[7][75][76], consolidatingworkloadsandturningoffunloadedserversareusedtoachieveenergy savings. However,mostworkhasfocusedonoptimizingonlyoneoratmosttwomanagement aspectsofdatacenters,byseparatelymanagingeithertheplatformlayere.g.,power andthermalmanagementorthevirtualizationlayere.g.,VMprovisioningand migration,applicationperformancemanagement.Theproposedtwo-levelcontrol systemseeChpater5utilizesinformationfromboththevirtualizationandtheplatform layers,forsimultaneouslyoptimizingmultipleobjectivesincludingmakingefcient usageofmultidimensionalresources,avoidingthermalhotspots,andreducingenergy consumption. 31

PAGE 32

CHAPTER3 AUTONOMICAPPLICATIONMANAGEMENTINGRIDSYSTEMS Thischapterpresentsourworkonapplyingautonomicmanagementtogrid middlewaretoachievefault-toleranceandimproveapplicationperformanceon non-dedicatedresourceswithvaryingloads.Thisworkisdevelopedandimplemented onatestbedforIn-VIGO,agrid-computinginfrastructurecreatedbytheACISLaboratory. 3.1ProblemDescription Intraditionalcomputingsystems,resourcemanagementandjobscheduling havebeenextensivelystudied.Themanagementsystemcomponentssuchas workloadschedulers,resourcemanagerandworkowengines,haveahugevariety ofimplementations.Thesesystemsaretypicallydesignedandoperatedinarelatively staticenvironmentoverwhichthemanagerhasacompletecontrol.Themanager canimplementthemechanismsandpoliciesasneededforefcientuseofthe resourcesunderhis/hercontrol.However,thisdoesnotapplytogridsystemsin whichmanagementmustdealwiththeheterogeneity,dynamism,anduncertaintyof gridresources. 3.1.1ChallengesofResourceManagementinGrids Gridcomputing[77]enablesuserstoshareresourcesdistributedacrossadministrative domains.Idleandlow-prioritysharedcyclesfromnon-dedicatedmachinesforman importantclassofgridresources.Twodistinguishingfeaturesofsuchresourcesare,a intermittentparticipationeithervoluntaryorduetofailure,andbhighlyvariableload duetotheirnon-dedicatednature.Thislackofperformanceguaranteesmakesitdifcult toexploitsuchresourcestosupportworkloadsthatrequireaspeciedqualityofservice QoS. 3.1.2WorkloadsofInterest Theworkloadsofinterestinthisworkarescienticinnature,batch-orientedand characterizedbyrelativelyshortexecutiontimesrangingfromtensofsecondstoless 32

PAGE 33

thanonehour.Educationalusagescenariosofgrid-computingsystemslikePUNCH[2] andIn-VIGO[3],offermanyexamplesoflargenumbersofusersrunningmanyrelatively smalljobsofscientictoolsoverconcentratedperiodsoftimese.g.beforehomework deadlines.Inspiteoftheshortdurationofthesejobs,theyplayanimportantrolein shapingtheuserexperienceswithagridsystem.Longresponsetimesforshortjobsare lesswelltoleratedthanlongexecutiontimesfortoolsknowntotakealongtimetorun. Unexpectedlongrunningtimesforsupposedlyquicktasksdodiscouragefurtheruseof agridsystem,andgenerateuserdiscontentandcustomerlosses.Ourgoalistoenable suchworkloadstoadapttothehighlydynamicandfault-pronegridenvironmentsthat resultfromsharingnon-dedicatedresources. Thefollowingsectionsdescribeanapproachtoautonomiccomputingthatis application-centricandleveragestheuseofvirtualizationtechnologyingridcomputing. TheideashavegeneralapplicabilitybutthecontextisthatofIn-VIGO,agridsystem thatprovidesapplicationserviceson-demandusingdynamicallyinstantiatedvirtual machines,networks,dataandapplications. 3.2In-VIGOIntroduction In-VIGO[3]In-VirtualInformationGridOrganizations,isagrid-computing infrastructuredesignedtosupportcomputationaltoolsforengineeringandscience researchongridresources.In-VIGOhasbeenconguredandtestedtorunavariety ofapplicationsinmaterialscience,bioinformatics,andcomputerachitectureetc.Its distinctivefeatureistheextensiveuseofvirtualizationtechnologiestoprovidesecure executionenvironmentsasneededbytoolsandusers.Adetaileddiscussionofits designandimplementationappearsin[3]. 3.2.1VirtualizationinIn-VIGO TheIn-VIGOarchitectureisbuiltuponcomponentsthatallowforvirtualizationof gridresourcesanduserinterfaces.Avirtuallesystemfacilitatesdatatransferand accessacrossgirdresources.Virtualmachinesprovideisolatedandcustomizable 33

PAGE 34

Figure3-1.High-levelblockdiagramofIn-VIGOshowinguse-relationshipsandcontrol interactionsamongitsmajorcomponents executionenvironmentforapplications.VirtualapplicationsVAPsallowmodication andextensionofanycomputationaltool'sbehavior.Bywrappingtheactualtoolswith additionalsoftware,theIn-VIGOVAPsenablecustomizationofaninterfacetoallowa physicalapplicationtobeseenbydifferentusersasmultipledifferentapplicationswith differentcapabilities. 3.2.2ArchitectureofIn-VIGO ThemajorcomponentsofIn-VIGOareshowninFigure3-1.Usersinteractwiththe middlewareviaaweb-browser-basedportal.Typically,auserinitiatesanapplication session,torunoneormoreinstancesofatoolongridresources.Eachapplication sessionismanagedbyaninstanceofaVirtualApplicationManagerVAMcomponent dynamicallycreatedfortheapplication.Theuser'sactions,forexamplesettingupinput parameters,importingdatales,executingtoolsandretrievingresults,aredirectedvia theUserInterfaceManagerUIMtotheVAM.TheVAMinteractswiththeResource ManagerRMtolaunchthenecessarytoolexecutionsontheavailableresources.The RMcomponentobtainsthestatusofavailableresourcesfromtheInformationSystem IScomponent,andallocatesresourcesforatoolexecution. 34

PAGE 35

Theresourcemanagementandjobschedulingfunctionsaremainlyimplemented intheVAMandRMcomponents.AVAMmanagesallactionsassociatedwithuser requestsduringanapplicationsessionandrespondstotheserequestsbyrequesting theRMtoprovidethenecessaryresourcesandlaunchingthetasks.AVAMhas session-specicinformation,e.g.inputparametersspeciedbytheuser,aswell asapplication-specicknowledge,i.e.thelogicneededtorespondtouseractions, speciedinanapplicationcongurationle.TheRMcomponentthatmanagesjob executionsonresourcesisanapplication-neutralgenericinterfacetoheterogeneous resourceAPIs. TheuseofvirtualizationinIn-VIGO,whichenableson-demandcreationofvirtual resourcesforapplication-speciccomputations,hasmanyadvantages.However,our experiencesshowthattheexistingIn-VIGOsystemstilllacksrobustandfault-tolerance featuresduetouctuationsofresourceavailabilityandusage.Investigationsof dependablecomputingtechniquesinthecontextofgridcomputing[78][79],have shownthatthepracticalwayofhandlingfaultyresourcesistomaketheclient,i.e.the submittingendpoint,responsiblefortheprogressandfailurehandlingofapplications. InIn-VIGOitistheVAMtheapplicationjobmanagementcomponent,thatthe proposedapproachaimstomakeautonomic,i.e.self-optimizingandself-healing. 3.3AutonomicVirtualApplicationManagement Thefollowingsectiondescribestheproposedtwo-levelcontrolapproachto autonomicapplicationmanagement,andtheimplementationviaextensionstoVAMsin In-VIGO. 3.3.1Two-LevelControlArchitecture Figure3-2providesahigh-levelviewofatwo-levelcontrolsysteminIn-VIGO.At theapplicationlevel,aVAMiscreatedforeachapplicationsessiontoimplementthe application-speciclogicrequiredtorespondtouser'srequests.EachVAMsessionis monitoredandcontrolledbyitsautonomicmanagerAM.InformationusedbyAMis 35

PAGE 36

Figure3-2.High-levelviewofAutonomicVirtualApplicationManagerAVAMin In-VIGO.ThisgureshowsmultipleVAMsthatconnecttotheResource Managertosubmitjobsonmultiplemachines. storedintheper-VAMknowledgebasealsocalledLocalKB.Thisincludesuser-and application-specicinformatione.g.expectedperformanceandinputparameterswhich isprovidedbytheusersthroughUIM,anddynamicresourceandjobstatusinformation monitoredbythejobandresourcesensors. Attheresourcelevel,theresourcemanagerRMcollectsresourceusage datasuchascurrentCPUloadandfreememorysize,measuredbyamonitoring daemononeachmachineandstoresittotheglobalknowledgebaseGlobalKBin Figure3-2.Besidesresourceinformation,theGlobalKBalsomaintainsapplications' executionhistorythatcanbelaterretrievedbyanAMtoestimateresourcedemands forapplicationexecutions.AglobalschedulerisapartoftheRM,assistingeachAMto discoverandallocatethemostappropriateresourcesforagivenjob. RatherthandirectlyforwardingapplicationtaskstotheRMcomponenttodetermine theresourceallocation,anAMmanagesresourceallocationlocallyfortheuser'sjobs, 36

PAGE 37

byinterpretingthecurrentexecution-specicinformationstoredintheLocalKBandthe applicationexecutionhistoryintheGlobalKB.Withthistwo-levelcontrolarchitecture, applicationsmakeschedulingdecisionsthemselves,adaptingtotheavailabilityof resourcesandoptimizingtheirownperformancewhiletheRMmaintainstheglobal resourceinformation. Thenextsectionexplainsindetailhowself-optimizationandself-healingare achievedbyanAManditsinteractionswiththeresourcemanager. 3.3.2AutonomicManager EachAMconsistsoftwocomponents,thelocalper-VAMresourcecoordinator andthelocaljobcontroller.Thelocalresourcecoordinatorusesalearningalgorithmto predicttheresourceusageforeachgiveninput-specicrunfrompreviousruns'history recordsextractedfromtheGlobalKB.Basedontheuser'srequirementsandpredicted resourceusage,thelocalresourcecoordinatorallocatestheproperresourcesforthejob bycooperatingwiththeglobalresourcescheduler.TheallocationofresourcesbyanAM fortasksallowsoptimalapplication-specicchoicefromthepoolofavailableresources. Thelocaljobcontrollerisresponsibleforcontrollingthejobexecutiontoachieve reliableandoptimizedperformance.Atthetimeofjobsubmission,ajobsensoris invokedonthechosenmachinebythejobcontrollertomonitorthejobstatus.Basedon themonitoredjobstatusandthepredictedresourceusage,thecontrollerdetermines theprogressofagivenjobrunandcomparesitwiththeuser'sQoSrequirementsto decidecontrolactionssuchasreschedulingthejobtootherresources.Theactual submissionandmanagementofajobontheselectedresourceisaccomplishedvia theRMcomponent,whichvirtualizesthelespaceoftheapplicationviatheGVFS componentseeFigure3-1.Eachtaskgetsitsowncopyofthelesystemtostoreits persistentstate,i.e.leinput/output.ThismakesIn-VIGOjobsidempotent,andasa resultthereisnoneedtorollbackjobstotheirinitialstatewhenreschedulingthemto alternateresources. 37

PAGE 38

Figure3-3.ThemajorcomponentsofAVAMandtheirinteractions. Figure3-3illustratesthemajorfunctionsofthelocalresourcecoordinatorand jobcontrollerinanAMandtheinteractionsamongthesefunctionsandwithother components.Thelocalresourcecoordinatorimplementsthreefunctionsnamed predict, evaluate ,and assign .Thefunctions analyze,monitor ,and execute areimplementedin thelocaljobcontroller.WhenusersexecutetoolsinIn-VIGO,theymayinputvarious parameters,requestdifferentresourcesandoptionallysetdifferentperformance expectationse.g.theexecutionofthegivenjobshouldnishin10minutes.Tochoose theappropriateresources,itisnecessarytoknowthespecicrequirements,suchas howmuchmemoryandCPUareneededforagivenrun.However,usersareusuallynot abletoprovideaccuratevaluesfortheseproperties.Itistheroleofthepredictfunction inanAMtoestimatetheresourceutilizationofanygivenjob,andhencefreeusersfrom providingsuchestimates. 3.3.3ResourceUsageandPerformancePrediction Inordertodeliverapplicationperformancesatisfyingusers'requirements, performancepredictionsuchasjobruntimeprovidescriticalinformationforjob 38

PAGE 39

schedulingdecisionsingridsystems.Alargebodyofresearchworkhasbeendoneon performanceandresourceusagepredictionsoncomputersystemse.g.,[80][81][82]. Thefollowingsummarizestwotypesofapproachesforperformanceprediction. Model-basedprediction: Workin[83][84]proposedaframeworkforHPCjob performancepredictionbasedonapplicationsignatureandmachineprole.An application'ssignatureisadetailedsummaryofoperationscarriedoutbytheapplication independentofanyparticularmachine,forexample,patternsofmemoryusageand communication.Amachineproleprovidestheratesatwhichamachinecanperform basicoperations.Anapplication'sperformanceispredictedbymappingitsapplication signatureontoamachineprole.Thisapproachrequiresdirectknowledgeofinternal designoftheapplicationandmachinearchitecture,althoughhighlyaccuratepredictions maybeachievedwiththedetailedinformationandanalysis. Historicaldata-basedprediction: Thisapproachderivesapplicationpredictions fromhistoricalobservationsofpreviousapplicationruns.Dindaetal.[85]describedand evaluatedasystemthatisusedtopredictrunningtimeofacomputation-boundtask onatime-sharedsystem,basedontheassumptionthatexecutiontimeofsuchtasks islineartotheloadofthehostingmachine.Itisshownthat,amongdifferentpredictive approaches,thesimplelineartime-seriesmodelsarethemostappropriateforhost-load predictionintermsofpredictivepowerandlowoverhead. Anothereffortusinghistoricaldatatopredictapplicationruntimeisbasedon similarityofdifferentjobruns.Byidentifyingsimilarjobswithcertainjobattributes, theseapproachesapplystatisticalmethodstogeneratepredictions.Locallearning techniqueshavebeeninvestigated,whichusehistoricaldatapointstobuildalocal modelforapproximation.Thekeyistodeneadistancemetrictomeasurenearness betweendatapointsfordescribingthejobs. Inourcase,oneveryimportantfactorthatmustbeconsideredinchoosingthe predictionmethodisitsoverheadsincethejobsofinterestarerelativelyshortlived.In 39

PAGE 40

ordertoavoidhighoverhead,thepredictfunctionisimplementedusingamemory-based learningalgorithm.In[81]threelearningalgorithmsareevaluated,andtheresults indicatethenearest-neighboralgorithmisthemostefcientone.Thisalgorithmis usedinthepredictionfunctionoftheresourcecoordinator.Thebasicideabehind thisalgorithmisreviewedinthenextparagraphsee[81]foralongerdiscussionand possibleimprovements. AllthejobexecutionhistoryisstoredintheGlobalKBthatcanbelaterqueried forestimatingresourcerequirementsandjobexecutionprogress.Theresourcese.g., CPUcyclesandmemoryconsumedbyaparticularjobrunoftendependontheinput parameterssuppliedtoaspecictool.Therefore,thesimilarityoftwojobrunsofa toolisdenedbythedistancemetricoftwosetsofinputvalues.Thedistancebetween agivenquery-pointandadata-pointintheinputspaceiscomputedasfollows.For astring-type,weuseexactmatch,andforavalue-type,wecomputethenormalized distanceusingthestandardEuclideanmetric.Thepredictfunctionappliesthefollowing steps: 1.Giventheapplicationtool'snameandinputvector V ,anSQLquerystatementis created. 2.TheSQLqueryisforwardedtotheGlobalKBwhichkeepstherecordsofthetool's previousruns.Foreveryresultthatsatisesthequery,thedistanceiscomputed. 3.Thenearestneighbortothegivenrun'sinputparametersisextracted,andthe wholerecordisselectedfromtheGlobalKBincludingresourcerequirementsuch asCPUcycles,memoryusageetc. 3.3.4ResourceSelection Afterthepredictfunctionestimatestheresourcerequirementsforaparticular jobrun,itqueriestheglobalschedulerforresourcesbasedonthepredictionand user-specicrequirementse.gaLinuxRedHat5.0systemwithatleast100MBof memory.Ifthereareresourcessatisfyingtherequirements,theglobalschedulerputs theminacandidatemachinelistandalsofetchestheirpropertiessuchasCPUspeeds, 40

PAGE 41

anddynamicstatussuchascurrentloadetc.whichareupdatedperiodicallybythe resourcemonitordaemonsseeFigure3-3.Thislistofresourcesalongwiththeirstatus informationisthensentbacktothelocalresourcecoordinator. The evaluate functiondetermineswhichmachineinthelistismorepreferablebased ontheresourceusagepredictionofthegivenjob.Forexample,ifthejobisI/Ointensive, aresourcethatisclosetotheleserverispreferable.ForCPU-intensivejobs,our goalistondtheresourceonwhichthegivenjobcanrunfast.Runtimedoesnotonly dependonamachine'sCPUspeed,butalsoisstronglyrelatedtothemachine'sload [81][85].Thereforeweusethefollowingequationstoestimateajob'sexecutiontimeon eachmachinebyassumingthatthecurrentloadofthemachinedoesnotchangeina relativelyshortperiodoftime. CPUtime = CPUcycles CPUspeed Executiontime = CPUtime + Load Usingtheaboveequations,theevaluatefunctioncalculatesthepredictedjob runtimesoneachmachineinthecandidatelist,andputsthebestonesinasorted preferencelist. 3.3.5JobControl Afterchoosingthebestresourceforajob,theassignfunctionisresponsible forsubmittingthejobtothechosenresource,andstoringthisjob'sinformationthe executingmachine'snameanditsCPUspeed,thejob'ssubmissiontime,andthe predictedexecutiontimeetc.intheLocalKB.Althoughtheresourcemanagerchooses themachinethatlooksthebestatthetimethejobissubmitted,themachine'sstatus suchasCPUloadmaychangedramaticallyduringthejob'sexecution.Ajobsensoris alsoinvokedbytheassignfunctiontomonitorthejob'sstatusonthemachinewhere thejobislaunchedandreportsthemtotheLocalKBataconstantinterval.Thejob 41

PAGE 42

sensorisimplementedbyascriptthatmeasuresprocess'sCPUtime,elapsedtime, CPUpercentage,andmemoryusageusingtheUnixutility ps SinceaVAMmaysubmitmultiplejobsonbehalfofauser,thejobcontrollerkeeps trackofeachjobinahashtable.Themonitorfunctioncheckseveryjobinthetable periodicallybyfetchingtheinformationfromtheLocalKB.First,thecontrollerchecks whetherthejobisrunning.Ifthejobnishessuccessfully,thecontrollercollectssome statisticdataaboutthisexecution,e.g.theapplication'sinputparameters,performance andresourceusages,andreportsittotheGlobalKnowledgeBaseasforhistorical records.Ifthejobisstillrunning,thecontrollerfetchesallthemonitoreddataabout theresource'sloadconditionandthejob'sprocesses,andthenestimatesthejob's progressbasedonageneralperformancemodel,explainedinthenextparagraph.If thecontrollerdetectsanyabnormalbehaviorsuchasmachinefailure,jobhanging, orintolerableperformance,itasksthelocalresourcecoordinatortoquerytheglobal scheduler;andthenittriestoallocateanotherresourceonwhichthejobcanstillnish beforetheuser-specieddeadline.Thejobreallocationalsousesthemechanism describedabove.Thenthecontrollerstopsthecurrentjobandresubmitsittoother resources. ForCPU-intensivejobs,howmuchCPUtimeisusedislargelyindependentfrom machineload.Therefore,CPUtimeischosenasthecriteriontomeasurehowjobs progress,giventhatthevaluesofCPUcyclesandCPUspeedareavailableintheLocal KB.Thefollowingperformancemodelisusedtoestimateajob'sexecutiontime: Runtime = CPUtime CPUpercentage whereCPUpercentagerepresentshowmuchavailableCPUtimecurrently consumedbythegivenjob,whichisupdatedbythejobsensor. Theanalyzerperiodicallycalculatestheremainingexecutiontimeonthecurrent machineusingthefollowingequationstopredictwhetherthisjob'srunwouldmeetthe 42

PAGE 43

expecteddeadline,where ElapsedCPUtime,ElapsedRuntime aretheCPUtimeand runtimealreadyconsumedbythejobrespectively,andthe TotalCPUtime and Total Runtime arethetotalCPUtimeestimatedbythepredictfunctionandthetotalexecution timecalculatedforthisrun: 8 > > > > < > > > > : UnfinishedCPUtime = TotalCPUtime )]TJ/F23 11.9552 Tf 11.956 0 Td [(ElapsedCPUtime UnfinishedRuntime = UnfinishedCPUtime CPUpercentage TotalRuntime = ElapsedRuntime + UnfinishedRuntime Iftheanalyzefunctiondetectsthatthemonitoredjobcannotnishbeforethe deadlinebycomparingthepredictedtotalexecutiontimewiththedeadlinesetbythe user,thejobcontrolleraskstheresourcecoordinatortoallocateabetterresource thatcansatisfytheuser'sgoalorbringaboutbenetsduetorescheduling.Sothejob controllerlooksforaresourcethatsatisesoneofthefollowingconditions: 1. ElapsedRuntime + Overhead + TotalCPUtime + Load
PAGE 44

hanging.Thedifcultyofdetectingsuchconditionisthatthejobsensormaystillbe abletocollectjob'sstatusandreportittotheLocalKnowledgeBase,althoughthejob isnolongermakinganyprogress.Sinceweareabletoestimatethejobexecutiontime withtheperformancemodeldescribedabove,atimethresholdtodetectjobhangingis settothreetimesthatofpredictedexecutiontime. Inordertoreducetheoverheadcausedbyqueryingthelargehistoricaldatain GlobalKnowledgeBase,historicalrecordsarecachedin-memory.DuringaVAM session,theuserislikelytosubmitmultiplesimilarjobs,i.e.thesameapplication withsimilarinputs.Hencewiththelocalrecords,theautonomicmanagercanpredict thesejobs'resourceusagesandperformanceveryquickly.Furthermore,itcanmark theresources'qualitybasedonthepreviousexperiencessothatitcanpreferthegood resourcesandavoidthebadonesforfuturejobs. 3.4Evaluation TheAVAMimplementationinIn-VIGOisevaluatedbyansweringthefollowing questions: 1.Canitefcientlyrespondtothedynamicresourceinformationandutilizeitto achievetheexpectedexecutionperformance? 2.Canitpreventandrecoverfromjobsubmissionandexecutionfailures? 3.4.1ExperimentalSetup TheevaluationexperimentswereconductedonatestbedoftheIn-VIGOsystem. ThecomputeresourcesconsistoftwoVMwareGSXserver2.5-supportedsingle processorvirtualmachines,hostedonaclusterof32Xeon2.4GHzprocessorswith 1GBmemoryand18GBdiskstorage,andaphysicalmachinewithdual927MHz PentiumIIIprocessors,512MBmemoryand23GBdiskstorage,allrunningRedhat7.3. TunProbNumericalCalculationoftheTransmissionProbabilityforOne-Dimensional ElectronTunneling,atoolinstalledinIn-VIGO,isusedastheworkload. 44

PAGE 45

Figure3-4.Averageandstandarddeviationofloadfortheunloaded,lightlyloaded ,loaded,andheavilyloadedresourcescenarios. 3.4.1.1Modelingnon-dedicatedresourcesingridenvironments Inourevaluation,backgroundloadonnon-dedicatedresourcesinagridenvironment isgeneratedbyCPU-intensiveprocesses,whoseinter-arrivaltimesandruntimesare bothmodeledasPoissonprocesses.Bychangingtheratiooftheprocesses'runtimes totheinter-arrivaltimes,wecreatedfourdifferentCPUloadsfromunloadedtorelatively heavyloadedseeFigure3-4.Lightloadisgeneratedwithprocesseswhoseaverage inter-arrivaltimessareroughlyequaltotheruntimess.Themediumloaded resourcerunsprocesseswhoseaverageruntimessarethreetimestheaverage inter-arrivaltimes.Theheavilyloadedresourcerunsprocesseswhoseruntimes sareroughlysixtimestheinter-arrivaltimes.Figure3-4showstheaveragesand standarddeviationsoftheloadsforthefourscenarios.Forthelasttwoscenarios,the standarddeviationsarerelativelylarge,indicatingthegeneratedloadsarealsohighly dynamic. Gridenvironmentstypicallyhavehundredsofnon-dedicatedresources.The followinganalysisdeterminesthechancesofndinganidlemachineinagroupof 2,16,32and256machinesarticiallyloadedwithCPU-intensivejobs.Wemeasure themachines'loadsoveraperiodoftimeinthefourscenariosdescribedabove, 45

PAGE 46

Table3-1.Theprobabilityofatleastonemachinebeingidleorhavingalightloadinfour loadingscenariosP1:Probabilityofatleastonemachineidle;P2:Probability ofatleastonemachinehavingaloadbelow1;P3:Probabilityofatleastone machinehavingaloadbelow2. Arrival RuntimeAvg load P1-P2-P3P1-P2-P3P1-P2-P3P1-P2-P3 ratemachinesmachinesmachinesmachines 0001-1-11-1-11-1-11-1-1 1/15s14s1.50.65-0.96-0.991-1-11-1-11-1-1 1/12s36s3.00.13-0.48-0.800.60-0.97-10.82-1-11-1-1 1/9s60s5.30.005-0.05-0.130.04-0.23-0.640.11-0.36-0.790.35-0.97-1 Figure3-5.ThepercentageofCPUtimepredictionerrorofnearest-neighboralgorithm withaincreasingnumberofTunProbruns. fromunloadedtorelativelyheavilyloaded,andtheresultsrevealthatthereisahigh possibilityofatleastonemachinebeingidleseeTable3-1.Whilethisanalysisis simplistic,itsupportstheintuitiveassumptionthatinarealsystemwithlargenumbers ofmachines,theprobabilityoftheavailabilityofanidlemachineatanygiventimeis high,providinganopportunitytoavoidperformancefailures.Onthisbasis,wemodel thenon-dedicatedresourcesinagridsystemwithtwoloadedmachinesandoneidle machinetoevaluatethemechanismsproposedinthispaper. 3.4.1.2Workloads TunProbisusedasanapplicationbenchmarkrepresentativeofCPU-intensive workloadswithshortexecutiontimes.TunProbrequiresfourinputparameters representingthedesiredone-dimensionaltunnelingbarrier,minimumandmaximum 46

PAGE 47

energiesandthenumberofenergystepsbetweenthemforwhichthetransmission probabilitymustbecalculated.TunProbisCPU-intensive,anditsresourceusage dependsonthreeoftheparameters,theminimumandmaximumenergyandthe numberofenergysteps,whichcanthereforebeusedforperformancepredictionviaa nearest-neighborlearningalgorithmevaluatedintherelatedwork[81].Thealgorithm usesdistancemetric P x q )]TJ/F23 11.9552 Tf 11.955 0 Td [(x i 2 x q isthequerypointand x i istheexistingdata point,formeasuringthedistancebetweentwoinputs.Figure3-5showsitsprediction accuracyforTunProbjobs,executedwiththeminimumenergy,maximumenergyand numberofenergystepsrandomlyselectedfrom0to10,0to1000,and0to1000000 respectively.Thenearest-neighboralgorithmlearnsfastandthepercentageoftheCPU timepredictionerrordropsbelow15%afterahundredruns.TheAVAMimplementation allowsplugginginofper-applicationperformancepredictors,andtheaboveresults indicatethatanearest-neighborlearningalgorithmbasedonthedistancemetricis suitableforTunProb. 3.4.1.3Schedulingstrategies ThreedifferentstrategieswereusedandcomparedforschedulingofTunProbjobs ontheIn-VIGOresources. Round-robin: Theround-robinstrategydoesnotconsidertheresources'load statusandsubmitsthejobsinaround-robinmannertothetwoloadedmachines. Bestwithoutrescheduling: Thebestcandidatewithoutreschedulingstrategy usesdynamicresourceinformationtochoosethebestresource,i.e.themore lightlyloadedofthetwoloadedmachines,forthejobduringjobsubmission.Itdoes nothowever,monitorthejob'sprogressortakeanyfurtheractions. Bestwithrescheduling: Thebestcandidatewithreschedulingstrategyinitially schedulesthejobonthebestresource,monitorsitsexecutionbehavior,predicts itsprogressandtakesappropriateactions,i.e.reschedulethejobtothelightly loadedmachineifitsuffersfromaperformancefault. ThejobQoSrequirementsmanagedbyCondition1and2seeSection3.3.5are aajobruntimedeadlinenolargerthan5timesofitsruntimeonanunloadedmachine, 47

PAGE 48

andbajobisrescheduledifitcanimprovecurrentperformancebyatleast30%.The alarmthreshold/counterforestablishingthatajobhasfailedissetat6.TheAVAM isevaluatedbycomparingtheruntimeofjobswiththedifferentschedulerstrategies infourloadedresourcescenarios.Wemeasuredtheperformanceimprovementdue totheAVAMwhenusingthebestcandidatewithreschedulingstrategy.Performance improvementduetotriggeringofCondition2,measuresAVAM'sabilitytorecoverfrom jobsubmissionfailures. 3.4.2ExperimentalResults 3.4.2.1Performanceimprovement Intherstsetoftheexperiments,wextheinputsvaluesofthebenchmark applicationasfollows:theenergystepis100000,theminimumenergyis2andthe maximumenergyis200.Theaverageruntimeofjobswiththeseinputparametersis about20sontheunloadedvirtualmachinesandabout40sontheunloadedphysical machine.RuntimeincreasesapproximatelylinearlywiththeCPUloadintroducedtothe machines.Thejob'sdeadlineissetto100s.Atthejobsubmissiontime,thetwovirtual machineswiththebackgroundloadareavailableresourcesforthescheduler.After thejobsubmission,theidlephysicalmachinealsobecomesavailableasacandidate resourceforrescheduling. Foreachloadingscenario,ftyrunsofTunProbweresubmittedtothetestbed continuously.Figure3-6showstheaverageruntimeandthestandarddeviationsof executingthebenchmarkwiththreedifferentstrategies.Figure3-7indicatesthatthe percentageofthejobsthatmeetthedeadlineof100sforthethreestrategies.Wecan observethatwhentheloadontheresourcesislight,allstrategiesworkwellbecause mostoftheresourcescansatisfythejobs'requirements.Whentheloadbecomes heavierandmoredynamic,thebestcandidatestrategiesperformmuchbetterthan theRound-robinstrategy,whiletheonewithreschedulingsubstantiallyoutperforms theonewithoutrescheduling.Thereasonsbehindtheobservationsare:rst,thebest 48

PAGE 49

Figure3-6.TheaverageexecutiontimesofTunProbjobswithxedinputsinthefour loadingscenarioswiththreestrategies:Round-robin,BestCandidate withoutReschedulingandwithRescheduling. Figure3-7.Thepercentageofjobsmeetingtheirdeadlinesetto100secondsinthe fourscenarioswiththreestrategies:Round-robin,Best-Candidatewithout ReschedulingandwithRescheduling. candidatestrategiescanalwaysndmoreappropriateresourcesforthejobsthanthe blindRound-robin;second,eventhoughthechosenbestcandidateseemstobethe bestresourceforthejobatthesubmissiontime,itmaynotbethebestduringthe job'sexecutionduetothehighlydynamiccomputingenvironment,soreschedulingwith relativelylowoverheadcanimprovethejob'sexecutiontimeandismorelikelytomeet 49

PAGE 50

Figure3-8.TheaverageexecutiontimesofTunProbjobswithvaryinginputsinthefour loadingscenarioswiththreestrategies:Round-robin,BestCandidate withoutReschedulingandwithRescheduling. Figure3-9.Thepercentageofjobswithvaryinginputsmeetingtheirdeadlinesettofour timesofexecutiontimeonanunloadedvirtualmachineinthefourloading scenarioswiththreestrategies:Round-robin,BestCandidatewithout ReschedulingandwithRescheduling. theexpecteddeadline.Inthethreeloadingscenarioswithrescheduling,6%,12%, and18%ofthejobshadimprovedperformancebutdidnotmeetthedeadline.The reschedulingofthesejobswastriggeredbyCondition2,i.e.jobswererescheduled becausetheycanimproveperformancebyatleast30%,indicatingthattheAVAMcan handlejobsubmissionfailureusingthistechnique. 50

PAGE 51

EffectofapplicationCPUtimepredictionerrorsfromthenearest-neighborlearning algorithmonperformanceisavoidedintheaboveexperimentsbyusingxedinput parametersforthebenchmark.Thisallowsustoevaluatethereschedulingframework whentheapplicationperformancemodelingonlytakesintoaccounterrorsinapplication runtimepredictionforvaryingload.TotaketheCPUtimepredictionerrorsfromthe nearest-neighborlearningalgorithmintoaccount,theabovesetofexperimentswas repeated,whilevaryingtheinputparametersenergystep,minimumenergyand maximumenergyoftheapplicationbyrandomlyselectingtheirvaluesfromuniformly distributedrangesof0to1000000,0to10,and0to1000,respectively.Thejobruntime deadlineissettobefourtimesoftheexecutiontimeonanunloadedmachine.Three virtualmachinesareusedforthisexperiment,inwhichtwoofthemareloadedwith backgroundprocesses.Beforethejobs'submissions,100runsofTunProbwithrandom inputparametersinthesameabovementionedrangeareusedtowarmthepredictor. Allthejobsaresubmittedtothetwoloadedvirtualmachines,andthethirdoneisused forrescheduling.Figure3-8andFigure3-9showsthattheaverageruntimesandthe percentagesofthejobsthatmeetthedeadlineforthethreestrategies.Performanceof thethreeschedulingstrategiesissimilartotherstsetofexperiments,reectingthefact thatsmallpredictionerrorshavelittleeffectontheefciencyofrescheduling. 3.4.2.2Sensitivitytoloadvariances Inthesecondsetofexperiments,wewanttoinvestigatehowefcientlyoursystem respondstotheloadchanges.ATunProbjobwiththefollowinginputs,energystep of500000,theminimumenergyof2andthemaximumenergyof200,issubmitted toanunloadedmachine.Thejob'saverageruntimeontheunloadedmachineis measuredtobe80s,sothedeadlineforjobcompletionissetto3minutes.Various amountofbackgroundloadisintroducedintothemachine20secondsafterthejob's submission.Figure3-10showsthebenchmarkapplicationexecutiontimeswithand withoutreschedulingunderdifferentlevelsofload.Thehorizontalaxisrepresents 51

PAGE 52

Figure3-10.Comparisonofexecutiontimeunderchangingloadwithandwithout rescheduling.Application Duration 1 istheruntimefortheTunProbjobon themachinewherethejobisoriginallysubmitted;execution Duration 2 is theruntimeonthemachinewherethejobisrescheduled. theamountofloadweintroducedinthemachine,whiletheverticalaxisisthejob's runtime.Theleftbarindicatestheexecutiontimewhenthejobcontinuestorunon themachineaftertheloadisintroduced.Therightbarrepresentsthejob'sexecution timemanagedbytheAVAM,whichreschedulesthejobtoabetterresourcebecause offailureorpoorperformance.Fromthegure,wecanseethatthegreatertheload introducedthequickerthesystemisatdetectingandreactingtoit.Thereasonforthis isthatwhentheloadincreasestoaveryhighvaluei.e.theloadisincreasedtoabove 5inourexperiments,thejobsensorfailstoupdatethejobstatusontimeandthusthe controllerturnsthealarmon.Afterseveralalarmswesetathresholdvalueof6inthe experimentstheAVAMregardsthejobasfailedandreschedulesthisjobtoanother machineimmediately.Otherwise,themanagerutilizesthemonitoreddatafromthe jobsensortoestimatewhetherthejobwouldnishbeforeitsdeadline.Thegurealso showstheoverheadforreschedulingthejob,whichaverageslessthan4secondinour experiments.Comparedwiththetotalexecutiontime,thisisinsignicantformostcases. 52

PAGE 53

Figure3-11.Comparisonofexecutiontimeunderdifferentloadintroductiontimewith andwithoutrescheduling.Application Duration 1 istheruntimeforthe TunProbjobonthemachinewherethejobisoriginallysubmitted;execution Duration 2 istheruntimeonthemachinewherethejobisrescheduled. Inthethirdsetofexperiments,afterthejobissubmittedtoanunloadedmachine, thesameamountofhighloadisintroduced,butatdifferentpointsduringthejob's execution.Theinputparametersforthebenchmarkarethesameasforthesecond experiment.InFigure3-11,thehorizontal-axisindicatestheelapsedtimeinseconds ofthejob'sexecutionwhentheloadisintroduced,whiletheverticalaxisisthejob's runtime.Thegainforreschedulingisdiminishedastheloadisintroducedlaterintothe jobexecution.TheAVAMrecognizesthisscenarioandavoidsunnecessaryrescheduling oftheapplication. 3.5RelatedWork Theexistingworkmostcloselyrelatedtoourworkfallsintothreecategories: resource-usageprediction,resourceallocation,andrescheduling. Topicsonresource-usagepredictionhavebeenexaminedbymanyresearchers. Mostoftheseapproachesrelyontheuseofpast-execution-timeknowledge.Devarakonda etal.[86]usesstatisticalclusteringandstate-transitionmodeltocharacterizeprocess resourceusage.Thepredictionschemeusestheknowledgeoftheprogram'sresource 53

PAGE 54

usageinitslastexecutiontogetherwithitsstate-transitionmodeltopredicttheresource usageinitsnextexecution.Theapproachdescribedinthischapterusesthepredictive applicationmodelingdevelopedby[81]becauseofitsefciencyanditsconsiderationof theapplications'inputparameters.ThistypeofinformationisavailableforIn-VIGOtools suchastheoneTunProbusedasabenchmark.Thepredictionmodelemploysalocal learningalgorithmforthepredictionofrun-specicresourceusageonthebasisofinput parameterssuppliedtotools.Thismodelworkswellforourcase. Researchershaveproposedanumberofsystemsorapproachesforresource discoveryandselectionindynamic,heterogeneouscomputingenvironments.Some projects[87][88]aimedtosupportcustom-specicsystemswhoseusers'specications canbedirectlyusedbyascheduler.Incontrast,thesystemdescribedinthischapter providesapplicationswithaself-learningabilitytopredicttheirresourcerequirements atruntime.TheApplication-LevelSchedulingProjectAppLes[38]hasdeveloped anapproachthatincorporatesstaticanddynamicresourceinformation,performance predictionsintoresourcescheduling.However,theirperformancemodelsareapplication-specic andnoteasytoreapplytonewapplications.Ourperformancepredictionapproachis basedonagenericmodelthatcanbeappliedtoanyapplication. Severalprojectshaveimplementedreschedulingexecutionofapplicationsto differentresources.Thesesystemseitheraimtouseunder-utilizedresources,toprovide faultresilience,ortoreducetheobtrusivenessintoworkstations.Themostrelated worktooursisdescribedin[89].Theirsystemcontinuouslymonitorsapplicationsand evaluatestheremainingexecutiontimeoftheapplication,andmigrationdecisionsare madewhenevertheapplicationsarenotmakingsufcientprogress.Thetwomain differencesbetweentheirapproachandoursolutionresideintheevaluationmodeland thereschedulingaction.Theirevaluationmodelalsodependsonapplication-specic performancemodelsandtheoverheadcausedbymigrationincludingreadingand 54

PAGE 55

writingcheckpointsismuchhigherthanrestartingajobinourcase.High-overhead migrationisnotfeasiblefortherelativelyshort-livedapplicationsconsideredinthiswork. 55

PAGE 56

CHAPTER4 AUTONOMICRESOURCEMANAGEMENTINVIRTUALIZEDDATACENTERS Gridenvironments,asdiscussedinChapter ?? ,anddatacentersfacedifferentbut yetrelatedchallengesinprovidingperformanceguaranteestoapplications.Resource providersindatacentersareexpectedtodeliverperformanceguaranteeswhile optimizingresourceutilization,whereasgridmiddlewarecannotexpectperformance guaranteesfromindividualresourcesandmustrelocateapplicationsasneeded. Thischapterpresentsourinitialworkonapplyingautonomiccontrolindatacenter environmenttooptimizeapplicationperformanceandresourceusage. 4.1ProblemDescription Datacentershavebecomeincreasinglyimportantforhostingbusiness-critical applications.Abusinessrelationshiptypicallyinvolvesdatacenterownersand applicationproviders.Adatacenterprovidesresourcesforhostingapplications andapplicationproviderspayforwhattheyuse.Itisoftendesirableforapplication providerstobeabletoleasedata-centerresourcesunderapay-as-you-gomodel, andforthedata-centerproviderstobeabletomultiplexsharedresourcesinawaythat guaranteestheexpectedperformanceofapplications.Torealizethis,thedatacenter mustprovideexibleandmanageableexecutionenvironmentsthatarecustomizedfor eachapplicationwithoutcompromisingitsabilitytoshareresourcesamongapplications anddeliveringtothemthenecessaryperformance,securityandisolation. 4.1.1VirtualizedDataCenters Thetraditionaldatacenterinfrastructureprovidesverylimitedoptionsforefcient manageabilityandimprovingresourceutilization.Intraditionaldatacenterenvironment, applicationsaredeployedatdifferentserverstoprovidenecessarysecurityand performanceisolation.Asmoreapplicationsaredeployed,thenumberofserversalso growsrapidly.Applicationshostedindatacenterstypicallyhavetime-varyingworkload, 56

PAGE 57

withhighpeak-to-averageratio.Over-provisioningusedformeetingpeakdemandwill leadtolowresourceutilizationandhighresourcewastage. Virtualizationbecomeskeytoprovidingexibleandmanageableexecution environmentsindatacenters.Itentailsthepossibilityofonephysicalserverhosting multipleindependentvirtualmachines[11][90][91],andtheabilityoftransparently movingapplicationsfromonephysicalservertoanotherthroughvirtualmachine migration.Fine-grainedvirtualmachineresourceallocationandreallocationarepossible inordertomeettheperformancetargetsofapplicationsrunningonvirtualmachines. Themanagementofvirtualmachine,e.g.lifecyclemanagementandresourceallocation, canbeconductedthroughtheinterfaceprovidedbythevirtualizationplatform.Ina virtualizeddatacenter,applicationsarehostedandmanagedintheirdedicatedresource containersimplementedasvirtualmachineswhichcanbedynamicallycreatedand providestrongisolationandsecurityandcustomizability.Insteadofallocatingdedicated serverstoapplicationsformeetingpeakdemand,resourcesaresharedamongmultiple applicationstoenableefcientresourceutilization. 4.1.2ChallengesofResourceManagementinVirtualizedDataCenters Applicationsservedbyadatacenterareusuallybusiness-criticalapplicationswith QualityofServiceQoSrequirements,e.g.e-commerceservices.Suchapplications havetime-varyingresourcedemandswithtypicallyhighpeak-to-meanratios,leadingto lowresourceutilizationifover-provisioningisusedtomeetpeakdemands.Theresource allocationneedstonotonlyguaranteethatavirtualmachinealwayshasenough resourcestomeetitsapplicationsperformancegoals,butalsopreventover-provisioning inordertoreducecostandallowtheconcurrenthostingofmanyapplications.Static allocationapproachesthatconsideraxedsetofapplicationsandresourcescannot beusedbecauseofchangingworkloadmixes,andsolutionsthatonlyconsiderruntime behaviorofindividualapplicationsfailtocapturethecompetitionforsharedresources byvirtualizedcontainers.Akeychallengeofresourcemanagementinvirtualizeddata 57

PAGE 58

Figure4-1.APay-as-you-godatacentermodelwithvirtualresourcecontainerstohost applicationsonasharedresourcepool,andatwo-levelcontroller architecturetoallocatephysicalresourcestocontainers. centersisthesimultaneouson-demandprovisioningofsharedresourcestovirtual machinesandthemanagementoftheircapacitiestomeetservicequalitytargetsat theleastcost.Thisworkproposestoachieveresourcemanagementneededtomeet servicelevelagreementsSLAsbyintegratingautonomicresource-controlfunctionsat twolevels-alocalcontrollerineachvirtualcontainerandaglobalcontrollerforthedata centerresourcepool. 4.2Two-LevelResourceControl Atwo-levelautonomicresourcemanagementsystemisdevelopedtoenable automaticandadaptiveresourceprovisioninginaccordancewithServiceLevel AgreementsSLAspecifyingdynamictradeoffsofservicequalityandcost.Inthe system,theresourcecontrolfunctionsareintegratedatdifferentlevelsofabstraction: virtualcontainersandresourcepools.Alocalcontroller,createdpervirtualcontainer, isresponsiblefordeterminingtheresourcesneededbyitsapplicationandmaking resourcerequestsaccordingly.Aglobalcontrollermanagesthevirtualcontainers hostedonthesamephysicalresources.Itrespondstothelocalcontrollers'requests andallocatesthesharedresourcestotheminawaythatmaximizesthetotalprotby leasingresourcestoalargenumberofcontainers. 58

PAGE 59

4.2.1ApplicationServiceLevelAgreementsSLAandResourceSLA Adatacenter,illustratedinFigure4-1,servesanumberofapplications.Each deliversadistinctservicetoitscustomersusingvirtualresourcesprovidedbyits dedicatedcontainer,whichisthevirtualmachinethathoststheapplication.The datacenterallocatesthephysicalresourcestoeachvirtualcontainersbasedonits application'sresourceneeds. TheapplicationSLAbetweenanapplicationprovideranditscustomersstates thequalityofserviceproviderspromisedtotheclients.Toachieveperformance isolationandguaranteeanapplication'sSLAindependentlyoftheloadsonother containers,alocalresourcecontrollerisemployedineachvirtualcontainertoestimate theresourcesneededbytheapplication'sworkloadandtomakeresourcerequests totheglobalcontroller.Bydoingso,thelocalcontrollerminimizesleasingcostsby avoidingover-provisioningfortheapplicationrunningonthecontainer.ResourceSLA betweenapplicationprovidersandthedatacenterownerspeciesboththecostof usingresourcesandthepenaltydueincasethedatacenterfailstodeliverresources neededbytheapplicationproviders.Theunderlyingassumptionisthatifthedata centerdoesnotallocateenoughphysicalresourcesrequestedbythelocalcontroller resultinginitsapplication'sSLAviolation,thedatacenterproviderwillbepenalized.The globalcontrollermakesallocationdecisionsamongcompetingrequests,tryingtoavoid violationsofResourceSLA. 4.2.2BenetsofTwo-LevelControl Thistwo-levelresourcecontrolsystemispreferredoverthemoreobvious centralizedapproachinwhichallthecontrolfunctionsareimplementedatone centralizedlocation.Sincelocalcontainersareindependentofeachother,heterogeneous localcontrollers'implementationsarepossible.Alloftheinternalcomplexitiesofcontrol functionsinvirtualcontainersarecompressedbylocalcontrollersintostraightforward resourcerequests,whichspecifytheamountofresourcesneeded.Thesystemhandles 59

PAGE 60

Figure4-2.Inputsandoutputsofavirtualcontainer. twodifferenttypesofoptimizationsindependently.Thelocalcontrollertriestominimize theresourceconsumedbythevirtualcontainertoreducetheresourcecostwhilestill satisfyingtheSLAsofitsclients.Theglobalcontrollerseekstomaximizeitsownprot, whichistherevenuereceivedfromallocatingitsresourcestovirtualcontainersminus thecostofpenaltiesincurredfromresourceSLAviolations.Thefollowingsections explainourapproachtothedesignofthelocalandglobalcontrollers. 4.3LocalController Interactionbetweenthelocalandglobalcontrollersenablesavirtualcontainerto augmentitsresourcesinresponsetoincreasedworkload,andtoreduceitsresources whentheyarenolongerneeded.Themaintaskofthelocalcontrolleristooptimizethe setofresourcesneededbyanapplicationrunninginthecontainer.Ourapproachtothe designofsuchacontrollerisbasedonfuzzylogictheory,asdiscussednext. 4.3.1Input-OutputModelofVirtualContainer Todeterminetheresourceneedsofanapplicationhostedinavirtualcontainer,the localcontrollerneedstolearnthebehaviorofthevirtualcontainerunderdynamically changingworkloads.Figure4-2showstheabstractedinputsandoutputsofavirtual containerthathostsarunningapplication.Thevirtualcontainerreceivestheapplication workloadfromitsclients,andutilizesthephysicalresourcesprovidedbythedatacenter resourcepooltoprocesstheworkload.TheachievedQoSoftheapplicationdepends ontheamountofallocatedresourcesandtheincomingworkload.Theinformationabout theapplication'sworkload,itsachievedperformanceanditsvirtualcontainer'sresource 60

PAGE 61

consumptionismonitoredbythesystemsensorsasFigure4-2illustrates.Depending onwhatinformationisavailablefromthesystem,twoapproachesareproposedfor estimatingresourceneeds:fuzzymodelingtocharacterizetherelationshipbetween workloadandresourceuseandfuzzypredictiontodetermineamappingfrom currentresourceobservationstofutureresourceneeds. 4.3.2BasicsofFuzzyLogic Fuzzylogic[92]isatooltodealwithuncertain,imprecise,orqualitativedecision-making problems.UnlikeinBooleanlogic,whereanelement x eitherbelongsordoesnot belongtoaset A ,infuzzylogicthemembershipof x in A hasadegreevalueina continuousintervalbetween0and1.Fuzzysetsaredenedbymembershipfunctions thatmapsetelementsintotheinterval[01]. Oneofthemostimportantapplicationsoffuzzylogicisthedesignoffuzzy rule-basedsystems.ThesesystemsuseIF-THENrulesalsocalledfuzzyrules whoseantecedentsandconsequentsusefuzzy-logicstatementstorepresentthe knowledgeorcontrolstrategiesofthesystem.Thecollectionoffuzzyrulesiscalleda rulebase.Therearemanysourcesforconstructingfuzzyrules,forexample,fromexpert experienceorbasedonanoperator'scontrolactions.Theapproachtakenforthedesign ofoursystemistolearnfuzzyrulesusingonlinemonitoringinformation,makingita so-calledself-learningfuzzysystem. Theprocessofapplyingfuzzyrulesonthesystemiscalledthefuzzyinference FISmechanism.Sincefuzzyrulesusefuzzyvaluestodescribethesystemvariables, twofunctionsarenecessaryfortranslatingbetweennumericvaluesandfuzzyvalues. Theprocessoftranslatinginputvaluesintooneormorefuzzysetsiscalledfuzzication. Defuzzicationistheinversetransformationwhichderivesasinglenumericvaluethat bestrepresentstheinferredfuzzyvaluesoftheoutputvariable. 61

PAGE 62

Figure4-3.Fuzzymodelingandinferencefunctionsinalocalresourcecontroller. 4.3.3FuzzyModeling Therstapproachusesfuzzylogicsystemstomodelthebehaviorofavirtual containerbyautomaticallylearningtherelationshipbetweenworkloadandthe correspondingresourceconsumptionwhenthedesiredQoSisachieved.Itrequires thesystemtoperiodicallymonitortheapplicationworkloadandtheirresourceusage, whicharethenusedasinput-outputdatapairforgeneratingfuzzyrules.Figure4-3 illustratesthekeyfunctionsforfuzzymodelinginthelocalcontroller.Thedatamonitored bythesensorsarerstprocessedbythelteringandclusteringfunctions.Themodeling functionconstructsfuzzyIF-THENrulesusingtheproduceddataclustersandkeeps themintheknowledgebase.Theclustercentersandrangesareusedtodetermine thefuzzymodel'sparameters.Finally,thefuzzyinferencefunctionsprocessthefuzzy ruleskeptintheknowledgebasetodeterminetheresourceneedsfromthecurrently monitoredworkload.Therestofthissectionexplainsthesefunctionsindetail. DataMonitoringandFiltering: Monitoringsensorsperiodicallymeasurethe applicationworkload w t ,itsperformance p t ,andtheresourceusage r t ofavirtual container.Foratypicaldatacenterapplication,itsworkloadcanusuallybedescribed bytherateandmixtureofitsclient'srequests.Forinstance,aWebserver'sworkload canbecharacterizedbytheHTTPrequestrateaswellastheratioofstaticWeb-content requeststodynamicones.Theperformancemetricsareoftendirectlytakenfromthe 62

PAGE 63

applicationSLA,e.g.thethroughputnumberofcompletedtransactionspersecond and/oraverageserviceresponsetime. Themetricsforresourceutilizationareassociatedwiththedifferenttypesof consumedphysicalresources,includingCPUpercentage,memorysize,diskstorage, diskI/Orateandnetworkbandwidth.However,anapplication'svirtualresourceusage thevaluescollectedinsideofthevirtualcontainerdoesnotnecessarilyrepresentits physicalresourceconsumption.Forexample,anapplication'snetworkI/Oconsumesnot onlythephysicalnetworkbandwidth,butalsothephysicalCPUcycles.Intheproposed approach,anapplication'sresourceusageisobtainedbydirectlymonitoringthephysical resourceconsumptionofitsvirtualcontainer.Thisissensiblebecauseintheenvisioned datacenteravirtualcontainerisdedicatedtoanapplication. Asequenceofinput-outputdatapairs w t ;r t areproducedbythesensorsat constanttimeintervalssecondsinourexperiments,andthenlteredbasedon thecorrespondingperformancemeasurements p t .Thelteringpolicyisthatadata pairmeasuredattime t iskeptorlteredoutdependingonwhethertheperformance measuredatthesametimesatisestheapplicationSLAornot,respectively.Performance issatisfactory,onlyiftheresourcesallocatedtothevirtualcontainerattime t issufcient forthegivenSLA.Inthiscase,themonitoredresourceutilizationrepresentstheactual resourceneeds,andthusthedatapaircanbeusedformodeling.Onthecontrary,an SLAviolationindicatesthattheallocatedresourcesarenotenoughtoachievetheSLA target.Inthiscase,theresourceconsumptioniscappedbytheallocatedcapacityso thatthemonitoredvaluesarelessthanthedesiredresourcedemandsandcannotbe usedinfuzzymodeling. DataClusteringandFuzzyRuleConstruction: Thecollectedpaireddataare rstclusteredtoproduceaconciserepresentationofthesystem'sbehaviorandthen thedataclustersareusedtobuildfuzzymodels.Severalclassicclusteringalgorithms 63

PAGE 64

canbeused,e.g.hierarchicalandk-meansclustering.Intheproposedlocalcontroller design,subtractiveclustering[93]ischosenforitsspeedandrobustness. Thisclusteringmethodassumesthateachdatapointisapotentialclustercenter andchoosesthedatacenterbasedonthedensityofsurroundingdatapoints.The algorithmselectsthedatapointwiththehighestdensityvaluetobetherstcluster centerandthenremovesalldatapointsinthevicinityoftherstclustercenterinorder todeterminethenextdataclusteranditscenterlocation.Thisprocesscontinuesuntil allthedataarewithinradiusofaclustercenter.Thevariable radius representsacluster center'srangeofinuenceineachofthedatadimensions,assumingthedatafallwithin aunithyperbox.Settingsmallradiusvaluesgenerallyresultinndingalargenumberof smallclusters.Thisvalueissetto0.5inthelocalcontroller'simplementation. Sinceeachproducedtwo-dimensionalclusterexempliesacharacteristicofsystem input-outputbehavioritcanbeusedasthebasisofafuzzyrulethatdescribesthe relationshipbetweenasystem'sinputandoutput.If n dataclustersareformed, n rules canbeproducedinwhichthe i thruleisexpressedas: IFinput w isincluster i ,THENoutput u isincluster i Eachclusterspeciesafuzzysetwithitsmembershipfunctionsdeterminedbythe clustercenterandrange.UsingtheGaussianmembershipfunction, i x = e )]TJ/F15 11.9552 Tf 7.782 8.087 Td [( x )]TJ/F23 11.9552 Tf 11.955 0 Td [(c i 2 2 2 i thecenterofthemembershipfunction c i equalsthecenterofcluster i andtheweightof membershipfunction i equalstheradiusofthatcluster. Themodeldescribedbytheabovefuzzyruleiscalledzero-orderSugeno-type fuzzymodel[17].Themodelingaccuracycanbeimprovedsignicantlybyusingthe rst-orderSugenomodel,inwhichtheoutputofeachruleisalinearfunctionoftheinput variables.Therulesarerewrittenasfollows, IFinput w isincluster i ,THENoutput u = aw + b forrule i Theparameters a and b inthelinearequationsareestimatedbytheleast-squares method. 64

PAGE 65

FuzzyInference: Oncethefuzzymodelrelatingworkloadtoresourcedemand islearnedfromtheselectedworkloadandresourceusagemeasurements,itcanbe usedinarule-basedfuzzyinferencemodulewhich,giventheapplication'sworkload, producestheestimatedapplication'sresourcedemandforthevirtualcontainer. ThefuzzyinferencemoduleconsistsoffourbasicfunctionsshowninFigure4-3. Theknowledgebaseincludesadatabasewhichcontainsthemembershipfunctionsof thefuzzysetsandarulebasewherethefuzzyrulesarespecied.Inthefuzzication function,theinput w t measuredfromthesensorismappedtofuzzyvaluesusing themembershipfunctions.Adecision-makingunit,calledthefuzzyinferenceengine, infersfromfuzzyinputstoresultingfuzzyoutputsaccordingtotherulesstoredinthe knowledgebase.Thedefuzzicationfunctionaggregatestheoutputsandconvertsthem toasingleoutputvalue.Thenaloutputistheweightedaverageofalloutputswiththe weightof i thrulebeingthemembershipvalueoftheinputincluster i Insummary,usingfuzzymodelingandfuzzyinferenceshowninFigure4-3,the localcontrollerestimatestheresourceneedsforthecurrentworkloadmeasuredbythe sensor,andsendsrequeststotheglobalcontrollertoeitheraskformoreresources ifthecurrentallocationisnotsufcienttosatisfytheapplicationSLAortowithdraw resourceswhennolongerneeded. AdaptiveModeling: Thediscussionsofarhasonlyconsideredofinemodel learningfromcollecteddata.Asworkloadorsystemconditionchanges,themodel describingthesystem'sbehaviorneedstocapturethechangesaccordingly.The adaptivemodelingisemployedbythelocalcontrollerinwhichthemodelisrepeatedly updatedbasedononlinemonitoredinformation. Theclusteringfunctiontakesnewdataintoconsiderationassoonastheyarrive afterthelteringandkeepsupdating,sothatup-to-dateclustersarealwaysprovided forthemodeling.Wheneverthedataclustersareupdated,theparametersofthe membershipfunctionsarechangedaccordinglyinthedatabase.Ifanewclusteris 65

PAGE 66

added,acorrespondingruleisthenaddedintotherulebase;andsimilarly,ifacluster nolongerexists,theruleassociatedwithitisremovedfromtherulebase. Inthecasewhentheallocatedresourcesareinsufcientfortheworkload,the monitoreddatabecomesdisqualiedandislteredoutbecauseoftheperformance degradation.Theshortageofqualieddatawillhurtthemodel'slearningspeedand quality.Toavoidthissituation,wheneverthelterfunctiondetectsthatthepercentage ofqualieddataislessthan50%duringatimewindow T issetto5minutesinthe prototype,thecontrollerasksforanadditionalpredenedpercentage%isusedin theprototypeofcurrentresourceallocationfromtheglobalcontrollertoimprovethe application'sperformancebacktothedesiredlevel. 4.3.4FuzzyPrediction Thefuzzy-modelingbasedapproachdescribedaboveautomaticallybuildsa mappingfromtheapplicationworkloadtothecorrespondingresourceneedsfor thedesiredQoS.Thisapproachisapplicableonlywhentheapplicationworkload canbecharacterizedandmonitored.However,datacenterstypicallyhostavariety ofapplications,therefore,itisunclearwhatshouldbeasetofstandardmetricsfor applicationperformanceduetothediversityofapplicationscoexistinginadatacenter. Insomecasesitishardtodescribeanapplicationworkloadusingafewmetricslike requestrate.Thesecondproposedapproachfuzzy-predictiononlyrequires informationabouttheresourceusagee.g.,CPUutilization,whichiseasytoobtain bymonitoringsystem-levelmetrics.Thebasicideaistodeterminefutureresource needsonthebasisofobservationsofpastresourceusage. FuzzyRuleConstruction: Thefuzzypredictionsystem,illustratedinFigure4-4, hassomecomponentsthataresimilartothoseusedinthefuzzymodelingapproach. Thefuzzyrulesaregeneratedfrommonitoreddataandstoredintherulebase.The fuzzyinferencesystemprocessesthelearnedfuzzyrulestoforecastfutureresource usebasedonthecurrentsystemobservations.Let r t t =1 ; 2 ; 3 ;:::; beresource 66

PAGE 67

Figure4-4.Fuzzypredictionfunctionsinalocalcontroller. Figure4-5.Three-inputtwo-outputdatapairexamples. usagemeasurementatsamplingtime.Theproblemcanbeformulatedas:attime t giventhelatestmmeasurements r t ;r t )]TJ/F15 11.9552 Tf 12.028 0 Td [(1 ; ;r t )]TJ/F23 11.9552 Tf 12.028 0 Td [(m +1 ,determinetheresource useatfuturetime r t +1 ;r t +2 ;:::;r t + n m and n arethenumberofinputsand outputsforafuzzyrule,respectively. Afuzzyruleisgeneratedfromaninput-outputdatapair,whosecomponentsare subsequencesofthesuccessiveresourceusagemeasurements,theinputsubsequence precedingtheoutputsubsequences.Figure4-5showsanexampleofthree-input two-output m =3 and n =2 inthiscasefuzzyrules.Totranslateinput-outputpairs intofuzzyrules,therststepistodividetheinputandoutputspacesintofuzzydomains. Assumingthattheinputandoutputrangesarenormalizedto[0,1],eachspaceis dividedinto 2 N +1 domains,denotedby R 1 ;R 2 ;:::;R 2 N +1 ,eachassignedafuzzy membershipfunction.Figure4-6givesanexampleofmembershipfunctionswherethe 67

PAGE 68

Figure4-6.Anexampleofdividingtheinput/outputspaceinto11fuzzydomainsandthe correspondingmembershipfunctions. input/outputspaceisdividedinto11domains N =5 isusedinourprototypewith triangularmembershipfunctions. Thenextstepistoassignagivendatapointtothefuzzydomainwiththehighest membershipdegreeusingthemembershipfunctionsdescribedabove.Forexample,for aninput-outputdatapair i and o i isassignedtodomain R 5 and o isconsideredtobe R 8 inFigure4-6.Finally,afuzzyruleisconstructedfromapairofinput-outputdataas follows, IF i 1 is R i 1 and i 2 is R i 2 ,...,and i m is R im ,THEN o 1 is R o 1 ,..., o n is R on .forrule i Therefore,everysequenceof m + n consecutiveresourceusagemeasurementscan beusedtogenerateafuzzyrulewhichmapstheinputspace i 1 ;i 2 ;:::;i m representing therecentsystemstatetotheoutputspace o 1 ;o 2 ;:::;o n representingthefuturestate. FuzzyRuleUpdate: Afuzzyruleisgeneratedateverysamplingtimeandeachrule isrepresentedasapointinthe m + n -dimensionalrulespace.Ifeveryruleisstored intherulebasethememoryrequirementswillbeexcessive,anditisprobablethatthere wouldbeconictingruleswhichhavethesameIFpartbutadifferentTHENpart.The rstproblemissolvedbypartitioninginput-outputspacesintoanitenumberofdomains asdescribedabovesothatatmostonerulei.e.,apointisstoredintherulebasefor eachdomain.Thenumberofrulesincreasesasnewinput-outputdataarecollected,but 68

PAGE 69

Figure4-7.Thefuzzy-ruleupdatingprocedure. itneverexceedsthemaximumnumberofdomainspartitionedinthe m + n -dimensional rulespace. Toovercomethesecondproblem,whenupdatingtherulebaseareliabilityindexis computedforeachruleas J i =thenumberofoccurrencesofrule i .Wheneveraruleis generated,thesystemscansalltherulesstoredintherulebase.Ifthereisamatching rulei.e.,aruleinthesamedomain,thevalueof J isincreasedby1.Otherwise,the newruleisaddedtotherulebaseand J isinitializedto1.Figure4-7illustratesthe procedureforupdatingrules.Ifthereexistconictingrules,theonetakeseffectis determinedbythevalueofthereliabilityindex.Therulewiththehighestreliabilityindex isactivated,indicatingthattheactivefuzzyruleappearsmorefrequentlythanother 69

PAGE 70

conictingrules.Iftwoconictingruleshavethesamevalueofreliabilityindex,theone thatappearedmostrecentlyisactivated. FuzzyInference: Giventhelatestresourceusagemeasurementsasinputs,as Figure4-4shows,thefuzzyinferenceengineprocessestheactivefuzzyruleskept intherulebasetodeterminetheoutputswhichconsistofthefutureresourceusage. Initially,thereisnoruleintherulebase.Aftertherst m + n measurementsare obtained,therstruleisgeneratedandstoredintherulebase.Afterwards,ateach samplingpoint,anewruleisconstructedandtherulebaseisupdatedfollowingthe updatingprocedureshowninFigure4-7.Thisupdatingproceduremakestheproposed fuzzypredictioncapableofself-learningtheresourceusagebehaviorofthemanaged virtualcontainer. Comparedwiththefuzzy-modelingapproach,bothmethodsessentiallylearnfrom historicaldatatobuildfuzzyrulesandcanadaptivelyupdatetheruleswhennewdata areavailablesothatitcanreectsystemchangesveryquickly.Nopriorknowledgeor mathematicalmodelsofthesystemarerequiredandtheybothareaone-passbuild-up procedurethatdoesnotneediterativetime-consumingtraining.Thedifferencebetween thetwoapproachesisthatthefuzzymodelingapproachmapsworkloadtoresource consumption,whilethefuzzypredictionmapstheobservationsofrecentresourceusage tothefutureresourceneeds. 4.4GlobalController Eachindividuallocalcontrollertriestominimizetheresourcecostbyonly requestingtheresourcesnecessaryformeetingtheapplicationSLA.Theglobal controllerreceivesrequestsforphysicalresourcesfromlocalcontrollersandallocates theresourcesamongthemasrequired.Itseekstomakeallocationsthatmaximizethe datacenter'sprot,whichis,basedontheproposedprotmodelinapay-as-you-go datacenter,therevenuereceivedbyallocatingtheresourcesminusthepenaltiesdueto resourceSLAviolations. 70

PAGE 71

Thelocalcontrollersperiodicallysendresourcerequeststotheglobalcontroller whichmakesallocationdecisionbasedonthereceivedrequestsandcurrentlyavailable resourcesinthedatacenter.Tosimplifytheproblem,weconsiderasingleresourcetype andasingleallocationperiod.Supposethat K virtualcontainersareconcurrentlyactive inthedatacenter.Let req k denotetheresourcesrequestedfromvirtualcontainer k ,and alc k betheamountofresourcesgrantedtoitbytheglobalcontroller.Thedatacenter receivesrevenueof rev k foreveryallocatedresourceunitoveranallocationperiod.But iftheglobalcontrollercannotsatisfytherequest req k ,thedatacenterpaysapenaltyof pnl k perunitofunmetresourcedemand,accordingtotheresourceSLA.Eachresource allocationdecisionmadebytheglobalcontrollerisexpressedasaresourceallocation vector alc 1 ;alc 2 ;:::;alc K ,andthetotalprotobtainedbythedatacenterforatime periodis, profit alc 1 ;alc 2 ;:::;alc K = K X k =1 [ rev k alc k )]TJ/F23 11.9552 Tf 11.955 0 Td [(pnl k req k )]TJ/F23 11.9552 Tf 11.956 0 Td [(alc k ] s:t: 0 alc k req k ; K X k =1 alc k A where A isthetotalavailableresourcecapacityinthedatacenter.Theprot equationcanberewrittenasfollows, profit alc 1 ;alc 2 ;:::;alc K = K X k =1 rev k + pnl k alc k )]TJ/F24 7.9701 Tf 16.737 14.944 Td [(K X k =1 pnl k req k rev k + pnl k isconsideredastheprotrateforvirtualcontainer k .Assumingthat theglobalcontrollercanallocateanyresourcefractiontothevirtualcontainers,agreedy algorithmthatallocatesresourcesintheorderofdecreasingprotratesisanoptimal allocationthisissimilartothecaseofafractionalknapsackproblem[94]. 71

PAGE 72

Tooptimizeprotovermultipletimeperiods,theallocationdecisionhastobe repeated.Equation4denesacumulativeprotwhichisthediscountedsumunder adiscountingfactor overatimehorizon T .Thefactormodelsthefactthatfutureprot isworthlessthancurrentprotbecauseoftheuncertaintyinthefuture.ForTallocation periods, max T X t =1 t profit t =max T X t =1 k X k =1 t [ rev k alc tk )]TJ/F23 11.9552 Tf 11.956 0 Td [(pnl k req k )]TJ/F23 11.9552 Tf 11.955 0 Td [(alc tk Basedontheaboveprotmodel,agreedystrategythatmaximizesthetotalprot foreveryperiodisstilloptimalbecausetheallocationdecisionforcurrentperioddoes notaffectthefutureperiods. 4.5ExperimentalEvaluation Thissectionsummarizestheexperimentalevaluationoftheproposedtwo-level controlsystemfordynamicresourceallocationinadatacenterenvironmentwith time-varyingworkloads.Section4.5.2and4.5.3discusstheexperimentsthatevaluate theabilityofthelocalcontrollertotracktheresourceneedsofchangingworkloads. Section4.5.4considersthemaximalprotapproachEquation4discussedin Sectionwhentheglobalcontrollermustallocatelimitedresourcesamongseveral competingvirtualcontainers. 4.5.1ExperimentalSetup DataCenterTestbed: Thetestbedisdeployedona16-CPUIBMx336based clusterthatprovidesvirtualcontainersforapplications.VMwareESXServer3.0.1is installedineachclusternodeequippedwithdualhyperthreadedIntelXeon3.2GHz CPUsand4GBmemory.Virtualmachinesarecreatedontheserversandusedas virtualcontainerstohostapplications.Theworkload-generatingclientsareplaced onVMware-Server-1.0.0-basedvirtualmachines,hostedonanotherclusterof32 dual-2.4GHzhyperthreadedIntelXeonnodes.Web-basedworkloadsaregeneratedby theclientsandissuedtotheapplicationsacrossaGigabitEthernetnetwork. 72

PAGE 73

ApplicationandWorkloads: TheJavaPetStore[95]waschosentorepresenta typicale-businessapplication.Itimplementsanonlinestorethatallowsuserstobrowse andmakeorders,andmanagerstomanageorders,suppliersandinventory.Synthetic HTTPworkloadsthatmimicthekeyaspectsofreal-worldworkloadsarecreatedwith variousclientsessionsissuedbyhttperf[96].Eachindividualsessionconsistsofa sequenceofrequestsgeneratedbyrepeatingandmixingthefollowingcustomeractions: gotothestorefront,signin,browseproducts,addsomeproductstoshoppingcart, andcheckout.Twokeyparametersareadjustedtovaryasession'sworkloadonthe application:theuserthink-timethetimebetweentwoconsecutiverequestscanbe changedtogeneratedifferentrequestrates;theratioofdynamicrequestse.g.,sign in,checkoutandsearchproducttostaticrequestse.g.,browsestaticWebpages andviewimagescanbevariedinordertochangetheworkloadcharacteristics.APerl programwasdevelopedtocreatedifferentworkloadsanddrivehttperftoissuethe requests. Tracescollectedfrom'98WorldCupsitesarealsousedintheexperimentsto representreal-worldworkloads.ThelogsprovidedbyanInternetrepository[97]consist ofabout1.3millionrequestsmadetothe'98WorldCupWebsitebetweenApril30, 1998andJuly26,1998.Areal-timelogreplayer's[98]isusedtogenerateworkloads accordingtothetrace. Global/LocalControllerPrototype: Thevirtualcontainersaremonitoredand controlledthroughtheWeb-service-basedmanagementinterfaceprovidedbyVMware ESXServer.Itallowstheallocationofaserver'sphysicalresourcesamongitshosted virtualmachinese.g.settingtheminimum,maximumandproportionalresourceshares ofavirtualmachine,andalsoprovidesthereal-timemonitoringofavirtualmachine's resourceutilization. Theproposedtwo-levelcontrollersareimplementedinJava,runningalongwith thevirtualcontainers.Everyvirtualmachinehasalocalcontrollertomanagethevirtual 73

PAGE 74

containeritprovides,andeveryESXservernodehasaglobalcontrollertomanagethe sharedresourcesforthevirtualcontainershostedonit.Thesensors,alsodevelopedin Java,monitortheworkloadrequestrateandmixture,theapplicationthroughputreply rate,andtheresourceconsumptionCPUusage.Themonitoringperiodissetto20 seconds.BecausetheconcernedworkloadsaremostlyCPUintensive,theexperiments focusontheutilizationandallocationofCPUresources. 4.5.2Fuzzy-ModelingApproach Thefollowingexplainstheexperimentsforvalidatingwhetherthelocalcontroller canaccuratelyestimateresourceneedsusingthefuzzy-modelingandfuzzy-prediction approachesunderdynamicallychangingworkloads. 4.5.2.1StaticWebrequests Intherstexperiment,theworkloadgeneratorissuesanewsessiontothePet Storeevery10seconds,uptoatotalof15sessions.Thesesessionsonlycontain requestsforstaticWebcontentwithauserthink-timerangingfrom0.1to1second, andeachsessionlastsaround1minute.Afteragroupof15sessionsarecompleted, anothergroupisgeneratedsimilarlybutwithadecreasingaveragethink-timeand henceanincreasingrequestrate.Thissetupemulatesthepresenceofburstin real-worldworkloads.Theentireexperimentlastsfor4000seconds. BecausetheworkloadsusedinthisexperimentconsistofonlystaticWeb-content requests,theCPUusageishighlycorrelatedwiththerequestrate,whichisthenused astheonlymetrictocharacterizetheworkload.Inthiscase,theinputandoutputto fuzzymodelingaretherequestrateandCPUusagemeasurements.Therst50pairs ofdatapointscollectedfromthesensorsareusedtoinitializethelearningofthefuzzy model.Afterwards,themodeliscontinuouslyupdatedevery200secondsduringwhich 10newdatapointsbecomeavailablefromthesensors.Figure4-8illustratesthemodel learnedatthebeginningandtheendoftheexperiment,whichshowsanapproximate 74

PAGE 75

Figure4-8.FuzzymodellearnedfromtheworkloadwithstaticWebrequestsatthe beginningmodel1andtheendmodel2oftheexperiment. linearrelationshipbetweentherequestrateandCPUusageintherangeof0to100 requests/second. ThelocalcontrollercontinuouslyestimatestheCPUdemandbasedonthecurrent workloadandthelatestlearnedfuzzymodel,anddynamicallyadjuststheCPUrequests totheglobalcontroller.Becausetheavailableresourcesaresufcientfortheonly virtualcontainerusedinthisexperiment,theglobalcontrolleralwaysallocatestheexact amountofCPUrequestedbythelocalcontroller.Toprovetheaccuracyofthefuzzy modeling,thesameexperimentisrepeatedonthevirtualcontainerwhichisstatically allocatedwithalargeamountofCPU.2GHzinordertoobtaintheidealthroughput forthesameworkload. ThethroughputfromthesetwoexperimentrunsarecomparedinFigure4-9, indicatingthattheactualthroughputobtainedbydynamicallocatingresourcesusingthe fuzzy-modelingapproachisalmostidenticaltotheidealthroughput.Comparedtothe staticallocationofCPUwiththepeakvalueoverprovisionbasedonthehighestload, thedynamicresourceallocationapproachsavesabout55%ofCPUcyclesotherwise neededforthisexperiment.Thisconrmsthattheonlinefuzzymodelingcanaccurately 75

PAGE 76

Figure4-9.Comparisonofthethroughputachievedbyusinglocalcontrollerandthe idealthroughputfortheworkloadwithstaticWebrequests. Figure4-10.Thesurfaceofthe3Dfuzzymodellearnedfromtheworkloadwithdynamic Webrequests. learntherelationshipbetweentheworkloadandresourcedemand,andeffectivelyguide theresourceallocationforthevirtualcontainer. 4.5.2.2DynamicWebrequests Inthesecondexperiment,theworkloadsaregeneratedsimilarlytotheprevious one,exceptthatdynamicWebrequestsarealsoconsidered.Everygroupofsessions differsnotonlyintherequestratebutalsotheproportionofdynamicrequestsinthe 76

PAGE 77

Figure4-11.Comparisonofthethroughputachievedbydynamicallcationusingfuzzy modelingandtheidealthroughputwithmaximalallocationfortheworkload withdynamicWebrequests. workload:theratioofdynamictostaticrequestsgrowsfrom0to1acrossthegroups. ServicingdynamicWebcontentrequiresprocessingbytheapplicationserverand database,andthustypicallyconsumesmoreresourcesthanservicingstaticWeb content.Ifthefuzzymodelingstillusesonlyrequestrateastheinput,theresulting modelcannoteffectivelyrepresenttheactualrelationshipbetweenworkloadand resourcedemand.Theexperimentresultsobservedbutnotshownherealsoconrm thatthethroughputachievedbyusingsuchamodelismuchworsethantheideal throughputfortheworkload. Incontrast,usingboththerequestrateanddynamic/staticrequestratioto characterizetheworkload,a3Dfuzzymodelcanbeconstructedtodescribethe relationshipbetweenworkloadandresourcedemandmoreaccurately.Figure4-10 showsthesurfaceofthemodellearnedattheendoftheexperiment.Oneofthe advantagesoffuzzymodelingdemonstratedbytheaboveexperimentsisthatfuzzy modelscaneffectivelylearnsimpleaswellasnon-linearandcomplexrelationships betweeninputsandoutputs. 77

PAGE 78

Figure4-12.Thesurfaceofthe3Dfuzzymodellearnedfromtheworkloadwithdynamic Webrequests. Figure4-11comparestheapplication'sthroughputtotheidealthroughput obtainablefortheworkload.Thegraphshowsthatthethroughputachievedisagain veryclosetoitsideallevelthedifferenceisunder6%.Itisalsonoticeablethatwhen theworkloadishighthedifferencebecomesrelativelylarger.Thisisbecauseofthe delaybetweenthechangeofworkloadandresourceallocation,whichislargelydue tothegranularityoftheonlinemonitoringandcontrol.Whentheworkloadisheavy, thisdelaycausestheapplication'sthroughputtouctuatealittlearoundtheidealone. However,theoverallerrorisstillverylow.About33%ofCPUcyclesaresavedbythis dynamicallocation,comparedtoaxedallocationwhereoverprovisionisbasedonthe highestload. 4.5.2.3Trace-basedworkload Inthethirdexperiment,the'98WorldCupWebsitetracecollectedonMay31from 5amto5pmlocaltimeinParisisusedtogenerateworkload,anditisplayedat12X speeduptoenhanceitsintensity.Allthedocumentsrequestedbythetracearecreated bythelogreplayertoolbasedonthesizesrecordedinthetrace.Becauseonlystatic Webpagesarerequestedinthetracereplaying,theworkloadischaracterizedbythe requestrate. 78

PAGE 79

Figure4-13.Comparisonofthethroughputachievedbyusinglocalcontrollerandthe idealthroughputforthetrace-basedworkload. Duringtheexperiment,therst30measurementsofworkloadandCPUconsumption areusedtoinitializethefuzzymodel.Afterthat,themodelisupdatedevery200 seconds.Figure4-13showsthattheapplication'sthroughputachievedbyusing thelocalcontrollerisclosetotheidealthroughputobtainablefortheworkloadthe differenceiswithin5%,indicatingtheeffectivenessofthefuzzymodelingapproach underreal-worldworkloads.Thedynamicallocationuseslessthan75%oftheCPU cyclesusedbyastaticapproachthatallocatesmaximumCPUfractionbasedonthe highestworkload. 4.5.3Fuzzy-PredictionApproach Similartothepreviousexperiment,threedaysofwebtracesfrom'98WorldCup Websitearechosentogenerateworkload.Thetracesareplayedat24Xspeedupto reduceexperimentduration.Duringtheexperiment,onlytheCPUutilizationofthe virtualcontainerismeasuredandfedtolocalcontrollerevery20seconds.Therst50 measurementsareusedtoinitializethefuzzyrules.Afterthat,therulebaseisupdated wheneverthenewCPUusagemeasurementisavailableevery20seconds.Every oneminute,thelocalcontrollerestimatestheCPUneedsforthenextminutebased onthefuzzyruleslearnedfromtheperviousobservationsandthensendsrequeststo 79

PAGE 80

Figure4-14.Comparisonofthethroughputachievedbydynamicallocationusingfuzzy predictionapproachandtheidealthroughputwithmaximalallocation. Figure4-15.TheCPUallocatedtothevirtualcontainerthroughtheinteractionoflocal andglobalcontroller. theglobalcontroller.ThentheglobalcontrolleradjuststheCPUallocationaccording totherequestsfromthelocalcontrollereveryminute.Figure4-14showstheresulting throughputforthetrace-workloadbyusingfuzzypredictionandthethroughputachieved byusingmaximalallocation.4GHz.Theresultsareverycloseandthedifferences betweenthemarelessthan1%onaverage.Figure4-15plotstheCPUallocatedtothe 80

PAGE 81

Figure4-16.CPUrequestsfromtwovirtualcontainersVC1,VC2thatsharelimited resources. virtualcontainerduringtheexperimentandabout44%resourcescanbesavedusing thedynamicallocation. Comparingthefuzzy-modelingandfuzzy-prediction,bothapproachescanproduce accurateshort-termresourcepredictionforlocalcontrollers.Fuzzymodelingapplies clusteringtechniquestoprovideconcisedatapresentation,resultingsmallerrulebase lessthantenfuzzyrulesduringtheexperimentsthanfuzzy-predictionapproachabout severaltensoffuzzyrulesintheexperiments. 4.5.4GlobalControllerValidation Thelastsetofexperimentsinvestigatestheglobalcontroller'sallocationoflimited resourcesamongmultiplecompetingvirtualcontainers.TwovirtualcontainersVC1, VC2runningonthesameservernodecompetefortheavailableCPUcyclesGHz. VC1servesaxedworkload,whichhasaconstantrequestrateof30requests/sec; whileVC2receivesanincreasingworkloadwitharequestraterisingfrom10upto60 requests/sec.TheworkloadsusedinthisexperimentonlyconsiderstaticWebrequests. Bothlocalcontrollersofthesetwocontainersemploythefuzzymodelingapproach todynamicallyestimatetheirCPUdemandsfortheworkloads,andtheamounts ofresourcesrequestedduringtheexperimentareplottedinFigure4-16.Thelocal 81

PAGE 82

Figure4-17.CPUallocationthatfavorsVC2. Figure4-18.CPUallocationthatfavorsVC1. controllerofVC1requestsaround500MHzofCPUthroughouttheentireexperiment; whileVC2increasesitsCPUrequestfromabout200MHztomorethan800MHzasits workloadgrows. WhentheCPUneededbyVC2goesbeyond500MHz,theglobalcontroller respondstotheresourceshortagebyreducingtheallocationtoVC1orVC2.The allocationpolicyoftheglobalcontrolleristomaximizeitsprotbyemployingthe greedyalgorithmdiscussedinSection4.5.Twosimplescenariosareconsideredin 82

PAGE 83

theexperiments.Intherstcase,theprotrateofVC2ishigherthanVC1;therefore, theglobalcontrollerdecidestosatisfytheresourcerequestsfromVC2byreducingthe allocationforVC1wheneveraCPUshortagehappens.Figure4-17showstheactual CPUallocationsforthetwocontainersthroughouttheexperiment.Thesecondcase considerstheoppositesituationwhereVC1hasahigherprotrateandthusisfavored intheresourceallocation.Inthiscase,VC2suffersfromtheresourceshortagewhen theglobalcontrollercannotallocateenoughresourcesforbothcontainersFigure4-18. 4.6RelatedWork Tothebestofourknowledgethereisnopriorworkusingafuzzymodeling approachtodatacenterresourcemanagement.Thefollowingbrieysummarizes otherworkwithsomecommonelementswiththeapproachdescribedinthischapter. Rule-basedsystems:Thisapproachusesasetofevent-condition-actionrules denedbysystemexpertsthataretriggeredwhensomepreconditionissatisede.g., whensomemetricsexceedapredenedthreshold.Forexample,theHP-UXWorkload Manager[99]allowstherelativeCPUutilizationofaresourcepartitiontobecontrolled withinauser-speciedrange,andtheapproachofRoliaetal.[100]observesresource utilizationconsumptionbyanapplicationworkloadandusessomexedthresholdto decidewhethercurrentallocationissufcientornotfortheworkload.Withthegrowing complexityofsystems,evenexpertsarendingitdifculttodenethresholdsand correctiveactionsforallpossiblesystemstates. Controltheory:Approachesbasedoncontroltheoryhavebeenappliedtoresource managementtoachieveperformanceguarantees.Mostoftheworkassumesalinear relationshipbetweentheQoSparametersandthecontrolparameters,andinvolves atrainingphasewithagivenworkloadtoperformsystemidentication.Typically, controlparametersmustbespeciedorconguredofineandonaper-workload basis.Abdelzaheretal.[101]investigatedthisapproachforQoSadaptationinWeb servers.In[62][64],anonlinearrelationbetweenresponsetimeandCPUallocation 83

PAGE 84

toaWebserverisstudied,andabimodalmodelisusedtoswitchbetweenunderload andoverloadoperatingregions.Todealwithtime-varyingworkloads,morerecentwork appliesadaptivecontroltheory,inwhichmodelsareautomaticallyadaptedtochanges usingonlinesystemidentication. Model-based:Previousresearchefforts[69][102][64][103]havebeentryingto modelcomputersystemsfromdifferentperspectives.Bennanietal.[71]predictsthe responsetimeandthroughputforbothonlineandbatchworkloadsusingmulticlass openqueueingnetworks.Liuetal.[104]usesARmodelstomapCPUentitlement percentagetothemeanresponsetimewithaxedworkload.Chandraetal.[69]uses atime-domainqueueingmodeltorelatetheresourcerequirementstoitsworkload. Someoftheseapproachesmakesimplifyingassumptionssuchasusingasinglequeue tomodelthewholesystem,whichmayfailtocapturecomplexitiesoftherelationship betweenapplicationworkloadandresourceusage.Somemodelsarevalidatedonly usingsimulations. ReinforcementlearningRL:Tesauro[105]proposedtousereinforcementlearning forautonomicresourceallocation.Comparedwithourfuzzy-logic-basedapproaches, bothcanautomaticallylearnfrompastexperienceswithoutanexplicitperformance model.However,RLusuallyuseslookuptabletostoretheinformationitobtained fromtrainingdata.Thesizeoftableincreasesexponentiallywiththenumberofstate variables,causingthescalabilityissue.Fuzzy-logicbasedsystemkeepsitsknowledge moreefcientlyintheformoffuzzyrulesandfuzzymembershipfunctions.RLmay alsohavealongtrainingtimeduetotheabsenceofdomainknowledgeorgood heuristics,whiletheconstructionoffuzzyrulebaseinourapproachdoesnotrequire time-consumingtraining.In[106],theauthorsproposedtouseahybridRLmethod combiningRLandqueuingmodels,inwhichRLtrainsofineondatacollectedwhile aqueuingmodelpolicycontrolsthesystemtoavoidperformancedegradationinlive onlinetraining. 84

PAGE 85

Fuzzycontrol:Diaoetal.[16]proposedaprot-orientedfeedbackcontrolsystem formaximizingSLAprotsinWebserversystems.Thecontrolsystemappliesfuzzy controltoautomatetheadmissioncontroldecisionsinawaythatbalancestheloss ofrevenueduetorejectedworkagainstthepenaltiesincurredifadmittedworkhas excessiveresponsetime. Theproposedresourcemanagementsysteminthischapterdiffersfromtheprior workinthefollowingaspects: Theresourcecontrolfunctionsareseparatedbetweenresourceproviderand applicationprovider,whichmakesthedesignofdatacenterresourcemanagement moreexibleandrobust.Eachlocalcontrollertriestomaximizeitsownprots byrequestingjustenoughresourcesforsatisfyingapplicationSLAsaswellas reducingunnecessaryresourcecost.Theglobalcontrollertakesintoaccountthe tradeoffbetweenrevenueobtainedfromsatisedresourcerequestsandcostfrom violationsofresourceSLAs. Fuzzy-logic-basedapproachesprovideagenericapproachtorepresentingthe relationshipbetweensystemvariables.Itcanbeeasilyappliedtoanytypeof applicationshostedinvirtualcontainers.Thisapproachmakesnounderlying assumptionoftheworkloadcharacteristics,andcanlearnanytypeofrelationship veryfast.Especially,thefuzzysystemcanefcientlymodelthenonlinearsystem withdynamicallychangingoperatingconditions. Theresourcemanagementprocessisautomaticwithoutanyhumanintervention. Thefuzzyrulesareautomaticallylearnedfromonlinemonitoringdataandthe knowledgebaseisupdatedcontinuouslyasnewdataarrives,enablingthesystem tocapturetransientorunexpectedworkloadchanges. 4.7Conclusions Thisworkpresentsaexibletwo-levelresourcemanagementsystemthatis abletoprovidehighqualityofservicewithmuchlowerresourceallocationcoststhan worst-caseprovisioning.Atapplicationlevel,inordertomakelocalcontrolleraccurately estimatetheresourcedemandsfordifferentworkloads,twofuzzylogicbasedmethods -fuzzymodelingandfuzzyprediction-areproposedtoguideresourceallocationbased ononlinemeasurements.Bothapproacheshavetheadaptivelearningabilityrequiring nodomainknowledgeaboutthesystem.Specically,thefuzzymodelingapproach 85

PAGE 86

characterizestherelationshipbetweentheworkloadsandthecorrespondingresource requirements,whilethefuzzypredictionbuildsamappingfromrecentresourceusage tofutureresourceneeds.Adaptivemeanstheknowledgecanbeeasilyupdated whennewinformationisavailabletoadapttothesystemchangesandreectthemost recentsystemconditions.Theglobalcontrollerattheresource-poolleveltriestondthe optimalresourceallocationbasedontheproposedprotmodel,towardsmaximizingthe totalprotofthedatacenter. Ourapproach,inconjunctionwithvirtualizationtechniques,canprovideapplication isolationandperformanceguaranteesinthepresenceofchangingworkloadsby dynamicallyallocatingresourcesatnetimegranularity,whichresultsinhighutilization andlowcostaswell.Theproposedresourcemanagementsystemisimplementedona virtualizeddatacentertestbedandevaluatedusingapplicationsthatarerepresentative ofe-businessandWeb-contentdeliveryscenarios.Bothsyntheticandreal-worldWeb workloadsareusedtoevaluatetheeffectivenessoftheapproach.Theexperimental resultsshowthatthesystemcansignicantlyreduceresourcecostwhilestillguaranteeing applicationQoSinvariousscenarios. 86

PAGE 87

CHAPTER5 MULTI-OBJECTIVEOPTIMIZATIONINVIRTUALMACHINEMANAGEMENT Chapter4introducedourinitialworkonresourcemanagementinvirtualizeddata centers.Itaddressesdynamicoptimizationofresourceallocationamongcompeting applicationsthatarerunningintheirvirtualmachinesandsharingphysicalresources.A simpliedprotmodelisusedbytheglobalcontroller,whichonlyconsiderstherevenue andcostofallocatingresourcestodifferentapplications.Thereremainsomequestions tobeanswered. 1.Whathappensifweconsiderotheroptimizationobjectivessuchasreducing operationalenergycost,avoidingthermalhotspots?Howcananoptimalsolution befoundwithmultiplepossiblyconictinggoals? 2.HowcanvirtualmachinesVMsbeplacedonphysicalserversandhowdoes theplacementaffecttheoptimizationofdatacenterresourcemanagement? Consideringalargenumberofvirtualmachinesandhosts,howcanthebest placementbefoundinanefcientway? 3.Howcanvirtualmachinemigrationsbeusedtoadapttodynamicallychanging environmentsindatacenterswithoutincurringhighoverhead? Inthischapterthesequestionsareaddressedasfollows.Section5.2.1and5.2.4 discussthemultipleobjectivesconsideredinmanagingvirtualmachineplacementand howtouseafuzzy-logic-basedapproachtocombinedifferentobjectives.Section5.2.3 and5.2.5presentanimprovedgeneticalgorithmtoefcientlyandgloballysearchthe spaceofvirtualmachineplacementsolutionswithfuzzymulti-objectiveevaluation.In section5.3dynamicvirtualmachinemigrationtoadapttochangesofsystemconditions andworkloadsisaddressedusingacross-layerapproach. 5.1ProblemDescription 5.1.1Multi-ObjectiveOptimization Agreatamountofworkhasbeendevotedtotechniquestooptimizeresource managementofadatacenter.Earlierworkmostlyfocusesonimprovingresource usagewhilemaintainingapplicationperformanceguarantees[107][50][108].Currently, 87

PAGE 88

powerconsumption[7][8][109]andthermaldissipation[110]aresignicantcontributors todatacenteroperationalcosts.Toreducethesecosts,theuseofvirtualizationto consolidateworkloadsandturningoffunloadedservershasbeenproposedtoachieve greaterenergysavings[6][75][52][111].Workin[112][9]proposedatemperature-aware workloadplacementapproachtominimizepeaktemperature.Mostoftheextant workfocusesononlyoneoratmosttwospecicaspectsofmanagement,suchas minimizingpowerconsumption,balancingthermaldistribution,ormaximizingresource usage.However,thesemaybeconictingobjectiveswhenconsideredalltogether.For example,tightlypackingvirtualmachinesontoasmallnumberofserversandturning offotherserversisaneffectivewaytoreduceenergycosts.However,concentrating workloadonasubsetofthesystemresourcescancauseheatimbalancesandcreate hotspots,whichmayimpactcoolingcostsanddegradeserverlifeandperformance.An effectiveVMplacementstrategyshouldconsidertradeoffsamongalltheseobjectives. 5.1.2VirtualMachineVMPlacementandMigration Twotypesofvirtualmachineplacementareconsideredinourwork.Initialor staticVMplacement,todecidehowtoplaceanumberofvirtualmachinesatonceinan unloadeddatacenter,consideringbothVMrequirementssuchasCPU,memoryandIO bandwidth,andphysicalhostcapacitiesandplatformrequirementssuchaspowerand temperature.DynamicVMplacement,todynamicallyreassignVMstohostsdueto thechangesofsystemconditionsorVMrequirementscausedbydynamicworkloads. Forbothtasks,theglobalcontrollertriestosimultaneouslyoptimizemultiplepotentially conictingobjectives,includingtheeliminationofthermalhotspots,theminimization oftotalpowerconsumption,andachievingdesiredapplicationperformance.However, thesetwotypesoftaskshavedifferentcharacteristicsandshouldbesolvedusing differentstrategies.TheVMinitialplacementhaslong-termeffectsbecauseextensive changeinVMplacementisimpracticalduetotheresourceoverheadincurredandtime consumedbymultipleVMmigrations.Also,theinitialplacementtypicallyoccursmuch 88

PAGE 89

lessfrequentlythandynamicplacement,e.gwhenthedatacenterstartsitsoperation, afterreset/idlestates,specialusageregimesandwhendynamicprovisioningleadsto unsatisfactorystates.IntheseandotherinitialVMprovisioningscenarios,theglobal controllercantakealongtimerelativetodynamicplacementtodeterminetheVM placement,whichisanNP-hardoptimizationproblem.Asophisticatedalgorithm,such astheoneproposedinourworkand[9][55]shouldbeusedtosearchthesolutionspace globallytoachievebetterperformance.Fordynamicplacement,toquicklyadaptto changesinthesystemorworkloads,thecontrollerisrequiredtomakedecisionsat runtimesoasto,forexample,reduceperformancelossesormitigatethermalanomalies. AnotherimportantconsiderationfordynamicVMplacementistheresourceoverhead includingpowerconsumptionandperformancelossincurredfrommigratingVMs. AcompletenewVMplacementwithoutconsideringcurrentplacementisimpractical. Beingoneoftheoptimizationmethodswiththeleastcomplexityandoverhead,anonline localsearchheuristicisapossiblealternativeforVMdynamicplacementandtheone proposedinourwork. ThefollowingtwosectionsdetailourworkoninitialVMplacementanddynamicVM migrationusingmulti-objectiveoptimizationapproach. 5.2InitialVirtualMachinePlacement Inthescenarioconsideredinthisworkeachuserrequeststheuseofoneor moreapplicationswithanexpectedqualityofservicethatrequiresacertainamountof resourcesandthedatacenterrespondstotherequestbydeployingavirtualmachine dedicatedtotheapplicationsandallocatingrequiredresourcestoit.Twotypesof resourcemappingareinvolvedthemappingfromapplicationworkloadtoresource requirementsandthemappingfromvirtualresourcestophysicalresources.Based ontheworkdescribedinChapter4,atwo-levelcontrolarchitectureseeFigure5-1 naturallysupportsthesetwomappingsthroughlocalcontrollersatthevirtual-machine levelandaglobalcontrollerattheresource-poollevel.Alocalcontrollerimplemented 89

PAGE 90

Figure5-1.Two-levelcontrolarchitectureandinformationowforautonomicresource managementinavirtualizeddatacenter. ineveryvirtualmachineisresponsiblefordeterminingtheamountofresourcesneeded byanapplicationandaskingformoreorlessresourcestoguaranteeapplication performanceatminimumcost.Aglobalcontrollerdeterminesvirtualmachineplacement andresourceallocation.TheworkdescribedinChapter4focusedonthedesignoflocal controllersthatusefuzzylogic-basedmodelingapproachestoadaptivelymodelthe relationshipbetweenapplicationworkloadsandtheirresourcedemands.Thischapter concentratesonthedesignoftheglobalcontrollerattheresourcelevel. 5.2.1Multi-ObjectiveVMPlacementDecision AsillustratedinFigure5-1,theglobalcontrollerreceivesresourcerequestsfrom userswhichareexpressedasVMswithspecicresourceneeds.ThesizeofaVMis representedasa d -dimensionalvectorinwhicheachdimensioncorrespondstoonetype oftherequestedresourcese.g.,CPU,memoryandstorage.Resourcesonphysical serversareallocatedasslicesalongmultipledimensionsaccordingtotheresource demandsofVMrequestsseeanexampleinFigure5-2.EachVMisassignedtoa sliceofaserverandtheresourcesconsumedbytheVMareboundedbythesizeofthis slice.Themonitoringsystemofthedatacentermeasuressysteminformationincluding resourceusage,powerconsumptionandtemperatureofeachserverandcollectsthem 90

PAGE 91

Figure5-2.AnexampleofresourcesallocatedtothreeVMsplacedintoasinglephysical machine. intoacentralizedprolingrepository.Theprolingandmodelingcomponentsutilizethe systemmeasurementstocreatemodelsofpowerandtemperature,whichareinturn usedbytheglobalcontrollertooptimizeitsplacementdecisions,whichconsidersthe followingfactors. ResourceWastage: Theresidualresourcesavailableoneachhostmayvary largelywithdifferentVMplacementsolutions.Inanticipationoffuturerequests,the resourcesleftoneachservershouldbebalancedalongdifferentdimensions.Otherwise, unbalancedresidualresourcesmaypreventanyfurtherVMplacement,thuswasting computingresources.AsFigure5-2illustrates,theoutsideshadedrectanglerepresents thetotalCPUandmemorycapacityofaphysicalserver.Thehost'sresourcecapacityis reducedalongeachdimensionbyplacingthreeVMsandallocatingresourcestothem. ThethreesmallrectanglesindicatetheamountofresourcesallocatedtoeachVM. Thecrosshatchedareaintheguredenotestheresidualresourcesavailableforfuture allocation.IntheexampleofFigure5-2thehosthasalotofunusedCPUcapacitybut littlememoryavailablecausingthehosttonotbeabletoacceptanynewVMbecauseof memoryscarcity. Tobalancetheresourceusagealongdifferentdimensions,thefollowingnotationis usedtocalculatethepotentialcostofwastedresources. R i representsthenormalized 91

PAGE 92

residualresourcei.e.,thepercentageofresidualresourcetothetotalcapacity alongdimension i .Usingsubscript k toidentifythedimensionthathasthesmallest normalizedresidualcapacity,thewastedresidualresourceonaserveriscalculated asthesumofdifferencesbetweenthesmallestnormalizedresidualresourceandthe others W = P i 6 = k [ R i )]TJ/F23 11.9552 Tf 12.292 0 Td [(R k ] .Therefore,thebiggerthedifferencesofresidualresources areamongdifferentdimensions,themoreresourcesarewasted. OperationalPower: Powerconsumptiontendstovarysignicantlywiththe actualcomputingactivity.Extensiveresearchworkhasbeendonetoestimatepower consumptionusingperformancecountersorsystemactivitymeasurements.Basedon theresultsfromprolingthepowerconsumptionofanIBMBladeCenterseedetailed experimentaldatainSection5.4.2,acommonlyusedlinearpowermodel[75][113]is usedinourworktoestimatethepowerconsumption.Inordertosaveenergy,servers areturnedoffwhentheyareunloadedalternatively,lowpowerstates[114]otherthan power-offcouldbeconsideredwithintheframeworkofourapproach..Thetotal operationalpower C consumedbytheserversiscalculatedas C = X j [ P j ] ;p j = 8 > < > : p 1 + p 2 U CPU j if U CPU j > 0 0 otherwise ,where P j and U CPU j denotethepowerconsumptionandCPUutilizationof j thserver. Thermaldissipation: Thermalperformanceisoneofthecriticalmetricsin datacentermanagement.Sharpspikesinserverutilizationmayresultindisruptive downtimesduetogeneratedhotspots.Accordingtothewell-knowndualitybetween heattransferandRCcircuitelectricalphenomena[110],athermalRCcircuitcanbe usedtomodelthesteadystatetemperatureofaprocessor T = PR + T amb where P denotesthepowerconsumption, R denotesthethermalresistance,and T amb isthe ambienttemperature.Usingthelinearrelationshipbetweenpowerconsumptionand CPUactivity,thetemperatureisrelatedtotheCPUloadofthehostaccordingtothe 92

PAGE 93

Table5-1.SymbolsusedinVMplacementproblemformulation. SymbolsExplanation M Numberofphysicalservers N NumberofVMrequests [ c CPU j c mem j ]j=[1... M ]Capacityvectorofthe j thserver [ r CPU j r mem j ]i=[1... N ]Resourcerequirementsofthe i thvirtual machine a ij 2 [0 ; 1] Allocationmatrixinwhich a ij =1 if vm i is allocatedtothe j thserver W j Resourcewastageofthe j thserver P j Powerconsumedbythe j thserver T j Temperatureofthe j thserver U CPU CPUutilization U mem Memoryutilization equation T = p 1 + p 2 U CPU R + T amb .ThislinearrelationshipbetweenCPUtemperature andCPUactivityisconrmedbyourprolingstudyofCPUtemperatureconductedon anIBMBladeCenterseedatainSection5.4.2.Furthermore,researchworkhasshown thatthecoolingcostincreasesifthetemperatureisunbalancedacrossadatacenter [112][9].Thermalmanagementindatacentersaimsatmitigatingindividualhotspots, keepingtemperaturewithinasafeoperatingrangeandbalancingtemperatureacross datacenterservers. Eachoftheabove-discussedfactorsrepresentsanoptimizationobjectiveduringVM placement.Basedontheobservedsystemstates,theglobalcontrollerutilizesthepower andthermalmodelstoestimatethefuturesystemstateandselectthebestplacement basedontheoptimizationcriteria.Manyresearcheffortshavefocusedonpowerand thermalmanagementleadingtoproposalsofdifferentpowerandthermalmodels.One advantageofourproposedglobalcontrolleristhatitcanincorporateanytypeofmodels, dependingontheinvestigatedsystems.Themodelsusedinourworkwereinferredfrom measurementsontheIBMBladeCentersystemmentionedinSection5.4. Theproposedmulti-objectiveVMplacementoptimizationproblemisformulatedas followsTable5-1liststhesymbolsusedinthisrestofthechapter: Goal: min P M j =1 W j and min P M j =1 P j and minmax T j 93

PAGE 94

Constraints: P N i =1 r CPU i a ij
PAGE 95

Figure5-3.AnexampleofVMplacementanditscorrespondingchromosome. toasageneration,anewsetofstringsiscreatedbygeneticoperationscrossover andmutationonthecurrentsolutionpooltoformthenewgeneration.Incrossover operations,twoindividualsi.e.solutionsareselectedasparentsandportionsoftheir stringsareusedtoproducenewstringsrepresentingnewsolutions.Mutationisanother commongeneticoperatorthatisappliedtoasinglechromosomeinwhichsomeofthe stringsarerandomlyselectedandchanged. 5.2.3GroupingGeneticAlgorithm Thegroupingproblemistogroupasetofitemsintoacollectionofmutuallydisjoint subsets.Falkenauer[116]pointedoutthataclassicgeneticalgorithmGAperforms poorlyongroupingproblemssuchasbin-packingandintroducedthegroupingGA GGA,whichisaGAheavilymodiedtosuitthestructureofgroupingproblems.A specialencodingschemeisusedinGGAinordertomaketherelevantstructureof groupingcorrespondtogenesinchromosomes.Inaddition,specialgeneticoperators forcrossoverandmutationareusedtosuitthestructureofchromosomes.Tofurther improvetheperformance,anewoperator,calledranking-crossover,ispropsoedand usedinthiswork,asexplainedinthefollowing. Encoding: Ingroupingproblems,theobjectivefunctionisdenedovergroups ratherthanisolatedobjects.Therefore,theencodingschemainGGAisgrouporiented. Figure5-3illustratesanexampleofvirtualmachineplacement.Ninevirtualmachines arepartitionedintothreegroupsoneperphysicalmachineandthecorresponding 95

PAGE 96

chromosomefeaturesthreegenes,eachofthemencodingagroupofvirtualmachines allocatedtooneofthreeservers. Crossover: TheaimoftheGGAcrossoveroperationistoproduceoffspringout oftwoparentsinsuchawaythatthechildreninheritasmuchaspossibleofuseful informationfrombothparents.TheGGAcrossoverrandomlyselectsaportionoftherst parenti.e.,someofthegroupsandinjectsitintothesecondone.SomeVMscould appeartwiceinthesolution,sothegroupsphysicalmachinescontainingtheminthe secondparentareeliminated.SomeoftheVMscouldbemissingasaresultofthis eliminationstepGGAusesalocalheuristic,suchasrst-twhichthisworktried,to reinsertthemissingVMs. However,thiscrossoveroperatordoesnotperformveryefcientlyinourcase becausetheinheritanceisperformedcompletelyblindlywithrandomselectionand insertion,anditisunlikelytoobtaingoodresultsfromarelativelysmallnumberof trials.Aranking-crossoverisproposedtoenablenewgeneratedsolutionstoinherit thegoodfeaturesfromtheirparentsmoreefciently.Therststepistoevaluate eachindividualgroupaphysicalserverwithitshostedVMsbasedonthreetypes ofefcienciesdiscussedbelowwhichcorrespondtothethreeaforementioned optimizationobjectives. Resourceusageefciency E resource : Itreectshowwelltheresourcesofdifferent typesareutilized.Thegoalistofullyutilizetheresourcesinalldimensions.Inthecase ofresourceswithCPUandmemory,theefciencyisdenedastheproductofCPU usageandmemoryusage,i.e., E resource = U CPU U mem Powerconsumptionefciency E power : Itreectshowmuchusefulworkisproduced undercertainpowerconsumption. E power = workload power = U CPU p 1 + p 2 U CPU p 1 + p 2 thefactor p 1 + p 2 isusedtomaketheefciencyvaluefallinto [0 1] range.Thepowerconsumption efciencymonotonicallyincreaseswithCPUusage,andreachesthehighestpointwhen CPUusageis100%. 96

PAGE 97

Thermalefciency E thermal : Alogisticfunctionisusedtocalculatethethermal efciencyas E thermal =1 = + e T )]TJ/F24 7.9701 Tf 6.586 0 Td [(Ts .Theefciencyvaluedecreasesrapidlywhen theCPUtemperaturegoesbeyondthesaferange T s issetto70 Cintheexperiments discussedinSection5.4. Thevaluesofallthesethreeefcienciesareinthe [0 1] range.Thegroup evaluationfunctionusestheaveragevalueofthethreeefcienciestoevaluategroupsin eachsolution.Thenextstepistocomposeanewsolutionbyselectingthegroupsfrom theparentsinadeceasingorderofevaluationvalues.Whenagroupisselected,itsVMs thatappearinapreviouslychosengroupareeliminated.Inthisway,thenewgenerated solutionsinheritthegoodstructuredgroupsandareabletoevolvetobettersolutions quickly. Mutation: GGA'smutationisalsogrouporiented.Afewgroupsphysicalmachines arerandomlyselectedandeliminated.Thevirtualmachinesinthosegroupsare insertedbackinarandomorderusingarst-theuristicalgorithm.However,the evaluationseedetailsinSection5.4showedthatthisoperationisnotveryusefulfor ourcasebecauseoftheblinddeletionandinsertion. 5.2.4FuzzyMulti-ObjectiveEvaluation Theproposedvirtualmachineplacementattemptstominimizeseveralpossibly conictingobjectives.InordertouseGGAtosolvemulti-objectiveproblems,thetness functionusedforselectingnewgenerationsofcandidatesolutionsshouldreectall objectives.ThepotentialsolutionsobtainedthroughGGAareevaluatedusingthe followingproposedfuzzy-logic-basedsystem. Considerourvirtualmachineplacementproblemwherethegoalistominimize resourcewastage,powerconsumptionandmaximumtemperature.Threelinguistic variablesresourcewastage w power p and temperature t -areintroducedand onelinguisticvalueisdenedforeachvariable,namely,fuzzysets smallresource wastage sw lowpowerconsumption lp and lowtemperature lt .Membership 97

PAGE 98

functionsofthesefuzzysetsaredecreasingfunctionsofthevariablevalues,sincethe smallerthevalue,thehigheristhedegreeofsatisfaction.Thesearchalgorithmseeks tondthesolutionsthatarenearesttoeachindividualgoal.Hence,theevaluationofa solutioncanbeexpressedbythefollowingfuzzyrule: IFsolution x hassmallresourcewastage sw ,ANDlowpowerconsumption lp ANDlowtemperature lt ,THEN x isagoodsolution. Themostdesirablesolutionistheonewiththehighestmembershipinthefuzzy sets sw;lp;lt .Usingtheorderedweighted-averagingfuzzyoperatorproposedbyYager [117],theabovefuzzyruleevaluatestothefollowing, x = min w x p x t x + )]TJ/F23 11.9552 Tf 12.706 0 Td [( avg w x p x t x issetto0.5intheevaluationexperimentsdescribed inSection5.4,inwhich w x p x ,and t x representthemembershipdegree ofsolution x inthefuzzysetsdenedby sw lp ,and lt ,respectively. x isthe membershipvalueforsolution x inthefuzzysetofgoodsolutions.Thesolutionwith thehighest x istheonethatbestmeetsallthegoalsandisreportedasthebest solution. Themembershipfunctionsforthefuzzyset sw;lp;lt arelineardecreasingfunctions. Thefollowingcalculationisproposedtodeterminethelowerandupperboundsofthe membershipfunctions.ThetotalCPUandmemoryrequirementsofallVMrequests arerepresentedby R CPU and R mem ,andtheCPUandmemorycapacityofeach physicalserveris c CPU and c mem .Theidealminimumnumberofserversneededto hostalltheVMsis m min =max R CPU =c CPU ;R mem =c mem byassumingthatVMscan bepartitionedandeachserverisfullyutilizedbytheVMsrunningonit,therefore thelowerboundofpowerconsumptionis P lower = m min p 1 + p 2 .Themaximum numberofserversforhostingVMsis m max =min M;N ,sotheupperboundofpower consumptionis P upper = m max p 1 + R CPU p 2 .Todeterminethelowerandupperbounds ofresourcewastage,weuse r CPU i and r mem i torepresentthepercentageofCPUand memoryrequirementsofthe i thVM.Thelowerboundoftotalresourcewastageis 98

PAGE 99

Figure5-4.Improvedgroupinggeneticalgorithmwithfuzzymulti-objectiveevaluationfor initialvirtualmachineplacement. W lower = j P r CPU i )]TJ/F29 11.9552 Tf 12.183 8.966 Td [(P r mem i j andtheupperboundis W upper = P j r CPU i )]TJ/F23 11.9552 Tf 11.955 0 Td [(r mem i j .Based onthethermalmodeldiscussedinSection5.2.1,thelowerandupperboundsofCPU temperatureare T lower = p 1 R + T amb and T upper = p 1 + p 2 R + T amb 5.2.5GGAwithFuzzyMulti-ObjectiveEvaluation Figure5-4showsthemajorproceduresoftheproposedGGAalgorithmwithfuzzy multi-objectiveevaluation.Thealgorithmconsistsoftwoparts,randomlygenerating anumberofsolutionstoformaninitialpopulationandreproducingnewgenerations ofsolutionsfromtheexistingsolutionpool.Forourproblem,theinitialpopulationis producedasfollows. S permutationsofVMrequestorderingsarerandomlygenerated. 99

PAGE 100

ForeachVMrequestsequence,therst-talgorithmisusedtoallocatetheVMsto thephysicalservers.Inthisway, S differentplacementsolutionsaregenerated. DuringeachgenerationofGGAasetofoffspringareproducedbytheranking-crossover operatordiscussedabove.Thisoperatorensuresthattheoffspringinherittheirparents' importantproperties.Thecrossoverandmutationratearethepercentageofnew solutionsgeneratedfromexistingsolutionpoolforeachgenerationusingcrossoverand mutationrespectively.Thegenerationselectionisbasedtheevaluationofeachsolution usingtheproposedfuzzy-logicbasedmulti-objectiveevaluation.Allthreeobjectives aretransformedtotheircorrespondingfuzzysetsrepresentedbytheirmembership functions.Byevaluatingthefuzzyrule,themembershipvalueofeachplacement solutionisregardedasitstnessvalue.Anumber S ofthebestplacementsolutionsare chosenfromthesolutionpoolcomprisingboththeparentsandtheiroffspringfornew generation.Therefore,theaveragetnessofthepopulationandthetnessofthebest individualsolutionincreaseineachgeneration. 5.3DynamicVirtualMachineMigration ThissectiondescribesourworkonthedynamicVMplacementproblem.Themain decisionsrequiredtosolvethisproblemarewhen,whichandwheretomoveVMs.In mostpriorstudies,thetriggerfordynamicVMmigrationonlydependsoneitherthe statesofVMsortheperformanceoftheirhostedapplicationsortheresourceusage oftheirhosts,withoutconsideringtheotherimportantinformationfromtheplatform layersuchaspowerusageefcienciesandtemperaturedistribution.Incorporatingthe informationfromboththevirtualizationlayerandtheplatformlayer,weidentiedthree conditionsfordynamicVMplacementincludingthermalemergency,resourcecontention andlowpowerefciency,whichwillbeexplainedindetailinthissection. ThedecisionsofwhichandwheretomigrateVMsarebasedontheproposed multi-objectiveoptimizationapproachwiththemigrationcostbeingtakenintothe consideration.Inaddition,areliableandrobustdetectionandselectionapproachusing 100

PAGE 101

Figure5-5.Cross-layercontrolarchitectureandinformationowfordynamicVM managementinavirtualizeddatacenter. sliding-windowandtrendanalysisisincorporatedintothedecision-makingprocessfor dynamicVMplacementtoachievestablesystemstateandavoidwastingresourcesand timeforunnecessarycontrolactions. 5.3.1Cross-layerProling,ModelingandControlling Figure5-5showstheproposedcross-layerapproachformanagingvirtualmachines andtheirhostsinavirtualizeddatacenter.Boththeplatformlayerandvirtualization layerhavemultiplesensorsformonitoringresourceusage,powerconsumptionand temperatureofservernodes,aswellasresourceutilizationofindividualVMshosted onthethosenodes.Itisalsopossibletoincorporateapplication-levelperformance informationretrievedfrominsideofVMs.However,itisunclearwhatshouldbeaset ofstandardmetricsforapplicationperformanceduetothediversityofapplications coexistinginadatacenter.Inordertoprovideageneralframeworkforalltypesof datacenterenvironments,applicationperformancedataisnotusedinoursystem implementation.Sensordataarecollectedfromacrossmultiplenodesintoacentralized prolingrepositorywhichcanbeaccessedbyaglobalcontrollerformanagingboth thephysical-resourcelayerandvirtualizationlayerinadatacenter.Theprolingand modelingcomponentsshowninthegureutilizethemonitoredsystemdatatocreate modelsofpowerandtemperature,whicharelaterusedbytheglobalcontrollerto optimizeitsdecisions. 101

PAGE 102

5.3.2ConditionsforDynamicVMPlacement Asexplainedintheintroduction,dynamicVMplacementisneededinthefollowing threeconditions: ThermalEmergency :Thetemperatureofeachservershouldbemaintained belowathreshold,becauseoverheatingofcomponentswillcausethermalcycling,and eventuallyhardwarefailures.Inaddition,highservertemperaturewillincreasecooling costsbecauseadditionalcoolingenergyisrequiredtoeliminatehotspots.Iflocalized hotspotsaredetected,virtualmachineswithhighintensiveworkloadsaremovedto relievethesituationandreducecoolingcosts. ResourceContention: PerformancedegradationmayresultfrommultipleVMs competingforresources.MovingtheVMsoutofthetroubledhostcanenablethemto obtainthenecessaryresourcestomaintaintheirperformanceaswellasalleviatethe resourcecontention. LowEnergyEfciency: Whenaserverbecomesidleoritsutilizationisverylow,it stillconsumesalargeportionofenergycomparedtowhenitisbusythisiswasted energyaslittleornousefulworkisdone.Insuchsituation,largeenergysavingscan resultfrommovingalltheVMsoutoftheidlehostandturningoffthehost. Figure5-6showsthecontrolowofthecontroller.Afterretrievingmonitoringdata fromtheprolingrepository,theinitialthreadcalledmonitorthreadcreatedbythe controllercheckstheabovethreeconditionsinturn.Thereasonforsequentialcheckup isbecausethecontroldecisionsfordifferentconditionsmayconictwitheachother.The checkstakeplacesequentiallyaccordingtotheirprioritiesasspeciedbythesystem administratorsandtheirordercanbeeasilychangedwhennecessary.Whenevera conditionismetonamonitoredserver,thecontrollerinitiatesanewthreadcalledaction threadtodeterminetheactionsandexecutethemonthechosenVMsandservers.The reasonforthisistoavoidblockingthemonitorthreadfromcheckingotherserversdue totime-consumingactionssuchasVMmigrations.However,conictsandinconsistency 102

PAGE 103

Figure5-6.Controlowofthecontroller. issuesmayarisewhenmultipleactionthreadsaretryingtodeterminedestinationhosts fortheirVMsandmigratethematthesametime.Forexample,thesameservermay bechosenasdestinationhostbymultipleactionthreadsandbecomeoverloadedif multipleVMsaremigratedtoit.Inaddition,adestinationhostexperienceshighresource usageduringVMmigration,causingthecontrollerincorrectlytriggerVMmigration. Therefore,theprocessofdestinationselectionisconsideredasacriticalsectionanda lockisappliedtosynchronizeitamongmultipleactionthreadssothatthereisatmost onethreadinselectinghostprocessatanygiventime.Onceaserverisselectedas destinationhostbyanactionthread,itistemporarilymarkeduntilmigrationisnished sothatotheractionthreadswillnotselectitasdestinationandthemonitorthread discardthemonitoreddatafromthatserverduringthemigrationtoavoidtriggering migrations. 103

PAGE 104

5.3.3ControllerFunctionality Therearebasicallythreefunctionsimplementedinthecontroller:condition detection,VMselectionanddestinationselection.Thefollowingwillexplainthemin detail. ConditionDetection: Thisfunctiondetermineswhenacontrolactionsuchas migratingVMsandturningon/offphysicalserversneedstobeperformed.Asshown inFigure5-6,thecontrollerperiodicallychecksthethreeconditionsmentionedinthe previoussectionusingtheprolingdatageneratedfromthesensors,andifacondition ismet,thecontrollerstartsanewthreadtofurtherinvestigatethesituation.Thisis anevent-detectionproblemandsimpledetectionmethodsuchasthreshold-based detectioncanbeused.Forexample,whenthecurrenttemperaturemonitoredona serverexceedsapredenedthreshold,theserverisidentiedasahotspotandthe controllerisactivatedinordertopossiblytriggeraVM-migrationaction.However,in atypicaldatacentersetting,thehostedapplicationworkloadschangedynamically overtime.Thiscausesthesystemconditions,suchasresourceutilization,power consumptionandCPUtemperature,touctuateovertime.Withoutrecognizing thistransientnatureofvariation,thesingle-thresholddetectionmaytriggermany unnecessaryactionsandevenworse,causesystemoscillationse.g.,continuousVM migrationsandmachineon/offswitches. Tomakeconditiondetectionbothreliableandtimely,atwo-leveldetectionapproach isproposedasfollows.Fortherst-leveldetection,thebasicideaisthattoavoidfalse detectionovertransientchanges,i.e.,persistentobservationsofthresholdviolations overaperiodoftimearerequiredtotriggerthedetection.Asliding-windowdetectionis applied,inwhichtheanalysisofthetime-varyingdatasuchasCPUtemperatureand resourceutilizationisperformedoverthevaluescoveredbyawindowofnitelength. Themonitoringdataaresampledataconstantinterval,andthedatawindowkeeps slidingovertime.Ineachwindow,analarmistriggeredifthepercentageofdataoutside 104

PAGE 105

ofapredenedthreshold T level )]TJ/F21 7.9701 Tf 6.587 0 Td [(1 islargerthanavalueV.Therst-leveldetectionmay beincorrectfordatashowinganincreasingordecreasingtrend.Thereasonisthat althoughmostoftheabsolutedatavaluesinthewindowareoutsideofthesaferange, thenear-futuredataislikelytofallbackintothesaferangebecauseofthetrend.The secondleveldetectionusestrendanalysisoverthedataintheslidingwindowtopredict thenear-futuredatavalues.Ifthepredictedvalueviolatesapre-speciedthreshold T level )]TJ/F21 7.9701 Tf 6.587 0 Td [(2 ,asecondalarmisgenerated.Whenthereisnopriorknowledgeaboutthose time-varyingdata,alinearmodelisusedfortrendestimationandthemodelparameters aredeterminedusingthelinearleast-squaresapproach.Onlyifbothalarmsfromthe two-leveldetectionaretriggered,thecontrollerstartsathreadtofurtherinvestigate thesituationontheserverswhichgeneratealarmsanddeterminesthecontrolactions. Detailsonhowtoselectwindowsizesandthresholdsunderdifferentconditionsare discussedlaterinthissection. VirtualMachineSelection: Themostimportantcontrolactionconsideredinthis paperisVMmigration,aconvenientwayofmovinghostedworkloadaroundadata centerinordertomitigateathermalemergency,alleviateresourcecontentionorturn offhoststosaveenergy.TheVMselectionfunctiondetermineswhichvirtualmachines shouldbemovedconsideringthebenetsandcostsofthemigrations.Thefollowing discussesthestrategiesforselectingVMsunderthreeconditions,namely: ThermalEmergency: TheobjectiveofmigratingVMsunderthisconditionisto bringthetemperatureoftheoverheatedserverintothesaferangeassoonaspossible. PreviousresearchworkhasshownthatCPUtemperatureishighlyrelatedtoCPU activity[118][112][9].Therefore,thoseVMswithhighCPUutilizationshouldbethe rsttomoveinordertolowertheCPUtemperatureefciently.Atthesametime,VM migrationincursprocessingandIOoverhead,whichinturnincreasepowerconsumption andservertemperature.ThemigrationprocessforaVMmainlyinvolvescopyingthe VM'smemoryleandimagetothedestinationandalsokeepingtrackofwhichpages 105

PAGE 106

arebeingmodiediflivemigrationisapplied.MigratingVMswithsmallermemory leorfootprintcanreducethemigrationtimeandoverhead.Consideringtheabove twoaspects,theVMsontheaggedserveraresortedindecreasingorderofthe ratiooftheirCPUutilizationtomemorysizeUSR.ChoosingtomigrateVMswiththe largestvaluesofUSRmaximizestheCPUloadreductionandtheimpliedloweringof temperatureperdatabytesmoved. ResourceContention: ThegoalofVMmigrationunderthisconditionistomake VMsandtheirhostedapplicationobtainsufcientresourcestomeetperformance guarantees.TheVMselectionalgorithmneedstoidentifywhichVMsarecompetingfor theinsufcientresourcessincemovinganidleVMdoesnotimprovethesituation.On theidentiedserver,eachVM'sutilizationoftheresourceunderpressureisretrieved andtheaverageutilizationofallVMsiscalculated.OnlytheVMswithutilizationhigher thantheaverageareconsideredtobecompetingVMsandothersarelteredout.Also consideringthemigrationcost,thecompetingVMsarefurtherorderedinincreasing orderoftheirmemorysize.TheVMswithsmallersizearechosenformigration. Lowenergyefciency: Whenthecontrolleridentiedanidleserveroraserverwith lowresourceutilization,alltheVMsrunningontheserverneedtobemovedoutinorder tobringtheserverdown. DestinationSelection: TodetermineanewdestinationhostfortheselectedVM, thehostselectionfunctionhasthefollowingconsiderations. Temperature: AfterVMmigration,thedestinationhost'sCPUtemperaturemay riseduetotheaddedVM'sworkload-theselectionfunctionmustmakesurethe temperatureisstillinthesaferangeafterthemigratedVMisdeployed.Furthermore, recentresearchworkhasshownthatthecoolingcostincreasesifthetemperatureis unbalancedacrossadatacenter[112][9].PlacingaVMinthecoolestservercanhelp balancethetemperaturedistribution.Utilizingthetemperaturemodelinferredfrom 106

PAGE 107

monitoredsystemdata,theselectionfunctionisabletopredicttheCPUtemperatureof thedestinationhostafterVMmigration. Power: AlthoughmovingaVManditsworkloaddoesnotchangethetotalpower consumptionunderahomogenousdatacentersetting,theselectionofdestinationmay affectfuturepowerusage.Forexample,movingaVMtoanidleserverwillkeepitfrom shuttingdowntosaveenergy.Onthecontrary,tightlyplacingVMsonasmallnumber ofserverscreatesmoreopportunitiesforotherserverstobeturnedoff.Fromthepoint viewofsavingpower,itispreferabletoplacetheVMinthemostoccupiedserverthat stillhasenoughresources.Forasystemconsistingofheterogeneousservers,placing aVMindifferentplacesmayresultindifferentpowerconsumption.Theselection algorithmcanusepowermodelslearnedfromthemodelingprocesstopredictfuture powerconsumptionofcandidatedestinationhosts. Performance: Toavoidgeneratingnewresourcecontentionandguaranteeeach VM'sperformance,preferenceisgiventothehoststhathavethelargestamountoffree resourcesthataresufcienttohostthemigratingVMs. Everyoneoftheaboveconsiderationstriestooptimizethedestinationselection fromdifferentperspectives.However,theselectionresultsfromoptimizingdifferent objectivesmayconictwitheachother.Oneofthemostcommonmethodsfor multi-objectiveoptimizationistoconvertmultipleobjectivesintoasingle-objective functionusingaweightedsum.Itishardtoidentifytheappropriateweightsfordifferent objectivesbecausethevaluesoftheobjectiveshavedifferentunitsandranges.Tosolve thisproblem,threeutilityfunctionsnamedtemperatureefciency,powerefciencyand performanceefciency,aredenedoverthreeobjectives,respectively. Temperatureefciency:Thetemperatureefciencydecreasesmonotonically withincreasingtemperaturesothathightemperatureindicateslowefciency andviceversa.TheobjectiveistokeepCPUtemperatureinthesaferange,so thisutilityfunctionisdesignedtodecreaseslowlywhenthetemperatureisfar belowthesafethresholdanddropsrapidlywhenapproachingorgoingbeyondthe threshold.Manynonlinearfunctionssuchasexponentialfunctionsandpolynomial 107

PAGE 108

functionscanservethispurpose.Forsimplicity,weuseapolynomialfunctionin theformof EFF T =1 )]TJ/F23 11.9552 Tf 12.538 0 Td [(T m m 2 ,inwhich Eff T representstheefciency valuefortemperature T ,and m isthedegreeoftheutilityfunction.Inorderto maketheefciencyvaluefallinto [0 1] interval, T isthennormalizedusingthe function ^ T = T )]TJ/F24 7.9701 Tf 6.587 0 Td [(T low T high )]TJ/F24 7.9701 Tf 6.587 0 Td [(T low ,inwhich [ T low ;T high ] isthesaferangeofCPUtemperature T .Choosinganydegree m inasmallrangee.g., m 5 doesnotresultin largevariationsinefciencyvaluessothatitdoesnotaffecttheselectionresults signicantly. Powerefciency:Thisefciencyreectshowmuchusefulworkisproducedbythe consumedpoweronanactiveserver.ByusingthelinearpowermodelwithCPU utilization P = p 1 + p 2 CPU parameter p 1 and p 2 areobtainedfromthemodeling process,thepowerefciencyisdenedas Eff P = Workload Power = CPU % p 1 + p 2 CPU % p 1 + p 2 Inthisutilityfunction,theworkloadisrepresentedbytheCPUutilizationCPU% andthefactor p 1 + p 2 isusedtomaketheutilityvaluefallinto [0 1] range.The powerefciencyincreasesmonotonicallywithincreasingCPUusage,andreaches thehighestpointwhenCPUusageis100%. Performanceefciency:Itreectstheextentofuseofresourcesofdifferenttypes. Topreventresourcecontention,theefciencydecreasesrapidlywhentheusageof oneormoreoftheresourcesapproachesandexceedsthemaximumallowedfor guaranteeingtheworkloadperformance.Thesettingofmaximalresourceusageis showninTable5-2.Similartotemperatureefciency,theresourceutilizationused intheutilityfunctionofdifferenttypesisnormalizedinto [0 1] range.Theutility valueofconsolidationissettotheminimumvalueofefciencyamongdifferent resourcetypes. Withalittleknowledgeaboutthesystem,theutilityfunctionsnotonlyhelpsmoothly expressthedegreeofpreferencedesirable,tolerable,undesirableetcunderdifferent valuesforeachobjective,butalsonormalizetherangesofalltheobjectivesintothe sameinterval[0,1].Therefore,itiseasytocombinethemintoasingleobjectivefunction consistingofthesumofthreeutilityfunctions. GiventheresourcerequirementsoftheselectedVM,thedestinationselection algorithmiteratesoveralltheavailableserversandretrievestheircurrentresource utilizationinformation,andpredictsthefuturestateincludingtemperature,power consumptionandresourceutilizationoneachserveraftertheselectedVMisdeployed usingthepowerandtemperaturemodelslearnedfromthemodelingfunctions.The algorithmthencalculatesthecombinedutilityvaluesforeachcandidatehostandthe 108

PAGE 109

serverthatreturnsthehighestvalueischosenasthedestinationhostforthemigrating VM. 5.3.4StabilizationConsiderationsandComplexityAnalysis Datacenterstypicallyhostavarietyofapplicationshavingdynamicallychanging workloads,andthereforetheresourcerequirementsoftheVMsandalsotheresource utilizationoftheirhostsmaychangedramaticallyovertime.Becauseofsuchvariations, aseeminglysuitableVMplacementormigrationdecisionscanquicklybecome inappropriatewithrespecttoperformance,temperatureorpowerondestinationhosts, whichinturnwouldgeneratemoreunnecessaryandcostlymigrationactions.Even worse,thesystemstate 1 mayoscillateduetocontinuousVMmigrationthatkeep triggeringnewmigrationsand/orserverturnon/offactions.Thedecisionsonwhenand wheretomoveVMs,andwhentoturnon/offserversshouldnotonlybebasedonthe currentconditions,butalsoonthedesirabilityofsuchadecisionoveracertaintime intothefuture.Toensuretheperformanceandstabilityofthecontroller,thefollowing approachesforparameterselectionareappliedinthethreefunctionsofthecontroller. Whentomove: Asdiscussedinthebeginningofabovesection,aslidingwindow detectionapproachisusedtoreducefalsealarmsontransientchanges.Asmallwindow sizeproducesfastandaggressivedetection,whilelargevaluescauseslowdetectionbut atthesametimecanlteroutmoretransientthresholdviolationsandavoidunprotable migrations,thereforethewindowsizeselectionhasanimpactontheperformanceof detections.Thetrendanalysisalsoavoidsthesituationwhentheresourcedemandsof themovedVMsfallbacktonormalrangesasbeforesoonafterthemigration,making 1 Wereferthestateofahostattimetasitsresourceusage,temperatureandpower consumptionatt,andthestateofadatacenteristhecollectionofthestatesofallhosts. ThestateofaVMisrepresentsbyitsresourcedemands.Astablestatemeansthatthe percentageofmonitoreddatarepresentingthestateinsideofapredenedrangeis largerthanavalueVoveraspeciedperiodoftime T 109

PAGE 110

thecontrolactionsunnecessaryandwastefulofresourcesandpower.Oneofthe necessaryconditionstoensureprotablemigrationsisthattherateatwhichstate changesofahostoraVMoccurisslowerthantherateofVMmigrationisperformed. Inanotherword,thestableintervali.e.,theperiodofadetectedhostoramovedVM stayinginstablestatemustbelongerthanthemigrationtime.IfaVMorhostisina stablestateforaperiodoftime T ,itispredictedthattostayinthatstateforanother Twithhighcondencesothatthefuturestableinterval T stable isestimatedas T .The necessaryconditionisthat T stable >T migration .Thedetectionalgorithmprescribesstable statesoftheVMsandtheirhostsmustbeobservedoveraslidingwindowbeforean alarmistriggered,therefore,thewindowsizeisregardedasthestableinterval.By makingthewindowsizelargerthanatypicalVMmigrationtimepreventsunnecessary migrationtoacertaindegree.Inourimplementation,themigrationtimeismeasured atabout30secondsforatypicalsizeVMwith1024Mmemory,theslidingwindow T window issetto2minuteswhichisconsideredlargeenoughtoproduceprotable migrationwhilenotdelayingthedetectiontoomuchinourimplementationsothat aVMispredictedtostayinitsstateafterthemigrationforatleastaperiodoftime T window )]TJ/F23 11.9552 Tf 12.449 0 Td [(T migration andthemigrationispredictedtobringsomebenettothesystem. Thesimilaranalysiscanbeappliedtoserverturningon/offactions.Thewindowsize fordetectinglowpowerefciencyissetto6minutes,whichisfourtimesoftheaverage periodforturningon/offabladenodemeasuredfromourIBMBladeCenter. Thethresholdsforconditiondetectionsusedinourprototypeimplementationare listedinTable5-3.Thereasonforsettinglevel-1thresholdslowerthanlevel-2thresholds isthatthedetectionalgorithmcancatchtheascendingtrendfasterandavoidtriggering thedescendingtrend.ThroughprolingvarietyofworkloadsonourIBMBladeCenter testbed,forexperimentpurposeweobservedthattheCPUtemperatureofablade nodevariesbetweenabout 20 55 dependingontheintensityoftheworkloads.The level-1andlevel-2thresholdsforthermalemergencyaresetto 48 and 50 .Forresource 110

PAGE 111

contention,weset90%ashighutilizationthresholdsforCPU,IOandnetworkthe reasonforsetting90%insteadof100%istogiveasmallbufferfortransientutilization changes. Wheretomove: Similartotheabovediscussion,wewanttomakesurethatthe destinationhostsareinastablestatetoavoidgeneratingnewhotspotsorresource contention,whichinturntriggersnewmigrationsandpossiblycausesystemoscillation. Toincorporatestabilityandtrendanalysisintodestinationselection,thealgorithm usesnotonlythemostrecentmonitoringdataoftheservers,butalsohistoricaldata forcalculatingthepredictedcombinedutilityvalues.Let [ d t )]TJ/F23 11.9552 Tf 12.661 0 Td [(W ;:::;d t )]TJ/F15 11.9552 Tf 12.662 0 Td [(1 ;d t ] representthetime-varyingmonitoringdataoveraperiodof W ,and t and t be theirtime-varyingmeanandstandarddeviation.Thetrend tr t iscalculatedusing least-squaresasdiscussedinSectionB.Attime t ,thepredicteddataiscalculatedas ^ d t = t + t + tr t W .Withthesecondterm,theselectionalgorithmprefersnotto selectahostwithhighlyvaryingsystemstate. Duringthemove: DuringtheVMmigration,theoriginalanddestinationhostsmay experiencetemporaryhighCPU,diskandnetworkIOusage.Toavoidtriggeringfurther VMmigrationonthosehosts,thecontrollerwillignorethemonitoringdatacollectedfrom themuntilthemigrationnishes. Toevaluatethecomplexityofthecontroller,supposethereareMphysicalmachines andeachmachinehostsNVMs.Thecontrollerperiodicallycheckseveryserverinthe datacenter,whichtakes O M time.Oncethedetectionistriggeredonaparticular host,theVMsearchalgorithmsortsallVMsaccordingtothecriteriadiscussedinthe abovesectionrunningonthathost,andthesortingalgorithmhas O NlogN complexity. ThecontrolleriteratesovereveryVMinthesortedorderandstopsifadestinationhost iffoundfortheVMbeingchecked.Sincethehostsearchingalgorithm'scomplexityis O M ,theworstcaseforselectinghostsforVMsis O M N ,inwhicheveryVMon thehostischecked.Thecombinedcomplexityofthealgorithmsusedinthecontrolleris 111

PAGE 112

O M N + O NlogN .ThemaximumnumberofVMscanbehostedonasinglehost islimitedbytheavailablephysicalresourcessuchasCPUandmemory,therefore,the combinedcomplexityislinearonthenumberoftotalphysicalhosts. 5.4ExperimentalEvaluation Thissectionsummarizestheexperimentalevaluationoftheproposedcontrol systemforvirtualmachineplacementandmigrationinadatacenterenvironment. Section5.4.1andSection5.4.2discussthevirtualizeddatacentersetupforthe experimentsandthemodelingdataobtainedfromprolingthetestbed,respectively. InSection5.4.3,theproposedmulti-objectiveGGAplacementalgorithmisevaluated usingasetofsimulationexperimentsandcomparedwithtraditionalbin-packingofine algorithmsandalsosingle-objectiveGGAapproachestoshowitsperformance, scalabilityandrobustnessoverawiderangeofenvironments.Thesimulationuses themodelingparametersobtainedfromprolinganIBMBladeCenter,sothattheresults canaccuratelycapturetherealsystembehavior.Section5.4.4discussestheevaluation oftheproposedapproachtodynamicVMplacementandmigrationonanexperimental testbedbuildupbyanIBMBladeCenter. 5.4.1ExperimentalSetup WebuiltourtestbedonanIBMBladeCenterwithnineHS21bladesconnected through2gigabitswitchmodulesforhostingvirtualmachines,andanIBMTotalstorage disksystemprovidingstorageforVMdiskimagesoverNFS.Eachbladenodehas twoXeonDual-CoreWoodcrest2.33GHzprocessorsand8GBRAM,oneofthemis reservedforrunningtheglobalcontroller,andeightofthemareusedforhostingLinux virtualmachinesinjectedwithdifferenttypesofworkloadofvariableintensities. 5.4.2ProlingandModeling Inordertoobtainthepowerandthermalmodelsrequiredbytheglobalcontroller, weusedIBM'sadvancedmanagementmodule[42]tomeasurepowerconsumptionand CPUtemperatureofBladeservers.TheIBMBladeCenterhas14HS21bladeseach 112

PAGE 113

Figure5-7.PowerconsumptionwithvaryingCPUutilization. Figure5-8.CPUtemperaturewithvaryingCPUutilization. withtwoXeonDual-Core2.33GHzprocessorswith4MBL2cache,8GBRAM,and 73GBSASdisk. Figure5-7showsthepowerconsumedbythebladeserverandFigure5-8plots theCPUtemperatureoffourcores 2 withrespecttoCPUutilizations.Itisclearthatthe 2 Thetemperaturemeasurementsusedformodelingcapturetheaveragetemperature ofthefourcoresonaserver. 113

PAGE 114

powerconsumptionislinearlyrelatedtotheCPUutilizationandtheCPUtemperatureis alsoanapproximatelylinearfunctionofCPUutilizationwithoutconsideringtheeffectof heatgeneratedbyotherserversnearby. 5.4.3EvaluationofMulti-ObjectiveInitialVMPlacement Weusethreesetsofexperimentstoevaluatetheproposedmulti-objectiveVM placementapproachwithrespecttoperformance,scalabilityandrobustness.The virtual-machineCPUrequestsmeasuredinGHzareuniformlydistributedoverthe set0.250.511.522.534andmemoryrequestsmeasuredinGBareuniformly distributedovertheset0.250.511.522.534tosimulatedifferentsizesofVM requests.ThenumberoftheavailableserversandtheVMrequestsarecongured duringthesetupphasetosimulatedifferentsizesoftheproblem.Table5-4liststhe parametersetupforthethreesetsofexperiments.Foreachsetting,wegenerated randominputsandrantheexperiments20timesandcomputedtheaverageresults exceptforFigure5-10discussedbelow. Performance: Intherstsetofexperiments,wecomparedtheproposedmulti-objective GGAapproachwithsixcompetingalgorithmsincludingfourwell-knowofinebin-packing heuristicsandtwosingle-objectiveGGASGGAapproaches. FFD-CPUandFFD-MEM: First-t-decreasingFFDplacesitemsinadecreasing orderofsize,andateachstep,thenextitemisplacedtotherstavailablebin. FFD-CPUrepresentstheFFDsolutionsortedbyvirtual-machineCPUrequirements andFFD-MEMistheFFDsolutionsortedbymemoryrequirements. BFD-CPUandBFD-MEM: Best-t-decreasingBFDplacesavirtualmachineinthe fullestserverthatstillhasenoughcapacity.BFD-CPUandBFD-MEMrepresenttheBFD solutionssortedbyCPUrequirementsandmemoryrequirements,respectively. SGGA-PandSGGA-T: BothalgorithmsuseGGAtosearchthesolutionspaceand thetnessvalueisevaluatedwithrespecttopowerforSSGA-P,ortotemperaturefor SSGA-T. 114

PAGE 115

Figure5-9.Performancecomparisonsofsevenplacementalgorithms.Comparisonof proposedmulti-objectiveoptimizationwithsixothercompetingsolutionswith respecttototalresourcewastage,powerconsumption,maximaltemperature andfuzzyevaluationvalues. MGGA: DifferentfromSGGA,thetnessvalueformulti-objectiveGGAMGGAis evaluatedconsideringallthethreeobjectivesincludingminimizingresourcewastage, powerconsumptionandmaximumtemperature. Figure5-9comparesthetotalresourcewastage,powerconsumption,and maximumtemperatureaswellasthetnessvalueforeachofthealgorithmsunder consideration.Thekeyobservationsconcerningthisgureareasfollows: FFD,BFDandSGGA-Pyieldthehighesttemperaturebecausetheyalltendto consolidateVMsintoasmallernumberofservers,resultinginhigherresourceutilization andhighertemperatureoftheservers.Amongthem,SGGA-Pproducesthelowest powerconsumptionbecausetheimprovedGGAalgorithmisabletosearchthesolution spacemoreefcientlyandgloballysothatitcanndtheplacementsolutionswitha 115

PAGE 116

Figure5-10.Solutionpointsobtainedfromsevenplacementalgorithmsfortwenty randomlygenerated128-machine250-VMinputs. smallernumberofusedserverscomparedwithFFDandBFD.Theresourcewastage ofSGGA-Pisalsolowbecausetheplacementtriestofullyutilizetheresourcesin alldimensions.Onthecontrary,SGGA-Tyieldsthelowesttemperaturebecausethe algorithmtendstoevenlydistributeVMrequeststoalloftheavailableservers,therefore theresourceutilizationofeachserverislowaswellastheCPUtemperature.Atthe sametime,SSGA-Tgeneratesthehighestpowerconsumptionbecausenoservers canbeturnedofftosaveenergy.MGGAproducesrelativelylowvaluesforpower consumption,peaktemperature,andresourcewastagebecauseittakesallobjectives intoconsiderationandstrivestondsolutionsthatoptimizeeveryobjectiveandachieve goodbalanceamongconictinggoals.ThebestperformanceofMGGAcomparedto othercompetingalgorithmsisalsoconrmedbythehighesttnessvalueshownin Figure5-9. Figure5-10illustratesthesolutionpointsobtainedbysevenplacementalgorithms fortwentydifferent128-machine250-VMinputsusingatwo-dimensionalgraph,inwhich x-axisrepresentsthepowerconsumptionandy-axisisfortheCPUtemperatureThe 116

PAGE 117

Figure5-11.FitnessvalueofMGGAfordifferentvaluesofSandG. dimensionforresourcewastageisomittedtomakethegureclear..Eachpointinthe gurerepresentsthesolutionobtainedbyoneofthealgorithmsforeveryinput.The pointsobtainedbySGGAresideatthetwoendsofthegure.Theyeitherhavethe highestpeaktemperatureandlowestpowerconsumptionbySGGA-P,orlowestpeak temperatureandhighestpowerconsumptionbySGGA-T.ThepointsofMGGAare locatedinthemiddleandachievebetterbalancebetweenthetwoconictinggoals. ThepointsobtainedbyFFDandBFDhavethesamepeaktemperatureasSGGA-P buthigherpowerconsumptionthanSGGA-P,showingthattheyarenotParetooptimal becausethesolutionsofSGGA-Paredominanttotheirsineveryobjective. Robustness: Theinitialsolutionsize S andthenumberofgenerations G are twoofthefundamentalparametersfortheGGAalgorithm.Theintuitionisthatthe performanceofthealgorithmimproveswithlargervaluesof S and G becausethereare moreexistingcandidatesolutionstoexploreandmoregenerationsofsolutionsbeing produced.Theprevioussetofexperimentsusessmallvaluesforbothparameters, whilethissetofexperimentsexploresthesensitivityoftheseresultstothevarious parametervalues.Figure5-11showsthetnessvaluesfordifferentvaluesof S and 117

PAGE 118

Figure5-12.FitnessvalueofMGGAfordifferentvaluesofcrossoverrate. G .Whentheinitialsolutionsizeisverysmalllessthan5,theperformanceofGGA doesnotimprovemuchevenwithalargenumberofgenerationsbecausetherearevery fewsolutionpointsavailableforGGAtoevolvewith.Whenthevalueof S exceeds12, themarginalbenetofincreasinginitialsolutionsizeisrapidlydecreasing,indicating thatGGAhasenoughpointstoproducebettersolutions.Anotherobservationisthatin mostcases,theperformancestopsimprovingafter G goesbeyondabout8,showing thattheproposedranking-crossoverGGAalgorithmcanquicklyimprovethesolutions andreachtheoptimalorsub-optimalpoints.Theseobservationsvalidaterobustness oftheproposedGGAapproachinthesensethattheperformanceobtainedbythe smallrepresentativesetparametervaluesisveryclosetolargerparametervalues. AsmentionedinSection5.2.3,crossoverandmutationrateareanothertwoimportant parametersoftheGGAalgorithm.Asetofexperimentswasconductedtoinvestigate theperformancewithrespecttothesetwoparameters.Figure5-12plotsthetness valuesofGGAwithvaryingvaluesofcrossoverrate.Theperformancestopsimproving afterthecrossoverrategoesbeyond0.8.Theexperimentstheresultsarenotshown 118

PAGE 119

Figure5-13.FitnessvalueofMGGAfordifferentvaluesofSandG. Figure5-14.FitnessvalueofMGGAfordifferentvaluesofcrossoverrate. inthepaperduetothespacelimitationalsoshowthatthemutationoperatordoesnot helpimprovetheperformancebecausetherandomdeletionandre-insertioncannot steertheexistingsolutionsbetter.Therefore,therateofcrossoverandmutationareset to0.8and0forallotherexperiments. 119

PAGE 120

Scalability: Thelastsetofexperimentsisusedtostudywhethertheproposed GGAalgorithmisscalableforlargesizeofdatacentersandVMrequests.Inthe experiments,thenumberofphysicalmachines M isvariedfrom50to1000andthe numberofVMrequests N from100to2000.Theexecutiontimeismeasuredon a2.00GHzPentiumMmachine.Figure5-13plotsthetimeofgeneratingtheinitial populationofsolutionsandsuccessivegenerationswithincreasingproblemsizes. Thealgorithmtakeslessthan3minutestosolvethedifcult1000-machine,2000-VM placementproblem.Thetimetogeneratenewsolutionsfora128-machine250-VM problemfordifferent S and G isshowninFigure5-14.Itisclearthattheexecutiontime isapproximatelylinearwithrespecttothevaluesof G and S AnalyzingthecomplexityoftheGGAalgorithm,itconsistsoftwomainparts.For theinitialsolutiongeneration,thealgorithmperformsrst-tonarandompermutation ofVMrequests.Thecomplexityis O SNlogN forgeneratinganumberof S solutions. Inthesuccessivesolutiongenerations,themostcostlyfunctionistheplacement evaluation.ThealgorithmevaluatesalltheVMsplacedoneachphysicalserversin everycandidatesolution,thereforethecomplexityis O NSG foranumberof N virtual machines, S solutionsand G generations.Combiningthesetwoparts,thecomplexityof GGAalgorithmis O SNlogN + O NSG ,whichyieldsapolynomialexecutiontime. 5.4.4EvaluationofDynamicVMMigration 5.4.4.1Prototypeimplementation TheBladecenterrunsVMwareServerasthevirtualizationplatform.TheVMsare conguredtohaveoneCPUand2GBdisksize.Theirmemorysizesarerandomly selectedfromtherangeof256Mto1024M,forgeneratingdifferentmigrationoverhead. AllthemonitoringsensorsareimplementedinPerlanddeployedasdaemonsrunning oneachhost.TheresourceutilizationincludingCPU,diskandnetworkIOinformation iscollectedfrom/procand/syssystemlesprovidedbyLinux;CPUtemperatureis monitoredusingthelm sensorstool;andpowerusageisobtainedthroughtheIBM 120

PAGE 121

AdvancedManagementModule.Thesensorsperiodicallyproducedataevery10 secondsinourimplementationandstorethemtoaprolingrepositoryontheshared storageoverNFS,whichcanbeaccessedbytheglobalcontrolleranytime. Thecontrolleralsorunsasadaemononthereservedbladenodeandallthe functionsdescribedinSectionIIIareimplementedwithPerlscripts.Periodically,itpulls outthemonitoringdatafromtheprolingrepository,checkswhethertheconditionsfor VMmigrationsorothersaremet,determinescontrolactionsandsendsthemtothe actuatorsimplementedonthehosts. 5.4.4.2Workloadgeneration Toemulateatypicaldatacentersetting,amixofdifferenttypesofworkloadsis generatedbydeployingandrunningmultiplebenchmarksonthevirtualmachines. SysBench isamodular,cross-platformandmulti-threadedbenchmarktoolforevaluating OSparametersthatareimportantforasystemrunningadatabaseunderintensive load.Ithassixtestmodes,allowingseveralsystemparameterstobevaried,including leI/Operformance,schedulerperformance,memoryallocationandtransferspeed, POSIXthreads,implementationperformance,anddatabaseserverperformanceOLTP benchmark.Threeofthetestmodesareusedintheexperiments,namelyCPU modetogenerateintegercomputation,leI/OmodeforintensivediskIOactivity, andOLTPmodetorepresentdatabaseapplication. Lookbusy isanotherapplication forgeneratingdifferenttypesofsyntheticloadsonaLinuxsystem.InCPUmode,the parametersettingsallowthesystemtoeithermaintainaconstantCPUusageatthe levelspecied,orperiodicallyvarywithinarangebyfollowingacurvefunctioncosine function.TheCPUusagelevel,curverangeandperiodcanbespeciedbytheusers. ThisbenchmarkisusedtogeneratedynamicallychangingCPUworkloads. Linpack is abenchmarkforperformingnumericallinearalgebraandcanbeusedtomeasureofa system'soatingpointcomputingpower.ItrepresentsanHPCapplicationinthedata centertestbed. 121

PAGE 122

AWorkloadgeneratorimplementedin Perl isusedtoinjectdifferentworkloads intoeachofthevirtualmachineshostedonthetestbed.ForeveryVM,thegenerator randomlyselectsabenchmarkandselectsitsparametersrandomlyfromtheirallowed ranges.Therefore,eachtestbednodeisassignedamixofvaryingworkloadswith differentintensities. 5.4.4.3Competingapproaches Toevaluatetheproposedapproachandcomparewithsomeexistingvirtualization managementapproach,anothertwocontrollersarealsoimplemented.Therstone ignoresstabilityissuesandexposesthebenetsoftheproposedapproachusing operationwindowsandpredictiontoavoidcontrolactionsthatleadtounnecessary migrationandunstableconguration.ThisrstcontrollertriggersaVMmigration wheneverthemonitoredtemperature,orresourceutilizationofahostexceedsacertain threshold.Thehostselectioninthecontrollerusesthemostrecentmonitoreddata collectedfromthesensorsinsteadofusingadatahistorytoincorporateitsvariation andtrend.Thesecondcontrollerusessingle-objectiveoptimizationfordestination hostselectionandexposesthebenetsoftheproposedapproachduetoitsuseof multi-objectiveoptimizationover.Thiscontrollerhasthreeselectionpolicies,1 coolest : selectthehosthavingthelowesttemperatureasthedestinationforVMmigration.2 idlest :selectthehostwiththelowestCPUloadasthedestination.3 fullest :selectthe hostwiththehighestCPUloadwhilestillhavingenoughCPUcapacityforthenewVM. TotesttheresponsetothethreedatacenterconditionsaforementionedinSection 5.3,besidestherandomworkloadsonbladenodes,theworkloadgeneratorintentionally createdaseriesofeventscorrespondingtohotspots,resourcecontention,andlow energyefciencyconditionsonrandomlychosenserversbyaddingorremoving certaintypesofworkloadsontheVMshostedontheservers,overa2-hourrun. Eacheventcanbeeithertransientbyrunningtheworkloadsforlessthan1minuteor stabletheworkloadslastatleast10minutes.Allthefollowingevaluationresultsare 122

PAGE 123

obtainedthoughaseriesofexperimentruns,ineachtherearerandomworkloadsplusa sequenceofeventsgeneratedontheservers.Theproposedmulti-objectiveoptimization withstabilizationconsiderationinVManddestinationhostselectionMOSapproach iscomparedwithamulti-objectiveoptimizationwithoutstabilizationconsideration MONS,single-objectiveoptimizationSOwithcoolestSOc,fullestSOf,andidlest SOipolicy,andnocontrolpolicyforeachexperimentrunwiththesamegenerated workloads.Sinceapplicationdowntimeisnotaconcerninourexperiments,wechose ofinemigrationalsocalledsuspend-copy-resumemigrationbecauseitisquickerand consumeslessresourcethanlivemigration.Itiseasytoincorporatelivemigrationinour testbedifnecessary. 5.4.4.4Evaluationresultsandanalysis Inthissection,theoverallbenetsofproposedapproachcomparedtono-control scenarioaresummarizedrst,followedbythemoredetailedresultsandanalysisof howthesystembenetsindividuallyseparatelyfrommulti-objectiveoptimizationand stabilization. Figure5-15showsthereal-timemonitoringdataCPUutilization,diskutilization, andCPUtemperatureforallthebladenodesduringanexperimentrun,which experiencedfourevents,atransientCPUcontention,astableIOcontention,atransient IOcontention,andastableCPUcontentionalsoahotspotevent.Thedataisupdated every10secondsFromFig.4a,itisseenthatwithoutanycontrolactions,server1 experiencedveryhighdiskIOusagearound 90% 100% fromabouttimeperiod60 tilltheendoftheexperiment.Server0begantohaveastablehighCPUloadstartingat timeperiod120,whichalsoresultedinCPUtemperatureviolationstartingfromaround timeperiod150.AlthoughServer7alsoexperiencedhighIOusage,itiscausedbyonly oneVMandisnotidentiedasanIOcontentioneventbythecontroller.Fig.4bshows thatthecontrollerreactedtothestableIOandCPUcontentionalsohotspoteventsby migratingselectedVMstothebestdestinationhostscanbefoundusingmulti-objective 123

PAGE 124

Figure5-15.MonitoringdataCPUutilization,diskutilization,andCPUtemperature fromthebladenodesduringanexperimentrun,whichexperiencedfour events,atransientCPUcontention,astableIOcontention,atransientIO contention,andastableCPUcontentionalsocreateahotspotevent. optimizationapproach.TheIOusageonserver1andCPUusageonserver2dropped backtosaferangearound 50 60% afterVMmigrations.ThetemporalhighIOusage aroundtimeperiod70onserver5,140onserver6and150onserver5arecaused byVMmigration.Thetemperatureonserver0wasalsokeptinthesaferange. Figure5-16showsthetimeperiodofthermalandresourceusageviolation experiencedbyallthebladenodesduringtheexperimentrunwithandwithout control.TheresultsarenormalizedtotheMOSapproach.AsseeninFigure5-16, thetemperatureviolationperiodisreducedby80%andresourceusageviolationperiod isreducedbyabout70%bydynamicVMmigrationusingMOSapproach.Figure5-17 showstheperformanceofaVMwhichwasrunningSysbencholtpmodeandhostedon 124

PAGE 125

Figure5-16.Comparisonoftotalthermalandresourceusageviolationperiodbetween MOSapproachandnocontrol. Figure5-17.ComparisonoftheperformanceofaVMrunningSysbencholtpmodeon server1undernocontrol,controlusingMOSandidealcase. server1whichexperiencedIOcontention.Theidealperformanceisobtainedbyrunning theVMexclusivelyonacompleteidleservernode.Itisseenthattheresponsetime increasesdramaticallyduetotheIOcontention.Thecontrollerdetectedthissituation andmigratedtheVMtoanotherhostusingMOS.Aftermigration,theperformanceofthe VMimprovedquicklyandgraduallygotclosetotheidealperformance.TheotherVMs runningSysbenchonserver1havethesimilarperformance,whicharenotshowninthe paper.Table5-5liststhetimingandresourceoverheadfordynamicVMmigration.The 125

PAGE 126

Figure5-18.Comparisonofproposedmulti-objectiveoptimizationwithstabilization considerationwithfourothercompetingsolutionsincludingmulti-objective optimizationwithoutstablizationconsideration,single-objectivcoptimization usingcoolest,idlest,andfullesthostselectionpolicy. VManddestinationhostselectionalgorithmexecutedbythecontrollertakeslessthan1 second,andconsumedverylittleresourceslessthan1%CPU.TheVMsuspensionon sourcehostandVMstartondestinationhosttakeafewsecondsandutilizeabout20% CPU.ThemainoverheadforaVMmigrationtakesplaceatcopyingVMles.Ittakes about20secondstocopyaVMwith256Mmemoryand40secondsforaVMwith512M memory.Duringthelecopying,bothsourcehostanddestinationhostexperiencevery highdiskandnetworkactivity. Figure5-18comparestheperformanceoftheproposedcontrollerusingmulti-objective optimizationwithstabilizationconsiderationMOSapproachwiththecontrollerwithout stabilizationconsiderationMONSandthecontrolleroptimizingsingle-objectivein hostselectionincludingcoolest,fullest,andidlestundervemetricsincludingthe totalnumberofperformedVMmigrations,thetotaltimeusedforVMmigrations,the totalconsumedpower,thetotalthermalviolationperiod,andtotalresourceusage violationperiodduringtheexperiment.AlltheresultsarenormalizedtotheMOS 126

PAGE 127

approach.AsseeninFigure5-18,MOStriggeredandperformedthefewestnumber ofVMmigrationsandconsumedtheleastmigrationtime.SincehighCPU,diskand networkoverheadareincurredduringVMmigrations,lowernumberofVMmigrations andlessmigrationtimesindicateslessresourcesarewastedforVMmovements.The totalthermalviolationperiodrecordsthetimethattheCPUtemperatureofanyblade nodeexceedsthesafeoperatingrange.Alltheapproachesexceptfullest selection policyachieveverylowthermalviolationperiod.Thereasonisthatfullest selection policytriestoconcentratetheworkloadsonfewerservers,resultinginhighCPUload oncertainhostsaswellastheirCPUtemperature.Thetotalresourceusageviolation periodisthetimethatresourceusageofanybladenodeisbeyondtheallowable resourceusageforguaranteeingapplicationperformance.Fromtheresults,MOS andMONSapproachesperformthebestunderthismetric.Theidlestandcoolest selectionpolicieshaverelativelyhigherviolationperiodandthemainreasonforthis isthatbothapproachestrytooptimizeasingleobjectiveanddoesnotconsiderthe effectsofotherfactors.Forexample,aserverthathoststheVMswithhighIOactivity mayhaveverylowCPUutilizationandCPUtemperatureaswell.Withcoolestoridlest selectionpolicy,acontrollermaymigrateaVMwithhighIOworkloadstosuchaserver, resultinginIOusageviolationandcausingfurtherVMmigrationsduetoIOcontention. Thefullest selectionpolicygivestheworstperformancebecausemovingVMsandtheir workloadstoahostalreadyhavinghighCPUloadincreasesthechanceofthatserver violatingCPUusageandincursmoremigrations. 5.5RelatedWork Theprobleminvestigatedinthisdissertation-mappingofVMstophysicalservers -isrelatedtoavarietyofresearchtopicsincludingworkloadplacementonshared resources,dynamicresourceallocationandtheclassicalbin-packingproblem. Theclassicalbin-packingproblemistodeterminehowtoputtheitemsinthe leastnumberofxed-spacebins.ThisNP-hardproblemhasbeenextensivelystudied 127

PAGE 128

see[119]forarecentsurvey.Onerelatedproblem,theapplicationplacementis theoreticallystudiedin[120]inwhichthegoalistomaximizethetotalnumberof applicationsthatcanbehostedonasharedhostingplatform.Theauthorsprove thatofine-APPisNP-hardbyreducingfromMultidimensionalKnapsackproblem. First-t,best-tandworst-tapproximationalgorithmsthatplaceapplicationsin nondecreasingorderoftheirrequirementsarediscussed.Cardosaetal.[121]has investigatedtheproblemofpower-efcientVMallocationinvirtualizedenterprise computingenvironments.Theyleveragemin,maxandsharesparameters,whichare supportedbythemostmodernVMmanagers.Theobjectivefunctiontobeoptimized includesthepowerconsumptionandutilitygainedfromexecutionofaVM,whichis assumedtobeknownapriori.Theauthorsprovideseveralheuristicsforthedened modelandexperimentalresults.Vermaetal.[122]studiedCPUusagecorrelation betweenapplicationsoverlongtermtodetermineVMplacementonashareddata centerinordertosavepoweraswellasreduceperformanceviolationsduetoVM consolidation. Morerecently,thermalandenergymanagementhavereceivedmuchattention, especiallyinlarge-scaledatacenterenvironments.Someresearchinvestigatesthe placingofapplicationsonenergy/thermal-efcientlocations[123].Atemperature-aware workloadplacementispresentedin[112][118][9].Sharmaetal.[124]addressesthe similarproblemandproposestomeasurecoolingefciencyforguidingtheworkload placement.Thereisalsosomeworklatelyonconservingpowerusage.Resource allocationiscombinedwithenergymanagementbyturningoffserverswithnoload, orlowloadafterunloadingtheservers[125][126].Somework[6][75][109][127]also considersusingdynamicvoltage/frequencyscalingDVFStofurtherimproveenergy efciency. Usingdynamicvirtualmachinemigrationcapabilityonavirtualizedplatformto improveefciencyforresourceusageandpowerconsumptionhasdrawnaincreasing 128

PAGE 129

attentionrecently.Fromthecommercialside,VMware'sDistributedResourceScheduler DRSusesVMmigrationtodynamicallybalancecomputingcapacityacrossacollection ofphysicalresources.VMwareDRSonlyrespondstoCPUandmemoryoverload cases,anddoesnotconsiderotherplatformfactorssuchastemperatureandpower consumption.VMwareDistributedPowerManagementDPMutilizesVMmigration toconsolidateworkloadonfewerphysicalmachinesandpowerofftheunusedhosts tosaveenergy.Therearealsomanyworkspublishedinrecentresearchliterature. ManyofthemconsiderthedynamicVMmigrationasanoptimizationproblem,for example,withanobjectivebeingtomaximizetotalresourcedemandsorminimize powerconsumption[55][56][57][58].Someofthemalsoconsidermigrationcost andperformanceguarantees[128][55][59].Khannaetal.[129]proposedanonline algorithmforreconguringVMplacementwhichistriggeredbytheviolationofresource utilizationthresholds.Thegoalistomaximizethegainsassociatedwithmigration. Woodetal.[59]tacklesasimilarproblem,inwhichmigrationisinitiatedtoavoid violationofapplicationSLAsorresourceutilizationthresholds.Themigrationalgorithm movesworkloadsfromthemostoverloadedserverstotheleast-loadedones,while minimizingdatatransferredduringmigration.Bobroffetal.[128]describesadynamic remappingVMtoPMalgorithm,inordertostatisticallysatisfySLAtargetsunder dynamicworkloads.pMapper[57]addressesthepowerandmigration-cost-aware applicationplacementprobleminheterogeneousserverclusters.Heuristicalgorithms adaptedfromrst-t-decreasingareproposedtoproducelocaloptimalsolutions. Entropy[58]makesuseoftheConstraintProgrammingCPparadigminordertond theoptimalplacementforVMsandthentriestoconstructaVMrecongurationplanwith theleastmigrationcost.Similarly,Mistral[55]addressesthedynamicVMplacement byrstndingtheminimumnumberofnodestohostVMsusingamodiedbin-packing algorithmandthenconstructingasearchgraphfromthecurrentcongurationwith best-rstsearchtechnique. 129

PAGE 130

Table5-6liststherecentworkondynamicVMmigrationandcomparesthemin relationtomigrationobjectives,migrationtriggerwhentomigrate,VManddestination selectionwhichandwheretomove.ThetriggerfordynamicVMmigrationinthe above-mentionedworkonlydependsonthestatesofVMs,theperformanceof theirhostedapplications,ortheresourceusageoftheirhosts,withoutconsidering informationsuchaspowerusageandtemperaturedistribution.Themigrationobjectives onlyconsideroneortwoaspectsofdatacenterssuchaspowerconsumptionand migrationcost.TheapproachesofndingwhichVMstobemovedandwheretomove themareeitherbasedonglobalsearchorlocalsearch.Althoughglobalsearchmay producebetterresults,itbecomesimpracticalforalargedatacenterduetothelarge searchspace.Inaddition,theuseofglobalsearchwithoutconsideringexistingVM placementcausesahighnumberofmigrationsandhighwasteofresources.Utilizing theinformationfromboththevirtualizationlayerandtheplatformlayer,weidentied threeconditionsfordynamicVMmigration,whicharethermalemergency,resource contentionandlowpowerefciency.TheselectionofVMsformigrationinourwork considersboththeefcienciesofimprovingsystemconditionsandmigrationcostunder differentconditions.Wealsoproposedtouseamulti-objectiveoptimizationapproach fordestinationselection.Inaddition,arobustdetectionandselectionapproachusinga sliding-windowandtrendanalysisisincorporatedintothedecision-makingprocessof thecontroller.Thisapproachleadstostablesystemstatesandpreventsthewasteof resourcesandtimeforunnecessarycontrolactions. 5.6Conclusions Enhancedvirtualizationtechnologyenablestheabilityofworkloadconsolidation andmigration,aswellasne-grainedcontrolofresourcesassignedtoasinglevirtual machine.Atthesametime,itbringstheneedformoreintelligentVMmanagementthat canadapttochangingworkloaddemandsandsatisfyvariousmanagementgoalsas well. 130

PAGE 131

Inthiswork,theproblemofVMplacementandmigrationisformulatedasa multi-objectivecombinatorialoptimizationproblemaimingtosimultaneouslyoptimize possiblyconictingobjectivesincludingmakingefcientusageofmultidimensional resources,avoidinghotspots,andreducingenergyconsumption.ForinitialVM placementinwhichanumberofvirtualmachinesareplacedatonceinanunloaded datacenter,modiedgeneticalgorithmisproposedanddevelopedtoeffectivelydeal withthepotentiallargesolutionspaceforlarge-scaledatacenters.Fuzzymulti-objective evaluationisappliedinthealgorithmtocombinetheconictinggoals.Wealso investigateddynamicVMplacementapproachestocopewithworkloadsthatover timechangetheirresourcerequirements.Insuchcases,theinitialmappingneedsto bemodiedforbetterallocationofexistingandnewvirtualmachines.However,the highcostincurredbyVMmigrationprohibitsunlimitedusageofthismechanism.The controllerneedstominimizetheimpactofmigrationaswellassatisfyotherobjectives andconstraints,whenmodifyinganexistingplacementandallocatingnewvirtual machines.Across-layercontrolsystemisproposedtomanagethedynamicmapping ofVMstophysicalresources.Thecontrolleruniestheinformationfromdifferentlayers todeterminecontrolactionssuchaswhen,whichandwhereVMsneedtobemoved andwhentoturnon/offphysicalhosts.Threeconditionsincludingthermalemergency, resourcecontentionandlowenergyefciencyareidentiedtoneeddynamicVM migration.DifferentstrategiesforselectionofwhichVMsshouldbemovedareused underdifferentconditions,toimprovetheefciencyofthemovementandreduce migrationoverhead.TheselectionofdestinationhostformigratedVMsispostedasa multi-objectiveoptimizationproblemandthreeutilityfunctions,eachrepresentingan optimizationobjective,areusedtocombinethemintosingleobjective.Stabilization issuesforconditiondetectionanddestinationhostselectionarealsoconsideredinthe designofthecontroller. 131

PAGE 132

Theprolingdataobtainedfrommeasurementsofpowerconsumptionand CPUtemperatureinanIBMBladeCenterareusedforbuildingthemodelsofpower consumptionandCPUtemperatureofbladeservers,whicharethenappliedinthe proposedplacementandmigrationalgorithm.FortestinginitialVMplacementovera widerangeofdatacentersettingandVMrequests,thesimulation-basedexperiments studiedtheproposedapproachwithrespecttoitsperformance,scalabilityand robustnessandshowedthesuperiorperformanceoftheproposedapproachcompared withwell-knownbin-packingalgorithmsandsingle-objectiveapproaches.Fordynamic VMplacement,anexperimentaltestbedisimplementedonanIBMBladeCenter.Amix ofdifferenttypesofworkloadsisgeneratedtoemulateatypicaldatacentersetting. Theperformanceoftheproposedcontrolleriscomparedwithanothertwocontroller, onewithoutstabilizationconsiderationandtheotherusingsingle-objectiveapproach. TheresultsshowthattheproposedapproachsignicantlyreducesunnecessaryVM migrationanditsassociatedresourceoverhead,avoidsunstablehostselection,andalso improvetheapplicationperformance,andefcienciesofresourceandpowerusageof datacenters. 132

PAGE 133

Table5-2.Thelistofutilityfunctions. NameUtilityfunctionsSymbols Temperature efciency Eff i T =1 )]TJ/F44 10.9091 Tf 10.909 0 Td [( T i )]TJ/F45 10.9091 Tf 10.909 0 Td [(T low T high )]TJ/F45 10.9091 Tf 10.909 0 Td [(T low m T i :Thetemperatureofserver i T low :Thetemperatureofanidleserver i C T high :Thetemperatureofanoverloadedserver i C m :degreeofutilityfunctionsetto3intheimplementation Performance efciency Eff i C = min Eff i CPU ;Eff i IO ;Eff i Net CPU i :TheCPUofserver i CPU low :TheCPUusageofanidleserver% CPU high :TheCPUusageofanoverloadedserver% Eff i CPU =1 )]TJ/F44 10.9091 Tf 10.909 0 Td [( CPU i )]TJ/F45 10.9091 Tf 10.909 0 Td [(CPU low CPU high )]TJ/F45 10.9091 Tf 10.91 0 Td [(CPU low m IO i :Thediskutilizationofserver i IO low :TheIOusageofanidleserver% CPU high :TheIOusageofanIOoverloadedserver% Eff i IO =1 )]TJ/F44 10.9091 Tf 10.909 0 Td [( IO i )]TJ/F45 10.9091 Tf 10.909 0 Td [(IO low IO high )]TJ/F45 10.9091 Tf 10.909 0 Td [(IO low m IO i :ThenetworkIOutilizationofserver i Net low :ThelowestnetworkIOusage Net high :ThehighestnetworkIOusageMbytes/sec Eff i Net =1 )]TJ/F44 10.9091 Tf 10.909 0 Td [( Net i )]TJ/F45 10.9091 Tf 10.909 0 Td [(Net low Net high )]TJ/F45 10.9091 Tf 10.909 0 Td [(Net low m m :degreeofutilityfunctionsetto3intheimplementation Power efciency Eff i P = CPU i p 1 + p 2 CPU i p 1 + p 2 CPU i :TheCPUusageofserver i p 1 andP 2 :constantsforlinearpowerfunction m :degreeofutilityfunctionsetto3intheimplementation multi-objective utility Eff i = Eff i T + Eff i C + Eff i P 133

PAGE 134

Table5-3.Thresholdsandwindowsizesforconditiondetection. Level-1Threshold TH level )]TJ/F7 6.9738 Tf 6.227 0 Td [(1 Level-2Threshold TH level )]TJ/F7 6.9738 Tf 6.227 0 Td [(2 ThermalEmergency46 C50 C ResourceContention CPU:85%CPU:90% IO:85%IO:90% Network:85%Network:90% LowEnergyEfciencyCPU:10%CPU:10% Percentage P 80% Windowsizeforthermal&resourcecontention120seconds Windowsizeforlowenergyefciency360seconds Table5-4.Parametersetupforthreesetsofexperiments. ExperimentsetProblemsizesGGAParameters Iperformance M =128 ;N =250 S =12 ;G =8 IIrobustness M =128 ;N =250 S =[2 100] ;G =[5 20] IIIscalability M =[50 1000] ;N =[100 2000] S =[5 20] ;G =[5 20] Table5-5.ThetimingandresourceoverheadfordynamicVMmigration. EventsTimesecondCPUoverheadDiskoverhead%NetworkoverheadMb VMandhostselection 1 1 00 VMsuspension 3 610 20 00 VMlecopy 20 30 MSource: 20 30 Source: 0 Source: 110 120 40 50 MDestination: 20 30 Destination: 60 100 Destination: 120 150 VMstart 1 210 25 00 134

PAGE 135

Table5-6.RelatedWorkonDynamicVirtualMachinePlacement. Paper MigrationObjectives MigrationTrigger VMSelectionandDestinationSelection Entropy [58] Minimizethenumberofactivenodes whilemaintaining performance. VM-state-basedifthe stateofaVMchanges :active,inactive PhaseI:ndthemin.numberofnodestohost alltheVMsusingdynamicprogramming.Phase II:constructamigrationplantoachievetheminimumnumberofhostswiththeleastmigration cost. pMapper [57] Minimizethepower andmigrationcosts. Server-ThresholdbasedifCPUutilizationofaserveris higherthanathreshold Localsearch1.ChoosethesmallestsizedVM fromtheserversthattriggerVMmigration2. SorttheVMstobemigratedfromallserversin ascendingorderoftheirsizeCPUutilization 3.Selectthemostenergy-efcientserveras destinationhost. PADD [56] Minimizeenergy consumptionwhile satisfyingSLAs Server-ThresholdbasediffreeCPU capacityofaserveris lowerthanathreshold Localsearch1.VMselection:maximumdemand, averagedemand,andminimumstandarddeviationofdemands2.Destinationselection:not specied Dynamic Placement [128] Providestatistical SLAguarantee whileminimizingthe numberofhosts Invokereplacement periodically Previousplacementisnottakingintoaccount1. AllVMsaresortedindescendingorderofpredicteddemandsARmode2.First-tpacking heuristictoplaceVMstoservers Mistral [55] Optimizetotalutility includingapplicationutility,power costs,andtransient adaptioncosts VM-Threshold-based WhenaVMworkloaddeviatesfroma speciedworkload band GlobalsearchinPhaseIpreviousplacement isnottakingintoaccountPhaseI:ndthemin. numberofnodestohostalltheVMsusingamodiedbin-packingalgorithm.PhaseII:constructa searchgraphfromcurrentcongurationanduse abest-rstsearchtoreducesearchspace vManage [130] Guaranteepower budgetwhenmigratingVMs VM-Threshold-based WhenaVMSLAis violated LocalSearch1.ChoosetheVMsthathaveSLA violations2.Selectahostthatsatisfyboththe VM'sdemandandpowerbudgetforaperiodof time Blackboxand Graybox [59] Eliminatehotspots resourcecontentioninorder tomaintainVM performance Black-box:ServerThreshold-basedif CPU/networkutilizationexceeda thresholdGrey-box: VM-Threshold-based ifapp.SLAisviolated Localsearch1.Sortserversindecreasingorder oftheirload.Withineachserver,sorttheVMs indecreasingorderoftheirload/sizememory size2.ChoosetheVMsinsortedorderfromthe mostloadedserver,andtrytomoveittotheleast loadedserver. 135

PAGE 136

CHAPTER6 COOPERATIVEAUTONOMICMANAGEMENTINDYNAMICDISTRIBUTEDSYSTEMS Theworkdescribedinlastthreechaptersappliedcentralizedmanagementatglobal controllevel.However,thecentralizedglobalmanagerintroducesasinglepointof failureandcanbecomeabottleneckinhandlingallinformationandmanagementtasks inlarge-scalesystems.Thischapterdiscussesourpreliminaryworkondecentralized managementinwhichanetworkofcooperativeautonomicmanagers,eachmanaginga subsetofresources,collaboratetomanagetheentiresystem. 6.1ProblemDescription Withtherapidgrowthofcomputingsystems,itbecomesimpracticaltousea centralizedcontrollertobuildalarge-scaleself-manageablesystem.First,theresource overheadandtimedelayforcollectingglobalsysteminformationintoacentralized locationincreasesdramaticallywiththesizeofthemanagedsystem.Second,The informationandalgorithmsneededtodecidewhatactionstotakearetoocomplexand applicationdependenttobehandledbyacentralizedcontroller.Extensiveresearch [131][132]hasfocusedonprovidingautonomiccapabilitiestoindividualsystem components,suchasdatabases,applicationserversandmiddlewarecomponents. Ingeneral,theseautonomiccomponentsuseanapplication-levelmanagerthatis capableofmonitoringand/orpredictingperformanceandmanagingresourcesas neededtodeliverreliableapplicationswiththeexpectedQualityofServiceQoS.One canenvisiontheuseoftheseorsimilarcomponentsandtheirautonomiccapabilitiesas thebasicbuildingblocksoflargedistributedsystems. Threequestionsariseinthiscontext.First,whatinteractionsshouldtakeplace amongindividualcomponents,inordertoachievesystem-levelself-management? Implicitinthisquestionistheneedforinformationsharingamongdifferentautonomic components.Second,whattypeofnetworkshouldbeusedtosupporttheinteractions? Implicitinthisquestionistheneedforthenetworktobehighlyscalableandrobust 136

PAGE 137

tofailures.Third,howshouldautonomicmanagersbedesignedtointeractwithother components,andenhancetheirabilitytoautonomicallyusethemanyresources availableindistributedsystems?Implicitinthisquestionistheneedforcooperation amongmanagerstoefcientlycollectandshareinformationaboutresources.The answerstothesethreequestionscanbeinformedbyresultsfromnetworkscience. Thisworkproposesanetwork-scienceinspiredapproachfordistributed-system self-managementarisingfrominteractionsamongtheautonomiccomponentsdeployed inthesystem.Thekeyfeaturesoftheproposeddesignofautonomicdistributed systemsaretheeffectiveuseofautonomiccomponents'limitedmonitoringand communicatingcapabilities,andtheirinteractiveadaptationtothesurrounding environmentonthebasisofinformationprovidedthroughanoverlaynetwork.The proposeddistributedautonomicsystemmodelhasthefollowingproperties: Self-adaptation:Thesystemcandynamicallyrespondtoachangingenvironmentto provideindividualapplicationmanagerswithinformationandresourcesneededfor achievingthedesiredQoS. Self-organization:Thedesigneddecentralizedcoordinationmechanismenables thesystemtoadapttodynamicchangeswithoutexternalintervention.Theglobal optimizationisachievedthroughlocalautonomousdecisionsandinteractions amonglocalmanagersbasedonlocalinformation. Robustness:Therearenocentralresourcesthatcouldbecomesinglepointsof failureorperformancebottlenecks.Recongurationmechanismsformonitoring resourcesandbuildingneighboringmanagerseffectivelydealwithdynamic resourceavailability. 6.2DecentralizedAutonomicManagementArchitecture 6.2.1GenericAutonomicElementModel Weconsiderahighlydynamicdistributedcomputingsystemconsistingofalarge collectionofautonomiccomponentscalledAutonomicElementswhichcanjoinand leavethesystematanytime.EachAutonomicElementAEconsistsofoneormore managedelementse.g.jobsandresourcesandanautonomicmanagerAM.The behaviorsofthecomponentsareindependentlymanagedbytheirautonomicmanagers 137

PAGE 138

Figure6-1.AdecentralizedautonomicsystemconsistingofAutonomicManagersAMs acrosstwodomains,eachwitharegistryindexingresourcesinthedomain. EachAMcontactsitsdomainregistrytochooseboththeresourcesto monitorcalledlocalresourcesandotherAMscalledneighborsto exchangelocalinformation. AMsbasedonmonitoredinformation.ForAMstomakeoptimaldecisionstowards desiredstates,theyrequireglobalknowledgeofthechangingsystemenvironment. However,inlargedistributedsystemsitisnotscalabletocollectandprovideglobal knowledgethroughacentrallocation. Tosolvethisproblem,individualAMsareextendedtomonitornotonlytheir managedelementsbutalsoasmallpieceoftheirsurroundingenvironmenthereon calledlocalresources.AsexplainedinSection6.2.3,localknowledge,onceshared amongAMs,providestheneededglobalknowledgetoeachAM.TheAMarchitecture Figure6-1consistsofseveralcomponentsandalocalknowledgebasewherethedata sharedbythemarestored.Thecomponentsarethefollowing: Monitor: itcollects,aggregatesandltersthestatusinformationfromitsmanaged elementsanditslocalresources. 138

PAGE 139

Controller: itmanagestheelement'behaviorbasedonanalysisandprediction usingthelocalknowledge. Communicator: itsupportsinformationexchangeswithotherautonomicmanagers. AsFigure6-1shows,eachAMonlyhasalocalviewofthewholeenvironment. However,interactionamongthecooperativemanagersprovideseveryAMwithaglobal viewofthesystem,asexplainednext. 6.2.2DistributedDomainRegistry Thecomputingresourcesofthesystemareorganizedintodomainswhichmay correspondtoadministrativedomainsorcouldconceivablybesmallerinsizee.g.the computersoftheACISLab.Adistributeddomainregistryinfrastructureisdesigned toprovidescalableandreliableneighborAMdiscoveryandresourcelocationservices forAMs.EachregistrymaintainsanindexofresourcesandalistofexistingAMsinits domain.Whenanautonomiccomponentjoinsthedomain,itsAMregistersitsuniqueID totheregistryandchoosessomeexistingAMstocommunicatewithandselectssome resourcesinthedomainasitslocalresources. Toimprovereliability,nearbydomainregistriesperiodicallyexchangeinformationso thatthelistsoflocalresourcesandAMsstoredineachregistryarereplicatedinsome places.Domainregistriessimplifytheneighborrelationshipmanagementinautonomic systems,andprovideusefuldomaininformation,suchasthetotalnumberofresources andAMspresentinthedomain. 6.2.3AutonomicManagerAMNetworkBuilding WhenanAMjoinsadomainitselects m existingAMsinthesamedomainas itspotentialneighbors.AMsinthesameneighborhoodcooperatewitheachother byexchanginginformation.Theneighborselectioncantakeplacerandomly,or preferentiallywhichmeansthatsomeAMsaremoreattractiveandhaveabetterchance togetneighbors.Whendepartingfromitsdomain,anAMunregistersitselfbydeleting itsIDfromthedomainregistryandsendingafarewellmessagetoalltheneighborsto 139

PAGE 140

terminatetherelationship.IncaseanAMneedsotherdomain'sinformation,itcanask itsdomainregistryfornearbyregistriestocontactAMsinotherdomains.Bybuildinga cross-domainneighborhood,AMscanquicklygetinformationfromotherdomains. Localresourceclaimanddisclaim: EachAMrandomlyselectsanumberof resourcesfromthedomainregistryandclaimsthemasitslocalresourcesbymarking thecorrespondingentriesintheresourcelistwithitsID.WhennewAMscomeinto thedomain,theytrytoselecttheunclaimedresourcesinordertomaximizethetotal numberofmonitoredresources.AnAMdisclaimsitsresourcesbyunmarkingthem intheregistrybeforeitsdeparture.ThenumberofresourcesanAMcanmonitoris boundedbyitscommunicationandcomputationcapability. Informationsharingandltering: Duringitslifecycle,eachAMbecomesa dynamicinformationsourcebyupdatingitslocalresources'time-changingstatus informationtothelocalknowledgebase.Thislocalinformationcanbepropagated throughmulti-AMcooperation,whereAMsperiodicallyexchangemessageswiththeir neighbors.EveryAMthatreceivesamessagefromaneighbormuststoreitandlater forwardittoitsotherneighbors. Twoapproachesareusedtogethertoreducethenumberofmessagestransmitted amongtheAMs.Oneistodeneanobsolescencerelation[133]betweenmessages:a message m 1 isrecognizedasobsoletebyanAMifithasmessage m 2 containingmore recentinformationthatsubsumes m 1 .Asaresult, m 1 isnolongerneededandcanbe safelydiscarded.Theotherwayistoevaluateeachmessage'svalue[134]indicating howusefulthemessageis,anddropthelow-valuemessagespreferentially.Inthecase ofdifferentAMshavingdistinctinterestsininformation,itisdesirabletopartitionthe AMsintodisjointgroupsandthendisseminateinformationwithingroups.Thiscanbe easilyachievedusingdomainregistriesbyaddingaeldintheregistrationlisttorecord eachAM'stypeofinterest,whichhelpsAMschoosetheirneighborsonlyfromthoseof thesametype. 140

PAGE 141

6.2.4AMNetworkDynamism TheAMneighborhoodsdenethetopologyofadynamicoverlaynetworkthat changesasAMscontinuouslyjoinandleavethesystem,inamannersimilartoa peer-to-peernetwork[135].TheAMsmustadapttheirbehaviorsandinteractionstothe changingstateg.Forexample,anAMleavingorcrashingmaycauseseriouseffects -claimedlocalresourcesmaybenolongermonitoredbyanyone,andsomeAMs maybecomeisolatedfromothers.Topreventandrepairthedamage,thefollowing mechanismsareproposed. AMdepartureandfailuredetection: IfanAMvoluntarilydecidestoleave,it informsalltheneighborsbysendingthemafarewellmessage.InthecaseofAMor networkfailure,sinceneighborsperiodicallycommunicatewitheachother,eachAM measurestheintervalbetweentwosuccessivemessagessentfromthesameneighbor andsetsatimeouttodetectthefailure. Dynamicresourceclaim: Byperiodicallycheckingthedomainregistry,AMs canobtainthedynamicdomaininformationsuchasthenumberofclaimedand unclaimedresourcesandthetotalnumberofAMscurrentlyinthesystem,andthen adjustthenumberofresourcesitshouldmonitortobalancethemonitoringloadover thenetwork.However,theinformationprovidedbydomainregistriesmightbeincorrect becauseofAM'sunpredictablefailure.Tosolvethisproblem,onceanAMdetectsits neighbor'sfailure,itinformsthedomainregistryandreclaimstheresourcesthatbecame unmonitoredbecauseofthefailure. Dynamicneighborhoodbuilding: WhenanAMisinformedofaneighbor's departureordetectsaneighbor'sfailure,itchoosesanotherAMasitsnewneighbor withprobabilitypsetto0.5asexplainedinSection6.3.3.Thissimplemechanism allowsAMstomaintainnetworkconnectivitybyestablishingnewneighborhoodoverthe network. 141

PAGE 142

6.3AnalyticalEvaluation 6.3.1NetworkModel Weusetheconceptualframeworkandnotationsfromcomplexrandomnetwork theory[136][137]tomodelthesystemandanalyzethestructuralorganization. ThedecentralizedautonomicsystemismodeledasanetworkinwhicheachAMis representedbyanode,andtwonodesarelinkediftheyareneighbors.Thedegree ofanoderepresentsthenumberofneighborsthenodehas.Weuselocalloadto indicatethenumberofresourcesclaimedbyanAM. Consideringasetofnodeswhichjoinandleavethedomaindynamicallyandhave distinctidentiers,weusethefollowingnotationswhere t denotesaninstantintime, n t :thetotalnumberofnodesattime t r t :thetotalnumberofresourcesattime t m i :theinitialnumberofneighborsthatthe i thnodeconnectstowhenjoiningthe network. k i t :thedegreeofthe i thnodeattime t o i t :thelocalloadofthe i thnodeattime t Thersttwoparametersdescribetheentirenetworkandcanbeobtaineddirectly fromthedomainregistry,whiletherestoftheparametersdescribethebehaviorof individualnodes. 6.3.2NodeJoiningandNeighborSelection Considerthecasewherethenetworkstartswithonenode,andthenateachstep, anewnodejoinsandconnectsto m existingnodes.Attime t thenetworkhasatotal of n t nodes n t m ,foralargesystem.Thefollowingequationscanbeeasily derivedsee[137]. Totalnumberoflinks: e t = n t m )]TJ/F23 11.9552 Tf 13.15 8.088 Td [(m 2 + m 2 n t m 142

PAGE 143

Averagedegree: k t = 2 e t n t 2 m Diametermaximalshortest-pathlengthbetweenanytwonodes: d t = ln n t ln k t ln n t ln2 m Equation6showsthattheshortest-pathlengthbetweenanytwonodesissmall evenforalargenetwork.Thissmallworldeffect[138]ensuresthatlocalinformation ofonenodecanbepropagatedtoanyothernodeveryquicklyeveninlargenetworks. Giventhetotalnumberofnodes,Equation6providesawaytoachieveanyexpected d byusing m asthetuningparameter. Differentneighborselectionpoliciesresultindifferentnetworkdegreedistributions. Therandomselectionresultsinexponentialdegreedistribution.Incontrast,the preferentiallinkingthelikelihoodofconnectingtoanodeisproportionaltothenode's degreeleadstoapower-lawdegreedistribution,alsoknownastheBarabasi-Albert modelorscale-freenetwork[136].Themajordifferencesbetweenthesetwonetworks aretheirrobustnessagainstrandomnetworkerrorsasdiscussedinthefollowing section. 6.3.3NodeLeavingandNeighborhoodRebuilding Sincenodesmayleavethenetworkatanytime,weneedtoexaminetherobustness ofthenetwork.Theeffectofrandomdamageonnetworkswassimulatedin[136] andtheresultsshowthatscale-freenetworksdisplayahighdegreeoftolerance againstrandomfailures.Forexponentialnetworks,Equation6indicatesthataverage degreedecreaseslinearlywithgrowing f thefractionofremovednodes,whichinturn increasesnetworkdiameterseeEquation6;thusitisincreasinglydifcultforthe remainingnodestocommunicatewitheachother. 143

PAGE 144

k 0 = k )]TJ/F23 11.9552 Tf 11.956 0 Td [(f k 0 = 2 e 0 n 0 2 kn 2 )]TJ/F15 11.9552 Tf 11.956 0 Td [( )]TJ/F23 11.9552 Tf 11.955 0 Td [(p kfn n )]TJ/F23 11.9552 Tf 11.955 0 Td [(f p =0 : 5 )374()222()374(! k Adynamicneighborhoodrebuildingmechanismisproposedtoavoidthisimpact. Whenanodeleavesthenetwork,afraction p ofitsneighborsestablishnewrelationships withothernodes.Equation6indicatesthatbychoosing p equalto0.5theaverage degreecanremainapproximatelyconstant,sodoesthenetworkdiameter. 6.3.4LocalLoadAdjustment Weuse o t = r t n t toexpresstheaverageratioofthenumberofresourcestothe networksizeattime t .Tobalancetheloadonallthenodes,whenanode i joinsthe network, o i t isinitializedasfollows: o i t = d o t e Themaximumnumberofresourceseachnodecanmonitorisboundedsoasto avoidoverloadingandalsoensurethattheresources'dynamicstatusinformationcan betimelycollected.Becausethevalueof o t maychangeasthenetworksizeand resourceavailabilityvary,eachnodeperiodicallycomparesitscurrentobservation degreewith o t andadjustsitaccordingly. 6.3.5CommunicationCost Eachnodeinthenetworksendsmessagestoitsneighborsatconstanttime interval.Withinformationltering,themessagesize s i canbeboundtoaxedvalue S Duringatimeunit,theglobalcommunicationcostofthenetworkis C = X k i t s i 2 e t S 2 m n t S 144

PAGE 145

whichgrowslinearlywiththenetworksize.Butfromtheperspectiveofasingle node,theaveragecommunicationcoststaysalmostconstant. 6.4CaseStudy 6.4.1Background Inordertovalidatetheproposedmodel,weusedIn-VIGO[3]middlewareto implementaDecentralizedAutonomicVirtualApplicationManagementDAVAM system.In-VIGOisagrid-computinginfrastructurethatusesvirtualizationtechnologies toprovidesecureapplicationexecutionenvironments.Inthiscontext,applicationsare themselvesvirtualizedandexecuteinvirtualmachines,hencetheDAVAMacronym. Figure6-2providesahigh-levelviewoftheroleoftheautonomicVirtualApplication ManagerAVAMinIn-VIGOdetailedinChapter3.Typically,auserinitiatesan applicationsessioninIn-VIGOtorunoneormoreinstancesofacomputationaltool ongridresources.Eachapplicationsessionismanagedbyamiddlewarecomponent, calledtheVirtualApplicationManagerVAM.FollowingthemodeldiscussedinSection 3.3,autonomicfeaturesincludingself-optimizationandself-healingareintegratedinto theAVAM.Itreliesonmonitoringofjobandresourceconditions,predictingviolationsof user-and/orsystem-expectedexecutiontimesandrestartingjobsinresourcescapable ofdeliveringacceptabletimes. Toachievedesiredperformance,eachAVAMrequiresglobalknowledgeofthe time-varyingresourcestatusinformation.However,thecentralizedapproachproposed in[139]usingaglobalcontrollertocollectandmaintaintheknowledgeofthewhole systemstatusdoesnotscalewellinlarge-scaledistributedsystems.Instead,aDAVAM systemisconstructedwithdistributed,cooperativeAVAMs,asdiscussednext. 6.4.2CooperativeAM Figure6-2showsthemajorfunctionsimplementedinanAVAMandtheirinformation ow.Thelocalknowledgebasestoresinformationsuchasdynamiclocalresources' status,applicationrun-timeperformance,thelistoftheneighborsandlocalresources 145

PAGE 146

Figure6-2.ThefunctionsandinformationowofacooperativeAutonomicVirtual ApplicationManager. claimedbytheAVAM.Acontrollerisdesignedtocontroltheinternalbehavioranda communicatortomanagetheinteractionswithotherAVAMs.Thefunctionsimplemented intheAVAMaresimilartothosedescribedinSection3.3.2.Iexplainthemhereforyour convenience. 6.4.3Controller Thecontrollerisresponsibleforcontrollingtheapplicationexecutiontoachieve reliableandoptimizedperformance.Thefollowingliststhefunctionsimplementedinthe controller: Predict: Tochoosetheappropriateresourcesforthetoolexecution,thecontroller needstoknowthespecicresourcerequirementsforeachgivenjob.Amemory-based learningalgorithmciteavam[81]isusedtopredictresourceutilizationinformation,such asCPUcycles,CPUutilizationandmemoryutilization.Thebasicideabehindthis algorithmisthattheresourcesconsumedbyaparticularjoboftendependontheinput parameterssuppliedtothetool.Therefore,thesimilarityoftwojobsisdenedbythe distancemetricoftwosetsofinputvaluesandresourceconsumptionispredictedbased onthetoolexecutionhistory. 146

PAGE 147

Select: Thecontrollerscansthroughthelistofresourcesinthelocalresourcetable andranksthembasedonthejob'sresourcerequirementsandtheresources'processing capacity.Thescorecalculatedasfollowsreectshowwellaresourcecanhostthejob. Theresourceswithzeroscorearenotconsideredsincetheycannotsatisfythedeadline speciedbytheuser.Iftheresourceisavirtualmachine,thepredictedruntimeisalso dividedbyitsphysicalhost'sload. predicted runtime i = 8 > < > : job cycles CPU speed i if job load 1 )]TJ/F23 11.9552 Tf 11.955 0 Td [(CPU load ; job cycles CPU speed i + CPU load otherwise : score i = 8 > < > : 1 )]TJ/F23 11.9552 Tf 13.151 8.088 Td [(predicted runtime i deadline i if predicted runtime i deadline i 0 otherwise Thecontrollercanchoosedifferentresourceselectionpoliciesfordifferent purposes.Inourcase,tooptimizetheperformanceofthejobsitmanages,thecontroller strivestoselecttheresourcewiththehighestscore.However,resourcecontentionmay happenifmultipleAVAMstrytosubmitjobstothesamebestresourcesimultaneously. Aso-called -randomruleisusedtodealwiththisproblem.Arandomlygeneratedsmall number distributedevenlyintherange[-0.1,0.1]isaddedtoeachresource'sscore, andthenSelectfunctionrankstheresourcelistwiththesemodiedscores.Bysetting asmallnumber ,the -randomruleisabletomitigateresourcecontentiontoacertain extent. Verify: Afteraresourceisselected,thisfunctionchecksthecurrentstatusofthe resourceandverieswhetheritsscoreisstillvalid.Ifnot,thecontrollerselectsthenext candidateresourceintherankedlistandrepeatsthisvericationprocess. 147

PAGE 148

Analyze: Afterajobissubmittedtothechosenresource,themonitorkeeps collectingthejob'srunningstatuse.g.,currentCPUtime,elapsedtime,andCPU utilizationconsumedbythejob,whichisusedbyAnalyzefunctiontoestimatethejob's progressandpredictthenishingtime.Thepredictedjobnishingtimeisgivenbythe followingformulasee[139]fordetails. finish time = elasped time + job cycles )]TJ/F23 11.9552 Tf 11.955 0 Td [(finished cycles CPU speed CPU load IfanyperformanceproblemisdetectedbytheAnalyzefunction,thecontrolleris responsiblefortakingappropriateactions.Forexample,ifitispredictedthatthejob cannotnishbeforethedeadline,thecontrollerwilltrytondabetterresourcethatcan satisfythejobrequirementsandreschedulesthejobtothatresource. Inthecasewhenalltheresourcesinonedomainareheavilyloadedandcannot satisfyjobrequirements,thecontrollertakesthefollowingstepstohandlethissituation: itcontactsnearbydomainregistriestoselectseveralAVAMsinotherdomainsasits cross-domainneighbors;anditcommunicateswiththeseneighborssoastoquickly gettheresourceinformationinotherdomainsanddetermineonwhichresourceit cansubmitthejob.Althoughinter-domaincommunicationmaybeslower,onlyafew interactionsarerequiredinthiscross-domainneighborselection,anditcangreatly speedupthejob'sexecutioninthisscenario,asdemonstratedbytheexperimentin Section6.6. 6.4.4MonitorandCommunicator Themonitorperiodicallycollectsdynamiclocalresources'statusinformation e.g.CPUutilization,memoryutilization,andaverageloadandupdatesittothelocal knowledgebase.Inaddition,themonitorcheckseveryjoblistedintheknowledgebase periodically.Ifthejobnishessuccessfully,themonitorcollectssomestatisticdata aboutthisexecutione.g.theapplication'sinputparameters,performanceandresource 148

PAGE 149

usage,andreportsittotheknowledgebaseforhistoricalrecords.Ifthejobisstill running,themonitorfeedsthemonitoreddataabouttheresourceandthejobtothe controllerforestimatingtheprogressofthegivenjob. TheCommunicatorisresponsibleforsendingandreceivingmessagestoandfrom neighbors.Therearefourtypesofmessagesexchangedbetweenneighbors. Joining/leaving: AnAVAMsendsmessagestoitsneighborstonotifyitsarrivalor departure.Themessagescarrythesender'sIDsothatthereceivercanadd/deleteitto orfromitsneighborlist. Localresourcetable: EachAVAMhasitsowncurrentviewoftheresources'status andstoresitinalocalresourcetable.Todisseminatethisinformation,everyAVAM periodicallyevery10secondsinourimplementationsendsitslocalresourcetableto theneighbors. Rewiring: Beforeleaving,anAVAMselectsafractionpsetto0.5inourcaseofits neighborsandsendsthemrewiringmessages.Thereceiversthenchoosesomeother AVAMsastheirnewneighbors. 6.4.5InformationFiltering Theresources'statusinformationcollectedbyanAVAM'smonitorandcommunicator mustbelteredbeforebeingaddedtothelocalresourcetabletoreducethemessage size.Torealizethis,eachrecordhasanageattributetoindicatethetimeelapsedsince thelastupdate.Iftworecordscontainthesameresource'sinformation,theolderone getslteredout. Informationlteringalsohappensbypurgingthelower-valuesrecordsfromthe table.ConcentratingonCPU-intensiveapplications,AVAMsareinterestedinresources withhighCPUprocessingpower.Thus,thevalueoftheithresourcerecordisdenedas follows.IfCPUutilizationstaysbelow100%,theCPUcapacityiscalculatedbytheCPU speedandutilization;otherwise,itiscomputedusingtheCPUloadthequeuelength 149

PAGE 150

oftherunnableprocesses.Aweightof0.01isusedtomakethesetwomeasurements comparable. value i = 8 > < > : CPU speed i )]TJ/F23 11.9552 Tf 11.955 0 Td [(CPU utilization i if CPU utilization i 1 CPU speed i CPU load i 0 : 01 otherwise Ifamonitoredresourceisavirtualmachine,itsCPUprocessingpowerisaffected bytheCPUloadofitsphysicalhost.Insuchcase,theAVAMmonitorsthestatusofboth thevirtualmachineandthephysicalhost.Thevaluecalculatedasaboveisreducedby afactorthatisinverselyproportionaltothephysicalhost'sload. Duetothedynamiccharacteristicsofgridresources,theolderaresourcerecord becomes,thelessaccurateitis.Therefore,whileevaluatingaresourcerecordduring thetablepurgingprocess,therecord'svalueisreducedbyafactorcorresponding toitsage,representedas =1 )]TJ/F24 7.9701 Tf 16.1 5.256 Td [(age max ,wherethemaxissetto60secondsinour implementation.Withthisinformationltering,alocalresourcetable'ssizeisoptimized byonlyretainingtheresourceswithhighCPUprocessingcapability. 6.5ExperimentalEvaluation 6.5.1ExperimentalSetup TheexperimentswereconductedonasubsetoftheIn-VIGOsystem.Thecomputer resourcesconsistof200VMware-servervirtualmachineseachhas128MBmemory andrunsRedHat7.3hostedonaclusterofdual2.4GHzhyper-threadedXeonnodes. AresourcecanbeusedtohostasingleAVAMandmultipleapplicationjobs simultaneously.Intheexperiments,aconsiderableamountofbackgroundloadwas alsointroducedintotheresourcesbylaunchingCPU-intensivejobs.Dynamicloading environmentswerecreatedbyrandomlychoosingandloadingdifferentsubsetsofthe resourcesrandomlychosenresources,unlessotherwisenotedevery50seconds. 150

PAGE 151

Figure6-3.ThecomparisonoftheexecutiontimeofTunProbjobswithdifferentnumber ofneighbors m during150seconds. ThedomainregistriesareimplementedwithMySQL.TunProbNumerical CalculationoftheTransmissionProbabilityforOne-DimensionalElectronTunneling,a toolavailableontheIn-VIGOportal,isusedasanapplicationbenchmarkrepresentative ofCPU-intensiveworkloads.IntheexperimentseachAVAMwasusedtomanagethe executionofoneormoreinstancesofTunProb. TheDAVAMsysteminitializationprocessstartswithoneAVAM.Thenateach incrementoftimeonesecondonenewAVAMisstarteduntiltheexpectedsystem sizeisreached.EachAVAMestablishesconnectionswith m randomly-chosenexisting AVAMsinitsdomain.Fromtheequation6wecanseethatsetting m asmallvalue -4isgoodenoughforanAVAMnetworkwhosesizeissmallerthan100.Thisisalso conrmedbythefollowingexperiments.EachAVAMmonitorsuptovevirtualmachines asitslocalresources,andupdatestheirstatusCPUload,CPUutilizationandfree memorysizeandtheirphysicalhosts'loadinitslocalresourcetableeverytenseconds. AVAMneighborsexchangetheirlocalresourcetableseverytensecondsandthetable canonlykeepuptotenresourcerecords. 151

PAGE 152

Figure6-4.ThecomparisonoftheTunProbjobsnishedby5AVAMswithdifferent numberofneighbors m during150seconds. 6.5.2ExperimentalEvaluationofEfciency TheefciencyoftheDAVAMsystemisreectedbyeachAVAMbeingabletoquickly obtainthecurrentstatusoftheentiresystemandndgoodresourcesforitsjobs.The rstexperimentinvestigateshowtheperformancechangeswithdifferentnumbersof neighborseachAVAMcontactswhenjoiningthedomain.FiftyAVAMswereinitially startedinthedomain,and10secondslateranotherveAVAMsjoinedthesamedomain andeachselectedavaluebetween0to6neighborstocommunicatewith.Afterten secondsoftheirarrivals,theveAVAMsbegantosubmitjobscontinuouslyuntiltheyleft thedomain140secondslater. Figure6-3andFigure6-4comparestheaveragejobruntimeandtheoverall throughputthetotalnumberofjobscompletedbythe5AVAMswithdifferentvaluesof m .Asexpectable,theworstperformanceoccurswheneachAVAMdoesnothaveany neighborsandonlyknowsthestatusofitslocalresources.Asthevalueofmincreases, theperformanceimprovesbecauseAVAMscanlearnmoreresources'information throughinteractionwiththeirneighborsandselectresourcesmorewisely.Figure6-4 alsoindicatesthat,whenthevalueof m exceedsve,thethroughputdropsbecausethe benetfromcontactingmoreneighborsisoutweighedbycommunicationoverhead. 152

PAGE 153

Figure6-5.ThecomparisonofthejobsaverageexecutiontimewithDAVAMandthe centralizedapproachesinthreedifferentloadingenvironments. 6.5.3ExperimentalEvaluationofScalability Inthesecondexperiment,westudiedthesystemscalabilitybycomparingthe performanceofDAVAMwithcentralizedmonitoringandround-robinapproaches.Forty AVAMsjointhedomainandeachonesubmitsjobscontinuouslyfor150seconds.In theDAVAMapproach,eachAVAMselectstwoneighbors.Theneighborselections, withandwithoutpreference,leadtotwotypesofnetworks,power-lawandexponential networks[136],respectively. The centralized-monitoring approachusesacentralmonitortocollectandstore resources'dynamicstatusinformation.EachAVAMchoosesthebestresourcecurrently availableinthedatabasetosubmititsjobs.Theround-robinapproachdoesnotneed anyresourcestatusinformationastheAVAMschooseresourcesfromthedatabase inaround-robinmanner.Theexperimentswereconductedinthreedifferentloading environments:low,mediumandhigh,inwhich30%,50%and70%ofrandomlychosen resourcesfromthedomainwereloadedwithCPU-intensiveprocesses,respectively. Figure6-5andFigure6-6showstheaveragejobruntimeandtheoverallthroughput ofthedifferentapproaches.Bothexponentialandpower-lawAVAMnetworksdeliver similarbestperformancebecausethesmallworldpropertymakessurethateach 153

PAGE 154

Figure6-6.Thecomparisonofthejobsnishedby40AVAMsfortheDAVAMandthe centralizedapproachesinthreedifferentloadingenvironments. AVAMinthenetworkcanobtainthelatestsystem-wideresourcestatusveryquickly. Furthermore,the -randomresourceselectionmechanismavoidsresourcecontention amongmultipleAVAMs.Incontrast,thecentralizedmonitoringapproachsuffers fromdatabase-accesscontentionbetweentheAVAMsandthecentralmonitor,and hencebehavesmuchworsethanDAVAM.Theround-robinapproachgivestheworst performancebecauseitdoesnotconsideranydynamicstatusinformationforresource selection. 6.5.4ExperimentalEvaluationofRobustness ThethirdexperimentstudiestherobustnessoftheDAVAMapproach,wherethe system-levelinformationisconstructedbythedistributedcooperativeAVAMs,incontrast withthecentralizedapproach,whereacentraldatabaseisusedtostoretheglobal knowledge.Intheexperiment,50AVAMswerestartedinasingledomainatthesame time.After200seconds,halfofthemleftandtheotherhalfcontinuedtoworkand submitjobsforanother200seconds. InDAVAMtheremainingAVAMsreacttosystemchangesbycontactingnew neighborsandreclaimingresourcesfromthedomainregistry.Theneighborhood rebuildingmechanismmaintainstheDAVAMnetworkconnectivity,andtheresource 154

PAGE 155

Figure6-7.TunProbjobsaverageexecutiontimebeforeandafterAVAMleaving,for DAVAMexponentialandpower-lawnetworks,andbeforeandafterfailureof acentraldatabasewhenitisusedforcentralizedmonitoring. Figure6-8.ThetotalnumberofTunProbjobsnishedby25AVAMsbeforeandafter AVAMleaving,forDAVAMexponentialandpower-lawnetworks,andbefore andafterfailureofacentraldatabasewhenitisusedforcentralized monitoring. reclaimingensuresthatmostoftheresourcesarestillmonitoredbyatleastoneAVAM. Figure6-7andFigure6-8comparestheaveragejobruntimeandtheoverallthroughput bythe25AVAMsbeforeandaftertheotherhalfoftheAVAMsleftthedomain.The resultsshowthat,forbothexponentialandpower-lawnetworks,theperformanceofthe remainingAVAMsisalmostunaffectedevenifahighnumberofAVAMsleftthesystem. 155

PAGE 156

Forthecentralizedmonitoringapproach,onthecontrary,ifthecentraldatabase fails,noneoftheAVAMscanretrieveanynewinformationfromthedatabase,sothey havetocontinueusingtheresourceschosenbeforethedatabasefailure.Figure6 showsthat,withoutthedynamicresourcestatusinformationprovidedbythedatabase, theperformancedropsdramatically.Similareffectscanbeobservedifthecentral monitorfails. 6.6Discussion Itmaypossibletodesignahierarchicalsystemtocircumventscalabilityissues causedbyapurelycentralizedapproachandalsoachievethesimilarperformancewith thep2papproach.However,inadynamicenvironmentwherenodescanjoinandleave atanytime,itisverydifculttoconstructandmaintainabalanced,optimalhierarchical structure.Moreover,thesupernodesrootnotesatthetoplevelinthehierarchical systemcanpotentiallycausesingle-pointsystemfailuresand/orleadtoisolatednodes inthesystem.Althoughreplicationcancompensateforpotentialunstablebehaviorof asupernode,itwilladdresourcecostsandcommunicationoverheadtokeepreplicas consistent. 6.7RelatedWork Agent-based[140][141][142]modelingisaverynaturalandexiblewaytomodel distributedinterconnectedsystems.In[143]severaldistributedandself-organizing algorithmsareproposedforplacementofservicesonservers.Foreachservicea servicemanagerisinstantiatedtocreatemultipleantsagentsandsendthemoutto theservernetwork.Theanttravelsfromoneservertoanother,choosingtheservers alongthepathbasedonlocallyavailableinformation.Theantthennallymakesa decision,basedontheknowledgeithasaccumulatedonitstravel.Servicemanager andthespawnedantsworkwithlocalinformation,whichensuresscalability.Similarly, Messor[142]proposedtouseantswanderingoverthenetworktoexploreload 156

PAGE 157

conditions.Thegoalistoachieveloadbalancingbyantsmovingjobsfromthemost overloadednodetounderloadedones. Inoursystem,eachautonomiccomponentcanbeidentiedasanagent,and theautonomicsystemasamulti-agentsystem.Eachautonomiccomponentisboth cooperativesharingitslocalknowledgewithneighborsandselshtryingtond andallocatethebestresourcesforitsownjobs.Theauthorsin[140]claimthatno obviousgaincanbeachievedfromcommunicationbetweenagents.Thereasonisthat ifalltheagentshaveabetterpictureofthewholesystem,theyalltendtousethe bestresourcesandthuscausecompetition.Incontrast,theresourcevericationand -randomselectionmechanismsappliedtooursystemcanpreventthisproblemand theireffectivenessisprovedbytheexperiments. Thepeer-to-peermodeloffersanalternativetothetraditionalclient-servermodelfor manylarge-scaleapplicationsindistributedsetting.Epidemicorgossipalgorithms [144][134]haveprovedtobeeffectivesolutionsfordisseminatinginformationin large-scalesystems.Thebasicideaisthateachprocessperiodicallychoosesa randomsubsetofprocessesinthesystemandsendsthemthenewinformationit hasreceived.Oneoftheissuesunderlyingthedeploymentoftraditionalepidemic algorithmsisthattheyrelyoneachprocesshavingknowledgeoftheglobalmembership whichisnotrealisticforlargegroupsofprocesses.Thereliableandefcientinformation disseminationinoursystemisinheritedfromthesimpleneighborrelationshipestablishment overthesystemanddynamicneighborhoodrebuildingmechanism.Themembership protocolisverysimpletodeploywithsupportfromthedecentralizeddomainregistry service. 6.8Conclusions Thischapterpresentsanautonomiccomputingsysteminwhichmultipleautonomic componentscollaboratetooptimizethebehaviorofthesystem.Ageneralautonomic managermodelisdesignedtocontrolthemanagedelements'internalstateand 157

PAGE 158

manageitsinteractionswiththesurroundingenvironment.Theautonomicmanageris lightweight,makingitsuitableformanydistributedsystems.Eachhasalocalviewof thesystemstateandcommunicatesperiodicallyitspartialknowledgetoitsneighbors, thuscontributingtobuildingacommon,sharedglobalviewofthesystemstate.A decentralizedregistryprovidesscalableandreliableneighborandresourcediscovery serviceforthesystem.Theoverlaynetworkstructuredbytheneighborrelationshipsis demonstratedtobehighlyreliableandefcient.Theresultsshowthatthedecentralized andcooperativenatureofthesystemyieldsanumberofdesirableproperties,including efciency,robustness,andscalabilityunderahighlydynamicenvironment.Thedesign decisionsweremotivatedbyresultsfromnetworksciencewhichalsoprovidedabasis fortheanalyticalandexperimentalevaluationofthedescribeddistributedautonomic managementsystem.Futureadvancesinnetworkscienceandautonomiccomputing willbeneteachother,tomanagecomplexIT-enabledsystemsandtoaddressnew managementchallengesofnetworkedofphysicalandsocialentities,respectively. 158

PAGE 159

CHAPTER7 CONCLUSIONSANDFUTUREWORK Themainobjectiveofthisdissertationistoachieveautonomicapplicationand resourcemanagementinvirtualizeddistributedcomputingsystems,targetinggridsand datacenters.Suchenvironmentsaretypicallyrunningalargevarietyofapplications withdifferentresourcerequirementsanddynamicworkloads,whichposegreat challengesforprovidingdesiredpropertiessuchasperformanceguarantees,efcient useofresources,andreducedoperationalandmanagementcosts.Anoveldesignof two-levelcontrolsystemisproposedtoreducethemanagementcomplexityandallow independentandexibleoptimizationandadaptation.Byincorporatingmethodsand techniquesfrommachinelearning,optimization,andcontroltheory,theproposedcontrol systemsachievetoincreasetheabilitytoadapttoanunpredictableenvironmentand managetomaximizeapplicationperformanceandminimizeassociatedcosts. Inthecontextofgridenvironments,themaincontributionofourworkistosupport workloadsthatrequireaspeciedqualityofserviceongridresourceswithvarying loadsandunpredictedavailability.Animplementationofthetwo-levelcontrolsystemis deployedintheIn-VIGOsystem,agridsystemforscienticapplicationsandextensively utilizesvirtualizationtechnologies.Alocalcontrolleractivatedforeachapplication sessionappliesamemory-basedlearningapproachtopredictitsjob'sresourceneeds andallocatestheproperresourcesforthejobusingtheresourcestatusinformation providedbytheglobalcontroller.Thelocalcontrolleralsomonitorsjobs'performance anddeterminescontrolactionssuchasreschedulingajobtobetterresourceswhenthe predictedjobperformancecannotmeetusers'requirements.Theautonomicapplication managementsystemistestedfortheapplicationsthatgenerateCPU-intensivejobs withshortexecutiontimes.Inthescenariosconsideredinthiswork,theresultsconrm thefollowingpoints:Theonlinelearningalgorithmisfastandthepredictionerroris lowbelow10%;Thetwo-levelcontrolsystemcanquicklyrespondtothedynamic 159

PAGE 160

changesofresourcestatusandimprovetheexecutionperformancesignicantly comparedtotheapproachwithoutusingresourceinformation;Withtheprompt reschedulingcontrolaction,thecontrolsystemcaneffectivelyrecoverfromperformance faultsandexecutionfailures. Thesecondpartofourworkfocusesonaddressingchallengesinautomating resourcemanagementinavirtualizeddatacenterenvironment.Comparedtoagrid environment,theresourcesindatacentersaretypicallymorereliableandlocatedat acentralizedlocation.However,theapplicationshostedindatacentersusuallyhave highlydynamicworkloadsthatarehardtopredict.Inaddition,therelationshipbetween theworkloadsandtheirresourceneedsaremorecomplexandmostoftimecannot bedescribedusinglinearmodels.Thefuzzy-logic-basedapproachesproposed inthisdissertationprovideagenericsolutiontolearningtherelationshipwithout makingunderlyingassumptionsonworkloadcharacteristicsorsystembehaviors. Theexperimentalevaluationshowsthatitcanefcientlymodelthenonlinearrelationship betweenworkloadsandresourceneedswithdynamicallychangingoperatingconditions veryfast.Theresourcecostconsumedbyeachvirtualmachineanditshosted applicationissignicantlyreducedwhiletheapplicationperformanceisstillguaranteed. Besidesoptimizingresourceusetoreducecosts,severalotheraspectsof datacenterssuchaspowerconsumptionandhotspotsissuesbecomeincreasingly important.Indeterminingvirtualmachineplacementtophysicalhosts,amulti-objective optimizationsolutionisproposedforsimultaneouslyoptimizingseveralpossibly conictingobjectives.Afuzzy-logic-basedevaluationapproachisusedtoconveniently combinedifferentobjectivesandanimprovedgeneticalgorithmisappliedforsearching theplacementsolutionspaceglobally.Conrmedbythesimulation-basedexperiments, theproposedapproachseeksgoodbalanceamongdifferentobjectivesandshows superiorperformancecomparedwithwell-knownbin-packingalgorithmsandsingle-objective approaches.Dynamicvirtualmachinemigrationisusedtoadapttochangesofsystem 160

PAGE 161

conditionsandapplicationworkloads.Theinformationacrossboththevirtualization layerandthephysicalresourcelayerareutilizedindeterminingwhen,whichand wheretomovevirtualmachines.Atwo-leveldetectionisappliedinthecontrollerfor fastdetectionwhileavoidingunnecessarymigrations.Themulti-objectiveoptimization approachisalsoappliedinselectinghostsformigratedvirtualmachines.System stabilizationisconsideredinthedesignofthecontrolsystem.Theexperiments conductedonanIBMBladeCenterconrmedthattheappliedstabilizationtechnique signicantlyreducesthenumberofvirtualmachinemigrations.Inaddition,the multi-objectiveselectionperformsbetterthansingle-objectivewithrespecttopower consumption,thermaldistribution,resourceusage,andmigrationoverheadinthe scenariosconsideredinthiswork. Theabovesummarizedthemajoraccomplishmentsinthisdissertation,however, thereareseverallimitationsinthisworkthatareworthdiscussing. 1.Intheworkonautonomicapplicationmanagementingridsystems,thelocal controllermonitorsandpredictstheprogressofitsjobsrunningonagridresource. Ifthejobispredictedtoexceedthedeadlines,thecontrollerstopsthecurrentjob andrestartsitonabetterresourcewherethejobisexpectedtonishsooner. Thisstop-restartjobreschedulingmechanismworkswellinthecasethatthe performancefaultisdetectedintheearlystageofajob'sexecutionortheexecution timeofthejobisverysmalle.g.,lessthantensofminutes.Ifajobhasbeen executedforalongtimee.g.,hoursorevendaysbeforeitisstoppedbythe controller,alltheworkthathasbeendoneislost.Analternativewayistosave thecurrentjob'sstatusi.e.,checkpointingandmigratethejob.Comparedto processmigration,virtualmachinemigrationprovidesawayofconvenientlymoving workloadswithoutmodifyingapplications.However,becauseoftheconsiderable migrationoverhead,thecontrollermustcarefullychoosebetweenthestop-restart andsuspend-restartreschedulingapproaches. 2.Inthetwo-levelresourcecontrolofvirtualizeddatacenters,localcontrollerspredict resourceneedsoftheirvirtualmachinesusingfuzzymodelingorfuzzyprediction approachesandtheglobalcontrollerallocatesresourcestovirtualmachinesbased ontheestimation.Iftheestimatedresourcedemandsaresmallerthanactual needs,theapplication'sperformancewillsufferduetotheresourcescarcity.On theotherhand,iftheestimatedvalueismuchlargerthantheactualneeds,the resourcesarewastedsothatthedatacenter'sprotisreduced.Therefore,the predictionaccuracyisexpectedtohaveanimpactonthesystemperformance, 161

PAGE 162

whichisnotfullyexploredinthisdissertationthecurrentworkusesasimple correctionactionwhichallocatesapredenedpercentageofresourcesincaseof performancedegradation. 3.Intheworkofvirtualmachineplacement,thethermalmanagementissimpliedby consideringonlyCPUtemperatureandthepowerconsumptiononlyconsiders thepowerconsumedbytheCPUsonthebladenodes.However,thereisa considerableamountofpowerconsumedbyCPUfanandcoolingfacilitiesinadata center,whichisalsoaffectedbyworkloadintensityanddistribution.BesideCPU temperaturewhichislargelydeterminedbyCPUactivity,thethermaldistribution ofadatacenterishighlyaffectedbyheatrecirculation,acommonphenomenonin datacenters. 4.Thecentralizedmanagementatthesecondleveloftheproposedcontrolsystem wouldbecomebottleneckforextremelylarge-scaledatacenterssuchascloud computingdatacenters.Chapter6investigateddecentralizedandcooperative resourcemanagementinthecontextoftheIn-VIGOgridsystem.However, thisapproachwouldnotperformwellinadatacenterenvironmentbecause ofthefollowingissues:1Howtoresolvetheconictsamongdifferentlocal controllers.Forexample,thesameservermaybechosenasdestinationby multiplecontrollerstomigratetheirvirtualmachinesandbecomeoverloaded. 2Howtodealwiththedelayofinformationsharingamonglocalcontrollers. Theaccuracyandpromptnessofresourceinformationarecriticaltodatacenter environments.3Randomneighborselectionprobablywon'tbetheoptimalway forconstructingamanagementnetwork.Thepositionoftheserverswillaffectthe systemperformanceduetonetworklatency. Eachoftheabovelimitationsisapotentialtopictobeinvestigatedinthefuture work.Inaddition,withtheideaofInfrastructure-as-a-serviceIaaS,Platform-as-a-Service PaaS,andSoftware-as-a-ServiceSaaS,cloudcomputingemergesasamainstream platform.Cloudcomputinghasitsconnectiontogridcomputingandvirtualizeddata centersbutwithitsdistinctcharacteristics,whichmakesitquitechallengingtoextend ourworkproposedinthisdissertationtoclouds.Thefollowingisthedenitionofcloud computingfromIanFoster,Alarge-scaledistributedcomputingparadigmthatisdriven byeconomiesofscale,inwhichapoolofabstracted,virtualized,dynamically-scalable, managedcomputingpower,storage,platforms,andservicesaredeliveredondemandto externalcustomersovertheInternet.[145].Differentfromgridcomputingwhichprovides aninfrastructurethatdeliversstorageandcomputeresources,cloudcomputingshiftsto 162

PAGE 163

theonethatiseconomybased,aimingtodelivermoreabstractresourcesandservices usuallythroughvirtualization.Comparedwithvirtualizeddatacenterenvironment consideredinthiswork,cloudcomputingprovidesdifferentlevelsofservices,which makesmonitoringmorechallenging.Usersareonlyexposedtohigh-levelserversand donothavethedetailedinformationabouttheresourcestatus.Thesameproblems potentiallyexistforClouddevelopersandadministrators,astheabstract/unied resourcesusuallygothroughvirtualizationandsomeotherlevelofencapsulation,and trackingtheissuesdownthesoftware/hardwarestackmightbemoredifcult.Network latencyandbandwidthmayalsoraiseissuesincloudcomputingsinceusersaccess theirservicesthroughwideareanetworks.ToensureagoodlevelofQoSdeliveredto theenduserswillbeoneofthemajorchallengesforcloudcomputing. 163

PAGE 164

REFERENCES [1]Ibmsolutionsgridforbusinesspartners. URL:http://joung.im.ntu.edu.tw/teaching/distributed systems/documents/IBM grid wp.pdf. [2]N.H.KapadiaandJ.A.B.Fortes,Punch:Anarchitectureforweb-enabled wide-areanetwork-computing, ClusterComputing ,vol.2,no.2,pp.153, 1999. [3]S.Adabala,V.Chadha,P.Chawla,R.Figueiredo,J.Fortes,I.Krsul, A.Matsunaga,M.Tsugawa,J.Zhang,M.Zhao,L.Zhu,andX.Zhu,From virtualizedresourcestovirtualcomputinggrids:thein-vigosystem, Future GenerationComputerSystems ,vol.21,pp.896,June2005. [4]Howdoyoudenethedatacenter. URL:http://www.networkworld.com/newsletters/datacenter/2006/0206datacenter1.html?page=1. [5]Howstreamingtheosandapplicationschangestherulesforserverprovisioning. URL:www.dell.com/downloads/global/power/ps3q07-20070601-Citrix.pdf. [6]Y.Chen,A.Das,W.Qin,A.Sivasubramaniam,Q.Wang,andN.Gautam, Managingserverenergyandoperationalcostsinhostingcenters, SIGMETRICSPerform.Eval.Rev. ,vol.33,no.1,pp.303,2005. [7]J.S.Chase,D.C.Anderson,P.N.Thakar,A.M.Vahdat,andR.P.Doyle, Managingenergyandserverresourcesinhostingcenters, SIGOPSOper. Syst.Rev. ,vol.35,no.5,pp.103,2001. [8]X.Fan,W.-D.Weber,andL.A.Barroso,Powerprovisioningfora warehouse-sizedcomputer, SIGARCHComput.Archit.News ,vol.35,pp.13, June2007. [9]Q.Tang,S.K.S.Gupta,andG.Varsamopoulos,Energy-efcientthermal-aware taskschedulingforhomogeneoushigh-performancecomputingdatacenters: Acyber-physicalapproach, IEEETrans.ParallelDistrib.Syst. ,vol.19,no.11, pp.1458,2008. [10]Estimatingtotalpowerconsumptioninserversintheu.s.andtheworld. URL:http://www.networkworld.com/newsletters/datacenter/2006/0206datacenter1.html?page=1. [11]B.Dragovic,K.Fraser,S.Hand,T.Harris,A.Ho,I.Pratt,A.Wareld,P.Barham, andR.Neugebauer,Xenandtheartofvirtualization,in Proc.theACMSymposiumonOperatingSystemsPrinciples ,October2003. [12]Vmware:Serveranddatacentervirtualizationproducts. URL:http://www.vmware.com/products/datacenter-virtualization.html. [13]Autonomiccomputingatibm.URL:http://www.research.ibm.com/autonomic/. 164

PAGE 165

[14]autonomicnervoussystem. URL:http://en.wikipedia.org/wiki/Autonomic nervous system. [15]Autonomiccomputingblueprint. URL:http://www-01.ibm.com/software/tivoli/autonomic/pdfs/AC Blueprint White Paper 4th.pdf. [16]Y.Diao,J.L.Hellerstein,andS.Parekh,Usingfuzzycontroltomaximizeprotsin servicelevelmanagement, IBMSystemsJournal ,vol.41,2002. [17]M.SugenoandT.Yasukawa,Afuzzy-logic-basedapproachtoqualitative modeling, IEEETransactionsonFuzzySystems ,vol.1,1993. [18]J.Xu,M.Zhao,J.Fortes,R.Carpenter,andM.Yousif,Ontheuseoffuzzy modelinginvirtualizeddatacentermanagement,in ProceedingsoftheFourth InternationalConferenceonAutonomicComputing ,Washington,DC,USA, pp.25,IEEEComputerSociety,2007. [19]M.Sakawa, GeneticAlgorithmsandFuzzyMultiobjectiveOptimization .Norwell, MA,USA:KluwerAcademicPublishers,2001. [20]J.P.Jones,B.Nitzberg,andB.Henderson,Workloadmanagement:Morethan justjobscheduling, ClusterComputing,IEEEInternationalConferenceon ,vol.0, p.113,2001. [21]Y.EtsionandD.Tsafrir,Ashortsurveyofcommercialclusterbatchschedulers, tech.rep.,HebrewUniversity,2005. [22]Mauiclusterscheuler.URL:http://www.clusterresources.com/products/maui/. [23]Loadlevelerclusterscheuler.URL: http://www-03.ibm.com/systems/clusters/software/loadleveler/index.html. [24]A.Bayucan,R.L.Henderson,C.Lesiak,B.Mann,T.Proett,andD.Tweten, Portablebatchsystem:Externalreferencespecication,tech.rep.,MRJ TechnologySolutions,November1999. [25]Lsfclusterscheduler.URL: http://www.platform.com/workload-management/high-performance-computing/lp. [26]J.P.JonesandB.Nitzberg,Schedulingforparallelsupercomputing:Ahistorical perspectiveofachievableutilization,in ProceedingsoftheJobScheduling StrategiesforParallelProcessing ,IPPS/SPDP'99/JSSPP'99,London,UK, pp.1,Springer-Verlag,1999. [27]D.FeitelsonandA.Weil,Utilizationandpredictabilityinschedulingtheibmsp2 withbacklling, ParallelProcessingSymposium,International ,vol.0,p.0542, 1998. 165

PAGE 166

[28]A.W.Mu'alemandD.G.Feitelson,Utilization,predictability,workloads,anduser runtimeestimatesinschedulingtheibmsp2withbacklling, IEEETransactions onParallelandDistributedSystems ,vol.12,pp.529,2001. [29]M.Zaharia,D.Borthakur,J.SenSarma,K.Elmeleegy,S.Shenker,and I.Stoica,Jobschedulingformulti-usermapreduceclusters,Tech.Rep. UCB/EECS-2009-55,EECSDepartment,UniversityofCalifornia,Berkeley, Apr2009. [30]D.ZotkinandP.J.Keleher,Job-lengthestimationandperformanceinbacklling schedulers,in Proceedingsofthe8thIEEEInternationalSymposiumonHigh PerformanceDistributedComputing ,HPDC'99,Washington,DC,USA,IEEE ComputerSociety,1999. [31]W.Tang,N.Desai,D.Buettner,andZ.Lan,Analyzingandadjustinguserruntime estimatestoimprovejobschedulingonthebluegene/p,in ParallelDistributed ProcessingIPDPS,2010IEEEInternationalSymposiumon ,pp.1,2010. [32]S.-H.Chiang,A.C.Arpaci-Dusseau,andM.K.Vernon,Theimpactofmore accuraterequestedruntimesonproductionjobschedulingperformance,in RevisedPapersfromthe8thInternationalWorkshoponJobSchedulingStrategies forParallelProcessing ,JSSPP'02,London,UK,pp.103,Springer-Verlag, 2002. [33]W.A.Ward,Jr.,C.L.Mahood,andJ.E.West,Schedulingjobsonparallel systemsusingarelaxedbackllstrategy,in RevisedPapersfromthe8thInternationalWorkshoponJobSchedulingStrategiesforParallelProcessing ,JSSPP'02, London,UK,pp.88,Springer-Verlag,2002. [34]S.-H.Chiang,A.C.Arpaci-Dusseau,andM.K.Vernon,Theimpactofmore accuraterequestedruntimesonproductionjobschedulingperformance,in RevisedPapersfromthe8thInternationalWorkshoponJobSchedulingStrategies forParallelProcessing ,JSSPP'02,London,UK,pp.103,Springer-Verlag, 2002. [35]Y.Zhang,M.S.Squillante,andR.K.Sahoo,Performanceimplicationsoffailures inlarge-scaleclusterscheduling,in In10thWorkshoponJobSchedulingStrategiesforParallelProcessing ,pp.233,2004. [36]A.J.Oliner,R.K.Sahoo,J.E.Moreira,M.Gupta,andA.Sivasubramaniam, Fault-awarejobschedulingforbluegene/lsystems,in InIEEEIPDPS,Intl. ParallelandDistributedProcessingSymposium ,pp.64,2004. [37]W.Tang,Z.Lan,N.Desai,andD.Buettner,Fault-aware,utility-basedjob schedulingonblue,gene/psystems,in ClusterComputingandWorkshops, 2009.CLUSTER'09.IEEEInternationalConferenceon ,pp.1,2009. 166

PAGE 167

[38]F.Berman,R.Wolski,H.Casanova,W.Cirne,H.Dail,M.Faerman,S.Figueira, J.Hayes,G.Obertelli,J.Schopf,G.Shao,S.Smallen,N.Spring,A.Su,and D.Zagorodnov,Adaptivecomputingonthegridusingapples, IEEETrans. ParallelDistrib.Syst. ,vol.14,no.4,pp.369,2003. [39]R.Wolski,Dynamicallyforecastingnetworkperformanceusingthenetwork weatherservice, ClusterComputing ,vol.1,no.1,pp.119,1998. [40]M.J.Litzkow,M.Livny,andM.W.Mutka,Condor-ahunterofidleworkstations, in Proc.8thInternationalConferenceonDistributedComputingSystems ,San Jose,CA,USA,pp.104,June1988. [41]M.Litzkow,T.Tannenbaum,J.Basney,andM.Livny,Checkpointandmigrationof unixprocessesinthecondordistributedprocessingsystem,tech.rep.,University ofWisconsin,Madison,1997. [42]F.Berman,A.Chien,K.Cooper,J.Dongarra,I.Foster,D.Gannon,L.Johnsson, K.Kennedy,C.Kesselman,J.Mellor-crummey,D.Reed,L.Torczon,and R.Wolski,Thegradsproject:Softwaresupportforhigh-levelgridapplication development, InternationalJournalofHighPerformanceComputingApplications vol.15,pp.327,2001. [43]R.L.Ribler,H.Simitci,andD.A.Reed,Theautopilotperformance-directed adaptivecontrolsystem, FutureGener.Comput.Syst. ,vol.18,pp.175, September2001. [44]F.Berman,H.Casanova,A.Chien,K.Cooper,H.Dail,A.Dasgupta,W.Deng, J.Dongarra,L.Johnsson,K.Kennedy,C.Koelbel,B.Liu,X.Liu,A.Mandal, G.Marin,M.Mazina,J.Mellor-Crummey,C.Mendes,A.Olugbile,J.M.Patel, D.Reed,Z.Shi,O.Sievert,H.Xia,andA.YarKhan,Newgridschedulingand reschedulingmethodsinthegradsproject, Int.J.ParallelProgram. ,vol.33, pp.209,June2005. [45]K.Krauter,R.Buyya,andM.Maheswaran,Ataxonomyandsurveyofgrid resourcemanagementsystemsfordistributedcomputing, Softw.Pract.Exper. vol.32,pp.135,February2002. [46]K.Keahey,I.Foster,T.Freeman,andX.Zhang,Virtualworkspaces:Achieving qualityofserviceandqualityoflifeonthegrid, ScienticProgramming ,vol.13, 2005. [47]S.Santhanam,P.Elango,A.Arpaci-Dusseau,andM.Livny,Deployingvirtual machinesassandboxesforthegrid,in WORLDS'05:Proceedingsofthe2nd conferenceonReal,LargeDistributedSystems ,Berkeley,CA,USA,pp.7, USENIXAssociation,2005. 167

PAGE 168

[48]M.A.Vouk,A.Rindos,S.F.Averitt,J.Bass,M.Bugaev,A.Kurth,A.Peeler, H.E.Schaffer,E.D.Sills,S.Stein,J.Thompson,andM.Valenzisi,Using vcltechnologytoimplementdistributedrecongurabledatacentersand computationalservicesforeducationalinstitutions, IBMJ.Res.Dev. ,vol.53, pp.509,July2009. [49]A.Bavier,M.Bowman,B.Chun,D.Culler,S.Karlin,S.Muir,L.Peterson, T.Roscoe,T.Spalink,andM.Wawrzoniak,Operatingsystemsupportfor planetary-scalenetworkservices,in Proceedingsofthe1stconferenceon SymposiumonNetworkedSystemsDesignandImplementation-Volume1 Berkeley,CA,USA,pp.19,USENIXAssociation,2004. [50]P.Padala,K.G.Shin,X.Zhu,M.Uysal,Z.Wang,S.Singhal,A.Merchant, andK.Salem,Adaptivecontrolofvirtualizedresourcesinutilitycomputing environments,in EuroSys'07:Proceedingsofthe2ndACMSIGOPS/EuroSys EuropeanConferenceonComputerSystems2007 ,NewYork,NY,USA, pp.289,ACM,2007. [51]X.Wang,D.Lan,G.Wang,X.Fang,M.Ye,Y.Chen,andQ.Wang, Appliance-basedautonomicprovisioningframeworkforvirtualizedoutsourcing datacenter, AutonomicComputing,InternationalConferenceon ,vol.0,p.29, 2007. [52]D.Kusic,J.O.Kephart,J.E.Hanson,N.Kandasamy,andG.Jiang,Powerand performancemanagementofvirtualizedcomputingenvironmentsvialookahead control,in ICAC'08:Proceedingsofthe2008InternationalConferenceon AutonomicComputing ,Washington,DC,USA,pp.3,IEEEComputer Society,2008. [53]E.KalyvianakiandT.Charalambous,Ondynamicresourceprovisioningfor consolidatedserversinvirtualizeddatacenters,in the8thInternationalWorkshop onPerformabilityModelingofComputerandCommunicationSystems ,2007. [54]Xenserver.URL:http://www.citrix.com/English/ps2/products/product.asp?contentID=683148. [55]G.Jung,M.A.Hiltunen,K.R.Joshi,R.D.Schlichting,andC.Pu,Mistral: Dynamicallymanagingpower,performance,andadaptationcostincloud infrastructures,in ICDCS'10:Proceedingsofthe2010IEEE30thInternationalConferenceonDistributedComputingSystems ,Washington,DC,USA, pp.62,IEEEComputerSociety,2010. [56]M.Y.Lim,F.Rawson,T.Bletsch,andV.W.Freeh,Padd:Powerawaredomain distribution,in ICDCS'09:Proceedingsofthe200929thIEEEInternationalConferenceonDistributedComputingSystems ,Washington,DC,USA,pp.239, IEEEComputerSociety,2009. 168

PAGE 169

[57]A.Verma,P.Ahuja,andA.Neogi,pmapper:powerandmigrationcostaware applicationplacementinvirtualizedsystems,in Proceedingsofthe9th ACM/IFIP/USENIXInternationalConferenceonMiddleware ,Middleware'08, NewYork,NY,USA,pp.243,Springer-VerlagNewYork,Inc.,2008. [58]F.Hermenier,X.Lorca,J.-M.Menaud,G.Muller,andJ.Lawall,Entropy:a consolidationmanagerforclusters,in VEE'09:Proceedingsofthe2009ACM SIGPLAN/SIGOPSinternationalconferenceonVirtualexecutionenvironments NewYork,NY,USA,pp.41,ACM,2009. [59]A.V.TimothyWood,PrashantShenoyandM.Yousif.,Black-boxandgray-box strategiesforvirtualmachinemigration,in ProceedingsoftheFourthSymposium onNetworkedSystemsDesignandImplementation ,2007. [60]T.F.Abdelzaher,J.A.Stankovic,C.Lu,R.Zhang,andY.Lu,Feedback performancecontrolinsoftwareservices, IEEEControlSystemsMagazine vol.23,p.2003,2003. [61]J.Xu,M.Zhao,J.Fortes,R.Carpenter,andM.Yousif,Autonomicresource managementinvirtualizeddatacentersusingfuzzylogic-basedapproaches, ClusterComputing ,vol.11,pp.213,September2008. [62]Z.W.X.Z.S.Singhal,Utilizationandslo-basedcontrolfordynamicsizingof resourcepartitions, LectureNotesinComputerScience ,2005. [63]X.Zhu,Z.Wang,andS.Singhal,Utility-drivenworkloadmanagementusing nestedcontroldesign,in ControlDesign,AmericanControlConference ,2006. [64]W.Xu,X.Zhu,S.Singhal,andZ.Wan,Predictivecontrolfordynamicresource allocationinenterprisedatacenters,in NetworkOperationsandManagement Symposium,2006.NOMS2006.10thIEEE/IFIP ,2006. [65]Y.Wang,X.Wang,M.Chen,andX.Zhu,Partic:Power-awareresponsetime controlforvirtualizedwebservers, ParallelandDistributedSystems,IEEE Transactionson ,vol.22,pp.323,feb2011. [66]X.Wang,M.Chen,C.Lefurgy,andT.Keller,Ship:Scalablehierarchicalpower controlforlarge-scaledatacenters,in ParallelArchitecturesandCompilation Techniques,2009.PACT'09.18thInternationalConferenceon ,pp.91,sept 2009. [67]L.ZhangandD.Ardagna,Slabasedprotoptimizationinautonomiccomputing systems,in ICSOC'04:Proceedingsofthe2ndinternationalconferenceon Serviceorientedcomputing ,NewYork,NY,USA,pp.173,ACM,2004. 169

PAGE 170

[68]X.ZhuandS.Singhal,Optimalresourceassignmentininternetdatacenters, in MASCOTS'01:ProceedingsoftheNinthInternationalSymposiuminModeling,AnalysisandSimulationofComputerandTelecommunicationSystems Washington,DC,USA,p.61,IEEEComputerSociety,2001. [69]A.Chandra,W.Gong,andP.Shenoy,Dynamicresourceallocationforshared datacentersusingonlinemeasurements,in SIGMETRICS'03:Proceedings ofthe2003ACMSIGMETRICSinternationalconferenceonMeasurementand modelingofcomputersystems ,NewYork,NY,USA,pp.300,ACM,2003. [70]P.S.A.Chandra,P.Goyal,Quantifyingthebenetsofresourcemultiplexingin on-demanddatacenters,in ProceedingsoftheeighteenthACMsymposiumon Operatingsystemsprinciples ,2001. [71]M.N.BennaniandD.A.Menasce,Resourceallocationforautonomicdata centersusinganalyticperformancemodels,in ICAC'05:Proceedingsofthe SecondInternationalConferenceonAutomaticComputing ,Washington,DC, USA,pp.229,IEEEComputerSociety,2005. [72]G.TesauroandJ.O.Kephart,Utilityfunctionsinautonomicsystems,in ICAC '04:ProceedingsoftheFirstInternationalConferenceonAutonomicComputing Washington,DC,USA,pp.70,IEEEComputerSociety,2004. [73]J.O.Kephart,H.Chan,R.Das,D.W.Levine,G.Tesauro,F.Rawson,and C.Lefurgy,Coordinatingmultipleautonomicmanagerstoachievespecied power-performancetradeoffs,in ICAC'07:ProceedingsoftheFourthInternationalConferenceonAutonomicComputing ,Washington,DC,USA,p.24,IEEE ComputerSociety,2007. [74]R.Das,J.O.Kephart,C.Lefurgy,G.Tesauro,D.W.Levine,andH.Chan, Autonomicmulti-agentmanagementofpowerandperformanceindata centers,in AAMAS'08:Proceedingsofthe7thinternationaljointconference onAutonomousagentsandmultiagentsystems ,Richland,SC,pp.107, InternationalFoundationforAutonomousAgentsandMultiagentSystems,2008. [75]E.M.Elnozahy,M.Kistler,andR.Rajamony,Energy-efcientserverclusters, in InProceedingsofthe2ndWorkshoponPower-AwareComputingSystems pp.179,2002. [76]E.Pinheiro,R.Bianchini,E.V.Carrera,andT.Heath, Dynamicclusterrecongurationforpowerandperformance ,pp.75.Norwell,MA,USA:Kluwer AcademicPublishers,2003. [77]I.Foster,C.Kesselman,andS.Tuecke,Theanatomyofthegrid:Enabling scalablevirtualorganizations, Int.J.HighPerform.Comput.Appl. ,vol.15, pp.200,August2001. 170

PAGE 171

[78]M.LivnyandD.Thain,Caveatemptor:Makinggridservicesdependablefromthe clientside,in Proc.2002PacicRimInternationalSymposiumonDependable ComputingPRDC02 ,pp.117,2002. [79]D.ThainandM.Livny,Buildingreliableclientsandservers,in TheGrid: BlueprintforaNewComputingInfrastructure I.FosterandC.Kesselman, eds.,pp.224,NewYork,NY:MorganKaufmann,2003. [80]I.ChungandJ.Hollingsworth,Usinginformationfrompriorrunstoimprove automatedtuningsystems,in Proc.theACM/IEEESC2004Conference pp.261,1999. [81]N.Kapadia,J.Fortes,andC.Brodley,Predictiveapplication-performance modelinginacomputationalgridenvironment,in Proc.IEEEInternational SymposiumonHighPerformanceDistributedComputingHPDC1999. ,1999. [82]P.Dinda,Thestatisticalpropertiesofhostload, ACMTrans.Comput.Syst. vol.20,no.1,pp.25,2002. [83]W.PfeifferandN.J.Wright,Modelingandpredictingapplicationperformanceon parallelcomputersusinghpcchallengebenchmarks,in 22ndIEEEInternational ParallelandDistributedProcessingSymposium,HyattRegencyHotel,Miami,FL, 2008 ,2008. [84]A.Snavely,X.Gao,C.Lee,L.Carrington,N.Wolter,J.Labarta,J.Gimenez,and P.Jones,Performancemodelingofhpcapplications,in ParallelComputing ,2003. [85]P.A.Dinda,Thestatisticalpropertiesofhostload, Sci.Program. ,vol.7,no.3-4, pp.211,1999. [86]M.V.DevarakondaandR.K.Iyer,Predictabilityofprocessresourceusage:A measurement-basedstudyonunix, IEEETrans.Softw.Eng. ,vol.15,no.12, pp.1579,1989. [87]J.Albrecht,D.Patterson,andA.Vahdat,Distributedresourcediscoveryon planetlabwithsword,in InWORLDS ,2004. [88]M.LivnyandM.Solomon,Matchmaking:Distributedresourcemanagementfor highthroughputcomputing,in InProceedingsoftheSeventhIEEEInternational SymposiumonHighPerformanceDistributedComputing ,pp.28,1998. [89]S.S.VadhiyarandJ.J.Dongarra,Aperformanceorientedmigrationframework forthegrid,in CCGRID'03:Proceedingsofthe3stInternationalSymposiumon ClusterComputingandtheGrid ,Washington,DC,USA,p.130,IEEEComputer Society,2003. [90]G.VenkitachalamandB.Lim,Virtualizingi/odevicesonvmwareworkstation's hostedvirtualmachinemonitor,in USENIXAnnualTechnicalConference ,2001. 171

PAGE 172

[91]J.Dike,Auser-modeportofthelinuxkernel,in Proc.4thAnnualLinuxShowcaseandConference ,2000. [92]L.Zadeh,Fuzzysets,in InformationandControl ,pp.338,1965. [93]S.Chiu,Fuzzymodelidenticationbasedonclusterestimation, Journalof IntelligentandFuzzySystems ,vol.2,no.3,pp.267,1994. [94]S.MartelloandP.Toth, Knapsackproblems:algorithmsandcomputerimplementations .NewYork,NY,USA:JohnWiley&Sons,Inc.,1990. [95]SunInc., JavaPetstore [96]D.MosbergerandT.Jin,httperfatoolformeasuringwebserverperformance, SIGMETRICSPerform.Eval.Rev. ,vol.26,no.3,pp.31,1998. [97]worldcupwebsiteaccesslogs.URL: ita.ee.lbl.gov/html/contrib/WorldCup.html. [98] Real-timeWebLogReplayer [99]Hp-uxworkloadmanager.URL:http://docs.hp.com/en/5990-8153/ch05s12.html. [100]J.Rolia,L.Cherkasova,andC.McCarthy,Conguringworkloadmanager controlparametersforresourcepools,in NetworkOperationsandManagement Symposium,2006.NOMS2006.10thIEEE/IFIP ,pp.127,2006. [101]T.Abdelzaher,K.G.Shin,andN.Bhatti,Performanceguaranteesforwebserver end-systems:Acontrol-theoreticalapproach, IEEETransactionsonParalleland DistributedSystems ,2002. [102]R.P.Doyle,J.S.Chase,O.M.Asad,W.Jin,andA.M.Vahdat,Model-based resourceprovisioninginawebserviceutility,in USITS'03:Proceedingsofthe 4thconferenceonUSENIXSymposiumonInternetTechnologiesandSystems Berkeley,CA,USA,pp.5,USENIXAssociation,2003. [103]B.Urgaonkar,G.Pacici,P.Shenoy,M.Spreitzer,andA.Tantawi,Ananalytical modelformulti-tierinternetservicesanditsapplications,in SIGMETRICS '05:Proceedingsofthe2005ACMSIGMETRICSinternationalconference onMeasurementandmodelingofcomputersystems ,NewYork,NY,USA, pp.291,ACM,2005. [104]X.Liu,X.Zhu,S.Singhal,andM.F.Arlitt,Adaptiveentitlementcontrolof resourcecontainersonsharedservers,in IntegratedNetworkManagement pp.163,IEEE,2005. 172

PAGE 173

[105]G.Tesauro,Onlineresourceallocationusingdecompositionalreinforcement learning,in Proceedings,TheTwentiethNationalConferenceonArticialIntelligenceandtheSeventeenthInnovativeApplicationsofArticialIntelligence Conference M.M.VelosoandS.Kambhampati,eds.,pp.886,AAAIPress/ TheMITPress,2005. [106]G.Tesauro,N.K.Jong,R.Das,andM.N.Bennani,Ontheuseofhybrid reinforcementlearningforautonomicresourceallocation, ClusterComputing vol.10,no.3,pp.287,2007. [107]D.Gmach,J.Rolia,andL.Cherkasova,Satisfyingservicelevelobjecticesina self-managingresourcepool,in Proceedingsofthe2009ThirdIEEEInternationalConferenceonSelf-AdaptiveandSelf-OrganizingSystems ,SASO'09, Washington,DC,USA,pp.243,IEEEComputerSociety,2009. [108]B.UrgaonkarandA.Chandra,Dynamicprovisioningofmulti-tierinternet applications,in ProceedingsoftheSecondInternationalConferenceonAutomaticComputing ,Washington,DC,USA,pp.217,IEEEComputerSociety, 2005. [109]R.Raghavendra,P.Ranganathan,V.Talwar,Z.Wang,andX.Zhu,Nopower struggles:coordinatedmulti-levelpowermanagementforthedatacenter, SIGARCHComput.Archit.News ,vol.36,pp.48,March2008. [110]A.KrumandF.Kreith, TheCRCHandbookofThermalEngineering .CRCPress, 2000. [111]C.Lefurgy,K.Rajamani,F.Rawson,W.Felter,M.Kistler,andT.W.Keller,Energy managementforcommercialservers, Computer ,vol.36,no.12,pp.39,2003. [112]J.Moore,J.Chase,P.Ranganathan,andR.Sharma,Makingschedulingcool: temperature-awareworkloadplacementindatacenters,in ATEC'05:ProceedingsoftheannualconferenceonUSENIXAnnualTechnicalConference Berkeley,CA,USA,pp.5,USENIXAssociation,2005. [113]Y.-W.B.Chia-HungLienandM.-B.Lin,Estimationbysoftwareforthepower consumptionofstreaming-mediaservers, IEEETrans.onInstrumentationand Measurement ,vol.56,no.5,2007. [114]K.KantandJ.Alexander,Proactivevs.reactiveidlepowercontrol,in Proceeding ofDesignandTestTechnologyConference ,2008. [115]D.E.Goldberg, GeneticAlgorithmsinSearch,OptimizationandMachineLearning .Boston,MA,USA:Addison-WesleyLongmanPublishingCo.,Inc.,1989. [116]E.FalkenauerandA.Delchambre,Ageneticalgorithmforbinpackingand linebalancing,in RoboticsandAutomation,1992.Proceedings.,1992IEEE InternationalConferenceon ,pp.1186vol.2,may.1992. 173

PAGE 174

[117]R.R.Yager,Onorderedweightedaveragingaggregationoperatorsinmulticriteria decisionmaking, IEEETrans.Syst.ManCybern. ,vol.18,no.1,pp.183, 1988. [118]L.RamosandR.Bianchini,C-oracle:Predictivethermalmanagementfordata centers,in HighPerformanceComputerArchitecture,2008.HPCA2008.IEEE 14thInternationalSymposiumon ,pp.111,feb.2008. [119]E.G.Coffman,Jr.,M.R.Garey,andD.S.Johnson, Approximationalgorithmsfor binpacking:asurvey ,pp.46.Boston,MA,USA:PWSPublishingCo.,1997. [120]B.Urgaonkar,A.Rosenberg,andP.Shenoy,Applicationplacementonacluster ofservers, InternationalJournalonFoundationsofComputerScience ,vol.18, no.5,2007. [121]M.Cardosa,M.R.Korupolu,andA.Singh,Sharesandutilitiesbasedpower consolidationinvirtualizedserverenvironments,in IM'09:Proceedingsofthe 11thIFIP/IEEEinternationalconferenceonSymposiumonIntegratedNetwork Management ,Piscataway,NJ,USA,pp.327,IEEEPress,2009. [122]A.Verma,G.Dasgupta,T.K.Nayak,P.De,andR.Kothari,Serverworkload analysisforpowerminimizationusingconsolidation,in USENIX'09:Proceedings ofthe2009conferenceonUSENIXAnnualtechnicalconference ,Berkeley,CA, USA,pp.28,USENIXAssociation,2009. [123]K.Le,R.Bianchini,M.Martonosi,andT.Nguyen,Cost-andenergy-awareload distributionacrossdatacenters,in ProceedingsoftheWorkshoponPower-Aware ComputingandSystems ,2009. [124]R.K.Sharma,C.E.Bash,C.D.Patel,R.J.Friedrich,andJ.S.Chase,Balance ofpower:Dynamicthermalmanagementforinternetdatacenters, IEEEInternet Computing ,vol.9,no.1,pp.42,2005. [125]E.Pinheiro,R.Bianchini,E.V.Carrera,andT.Heath,Loadbalancingand unbalancingforpowerandperformanceincluster-basedsystems,in InProceedingsoftheInternationalWorkshoponCompilersandOperatingSystemsforLow Power ,2001. [126]N.Tolia,Z.Wang,M.Marwah,C.Bash,P.Ranganathan,andX.Zhu,Delivering energyproportionalitywithnonenergy-proportionalsystems:optimizingthe ensemble,in HotPower'08:Proceedingsofthe2008conferenceonPoweraware computingandsystems ,Berkeley,CA,USA,pp.2,USENIXAssociation, 2008. [127]V.Sharma,A.Thomas,T.Abdelzaher,K.Skadron,andZ.Lu,Power-aware qosmanagementinwebservers,in RTSS'03:Proceedingsofthe24thIEEE InternationalReal-TimeSystemsSymposium ,Washington,DC,USA,p.63, IEEEComputerSociety,2003. 174

PAGE 175

[128]N.Bobroff,A.Kochut,andK.Beaty,Dynamicplacementofvirtualmachinesfor managingslaviolations.,in IntegratedNetworkManagement ,pp.119,IEEE, 2007. [129]A.Karve,T.Kimbrel,G.Pacici,M.Spreitzer,M.Steinder,M.Sviridenko,and A.Tantawi,Dynamicplacementforclusteredwebapplications,in WWW'06: Proceedingsofthe15thinternationalconferenceonWorldWideWeb ,NewYork, NY,USA,pp.595,ACM,2006. [130]S.Kumar,V.Talwar,V.Kumar,P.Ranganathan,andK.Schwan,vmanage: looselycoupledplatformandvirtualizationmanagementindatacenters,in ICAC '09:Proceedingsofthe6thinternationalconferenceonAutonomiccomputing NewYork,NY,USA,pp.127,ACM,2009. [131]H.Liu,Acomponent-basedprogrammingmodelforautonomicapplications, in ICAC'04:ProceedingsoftheFirstInternationalConferenceonAutonomic Computing ,Washington,DC,USA,pp.10,IEEEComputerSociety,2004. [132]S.R.White,J.E.Hanson,I.Whalley,D.M.Chess,A.Segal,andJ.O.Kephart, Autonomiccomputing:Architecturalapproachandprototype, Integr.Comput.AidedEng. ,vol.13,no.2,pp.173,2006. [133]J.Pereira,L.Rodrigues,andR.Oliveira,Semanticallyreliablemulticast: Denition,implementation,andperformanceevaluation, IEEETrans.Comput. vol.52,no.2,pp.150,2003. [134]P.T.Eugster,R.Guerraoui,A.-M.Kermarrec,andL.Massoulieacute,Epidemic informationdisseminationindistributedsystems, Computer ,vol.37,no.5, pp.60,2004. [135]R.SteinmetzandK.Wehrle, Peer-to-PeerSystemsandApplicationsLecture NotesinComputerScience .Secaucus,NJ,USA:Springer-VerlagNewYork,Inc., 2005. [136]A.RekaandBarabasi,Statisticalmechanicsofcomplexnetworks, Rev.Mod. Phys. ,vol.74,pp.47,June2002. [137]S.DorogovtsevandJ.Mendes,Evolutionofnetworks, AdvancesinPhysics vol.51,p.1079,2002. [138]D.J.WattsandS.H.Strogatz,Collectivedynamicsof'small-world'networks., Nature ,vol.393,pp.440,June1998. [139]J.Xu,S.Adabala,andJ.A.B.Fortes,Towardsautonomicvirtualapplications inthein-vigosystem,in Proc.SecondInternationalConferenceonAutonomic ComputingICAC2005 ,pp.15,2005. [140]N.R.Jennings,Anagent-basedapproachforbuildingcomplexsoftware systems, Commun.ACM ,vol.44,no.4,pp.35,2001. 175

PAGE 176

[141]A.Schaerf,Y.Shoham,andM.Tennenholtz,Adaptiveloadbalancing:A studyinmulti-agentlearning, JournalofArticialIntelligenceResearch ,vol.2, pp.475,1995. [142]A.Montresor,H.Meling,andO.Babaoglu,Messor:Load-balancingthrougha swarmofautonomousagents,in InProceedingsof1stWorkshoponAgentand Peer-to-PeerSystems ,pp.125,2002. [143]A.Andrzejak,A.Andrzejak,S.Graupner,S.Graupner,V.Kotov,V.Kotov,H.Trink, andH.Trinks,Algorithmsforself-organizationandadaptiveserviceplacementin dynamicdistributedsystems,tech.rep.,2002. [144]K.P.Birman,M.Hayden,O.Ozkasap,Z.Xiao,M.Budiu,andY.Minsky,Bimodal multicast, ACMTrans.Comput.Syst. ,vol.17,no.2,pp.41,1999. [145]I.T.Foster,Y.Zhao,I.Raicu,andS.Lu,Cloudcomputingandgridcomputing 360-degreecompared, CoRR ,vol.abs/0901.0131,2009. 176

PAGE 177

BIOGRAPHICALSKETCH JingXugrewupinJilin,abeautifulcityinNortheastChina.Attheageofnineteen, sheleftherhometowninpursuitofhighereducation,whichhascontinuedtilltoday. ThroughveyearsofstudyattheUniversityofScienceandTechnologyofChina,she earnedthebachelordegreefromtheDepartmentofAutomationwithhonor.Afterwards, shebeganherPh.D.studywithProf.Jos eFortesinACISlabattheUniversityof Florida,whereshehasbeendoingresearchintheareasofdistributed/gridcomputing, autonomiccomputing,andvirtualization.ShereceivedherPh.D.inelectricaland computerengineeringfromtheUniversityofFloridain2011.Currently,sheisanadjunct facultyintheSchoolofComputerandInformationSciences,FloridaInternational University,Miami. JinghasbeenmarriedtoMingsinceMay,2008.Theyhavetwowonderfulchildren, EileenandElwen. 177