Citation
Directed Test Generation for Hybrid Systems

Material Information

Title:
Directed Test Generation for Hybrid Systems
Creator:
Proch, Sudhi Ranjan
Publisher:
University of Florida
Publication Date:
Language:
English

Thesis/Dissertation Information

Degree:
Master's ( M.S.)
Degree Grantor:
University of Florida
Degree Disciplines:
Electrical and Computer Engineering
Committee Chair:
MISHRA,PRABHAT KUMAR
Committee Co-Chair:
MAGHARI,NIMA
Committee Members:
STITT,GREG M
Graduation Date:
5/3/2014

Subjects

Subjects / Keywords:
Centroids ( jstor )
Hybrid systems ( jstor )
Learning modalities ( jstor )
Mathematical variables ( jstor )
Modeling ( jstor )
Simulations ( jstor )
Thermostats ( jstor )
Trajectories ( jstor )
Tree growth ( jstor )
Velocity ( jstor )
clustering
hybrid-system
learning
model-checking
rapidly-exploring-random-tree
rrt
test-generation
validation

Notes

General Note:
Validation of hybrid systems is challenging due to the combination of both discrete and continuous dynamics. Simulation is the most widely used form of system validation using a combination of random and constrained-random tests. Directed tests are promising since orders-of-magnitude less number of directed tests can achieve same coverage goal as random simulation. Automated methods for development of directed tests are required to keep pace with the increasing design complexity. This thesis developed efficient directed test generation methods for hybrid systems. The test generation methods are based on two popular techniques used by current researchers namely, RRT (Rapidly Exploring Random Tree) and model checking. In contrast to current RRT methods that explore the state space in forward direction (from initial region to functional scenario), my method explores the state space in the reverse direction (from functional scenario to initial region). Experimental results demonstrate that reverse RRT technique is upto 33 times (average 10 times) faster than currently used forward RRT methods. I also developed a model checking based test generation method that addressed the tradeoff between accuracy of the result and test generation time for affine-hybrid systems. The reverse reachability based method produces a reasonably accurate testcase (within 18% error margin) in a fraction of time (15 times faster) normally required for generating a high accuracy testcase. To further reduce the test generation time, this thesis proposed a clustering and learning technique to improve test generation time involving similar functional scenarios. The learning based test generation method provides a significant reduction in test generation time (1.6 times on average) by exploiting similarity across test generation instances. The proposed approaches can significantly reduce the overall validation of hybrid systems.

Record Information

Source Institution:
UFRGP
Rights Management:
Copyright Proch, Sudhi Ranjan. Permission granted to the University of Florida to digitize, archive and distribute this item for non-profit research and educational purposes. Any reuse of this item in excess of fair use or other copyright exemptions requires permission of the copyright holder.
Embargo Date:
5/31/2016

Downloads

This item has the following downloads:


Full Text
xml version 1.0 encoding UTF-8
REPORT xmlns http:www.fcla.edudlsmddaitss xmlns:xsi http:www.w3.org2001XMLSchema-instance xsi:schemaLocation http:www.fcla.edudlsmddaitssdaitssReport.xsd
INGEST IEID EX6CB8BJG_5EOSU6 INGEST_TIME 2014-10-03T22:47:42Z PACKAGE UFE0046733_00001
AGREEMENT_INFO ACCOUNT UF PROJECT UFDC
FILES



PAGE 1

DIRECTEDTESTGENERATIONFORHYBRIDSYSTEMSBySUDHIRANJANPROCHATHESISPRESENTEDTOTHEGRADUATESCHOOLOFTHEUNIVERSITYOFFLORIDAINPARTIALFULFILLMENTOFTHEREQUIREMENTSFORTHEDEGREEOFMASTEROFSCIENCEUNIVERSITYOFFLORIDA2014

PAGE 2

c2014SudhiRanjanProch 2

PAGE 3

Idedicatethistomyparents 3

PAGE 4

ACKNOWLEDGMENTS MyrstthanksgoestomythesisadvisorDr.PrabhatMishra.Withouthissupport,constructivecriticismandencouragement,thisworkwouldnothavetakenthisshape.IalsothankmythesiscommitteemembersDr.GregoryStittandDr.NimaMaghariforprovidingvaluablefeedbackandreviewcommentstohelpimprovethiswork.Mysincerethanksgoestomylabmatesforhelpingmestaymotivatedandprovidingaworkculturewherethisworkwaspossible.Theloveandsupportofmyfamilyprovidesameaningtoallthiswork.Itisasourceofconstantmotivationinmylife.Myheartfeltthankshence,goestomyfamily. 4

PAGE 5

TABLEOFCONTENTS page ACKNOWLEDGMENTS .................................. 4 LISTOFTABLES ...................................... 7 LISTOFFIGURES ..................................... 8 ABSTRACT ......................................... 9 CHAPTER 1INTRODUCTION ................................... 11 1.1Simulation-basedValidationusingTestVectors ............... 11 1.2TestGenerationforHybridSystems ..................... 12 1.3ThesisContributionandOrganization .................... 13 2HYBRIDSYSTEMMODELANDRELATEDWORK ................ 15 2.1HybridSystemModel ............................. 15 2.2DirectedTestGenerationProblem ...................... 16 2.3RelatedWork .................................. 16 2.3.1ValidationusingModelChecking ................... 16 2.3.2ValidationusingRRT .......................... 17 3TESTGENERATIONUSINGRRT ......................... 19 3.1ReverseRRTTestGenerationMethod .................... 20 3.2AlgorithmImplementation ........................... 22 3.2.1GoalSampleGeneration ........................ 22 3.2.2DynamicBiasAdjustment ....................... 22 3.2.3NearestNodeSelection ........................ 23 3.3CaseStudies .................................. 24 3.3.1BouncingBall .............................. 24 3.3.2Thermostat ............................... 26 3.4ComparisonwithForwardRRTMethods ................... 28 4TESTGENERATIONUSINGMODELCHECKING ................ 29 4.1TestGenerationMethod ............................ 30 4.1.1TestGenerationforAfne-hybridSystems .............. 31 4.1.2TestGenerationforLinearSystems .................. 34 4.2CaseStudies .................................. 34 4.2.1BouncingBall .............................. 35 4.2.1.1Forwardreachabilitymethod ................ 35 4.2.1.2Reversereachabilitymethod ................ 36 4.2.2ThermostatSystem ........................... 41 5

PAGE 6

4.3ComparisonwithRRTBasedTestGeneration ................ 41 5TESTGENERATIONUSINGLEARNINGANDCLUSTERINGTECHNIQUES 45 5.1TestGenerationMethod ............................ 45 5.2AlgorithmImplementation ........................... 47 5.2.1TestScenarioClustering ........................ 47 5.2.2LearningBasedTestGeneration ................... 50 5.2.2.1Regioncreation ....................... 50 5.2.2.2Goalsamplegeneration ................... 50 5.2.2.3Randomgoaladjustment .................. 52 5.2.2.4Findnearesttreenode ................... 53 5.3CaseStudies .................................. 53 5.3.1BouncingBall .............................. 53 5.3.2ThermostatSystem ........................... 54 5.3.3ComparisonwithMethodswithoutLearning ............. 55 6CONCLUSION .................................... 57 REFERENCES ....................................... 59 BIOGRAPHICALSKETCH ................................ 61 6

PAGE 7

LISTOFTABLES Table page 3-1TestgenerationtimeandmemoryusagecomparisonforRRTmethods ..... 27 4-1Testgenerationtimeandaccuracydataforbouncingballsystem ........ 39 4-2TestgenerationtimecomparisonforRRTandmodelchecking ......... 44 5-1Testgenerationtimecomparisonbetweenlearningbased(lrn)andwithoutlearning(nolrn)methods ............................... 56 7

PAGE 8

LISTOFFIGURES Figure page 1-1Simulation-basedvalidationusingtestvectors ................... 12 1-2Differenttrajectoriesforabouncingballsystem .................. 13 3-1TreegrowthinforwardRRT(existingapproaches) ................ 19 3-2TreegrowthinreverseRRT(Algorithm 1 ) ..................... 21 3-3Probabilitydistributioncurvesfordifferentstandarddeviationsinaboundedregion[0,10]withmean2 .............................. 23 3-4Modelofabouncingballsystem .......................... 24 3-5Testcaseproducedforbouncingballsystem .................... 25 3-6Modelofathermostatsystem ............................ 26 3-7Testcaseproducedforthermostatsystem ..................... 27 4-1Forwardreachabilitygrowthinmodelchecking .................. 30 4-2Reversereachabilitygrowthinmodelchecking .................. 31 4-3Constraintsimplicationandgeometriccentroidmethod ............. 33 4-4PHAVermodelforbouncingballforwardreachabilityanalysis .......... 36 4-5PHAVermodelforbouncingballreversereachabilityanalysis .......... 36 4-6Testcaseforbouncingballwithpartitionsizex3=1 40 ............... 37 4-7Testcaseforbouncingballwithpartitionsizex3=1 100 ............... 38 4-8Testcaseforbouncingballwithpartitionsizex3=1 500 ............... 38 4-9HyTechmodelforthermostatsystem ........................ 42 4-10Testcaseconstructedfromthetraceofthermostatmodel ............ 42 5-1DirectedTestGenerationusingLearningandClustering ............. 46 5-2Randomdistributionofscenariosforclustering .................. 49 5-3Clusterdistributionhistogram ............................ 49 5-4Goalgenerationforlearningbasedtestgeneration ................ 52 5-5Bouncingballtestcaseswithlearningandnolearning .............. 54 5-6Thermostattestcaseswithlearningandnolearning ............... 55 8

PAGE 9

AbstractofThesisPresentedtotheGraduateSchooloftheUniversityofFloridainPartialFulllmentoftheRequirementsfortheDegreeofMasterofScienceDIRECTEDTESTGENERATIONFORHYBRIDSYSTEMSBySudhiRanjanProchMay2014Chair:PrabhatKumarMishraMajor:ElectricalandComputerEngineeringValidationofhybridsystemsischallengingduetothecombinationofbothdiscreteandcontinuousdynamics.Simulationisthemostwidelyusedformofsystemvalidationusingacombinationofrandomandconstrained-randomtests.Directedtestsarepromisingsinceorders-of-magnitudelessnumberofdirectedtestscanachievesamecoveragegoalasrandomsimulation.Automatedmethodsfordevelopmentofdirectedtestsarerequiredtokeeppacewiththeincreasingdesigncomplexity.Thisthesisdevelopedefcientdirectedtestgenerationmethodsforhybridsystems.Thetestgenerationmethodsarebasedontwopopulartechniquesusedbycurrentresearchersnamely,RRT(RapidlyExploringRandomTree)andmodelchecking.IncontrasttocurrentRRTmethodsthatexplorethestatespaceinforwarddirection(frominitialregiontofunctionalscenario),mymethodexploresthestatespaceinthereversedirection(fromfunctionalscenariotoinitialregion).ExperimentalresultsdemonstratethatreverseRRTtechniqueisupto33times(average10times)fasterthancurrentlyusedforwardRRTmethods.Ialsodevelopedamodelcheckingbasedtestgenerationmethodthataddressedthetradeoffbetweenaccuracyoftheresultandtestgenerationtimeforafne-hybridsystems.Thereversereachabilitybasedmethodproducesareasonablyaccuratetestcase(within18%errormargin)inafractionoftime(15timesfaster)normallyrequiredforgeneratingahighaccuracytestcase.Tofurtherreducethetestgenerationtime,thisthesisproposedaclusteringandlearningtechniqueto 9

PAGE 10

improvetestgenerationtimeinvolvingsimilarfunctionalscenarios.Thelearningbasedtestgenerationmethodprovidesasignicantreductionintestgenerationtime(1.6timesonaverage)byexploitingsimilarityacrosstestgenerationinstances.Theproposedapproachescansignicantlyreducetheoverallvalidationofhybridsystems. 10

PAGE 11

CHAPTER1INTRODUCTIONFunctionalvalidationisanessentialpartofanelectronicsystemdevelopmentcycle.Increasingdesigncomplexitynecessitatesthatvalidationtechniquesmustalsoevolvetoaddressthechallenge.Electronicdesignscanbebroadlyclassiedasdigital(discretedynamics)andanalog(continuousdynamics).Hybridsystemsareparticularlychallengingbecausetheycombineboththecontinuousanddiscretedynamics.Simulationusingasetoftestvectorsisanintegralpartofvalidatinghybridsystems.Developingautomatedmethodsofgeneratingtestvectorsisthuscriticalinaddressingthevalidationcomplexityofhybridsystems.Thischapterisorganizedasfollows.Section 1.1 introducestheconceptofsimulationbasedvalidationusingtestvectors.Section 1.2 providesmotivationforapplyingdirectedtestgenerationtechniquestohybridsystems.Finally,Section 1.3 highlightsthespecicthesiscontributionsandoutlinesthethesisorganization. 1.1Simulation-basedValidationusingTestVectorsValidationofasystemusingsimulationoftestvectorsisoneofthemostprevalenttechniquesusedduringelectronicsystemdevelopment.AsshowninFig. 1-1 ,atestgeneratorappliesaseriesoftestvectors(stimulus)onaDesignUnderTest(DUT).ApropertyandcoveragemonitorcheckstheresponseoftheDUTandensuresthatitmatchestheexpectedoutput.Testvectorgenerationcanbeclassiedintothreebroadcategories:random,constrained-randomanddirected.Randomtestgenerators,duetotheircapabilitytogenerateawidevarietyoffunctionalscenarios,areefcientinvericationofunknownerrors.Constrained-randomtestgenerationisanattempttosteeragenericrandomtestgeneratortowardsgeneratingtestvectorsthatarelikelytoactivateasetofimportantfunctionalscenarios.Dependingonthenatureofafunctionalscenario,constraintgenerationcanbecomplex.Moreover,duetoprobabilisticnatureoftheseconstraints,thegeneratedtestsmaynotactivatethetargetfunctional 11

PAGE 12

Figure1-1. Simulation-basedvalidationusingtestvectors scenarios.Adirectedtestgenerator,ontheotherhand,generatesonetestcasetotargetaspecicfunctionalscenario.Clearly,orders-of-magnitudelessnumberofdirectedtestscanachievethesamecoveragegoalasrandomorconstrained-randomtests,andtherefore,candrasticallyreducetheoverallsimulationtimeandvalidationeffort.However,directedtestgenerationismostlyperformedbyhumanintervention.Hand-writtentestsrequirelaboriousandtime-consumingeffortofvericationengineerswhohavedeepknowledgeofthedesignunderverication.Therefore,itisimportanttodevelopautomatedtechniquesforgenerationofdirectedtests. 1.2TestGenerationforHybridSystemsLetusconsiderasimplehybridsystem,abouncingball1.AsshowninFig. 1-2 ,theballcanbelaunchedwithdifferentvelocitiesandfromastartingpointofdifferentheight,resultingindifferentbouncetrajectories.Theballbouncesonimpactwithdifferentplanksontheground.Thegapsbetweenplanks(holes)mustbeavoidedduringtravel.Aninterestingfunctionalscenarioherecouldbethatofatrajectorywhereballonly 1Chapter 3 describesthebouncingballsystemindetail 12

PAGE 13

Figure1-2. Differenttrajectoriesforabouncingballsystem hitsplank2and3beforereachingtheotherside.Onecanimaginemanymoresuchfunctionalscenariosofincreasingcomplexitythatmightbeofinterest.Arandomtestgeneratorinthiscasewouldgeneratenumerousrandombouncetrajectoriesbeforediscoveringthedesirablefunctionality.Adirectedtestgeneratorontheotherhandwouldgenerateexactlyonetestcasetosatisfyonefunctionalscenario.Itshouldbeobviousthatthenumberofdirectedtestsrequiredtoreachacoveragegoalwouldbeseveralordersofmagnitudelessthanusingrandomtestvectors.OnecanalsovisualizeusingFig. 1-2 thatmanyofthebouncetrajectoriesarelikelytoexhibitconsiderablesimilarityduringtravel.Takingadvantageoflearningfromprevioustestcasegenerationcanfurtheraidadirectedtestgeneratortolowerthetestgenerationtimefornewtestcases.Whiledirectedtestgenerationisawellstudiedproblemfordigitaldesigns[ 1 ],incaseofhybridsystemsitismostlyperformedbyhumaninterventionthatiscumbersomeanderrorprone.Infact,itmaynotbepossibletodevelopadirectedtestmanuallyifthedesigniscomplexand/oritinvolvescomplexinteractions. 1.3ThesisContributionandOrganizationThisthesisdevelopedautomatedmethodsforgeneratingdirectedtestcasesforhybridsystemsusingtwodifferenttechniques.OneofthemethodsisbasedonRapidlyExploringRandomTree(RRT)algorithmfromtheroboticspathplanningdomainwhile 13

PAGE 14

theotheroneisbasedonhybridmodelcheckingalgorithm.Thisthesismadethreeimportantcontributions: DevelopmentofadirectedtestgenerationmethodusingRRTalgorithmthatemploystreeexplorationinreversedirection. Developmentofanaccuratedirectedtestgenerationmethodusingreversereachabilityanalysisofhybridmodelcheckingalgorithm. Developmentofaclusteringandlearningbasedtestgenerationmethodthatreducesthetestcasegenerationeffortfornewtestcasesbylearningfrompreviouslygeneratedtestcasesofacluster.Experimentalresultsusingeachmethoddemonstratedtheeffectivenessoftheproposedtestgenerationtechniques.ThisthesisalsoprovidesarelativecomparisonoftheRRTandmodelcheckingbasedtestgenerationmethodsintermsoftheirruntimeefciencyandaccuracyoftheresultsforseveralpopularhybridsystems.Thethesisisorganizedasfollows.Chapter 2 introducesthehybridsystemmodelanddescribesrelatedworkinthisarea.Chapter 3 describesthereverseRRTbasedtestgenerationmethod.Chapter 4 explainsthetestgenerationmethodusinghybridmodelcheckingalgorithm.Chapter 5 describesalearningandclusteringalgorithmthatallowsmytestgenerationmethodtoprovideadditionalreductionintestgenerationtimefornewtestcases.Finally,Chapter 6 concludesthethesis. 14

PAGE 15

CHAPTER2HYBRIDSYSTEMMODELANDRELATEDWORKModelingofhybridsystemplaysanimportantroletoenableautomatedgenerationofdirectedtests.Themodelforhybridsystemmustbeabletodescribeboththediscreteandcontinuousdynamicssimultaneously.Section 2.1 introducesthesystemmodelusedinthiswork.Section 2.2 usesthemodeltoformallystatetheproblemofdirectedtestgenerationforhybridsystems.Section 2.3 describesrelatedworkinthisareausingbothRRTandmodelcheckingbasedmethods. 2.1HybridSystemModelThehybridsystemmodelMisdenedbythetuple(X,Q,U,f,Inv,Iinit,E,G,R)where: X:nitesetofboundedcontinuousvariablesRn Q:nitesetofdiscretestatesfq1,q2,q3....qngZ U:nitesetofboundedcontrolinputsRm f=ffqjq2Qgsuchthatfqisavectoreldthatdenesthetimederivativeofthecontinuousvariablex2Xgivenby_x=fq(x,u).Itisassumedthatu2Uispiecewisecontinuous.Additionally,allfqareassumedtobeLipschitzcontinuous[ 2 ]andintegrableinreversetime. Inv:assignstoeachqaninvariantset. Iinit:istheinitialregionofinterestforthesystem. EQQ:isasetofedgesdenotingthediscretetransitionofthesystem. G:isasetofguardconditionsassignedtoeachedgee=(q,q0)2E.Atransitionistakenwhentheguardconditionissatised. R:(x,q)!(x0,q0)isaresetmapdenedforeachedgee=(q,q0)2E.Itdeneshowxchangeswhenthattransitionistaken.Stateofthesystemisdenedby(x,q)withtheinitialstatebeing(xinit,qinit)2Iinit.ItisassumedthatresetmapRcanbeinvertedsuchthatthereisacorrespondingrelationR0foreachedgee=(q,q0)2EwhichisinverseofR.R0denestransitionfromx0 15

PAGE 16

toxforthecausalityreversedsystemsuchthatnormalsystemoperationwillcausetransformation(x,q)!(x0,q0)withresetmapR. 2.2DirectedTestGenerationProblemGivenamodelM=(X,Q,U,f,Inv,Iinit,E,G,R)forhybridsystemandaspecicfunctionalscenarioS,thegoalistocomeupwithonepossiblesetofcontrolinputs(u1,u2,...,un)andastartlocationI02Iinitsuchthatthesetofinputsappliedoveranitetimeintervalf0!TgguidethesystemfromI0toastatewherefunctionalscenarioSissatised.AtestcaseforfunctionalscenarioSwouldthenconsistoftheset(I0,u1,u2,...,un).Forsystemswithoutanycontrolinputs,thetestcaseconsistsonlyofthestartlocationI0suchthatfunctionalscenarioSisactivatedifthesystemevolvesunforcedfromI0forsomenitetimeintervalT.ItisassumedthatthesystemstartregionIinitandthefunctionalscenarioScanbespeciedusinginequalityoperatorsf,gandlogicaloperators(^,_)overthesystemvariablesX.AtestcaseisconsideredvalidonlyifitoriginatesfromIinitanditactivatesthefunctionalscenario. 2.3RelatedWorkRelatedworkcanbeclassiedintwomaincategories.Oneoftheapproachesaddressfunctionalvalidationofhybridsystemsbydoingstatereachabilityanalysisusingmodelcheckingalgorithm.TheotherclassoftoolsemployRRTalgorithmbasedrandomsearchtechniques. 2.3.1ValidationusingModelCheckingHybridmodelcheckingalgorithmspartitionthestatespaceofthesystemintonitesectionsandattempttodoreachabilityanalysisforaspecicsystemstate(targetstate).HyTech[ 3 ]isoneoftheearliestmodelcheckingtoolsforhybridsystem.Thistoolcanperformreachabilityanalysisbothinforwardandbackwarddirection.Thesystemregionofinterest(targetstate)canbespeciedinaCTL(ComputationalTreeLogic)likemanner.Ithastheabilitytonotonlydoyes/noanalysiswithrespecttoreachabilityof 16

PAGE 17

states,butalsogeneratesatracefromtheinitialstatetothetargetstatewhichbecomesacounterexample.However,thistoolworksonlyforlinearhybridsystemswithconstantboundsontheratechangeofvariables.PHAVer[ 4 ]addressedtheshortcomingsofHyTechbysupportingafnedynamicswheretheratechangeofvariablescanbedependentonothersystemvariables(dynamicsoftheform_x=x+A).Whiletoolsforlinearhybridsystems(likeHyTech)cangiveexactanswers,over-approximationofthereachablesetsmustbemadeforsystemswithafnedynamics.Incontrasttoastaticstatepartitioningapproachofsometools[ 5 ],PHAVerperformsdynamicpartitioningofthestatespacewhiledoingreachabilityanalysis.Thetoolallowsbothforwardandbackwardreachabilityanalysis.Theaccuracyoftheresultscanbeincreasedbymakingnegrainedpartitions.However,theincreaseinaccuracycomesatthecostofincreasingthetestgenerationtimebyseveralordersofmagnitude.PHAVerdoesnotsupporttracegenerationforthereachablestates.Hence,itcannotgenerateacounterexampleforsystemswithcontrolvariables.InChapter 4 ,IdescribeamethodofgeneratinganaccuratetestcaseusingPHAVerwhilestillkeepingthetestgenerationtimewithinreasonablebounds.CheckMate[ 6 ]isanotherpopulartoolthatperformsmodelcheckingbasedreachabilityanalysisusingCTLlikeformulas.Whilethistoolsupportsawiderangeofsystemtypes(linearandnonlinear),itcouldnotbeappliedonourhybridsystemmodels-generatessomeerrormessagesandterminates.OthertoolsinthisclassincludeSpaceEx[ 7 ]andd/dt[ 8 ].Bothofthesetoolsarecapableofsupportinglinearandnonlineardynamics.However,thesetoolslackthecapabilitytodotracegenerationandreversereachabilityanalysis.TheselimitationsmakethemunsuitablefortestgenerationasexplainedinChapter 4 2.3.2ValidationusingRRTTheotherclassofvalidationtoolsarebasedonrandomsearchtechniques.RRT(RapidlyExploringRandomTree)[ 9 ]isoneofthepopularalgorithmsinthisclass, 17

PAGE 18

whichisusedbymanyresearchers[ 10 13 ].Both[ 11 ]and[ 12 ]useRRTalgorithmthatstartsfromapointintheinitialregionandgrowsintheforwarddirectiontowardsatargetgoalregion.In[ 11 ],acoveragebasedcriteriaisusedtobiasthegoalregionsampling.In[ 12 ],theresultsofalearningphase,basedonstablesystemstates,isusedforgoalsamplebiasing.Whileallothermethodsusestaticgoalbiasgenerationschemesinforwarddirectionwiththeprimaryobjectiveofstatereachabilityanalysis,[ 13 ]hasatechniquebasedondynamicadjustmentofgoalbiasthatisprimarilydirectedtowardstestgenerationproblem.Thetechniqueof[ 13 ]however,isstillbasedonforwardRRTalgorithm.AsdescribedinChapter 3 ,forwardRRTalgorithms,evenwiththeirgoalbiasingheuristics,becomeinefcientwhilesearchingforaspecicsystemstateastheyrelyonrandomsearchinthewholestatespace.In[ 10 ],abidirectionalRRTtree(forwardandbackward)growthalgorithmisusedbuttheirfocusishybridsystemanalysisandnottestcasegeneration.NoneoftheexistingtechniquesemployareverseRRTgrowthalgorithm(startingfromthefunctionalscenarioregion)forthepurposeoftestgeneration.MymethodinChapter 3 demonstratesthatreverseRRTbasedtestgenerationtechniquesperformsignicantlybetterthantheforwardRRTapproaches.Inaddition,noneoftheexistingmethodsattempttolearnfrompriorgeneratedtestcasessothattestcasesfornewfunctionalscenarioscanbegeneratedmoreefciently.InChapter 5 ,Idescribeaclusteringandlearningtechniquethatisintendedtospeedupthetestgenerationtimefornewfunctionalscenariosbylearningfromsystemtrajectoriesthathavealreadybeenexploredforpriortestcases. 18

PAGE 19

CHAPTER3TESTGENERATIONUSINGRRTThischapterdescribesthedirectedtestgenerationtechniquebasedonRapidlyExploringRandomTree(RRT)algorithm.RRTalgorithmiswidelyusedinroboticspathplanningdomainwheretheyhaveprovedtobeveryefcientinrandomlysearchingforpossiblesystemtrajectoriesconnectingtwodesirableregionsinstatespace.RRTalgorithmwhenemployedfortestcasegenerationofhybridsystems,exploresthepossiblestatetrajectoriesbetweentheinitialsystemstateandthestateofinterest(goalstate).Ifsuchaconnectingtrajectoryisfoundduringexploration,atestcasecanbeconstructedthatleadsthesystemfromaknowninitialstatetothegoalstate.Thischapterisorganizedasfollows.Section 3.1 providesanoverviewofreverseRRTbasedtestgenerationmethodandassociatedalgorithm.Section 3.2 providesimplementationdetailsabouteachofthemaincomponentsofthealgorithm.Section 3.3 demonstratestheapplicabilityofthemethodviacasestudiesoftwohybridsystems(bouncingballandthermostatsystem).Finally,Section 3.4 providesacomparisonbetweenreverseRRTandforwardRRTmethods. Figure3-1. TreegrowthinforwardRRT(existingapproaches) 19

PAGE 20

Input: i)Thedesignmodel,Mii)AfunctionalscenarioSintheformanXnbn Output: i)AninitialregionI02Iinitii)PathfromI0toSaspartoftreestructure)]TJ ET BT /F6 11.955 Tf 30.06 -69.51 Td[(foriisfrom1toMax iterdo1 I0=fg,)-278(=fg;2 AddNodeUDF(S)to)]TJ /F1 11.955 Tf 6.78 0 Td[(;3 forjisfrom1toMax nodesdo4 ifIinit=2)]TJ /F6 11.955 Tf 10.09 0 Td[(then5 Xrand=Goal Sample(P(x,init,));6 Xnear=Nearest Node(Xrand,)]TJ /F1 11.955 Tf 12.26 0 Td[();7 Xsim=fRt0f(xnear,UDF(u))g;8 Xnew=Nearest Node(Xrand,Xsim);9 ifXnew=2)]TJ /F6 11.955 Tf 10.1 0 Td[(then10 addXnewto)]TJ /F1 11.955 Tf 6.78 0 Td[(;11 end12 else13 (())=Dynamic Bias Adjust();14 end15 end16 else17 I0=Iinit2)]TJ /F1 11.955 Tf 6.77 0 Td[(;18 break;19 end20 end21 ifIinit2)]TJ /F6 11.955 Tf 10.09 0 Td[(then22 break;23 end24 end25 returnI0,)]TJ /F1 11.955 Tf 12.26 0 Td[(;26 Algorithm1:DirectedtestgenerationUsingReverseRRTmethod 3.1ReverseRRTTestGenerationMethodDirectedtestgenerationmethodviareverseRRTtreegrowthisshowninAlgorithm 1 .Thealgorithmconstructsatreestructure()]TJ /F1 11.955 Tf 6.77 0 Td[()consistingofnodesandedges.Eachnodestoresthesystemstatewhileedgesdenotethecontrolinputvaluesthatleadonesystemstatetotheother.ForasystemwithstatevariablesX1andX2,Fig. 3-1 andFig. 3-2 showsearchtreegrowthinforward(existingapproaches)andreverseRRT(Algorithm 1 ),respectively.IncontrasttoforwardRRTalgorithm(Fig. 3-1 ),thatsetsa 20

PAGE 21

Figure3-2. TreegrowthinreverseRRT(Algorithm 1 ) specicinitialsystemstateastherootofthetree(startpoint),Algorithm 1 (Fig. 3-2 )setstherootofthetreeinthesystemstatewheretargetfunctionalscenario(denotedbyS)issatised.Thealgorithmthentriestogrowthetreetowardsaregionofinterest.ThisregionofinterestincaseofAlgorithm 1 istheinitialregionIinit.AteachstepoftheiterationanewnodeXnewisaddedtothetree,startingfromanearestnodeXnearwithrespecttoarandomgoalregionpointXrand.XrandisselectedasperadistributionfunctionP(x,init,)whoseaimistobiasthetreegrowthtowardstheregionofinterest.StartingfromthestateXnear,systemissimulatedforashortdurationbyvariationofcontrolinputsU,resultinginaset(Xsim)ofintermediatenodes.XnewisthenselectedfromXsimbymeansofametricfunctionthatgrowsthetreetowardsXrand.UDFasmentionedinAlgorithm 1 referstotheuniformrandomdistributionfunction.ItshouldbenotedthatwhenAlgorithm 1 growstheRRTinreverse,ateverystepitistryingtoexploreoneoutofmanypossibleconditionsthatcanmakepresentstatereachablebyvariationofcontrolinputs.AhybridsystemthatiscomposedofdiscretetransitionsandLipschitzcontinuousfunctionisexpectedtobecompletelyreversibleinbothforwardandreversedirections.Thus,thistestgenerationalgorithmremainsunaffectedbythefactthattherecouldbemultiplewaysofreachingafunctionalscenariofromaninitial 21

PAGE 22

region,asitisonlyrequiredtoexploreoneofthemanywaysinwhichsuchapathcanbetraversed. 3.2AlgorithmImplementationFunctionsGoal Sample(),Dynamic Bias Adjust()andNearest Node()arethethreemajorcomponentsofthereverseRRTAlgorithm 1 .Next,wedescribethesecomponentsindetail. 3.2.1GoalSampleGenerationGoalsamplegenerationschemeofAlgorithm 1 (functionGoal Sample)attemptstobiasthetreegrowthsothatitquicklyreachestheinitialregion(Iinit)ofthesystem.Thefunctionusesboundednormaldistributionfunctionforgoalgeneration.ThespreadofthedistributionfunctioncanbedynamicallyadjustedtoresultinrelativelymoreuniformdistributionasexplainedinSection 3.2.2 .ThegoaldistributionfunctionPforeachsystemvariablexinthiscaseisgivenbyP=8>><>>:N(x,,)+Cnorm,blxbr0otherwiseN(x,,)isthenormaldistributionfunctionwithmeanandstandarddeviationgivenbyfunction().Theadaptivefunction()controlsthestandarddeviationofthespreadfromarangeofuserprovidedvalues.Themeanofavariablespeciedbyinequalitiesblxbristakenasx=bl+br 2.Cnormisanormalizationfactorthatisaddedtomaketheareaunderthedistributionbecomeunityfortheboundedinterval.Fig. 3-3 showstheplotofP(x,,)fordifferentstandarddeviationvalueswithaparticularmeaninaboundedregionofavariable. 3.2.2DynamicBiasAdjustmentDuringtreegrowthphase(lines10)]TJ /F3 11.955 Tf 12.62 0 Td[(15inAlgorithm 1 ),aroutinekeepstrackofinstanceswhengeneratedgoalfailstoextendthetree.ReferringtoFig. 3-2 ,treegrowthisunsuccessfulwhenselectednodeXnewisalreadycontainedinthetree.Dynamicbias 22

PAGE 23

Figure3-3. Probabilitydistributioncurvesfordifferentstandarddeviationsinaboundedregion[0,10]withmean2 adjustment(functionDynamic Bias Adjust)helpsinthiscasetosteerthetreetowardsthedirectionwhereitkeepsgrowing.AsdescribedinSection 3.2.1 anddepictedinFig. 3-3 ,goalsamplegenerationfortestcasesiscontrolledbytheboundednormaldistributionfunctionP(x,,).Thedynamicbiasisadjustedbythestandarddeviationfunction()sothatitbecomesuniformincaseofundesirabletreegrowthdirection.Theschemeofdynamicallyadjustingthespreadofgoalsamplesisareasonablecompromisebetweenfasterconvergenceandavoidinglocalminima. 3.2.3NearestNodeSelectionFunctionNearest Node(),showninAlgorithm 1 ,isimplementedbyusingametricfunction.Toenableguidedtreegrowth,thisfunctionprovidesameasureofdistanceforatreenodewithrespecttothegoalregion(Iinit).Euclideannormisusedasthemetricfunction.TheEuclideannormfordistancebetweentwondimensionalvectors~xand~yisgivenby(~x,~y)=(x1)]TJ /F4 11.955 Tf 11.95 0 Td[(y1)2+(x2)]TJ /F4 11.955 Tf 11.95 0 Td[(y2)2+....(xn)]TJ /F4 11.955 Tf 11.95 0 Td[(yn)21 2 23

PAGE 24

Forsituations,whenthegeneratedtestcasedoesnotmeettherequirementsofinitialregionandhenceisnotavalidtestcase,thefunctionalscenarioregionisexploredforauniformlyrandomdifferentstartlocation. 3.3CaseStudiesThissectiondemonstratestheapplicabilityoftheproposedtestgenerationapproachthroughcasestudiesoftwodifferenthybridsystems,namelybouncingballandthermostat. Figure3-4. Modelofabouncingballsystem 3.3.1BouncingBallThesystemshowninFig. 3-4 modelsabouncingballsystemsimilartotheoneshowninFig. 1-2 .Thestatevariables(x1,x2,x3)denotetheverticalposition,horizontalpositionandthevelocityoftheball.Thesystemhasasinglediscretestate(q=1)labeled\bounce.Asitisanunforcedsystem,Uisanemptyset.DynamicsofthesystemvariablesareshowninFig. 3-4 asequationsfor_x1,_x2,and_x3.Edgee1hasguardconditiong1:(x1=0^x3<0).Itindicatesthatthetransitionhappenswhentheballhitstheoor(x1=0)withanegativevelocity.Resetmapr1indicatesthatthevelocityoftheballreversesandreducestohalfthepreviousvalueontransitione1.Considergeneratingatestcaseforthissystemwhenballhitssomespecicregion(planksection)ontheoorwithcertainvelocitysothattheballbouncestowardsaspecicsectionontheoppositewall.Thisscenarioisstatedas: 24

PAGE 25

\Startingfromaheightrangeof[0,1.35]andvelocityrangeof[)]TJ /F3 11.955 Tf 9.29 0 Td[(2.2,1.1]generateatestcasetohitoorsection[3.9,4.1]suchthatthevelocityoftheballwhilehittingthisregionisintherange[)]TJ /F3 11.955 Tf 9.3 0 Td[(0.4,)]TJ /F3 11.955 Tf 9.3 0 Td[(0.2].Functionalscenarioregion(S)becomesthestartregionandisformulatedas:S:(0x10)^(3.9x24.1)^()]TJ /F3 11.955 Tf 9.3 0 Td[(0.4x3)]TJ /F3 11.955 Tf 21.92 0 Td[(0.2)^(q=1)Initialregion(Iinit),towardswhichtreegrows,isformulatedas:Iinit:(0x11.35)^(0x20)^()]TJ /F3 11.955 Tf 9.29 0 Td[(2.2x31.1)^(q=1)ThetestcasegeneratedbyAlgorithm 1 isshownhighlightedinFig. 3-5 .Thealgorithmidentiespoint[x1,x2,x3,q]=[1.0313,0,0.3465,1]withinIinitthatwillsatisfythescenario.AveragememoryusageandtestcasegenerationtimeareshowninTable 3-1 formultipleattemptstogeneratethistestcase.Fig. 3-5 alsohighlightstherandomlyexploredstatespaceusedforgoalselectionduringtestcasegeneration.Itshouldbe Figure3-5. Testcaseproducedforbouncingballsystem notedthatsincethisisasystemwithnocontrolinputs(unforced),thetreegrowsinapredictablemannerdependingonthestartconditionsofthesystem.AlthoughthisexampleissimpleforexplorationusingRRTalgorithm,itdemonstratestheapplicability 25

PAGE 26

ofAlgorithm 1 tosuchasystem.Arelativelymorecomplex,controlledsystem,isdescribedinnextsection. 3.3.2ThermostatConsideranexampleofathermostatsystemwithmodelasshowninFig. 3-6 .It Figure3-6. Modelofathermostatsystem hastwodiscretestatesnamely,\onand\off.Systemvariablesarefx1,x2gwherex1isthesystemtemperatureandx2isthetotalamountoftimesystemisinoperation.Theheatingandcoolingratesin\on(q=1)and\off(q=2)statesaregivenbyU=fuon,uog,whereuon=[2,4]anduo=[)]TJ /F3 11.955 Tf 9.3 0 Td[(3,)]TJ /F3 11.955 Tf 9.3 0 Td[(1].Systemtransitionsfrom\onstateto\offstatewheneversystemtemperaturebecomesequaltoupperboundof3.Similarly,transitionfrom\offto\onstatehappenswhensystemtemperatureequalsalowerboundof1.Noneofthevariableschangevalueswhentransitionistaken.Letusconsiderthetestgenerationforascenariowhen\Startsystemtemperatureisintherange[1,2]andtemperaturereaches2.11intheonstateafteraruntimeof5.7timeunits.Functionalscenarioregion(S)andinitialregion(Iinit)areformulatedinthiscasewiththefollowinginequalities:S:(2.11x12.11)^(5.7x25.7)^(q=1)Iinit:(1x12)^(0x20)^(q=1_2)ApplicationofAlgorithm 1 generatesapoint[1.41,0,1]withinIinitasthetestcasestartpointinthiscase.Italsoreturnsthetreestructurefromwhichheating/coolingrates 26

PAGE 27

Figure3-7. Testcaseproducedforthermostatsystem andcorrespondingtimescanbeextracted.TheextractedtestcaseisshownhighlightedinFig. 3-7 .AreaofthestatespaceexploredforgeneratingthegoalpointsisalsohighlightedinFig. 3-7 .AveragememoryusageandtestcasegenerationtimearelistedinTable 3-1 formultipleattemptstogeneratethistestcase. Table3-1. TestgenerationtimeandmemoryusagecomparisonforRRTmethods SystemTestTestGenerationtime(sec)MemoryUsed(MByte) Fwd.RRTRev.RRTChg.Fwd.RRTRev.RRTChg.[ 13 ](Alg 1 )(x)[ 13 ](Alg 1 )(%) bouncing1926.757.216.2512.1509.80.5ball2152.547.33.2510.2509.80.13>1Hr?52.7)]TJ /F17 7.97 Tf 9.3 4.34 Td[(?>1Hr?510.1)]TJ /F17 7.97 Tf 9.3 4.34 Td[(?4123.355.52.2509.9509.70.05>1Hr?48.6)]TJ /F17 7.97 Tf 9.3 4.34 Td[(?>1Hr?509.8)]TJ /F17 7.97 Tf 9.3 4.34 Td[(?Avg.400.853.37.5510.8509.80.2 thermostat1587.317.832.9516.7509.81.32>1Hr?53.3)]TJ /F17 7.97 Tf 9.3 4.34 Td[(?>1Hr?510.9)]TJ /F17 7.97 Tf 9.3 4.34 Td[(?3>1Hr?27.3)]TJ /F17 7.97 Tf 9.3 4.33 Td[(?>1Hr?510.1)]TJ /F17 7.97 Tf 9.3 4.33 Td[(?44317.8428.010.1541.3517.24.45363.756.76.4515.6511.00.9Avg.1756.3167.510.5524.5512.72.3 Note:`?'impliesforwardRRTfailed 27

PAGE 28

3.4ComparisonwithForwardRRTMethodsTable 3-1 providesacomparisonoftestgenerationtimeandmemoryusageofAlgorithm 1 withmyimplementationofexistingforwardRRTalgorithmasdescribedin[ 13 ].FivetestcasesweregeneratedforeachofthesystempresentedinSection 3.3 .ResultsarelistedinTable 3-1 .Eachalgorithmwasallowedtorunforamaximumtimeofapprox.1houronalinux2.4GHzmachine.ItisclearlyvisiblefromtheresultsthatreverseRRT(Algorithm 1 )isupto33timesfaster(10timesonaverage)andrequireslessmemory(upto4%,2%onaverage)comparedtoforwardRRT.Itmustbehighlightedthatinmanycases,theforwardRRTalgorithmdidnotndtherequiredtestcaseevenafterrunningforthemaximumallowedtime.Thesecases,highlightedwith`?'inTable 3-1 ,arenotconsideredforimprovementcomputations.Inothercases,forwardRRTgeneratedanapproximatetestcasewhichfailedtoactivatethefunctionalscenarioinreality.TheseresultsareexpectedbecauseAlgorithm 1 startsfromthefunctionalscenarioregionandgrowstowardsawideinitialregion.AwideinitialregionhelpsquickconvergenceofrandomsearchforreverseRRTalgorithmbecausendinganypointintheinitialregionduringrandomsearchwouldresultinavalidtestcasethatleadstotargetfunctionalscenario.ForwardRRTalgorithmontheotherhandfacestwomajorchallenges.First,ithastotryalargenumbernumberoftrajectoriessincetheinitialregionistypicallylargeandeachpointintheinitialregioncanbeapotentialstartingpointofatrajectory.Also,ittriestorandomlysearchthetargetfunctionalscenariointhecompletesystemspaceandtakestimetoconverge(targetfunctionalscenarioistypicallyasmallregionorapointinalargesearchspace),oftensettlingforanapproximatelyclosepoint. 28

PAGE 29

CHAPTER4TESTGENERATIONUSINGMODELCHECKINGModelcheckingisoneofthemostpromisingformalvericationtechniquesforvalidationofhybridsystemsbasedonsymbolicanalysisofthesystemstatespace.ThisisincontrasttoRRTalgorithmbasedmethods(Chapter 3 )whichexplorethestatespacethroughsimulation.Designvalidationmethodsbasedonsymbolicmodelcheckingarewellestablishedinthediscretedomain[ 14 ]wheretheyprovidearobustalternativetosimulationbasedvalidationmethods.Symbolicmodelcheckinghoweveronlyworksfornitestatespacewherestatesareconnectedbyatransitionrelation.Whilethestatespaceforapurediscretesystemisnite,thestatespaceofahybridsystemisinniteifthereisnolimitontheprecisionofcontinuousvariablevalues.Statespaceabstractionsarehencerequiredtomakeitnitebeforestandardmodelcheckingalgorithmscanbeapplied.Forhybridsystems,thetransitionrelationbetweenstatesisestablishedbysimulationofafewselectedpointsinastate.Modelcheckingmethodsforhybridsystemsarenotguaranteedtoterminatetheoretically,butformostpracticalproblemsareabletogeneratetherequiredresults.Foradetailedunderstandingofthetheoryofmodelcheckingforhybridsystemsonecanreferto[ 15 ]and[ 16 ].Thischapterintroduceshowhybridmodelcheckingtoolscanbeusedforthepurposeofdirectedtestgeneration.Thestandardforwardreachabilityalgorithmofthetoolshowever,failstogenerateavalidtestcaseinmanyscenarios,especiallywhenthesystemdynamicsarenotlinear.Section 4.1 describesatestgenerationmethodthataddressesthelimitationsofforwardreachabilitymethodbyemployingreversereachabilityanalysisstartingfromthefunctionalscenarioregion.ExperimentalresultsinSection 4.2 demonstratetheeffectivenessofthemethodthroughcasestudiesofbouncingballandthermostatsystem.Finally,Section 4.3 comparesthereversemodelcheckingbasedtestgenerationmethodwiththereverseRRTmethodofChapter 3 29

PAGE 30

4.1TestGenerationMethodModelcheckingtoolsforhybridsystemattempttoexploreallpossiblesystemtrajectoriesstartingfromasetofinitialstates.Fig. 4-1 demonstratesthisconceptfora2variablesystem(x1andx2).Startingwithagivensetofinitialstatesatstep0,thetoolcomputesasetofallpossiblenextstatesthatcanbereachedinstep1.Thisprocessrepeatstillatargetstateisfoundreachable.Thetransitionrelationbetweenstatesateachstepisbasedonthecontinuousdynamicsofstatevariables(equationsfor_x1and_x2inthiscase).Reachabilityanalysisprocessintheforwarddirection(initialstatestotargetstate)asshowninFig. 4-1 canbeusedforthepurposeofdirectedtestgeneration.Thefunctionalscenariobecomesthetargetstateforwhichthereachabilityanalysisisperformed.Iffoundtobereachable,atestcasecanthenbeconstructedbytracingthepathfrominitialstatestotargetstate.ItisevidentfromFig. 4-1 thatwhile Figure4-1. Forwardreachabilitygrowthinmodelchecking explorationofallpossiblesystemtrajectoriesprovidesadenitiveanswerintermsofreachabilityofthetargetstate,italsocausesthecomputationcomplexitytogrowquicklywitheachexplorationstep.ExperimentalresultsofSection 4.2 conrmthatinmanycases,especiallywhenstatedynamicsarenotlinear,forwardreachabilityanalysismethodfailstondatestcaseinreasonabletimeevenwithamoderatelevelofaccuracyrequirement. 30

PAGE 31

Reversereachabilitybasedtestgenerationmethodaddressestheshortcomingsoftheforwardreachabilitymethodbystartingthestateexplorationfromthefunctionalscenarioregionandtryingtoreachtowardstheinitialregion.Atypicaltestgenerationscenarioinvolvesalargeinitialregionwhereasthetargetfunctionalscenarioisasmallregion(orapoint).Thereversereachabilityisexpectedtoperformbetterthanforwardreachabilityfortworeasons.First,itiseasierforthereversereachabilitytoreachoneofthestatesintheinitialregion(typicallylarge)thantheforwardreachabilityalgorithmtryingtoreachasmalltarget.Moreover,reversereachabilitycanstartfromonepoint(functionalscenario)whereastheforwardreachabilityhastotryexplorationfrommultiplepointswithinthelargeinitialregion.Fig. 4-2 providesavisualizationofthisprocess.Sections 4.1.1 and 4.1.2 providedetailsofthemethodforbothafne-hybridsystemsandlinearhybridsystemsrespectively.Inaddition,Section 4.1.1 alsodiscussesamethodthatprovidessignicantreductionintestgenerationtimeforafne-hybridsystemswhileincurringminimallossinaccuracyoftheresult. Figure4-2. Reversereachabilitygrowthinmodelchecking 4.1.1TestGenerationforAfne-hybridSystemsEachpartitionedstateofahybridsystemismodeledasaconvexareaboundedbylinearconstraints.Inordertocontainthecomputationalcomplexityforafne-hybridsystems(likebouncingballofSection 4.2.1 )constraintsimplicationisrequiredateachstepofthestateexploration(Fig. 4-1 ).Thissimplicationprocesshowever,also 31

PAGE 32

causesanover-approximationerrorthatgrowswitheachexplorationstep.Inordertocontainthiserror,nerstatepartitionscanbeused.However,thiscomesatthecostofincreasingthetotalnumberofpossiblesystemstatesandtranslatestoverylongtestgenerationtime.Mytestgenerationmethodaddressesthislimitationbyemployingreachabilityanalysisinreversedirection.AsshowninFig. 4-2 ,auniformlyrandomstatepointinthefunctionalscenarioregionbecomesthestartpointforstateexploration.Sincetheexplorationstartsfromaspecicsystempointinsteadofaboundedregion,statepartitiongranularityhasnoeffectonthenumberofpossiblestartstates,asaspecicstatepointisthesmallestpossiblesizethatcanbespecied.Reducingthestartstateinthismethodtoasinglestatepointversusaboundedregioninforwardreachabilitymethodensuresthatthereversereachabilityanalysisconvergesinseveralordersofmagnitudelesstimeevenwithhighaccuracysettings.Theinitialregionofthesystemnowbecomesthetargetofreachabilityanalysis.Modelcheckingtoolsinthiscasereturnalistofconvexpolyhedrastructuresthatareattheintersectionofcomputedreachablestatesandtheinitialregion.Thereturnedsolutionisshownas\initialregioninFig. 4-2 .Applicationofreversereachabilityanalysishowever,doesnotaddressthesensitivityofresultaccuracytothepartitiongranularity.InFig. 4-2 ,theaccuracyoftheanalysisdeterminesthesizeofintersectionregionlabelledas\initialregion.Experimentalresultsconrm(SeeFigs. 4-6 4-7 and 4-8 )thathigheraccuracyofresultrequiresanepartitiongranularityeveninreversereachabilitycase.Thismeansthatifthepartitionsizeisnotsmallenoughthenselectinganyrandompolyhedrastructurefromthelistof\initialregion,returnedbyreversereachabilitymethod,doesnotguaranteereachingfunctionalscenarioregioninreality(falsepositiveresult).Smallerpartitionsizesandhencehighaccuracyoftheresulttranslatestoverylongtestgenerationtime.Totradeoffbetweentestgenerationtimeandaccuracyoftheresult,Iproposecreatingaunionofallthereturnedpolyhedralstructuresin\initialregionand 32

PAGE 33

simplifyingtheregionconstraints.Fig. 4-3 showstheconceptofconstraintsimplicationprocess.TwoconvexpolyhedralstructuresR1andR2areuniedbyretainingonlytheextremevaluesoftheprojectionsinthedirectionofstatevariablesx1andx2.Theresultisarectangularareaspeciedbyequations:(x1lx1x1u)^(x2lx2x2u).Theedgesoftherectangularregionarespeciedby(x1u,x1l)and(x2u,x2l),which Figure4-3. Constraintsimplicationandgeometriccentroidmethod denethelengthoftherectangularregioninthedirectionofsystemvariablesx1andx2,respectively.Thegeometriccentroidoftheresultantsimpliedstructureistakenasananswerinthismethod.Geometriccentroidcoordinatescanbecalculatedbytakingthecenter-pointoftheedgelengthsalongthedirectionofsystemvariables.Section 4.2.1.2 describesthegeometriccentroidcalculationforthebouncingballsystem.ItisevidentfromtheexperimentalresultsofSection 4.2 thatthismethodprovidesamoderatelossofaccuracyoftheresultbutresultsinadrasticreductioninthetestgenerationtime.Byselectingarandomstartpointinfunctionalscenarioregionthatissufcientlyawayfromtheregionboundaries,themoderatelossofaccuracyofthismethodcanbecompensatedandresultsinanaccuratetestcase. 33

PAGE 34

4.1.2TestGenerationforLinearSystemsForlinearhybridsystems,themodelcheckingtoolsarecapableofprovidingexactsolutionasnoover-approximationofthereachablestatesneedstobedone.Thestatespacemustbepartitionedbeforestandardmodelcheckingalgorithmscouldbeapplied.ReferringtoFigs. 4-1 and 4-2 ,itcanbeassertedthatthetestgenerationmethodusingreversereachabilityisexpectedtoreducetestgenerationtimeforlinearhybridsystems.Thisisbecausethereversereachabilitystartsfromaspecicsystemstatepointinthefunctionalscenarioregionandattemptstoreachoneofthefewinitialstateswhichisexpectedtotakelesstimethanndingaspecicsystemstateoutofmanypossiblestatesintheforwardreachabilityanalysis.Inthiscasehowever,thereisnoneedtousethegeometriccentroidmethodasthetoolsarecapableofreturningtheexactsolution.Ittakesaverysmalltimetondasolutionintheforwarddirection(thermostatsysteminSection 4.2.2 ).Exploringthesametestcaseusingreversereachabilityisunlikelytoproduceameasurablerelativedifferenceinthetestgenerationtime. 4.2CaseStudiesTheexperimentalresultsinthissectionaregeneratedusingHyTech[ 3 ]andPHAVer[ 4 ]tools.FromthelistofexistingmodelcheckingtoolsmentionedinSection 2.3 ,nosingletoolcouldbefoundthatwouldworkforalloftheanalyzedsystems.SinceHyTechonlyworksforlinearhybridsystem,itwasusedtoproducetestcaseforthethermostatsystem(Section 4.2.2 ).Theothersystemthatwasanalyzed,namelybouncingball,(Section 4.2.1 )cannotbeanalyzedbyHyTechasitsupportsvariabledynamicsoftheform_x=x(afnesystem).Forthebouncingballsystem,PHAVerwasselectedtogeneratethetestcase.EventhoughPHAVertoolsupportslinearaswellasafnedynamics,itcouldnotbeusedtogeneratetestcaseforthermostatsystemasitlacksthecapabilitytogenerateatraceconnectingthestates.Withoutthisinformation,thecontrolvariablevaluesthatarerequiredforconstructingatestcasecouldnotbedetermined. 34

PAGE 35

Bouncingballsystemisfreefromcontrolvariables,hencereachabilityanalysisasdonebyPHAVerissufcientfortestcasegeneration. 4.2.1BouncingBallThemodelforthebouncingballsystemisshowninFig. 3-4 .Systemischaracterizedbystatevariables(x1,x2,x3)whichdenotetheverticalposition,horizontalpositionandthevelocityoftheball,respectively.Asthissystemhasdynamicsoftheform_x=x,PHAVerwasusedtogenerateatestcaseforthefollowingfunctionalscenario:\Startingfromaheightrangeof[0,1.35]andvelocityrangeof[)]TJ /F3 11.955 Tf 9.29 0 Td[(2.2,1.1]generateatestcasetohitoorsection[3.9,4.1]suchthatthevelocityoftheballwhilehittingthisregionisintherange[)]TJ /F3 11.955 Tf 9.3 0 Td[(0.4,)]TJ /F3 11.955 Tf 9.3 0 Td[(0.2] 4.2.1.1ForwardreachabilitymethodForwardreachabilityanalysisofPHAVerprovidesayes/noanswerincaseatargetstate(functionalscenariointhiscase)isreachablefromaninitialregion.Inordertodeterminetheinitialregionvaluesthatresultinthetargetstatebeingreachable,copyvariableswerecreatedforeachofthestatevariables.Thesevariablesaredesignatedas(x4,x5,x6).Theoriginal3variablesystemhence,getstransformedasa6variablesystem.Thedynamicsofthe3copyvariablesaresetto0(equationsforx04,x05,x06)sothattheymaintaintheirinitialstatesthroughthecompletestateexplorationprocess.Fig. 4-4 describesthePHAVermodelforthiscase.EquationsforinitialstateandnalstateofthesystemarealsolistedinFig. 4-4 .Thecompositionofthesestatesisdeterminedbyfunctionalscenarioforwhichatestcasemustbegenerated.Usingforwardreachabilityfrominitialstateofthemodel(commandreachableaslistedin[ 17 ]),atestcasecouldnotbegeneratedforthissystemduetothetooltakingmorethanthetimeoutperiodtoprovideananswer.ThisisexpectedaspertheanalysisofforwardreachabilityusingmodelcheckingasexplainedinSection 4.1 .Regionpartitioninginallthestatevariabledirectionswasattemptedwiththesmallestsizeof0.5.Timeoutperiodoftherunswassetto30minutesonalinux2.4GHzmachine. 35

PAGE 36

Figure4-4. PHAVermodelforbouncingballforwardreachabilityanalysis 4.2.1.2ReversereachabilitymethodPHAVertoolalsoprovidesthecapabilitytodoreversereachabilityanalysisbyreversingthecausality(timeandconstraints).ThesystemmodelforreversereachabilityanalysisusingPHAVerisshowninFig. 4-5 .Theinitialstateofthesystemisnowa Figure4-5. PHAVermodelforbouncingballreversereachabilityanalysis 36

PAGE 37

randomlyselectedpoint(x1=0;x2=4.07;x3=)]TJ /F3 11.955 Tf 9.3 0 Td[(0.28)inthetargetfunctionalscenario.Experimentwasrunfor5suchuniformlyrandompointsinthefunctionalscenarioregionresultingin5testcasesthatsatisfytherequirements.Foreachtestcase,thepartitionsizeofeachofthesystemvariableswasvaried.Fig. 4-6 4-7 and 4-8 showoneofthetestcasesproducedfordifferentconstraintpartitioningsizesinthex3direction(velocity).Itcanbeclearlyvisualizedthattheaccuracyofthetestcaseimprovesasthepartitionsizebecomessmaller. Figure4-6. Testcaseforbouncingballwithpartitionsizex3=1 40 Table 4-1 summarizestheresultforall5testcasesforvariouspartitionsizesintime(x2)andvelocity(x3)direction.Partitionindirectionofheight(x1)didnotresultingooddataforanypartitionsize.ItisobviousfromthedataofTable 4-1 thatthetimetogeneratethetestcaseincreaseslinearlyasthepartitionsizebecomessmall.Inadditiontomeasuringthetestcasegenerationtime,theaccuracyofreturnedvalueswasalsomeasuredbysimulatingthereturnedvaluesfromPHAVerinaMATLABprogram.TheMATLABprogramwasdesignedtorandomlysamplethereturnedlistofsolutionpolygonsfor10statepointsandidentifythecaseswheretheydonothitthefunctionalscenarioregion.Asexpected,theaccuracyoftheresultisnot100%formanypartition 37

PAGE 38

Figure4-7. Testcaseforbouncingballwithpartitionsizex3=1 100 Figure4-8. Testcaseforbouncingballwithpartitionsizex3=1 500 38

PAGE 39

Table4-1. Testgenerationtimeandaccuracydataforbouncingballsystem TestcasePartitionaxisx3Partitionaxisx21 401 1001 5001 401 1001 500 tAcctAcctAcctAcctAcctAcc(sec)(%)(sec)(%)(sec)(%)(sec)(%)(sec)(%)(sec)(%) TC110.069.020.086.0?345.0100.0?53.075.0?94.0100.0?--TC216.080.031.0100.0?--212.087.5?478.078.6?3625.0100.0TC36.047.611.077.8?172.0100.0?16.075.0?37.025.0?367.0100.0TC45.447.411.088.9?159.0100.0?15.075.0?34.09.1?382.0100.0TC519.059.742.086.9?804.0100.0?345.04.90520.078.6?9822.0100.0Avg.11.360.723.087.9?370.0100.0?128.263.50232.658.2?3549.0100.0 Note:`?'signiessuccessofgeometriccentroidmethodinmeetingscenariorequirement 39

PAGE 40

sizegranularities.Thisindicatesthattoolisgeneratingfalsepositives.Theaccuracyoftheresulthowever,approaches100%asthepartitionsizebecomesmaller(forsize1 500inthisexperiment).ResultsofTable 4-1 pointatthetradeoffbetweentheaccuracyofthegeneratedtestcaseandthetimetakenforgeneration.However,itisobservedthatevenwhentheaccuracyoftheresultisnotperfectasisthecaseforpartitionsizeof1 100,usingthegeometriccentroidmethodasexplainedinSection 4.1.1 ,resultsinananswerclosetotheexpectedsolution.Forexample,incaseofTC4showninFig. 4-7 ,thetoolreturned9polygonregionsintheinitialregionofthesystem,asthesolution.Eachpolygonisaconvexregionboundedbylinearconstraintsofthe3systemvariables,namelyheight(x1),time(x2)andvelocity(x3).Selectingtheextremeconstraintprojectionvaluesforeachofthevariablesfromallthe9polygonsresultsinvalues:1.10x11.29;0x20;)]TJ /F3 11.955 Tf 9.3 0 Td[(0.02x30.19.Arithmeticmeanofeachofthevariablerangesresultsintheanswerwhichisgeometricallyatthecentroidofthesimpliedunionof9polygons(seeFig. 4-3 forreference).Thereturnedsolutionvaluesinthiscaseare:x1=1.19;x2=0;x3=0.08.Randomtargetpoint,inthefunctionalscenarioregion,selectedforthistestcaseisx1=0;x2=3.97;x3=)]TJ /F3 11.955 Tf 9.3 0 Td[(0.39.Theexpectedsolutionforthistargetpointis:x1=1.2;x2=0;x3=0.07.Itcanbeobservedthattheapproximatesolutionreturnedbythegeometriccentroidmethodiswithin18%errormarginoftheexpectedsolutionforvelocity(x3)and1.5%forheight(x1).InTable 4-1 ,allinstanceswheregeometriccentroidmethodissuccessfulinsatisfyingfunctionalscenarioarehighlightedwitha\?.Itshouldbenotedthatforthesamecasewhenpartitionsizeisreducedto1 500,allsolutionpolygonsresultincorrecttestcase(accuracyoftheresultbecomes100%).However,theimprovementinaccuracy(ignoringanymachinearithmeticapproximationerrors)isatthecostofincreasingthetestgenerationtimeby15times(11secondsversus159seconds).Thegeometriccentroidmethodhence,providesasolutionwithmoderatelossinaccuracywithoutincurringthepenaltyofhigh 40

PAGE 41

testgenerationtimerequiredforhighaccuracy.Byselectingrandomstartpointsinthefunctionalscenarioregionthatarenearthecentroid,themoderatelossinaccuracywouldstillresultinanaccuratetestcase. 4.2.2ThermostatSystemThemodelforthethermostatsystemisshowninFig. 3-6 .Systemischaracterizedbystatevariables(x1,x2)whichdenotethesystemtemperatureandtimerespectively.Thesystemalsohas2discretestatesnamely,\onand\off.Theheatingandcoolingrateintervalsaregivenbyintervalsuon=[2,4]anduo=[)]TJ /F3 11.955 Tf 9.3 0 Td[(3,)]TJ /F3 11.955 Tf 9.3 0 Td[(1],forthe\on(q=1)and\off(q=2)states,respectively.HyTechtool[ 3 ]wasusedtogeneratetestcaseforthefollowingscenario:\Startsystemtemperatureisintherange[1,2]andtemperaturereaches2.11intheonstateafteraruntimeof5.7timeunits.HyTechmodelforthissystemisshowninFig. 4-9 .Aftergeneratingthereachablestatesfrominitialregionofthesystem(representedasvariableinit reginthemodel),tracecommandisusedtogenerateapathfrominit regtonal reg.Thetoolgeneratesatracelogfromwhichanaccuratetestcasecanbeconstructed.TheconstructedtestcaseisshowninFig. 4-10 .Thetoolspentatotalof0.02secondsintestcasegenerationconsuming7.75MBytesofmemoryona2.4GHzlinuxmachine.Multipleattemptstogenerateatestcaseforthisscenarioresultedinthesametestcasebeinggeneratedbythetool.Sincetheabsolutetestgenerationtimeissosmallforthisscenario,noattemptwasmadetogeneratethetestcaseusingreversereachabilityasitisnotexpectedtorevealanymeasurabledifferenceinruntimewhencomparedwithforwardreachabilityresults.However,basedonthereasoninggiveninSection 4.1 ,itisexpectedthatreversereachabilitymethodwouldresultinreductionoftestgenerationtimeinthiscaseaswell. 4.3ComparisonwithRRTBasedTestGenerationAsdescribedinChapter 3 ,RRTbasedtestgenerationmethodsuserandomsimulationtechniquetondsystemtrajectoriesconnectingtwospecicpointsinthe 41

PAGE 42

Figure4-9. HyTechmodelforthermostatsystem Figure4-10. Testcaseconstructedfromthetraceofthermostatmodel 42

PAGE 43

statespace.IfaRRTrunisnotabletondtherequiredtrajectoryinaparticularrun,itneedstoberunagainstartingfromadifferentstartpoint.Modelcheckingbasedmethodsontheotherhand,usesymbolicabstractionstopartitionthestatespaceinniteregions.Itisthenattemptedtoexploreallpossiblesystemtrajectoriesbycreatingamapofinterconnectedstateregionsthusprovidingadenitiveanswerintermsofreachabilityoftargetstate.However,duetotheirrequirementofnitepartitionsizesandsimplicationofconstraintsduringstateexploration,theymightprovideafalsepositiveresult,especiallyforsystemwithafneornonlineardynamics.RRTbasedmethodsontheotherhandprovideanaccurateresultevenforafne-hybridsystemsastheyrelyonsimulationofspecicsystemstateswithoutanyconstraintsimplicationsofthestateregions.ManipulationofthepartitionsizeincaseofmodelcheckingtoolsalsoenablesthemtocreateareasonablyaccuratetestcaseinafractionoftimetakenbyRRTbasedtestgenerationmethods.Table 4-2 providesacomparisonofaveragetestgenerationtimeforbothbouncingballandthermostatsystemusingmodelcheckingandRRTmethods.AverageruntimedataforreverseRRThasbeentakenfromTable 3-1 ofSection 3.3 .FormodelcheckingwerefertodatafromTable 4-1 (partitioninx3directionwithsize1 100)andSection 4.2.2 .Table 4-2 datashowsthatforanafne-hybridsystemlikebouncingball,reversereachabilitybasedmodelcheckingmethodtakesonanaverage2timeslesstimeingeneratingatestcasewhencomparedwithreverseRRT.However,theaccuracyoftheresultincaseofmodelcheckingmethodisnotguaranteedunlessspecialcareistaken(likegeometriccentroidmethodofSection 4.1.1 ).Foralinearsystemlikethermostat,themodelcheckingtoolsareonanaverage8000timesfasterthanreverseRRTmethodsingeneratinganaccuratetestcaseastheydonotsufferfromover-approximationerrors.Insummary,RRTbasedmethodsaremoresuitableforafne-hybridsystemswhenanaccuratesolutionisrequired.ModelcheckingtoolsontheotherhandaremoresuitablethanRRTmethodsfortestgenerationoflinear 43

PAGE 44

Table4-2. TestgenerationtimecomparisonforRRTandmodelchecking SystemTestgenerationtime(sec)Improvement(times)Rev.RRTmodelcheckingRRT/ModelChecking bouncingball53.3423.002.32Thermostat167.530.028376.50 hybridsystems.Inaddition,modelcheckingtoolsprovideanapproximatetestcaseforafne-hybridsystemsinafractionofatimetakenbyRRTmethods. 44

PAGE 45

CHAPTER5TESTGENERATIONUSINGLEARNINGANDCLUSTERINGTECHNIQUESTestgenerationtechniquesdescribedinpreviouschaptersaimtogenerateasingletestcaseforaspecicfunctionalscenario.Anymoderatelycomplexhybridsystemhowever,islikelytohavemorethanonefunctionalscenariothatmustbevalidated.Itishencelogicaltodeveloptechniquesthatcanleadtoincreasedefciencyintestgenerationbylearningfromaclusteroffunctionalscenarios.Thischapterisorganizedasfollows.Section 5.1 describestheoveralltestgenerationmethodincludingthealgorithm.Section 5.2 providesimplementationdetailsabouteachofthemainalgorithmcomponentsincludingtestcaseclustering(Section 5.2.1 )andlearning(Section 5.2.2 ).Finally,Section 5.3 demonstratestheeffectivenessofthemethodsbyprovidingresultsforbouncingballandthermostatsystemandcomparestheresultwiththetestgenerationmethodwithoutlearning(Section 3.1 ). 5.1TestGenerationMethodTestgenerationmethodusingclusteringandlearningtechniquesisshowninFig. 5-1 .Twoimportantstepsofourmethodology,namelyclusteringoffunctionalscenariosandlearningbasedtestgenerationmethod,arealsohighlightedinFig. 5-1 .Itshouldbeemphasizedthateachofthesestepsareindependent.However,theyareequallyimportant.Whenconsideringmultipletestgenerationinstances,itisnotpossibletolearnfromeachotherunlesstheyhavesimilarityintestgenerationpath.Forexample,incaseofRRTbasedmethods,onetestgenerationinstancecanlearnfromanotheronlyiftheyhavereasonableoverlapintheirtrajectory.Unlesstestgenerationisdone,wedonotknowtheirexacttrajectoryoroverlap.Therefore,itisamajorchallengetoclusterasetofsimilartestgenerationinstancesbeforeactualtestgeneration.Onceclusteringisdone,itisalsoequallyimportanttodevelopefcientmethodstoexploitlearningduringtestgenerationofaclusterofsimilarinstancestosignicantlyreducetheoveralltestgenerationtime.Inthiswork,Ihaveusedaclusteringmethodbasedontheproximityof 45

PAGE 46

functionalscenarioandinitialregion,whilemylearningbasedtestgenerationisbasedonreverseRRTalgorithm(Chapter 3 ). Figure5-1. DirectedTestGenerationusingLearningandClustering Algorithm 2 providestheimplementationframeworkforthetestgenerationmethodofFig. 5-1 .ThealgorithmtakessystemmodelM(Section 2.1 )asinputalongwithalistoffunctionalscenariosSL.Theobjectiveofalgorithmistogenerateatestcaseforeachofthescenarios.GeneratedlistoftestcasesisreturnedasstructureLTC.Scenariosarerstrunthroughaclusteringroutine(Make Cluster).Theclusteringroutine(Section 5.2.1 )groupsscenariosinSLbasedontheirkeypropertieswiththeaimofmaximizinglearningbenetforallclustermembers.Ifascenarioistherstoneintheclusterorifitcouldnotbeappropriatelyclusteredwithotherscenarios(sizeofclusteris1),thenroutineGen Nolearn TC()generatesatestcasewithoutemployinganylearningstrategies.However,incasethescenariobeinggeneratedispartofaclusterforwhichanearliertestcasewasgenerated,thenGen Learn TC()routineemployslearning 46

PAGE 47

Input: i)Thedesignmodel,Mii)ListofnfunctionalscenariosSL=fS1,S2,...,Sng Output: i)Listofntestcases,LTC=fTC1,TC2,...,TCng LTC=fg;1 Sclust=fSC1,SC2,...,SCmg=Make Cluster(SL,Pclust);2 whereSCi=fS1,S2,...,SpgisaclusteredlistofscenariosinSL3 foreachClusterSCiinSclustdo4 TCclust=fg;5 forjisfrom1tosizeofcluster(SCi)do6 ifsizeofTCclust1then7 TCj=Gen Learn TC(Sj,M,TCclust);8 end9 else10 TCj=Gen Nolearn TC(Sj,M);11 end12 ifTCjisVALIDthen13 AddTCjtoTCclust;14 end15 end16 AddTCclusttoLTC;17 end18 returnLTC;19 Algorithm2:Testgenerationusingclusteringandlearningtechniques methods(Section 5.2.2 )togainfurtherimprovementintestgenerationtimeforthenewscenario. 5.2AlgorithmImplementationThissectionprovidesimplementationdetailsabouteachoftheimportantroutineslistedinAlgorithm 2 .Amethodforgeneratingasingletestcaseinstance(routineGen Nolearn TC)withoutlearninghasalreadybeendescribedinAlgorithm 1 (Chapter 3 ).ThissectiondescribesthenewcomponentsofAlgorithm 2 ,namelyfunctionsMake Cluster()(Section 5.2.1 )forscenarioclusteringandfunctionGen Learn TC()(Section 5.2.2 )forlearningbasedtestgenerationwithinacluster. 5.2.1TestScenarioClusteringOneoftheimportantobjectivesofclusteringmethodistogroupmultiplefunctionalscenariossuchthatthereisconsiderablesimilarityoftestcasetrajectoriesamongthe 47

PAGE 48

clustermembers.Learningbasedtestcasegenerationmethod(Section 5.2.2 )exploitsthissimilarityoftrajectoriesleadingtoreductionintestgenerationtimefornewertestcaseswithinacluster.Itmustbenotedthattheclusteringmethodisemployedbeforeactualtestcasesaregenerated,hencetestcasetrajectoriesarenotknownapriori.Clusterfunctionthereforeshouldbebasedonasimilaritymeasurethathashighlikelihoodofgeneratingmatchedtestcasetrajectories.HierarchicalclusteringandK-meansclusteringareexamplesofstandardclusteringtechniquesusedbyresearchers.Forthiswork,hierarchicalclusteringtechniquewasselectedastheclusteringfunction(Make Cluster)ofAlgorithm 2 .ThesimilaritymeasureusedintheclusteringfunctionisbasedontheEuclideandistanceoffunctionalscenarioandinitialregionpointsforeachofthetestcaseinstances.ExperimentalresultsofSection 5.3 conrmthatproximityoffunctionalscenarios,thatarespeciedbyrelationaloperatorsofsystemstatevariables,resultsinsystemtrajectoriesthatareconsiderablysimilar.Randomdistributionof100functionalscenariosfora2variablesystemisshowninFig. 5-2 .Thefunctionalscenarioregion(S)andInitialregion(Iinit)arespeciedbythefollowinginequalities:S:(1x13)^(5x210)Iinit:(1x13)^(0x20)SimilaritymeasureforclusteringwasselectedasEuclideandistancebetweenthefunctionalscenarioregionpoints.Byselectingappropriatevaluefortheclustercutoffparameter,differentclusterdistributionscanbegenerated.FortheclusterdistributionshowninFig. 5-3 ,aclustercutoffvalueof0.8wasused.Itshowsthatforthisparticularclustercutoffvalue,total39clustersaregeneratedwiththesizedistributionasshowninFig. 5-3 .Largestclustersizeis6andthereisonlyonesuchcluster.Majorityofclustersareofsize2and3aswouldbeexpectedforauniformlyrandomdistributionprole.Itshouldbenotedthattheactualfunctionalscenariodistributioninthesystem 48

PAGE 49

Figure5-2. Randomdistributionofscenariosforclustering Figure5-3. Clusterdistributionhistogram statespaceisgovernedbythesystemspecication,asshowninFig. 5-1 .Sinceallthescenariosinthiscasehadidenticalinitialregion(Iinit),itdidnotplayaroleindeterminingthesimilaritymeasurebetweenthescenarios.Thisclusteringmethodhowevercanalsobeappliedtosituationswhereeachofthescenarioshavedifferentinitialregionspecication.SimilarclusteringmethodwasusedforthecasestudiespresentedinSection 5.3 49

PAGE 50

5.2.2LearningBasedTestGenerationSignicantreductionintestgenerationtimehasbeenobserved(Section 5.3 )bythelearningbasedtestgenerationmethodofAlgorithm 3 thatpositivelyexploitsthelearningfrompriortestcaseinstanceswithinacluster.NextsectionsprovidedetailsabouteachoftheimportantstepsofAlgorithm 3 5.2.2.1RegioncreationFunctionCreate Regions()ofAlgorithm 3 partitionsthestatespaceofthesystemalongaparticulardimensiontofacilitateefcienttreegrowthusinggoalgenerationadjustment.Section 5.2.2.3 andFig. 5-4 describehowthisregioninformationisusedingoalgenerationforlearningbasedtestcasegeneration.Algorithm 3 currentlysupportspartitioningthesystemstatespacealonganyoneoftheuserselecteddirectionswithaxedpartitionsize. 5.2.2.2GoalsamplegenerationFortestcasesthatcanbenetfromlearningwithinacluster,thegoalgenerationfunction(Gen Goals)inAlgorithm 3 selectsauniformlyrandomtestcasenodefromadynamicallyselectedregionofapreviouslygeneratedtestcaseofthecluster(TCclust).Thedynamicselectionoftestcaseregion(Section 5.2.2.3 )isresponsibleforinuencingthedirectionofgoalgenerationleadingtoefcienttreegrowth.Fig. 5-4 showsanexampletreegrowththatisexpectedfromusingalearningbasedgoalgeneration.Thegoalsample(Xrand)foraparticulariterationinAlgorithm 3 isselectedfromoneofthenodesofaprevioustestcasenamedTC1thatliesinregion3ofthestatespace.Asaresult,thetestcasethatisbeinggenerated(TC2)growstowardstheselectedgoalregionofprevioustestcase(TC1)andfollowsit'strajectory.Thisintendedreuseoftheprevioustestcasetrajectoryenablesthenewtestcasegenerationroutinetorestricttherandomgoalsearchinaverysmallregionofthestatespace.AsconrmedbyresultsofTable 5-1 insection 5.3 ,reductioninrandomsearchspacetranslatestosignicantsavingsintestcasegenerationtime. 50

PAGE 51

Input: i)Thedesignmodel,Mii)AfunctionalscenarioSintheformanXnbniii)AlistofpreviouslygeneratedtestcasesofclusterTCclust Output: i)AninitialregionI02Iinitii)PathfromI0toSaspartoftreestructure)]TJ ET BT /F2 11.955 Tf 26.74 -121.4 Td[(fregiong=Create Regions(Xn,direction);1 I0=fg,)-277(=fg;2 AddNodeUDF(S)to)]TJ /F1 11.955 Tf 6.77 0 Td[(;3 forjisfrom1toMax nodesdo4 ifIinit=2)]TJ /F6 11.955 Tf 10.09 0 Td[(then5 ifTCclustfullyExploredthen6 Gen Nolearn TC();7 end8 else9 Xrand=Gen Goals(TCclust,region));10 Xnear=Find Near Node(Xrand,)]TJ /F1 11.955 Tf 12.26 0 Td[();11 Xsim=fRt0f(Xnear,UDF(inputs))g;12 Xnew=Find Near Node(Xrand,Xsim);13 ifXnew=2)]TJ /F6 11.955 Tf 10.1 0 Td[(then14 addXnewto)]TJ /F1 11.955 Tf 6.78 0 Td[(;15 end16 else17 Goal Adjust(region,threshold);18 end19 end20 end21 else22 I0=Iinit2)]TJ /F1 11.955 Tf 6.77 0 Td[(;23 break;24 end25 end26 ifIinit2)]TJ /F6 11.955 Tf 10.1 0 Td[(then27 returnI0,)]TJ /F1 11.955 Tf 12.26 0 Td[(;28 returnsuccess;29 end30 else31 returnfailure;32 end33 Algorithm3:Testgenerationwithlearning 51

PAGE 52

Figure5-4. Goalgenerationforlearningbasedtestgeneration 5.2.2.3RandomgoaladjustmentFunctionGoal Adjust()ofAlgorithm 3 isresponsiblefordynamicadjustmentoftheregionwhererandomgoalsaregenerated.Fortestcasesthatcantakeadvantagefrompriorlearning,goalsamplegenerationconsistsofrandomlyselectinganodeofapriortestcase(learnedtestcase)oftheclusterTCclust.Theprogramdatastructuremaintainsacountforeachregionofthelearnedtestcasethatwasusedforgoalgeneration.Thecountisincrementediftheselectedgoalfailedtoextendthetreegrowth.Ifthecountforalllearnedtestcasesinaparticularregionexceedsauserdenedthreshold(badgoalthreshold),thegoalgenerationbiasischangedtoselectrandomgoalsfromoneoftheneighboringregionsofthecurrentregion.ReferringtoFig. 5-4 ,regions5and4haveabadgoalcountof20whichisgreaterthanorequaltothethresholdinthiscase,hencethegoalgenerationfortheiterationshownisdynamicallyshiftedtooneofit'sneighboringregions,namelyregion3.Theprocessisrepeatedunlesseithertestcasegenerationsucceeds(treereachesIinit)orallregionshaveabadgoalcountmorethanthethreshold.Ifallregionsreachabadgoalcountmorethanthethreshold,thengoal 52

PAGE 53

generationschemeshiftstothemethodusedfortestcasegenerationwithoutlearning(Gen Nolearn TC)asexplainedinSection 3.2 5.2.2.4FindnearesttreenodeFunctionFind Near Node()ofAlgorithm 3 isresponsibleforselectingtheexistingnodeofthetreethatisnearesttotherandomlygeneratedgoal.Euclideandistancebetweentwovectorsisusedasthemetricfunctioninthiscase. 5.3CaseStudiesThissectionprovidesexperimentalresultsfortestgenerationtechniqueofAlgorithm 2 andAlgorithm 3 throughcasestudiesoftwohybridsystems,namelybouncingballandthermostat. 5.3.1BouncingBallThemodelforthebouncingballsystemisshowninFig. 3-4 .Systemischaracterizedbystatevariables(x1,x2,x3)whichdenotetheverticalposition,horizontalpositionandthevelocityoftheball,respectively.Itisintendedtogeneratetestcasesforrandomlydistributedfunctionalscenariosforthissystemwherefunctionalscenarioregion(S)andinitialregion(Iinit)aregivenby:S:(0x10.1)^(3x24)^()]TJ /F3 11.955 Tf 9.3 0 Td[(0.5x30.5)^(q=1)Iinit:(0x110)^(0x20)^()]TJ /F3 11.955 Tf 9.3 0 Td[(10x310)^(q=1).AfterclusteringthescenariosusingthemethodofSection 5.2.1 ,threerandomscenarioclusterswereselectedforgeneratingtestcases.Clustercutoffvaluewassetto0.8.Thestatespacewaspartitionedalongvariablex2(horizontaldistance)withapartitionsizeof1.Badgoalthresholdissetto10.GeneratedtestcasesforoneoftheclustersareshowninFig. 5-5 .Table 5-1 liststhetestgenerationtimesforallthe3clusters.Forcomparingtheresultswithtestgenerationmethodthatdoesnotuseanylearningandclusteringtechnique,atestcasewasgeneratedforoneofthefunctionalscenariosofthissystemusingAlgorithm 1 ofChapter 3 .ThegeneratedtestcaseisalsoshowninFig. 5-5 .Fig. 5-5 alsoshowsthestatespacethatwasrandomlyexploredfor 53

PAGE 54

treegrowth(Xrandgeneration)inbothcases.AvisualcomparisonofplotsforlearningandnolearningbasedmethodsinFig. 5-5 demonstratesthereductionintherandomexplorationofthespaceusinglearningbasedmethod(thereductionwasmeasuredtobeoftheorderof1.4timesonaverageforthiscluster).Fig. 5-5 plotstheaveragestateexplorationdataofthecompleteclusterforthelearningbasedtestgenerationmethod.Otherclustersproducecomparableresults. A-testcaseswithlearning B-testcasewithnolearningFigure5-5. Bouncingballtestcaseswithlearningandnolearning 5.3.2ThermostatSystemThemodelforthethermostatsystemisshowninFig. 3-6 .Systemischaracterizedbystatevariables(x1,x2)whichdenotethesystemtemperatureandtimerespectively.Thesystemalsohas2discretestatesnamely\onand\off.Thesystemisdesignedtorepeatedlytransitionbetweenthe\onand\offstateinordertoregulatethetemperaturebetweenanupperboundof3andalowerboundof1.Itisintendedtogeneratetestcasesforrandomlydistributedfunctionalscenariosforthissystemwherefunctionalscenarioregion(S)andinitialregion(Iinit)aregivenby:S:(1x13)^(5x210)^(q=1_2)Iinit:(1x13)^(0x20)^(q=1_2) 54

PAGE 55

AfterclusteringthescenariosusingthemethodofSection 5.2.1 ,threerandomscenarioclusterswereselectedforgeneratingtestcases.Clusteringdistanceforhierarchicalclusteringmethodwassetto0.8.Thestatespacewaspartitionedalongvariablex2(time)withapartitionsizeof0.1.Badgoalthresholdissetto15.GeneratedtestcasesareshowninFig. 5-6 foroneoftheclusters.Table 5-1 liststhetestgenerationtimesforallthe3clusters.Forcomparingtheresultswithtestgenerationmethodthatdoesnotuseanylearningandclusteringtechnique,atestcasewasgeneratedforoneofthefunctionalscenariosofthissystemusingAlgorithm 1 ofChapter 3 .ThegeneratedtestcaseisalsoshowninFig. 5-6 .Fig. 5-6 alsoshowsthestatespacethatwasrandomlyexploredfortreegrowth(Xrandgeneration)inbothcases.AvisualcomparisonofplotsforlearningandnolearningbasedmethodsinFig. 5-6 demonstratesthatthelearningbasedmethodsignicantlyreducestherandomsearchofthestatespace.Fig. 5-6 plotstheaveragestateexplorationdataofthecompleteclusterforthelearningbasedtestgenerationmethod.Otherclustersproducecomparableresults. A-testcaseswithlearning B-testcasewithnolearningFigure5-6. Thermostattestcaseswithlearningandnolearning 5.3.3ComparisonwithMethodswithoutLearningTable 5-1 showstheimprovementintestgenerationtimebycomparingtheclusteringandlearningbasedtestgenerationmethodofAlgorithm 2 andAlgorithm 55

PAGE 56

Table5-1. Testgenerationtimecomparisonbetweenlearningbased(lrn)andwithoutlearning(nolrn)methods SystemClusterClusterTotalTimeperTimeperChangeNo.SizetimelrnTC-lrnTC-nolrnnolrn/lrn(sec)(sec)(sec)(times) bouncing1382.427.532.91.2ball2375.925.326.11.03386.428.834.81.2Avg.1.2thermostat141392.3348.1567.81.6241493.9373.5591.61.6331858.2619.41039.11.7Avg.1.6 3 withthatofAlgorithm 1 ofChapter 3 .Testgenerationmethodwithnouseoflearningtechniques(Algorithm 1 ofChapter 3 )targetseachfunctionalscenarioindividually.Incomparison,learningandclusteringbasedmethodaddressestheproblembycreatingaclusterofrelatedfunctionalscenariosofthesystem.Testgenerationtimesavingisrealizedbylearningandsharingtheinformationbetweenmultipleinstances.Thislearningbasedtestgenerationensuresthatnewtestcasesfollowthetrajectoryofsuccessfullygeneratedpriortestcasesasmuchaspossible.Fig. 5-5 andFig. 5-6 demonstratethatbyemployingmaximumreuseofthelearnedtestcasetrajectories,thelearningbasedtestgenerationmethodreducestherandomsearchtoaverysmallregionofthestatespace.Timesavedinrandomexplorationthusconvertsdirectlytoasignicantreductionintestgenerationtime.Experimentresultsindicatethatareductionintestgenerationtimeofuptoafactorof1.7(average1.6times)canberealizedusingclusteringandlearningbasedtechniques.Itshouldalsobenotedthatlargeclustersizesresultinrelativelygreatersavingsastheeffortspentinlearningbenetsalargernumberofscenarioinstances.Thesavingsofeachclusteraddsuptoresultinamuchgreateroverallsavingsforthecompletesystem. 56

PAGE 57

CHAPTER6CONCLUSIONDirectedtestscanachievesamecoveragegoalwithanorders-of-magnitudelessnumberofvectorsascomparedtorandomtestvectors.ExistingvalidationmethodsforhybridsystemsareprimarilybasedonRRT(RapidlyExploringRandomTree)algorithmandmodelchecking.ThisthesispresentedautomatedwaysofgeneratingefcientdirectedtestcasesforhybridsystemsusingbothRRTandmodelcheckingmethods.IncontrasttoexistingforwardgrowthRRTbasedtestgeneration,mymethodemploysreversegrowthRRTalgorithmstartingfromthefunctionalscenarioregionandgrowstowardstheinitialregion.Icombinedthistechniquewithanadaptivebiasadjustmentmethodthatisefcientinrandomlyexploringthesystemstatespace.Multiplefunctionalscenariosthatmustbevalidatedforasystem,provideopportunitieswheretestgeneratorscanlearnfrompreviouslygeneratedtestcases.Thislearninghelpstooptimizegenerationalgorithmforadditionalsavingswhilegeneratinganewtestcase.IalsopresentedefcientclusteringandlearningalgorithmsthatprovidesignicantadditionalsavingsforreverseRRTbasedtestgenerationmethods.ExperimentalresultsusingmodelsofvaryingcomplexitiesdemonstratethatmyreversegrowthRRTalgorithmisupto33timesfasterandrequireslessmemory(upto4%)whencomparedwithexistingforwardgrowthRRTalgorithm.Inaddition,clusteringandlearningbasedmethodscanprovideanadditionaltestgenerationtimereductionof1.6timesonaverageoverreverseRRTmethodsthatdonotemployanylearningtechnique.Whileexistingmodelcheckingbasedmethodsareveryefcientandaccurateingeneratingtestcaseforlinearhybridsystemsthathaveaconstantboundonthestatevariabledynamics,theyemployover-approximationofreachablesetswhenanalyzingafnehybridsystems(statedynamicsoftheform_x=x+A),thusresultinginnonexactresults.Mytestgenerationmethodusingmodelcheckingdemonstratedreversereachabilitybasedanalysisforbothlinearandafne-hybridsystemsthatstartsfroma 57

PAGE 58

specicstatepointinthefunctionalscenarioregion.Inordertoaddresstheaccuracyoftheresultsforafne-hybridsystems,ageometriccentroidmethodwasproposedthatresultsinmoderatelossinaccuracyoftheresult(worstcaseof18%error)whileproducingamultipleorderofreductionintestgenerationtime(upto15times).Thelossofaccuracycanbecompensatedbycarefulselectionofstartpointinthefunctionalscenarioregion.AcomparisonofRRTandmodelcheckingbasedtestgenerationmethodsconrmsthatRRTbasedmethodsarebettersuitedforafne-hybridsystemswhenanaccuratesolutionisrequired.ModelcheckingbasedtestgenerationtoolsontheotherhandaremuchmoreefcientthanRRTmethodsforlinearsystems.Inaddition,modelcheckingtoolsprovideanapproximatetestcaseforafne-hybridsystemsinafractionofatimetakenbyRRTmethods. 58

PAGE 59

REFERENCES [1] M.Chen,X.Qin,H.Koo,andP.Mishra,System-levelvalidation:high-levelmodelinganddirectedtestgenerationtechniques,2012. [2] J.Lygeros,Lecturenotesonhybridsystems,tech.rep.,2004. [3] T.Henzinger,P.Ho,andH.Wong-toi,Hytech:Amodelcheckerforhybridsystems,SoftwareToolsforTechnologyTransfer,vol.1,pp.460,1997. [4] G.Frehse,Phaver:Algorithmicvericationofhybridsystemspasthytech,2005. [5] W.Hartong,L.Hedrich,andE.Barke,Modelcheckingalgorithmsforanalogverication,2002. [6] B.SilvaandB.Krogh,Formalvericationofhybridsystemsusingcheckmate:acasestudy,inProc.AmericanControlConference,pp.1679Vol3,2000. [7] G.Frehse,C.Guernic,A.Donze,S.Cotton,R.Ray,O.Lebeltel,R.Ripado,A.Girard,T.Dang,andO.Maler,Spaceex:Scalablevericationofhybridsystems,inProc.23rdInternationalConferenceonComputerAidedVerica-tion(CAV),2011. [8] E.Asarin,T.Dang,andO.Maler,Thed/dttoolforvericationofhybridsystem,2002. [9] S.LavalleandJ.Kuffner,Rapidly-exploringrandomtrees:Progressandprospects,inAlgorithmicandComputationalRobotics:NewDirections,pp.293,2000. [10] M.Branicky,M.Curtiss,J.Levine,andS.Morgan,Rrtsfornonlinear,discrete,andhybridplanningandcontrol,inIEEEConf.onDecisionandControl,pp.9,2003. [11] T.DangandT.Nahhal,Coverage-guidedtestgenerationforcontinuousandhybridsystems,Form.MethodsSyst.Des.,vol.34,pp.183,2009. [12] S.Ahmadyan,J.Kumar,andS.Vasudevan,Goal-orientedstimulusgenerationforanalogcircuits,inDAC,pp.1018,2012. [13] J.KimandJ.Esposito,Adaptivesamplebiasforrapidly-exploringrandomtreeswithapplicationstotestgeneration,AmericanControlConference,vol.2005,2005. [14] E.Clarke,O.Grumberg,andD.Peled,ModelChecking.Cambridge,MA,USA:MITPress,1999. [15] T.Henzinger,Thetheoryofhybridautomata,inLogicinComputerScience,1996. 59

PAGE 60

[16] R.Alur,C.Courcoubetis,N.Halbwachs,T.Henzinger,P.Ho,X.Nicollin,A.Olivero,J.Sifakis,andS.Yovine,Thealgorithmicanalysisofhybridsystems,TheoreticalComputerScience,vol.138,pp.3,1995. [17] G.Frehse,LanguageOverviewforPHAVer.2006. 60

PAGE 61

BIOGRAPHICALSKETCH SudhiRanjanProchreceivedhisBachelorofEngineeringinElectronicsfromMotilalNehruNationalInstituteofTechnology,Indiain1998.Foralmost14yearsheworkedinthechipdesignindustryforvariousemployersasadesignandvalidationengineer.HecompletedhisMasterofScienceinElectricalandComputerEngineeringfromUniversityofFloridain2014. 61