Hummerfedora16

TestingIdempotenceforInfrastructureasCodeWaldemarHummer1,FlorianRosenberg2,FabioOliveira2,andTamarEilam21DistributedSystemsGroup,ViennaUniversityofTechnology,Austriahummer@dsg.
tuwien.
ac.
at2IBMT.
J.
WatsonResearchCenter,YorktownHeights,NY,USA{rosenberg,fabolive,eilamt}@us.
ibm.
comAbstract.
Duetothecompetitivenessofthecomputingindustry,soft-waredevelopersarepressuredtoquicklydelivernewcodereleases.
Atthesametime,operatorsareexpectedtoupdateandkeepproductionsys-temsstableatalltimes.
Toovercomethedevelopment–operationsbar-rier,organizationshavestartedtoadoptInfrastructureasCode(IaC)toolstoecientlydeploymiddlewareandapplicationsusingautoma-tionscripts.
Theseautomationscompriseaseriesofstepsthatshouldbeidempotenttoguaranteerepeatabilityandconvergence.
Rigoroustestingisrequiredtoensurethatthesystemidempotentlyconvergestoade-siredstate,startingfromarbitrarystates.
Weproposeandevaluateamodel-basedtestingframeworkforIaC.
Anabstractedsystemmodelisutilizedtoderivestatetransitiongraphs,basedonwhichwesystemati-callygeneratetestcasesfortheautomation.
Thetestcasesareexecutedinlight-weightvirtualmachineenvironments.
OurprototypetargetsonepopularIaCtool(Chef),buttheapproachisgeneral.
WeapplyourframeworktoalargebaseofpublicIaCscriptswrittenbyoperators,showingthatitcorrectlydetectsnon-idempotentautomations.
Keywords:MiddlewareDeployment,SoftwareAutomation,Idempo-tence,Convergence,InfrastructureasCode,SoftwareTesting.
1IntroductionTheever-increasingneedforrapidlydeliveringcodechangestosatisfynewre-quirementshasledmanyorganizationstorethinktheirdevelopmentpractices.
Acommonimpedimenttothisdemandforquickcodedeliverycyclesisthewell-knowntensionbetweensoftwaredevelopersandoperators:theformerareconstantlypressuredtodelivernewreleases,whereasthelattermustkeeppro-ductionsystemsstableatalltimes.
Notsurprisingly,operatorsarereluctanttoacceptchangesandtendtoconsumenewcodeslowerthandeveloperswouldlike.
Inordertorepeatedlydeploymiddlewareandapplicationstotheproductionenvironment,operationsteamstypicallyrelyonautomationlogic(e.
g.
,scripts).
Asnewapplicationreleasesbecomeavailable,thislogicmayneedtoberevisitedtoaccommodatenewrequirementsimposedontheproductioninfrastructure.
Asautomationlogicistraditionallynotdevelopedfollowingthesamerigorofsoft-wareengineeringusedbyapplicationdevelopers(e.
g.
,modularity,re-usability),D.
EyersandK.
Schwan(Eds.
):Middleware2013,LNCS8275,pp.
368–388,2013.
cIFIPInternationalFederationforInformationProcessing2013TestingIdempotenceforInfrastructureasCode369automationstendtoneverachievethesamelevelofmaturityandquality,incur-ringanincreasedriskofcompromisingthestabilityofthedeployments.
Thisstate-of-aairshasbeenfuelingtheadoptionofDevOps[1,2,3]practicestobridgethegapbetweendevelopersandoperators.
OneofthepillarsofDevOpsisthenotionofInfrastructureasCode(IaC)[1,4],whichfacilitatesthedevelop-mentofautomationlogicfordeploying,conguring,andupgradinginter-relatedmiddlewarecomponentsfollowingkeyprinciplesinsoftwareengineering.
IaCau-tomationsareexpectedtoberepeatablebydesign,sotheycanbringthesystemtoadesiredstatestartingfromanyarbitrarystate.
Torealizethismodel,state-of-the-artIaCtools,suchasChef[5]andPuppet[6],providedeveloperswithseveralabstractionstoexpressautomationstepsasidempotentunitsofwork.
Thenotionofidempotencehasbeenidentiedasthefoundationforrepeat-able,robustautomations[7,8].
Idempotenttaskscanbeexecutedmultipletimesalwaysyieldingthesameresult.
Importantly,idempotenceisarequirementforconvergence[7],theabilitytoreachacertaindesiredstateunderdierentcir-cumstancesinpotentiallymultipleiterations.
Thealgebraicfoundationsoftheseconceptsarewell-studied;however,despite(1)theirimportanceaskeyelementsofDevOpsautomationsand(2)thecriticalroleofautomationstoenablefrequentdeploymentofcomplexinfrastructures,testingofidempotenceinrealsystemshasreceivedlittleattention.
Tothebestofourknowledge,noworktodatehasstudiedthepracticalimplicationsofidempotenceorsoughttosupportdevelop-ersascertainthattheirautomationsidempotentlymakethesystemconverge.
WetacklethisproblemandproposeaframeworkforsystematictestingofIaCautomationscripts.
Givenaformalmodeloftheproblemdomainandinputcov-eragegoalsbasedonwell-denedcriteria,aStateTransitionGraph(STG)oftheautomationundertestisconstructed.
TheresultingSTGisusedtoderivetestcases.
AlthoughourprototypeimplementationisbasedonChef,theapproachisdesignedforgeneralapplicability.
WerelyonAspect-OrientedProgramming(AOP)toseamlesslyhookthetestexecutionharnessintoChef,withpracti-callynocongurationeortrequired.
Sinceecientexecutionoftestcasesisakeyissue,ourprototypeutilizesLinuxcontainers(LXC)aslight-weightvirtualmachine(VM)environmentsthatcanbeinstantiatedwithinseconds.
Ourex-tensiveevaluationcoverstestingofroughly300publiclyavailable,real-lifeChefscripts[9].
Afterexecuting3671testcases,ourframeworkcorrectlyidentied92ofthosescriptsasnon-idempotentinourtestenvironment.
Next,weprovidesomebackgroundonChefandhighlighttypicalthreatstoidempotenceinautomations(§2),presentanoverviewofourapproach(§3),de-tailtheunderlyingformalmodel(§4),delveintoSTG-basedtestcasegenerationandexecution(§5),unveilourprototypeimplementation(§6),discussevalua-tionresults(§7),summarizerelatedwork(§8),andwrapupthepaper(§9).
2BackgroundandMotivationInthissectionweexplaintheprinciplesbehindmodernIaCtoolsandtheim-portanceoftestingIaCautomationsforidempotence.
AlthoughwecouchourdiscussioninthecontextofChef[5],thesameprinciplesapplytoallsuchtools.
370W.
Hummeretal.
ChefBackground.
InChefterminology,automationlogiciswrittenasrecipes,andacookbookpackagesrelatedrecipes.
Followingadeclarativeparadigm,recipesdescribeaseriesofresourcesthatshouldbeinaparticularstate.
Listing1.
1showsasamplerecipeforthefollowingdesiredstate:directory"/tmp/mydir"mustexistwiththespeciedpermissions;package"tomcat6"mustbeinstalled;OSservice"tomcat6"mustrunandbeconguredtostartatboottime.
Eachresourcetype(e.
g.
,package)isimplementedbyplatform-dependentprovidersthatproperlyconguretheassociatedresourceinstances.
Chefen-surestheimplementationofresourceprovidersisidempotent.
Thus,evenifoursamplerecipeisexecutedmultipletimes,itwillnotfailtryingtocreateadirec-torythatalreadyexists.
Thesedeclarative,idempotentabstractionsprovideauniformmechanismforrepeatableexecution.
Thismodelofrepeatabilityisim-portantbecauserecipesaremeanttoberunperiodicallytooverrideout-of-bandchanges,i.
e.
,preventdriftsfromthedesiredstate.
Inotherwords,arecipeisexpectedtocontinuouslymakethesystemconvergetothedesiredstate.
§¤1directory"tmp/mydir"do2owner"root"3group"root"4mode07555action:create6end7package"tomcat6"do8action:install9end10service"tomcat6"do11action[:start,:enable]12endListing1.
1.
DeclarativeChefRecipe§¤1bash"buildphp"do2cwdConfig[:filecachepath]3code2000downloads),weidentiedanon-trivialidempotencebugrelatedtoincorrectlepermissions.
Theversionnumberindicatesthatthecookbookhasundergoneanumberofrevisionsandxes,butthisissuewasapparentlynotdetected.
Table6.
TasksinChefCookbooktomcat6TaskResourceTypeDescriptiona9directoryCreatesdirectory/etc/tomcat6/a16bashCopieslesto/etc/tomcat6/asusertomcat;onlyexecutedif/etc/tomcat6/tomcat6.
confdoesnotexist.
a21fileWritesto/etc/tomcat6/logging.
propertiesasuserroot.
a22serviceEnablestheservicetomcat(i.
e.
,automaticstartatboot)a23fileCreatesle/etc/tomcat6/tomcat6.
confThecrucialtasksareoutlinedinTable6(theentireautomationconsistsof25tasks).
ApplyingthetestcoveragesettingsfromSection7.
1,thetestsuiteforthiscookbookconsistsof23testcases,outofwhichtwotestcases(denotedt1,t2)failed.
Testt1isconguredtoruntasksequencea1,.
.
.
,a21,a1,.
.
.
,a25(simulatingthattheautomationisterminatedandrepeatedaftertaska21),andtestt2isconguredwithtasksequencea1,.
.
.
,a22,a1,.
.
.
,a25(restartingaftertaska22).
Bothtestcasesfailedatthesecondexecutionoftaska16,denotede(a16)[2]inourmodel,whichcopiescongurationlestoadirectorypreviouslycreatedbytaska9.
Inthefollowingweclarifywhyandhowthisfaulthappens.
Thereasonwhyt1andt2failedwhenexecutinge(a16)[2]isthatatthetimeofexecutionthele/etc/tomcat6/logging.
propertiesisownedbyuserroot,anda16attemptstowritetothesameleasusertomcat(resultingin"permissiondenied"fromtheoperatingsystem).
Weobservethattaska21alsowritestothesamele,butincontrasttotaska16notasusertomcat,butasuserroot.
Atexe-cutione(a21)[1],thecontentofthelegetsupdatedandtheleownershipissettoroot.
Hence,thecookbookdeveloperhasintroducedanimplicitdependencybetweentasksa16anda21,whichleadstoidempotenceproblems.
Notethattheother21testcasesdidnotfail.
Clearly,alltestcasesinwhichtheautomationisrestartedbeforetheexecutionoftaska21arenotaectedbythebug,sincetheownershipoftheledoesnotgetoverwritten.
Theremainingtestcasesinwhichtheautomationwasrestartedaftera21(i.
e.
,aftera23,a24,anda25)didnotfailduetoaconditionalstatementnotifwhichensuresthata16isonlyexecutedif/etc/tomcat6/tomcat6.
confdoesnotexist.
ChefCookbookmongodb-10gen.
Thethirdinterestingcasewediscussiscook-bookmongodb-10gen(installsMongoDB),forwhichourframeworkallowedusto384W.
Hummeretal.
detectanidempotencebugintheChefimplementationitself.
TherelevanttasksareillustratedinTable7:a11installspackagemongodb-10gen,a12createsadirec-tory,anda13createsanothersub-directoryandplacescongurationlesinit.
Ifinstalledproperly,thepackagemongodb-10gencreatesuserandgroupmongodbonthesystem.
However,sincethecookbookdoesnotconguretherepositoryprop-erly,thispackagecannotbeinstalled,i.
e.
,taska11failedinourtests.
Now,astaska12isexecuted,itattemptstocreateadirectorywithuser/groupmongodb,whichbothdonotexistatthattime.
Letusassumethetestcasewithtasksequencea1,a13,a1,a13.
Asitturnsout,therstexecutionofa13cre-ates/data/mongodbwithuser/groupsettoroot/mongodb(eventhoughgroupmongodbdoesnotexist).
Onthesecondexecutionofa12,however,Chefagaintriestosetthedirectory'sownershipandreportsanerrorthatusermongodbdoesnotexist.
ThisbehaviorisclearlyagainstChef'snotionofidempotence,becausetheerrorshouldhavebeenreportedonthersttaskexecutionalready.
Infact,ifthecookbookwasrunonlyonce,thiscongurationerrorwouldnotbedetected,butmayleadtoproblemsatruntime.
Wesubmittedabugreport(OpscodeticketCHEF-4236)whichhasbeenconrmedbyChefdevelopers.
Table7.
TasksinChefCookbookmongodb-10genTaskResourceTypeDescriptiona11packageInstallspackagemongodb-10gena12directoryCreatesdirectory/dataa13remotedirectoryCreatesdirectory/data/mongodbasuser/groupmongodbLessonsLearned.
Thekeytake-awaymessageoftheseillustrativereal-worldexamplesisthatautomationsmaycontaincompleximplicitdependencies,whichIaCdevelopersareoftennotawareof,butwhichcanbeecientlytestedbyourapproach.
Forinstance,theconditionalnotifina16ofrecipetomcat6wasintroducedtoavoidthattheconglegetsoverwritten,butthedeveloperwasapparentlynotawarethatthischangebreakstheidempotenceandconvergenceoftheautomation.
Thisexampledemonstratesnicelythatsomeidempotenceandconvergenceproblems(particularlythoseinvolvingdependenciesamongmultipletasks)cannotbeavoidedsolelybyprovidingdeclarativeandidempotentresourceimplementations(e.
g.
,asprovidedinChef)andhencerequiresystematictesting.
7.
3IdempotenceforDierentTaskTypesTable8showsthenumberofidentiednon-idempotenttasks(denoted#NI)fordierenttasktypes.
ThetasktypescorrespondtotheChefresourcesusedintheevaluatedcookbooks.
Thesetofscriptingtasks(execute,bash,script,rubyblock)makesupfor90ofthetotal263non-idempotenttasks,whichcon-rmsoursuspicionthatthesetasksareerror-prone.
Interestingly,theservicetasktypealsoshowsmanynon-idempotentoccurrences.
Lookingfurtherintothisissue,weobservedthatservicetasksoftencontaincustomcodecommandstostart/restart/enableservices,whicharepronetoidempotenceproblems.
TestingIdempotenceforInfrastructureasCode385Table8.
Non-IdempotentTasksByTaskTypeTaskType#NITaskType#NITaskType#NIservice66directory10link3execute44remotefile10bluepillservice2package30gempackage7cookbookfile2bash27file5git2template19pythonpip5user2script15rubyblock4aptpackage17.
4IdempotenceforDierentCookbookVersionsWeanalyzedtheevolutionofthe20mostpopularChefcookbooks.
There-sultsinTable9leaveoutcookbookswithemptydefaultrecipes(application,openssl,users)andcookbookswithoutanynon-idempotenttasks:mysql,java,postgresql,build-essential,runit,nodejs,git,ntp,python,revealcloud,graylog2.
Forthecookbooksundertest,newreleasesxedidempotenceissues,oratleastdidnotintroducenewissues.
Ourtoolautomaticallydeterminesthesedata,henceitcanbeusedtotestautomationsforregressionsandnewbugs.
Table9.
EvolutionofNon-IdempotentTasksByIncreasingVersionCookbooki-9i-8i-7i-6i-5i-4i-3i-2i-1iapache2(i=1.
4.
2)1110000000nagios(i=3.
1.
0)1100000000zabbix(i=0.
0.
40)2222222222php(i=1.
1.
4)1100000000tomcat6(i=0.
5.
4)33333321riak(i=1.
2.
1)11111100008RelatedWorkExistingworkhasidentiedtheimportanceofidempotenceforbuildingreli-abledistributedsystems[13]anddatabasesystems[14].
Overthelastyears,theimportanceofbuildingtestablesystemadministration[8]basedonconvergentmodels[15,7]becamemoreprevalent.
cfengine[16]wasamongthersttoolsinthisspace.
Morerecently,otherIaCframeworkssuchasChef[5]orPuppet[6]heavilyrelyontheseconcepts.
However,automatedandsystematictestingofIaCforverifyingidempotenceandconvergencehasreceivedlittleattention,despitetheincreasingtrendofautomatingmulti-nodesystemdeployments(i.
e.
,contin-uousdelivery[17])andplacementofvirtualinfrastructuresintheCloud[18].
ExistingIaCtestframeworksallowdeveloperstomanuallywritetestcodeusingcommonBehavior-DrivenDevelopment(BDD)techniques.
ChefSpec[19]orCucumber-puppet[20]allowtoencodethedesiredbehaviorforverifyingin-dividualautomationtasks(unittesting).
TestKitchen[21]goesonestepfurtherbyenablingtestingofmulti-nodesystemdeployments.
Itprovisionsisolatedtest386W.
Hummeretal.
environmentsusingVMswhichexecutetheautomationundertestandverifytheresultsusingtheprovidedtestframeworkprimitives.
Thiskindoftestingisamanualandlaborintensiveprocess.
OurframeworktakesadierentapproachbysystematicallygeneratingtestcasesforIaCandexecutingtheminascalablevirtualizedenvironment(LXC)todetecterrorsandidempotenceissues.
Extensiveresearchisconductedonautomatedsoftwaredebuggingandtest-ingtechniques,includingmodel-basedtesting[22]orsymbolicexecution[23],aswellastheirapplicationtospecializedproblemareas,forinstancecontrolowbased[24]ordataowbased[25]testingapproaches.
Mostexistingworkandtools,however,arenotdirectlyapplicabletothedomainofIaC,fortwomainreasons:(i)IaCexposesfairlydierentcharacteristicsthantraditionalsoftwaresystems,i.
e.
,idempotenceandconvergence;(ii)IaCneedstobetestedinrealen-vironmentstoensurethatsystemstatechangestriggeredbyautomationscriptscanbeassertedaccordingly.
Suchtestsarehardtosimulate,hencesymbolicexecutionwouldhavelittlepracticalvalue.
Eventhoughdry-runcapabilitiesexist(e.
g,Chef'swhy-runcapability),theycannotreplacesystematictesting.
Theapplicabilityofautomatedtestingisakeyrequirementidentiedbyotherapproaches[26,27,28],whetherthetesttargetissystemsoftwareorIaC.
Existingapproachesformiddlewaretestinghavelargelyfocusedonperfor-manceandeciency.
Casaleetal.
[29]useautomaticstresstestingformulti-tiersystems.
Theirworkplacesburstyservicedemandsonsystemresources,inordertoidentifyperformancebottlenecksaswellaslatencyandthroughputdegrada-tions.
Otherworkfocusesontestingmiddlewareforelasticity[30],whichisbecomingakeypropertyforCloudapplications.
Bucuretal.
[26]proposeanautomatedsoftwaretestingapproachthatparallelizessymbolicexecutionsforeciency.
Thesystemundertestcaninteractwiththeenvironmentviaa"sym-bolicsystemcall"layerthatimplementsasetofcommonPOSIXprimitives.
Theirapproachcouldpotentiallyenhanceourworkandmayspeeduptheper-formance,butrequiresacompleteimplementationofthesystemcalllayer.
Otherapproachesdealwithndingandxingcongurationerrors[31,32].
Faultscausedbycongurationerrorsareoftenintroducedduringdeploymentandremaindormantuntilactivatedbyaparticularaction.
Detectingsucherrorsischallenging,buttoolslikeAutoBash[32]orChronus[31]caneectivelyhelp.
AnaturalextensionwouldbetoalsotakeintoaccounttheIaCscriptstondthecongurationparameterthatpotentiallycausedtheproblem.
Burgetal.
[28]proposeautomatedsystemtestsusingdeclarativevirtualmachines.
Declarativespecicationsdescribeexternaldependencies(e.
g.
,accesstoexternalservices)togetherwithanimperativetestscript.
Theirtoolthenbuildsandinstantiatesthevirtualmachinenecessarytorunthescript.
Ourapproachleveragespre-builtcontainersinLXC;dynamicallycreatingadeclarativespecicationwouldbepossiblebutbuildingaVMismorecostlythanbringingupanLXCcontainer.
9ConclusionWeproposeanapproachformodel-basedtestingofInfrastructureasCode,aim-ingtoverifywhetherIaCautomations,suchasChefrecipes,canrepeatedlyTestingIdempotenceforInfrastructureasCode387makethetargetsystemconvergetoadesiredstateinanidempotentmanner.
GiventheIaCmodelofperiodicre-executions,idempotenceisacriticalprop-ertywhichensuresrepeatabilityandallowsautomationstostartexecutingfromarbitraryinitialorintermediatestates.
Ourextensiveevaluationwithreal-worldIaCscriptsfromtheOpsCodecommunityrevealedthattheapproacheectivelydetectsnon-idempotence.
Outofroughly300testedChefscripts,almostathirdwereidentiedasnon-idempotent.
Inaddition,wewereabletodetectandreportabugintheChefimplementationitself.
Ournovelapproachopensupexcitingfutureresearchdirections.
First,wewillextendourprototypetohandletheexecutionofdistributedautomationswithcross-nodedependencies,whichisoftenusedtodeploymulti-nodesystems.
Second,weplantoapplytheapproachtootherIaCframeworkslikePuppet,whoseexecutionmodeldoesnotassumetotaltaskordering.
Third,weenvisionthatsystematicdebugging/analysiscanbepushedfurthertoidentifyimplicitdependenciesintroducedbyIaCdevelopers.
Moreover,wearecurrentlyextend-ingthestatecapturingmechanismtodetectne-grainedchangesonsystemcalllevel.
Thehypothesisisthattheimprovedmechanismcanleadtodetectionofadditionalnon-idempotencecasesstemmingfromsideeectswecurrentlymiss.
References1.
H¨uttermann,M.
:DevOpsforDevelopers.
Apress(2012)2.
Loukides,M.
:WhatisDevOpsO'ReillyMedia(2012)3.
Schaefer,A.
,Reichenbach,M.
,Fey,D.
:ContinuousIntegrationandAutomationforDevops.
IAENGTrans.
onEngineeringTechnologies170,345–358(2013)4.
Nelson-Smith,S.
:Test-DrivenInfrastructurewithChef.
O'Reilly(2011)5.
Opscode:http://www.
opscode.
com/chef/6.
PuppetLabs:http://puppetlabs.
com/7.
Couch,A.
L.
,Sun,Y.
:Onthealgebraicstructureofconvergence.
In:Brunner,M.
,Keller,A.
(eds.
)DSOM2003.
LNCS,vol.
2867,pp.
28–40.
Springer,Heidelberg(2003)8.
Burgess,M.
:Testablesystemadministration.
Commun.
ACM54(3),44–49(2011)9.
OpscodeCommunity:http://community.
opscode.
com/10.
Utting,M.
,Pretschner,A.
,Legeard,B.
:Ataxonomyofmodel-basedtestingap-proaches.
SoftwareTesting,VericationandReliability22(5),297–312(2012)11.
Outt,J.
,Liu,S.
,Abdurazik,A.
,Ammann,P.
:Generatingtestdatafromstate-basedspecications.
SoftwareTesting,VericationandReliability13,25–53(2003)12.
Nie,C.
,Leung,H.
:Asurveyofcombinatorialtesting.
ACMComp.
Surv.
(2011)13.
Helland,P.
:Idempotenceisnotamedicalcondition.
ACMQueue10(4)(2012)14.
Helland,P.
,Campbell,D.
:Buildingonquicksand.
In:ConferenceonInnovativeDataSystemsResearch,CIDR(2009)15.
Traugott,S.
:Whyordermatters:Turingequivalenceinautomatedsystemsad-ministration.
In:16thConferenceonSystemsAdministration(LISA),pp.
99–120(2002)16.
Zamboni,D.
:LearningCFEngine3:Automatedsystemadministrationforsitesofanysize.
O'ReillyMedia,Inc.
(2012)17.
Humble,J.
,Farley,D.
:ContinuousDelivery:ReliableSoftwareReleasesthroughBuild,Test,andDeploymentAutomation.
Addison-WesleyProfessional(2010)388W.
Hummeretal.
18.
Giurgiu,I.
,Castillo,C.
,Tantawi,A.
,Steinder,M.
:Enablingecientplacementofvirtualinfrastructuresinthecloud.
In:Narasimhan,P.
,Triantallou,P.
(eds.
)Middleware2012.
LNCS,vol.
7662,pp.
332–353.
Springer,Heidelberg(2012)19.
ChefSpec:https://github.
com/acrmp/chefspec20.
Cucumber-puppet:http://projects.
puppetlabs.
com/projects/cucumber-puppet21.
TestKitchen:https://github.
com/opscode/test-kitchen22.
Pretschner,A.
:Model-basedtesting.
In:Proceedingsofthe27thInternationalCon-ferenceonSoftwareEngineering,ICSE2005,pp.
722–723(2005)23.
Cadar,C.
,Godefroid,P.
,etal.
:Symbolicexecutionforsoftwaretestinginpractice:preliminaryassessment.
In:33rdInt.
Conf.
onSoftwareEngineering,ICSE(2011)24.
BenavidesNavarro,L.
D.
,Douence,R.
,S¨udholt,M.
:Debuggingandtestingmiddle-warewithaspect-basedcontrol-owandcausalpatterns.
In:Issarny,V.
,Schantz,R.
(eds.
)Middleware2008.
LNCS,vol.
5346,pp.
183–202.
Springer,Heidelberg(2008)25.
Hummer,W.
,Raz,O.
,Shehory,O.
,Leitner,P.
,Dustdar,S.
:Testingofdata-centricandevent-baseddynamicservicecompositions.
In:Softw.
Test.
,Verif.
&Reliab.
(2013)26.
Bucur,S.
,Ureche,V.
,Zamr,C.
,Candea,G.
:Parallelsymbolicexecutionforauto-matedreal-worldsoftwaretesting.
In:ACMEuroSys.
Conf.
,pp.
183–198(2011)27.
Candea,G.
,Bucur,S.
,Zamr,C.
:Automatedsoftwaretestingasaservice.
In:1stACMSymposiumonCloudComputing(SoCC),pp.
155–160(2010)28.
vanderBurg,S.
,Dolstra,E.
:Automatingsystemtestsusingdeclarativevirtualmachines.
In:21stInt.
SymposiumonSoftwareReliabilityEngineering(2010)29.
Casale,G.
,Kalbasi,A.
,Krishnamurthy,D.
,Rolia,J.
:Automaticstresstestingofmulti-tiersystemsbydynamicbottleneckswitchgeneration.
In:Bacon,J.
M.
,Cooper,B.
F.
(eds.
)Middleware2009.
LNCS,vol.
5896,pp.
393–413.
Springer,Heidelberg(2009)30.
Gambi,A.
,Hummer,W.
,Truong,H.
L.
,Dustdar,S.
:TestingElasticComputingSystems.
IEEEInternetComputing(2013)31.
Whitaker,A.
,Cox,R.
,Gribble,S.
:Congurationdebuggingassearch:ndingtheneedleinthehaystack.
In:Symp.
onOp.
Sys.
Design&Impl(OSDI),p.
6(2004)32.
Su,Y.
Y.
,Attariyan,M.
,Flinn,J.
:AutoBash:improvingcongurationmanagementwithoperatingsystemcausalityanalysis.
In:SOSP(2007)