goalscentos6.0

ExtractingFlexible,ReplayableModelsfromLargeBlockTracesV.
Tarasov1,S.
Kumar1,J.
Ma2,D.
Hildebrand3,A.
Povzner3,G.
Kuenning2,andE.
Zadok11StonyBrookUniversity,2HarveyMuddCollege,and3IBMAlmadenResearchAbstractI/Otracesaregoodsourcesofinformationaboutreal-worldworkloads;replayingsuchtracesisoftenusedtoreproducethemostrealisticsystembehaviorpossible.
Buttracestendtobelarge,hardtouseandshare,andinexibleinrepresentingmorethantheexactsystemconditionsatthepointthetraceswerecaptured.
Often,however,researchersarenotinterestedintheprecisede-tailsstoredinabulkytrace,butratherinsomestatisti-calpropertiesfoundinthetraces—propertiesthataffecttheirsystem'sbehaviorunderload.
Wedesignedandbuiltasystemthat(1)extractsmanydesiredpropertiesfromalargeblockI/Otrace,(2)buildsastatisticalmodelofthetrace'ssalientcharacteristics,(3)convertsthemodelintoaconcisedescriptioninthelanguageofoneormoresyntheticloadgenerators,and(4)canaccuratelyreplaythemodelsintheseloadgener-ators.
Oursystemismodularandextensible.
Weexper-imentedwithseveraltracesofvaryingtypesandsizes.
Ourconcisemodelsare4–6%oftheoriginaltracesize,andourmodelingandreplayaccuracyareover90%.
1IntroductionTracesareatime-honoredwaytocollectinformationaboutreal-worldworkloads.
Theinformationcontainedintracesallowsaworkloadtobecharacterizedusingfac-torssuchastheexactsizeandoffsetofeachI/Orequest,read/writeratio,orderingofrequests,etc.
Byreplayingatrace,userscanevaluatereal-worldsystembehavior,optimizeasystembasedonthatbehavior,andcomparetheperformanceofdifferentsystems[21,23,25,34].
Despitethebenetsoftraces,theyarehardtouseinpractice.
Atracecollectedononesystemcannoteasilybescaledtomatchthecharacteristicsofanother.
Itisdif-culttomodifytracessystematically,e.
g.
,bychangingoneworkloadparameterbutleavingallothersconstant.
Tracesarehardtodescribeandcompareintermsthatareeasilyunderstoodbysystemimplementors.
Largetracelesaretime-consumingtodistributeandcanaffectthesystem'sbehaviorduringreplaybypollutingthepagecacheorcausinganI/Obottleneck[20].
Inreviewingrelatedwork,weobservedthatinmanycasesreplayingtheexacttraceisnotrequired.
Instead,itisoftensufcienttouseasyntheticworkloadgener-atorthataccuratelyreproducescertainspecicproper-ties.
Forexample,aparticularsystemmightbemoresensitivetotheread-writeratiothantooperationsize.
Inthissituationonedoesnotreallyneedtoreplaythetraceprecisely;asyntheticworkloadthatemulatesthatread-writeratiowouldsufce.
Ofcourse,thisexampleissimplistic,andinmanycasesonewouldbeinterestedinmorecomplexcombinationsoftheworkloadparame-ters.
However,thegeneralideathatonlysomepropertiesofthetraceaffectsystembehaviorremainsvalid.
Becausemanysystemsrespondonlytoafewpa-rameters,researchershavedevelopedmanybenchmarksandsyntheticworkloadgenerators,suchasIOzone[7],Filebench[12],andIometer[33],whichavoidmanyofthedecienciesoftraces.
Butitcanbedifculttocongureabenchmarksothatitproducesarealisticworkload;simpleonesarenotsufcientlyexible,whilepowerfuloneslikeFilebenchoffersomanyoptionsthatitcanbedauntingtoselectthecorrectsettings.
Inthisworkweproposetollthegapbetweentracesandbenchmarksbyconvertingtracesintothelanguagesofthebenchmarks.
Wefocushereonblocktracesduetotheirrelativesimplicity,butweplantoextendthisworktoothertracetypes,e.
g.
,lesystemandNFS.
Oursystemcreatesauniversalrepresentationofthetrace,expressedasamulti-dimensionalmatrixinwhicheachdimensionrepresentsthestatisticaldistributionofatraceparameterorafunction.
Eachparameterischo-sentorepresentaspecicworkloadproperty.
Weimple-mentedthemostcommonlyusedproperties,suchasI/Osize,inter-arrivaltime,seekdistance,read-writeratio,etc.
Enduserscaneasilyaddnewonesasdesired.
Foreachbenchmark,asmallpluginconvertstheuniversaltracematrixintothespecicbenchmark'slanguage.
Manyworkloadsvarysignicantlyduringthetracingperiod.
Toaddressthisissue,oursystemsupportstracechunkingacrosstime.
Withineachchunk,theworkloadisconsideredtobestableanduniformandisexpressedasaseparatematrix.
Weusechunkdeduplicationtosavespaceinperiodswheretheworkloadisthesame.
Weevaluatedtheaccuracyofoursystembygenerat-ingmodelsfromseveralpubliclyavailabletraces.
Werstreplayedeachtraceonatestsystem,observingthroughput,latency,I/Oqueuelengthandutilization,powerconsumption,requestsizes,CPUandmemoryus-age,andthenumbersofinterruptsandcontextswitches.
Thenweemulatedthetracebyrunningbenchmarkswithgeneratedparametersonthesamesystem,collectedthesameobservations,andcomparedtheresults.
Ourerrorwaslessthan10%onaverage,and15%atmost;itcanbecontrolledbyvaryingseveralparameters.
Forabasicsetofmetrics,weconverteda1.
4GBtracetotheFilebenchlanguageinonly30s.
Theresultingtracedescriptionwas60MB,or23.
3*smaller.
12BackgroundandMotivationStatisticsMatter.
Tracereplayisacommonevalua-tiontechniquebecause,unlikeanyothertestingmethod,bydenitiontracesrepresentreality.
However,thisreal-ismcomesataprice:thetracerepresentsoneinstanceofonesystematonepointintime.
Thenextday'sworkloadwillinevitablybedifferent,aswillthesameworkloadonasystemwithdifferenthardware,competingworkloads,etc.
Intheworstcase,thesevariationsmightcauseasys-temtobeunintentionallyoptimizedforanatypicaloper-atingpoint.
Evenifatraceaccuratelyrepresentsatargetworkload,rapidchangesinhardwareperformancemakeitdifculttoevaluateadesignonamodernmachineus-ingmeasurementsandtracescapturedonadifferentsys-temonlyafewyearsearlier.
Ourkeyobservationisthatformanypurposes,statis-ticsarewhatmatter.
Theexactorderingofoperations,theirprecisetiming,theblocksorlesaccessed,andmanyotherdetailsrecordedinatracearevariableandwouldchangeifitwerere-recorded.
Thus,whenwere-playatrace,wedonotnecessarilywanttoreproduceeverydetailaspreciselyaspossible;instead,wewouldliketoaccuratelyrepresentitsstatisticalproperties.
Anadvantageofthinkingoftracesstatisticallyisthattheybecomemuchmoreexible.
Forexample,atracecollectedadecadeagowouldrecordaccessestoonlyafractionoftheblocksonamoderndisk,andataverydifferentrate.
Comparedtoabulkytrace,astatisticalde-scriptionismuchsimplertoscaletoamodernmachineandthereforeprovidesaconvenientabstractionforper-formingsystematicevaluationofmanysystems.
Generatingagooddescriptionrequiresrepresentativetracepropertiestobeselected.
Ingeneral,themostap-propriatepropertiesdependonthesystembeingtested,soitisimpossibletocreateacompletelist.
Formostpurposes,however,theparametersofinterestarewelldenedandwidelyadopted,e.
g.
,I/Orateanddistribu-tion,read/writeratio.
Thus,astatisticalmodelofatraceshouldbeabletocapturethoseparameters,andshouldbeabletodescribetheminsufcientdetailsothatnoimportantinformationislost.
Inparticular,weshouldnotreducecomplex,empiricallyobserveddistributionstooverlysimplemathematicalmodels,suchasPoissonarrivalprocesses,withoutjustication.
Someworkloadsmayalsoexhibitnonstandard,orevenundiscovered,propertiesthatmightaltersystembehavior.
Itisthereforeadvisabletopreservetheorig-inaltracestoensurethesepropertiesareretained.
Aworkloadgeneratorcanbeadaptedtoincludesuchchar-acteristicsoncetheyareidentied.
SystemResponse.
Toevaluateasystemempirically,workloadsareappliedandappropriatemetricsmeasureitsresponse.
Performanceisoftencharacterizedbythroughput,latency,CPUutilization,I/Oqueuelength,andmemoryusage[39,45].
Powerconsumptioncharac-terizesenergyefciency[29,36].
Inmanypapers,thesemetricsaresummarizedbystatisticssuchasaveragesordistributions.
Butasweargueabove,itisoftenpossibletoaccuratelyevaluatethesemetricswithoutresortingtoafullanddetailedtracereplay.
Ifthesystemresponsetoatraceemula-tionissimilartothatofafullreplay,thenemulationcanreplacefullreplaywithoutbiasingtheresults.
Toevaluatetheaccuracyofourtraceextractionandmodelingsystem,wesurveyedpapersinUsenixFASTconferencesfrom2008–2011andnotedthatthefre-quentlyusedmetricsfellintofourcategories:(1)throughputandlatency;(2)I/OutilizationandaverageI/Oqueuelength;(3)CPUutilizationandmemoryus-age;and(4)powerconsumption.
Mostofthesurveyedpapersincluded1–2ofthesemetrics,butinourstudyweevaluateallfourtypestoensureacomprehensivecom-parison.
Weclaimthatifallresponsemetricsaresimilar,thenthetraceismodeledproperly.
Wefeelthatoursetofmetricsissufcientlyrepresentativeandcomprehen-sivetoproducereliableresults.
Thereisstillachancethatanunmeasuredresponseparametermaydiffer;butoursystemismodularandeasilyextensibletoemulateanyadditionalmetricsonedesires.
ReplayMethods.
Weusesystemresponsetoevaluateourtraceemulationaccuracy.
However,asystem'sre-sponsedependsonthereplaymethod,andvariesbasedonthegoalofthestudy.
Tostudypeakperformance,tracesareoftenaccelerated[31,40,44,48].
Forpowerefciency,tracesareusuallyreplayedverbatimtopre-serverealisticidleperiods[5,9].
Tostressspecicsub-systems,asubsetofthetraceissometimesreplayed[38].
Ourworkloadmodelscanemulateexistingtrace-replaymethodsaswellasmoresophisticatedones.
3DesignOurvedesigngoals,indecreasingpriority,are:1.
Accuracy:Ensurethattracereplayandtraceemu-lationyieldmatchingevaluationresults.
2.
Flexibility:First,leverageexistingpowerfulwork-loadgenerators,ratherthancreatingnewones.
Therefore,tracesshouldbetranslatedintomodelsthatcanbeaccuratelydescribedusingthecapabili-tiesofexistingbenchmarks.
Second,allowuserstochooseanythingfromaccurateyetbulkymodelstosmallerbutlesspreciseones.
3.
Extensibility:Allowthemodeltoincludeaddi-tionalpropertieschosenbytheuser.
4.
Conciseness:Theresultingmodelshouldbemuchsmallerthantheoriginaltrace.
5.
Speed:Thetimetotranslatelargetracesshouldbereasonableevenonamodestmachine.
2FeatureExtraction.
Therststepinourmodel-buildingprocessistoextractimportantfeaturesfromthetrace.
Werstdiscusshowweextractparametersfromworkloadswhosestatisticalcharacteristicsdonotchangeovertime,i.
e.
,stationaryworkloads.
Thenwedescribehowtoemulateanon-stationaryworkload.
Eachblocktracerecordhasasetofeldstodescribetheparametersofagivenrequest.
Fieldsmayincludetheoperationtype,offsetorblocknumber,I/Osize,times-tamp,etc.
Ourtranslatoriseld-oblivious:itconsiderseveryparameterasanumber.
Wedesignatetheseparam-etersasann-dimensionalvectorp=(p1,p2,.
.
.
,pn).
Wedeneafeaturefunctionvectoronp:f=(f1(p,s1),f2(p,s2),.
.
.
,fm(p,sm))=f(p,sf)Eachfeaturefunctionrepresentsananalysisofsomepropertyofthetrace;sirepresentsprivatestatedataforthei-thfeaturefunction,whichletsusdenefeaturesacrossmultipletraceentriesandparameters.
Forexample,assumethatp1andp2representtheI/Osizeandoffsetelds,respectively.
Wecanthendenethesimplefeaturefunctionsf1—justtheI/Osizeitself—andf2—thelogarithmicinter-arrivaldistance(offsetdif-ferencebetweentwoconsecutiverequests):f1=f1(p,s1)=p1f2=f2(p,s2)=log(p2s2.
prevoffset)Inourtranslator,theuserrstchoosesasetofmfea-turefunctions.
Evaluatingthesefunctionsonasingletracerecordresultsinavectorthatrepresentsapointinanm-dimensionalfeaturespace.
Thetranslatordividesthefeaturespaceintobucketsofuser-speciedsize,andcollectsahistogramoffeatureoccurrencesinamulti-dimensionalmatrix—thefeaturematrix—thatexplicitlycapturestherelevantstatisticsoftheworkload,andim-plicitlyrecordstheircorrelations.
Forexample,usingthetwofeaturefunctionsabove,plusathirdthatencodestheoperation(0forreads,1forwrites),theresultingfeaturematrixmightlookliketheoneinFigure1.
Inthiscase,thetraceheld52requestsofsizelessthan4KBandinter-arrivaldistancelessthan1KB;ofthose,38werereadsand14werewrites.
Bychoosingasetoffeaturefunctions,userscanad-justtheworkloadrepresentationtocaptureanyimpor-tanttracefeatures.
Byselectinganappropriatebucketgranularity,userscancontroltheaccuracyoftherepre-sentation,tradingoffprecisionforcomputationalcom-plexityinthetranslatorandmatrixsize.
Stage1inFig-ure2showsthetranslator'sroleintheoveralldesign.
Oncethefeaturematrixhasbeencreated,thetransla-torcanperformanumberofadditionaloperationsonit:projection,summationalongdimensions,computationofconditionalprobabilities,andnormalization.
Theseoperationscanbeusedbythebenchmarkplugins(de-scribedbelow)tocalculateparameters.
Forexample,usingthematrixinFigure1,apluginmightrstsumacrossthedistance-vs.
-sizeplanetocalculatethetotalnumbersofreadsandwrites,normalizethesetondP(read),andthengeneratebenchmarkcodetocondition-alizeI/Osizeontheoperationtype.
Clearly,thechoiceoffeaturefunctionsaffectsthequalityoftheemulation;currentlytheinvestigatormustdothisbasedontheinsightintotheparticularsystemofinterest,e.
g.
,whetherithasbeenoptimizedforcertainworkloadsthatcanbereectedinanappropriatefea-turefunction.
Wehaveimplementedalibraryofoveradozenstandardfeaturefunctionsbasedonthosecom-monlyfoundintheliterature[10,11,26,30],includingoperationtype,I/Osize,offsetdistribution,inter-arrivaldistance,inter-arrivaltime,processidentier,etc.
Newfeaturefunctionscaneasilybeaddedasneededtocap-turespecializedsystemcharacteristics.
BenchmarkPlugins.
Onceafeaturematrixhasbeenconstructedfromatrace,itispossibletouseitdirectlyasinputtoaworkloadgenerator.
However,ourgoalinthisresearchisnottocreateyetanothergenerator.
Instead,webelievethatitisbesttobuildontheworkofothersbyusingexistingworkloadgeneratorsandbenchmarks.
Thisapproachallowsustoeasilyreusealltheexten-sivefacilitiesthatthesebenchmarksprovide.
Manyex-istingbenchmarksofferawaytoconguretheworkloadthattheygenerate;someoffercommand-linecongura-tionparameters(e.
g.
,IOzone[7]andIometer[33])whileothersofferamoreextensivelanguageforthatpurpose(e.
g.
,Filebench[12]ando[13]).
Mostexistingbenchmarksusestatisticalmodelstogenerateaworkload.
Someofthemuseaverageparame-Figure1:Workloadrepresentationusingafeaturematrix3Figure2:OverallSystemDesigntervalues;othersusemorecomplexdistributions.
Inallcases,ourfeaturematricescontainalltheinformationneededtocontrolthemodelsusedbythesebenchmarks.
Asimpleplugintranslatesthefeaturematrixintoaspe-cicbenchmark'sparametersorlanguage.
Forsomebenchmarks,theexpressivenessoftheparametersmightlimittheachievableaccuracy,buteventhenthepluginwillhelpchoosethebestsettingstoemulatetheoriginaltrace'sworkload.
Stage3inFigure2demonstratestheroleofthebenchmarkpluginsintheoveralldesign.
Forourinitialinvestigations,wehaveimplementedpluginsforFilebenchandIOzone.
WechoseFilebenchforitsexibility,andIOzonebecauseitismoresuitableformicro-benchmarking.
Wefoundthatitwaseasytoaddapluginforanewbenchmark,sinceonlyasinglefunctionhastoberegisteredwiththetranslator.
Thesizeofthefunctiondependsonthenumberoffeaturefunctionsandthecomplexityofthetargetbenchmark.
Chunking.
Manyreal-worldtracesarenon-stationary:theirstatisticalcharacteristicsvaryovertime.
Thisises-peciallytruefortracesthatcoverseveralhours,days,orweeks.
However,mostworkloadgeneratorsapplyastationaryload,andcannotvaryitovertime.
Wead-dressthisissuewithtracechunking:splittingatraceintochunksbytime,suchthatthestatisticsofanygivenchunkarerelativelystable.
Findingchunkboundariesisdifcult,sowerstuseaconstantuser-denedchunksize,measuredinseconds.
Foreachchunk,wecomputeafeaturematrixindependently;thisresultsinasequenceofmatrices.
Wethenconvertthesexedchunksintovariable-sizedonesbyfeedingthematricestoadedupli-catorthatmergesadjacentsimilarmatrices(Stage2inFigure2).
Thisoptimizationworkswellbecausemanytracesremainstableforextendedperiodsbeforeshiftingtoadifferentworkloadmode.
Wenormalizethematri-cesbeforecomparingthem,sothattheabsolutenumberofrequestsinachunkdoesnotaffectthecomparison.
Weusethemaximumdistancebetweenmatrixcellsasametricofsimilarity.
Whentwomatricesarefoundtobesimilar,weaveragetheirvaluesandusetheresulttorep-resenttheworkloadsinthecorrespondingtimechunks.
Besidesdetectingvaryingworkloadphases,thededu-plicationprocessalsoreducesthemodelsize.
Toachieveevenfurthercompression,wesupportall-waysdedupli-cation:everychunkinatraceisdeduplicatedagainsteveryotherchunk(notjustadjacentones).
Alongwiththematrices,wegenerateatime-to-matricesmapthatservesasanadditionalinputtothebenchmarkplugins.
Ifthetargetbenchmarkisunabletosupportamulti-phaseworkload,theplugingeneratesmultipleinvocationswithappropriateparameters.
IntheexampleinFigure2,wesetthetracedurationto60sandtheinitialchunksizeto10s,sothetransla-torgeneratedsixmatrices.
Afterall-waysdeduplication,onlytworemained.
4ImplementationTracesfromdifferentsourcesoftenhavedifferentfor-mats.
Wewantedourtranslatortobeefcientandportable.
WechosetheefcientandexibleDataSeriesformat[2]—recommendedbytheStorageNetworkingIndustryAssociation(SNIA)—andweselectedSNIA'sdraftblock-tracesemantics[37].
Wewroteconverterstoallowexperimentationwithexistingtracesinotherformats.
Wealsocreatedablock-tracereplayerforDataSeries,whichsupportsseveralcommonlyusedre-playmodes.
Intotalwewroteabout3,700LoC:1,500inthetranslator,800intheconverters,1,000intheDataSeriesreplayer,and400intheFilebenchandIO-zoneplugins.
Weplantoreleasethesepublicly.
5EvaluationToevaluatetheaccuracy,conversionspeed,andcom-pressionofoursystem,weusedmultiplemicro-benchmarksandavarietyofrealtraces.
Inthispaperwepresentevaluationresultsbasedontwotraces:Fi-nance1[28]andMS-WBS[22].
TheFinance1tracecapturestheactivityofseveralOLTPapplicationsrun-ningattwolargenancialinstitutions.
TheMS-WBStraceswerecollectedfromdailybuildsoftheMicrosoftWindowsServeroperatingsystem.
Thehigh-levelchar-acteristicsofthetracesarepresentedinTable1.
Itisfairtoassumethattheaccuracyofourtransla-tormightdependonthesystemunderevaluation.
Inourexperimentsweusedaspectrumofblockdevices:CharacteristicFinance1MS-WBSDuration12hours1.
5hoursReads/Writes(106)1.
2/4.
10.
7/0.
6AvgI/Osize3.
5KB20KBSeq.
Requests11%47%Table1:High-levelcharacteristicsoftheusedtraces4Figure3:Readsandwritespersecond,SetupP,Fin1trace.
Figure4:Diskpowerconsumption,SetupP,MS-WBStrace.
Figure5:MemoryandCPUusage,SetupP,Fin1trace.
variousdiskdrives,ashdrives,RAIDs,andevenvir-tualblockdevices.
Inthispaperwepresentresultsfromtwoextremesofthespectrum.
Intherstexperimentalsetup—SetupP—weusedaPhysicalmachinewithanexternalSCSISeagateCheetah300GBdiskdrivecon-nectedthroughanAdaptec39320controller.
Thefactthatthedrivewaspoweredexternallyallowedustomea-sureitspowerconsumptionusingaWattsUpmeter[43].
Thesecondexperimentalsetup(SetupV)isanenterprise-classsystemthathasaVirtualmachinerun-ningundertheVMwareESX4.
1Hypervisor.
TheVMaccessesitsvirtualdisksonanNFSserverbackedbyaGPFSparallellesystem[19,35].
TheVMrunsCentOS6.
0;theESXandGPFSserversareIBMSystemx3650's,withGPFSusingaDS4700storagecontroller.
AccuracymetricswererecordedattheNFS/GPFSserver.
Onbothsetups,werstreplayedtracesandthenemu-latedthemusingFilebench.
Inallexperimentswesetthechunksizeto20sandenabledallfeaturefunctions.
Wechosethematrixgranularityforeachdimensionexper-imentally,bygraduallydecreasingituntiltheaccuracybegantodrop.
DuringallrunswecollectedtheaccuracyparametersspeciedinSection2usingtheiostat,vm-stat,andwattsuptools;weplottedgraphsshowingthevalueofeachaccuracyparameterversustimeforbothreplayandemulation.
Duetolimitedspace,weonlypresentthegraphsforafewrepresentativeaccuracypa-rameters.
However,wegivetheaverageandmaximumemulationerrorforallexperiments.
Figure3depictshowthethroughput—forbothreadsandwrites—changeswithtimefortheFinance1trace.
Thereplaywasperformedwithinniteacceleration;ittookabout5hourstocompleteonSetupP.
Thetraceemulationlinecloselyfollowsthereplayline;theRootMeanSquare(RMS)distanceislowerthan6%andthemaximumdistanceisbelow15%.
Inthebeginningoftherun,readthroughputwas4timeshigherthenlaterinthetrace.
Byinspectingthemodelwefoundthattheworkloadexhibitshighsequentialityinthebegin-ningofthetrace.
Afterstartup,thereadthroughputfallsto50–100ops/s,whichisreasonableforanOLTP-likeworkloadandourhardware.
Thewriteperformanceis2–2.
5timeshigherthanforread,duetothecontroller'swrite-backcachethatmakeswritesmoresequential.
Figure4depictsdisk-drivepowerconsumptioninSetupPduringa10-minutenon-acceleratedreplayandemulationoftheMS-WBStrace.
Intherst5min-utestraceactivitywaslow,resultinginlowpowerusage.
Later,aburstofrandomdiskrequestsincreasedpowerconsumptionbyalmost40%.
Theemulationlinedevi-atesfromthereplaylinebyanaverageof6%.
InSetupV,theGPFSserverwascachingrequestscomingfromavirtualmachine.
Asaresult,theruntimeoftheFin1tracewasonly75minutes.
ThememoryandCPUconsumptionoftheGPFSserverduringthistimeareshowninFigure5.
Memoryusagerisessteadily,in-creasingbyabout500MBbytheendoftherun,whichistheworking-setsizeoftheFin1trace.
Discrepanciesbe-tweenreplayandemulationarewithin10%,buttherearevisibledeviationsattimeswhenthememoryusagestepsup.
WeattributethistothecomplexityoftheGPFS'scachepolicy,whichisaffectedbyaworkloadparame-terthatwedidnotemulate.
CPUutilizationremainedsteadilyabout10%forbothreplayandemulation.
Figure6summarizestheerrorsforallparameters,forbothsetupsandtraces.
Themaximumemulationerrorwasbelow15%andRMSdistancewas10%onaverage.
Althoughthemaximumdiscrepancymightseemhigh,Figure3showssufcientbehavioralaccuracy.
Theselectionoffeaturematrixdimensionsisvitalforachievinghighaccuracy.
Ifasystemissensitivetoaworkloadpropertythatismissinginthefeaturematrix,accuracycansuffer.
Forexample,disk-andSSD-basedstoragesystemsmayhaveradicallydifferentqueuingandprefetchingpolicies.
Toensurehigh-delityreplays5(a)SetupP,Fin1trace(b)SetupP,MS-WBStrace(c)SetupV,Fin1trace(d)SetupV,MS-WBStraceFigure6:RootMeanSquare(RMS)andmaximumrelativedistancesofaccuracyparametersfortwotracesandtwosystems.
acrossbothtypesofsystems,thefeaturematrixshouldcapturetheimpactofappropriateparameters.
Thechunksizeandmatrixgranularityalsoaffectthemodel'saccuracy.
Ourgeneralstrategyistoselecttheseparametersliberallyatrst(e.
g.
,100schunksizeand1MBgranularityforI/Osize)andthengraduallyandrepeatedlyrestrictthem(e.
g.
,10schunksize,1KBI/Osize)asneededuntilthedesiredaccuracyisachieved.
Onecanalwaysbeguaranteedtogethighenoughaccu-racyifsufcientlysmallnumbersareused.
ConversionSpeedandModelSize.
Thespeedofconversionandthesizeoftheresultingmodeldependonthetracelengthandthetranslatorparameters.
Onour2.
5GHzserver,traceswereconvertedatabout50MB/s,whichisclosetothethroughputofthe7200RPMdiskdrive.
Theresultingmodelwithoutdeduplicationwasofapproximately10–15%sizeoftheoriginaltrace.
Dedu-plicationremovedover60%ofthechunksinboththeFin1andMS-WBStraces,resultinginanalmodelsizereductionof94–96%.
Allsizesweremeasuredaftercompressingbothtracesandmodelsusinggzip.
6RelatedWorkThebodyofresearchrelatedtotracesislarge;weciteonlyarepresentativesample.
Manystudieshavefo-cusedonaccuratetracecollectionwithminimuminter-ference[1,4,24,31,32].
Otherresearchershavepro-posedtrace-replayingframeworksatdifferentlayersinthestoragestack[3,20,48,48,49].
Sinceatracecontainsinformationabouttheworkloadappliedtothesystem,anumberofworksfocusedontrace-drivenworkloadchar-acterization[22,23,25,34].
N.
Yadwadkarproposedtoidentifyanapplicationbasedonitstrace[46].
Afteraworkloadischaracterized,afewresearchershavesuggestedaworkloadmodelthatallowsthemtogeneratesyntheticworkloadswithidenticalcharacteris-tics[6,14–18,41,42,47].
Theseworksaddressonlyoneortwoworkloadproperties,whereaswepresentagen-eralframeworkforanynumberofproperties.
Also,wechunkdataandgenerateworkloadexpressionsforthelanguagesofalreadyexistingbenchmarks.
ThetwoprojectsmostcloselyrelatedtooursareDis-tiller[27]andChen'sWorkloadAnalyzer[8].
Dis-tiller'smaingoalistoidentifyimportantworkloadprop-erties.
Wecanusethisinformationtointelligentlyde-nedimensionsforourfeaturematrix.
Chenusesma-chinelearningtechniquestoidentifythedependenciesbetweenworkloadfeatures.
However,theauthorsdonotemulatetracesbasedontheextractedinformation.
7ConclusionsandFutureWorkWehavecreatedasystemthatextractsexibleworkloadmodelsfromlargeI/Otraces.
Throughthenoveluseofchunking,wesupporttraceswithtime-varyingstatisticalproperties.
Inaddition,traceextractionistunable,allow-ingmodelaccuracyandsizetobetradedoffagainstcre-ationtime.
ExistingI/Obenchmarkscanreadilyusethegeneratedmodelbyimplementingaplugin.
Oureval-uationwithFilebenchandseveralblocktracesdemon-stratedthattheaccuracyofgeneratedmodelsapproaches95%,whilethemodelsizeislessthan6%oftheoriginaltracesize.
Suchconcisemodelsalloweasycomparison,scalingandothermodications.
Inthefutureweplantosupportle-system-leveltraces,buildmulti-layermodels,andaddexibilityintheanalysisphase.
Ourcurrentchunkingmethodissim-pleandwewanttoinvestigatealternativechunkingtech-niques.
Wewillalsoworkonagraphicaltoolformanualtracechunking.
Toavoidmanualselectionofthetransla-tor'sparameters,wewanttoexplorevariousarticialin-telligenceapproaches.
Tofurtherreducethemodelsize,weplantoimprovethecompressionratiobymatchingempiricaldistributionsinthefeaturematrixtoexplicitmathematicalfunctions.
Werecognizethatourlistofac-curacymetricsisnotcompleteandwanttoexperimentwithotheraccuracyparameters(e.
g.
,latencydistribu-tions).
Wealsoplantodeveloptoolsandtechniquesthatwillsimplifyvariousoperationsonourmodels,suchastimeandsizescaling,andcomparisontoothermodels.
6References[1]E.
Anderson.
Capture,conversion,andanalysisofanintenseNFSworkload.
InProceedingsoftheSeventhUSENIXConferenceonFileandStorageTechnologies(FAST'09),2009.
[2]E.
Anderson,M.
Arlitt,C.
Morrey,andA.
Veitch.
DataSeries:anefcient,exible,dataformatforstructuredserialdata.
ACMSIGOPSOperatingSystemsReview,43(1),January2009.
[3]E.
Anderson,M.
Kallahalla,M.
Uysal,andR.
Swaminathan.
Buttress:AtoolkitforexibleandhighdelityI/Obenchmarking.
InProceed-ingsoftheThirdUSENIXConferenceonFileandStorageTechnologies(FAST'04),2004.
[4]A.
Aranya,C.
P.
Wright,andE.
Zadok.
Tracefs:alesystemtotracethemall.
InProceedingsoftheThirdUSENIXConferenceonFileandStorageTechnologies(FAST'04),2004.
[5]T.
Bisson,S.
A.
Brandt,andD.
D.
E.
Long.
Ahybriddisk-awarespin-downalgorithmwithI/Osubsys-temsupport.
InProceedingsoftheIEEE2007Per-formance,Computing,andCommunicationsCon-ference(IPCCC),2007.
[6]P.
Bodik,A.
Fox,M.
Franklin,M.
Jordan,andD.
Patterson.
Characterizing,modeling,andgener-atingworkloadspikesforstatefulservices.
InPro-ceedingsoftheFirstACMSymposiumonCloudComputing(SOCC),2010.
[7]D.
Capps.
IOzonelesystembenchmark.
www.
iozone.
org.
[8]Y.
Chen,K.
Srinivasan,G.
Goodson,andR.
Katz.
Designimplicationsforenterprisestoragesystemsviamulti-dimensionaltraceanalysis.
InProceed-ingsofthe23rdACMSymposiumonOperatingSystemPrinciples(SOSP'11),2011.
[9]F.
Douglis,P.
Krishnan,andB.
Bershad.
Adaptivediskspin-downpoliciesformobilecomputers.
InProceedingsoftheSecondSymposiumonMobileandLocation-IndependentComputing,1995.
[10]M.
EblingandM.
Satyanarayanan.
SynRGen:Anextensiblelereferencegenerator.
InProceed-ingsofthe1994ACMSIGMETRICSConferenceonMeasurementandModelingofComputerSys-tems,1994.
[11]D.
Ellard,J.
Ledlie,P.
Malkani,andM.
Seltzer.
PassiveNFStracingofemailandresearchwork-loads.
InProceedingsoftheSecondUSENIXCon-ferenceonFileandStorageTechnologies(FAST'03),2003.
[12]Filebench.
http://filebench.
sourceforge.
net.
[13]o—exibleI/Otester.
http://freshmeat.
net/projects/fio/.
[14]A.
Ganapathi,Y.
Chen,A.
Fox,R.
Katz,andD.
Patterson.
Statistics-drivenworkloadmodelingforthecloud.
InProceedingsoftheInternationalWorkshoponInformationandSoftwareasServices(WISS),2010.
[15]G.
Ganger.
Generatingrepresentativesyntheticworkloads:anunsolvedproblem.
InProceed-ingsofComputerMeasurementGroupConference(CMG),1995.
[16]M.
GomezandV.
Santonja.
Anewapproachinthemodelingandgenerationofsyntheticworkloads.
InProceedingsofthe8thSymposiumonMod-eling,AnalysisandSimulationofComputerandTelecommunicationSystems(MASCOTS),2000.
[17]B.
HongandT.
Madhyastha.
Therelevanceoflong-rangedependenceindisktrafcandimpli-cationsfortracesynthesis.
InProceedingsofthe22ndIEEE/13thNASAGoddardConferenceonMassStorageSystemsandTechnologies(MSST),2005.
[18]B.
Hong,T.
Madhyastha,andB.
Zhang.
Cluster-basedinput/outputtraceanalysis.
InProceedingsof24thIEEEInternationalPerformance,Comput-ing,andCommunicationsConference(IPCCC),2005.
[19]IBM.
IBMscaleoutmetworkattachedstorage.
www.
ibm.
com/systems/storage/network/sonas/.
[20]N.
Joukov,T.
Wong,andE.
Zadok.
Accurateandefcientreplayingoflesystemtraces.
InPro-ceedingsoftheFourthUSENIXConferenceonFileandStorageTechnologies(FAST'05),2005.
[21]S.
Kavalanekar,D.
Narayanan,S.
Sankar,E.
Thereska,K.
Vaid,andB.
Worthington.
Measuringdatabaseperformanceinonlineser-vices:atrace-basedapproach.
InProceedingsofTPCTechnologyConferenceonPerformanceEvaluationandBenchmarking(TPCTC),2009.
[22]S.
Kavalanekar,B.
Worthington,Q.
Zhang,andV.
Sharda.
Characterizationofstoragework-loadtracesfromproductionwindowsservers.
InProceedingsofIEEEInternationalSymposiumonWorkloadCharacterization(IISWC),2008.
[23]T.
Kimbrel,A.
Tomkins,R.
Patterson,B.
Bershad,P.
Cao,E.
Felten,G.
Gibson,A.
Karlin,andK.
Li.
Atrace-drivencomparisonofalgorithmsforparal-lelprefetchingandcaching.
InProceedingsoftheSecondSymposiumonOperatingSystemsDesignandImplementation(OSDI1996),1996.
7[24]A.
Konwinski,J.
Bent,J.
Nunez,andM.
Quist.
TowardsanI/Otracingframeworktaxonomy.
InInProceedingsoftheInternationalWorkshoponPetascaleDataStorage(PDSW),2007.
[25]G.
H.
Kuenning,G.
J.
Popek,andP.
Reiher.
Ananalysisoftracedataforpredictivelecachinginmobilecomputing.
InProceedingsoftheSummer1994USENIXConference,1994.
[26]Z.
Kurmas.
GeneratingandAnalyzingSyntheticWorkloadsusingIterativeDistillation.
PhDthesis,GeorgiaInstituteofTechnology,2004.
[27]Z.
Kurmas,K.
Keeton,andK.
Mackenzie.
Synthe-sizingrepresentativeI/Oworkloadsusingiterativedistillation.
InProceedingsofIEEE/ACMInterna-tionalSymposiumonModeling,AnalysisandSim-ulationofComputerTelecommunicationsSystems(MASCOTS),2003.
[28]LASS.
UMasstracepepository.
http://traces.
cs.
umass.
edu.
[29]T.
LiandL.
K.
John.
Run-timemodelingandesti-mationofoperatingsystempowerconsumption.
InProceedingsofthe2003ACMSIGMETRICSCon-ferenceonMeasurementandModelingofCom-puterSystems,2003.
[30]Z.
Li,Z.
Chen,S.
M.
Srinivasan,andY.
Zhou.
C-Miner:Miningblockcorrelationsinstoragesys-tems.
InProceedingsoftheThirdUSENIXConfer-enceonFileandStorageTechnologies(FAST'04),2004.
[31]M.
P.
Mesnier,M.
Wachs,R.
R.
Sambasivan,e.
Lopez,J.
Hendricks,G.
R.
Ganger,andD.
O'Hallaron.
//TRACE:paralleltracereplaywithapproximatecausalevents.
InProceedingsoftheFifthUSENIXConferenceonFileandStorageTechnologies(FAST'07),2007.
[32]R.
Moore.
AuniversaldynamictraceforLinuxandotheroperatingsystems.
InProceedingsofthe2001USENIXAnnualTechnicalConference(ATC),2001.
[33]OSDL.
Iometerproject.
www.
iometer.
org.
[34]J.
Ousterhout,H.
Costa,D.
Harrison,J.
Kunze,M.
Kupfer,andJ.
Thompson.
Atrace-drivenanal-ysisoftheUNIX4.
2BSDlesystem.
InProceed-ingsoftheTenthACMSymposiumonOperatingSystemPrinciples(SOSP),1985.
[35]F.
SchmuckandR.
Haskin.
GPFS:Ashared-disklesystemforlargecomputingclusters.
InPro-ceedingsoftheFirstUSENIXConferenceonFileandStorageTechnologies(FAST'02),2002.
[36]P.
Sehgal,V.
Tarasov,andE.
Zadok.
Evaluatingperformanceandenergyinlesystemserverwork-loadsextensions.
InProceedingsoftheEighthUSENIXConferenceonFileandStorageTech-nologies(FAST'10),2010.
[37]StorageNetworkingIndustryAssociation(SNIA).
BlockI/Otracecommonsemantics(workingdraft).
www.
snia.
org/sites/default/files/BlockIOSemantics-v1.
0r11.
pdf,February2010.
[38]C.
A.
N.
Soules,G.
R.
Goodson,J.
D.
Strunk,andG.
R.
Ganger.
Metadataefciencyinver-sioninglesystems.
InProceedingsoftheSec-ondUSENIXConferenceonFileandStorageTech-nologies(FAST'03),2003.
[39]A.
Traeger,N.
Joukov,C.
P.
Wright,andE.
Zadok.
ANineYearStudyofFileSystemandStorageBenchmarking.
ACMTransactionsonStorage(TOS),4(2):25–80,May2008.
[40]B.
Trushkowsky,P.
Bodik,A.
Fox,M.
J.
Franklin,M.
I.
Jordan,andD.
A.
Patterson.
TheSCADSdi-rector:scalingadistributedstoragesystemunderstringentperformancerequirements.
InProceed-ingsoftheNinethUSENIXConferenceonFileandStorageTechnologies(FAST'11),2011.
[41]M.
Wang,A.
Ailamaki,andC.
Faloutsos.
Captur-ingthespatio-temporalbehaviorofrealtrafcdata.
InProceedingsofPerformance,2002.
[42]M.
Wang,T.
Madhyastha,N.
Chan,andS.
Pa-padimitriou.
Dataminingmeetsperformanceeval-uation:fastalgorithmsformodelingbursttrafc.
InProceedingsof16thInternationalConferenceonDataEngineering(ICDE),2002.
[43]WattsupPROESPowerMeter.
www.
wattsupmeters.
com/secure/products.
php.
[44]C.
Weddle,M.
Oldham,J.
Qian,A.
A.
Wang,P.
Reiher,andG.
Kuenning.
PARAID:agear-shiftingpower-awareRAID.
InProceedingsoftheFifthUSENIXConferenceonFileandStorageTechnologies(FAST'07),2007.
[45]S.
Wu,H.
Jiang,D.
Feng,L.
Tian,andB.
Mao.
WorkOut:I/OworkloadoutsourcingforboostingRAIDreconstructionperformance.
InProceedingsoftheSeventhUSENIXConferenceonFileandStorageTechnologies(FAST'09),2009.
[46]N.
Yadwadkar,C.
Bhattacharyya,andK.
Gopinath.
Discoveryofapplicationworkloadsfromnet-workletraces.
InProceedingsoftheEighthUSENIXConferenceonFileandStorageTech-nologies(FAST'10),2010.
8[47]J.
Zhang,A.
Sivasubramaniam,H.
Franke,N.
Gau-tam,Y.
Zhang,andS.
Nagar.
Synthesizingrepre-sentativeI/OworkloadsforTPC-H.
InProceed-ingsofInternationalSypmposiumonHighPerfor-manceComputerArchitecture(HPCA),2004.
[48]N.
Zhu,J.
Chen,andT.
Chiueh.
TBBT:scalableandaccuratetracereplayforleserverevalua-tion.
InProceedingsoftheFourthUSENIXConfer-enceonFileandStorageTechnologies(FAST'05),2005.
[49]N.
Zhu,J.
Chen,T.
Chiueh,andD.
Ellard.
AnNFStraceplayerforlesystemevaluation.
TechnicalReportTR-14-03,HarvardUniversity,December2003.
9