infrastructuremanageengine

ProposalonNetwork-WideRollbackSchemeforFastRecoveryfromOperatorErrorsKiyohitoYoshihara,DaisukeArai,AkiraIdoue,andHirokiHoriuchiKDDIR&DLaboratoriesInc.
,2-1-15OharaFujimino-shiSaitama356-8502,Japan{yosshy,di-arai,idoue,hr-horiuchi}@kddilabs.
jpAbstract.
Thispaperproposesanewnetwork-widerollbackschemeforfastrecoveryfromoperatorerrors,towardthehighavailabilityofnet-worksandservices.
Atechnicalissuearisesfromthefactthatoperators,whomanipulateoneormorediversedevicesandservicesduetotheirnetwork-widedependencyinatypicalmanagementtask,arethemajorcauseoffailure.
Thelackofsystemsortoolsfullyaddressingtheissuemotivatedustodevelopanewscheme.
Theunderlyingideaisthat,foranyoperationaldeviceorservice,theobservablebehaviorisidenticalwheneverthesamesettingiscongured.
Highavailabilitywillthusbeachievedbyrollingthesettingsthatmaycauseanabnormalstatebyanoperatorerror,backtopastoneswithwhichdevicesandserviceswerestable.
Certainpoliciesforthenetwork-widerollbackareidentiedandaprototypeimplementationandpreliminaryresultswillbepresented.
1IntroductionAsseenfromitsglobalacceptance,theInternetisbecomingasanotherformofpromisinginfrastructurelikeelectricity,waterandgas.
Atthesametime,theever-increasingscaleofthenetworksconstitutedbydiversedevicesandservicesentailsadditionalopportunitiesfornetworkoperations.
Weoperateoneormoredevicesandservicesinatypicalmanagementtask,duetotheirnetwork-widedependency.
Forinstance,whenweinstallanewWebserver,wecongurearouter,theDNSserverandarewallinorder,aswellastheWebserver.
Incontrast,itisreportedin[1]thatoperatorsaretheleadingcauseoffail-ure,accountingfor51%,whichwasroughlyestimatedbasedonthenumberofidentiedoutagesinthreeInternetsitesof500to5000computers.
Inordertomakethenetworksandservicesdependableenoughtobeusedwithavailabilityequivalenttoexistinginfrastructures,thenewtechnicalissuearisingistoenablefastrecoveryfromoperatorerrorswhileconsideringthediversityofdevicesandservicesaswellastheirnetwork-widedependency.
Certainsystems[2,3]havebeendevelopedtoaddresstheissue,whichpre-supposeerrorsonthepartoftheoperator,provideundoutilitiesthatallowtheoperatortorollaservicebacktoapreviousstate,andminimizetheMeanTimeToRepair(MTTR)forhigheravailability;however,thesystemstakeintoac-countneitherthediversityofservicesnortheirdependency.
ThedevelopmentA.
Clemm,L.
Z.
Granville,andR.
Stadler(Eds.
):DSOM2007,LNCS4785,pp.
199–202,2007.
cIFIPInternationalFederationforInformationProcessing2007200K.
Yoshiharaetal.
ofadditionalsoftwarecomponentsforeachserviceisrequiredfortheadoption.
Moreover,theiroperationalscopeisrestrictedtoasingleservice,andtheresult-ingstateisnotnecessarilyconsistentwiththatafterpastmanagementtasks.
Wemightexploittools[4,5]formulti-vendorroutersandswitches;however,theyarethesameasthesystems[2,3]intermsofthelimitedoperationalscope.
Thispaperproposesanewnetwork-widerollbackschemeforfastrecoveryfromoperatorerrors,asworkinprogress.
Theproposedschemewillbedieren-tiatedby(1)thepracticalitybywhichwecandeploytheproposedschemewithnomodicationtoexistingdiversedevicesandservicesand(2)thenetwork-widecongurationmanagement,viawhichwecanrollanentiremanagednetworkbacktoasafestate,consistentlywithpastmanagementtasks.
Inthesubsequentsections,wewillinitiallypresenttheproposedscheme,togetherwithsomepoli-ciesspecictonetwork-widerollback,andthenshowthecurrentstateoftheprototypeimplementationandpreliminaryresults.
2ProposalonNetwork-WideRollbackSchemeforFastRecoveryfromOperatorErrors2.
1DesignPrincipleTheideaoftheproposedschemecomesfromastraightforwardfact:foranyoperationaldeviceorservice,itsobservablebehaviorisidenticalwheneverthesamesettingiscongured.
Thus,wecanexpectfastrecoveryfromoperatorerrorswheninanabnormalstateafteramanagementtask,byrollingthecurrentsettingsbacktopastones,withwhichmanageddevicesandserviceswereinastablestate.
Weextendthisideatoanetworkwiththedesignprinciplesbelow.
1.
Wedevelopanewservertoachievenetwork-widerollback.
Theserverhooksallcommandrequeststoandresponsesfrommanageddevicesandservicesduringamanagementtask,andbackupsallsettingsofthedevicesandser-vices,includingthosethatwerenotoperatedwheneverataskiscompleted.
AnIDisgiven,alongwithbackuptimeandanassociablecomment,inordertoidentifyastateinthenetwork-widerollback.
2.
Foradeviceandservice,wedescribetwoscriptsforthebackupandrollbackofsetting,andregisterthemaswellasthehuman-readablename,IPaddressandlogincredentialwiththeserver.
Runningoneormorescriptscanseeanetworkofdiversedevicesandservicesrolledbacktoastablestate,withnomodicationtothedevicesandservices.
3.
Tocopewithaddition/deletionofadeviceandservice,wedenecompensa-tionpoliciesasretainingastatestablyafterthenetwork-widerollbackoversuchchanges.
Asimpleexampleofthepolicyistoskiptherollbackforadevicenotexistedbefore.
Theresultingstateshouldbeconsistentbutmaynotalwaysbetheoneoccurredinthepastmanagementtasks.
4.
Wecannotalwaysrollallnecessarysettingsbackatwill,duetoatransientstateduringtherollback:e.
g.
inabilitytoreachamanageddeviceafewhopsawayfromtheserver,viathechangeofaroutingtableofarouteronthepath.
Withthisinmind,wedeneorderingpoliciestoavoidsuchside-eects.
ProposalonNetwork-WideRollbackScheme201(4)AlarmNotificationacktoPastSettings(3)BackupSettingsofDevicesandServicesforEachManagementTaskT=t0T=t2T=t1Time(T)versusxyz123StableSettingsatTmanageengine.
adventnet.
com/products/device-expert/index.
html