maintainsphysicalmemory

FinalMemoryOvercommit--WithouttheCommitment(XenSummit2008)Page1of3MemoryOvercommit…withoutthecommitmentDanMagenheimer,OracleCorp.
ExtendedAbstractforXenSummit,June2008IntroductionValuableresourcesinphysicalserversareoftenunderutilizedandvirtualizationcanbeemployedtoconsolidatemultiplephysicalserverssothatresourcesaremorefullyutilized.
IfaVMMcreatestheillusionthattherearemoreinstances(orbandwidth)ofaresourcethanactuallyphysicallyavailable,thatresourceisconsideredtobeovercommitted(oroversubscribed).
Forexample,ifthreeuniprocessorserversarevirtualized(asguests)andconsolidatedontoonedual-processorserver,theCPUresourceisconsideredovercommitted.
IfallthreeguestseachneedtorunworkloadsthatalwaysrequireanentireCPU,theseguestswouldbeconsideredpoorcandidatesforvirtualization.
But,virtualizationworksmostofthetimebecausemostresourcesonmanyphysicalserversarebadlyunderutilized.
Onekeyresourcehoweverisanotableexception:physicalmemory.
UnlikeCPUsandI/Ocards,anoperatingsystemonaphysicalsystemgenerallyutilizesitsphysicalmemoryfully.
Pagesofmemorythatarenotusedforkernel/applicationcodeordataareusedtocachediskdata.
Sincedisksareextremelyslowanditiscommonforthesamediskdatatobeusedagainandagain,awell-managedcacheofdiskpagescandramaticallyimproveperformanceofmanyworkloads.
Consequently,whenphysicalserversareconsolidated,itisconsideredunwisetoovercommitphysicalmemory.
Indeed,Xendoesnotsupportmemoryovercommitmentatall.
Forexample,ifonewishestoconsolidatemultiple1GBguestsona4GBXenserver,onlythreeguestscanbeaccomodated.
MemoryOvercommitmentElsewhereSomeVmwareproductsoptionallysupportmemoryovercommit.
Usingblack-boxmechanismsdescribedin[10],uptofivetimesasmanyguestscanberunwhenmemoryovercommitisenabled.
Inarecentblogposting[4],VmwareexploitsthiscapabilitytoarguethattheirproductsarelessexpensivethanXenorMicrosoftVirtualServerasmeasuredby"cost-per-virtual-machine".
Manyresponsestothisposting,argued:Realworldworkloadsdon'tbenefitMemoryischeap:justaddmorephysicalmemoryVmwaredoesn'trecommendturningitonanywayMigrationdoesn'tworkwithmemoryovercommitmentButthetruthisthatmanyworkloadsdobenefitfrommemoryovercommit,additionalmemoryisn'tapanacaea,Microsoftisworkingonit[8],andsoiskvm[5].
SowhynotXenMakingitmaximallyefficientisveryhardand(evenwithVmware)canleadtostabilityissues,functionalitylimitations,andsomelossofperformance.
ButcanXengetmostofthebenefit,withoutincurringthesecostsMemoryOvercommitmentMechanismsWaldspurgerin[10]describesthreeVMwaretechniques:Ballooningutilizesaspecialdriverthat"inflates"byrequestingmemoryfromthekernelandreturningittotheVMM.
Xenprovidesafunctioningballoondriverandsomelimitedtoolsforinteractingwithitbutnomanagement.
Wewillreturntoballooningshortly.
Content-basedpagesharingtransparentlydiscoversidenticalpagesandcombinesthem,usingcopy-on-writetoseparatethemagainwhennecessary.
ThishasbeenexploredonXenin[6]and[7].
Itiscomplextoimplement,haslimitedbenefitinnon-idledomains,andhasaperformanceimpact.
Indemandpaging,theVMMmaintainsaswapareaandcanpageoutmemorywithoutguestinvolvement.
Becauseoftheso-called"semanticgap"[2],aVMM-basedpagereplacementalgorithmcan'tbeefficientandcanleadtodoublepaging.
SinceXenhasnoI/Ocapabilityinthehypervisor,domain0wouldneedtomanageasystem-wideswaparea,whichleadstoevenmorecomplexityandinefficiency.
[9]suggeststhatmemoryhotplugcouldbeusedasanunderlyingmechanismand[3]suggeststhatentireguestsmightbeswappedout--justasentireprocesseswereswappedinearlyoperatingsystems;guestsareswappedinwhentriggeredbysomethinglikewake-on-LAN.
IsXenBallooningEnoughJustaswithCPUsandI/O,thephysicalmemoryinmostphysicalserversmayalsooftenbeunderutilized.
WhenaserverhasidleCPUs,muchofphysicalmemoryisholdingstaleapplicationdataordiskpagesthatmayneverbeusedagain.
Today'soperatingsystemsdoafinejobofoptimizingmemoryusagebyretainingpagesthatmightbeusedagainbutnooperatingsystemcanaccuratelypredictthefuture,sosome--andinmanycasesmost--ofitsguessesarewrong.
Asaresult,muchofthephysicalmemoryisidlememorythatisjusttakingupspaceandFinalMemoryOvercommit--WithouttheCommitment(XenSummit2008)Page2of3won'tbeusedagain.
Inotherwords,physicalmemorymaybefullyutilizedbutitisoftennotefficientlyutilized.
Aballoondriver,whenproperlymanaged,complementsnicelywiththeLinuxkernel'spagereplacementmechanisms.
Whentheballoonisinflatedbyonepage,thekernelsurrendersthepagethatitbelievesisleastlikelytobeusedagain.
Iftheballooncanbeinflateduntiljustbeforetheguest"hurts"formemory,theguest'sidlememorywillbeminimized,andXencanusethatotherwiseidlememoryforanotherguest.
Whenaguestneedsmorememory,itmustbeabletoquicklydeflatetheballoon;butifadditionalmemoryisunavailable,aproperlyconfiguredLinuxguestswapspagestodisk,justasifitreachedafixedlimitofphysicalmemory.
Thenwhenmemorylaterbecomesavailable,Linuxcanmakeuseofthatmemorytoswappagesinfromdiskifitneedsto.
In[11],gray-boxdataiscollectedtomanagemigrationstrategies.
Byusingasimilarapproachtoballooning,dataprovidedbyLinuxitselfcanbeusedtocontrolballooninflationanddeflation.
Specifically,theCommitted_ASstatisticin/proc/meminfoprovidesareasonableestimateofmemory"need"thatisconservative,reasonablyaccurate,anddynamic.
Byusingthisvaluetoinflateanddeflatetheballoonautomatically,aprocesswecallselfballooning,idlememoryappearstoberoughlyminimized.
Whataboutout-of-memory(OOM)conditionsAssumingLinuxhasbasicallyenoughmemoryand/orisgivenenoughtimetoadapttolow-memoryconditions,OOMsdon'thappen.
Aminimummemoryvalue(varyingdependingontotalmemoryavailable)wasrecentlyadded[1]totheXenballoondrivertoensuretheballoondoesn'tinflatetoomuch.
Andincludinghysteresisintheballooningalgorithm,ballooninflationcanberate-limitedsothatmemoryisneverreducedtooquicklyforLinuxtoadapt.
ImplementationAtfirst,wetriedmakingchangesintheXenballoondriveritself.
Becausesomekeymemorystatisticvariablesarenotexportedfromthekernel(eventhoughtheyareavailablein/proc),wedecidedtomoveallballoonmanagementintouser-mode.
Weimplemented,xenballoond,aguest-residentservice(daemon)writtenentirelyinbashscriptthatsupportsbothselfballooningand"directedballooning".
Forselfballooning,theguestneedonlyhavetheballoondriverinstalledeitherin-kernel(paravirtualized)orasakernelmodule(hardwarevirtualized).
Fordirectedballooning,theguestmustalsohaveafewxenstoretoolsinstalled;inthiscase,memorystatistics(proc/meminfoand/proc/vmstat)arecommunicatedviaxenbustodomain0andballoonsizecanbecalculatedandcontrolledbydomain0.
Wehavenotyetimplementedadirectedballooningmanagementtoolindomain0asselfballooningseemstobeworkingverynicelysofar.
TestingTotestxenballoond,wewroteasyntheticworkloadcalled"eatmem"thatmalloc'sarandom-sizedchunkofmemory(upto512MB)andthenwritestoeachpagerepeatedlyarandomnumberoftimes,thenfreesthememoryandsleepsforarandomnumberofseconds(upto63).
Thenthissequenceisrepeatedforeverandweuse/etc/rc.
localtoinvokethisworkloadautomaticallyforthelifetimeofthedomain.
Wealsomodifiedeatmemtousexenbustocommunicatehowlargethechunkofmemoryissothatwecanmonitoritfromdomain0.
Notethateachguestisassumedtobeconfiguredwithanadequatelysized(virtual)swapdisk.
Onthedomain0side,wewroteascripttolaunchdomainssequentially,watchingmemoryusagetodeterminewhenitisOKtolaunchanotherone.
Thenwewroteanotherscripttoprintcertainxenbuskeysforeachdomainandweusethe"watch"command(withdifferenceshighlighted)toprovideaxentop-likedisplaytomonitorthesevalues.
(SeeFigure1.
)Guestscanbeobserveddoingsomeswappingwheneatmemsuddenlyrequirestheballoontoberapidlydeflated,butthephenomenomistransientandinfrequent.
Thoughthisisfarfromanindustrial-strengthtestscaffold,itwassufficienttoverifythatselfballooningworks,thatmetricscanbemonitoredfromdomain0,andthatLinuxwascapableofhandlingrandomdynamiclargeinflationanddeflationoftheballoon.
Weranthisontwodifferent2GBmachinesrunningOracleVM(oneanolderlaptop)andoneachsuccessfullylaunchedsevenpvmdomainswhichwereconfiguredwithmemory=512andmaxmem=1024.
Onanothertestwithmoreaggressiveballooninflationassumptions,wewereabletolaunchfourteen512MBidledomains.
ConclusionsWhilecontent-basedpage-sharing,VMM-baseddemandpaging,andhotplugmemoryareallglamorousmechanismsthatcanbeusedtoimprovememoryefficiency,thesimpleexistingballoondriverprovidedbyXen,whencombinedwithgray-boxdatacollectedbyafewscripts,issufficienttoimplementreasonablememoryovercommit.
MoremeasurementandtestingisongoinginOracle'sOnDemandgroup,butwebelievethatthisverysimplesolutiondeliversthevastmajorityofthevalueofmemoryovercommitwithamuchsmallercost.
FinalMemoryOvercommit--WithouttheCommitment(XenSummit2008)Page3of3References1.
J.
Beulich,[PATCH]linux/balloon:don'tallowballooningdownadomainbelowareasonablelimit,,Xendevelopersarchive,http://lists.
xensource.
com/archives/html/xen-devel/2008-04/msg00143.
html2.
P.
Chenetal.
WhenVirtualisBetterthanReal.
InProceedingsHOTOS'01.
http://www.
eecs.
umich.
edu/~pmchen/papers/chen01.
pdf3.
K.
Fraser,RE:MemoryOvercommit,Xendevelopersarchivehttp://lists.
xensource.
com/archives/html/xen-devel/2005-12/msg00409.
html4.
E.
Horschman,CheapHypervisors:AFineIdea--Ifyoucanaffordthem,blogposting,http://blogs.
vmware.
com/virtualreality/2008/03/cheap-hyperviso.
html5.
A.
Kivity,Memoryovercommitwithkvm,http://avikivity.
blogspot.
com/2008/04/memory-overcommit-with-kvm.
html6.
J.
Klosteretal,Onthefeasibilityofmemorysharing:contentbasedpagesharinginthexenvirtualmachinemonitor,TechnicalReport,2006.
http://www.
cs.
aau.
dk/library/files/rapbibfiles1/1150283144.
pdf7.
G.
Milos,MemoryCOWinXen,PresentationatXenSummit,Nov2007.
http://www.
xen.
org/files/xensummit_fall2007/18_GregorMilos.
pdf8.
J.
Oneill,ExpensiveHypervisors,abadideaevenifyoucanaffordthem.
blogpostinghttp://blogs.
technet.
com/jamesone/archive/2008/03/13/expensive-hypervisors-a-bad-idea-even-if-you-can-afford-them.
aspx9.
J.
Schoppetal,ResizingMemorywithBalloonsandHotplug,OttawaLinuxSymposium2006,https://ols2006.
108.
redhat.
com/reprints/schopp-reprint.
pdf10.
C.
Waldspurger.
MemoryResourceManagementinVmwareESXServer,InProceedingsOSDI'02,http://www.
usenix.
org/events/osdi02/tech/walspurger/waldspurger.
pdf11.
T.
Wood,etal.
Black-boxandGray-boxStrategiesforVirtualMachineMigration,InProceedingsNSDI'07.
http://www.
cs.
umass.
edu/~twood/pubs/NSDI07.
pdfFigure1.
Memory-overcommitedmachineshowingmemoryusagemonitorforselfballoonedguests