illustratedfreehost
freehost 时间:2021-04-10 阅读:()
UnderstandingMemoryResourceManagementinVMwareESXServerWHITEPAPER2VMwarewhitepaperTableofContents1.
Introduction32.
ESXMemoryManagementOverview42.
1Terminology42.
2MemoryVirtualizationBasics42.
3MemoryManagementBasicsinESX53.
MemoryReclamationinESX63.
1Motivation63.
2TransparentPageSharing(TPS)73.
3Ballooning83.
4HypervisorSwapping93.
5WhentoReclaimHostMemory104.
ESXMemoryAllocationManagementforMultipleVirtualMachines115.
PerformanceEvaluation135.
1ExperimentalEnvironment135.
2TransparentPageSharingPerformance145.
3Ballooningvs.
Swapping145.
3.
1LinuxKernelCompile155.
3.
2Oracle/Swingbench165.
3.
3SPECjbb175.
3.
4MicrosoftExchangeServer2007186.
BestPractices197.
References193VMwarewhitepaper1.
IntroductionVMwareESXisahypervisordesignedtoefficientlymanagehardwareresourcesincludingCPU,memory,storage,andnetworkamongmultipleconcurrentvirtualmachines.
ThispaperdescribesthebasicmemorymanagementconceptsinESX,theconfigurationoptionsavailable,andprovidesresultstoshowtheperformanceimpactoftheseoptions.
Thefocusofthispaperisinpresentingthefundamentalconceptsoftheseoptions.
Moredetailscanbefoundin"MemoryResourceManagementinVMwareESXServer"[1].
ESXuseshigh-levelresourcemanagementpoliciestocomputeatargetmemoryallocationforeachvirtualmachine(VM)basedonthecurrentsystemloadandparametersettingsforthevirtualmachine(shares,reservation,andlimit[2]).
Thecomputedtargetallocationisusedtoguidethedynamicadjustmentofthememoryallocationforeachvirtualmachine.
Inthecaseswherehostmemoryisovercommitted,thetargetallocationsarestillachievedbyinvokingseverallower-levelmechanismstoreclaimmemoryfromvirtualmachines.
Thispaperassumesapurevirtualizationenvironmentinwhichtheguestoperatingsystemrunninginsidethevirtualmachineisnotmodifiedtofacilitatevirtualization(oftenreferredtoasparavirtualization).
KnowledgeofESXarchitecturewillhelpyouunderstandtheconceptspresentedinthispaper.
Inordertoquicklymonitorvirtualmachinememoryusage,theVMwarevSphereClientexposestwomemorystatisticsintheresourcesummary:ConsumedHostMemoryandActiveGuestMemory.
Figure1:HostandGuestMemoryusageinvSphereClientConsumedHostMemoryusageisdefinedastheamountofhostmemorythatisallocatedtothevirtualmachine,ActiveGuestMemoryisdefinedastheamountofguestmemorythatiscurrentlybeingusedbytheguestoperatingsystemanditsapplications.
Thesetwostatisticsarequiteusefulforanalyzingthememorystatusofthevirtualmachineandprovidinghintstoaddresspotentialperformanceissues.
Thispaperhelpsanswerthesequestions:WhyistheConsumedHostMemorysohighWhyistheConsumedHostMemoryusagesometimesmuchlargerthantheActiveGuestMemoryWhyistheActiveGuestMemorydifferentfromwhatisseeninsidetheguestoperatingsystemThesequestionscannotbeeasilyansweredwithoutunderstandingthebasicmemorymanagementconceptsinESX.
UnderstandinghowESXmanagesmemorywillalsomaketheperformanceimplicationsofchangingESXmemorymanagementparametersclearer.
ThevSphereClientcanalsodisplayperformancechartsforthefollowingmemorystatistics:active,shared,consumed,granted,overhead,balloon,swapped,swappedinrate,andswapped-outrate.
Acompletediscussionaboutthesemetricscanbefoundin"MemoryPerformanceChartMetricsinthevSphereClient"[3]and"VirtualCenterMemoryStatisticsDefinitions"[4].
Therestofthepaperisorganizedasfollows.
Section2presentstheoverviewofESXmemorymanagementconcepts.
Section3discussesthememoryreclamationtechniquesusedinESX.
Section4describeshowESXallocateshostmemorytovirtualmachineswhenthehostisundermemorypressure.
Section5presentsanddiscussestheperformanceresultsfordifferentmemoryreclamationtechniques.
Finally,Section6discussesthebestpracticeswithrespecttohostandguestmemoryusage.
4VMwarewhitepaper2.
ESXMemoryManagementOverview2.
1TerminologyThefollowingterminologyisusedthroughoutthispaper.
Hostphysicalmemory1referstothememorythatisvisibletothehypervisorasavailableonthesystem.
Guestphysicalmemoryreferstothememorythatisvisibletotheguestoperatingsystemrunninginthevirtualmachine.
Guestvirtualmemoryreferstoacontinuousvirtualaddressspacepresentedbytheguestoperatingsystemtoapplications.
Itisthememorythatisvisibletotheapplicationsrunninginsidethevirtualmachine.
Guestphysicalmemoryisbackedbyhostphysicalmemory,whichmeansthehypervisorprovidesamappingfromtheguesttothehostmemory.
Thememorytransferbetweentheguestphysicalmemoryandtheguestswapdeviceisreferredtoasguestlevelpagingandisdrivenbytheguestoperatingsystem.
Thememorytransferbetweenguestphysicalmemoryandthehostswapdeviceisreferredtoashypervisorswapping,whichisdrivenbythehypervisor.
2.
2MemoryVirtualizationBasicsVirtualmemoryisawell-knowntechniqueusedinmostgeneral-purposeoperatingsystems,andalmostallmodernprocessorshavehardwaretosupportit.
Virtualmemorycreatesauniformvirtualaddressspaceforapplicationsandallowstheoperatingsystemandhardwaretohandletheaddresstranslationbetweenthevirtualaddressspaceandthephysicaladdressspace.
Thistechniquenotonlysimplifiestheprogrammer'swork,butalsoadaptstheexecutionenvironmenttosupportlargeaddressspaces,processprotection,filemapping,andswappinginmoderncomputersystems.
Whenrunningavirtualmachine,thehypervisorcreatesacontiguousaddressablememoryspaceforthevirtualmachine.
Thismemoryspacehasthesamepropertiesasthevirtualaddressspacepresentedtotheapplicationsbytheguestoperatingsystem.
Thisallowsthehypervisortorunmultiplevirtualmachinessimultaneouslywhileprotectingthememoryofeachvirtualmachinefrombeingaccessedbyothers.
Therefore,fromtheviewoftheapplicationrunninginsidethevirtualmachine,thehypervisoraddsanextralevelofaddresstranslationthatmapstheguestphysicaladdresstothehostphysicaladdress.
Asaresult,therearethreevirtualmemorylayersinESX:guestvirtualmemory,guestphysicalmemory,andhostphysicalmemory.
TheirrelationshipsareillustratedinFigure2(a).
Figure2:Virtualmemorylevels(a)andmemoryaddresstranslation(b)inESX(a)VM(b)GuestvirtualmemoryApplicationOperatingSystemHypervisorHypervisorGuestphysicalmemoryHostphysicalmemoryGuestOSPageTablesguestvirtual-to-guestphysicalShadowPageTablesguestvirtual-to-guestphysicalpmapguestphysical-to-hostphysicalAsshowninFigure2(b),inESX,theaddresstranslationbetweenguestphysicalmemoryandhostphysicalmemoryismaintainedbythehypervisorusingaphysicalmemorymappingdatastructure,orpmap,foreachvirtualmachine.
Thehypervisorinterceptsallvirtualmachineinstructionsthatmanipulatethehardwaretranslationlookasidebuffer(TLB)contentsorguestoperatingsystempagetables,whichcontainthevirtualtophysicaladdressmapping.
TheactualhardwareTLBstateisupdatedbasedontheseparateshadowpagetables,whichcontaintheguestvirtualtohostphysicaladdressmapping.
Theshadowpagetablesmaintainconsistencywiththeguestvirtualtoguestphysicaladdressmappingintheguestpagetablesandtheguestphysicaltohostphysicaladdress1Thetermshostphysicalmemoryandhostmemoryareusedinterchangeablyinthispaper.
Theyarealsoequivalenttothetermmachinememoryusedin[1].
5VMwarewhitepapermappinginthepmapdatastructure.
Thisapproachremovesthevirtualizationoverheadforthevirtualmachine'snormalmemoryaccessesbecausethehardwareTLBwillcachethedirectguestvirtualtohostphysicalmemoryaddresstranslationsreadfromtheshadowpagetables.
Notethattheextralevelofguestphysicaltohostphysicalmemoryindirectionisextremelypowerfulinthevirtualizationenvironment.
Forexample,ESXcaneasilyremapavirtualmachine'shostphysicalmemorytofilesorotherdevicesinamannerthatiscompletelytransparenttothevirtualmachine.
Recently,somenewgenerationCPUs,suchasthirdgenerationAMDOpteronandIntelXeon5500seriesprocessors,haveprovidedhardwaresupportformemoryvirtualizationbyusingtwolayersofpagetablesinhardware.
Onelayerstorestheguestvirtualtoguestphysicalmemoryaddresstranslation,andtheotherlayerstorestheguestphysicaltohostphysicalmemoryaddresstranslation.
Thesetwopagetablesaresynchronizedusingprocessorhardware.
Hardwaresupportmemoryvirtualizationeliminatestheoverheadrequiredtokeepshadowpagetablesinsynchronizationwithguestpagetablesinsoftwarememoryvirtualization.
Formoreinformationabouthardware-assistedmemoryvirtualization,see"PerformanceEvaluationofIntelEPTHardwareAssist"[5]and"PerformanceEvaluationofAMDRVIHardwareAssist.
"[6]2.
3MemoryManagementBasicsinESXPriortotalkingabouthowESXmanagesmemoryforvirtualmachines,itisusefultofirstunderstandhowtheapplication,guestoperatingsystem,hypervisor,andvirtualmachinemanagememoryattheirrespectivelayers.
Anapplicationstartsandusestheinterfacesprovidedbytheoperatingsystemtoexplicitlyallocateordeallocatethevirtualmemoryduringtheexecution.
Inanon-virtualenvironment,theoperatingsystemassumesitownsallphysicalmemoryinthesystem.
Thehardwaredoesnotprovideinterfacesfortheoperatingsystemtoexplicitly"allocate"or"free"physicalmemory.
Theoperatingsystemestablishesthedefinitionsof"allocated"or"free"physicalmemory.
Differentoperatingsystemshavedifferentimplementationstorealizethisabstraction.
Oneexampleisthattheoperatingsystemmaintainsan"allocated"listanda"free"list,sowhetherornotaphysicalpageisfreedependsonwhichlistthepagecurrentlyresidesin.
Becauseavirtualmachinerunsanoperatingsystemandseveralapplications,thevirtualmachinememorymanagementpropertiescombinebothapplicationandoperatingsystemmemorymanagementproperties.
Likeanapplication,whenavirtualmachinefirststarts,ithasnopre-allocatedphysicalmemory.
Likeanoperatingsystem,thevirtualmachinecannotexplicitlyallocatehostphysicalmemorythroughanystandardinterfaces.
Thehypervisoralsocreatesthedefinitionsof"allocated"and"free"hostmemoryinitsowndatastructures.
Thehypervisorinterceptsthevirtualmachine'smemoryaccessesandallocateshostphysicalmemoryforthevirtualmachineonitsfirstaccesstothememory.
Inordertoavoidinformationleakingamongvirtualmachines,thehypervisoralwayswriteszeroestothehostphysicalmemorybeforeassigningittoavirtualmachine.
Virtualmachinememorydeallocationactsjustlikeanoperatingsystem,suchthattheguestoperatingsystemfreesapieceofphysicalmemorybyaddingthesememorypagenumberstotheguestfreelist,butthedataofthe"freed"memorymaynotbemodifiedatall.
Asaresult,whenaparticularpieceofguestphysicalmemoryisfreed,themappedhostphysicalmemorywillusuallynotchangeitsstateandonlytheguestfreelistwillbechanged.
Thehypervisorknowswhentoallocatehostphysicalmemoryforavirtualmachinebecausethefirstmemoryaccessfromthevirtualmachinetoahostphysicalmemorywillcauseapagefaultthatcanbeeasilycapturedbythehypervisor.
However,itisdifficultforthehypervisortoknowwhentofreehostphysicalmemoryuponvirtualmachinememorydeallocationbecausetheguestoperatingsystemfreelistisgenerallynotpubliclyaccessible.
Hence,thehypervisorcannoteasilyfindoutthelocationofthefreelistandmonitoritschanges.
Althoughthehypervisorcannotreclaimhostmemorywhentheoperatingsystemfreesguestphysicalmemory,thisdoesnotmeanthatthehostmemory,nomatterhowlargeitis,willbeusedupbyavirtualmachinewhenthevirtualmachinerepeatedlyallocatesandfreesmemory.
Thisisbecausethehypervisordoesnotallocatehostphysicalmemoryoneveryvirtualmachine'smemoryallocation.
Itonlyallocateshostphysicalmemorywhenthevirtualmachinetouchesthephysicalmemorythatithasnevertouchedbefore.
Ifavirtualmachinefrequentlyallocatesandfreesmemory,presumablythesameguestphysicalmemoryisbeingallocatedandfreedagainandagain.
Therefore,thehypervisorjustallocateshostphysicalmemoryforthefirstmemoryallocationandthentheguestreuses6VMwarewhitepaperthesamehostphysicalmemoryfortherestofallocations.
Thatis,ifavirtualmachine'sentireguestphysicalmemory(configuredmemory)hasbeenbackedbythehostphysicalmemory,thehypervisordoesnotneedtoallocateanyhostphysicalmemoryforthisvirtualmachineanymore.
Thismeansthatthefollowingalwaysholdstrue:VM'shostmemoryusage/dev/null"Numberofruns:4VMconfiguration:1vCPU,512MBmemorySwingbenchDatabase:Oracle11gFunctionalbenchmark:OrderEntryNumberofusers:30Runtime:20minutesNumberofruns:3VMconfiguration:4vCPUs,4GmemoryExchange2007Server:MicrosoftExchange2007Loadgenclient:2000heavyexchangeusersVMconfiguration:4vCPUs,12GmemoryTheguestoperatingsystemrunninginsidetheSPECjbb,kernelcompile,andSwingbenchvirtualmachineswas64-bitRedHatEnterpriseLinux5.
2Server.
TheguestoperatingsystemrunninginsidetheExchangevirtualmachinewasWindowsServer2008.
ForSPECjbb2005andSwingbench,thethroughputwasmeasuredbycalculatingthenumberoftransactionspersecond.
Forkernelcompile,theperformancewasmeasuredbycalculatingtheinverseofthecompilationtime.
ForExchange,theperformancewasmeasuredusingtheaverageRemoteProcedureCall(RPC)latency.
Inaddition,forSwingbenchandExchange,theclientapplicationswereinstalledinaseparatenativemachine.
14VMwarewhitepaper5.
2TransparentPageSharingPerformanceInthisexperiment,twoinstancesofworkloadswererun.
Theoverallperformanceofworkloadswithpagesharingenabledtothosewithpagesharingdisabledwerecompared.
Therewasafocusonevaluatingtheoverheadofpagescanning.
Sincethepagescanrate(numberofscannedpagespersecond)islargelydeterminedbytheMem.
ShareScanTime,inadditiontothedefault60minutes,theminimalMem.
ShareScanTimeof10minuteswastested,whichpotentiallyintroducesthehighestpagescanningoverhead.
Figure9:Performanceimpactoftransparentpagesharing0.
90.
920.
960.
940.
981.
001.
021.
041.
0000.
9940.
9980.
9981.
0020.
991PshareDefaultPshareOPshare(ShareScanTime10)NormalizedThroughputSPECjbbKernelCompileSwingbenchFigure9confirmsthatenablingpagesharingintroducesanegligibleperformanceoverheadinthedefaultsettingandonly<1percentoverheadwhenMem.
ShareScanTimeis10minutesforallworkloads.
Pagesharingsometimesimprovesperformancebecausethevirtualmachine'shostmemoryfootprintisreducedsothatitfitstheprocessorcachebetter.
Besidespagescanning,breakingCoWpagesisanothersourceofpagesharing.
Unfortunately,suchoverheadishighlyapplicationdependentanditisdifficulttoevaluateitthroughconfigurableoptions.
Therefore,theoverheadofbreakingCoWpageswasomittedinthisexperiment.
5.
3Ballooningvs.
SwappingInthefollowingexperiments,VMmemoryreclamationwasenforcedbychangingeachvirtualmachine'smemorylimitvaluefromthedefaultunlimitedtovaluesthataresmallerthantheconfiguredvirtualmachinememorysize.
Pagesharingwasturnedofftoisolatetheperformanceimpactofballooningorswapping.
Sincethehostmemoryismuchlargerthanthevirtualmachinememorysize,thehostfreememoryisalwaysinthehighstate.
Hence,bydefault,ESXonlyusesballooningtoreclaimmemory.
Theballoondriverwasturnedofftoobtaintheperformanceofusingswappingonly.
Theballoonedandswappedmemorysizeswerealsocollectedwhenthevirtualmachineransteadily.
15VMwarewhitepaper5.
3.
1LinuxKernelCompileFigure10presentsthethroughputofthekernelcompileworkloadwithdifferentmemorylimitswhenusingballooningorswapping.
Thisexperimentwascontrivedtouseonlyballooningorswapping,notboth.
Whilethiscasewillnotoftenoccurinproductionenvironments,itshowstheperformancepenaltyduetoeithertechnologyonitsown.
Thethroughputisnormalizedtothecasewherevirtualmachinememoryisnotreclaimed.
Figure10:Performanceofkernelcompilewhenusingtheballooningvs.
theswapping00.
20.
60.
40.
81.
01.
20100300200400500600BalloonedsizeSwappedsizeThroughout(Balloononly)NormalizedThroughputBallooned/SwappedMemory(MB)Memorylimit(MB)512448384320256192128Throughput(Swappingonly)Byusingballooning,thekernelcompilevirtualmachineonlysuffersfrom3percentthroughputlossevenwhenthememorylimitisaslowas128MB(1/4oftheconfiguredvirtualmachinememorysize).
Thisisbecausethekernelcompileworkloadhasverylittlememoryreuseandmostoftheguestphysicalmemoryisusedasbuffercachesforthekernelsourcefiles.
Withballooning,theguestoperatingsystemreclaimsguestphysicalmemoryupontheballoondriver'sallocationrequestbydroppingthebufferpagesinsteadofpagingthemouttotheguestvirtualswapdevice.
Becausedroppedbufferpagesarenotreusedfrequently,theperformanceimpactofusingballooningistrivial.
However,withhypervisorswapping,theselectedguestbufferpagesareunnecessarilyswappedouttothehostswapdeviceandsomeguestkernelpagesareswappedoutoccasionally,makingtheperformanceofthevirtualmachinedegradewhenmemorylimitdecreases.
Whenthememorylimitis128MB,thethroughputlossisabout34percentintheswappingcase.
Ballooninflationisabetterapproachtomemoryreclamationfromaperformanceperspective.
Figure10,illustratesthatasmemorylimitdecreases,theballoonedandswappedmemorysizesincreasealmostlinearly.
Thereisadifferencebetweentheballoonedmemorysizeandtheswappedmemorysize.
Intheballooningcases,whenvirtualmachinememoryusageexceedsthespecifiedlimit,theballoondrivercannotforcetheguestoperatingsystemtopageoutguestphysicalpagesimmediatelyunlesstheballoondriverhasusedupmostofthefreeguestphysicalmemory,whichputstheguestoperatingsystemundermemorypressure.
Intheswappingcases,however,aslongasthevirtualmachinememoryusageexceedsthespecifiedlimit,theextraamountofpageswillbeswappedoutimmediately.
Therefore,theballoonedmemorysizeisroughlyequaltothevirtualmachinememorysizeminusthespecifiedlimit,whichmeansthatthefreephysicalmemoryisincluded.
Theswappedmemorysizeisroughlyequaltothevirtualmachinehostmemoryusageminusthespecifiedlimit.
Inthekernelcompilevirtualmachine,sincemostoftheguestphysicalpagesareusedtobuffertheworkloadfiles,thevirtualmachine'seffectivehostmemoryusageisclosetothevirtualmachinememorysize.
Hence,theswappedmemorysizeissimilartotheballoonedmemorysize.
16VMwarewhitepaper5.
3.
2Oracle/SwingbenchFigure11presentsthethroughputofanOracledatabasetestedbytheSwingbenchworkloadwithdifferentmemorylimitswhenusingballooningvs.
swapping.
Thethroughputisnormalizedtothecasewherevirtualmachinememoryisnotreclaimed.
Figure11:PerformanceofSwingbenchwhenusingballooningvs.
swapping00.
20.
60.
40.
81.
01.
2050015001000200025003000BalloonedsizeSwappedsizeThroughout(Balloononly)NormalizedThroughputBallooned/SwappedMemory(MB)Memorylimit(MB)3840358433283072281625602048179215362304Throughput(Swappingonly)Asinthekernelcompiletest,usingballooningbarelyimpactsthethroughputoftheSwingbenchvirtualmachineuntilthememorylimitdecreasesbelow2048MB.
ThisoccurswhentheguestoperatingsystemstartstopageoutthephysicalpagesthatareheavilyreusedbytheOracledatabase.
Incontrasttousingballooning,usingswappingcausessignificantthroughputpenalty.
Thethroughputlossisalready17percentwhenthememorylimitis3584MB.
Inhypervisorswapping,someguestbufferpagesareunnecessarilyswappedoutandsomeguestkernelorperformance-criticaldatabasepagesarealsounintentionallyswappedoutbecauseoftherandompageselectionpolicy.
FortheSwingbenchvirtualmachine,thevirtualmachinehostmemoryusageisveryclosetothevirtualmachinememorysize,sotheswappedmemorysizeisveryclosetotheballoonedmemorysize.
17VMwarewhitepaper5.
3.
3SPECjbbFigure12presentsthethroughputoftheSPECjbbworkloadwithdifferentmemorylimitswhenusingballooningvs.
swapping.
Thethroughputisnormalizedtothecasewherevirtualmachinememoryisnotreclaimed.
Figure12:PerformanceofSPECjbbwhenusingtheballooningvs.
theswapping00.
20.
60.
40.
81.
01.
2050015001000200025003000BalloonedsizeSwappedsizeThroughout(Balloononly)NormalizedThroughputBallooned/SwappedMemory(MB)Memorylimit(MB)3072281625602304204817921536Throughput(Swappingonly)Fromthisfigure,itisseenthatwhenthevirtualmachinememorylimitdecreasesbeyond2816MB,thethroughputofSPECjbbdegradessignificantlyinbothballooningandswappingcases.
Whenthememorylimitisreducedto2048MB,thethroughputlossesare89percentand96percentforballooningandswappingrespectively.
SincetheconfiguredJVMheapsizeis2.
5GB,theactualvirtualmachineworkingsetsizeisaround2.
5GBplusguestoperatingsystemmemoryusage(about300MB).
Whenthevirtualmachinememorylimitfallsbelow2816MB,thehostmemorycannotbacktheentirevirtualmachine'sworkingset,sothatvirtualmachinestartstosufferfromguest-levelpagingintheballooningcasesorhypervisorswappingintheswappingcases.
SinceSPECjbbisanextremelymemoryintensiveworkload,itsthroughputislargelydeterminedbythevirtualmachinememoryhitrate.
Inthisinstance,virtualmachinememoryhitrateisdefinedasthepercentageofguestmemoryaccessesthatresultinhostphysicalmemoryhits.
AhighermemoryhitratemeanshigherthroughputfortheSPECjbbworkload.
Sinceballooningandhostswappingsimilarlydecreasememoryhitrate,bothguestlevelpagingandhypervisorswappinglargelyhurtSPECjbbperformance.
Surprisingly,whenthememorylimitis2506MBor2304MB,usingswappingobtainshigherthroughputthanthatofusingballooning.
Thisseemstobecounterintuitivebecausehypervisorswappingtypicallycausesahigherperformancepenalty.
OnereasonableexplanationisthattherandompageselectionpolicyusedinhypervisorswappinglargelyfavorstheaccesspatternsoftheSPECjbbvirtualmachine.
Morespecifically,withballooning,whentheguestoperatingsystem(Linuxinthiscase)pagesoutguestphysicalpagestosatisfytheballoondriver'sallocationrequest,itchoosesthetargetsusinganLRU-approximatedpolicy.
However,theSPECjbbworkloadoftentraversesalltheallocatedguestphysicalmemoryiteratively.
Forexample,theJVMgarbagecollectorperiodicallyscanstheentireheaptofreememory.
Thisbehavioriscategorizedtoawell-knownLRUpathologicalcaseinwhichthememoryhitratedropsdramaticallyevenwhenthememorysizeisslightlysmallerthantheworkingsetsize.
Incontrast,whenusinghypervisorswapping,theswappedphysicalpagesarerandomlyselectedbythehypervisor,whichmakesthememoryhitratereducemoregraduallyasthememorylimitdecreases.
Thatiswhyusingswappingachieveshigherthroughputwhenthememorylimitissmallerthanthevirtualmachine'sworkingsetsize.
However,whenthememorylimitdropsto2304MB,thevirtualmachinememoryhitrateisequivalentlylowinbothswappingandballooningcases.
Usingswappingstartstocauseworseperformancecomparedtousing18VMwarewhitepaperballooning.
Notethattheabovetwoconfigurationswhereswappingoutperformsballooningarerarepathologicalcasesforballooningperformance.
Inmostcases,usingballooningachievesmuchbetterperformancecomparedtousingswapping.
SinceSPECjbbvirtualmachine'sworkingsetsize(~2.
8GB)ismuchsmallerthantheconfiguredvirtualmachinememorysize(4GB),theballoonedmemorysizeismuchhigherthantheswappedmemorysize.
5.
3.
4MicrosoftExchangeServer2007ThissectionpresentshowballooningandswappingimpacttheperformanceofanExchangeServervirtualmachine.
ExchangeSeverisamemoryintensiveworkloadthatisoptimizedtousealltheavailablephysicalmemorytocachethetransactionsforfewerdiskI/Os.
TheExchangeServerperformancewasmeasuredusingtheaverageRemoteProcedureCall(RPC)latencyduringtwohoursstablerun.
TheRPClatencygaugestheserverprocessingtimeforanRPCfromLoadGen(theclientapplicationthatdrivestheExchangeserver).
Therefore,lowerRPClatencymeansbetterperformance.
TheresultsarepresentedinFigure13.
Figure13:AverageRPClatencyofExchangewhenusingballooningvs.
usingswapping012111098765433212045678Ballooning4.
65.
96.
16.
857.
3HypervisorSwappingAverageRPClatency(ms)Memorylimit(GB)(a)01211109604080100120140160HypervisorSwapping4.
6143xAverageRPClatency(ms)Memorylimit(GB)(b)7.
48Figure13(a),illustratesthatwhenthememorylimitdecreasesfrom12GBto3GB,theaverageRPClatencyisgraduallyincreasedfrom4.
6msto7.
3mswithballooning.
However,asshowninFigure13(b),theRPClatencyisdramaticallyincreasedfrom4.
6msto143mswhensolelyswappingout2GBhostmemory.
Whenthememorylimitisreducedto9GB,hypervisorswappingmakestheRPClatencytoohigh,whichresultedinthefailureoftheLoadGenapplication(duetotimeout).
Overall,thisfigureconfirmsthatusingballooningtoreclaimmemoryismuchmoreefficientthanusinghypervisorswappingfortheExchangeServervirtualmachine.
19VMwarewhitepaper6.
BestPracticesBasedonthememorymanagementconceptsandperformanceevaluationresultspresentedintheprevioussections,thefollowingaresomebestpracticesforhostandguestmemoryusage.
Donotdisablepagesharingortheballoondriver.
Asdescribed,pagesharingisalightweighttechniquewhichopportunisticallyreclaimstheredundanthostmemorywithtrivialperformanceimpact.
Inthecaseswherehostsareheavilyovercommitted,usingballooningisgenerallymoreefficientandsafercomparedtousinghypervisorswapping,basedontheresultspresentedinSection5.
3.
ThesetwotechniquesareenabledbydefaultinESX4andshouldnotbedisabledunlessthebenefitsofdoingsoclearlyoutweighthecosts.
Carefullyspecifythememorylimitandmemoryreservation.
Thevirtualmachinememoryallocationtargetissubjecttothelimitandreservation.
Ifthesetwoparametersaremisconfigured,usersmayobserveballooningorswappingevenwhenthehosthasplentyoffreememory.
Forexample,avirtualmachine'smemorymaybereclaimedwhenthespecifiedlimitistoosmallorwhenothervirtualmachinesreservetoomuchhostmemory,eventhoughtheymayonlyuseasmallportionofthereservedmemory.
Ifaperformance-criticalvirtualmachineneedsaguaranteedmemoryallocation,thereservationneedstobespecifiedcarefullybecauseitmayimpactothervirtualmachines.
Hostmemorysizeshouldbelargerthanguestmemoryusage.
Forexample,itisunwisetorunavirtualmachinewitha2GBworkingsetsizeinahostwithonly1GBhostmemory.
Ifthisisthecase,thehypervisorhastoreclaimthevirtualmachine'sactivememorythroughballooningorhypervisorswapping,whichwillleadtopotentiallyseriousvirtualmachineperformancedegradation.
Althoughitisdifficulttotellwhetherthehostmemoryislargeenoughtoholdallofthevirtualmachine'sworkingsets,thebottomlineisthatthehostmemoryshouldnotbeexcessivelyovercommittedmakingtheguestshavetocontinuouslypageoutguestphysicalmemory.
Usesharestoadjustrelativeprioritieswhenmemoryisovercommitted.
Ifthehost'smemoryisovercommittedandthevirtualmachine'sallocatedhostmemoryistoosmalltoachieveareasonableperformance,theusercanadjustthevirtualmachine'ssharestoescalatetherelativepriorityofthevirtualmachinesothatthehypervisorwillallocatemorehostmemoryforthatvirtualmachine.
SetappropriateVirtualMachinememorysize.
Thevirtualmachinememorysizeshouldbeslightlylargerthantheaverageguestmemoryusage.
Theextramemorywillaccommodateworkloadspikesinthevirtualmachine.
Notethatguestoperatingsystemonlyrecognizesthespecifiedvirtualmachinememorysize.
Ifthevirtualmachinememorysizeistoosmall,guest-levelpagingisinevitable,eventhoughthehostmayhaveplentyoffreememory.
Instead,theusermayconservativelysetaverylargevirtualmachinememorysize,whichisfineintermsofvirtualmachineperformance,butmorevirtualmachinememorymeansthatmoreoverheadmemoryneedstobereservedforthevirtualmachine.
7.
References[1]CarlA.
Waldspurger.
"MemoryResourceManagementinVMwareESXServer".
ProceedingofthefifthSymposiumonOperatingSystemDesignandImplementation,Boston,Dec2002.
[2]vSphereResourceManagementGuide.
VMware.
http://www.
vmware.
com/pdf/vsphere4/r40/vsp_40_upgrade_guide.
pdf[3]MemoryPerformanceChartStatisticsinthevSphereClient.
http://communities.
vmware.
com/docs/DOC-10398[4]VirtualCenterMemoryStatisticsDefinitions.
http://communities.
vmware.
com/docs/DOC-5230[5]PerformanceEvaluationofIntelEPTHardwareAssist.
VMware.
http://www.
vmware.
com/resources/techresources/10006[6]PerformanceEvaluationofAMDRVIHardwareAssist.
VMware.
http://www.
vmware.
com/resources/techresources/1079[7]Thebuffercache.
http://tldp.
org/LDP/sag/html/buffer-cache.
html4VMwareToolsmustbeinstalledinordertoenableballooning.
Thisisrecommendedforallworkloads.
UnderstandingMemoryResourceManagementinVMwareESXServerSource:TechnicalMarketing,SDRevision:20090820VMware,Inc.
3401HillviewAvePaloAltoCA94304USATel877-486-9273Fax650-427-5001www.
vmware.
comCopyright2009VMware,Inc.
Allrightsreserved.
ThisproductisprotectedbyU.
S.
andinternationalcopyrightandintellectualpropertylaws.
VMwareproductsarecoveredbyoneormorepatentslistedathttp://www.
vmware.
com/go/patents.
VMwareisaregisteredtrademarkortrademarkofVMware,Inc.
intheUnitedStatesand/orotherjurisdictions.
Allothermarksandnamesmentionedhereinmaybetrademarksoftheirrespectivecompanies.
VMW_ESX_Memory_09Q3_WP_P20_R3
Introduction32.
ESXMemoryManagementOverview42.
1Terminology42.
2MemoryVirtualizationBasics42.
3MemoryManagementBasicsinESX53.
MemoryReclamationinESX63.
1Motivation63.
2TransparentPageSharing(TPS)73.
3Ballooning83.
4HypervisorSwapping93.
5WhentoReclaimHostMemory104.
ESXMemoryAllocationManagementforMultipleVirtualMachines115.
PerformanceEvaluation135.
1ExperimentalEnvironment135.
2TransparentPageSharingPerformance145.
3Ballooningvs.
Swapping145.
3.
1LinuxKernelCompile155.
3.
2Oracle/Swingbench165.
3.
3SPECjbb175.
3.
4MicrosoftExchangeServer2007186.
BestPractices197.
References193VMwarewhitepaper1.
IntroductionVMwareESXisahypervisordesignedtoefficientlymanagehardwareresourcesincludingCPU,memory,storage,andnetworkamongmultipleconcurrentvirtualmachines.
ThispaperdescribesthebasicmemorymanagementconceptsinESX,theconfigurationoptionsavailable,andprovidesresultstoshowtheperformanceimpactoftheseoptions.
Thefocusofthispaperisinpresentingthefundamentalconceptsoftheseoptions.
Moredetailscanbefoundin"MemoryResourceManagementinVMwareESXServer"[1].
ESXuseshigh-levelresourcemanagementpoliciestocomputeatargetmemoryallocationforeachvirtualmachine(VM)basedonthecurrentsystemloadandparametersettingsforthevirtualmachine(shares,reservation,andlimit[2]).
Thecomputedtargetallocationisusedtoguidethedynamicadjustmentofthememoryallocationforeachvirtualmachine.
Inthecaseswherehostmemoryisovercommitted,thetargetallocationsarestillachievedbyinvokingseverallower-levelmechanismstoreclaimmemoryfromvirtualmachines.
Thispaperassumesapurevirtualizationenvironmentinwhichtheguestoperatingsystemrunninginsidethevirtualmachineisnotmodifiedtofacilitatevirtualization(oftenreferredtoasparavirtualization).
KnowledgeofESXarchitecturewillhelpyouunderstandtheconceptspresentedinthispaper.
Inordertoquicklymonitorvirtualmachinememoryusage,theVMwarevSphereClientexposestwomemorystatisticsintheresourcesummary:ConsumedHostMemoryandActiveGuestMemory.
Figure1:HostandGuestMemoryusageinvSphereClientConsumedHostMemoryusageisdefinedastheamountofhostmemorythatisallocatedtothevirtualmachine,ActiveGuestMemoryisdefinedastheamountofguestmemorythatiscurrentlybeingusedbytheguestoperatingsystemanditsapplications.
Thesetwostatisticsarequiteusefulforanalyzingthememorystatusofthevirtualmachineandprovidinghintstoaddresspotentialperformanceissues.
Thispaperhelpsanswerthesequestions:WhyistheConsumedHostMemorysohighWhyistheConsumedHostMemoryusagesometimesmuchlargerthantheActiveGuestMemoryWhyistheActiveGuestMemorydifferentfromwhatisseeninsidetheguestoperatingsystemThesequestionscannotbeeasilyansweredwithoutunderstandingthebasicmemorymanagementconceptsinESX.
UnderstandinghowESXmanagesmemorywillalsomaketheperformanceimplicationsofchangingESXmemorymanagementparametersclearer.
ThevSphereClientcanalsodisplayperformancechartsforthefollowingmemorystatistics:active,shared,consumed,granted,overhead,balloon,swapped,swappedinrate,andswapped-outrate.
Acompletediscussionaboutthesemetricscanbefoundin"MemoryPerformanceChartMetricsinthevSphereClient"[3]and"VirtualCenterMemoryStatisticsDefinitions"[4].
Therestofthepaperisorganizedasfollows.
Section2presentstheoverviewofESXmemorymanagementconcepts.
Section3discussesthememoryreclamationtechniquesusedinESX.
Section4describeshowESXallocateshostmemorytovirtualmachineswhenthehostisundermemorypressure.
Section5presentsanddiscussestheperformanceresultsfordifferentmemoryreclamationtechniques.
Finally,Section6discussesthebestpracticeswithrespecttohostandguestmemoryusage.
4VMwarewhitepaper2.
ESXMemoryManagementOverview2.
1TerminologyThefollowingterminologyisusedthroughoutthispaper.
Hostphysicalmemory1referstothememorythatisvisibletothehypervisorasavailableonthesystem.
Guestphysicalmemoryreferstothememorythatisvisibletotheguestoperatingsystemrunninginthevirtualmachine.
Guestvirtualmemoryreferstoacontinuousvirtualaddressspacepresentedbytheguestoperatingsystemtoapplications.
Itisthememorythatisvisibletotheapplicationsrunninginsidethevirtualmachine.
Guestphysicalmemoryisbackedbyhostphysicalmemory,whichmeansthehypervisorprovidesamappingfromtheguesttothehostmemory.
Thememorytransferbetweentheguestphysicalmemoryandtheguestswapdeviceisreferredtoasguestlevelpagingandisdrivenbytheguestoperatingsystem.
Thememorytransferbetweenguestphysicalmemoryandthehostswapdeviceisreferredtoashypervisorswapping,whichisdrivenbythehypervisor.
2.
2MemoryVirtualizationBasicsVirtualmemoryisawell-knowntechniqueusedinmostgeneral-purposeoperatingsystems,andalmostallmodernprocessorshavehardwaretosupportit.
Virtualmemorycreatesauniformvirtualaddressspaceforapplicationsandallowstheoperatingsystemandhardwaretohandletheaddresstranslationbetweenthevirtualaddressspaceandthephysicaladdressspace.
Thistechniquenotonlysimplifiestheprogrammer'swork,butalsoadaptstheexecutionenvironmenttosupportlargeaddressspaces,processprotection,filemapping,andswappinginmoderncomputersystems.
Whenrunningavirtualmachine,thehypervisorcreatesacontiguousaddressablememoryspaceforthevirtualmachine.
Thismemoryspacehasthesamepropertiesasthevirtualaddressspacepresentedtotheapplicationsbytheguestoperatingsystem.
Thisallowsthehypervisortorunmultiplevirtualmachinessimultaneouslywhileprotectingthememoryofeachvirtualmachinefrombeingaccessedbyothers.
Therefore,fromtheviewoftheapplicationrunninginsidethevirtualmachine,thehypervisoraddsanextralevelofaddresstranslationthatmapstheguestphysicaladdresstothehostphysicaladdress.
Asaresult,therearethreevirtualmemorylayersinESX:guestvirtualmemory,guestphysicalmemory,andhostphysicalmemory.
TheirrelationshipsareillustratedinFigure2(a).
Figure2:Virtualmemorylevels(a)andmemoryaddresstranslation(b)inESX(a)VM(b)GuestvirtualmemoryApplicationOperatingSystemHypervisorHypervisorGuestphysicalmemoryHostphysicalmemoryGuestOSPageTablesguestvirtual-to-guestphysicalShadowPageTablesguestvirtual-to-guestphysicalpmapguestphysical-to-hostphysicalAsshowninFigure2(b),inESX,theaddresstranslationbetweenguestphysicalmemoryandhostphysicalmemoryismaintainedbythehypervisorusingaphysicalmemorymappingdatastructure,orpmap,foreachvirtualmachine.
Thehypervisorinterceptsallvirtualmachineinstructionsthatmanipulatethehardwaretranslationlookasidebuffer(TLB)contentsorguestoperatingsystempagetables,whichcontainthevirtualtophysicaladdressmapping.
TheactualhardwareTLBstateisupdatedbasedontheseparateshadowpagetables,whichcontaintheguestvirtualtohostphysicaladdressmapping.
Theshadowpagetablesmaintainconsistencywiththeguestvirtualtoguestphysicaladdressmappingintheguestpagetablesandtheguestphysicaltohostphysicaladdress1Thetermshostphysicalmemoryandhostmemoryareusedinterchangeablyinthispaper.
Theyarealsoequivalenttothetermmachinememoryusedin[1].
5VMwarewhitepapermappinginthepmapdatastructure.
Thisapproachremovesthevirtualizationoverheadforthevirtualmachine'snormalmemoryaccessesbecausethehardwareTLBwillcachethedirectguestvirtualtohostphysicalmemoryaddresstranslationsreadfromtheshadowpagetables.
Notethattheextralevelofguestphysicaltohostphysicalmemoryindirectionisextremelypowerfulinthevirtualizationenvironment.
Forexample,ESXcaneasilyremapavirtualmachine'shostphysicalmemorytofilesorotherdevicesinamannerthatiscompletelytransparenttothevirtualmachine.
Recently,somenewgenerationCPUs,suchasthirdgenerationAMDOpteronandIntelXeon5500seriesprocessors,haveprovidedhardwaresupportformemoryvirtualizationbyusingtwolayersofpagetablesinhardware.
Onelayerstorestheguestvirtualtoguestphysicalmemoryaddresstranslation,andtheotherlayerstorestheguestphysicaltohostphysicalmemoryaddresstranslation.
Thesetwopagetablesaresynchronizedusingprocessorhardware.
Hardwaresupportmemoryvirtualizationeliminatestheoverheadrequiredtokeepshadowpagetablesinsynchronizationwithguestpagetablesinsoftwarememoryvirtualization.
Formoreinformationabouthardware-assistedmemoryvirtualization,see"PerformanceEvaluationofIntelEPTHardwareAssist"[5]and"PerformanceEvaluationofAMDRVIHardwareAssist.
"[6]2.
3MemoryManagementBasicsinESXPriortotalkingabouthowESXmanagesmemoryforvirtualmachines,itisusefultofirstunderstandhowtheapplication,guestoperatingsystem,hypervisor,andvirtualmachinemanagememoryattheirrespectivelayers.
Anapplicationstartsandusestheinterfacesprovidedbytheoperatingsystemtoexplicitlyallocateordeallocatethevirtualmemoryduringtheexecution.
Inanon-virtualenvironment,theoperatingsystemassumesitownsallphysicalmemoryinthesystem.
Thehardwaredoesnotprovideinterfacesfortheoperatingsystemtoexplicitly"allocate"or"free"physicalmemory.
Theoperatingsystemestablishesthedefinitionsof"allocated"or"free"physicalmemory.
Differentoperatingsystemshavedifferentimplementationstorealizethisabstraction.
Oneexampleisthattheoperatingsystemmaintainsan"allocated"listanda"free"list,sowhetherornotaphysicalpageisfreedependsonwhichlistthepagecurrentlyresidesin.
Becauseavirtualmachinerunsanoperatingsystemandseveralapplications,thevirtualmachinememorymanagementpropertiescombinebothapplicationandoperatingsystemmemorymanagementproperties.
Likeanapplication,whenavirtualmachinefirststarts,ithasnopre-allocatedphysicalmemory.
Likeanoperatingsystem,thevirtualmachinecannotexplicitlyallocatehostphysicalmemorythroughanystandardinterfaces.
Thehypervisoralsocreatesthedefinitionsof"allocated"and"free"hostmemoryinitsowndatastructures.
Thehypervisorinterceptsthevirtualmachine'smemoryaccessesandallocateshostphysicalmemoryforthevirtualmachineonitsfirstaccesstothememory.
Inordertoavoidinformationleakingamongvirtualmachines,thehypervisoralwayswriteszeroestothehostphysicalmemorybeforeassigningittoavirtualmachine.
Virtualmachinememorydeallocationactsjustlikeanoperatingsystem,suchthattheguestoperatingsystemfreesapieceofphysicalmemorybyaddingthesememorypagenumberstotheguestfreelist,butthedataofthe"freed"memorymaynotbemodifiedatall.
Asaresult,whenaparticularpieceofguestphysicalmemoryisfreed,themappedhostphysicalmemorywillusuallynotchangeitsstateandonlytheguestfreelistwillbechanged.
Thehypervisorknowswhentoallocatehostphysicalmemoryforavirtualmachinebecausethefirstmemoryaccessfromthevirtualmachinetoahostphysicalmemorywillcauseapagefaultthatcanbeeasilycapturedbythehypervisor.
However,itisdifficultforthehypervisortoknowwhentofreehostphysicalmemoryuponvirtualmachinememorydeallocationbecausetheguestoperatingsystemfreelistisgenerallynotpubliclyaccessible.
Hence,thehypervisorcannoteasilyfindoutthelocationofthefreelistandmonitoritschanges.
Althoughthehypervisorcannotreclaimhostmemorywhentheoperatingsystemfreesguestphysicalmemory,thisdoesnotmeanthatthehostmemory,nomatterhowlargeitis,willbeusedupbyavirtualmachinewhenthevirtualmachinerepeatedlyallocatesandfreesmemory.
Thisisbecausethehypervisordoesnotallocatehostphysicalmemoryoneveryvirtualmachine'smemoryallocation.
Itonlyallocateshostphysicalmemorywhenthevirtualmachinetouchesthephysicalmemorythatithasnevertouchedbefore.
Ifavirtualmachinefrequentlyallocatesandfreesmemory,presumablythesameguestphysicalmemoryisbeingallocatedandfreedagainandagain.
Therefore,thehypervisorjustallocateshostphysicalmemoryforthefirstmemoryallocationandthentheguestreuses6VMwarewhitepaperthesamehostphysicalmemoryfortherestofallocations.
Thatis,ifavirtualmachine'sentireguestphysicalmemory(configuredmemory)hasbeenbackedbythehostphysicalmemory,thehypervisordoesnotneedtoallocateanyhostphysicalmemoryforthisvirtualmachineanymore.
Thismeansthatthefollowingalwaysholdstrue:VM'shostmemoryusage/dev/null"Numberofruns:4VMconfiguration:1vCPU,512MBmemorySwingbenchDatabase:Oracle11gFunctionalbenchmark:OrderEntryNumberofusers:30Runtime:20minutesNumberofruns:3VMconfiguration:4vCPUs,4GmemoryExchange2007Server:MicrosoftExchange2007Loadgenclient:2000heavyexchangeusersVMconfiguration:4vCPUs,12GmemoryTheguestoperatingsystemrunninginsidetheSPECjbb,kernelcompile,andSwingbenchvirtualmachineswas64-bitRedHatEnterpriseLinux5.
2Server.
TheguestoperatingsystemrunninginsidetheExchangevirtualmachinewasWindowsServer2008.
ForSPECjbb2005andSwingbench,thethroughputwasmeasuredbycalculatingthenumberoftransactionspersecond.
Forkernelcompile,theperformancewasmeasuredbycalculatingtheinverseofthecompilationtime.
ForExchange,theperformancewasmeasuredusingtheaverageRemoteProcedureCall(RPC)latency.
Inaddition,forSwingbenchandExchange,theclientapplicationswereinstalledinaseparatenativemachine.
14VMwarewhitepaper5.
2TransparentPageSharingPerformanceInthisexperiment,twoinstancesofworkloadswererun.
Theoverallperformanceofworkloadswithpagesharingenabledtothosewithpagesharingdisabledwerecompared.
Therewasafocusonevaluatingtheoverheadofpagescanning.
Sincethepagescanrate(numberofscannedpagespersecond)islargelydeterminedbytheMem.
ShareScanTime,inadditiontothedefault60minutes,theminimalMem.
ShareScanTimeof10minuteswastested,whichpotentiallyintroducesthehighestpagescanningoverhead.
Figure9:Performanceimpactoftransparentpagesharing0.
90.
920.
960.
940.
981.
001.
021.
041.
0000.
9940.
9980.
9981.
0020.
991PshareDefaultPshareOPshare(ShareScanTime10)NormalizedThroughputSPECjbbKernelCompileSwingbenchFigure9confirmsthatenablingpagesharingintroducesanegligibleperformanceoverheadinthedefaultsettingandonly<1percentoverheadwhenMem.
ShareScanTimeis10minutesforallworkloads.
Pagesharingsometimesimprovesperformancebecausethevirtualmachine'shostmemoryfootprintisreducedsothatitfitstheprocessorcachebetter.
Besidespagescanning,breakingCoWpagesisanothersourceofpagesharing.
Unfortunately,suchoverheadishighlyapplicationdependentanditisdifficulttoevaluateitthroughconfigurableoptions.
Therefore,theoverheadofbreakingCoWpageswasomittedinthisexperiment.
5.
3Ballooningvs.
SwappingInthefollowingexperiments,VMmemoryreclamationwasenforcedbychangingeachvirtualmachine'smemorylimitvaluefromthedefaultunlimitedtovaluesthataresmallerthantheconfiguredvirtualmachinememorysize.
Pagesharingwasturnedofftoisolatetheperformanceimpactofballooningorswapping.
Sincethehostmemoryismuchlargerthanthevirtualmachinememorysize,thehostfreememoryisalwaysinthehighstate.
Hence,bydefault,ESXonlyusesballooningtoreclaimmemory.
Theballoondriverwasturnedofftoobtaintheperformanceofusingswappingonly.
Theballoonedandswappedmemorysizeswerealsocollectedwhenthevirtualmachineransteadily.
15VMwarewhitepaper5.
3.
1LinuxKernelCompileFigure10presentsthethroughputofthekernelcompileworkloadwithdifferentmemorylimitswhenusingballooningorswapping.
Thisexperimentwascontrivedtouseonlyballooningorswapping,notboth.
Whilethiscasewillnotoftenoccurinproductionenvironments,itshowstheperformancepenaltyduetoeithertechnologyonitsown.
Thethroughputisnormalizedtothecasewherevirtualmachinememoryisnotreclaimed.
Figure10:Performanceofkernelcompilewhenusingtheballooningvs.
theswapping00.
20.
60.
40.
81.
01.
20100300200400500600BalloonedsizeSwappedsizeThroughout(Balloononly)NormalizedThroughputBallooned/SwappedMemory(MB)Memorylimit(MB)512448384320256192128Throughput(Swappingonly)Byusingballooning,thekernelcompilevirtualmachineonlysuffersfrom3percentthroughputlossevenwhenthememorylimitisaslowas128MB(1/4oftheconfiguredvirtualmachinememorysize).
Thisisbecausethekernelcompileworkloadhasverylittlememoryreuseandmostoftheguestphysicalmemoryisusedasbuffercachesforthekernelsourcefiles.
Withballooning,theguestoperatingsystemreclaimsguestphysicalmemoryupontheballoondriver'sallocationrequestbydroppingthebufferpagesinsteadofpagingthemouttotheguestvirtualswapdevice.
Becausedroppedbufferpagesarenotreusedfrequently,theperformanceimpactofusingballooningistrivial.
However,withhypervisorswapping,theselectedguestbufferpagesareunnecessarilyswappedouttothehostswapdeviceandsomeguestkernelpagesareswappedoutoccasionally,makingtheperformanceofthevirtualmachinedegradewhenmemorylimitdecreases.
Whenthememorylimitis128MB,thethroughputlossisabout34percentintheswappingcase.
Ballooninflationisabetterapproachtomemoryreclamationfromaperformanceperspective.
Figure10,illustratesthatasmemorylimitdecreases,theballoonedandswappedmemorysizesincreasealmostlinearly.
Thereisadifferencebetweentheballoonedmemorysizeandtheswappedmemorysize.
Intheballooningcases,whenvirtualmachinememoryusageexceedsthespecifiedlimit,theballoondrivercannotforcetheguestoperatingsystemtopageoutguestphysicalpagesimmediatelyunlesstheballoondriverhasusedupmostofthefreeguestphysicalmemory,whichputstheguestoperatingsystemundermemorypressure.
Intheswappingcases,however,aslongasthevirtualmachinememoryusageexceedsthespecifiedlimit,theextraamountofpageswillbeswappedoutimmediately.
Therefore,theballoonedmemorysizeisroughlyequaltothevirtualmachinememorysizeminusthespecifiedlimit,whichmeansthatthefreephysicalmemoryisincluded.
Theswappedmemorysizeisroughlyequaltothevirtualmachinehostmemoryusageminusthespecifiedlimit.
Inthekernelcompilevirtualmachine,sincemostoftheguestphysicalpagesareusedtobuffertheworkloadfiles,thevirtualmachine'seffectivehostmemoryusageisclosetothevirtualmachinememorysize.
Hence,theswappedmemorysizeissimilartotheballoonedmemorysize.
16VMwarewhitepaper5.
3.
2Oracle/SwingbenchFigure11presentsthethroughputofanOracledatabasetestedbytheSwingbenchworkloadwithdifferentmemorylimitswhenusingballooningvs.
swapping.
Thethroughputisnormalizedtothecasewherevirtualmachinememoryisnotreclaimed.
Figure11:PerformanceofSwingbenchwhenusingballooningvs.
swapping00.
20.
60.
40.
81.
01.
2050015001000200025003000BalloonedsizeSwappedsizeThroughout(Balloononly)NormalizedThroughputBallooned/SwappedMemory(MB)Memorylimit(MB)3840358433283072281625602048179215362304Throughput(Swappingonly)Asinthekernelcompiletest,usingballooningbarelyimpactsthethroughputoftheSwingbenchvirtualmachineuntilthememorylimitdecreasesbelow2048MB.
ThisoccurswhentheguestoperatingsystemstartstopageoutthephysicalpagesthatareheavilyreusedbytheOracledatabase.
Incontrasttousingballooning,usingswappingcausessignificantthroughputpenalty.
Thethroughputlossisalready17percentwhenthememorylimitis3584MB.
Inhypervisorswapping,someguestbufferpagesareunnecessarilyswappedoutandsomeguestkernelorperformance-criticaldatabasepagesarealsounintentionallyswappedoutbecauseoftherandompageselectionpolicy.
FortheSwingbenchvirtualmachine,thevirtualmachinehostmemoryusageisveryclosetothevirtualmachinememorysize,sotheswappedmemorysizeisveryclosetotheballoonedmemorysize.
17VMwarewhitepaper5.
3.
3SPECjbbFigure12presentsthethroughputoftheSPECjbbworkloadwithdifferentmemorylimitswhenusingballooningvs.
swapping.
Thethroughputisnormalizedtothecasewherevirtualmachinememoryisnotreclaimed.
Figure12:PerformanceofSPECjbbwhenusingtheballooningvs.
theswapping00.
20.
60.
40.
81.
01.
2050015001000200025003000BalloonedsizeSwappedsizeThroughout(Balloononly)NormalizedThroughputBallooned/SwappedMemory(MB)Memorylimit(MB)3072281625602304204817921536Throughput(Swappingonly)Fromthisfigure,itisseenthatwhenthevirtualmachinememorylimitdecreasesbeyond2816MB,thethroughputofSPECjbbdegradessignificantlyinbothballooningandswappingcases.
Whenthememorylimitisreducedto2048MB,thethroughputlossesare89percentand96percentforballooningandswappingrespectively.
SincetheconfiguredJVMheapsizeis2.
5GB,theactualvirtualmachineworkingsetsizeisaround2.
5GBplusguestoperatingsystemmemoryusage(about300MB).
Whenthevirtualmachinememorylimitfallsbelow2816MB,thehostmemorycannotbacktheentirevirtualmachine'sworkingset,sothatvirtualmachinestartstosufferfromguest-levelpagingintheballooningcasesorhypervisorswappingintheswappingcases.
SinceSPECjbbisanextremelymemoryintensiveworkload,itsthroughputislargelydeterminedbythevirtualmachinememoryhitrate.
Inthisinstance,virtualmachinememoryhitrateisdefinedasthepercentageofguestmemoryaccessesthatresultinhostphysicalmemoryhits.
AhighermemoryhitratemeanshigherthroughputfortheSPECjbbworkload.
Sinceballooningandhostswappingsimilarlydecreasememoryhitrate,bothguestlevelpagingandhypervisorswappinglargelyhurtSPECjbbperformance.
Surprisingly,whenthememorylimitis2506MBor2304MB,usingswappingobtainshigherthroughputthanthatofusingballooning.
Thisseemstobecounterintuitivebecausehypervisorswappingtypicallycausesahigherperformancepenalty.
OnereasonableexplanationisthattherandompageselectionpolicyusedinhypervisorswappinglargelyfavorstheaccesspatternsoftheSPECjbbvirtualmachine.
Morespecifically,withballooning,whentheguestoperatingsystem(Linuxinthiscase)pagesoutguestphysicalpagestosatisfytheballoondriver'sallocationrequest,itchoosesthetargetsusinganLRU-approximatedpolicy.
However,theSPECjbbworkloadoftentraversesalltheallocatedguestphysicalmemoryiteratively.
Forexample,theJVMgarbagecollectorperiodicallyscanstheentireheaptofreememory.
Thisbehavioriscategorizedtoawell-knownLRUpathologicalcaseinwhichthememoryhitratedropsdramaticallyevenwhenthememorysizeisslightlysmallerthantheworkingsetsize.
Incontrast,whenusinghypervisorswapping,theswappedphysicalpagesarerandomlyselectedbythehypervisor,whichmakesthememoryhitratereducemoregraduallyasthememorylimitdecreases.
Thatiswhyusingswappingachieveshigherthroughputwhenthememorylimitissmallerthanthevirtualmachine'sworkingsetsize.
However,whenthememorylimitdropsto2304MB,thevirtualmachinememoryhitrateisequivalentlylowinbothswappingandballooningcases.
Usingswappingstartstocauseworseperformancecomparedtousing18VMwarewhitepaperballooning.
Notethattheabovetwoconfigurationswhereswappingoutperformsballooningarerarepathologicalcasesforballooningperformance.
Inmostcases,usingballooningachievesmuchbetterperformancecomparedtousingswapping.
SinceSPECjbbvirtualmachine'sworkingsetsize(~2.
8GB)ismuchsmallerthantheconfiguredvirtualmachinememorysize(4GB),theballoonedmemorysizeismuchhigherthantheswappedmemorysize.
5.
3.
4MicrosoftExchangeServer2007ThissectionpresentshowballooningandswappingimpacttheperformanceofanExchangeServervirtualmachine.
ExchangeSeverisamemoryintensiveworkloadthatisoptimizedtousealltheavailablephysicalmemorytocachethetransactionsforfewerdiskI/Os.
TheExchangeServerperformancewasmeasuredusingtheaverageRemoteProcedureCall(RPC)latencyduringtwohoursstablerun.
TheRPClatencygaugestheserverprocessingtimeforanRPCfromLoadGen(theclientapplicationthatdrivestheExchangeserver).
Therefore,lowerRPClatencymeansbetterperformance.
TheresultsarepresentedinFigure13.
Figure13:AverageRPClatencyofExchangewhenusingballooningvs.
usingswapping012111098765433212045678Ballooning4.
65.
96.
16.
857.
3HypervisorSwappingAverageRPClatency(ms)Memorylimit(GB)(a)01211109604080100120140160HypervisorSwapping4.
6143xAverageRPClatency(ms)Memorylimit(GB)(b)7.
48Figure13(a),illustratesthatwhenthememorylimitdecreasesfrom12GBto3GB,theaverageRPClatencyisgraduallyincreasedfrom4.
6msto7.
3mswithballooning.
However,asshowninFigure13(b),theRPClatencyisdramaticallyincreasedfrom4.
6msto143mswhensolelyswappingout2GBhostmemory.
Whenthememorylimitisreducedto9GB,hypervisorswappingmakestheRPClatencytoohigh,whichresultedinthefailureoftheLoadGenapplication(duetotimeout).
Overall,thisfigureconfirmsthatusingballooningtoreclaimmemoryismuchmoreefficientthanusinghypervisorswappingfortheExchangeServervirtualmachine.
19VMwarewhitepaper6.
BestPracticesBasedonthememorymanagementconceptsandperformanceevaluationresultspresentedintheprevioussections,thefollowingaresomebestpracticesforhostandguestmemoryusage.
Donotdisablepagesharingortheballoondriver.
Asdescribed,pagesharingisalightweighttechniquewhichopportunisticallyreclaimstheredundanthostmemorywithtrivialperformanceimpact.
Inthecaseswherehostsareheavilyovercommitted,usingballooningisgenerallymoreefficientandsafercomparedtousinghypervisorswapping,basedontheresultspresentedinSection5.
3.
ThesetwotechniquesareenabledbydefaultinESX4andshouldnotbedisabledunlessthebenefitsofdoingsoclearlyoutweighthecosts.
Carefullyspecifythememorylimitandmemoryreservation.
Thevirtualmachinememoryallocationtargetissubjecttothelimitandreservation.
Ifthesetwoparametersaremisconfigured,usersmayobserveballooningorswappingevenwhenthehosthasplentyoffreememory.
Forexample,avirtualmachine'smemorymaybereclaimedwhenthespecifiedlimitistoosmallorwhenothervirtualmachinesreservetoomuchhostmemory,eventhoughtheymayonlyuseasmallportionofthereservedmemory.
Ifaperformance-criticalvirtualmachineneedsaguaranteedmemoryallocation,thereservationneedstobespecifiedcarefullybecauseitmayimpactothervirtualmachines.
Hostmemorysizeshouldbelargerthanguestmemoryusage.
Forexample,itisunwisetorunavirtualmachinewitha2GBworkingsetsizeinahostwithonly1GBhostmemory.
Ifthisisthecase,thehypervisorhastoreclaimthevirtualmachine'sactivememorythroughballooningorhypervisorswapping,whichwillleadtopotentiallyseriousvirtualmachineperformancedegradation.
Althoughitisdifficulttotellwhetherthehostmemoryislargeenoughtoholdallofthevirtualmachine'sworkingsets,thebottomlineisthatthehostmemoryshouldnotbeexcessivelyovercommittedmakingtheguestshavetocontinuouslypageoutguestphysicalmemory.
Usesharestoadjustrelativeprioritieswhenmemoryisovercommitted.
Ifthehost'smemoryisovercommittedandthevirtualmachine'sallocatedhostmemoryistoosmalltoachieveareasonableperformance,theusercanadjustthevirtualmachine'ssharestoescalatetherelativepriorityofthevirtualmachinesothatthehypervisorwillallocatemorehostmemoryforthatvirtualmachine.
SetappropriateVirtualMachinememorysize.
Thevirtualmachinememorysizeshouldbeslightlylargerthantheaverageguestmemoryusage.
Theextramemorywillaccommodateworkloadspikesinthevirtualmachine.
Notethatguestoperatingsystemonlyrecognizesthespecifiedvirtualmachinememorysize.
Ifthevirtualmachinememorysizeistoosmall,guest-levelpagingisinevitable,eventhoughthehostmayhaveplentyoffreememory.
Instead,theusermayconservativelysetaverylargevirtualmachinememorysize,whichisfineintermsofvirtualmachineperformance,butmorevirtualmachinememorymeansthatmoreoverheadmemoryneedstobereservedforthevirtualmachine.
7.
References[1]CarlA.
Waldspurger.
"MemoryResourceManagementinVMwareESXServer".
ProceedingofthefifthSymposiumonOperatingSystemDesignandImplementation,Boston,Dec2002.
[2]vSphereResourceManagementGuide.
VMware.
http://www.
vmware.
com/pdf/vsphere4/r40/vsp_40_upgrade_guide.
pdf[3]MemoryPerformanceChartStatisticsinthevSphereClient.
http://communities.
vmware.
com/docs/DOC-10398[4]VirtualCenterMemoryStatisticsDefinitions.
http://communities.
vmware.
com/docs/DOC-5230[5]PerformanceEvaluationofIntelEPTHardwareAssist.
VMware.
http://www.
vmware.
com/resources/techresources/10006[6]PerformanceEvaluationofAMDRVIHardwareAssist.
VMware.
http://www.
vmware.
com/resources/techresources/1079[7]Thebuffercache.
http://tldp.
org/LDP/sag/html/buffer-cache.
html4VMwareToolsmustbeinstalledinordertoenableballooning.
Thisisrecommendedforallworkloads.
UnderstandingMemoryResourceManagementinVMwareESXServerSource:TechnicalMarketing,SDRevision:20090820VMware,Inc.
3401HillviewAvePaloAltoCA94304USATel877-486-9273Fax650-427-5001www.
vmware.
comCopyright2009VMware,Inc.
Allrightsreserved.
ThisproductisprotectedbyU.
S.
andinternationalcopyrightandintellectualpropertylaws.
VMwareproductsarecoveredbyoneormorepatentslistedathttp://www.
vmware.
com/go/patents.
VMwareisaregisteredtrademarkortrademarkofVMware,Inc.
intheUnitedStatesand/orotherjurisdictions.
Allothermarksandnamesmentionedhereinmaybetrademarksoftheirrespectivecompanies.
VMW_ESX_Memory_09Q3_WP_P20_R3
- illustratedfreehost相关文档
- detailsfreehost
- summedfreehost
- determinefreehost
- endogefreehost
- congestionfreehost
- cgfreehost
搬瓦工(季付46.7美元)新增荷兰(联通线路)VPS,2.5-10Gbps
搬瓦工最近新增了荷兰机房中国联通(AS9929、AS4837)线路的VPS产品，选项为NL - China Unicom Amsterdam(ENUL_9)，与日本软银和洛杉矶(DC06)CN2 GIA配置一致可以互换，属于高端系列，2.5Gbps-10Gbps大带宽，最低季付46.7美元起。搬瓦工VPS隶属于老牌IT7公司旗下，主要提供基于KVM架构VPS主机，数据中心包括美国洛杉矶、凤凰城、纽...
gcorelabs：CDN业务节点分布100多个国家地区，免费版提供1T/月流量
卢森堡商家gcorelabs是个全球数据中心集大成的运营者，不但提供超过32个数据中心的VPS、13个数据中心的cloud（云服务器）、超过44个数据中心的独立服务器，还提供超过100个数据中心节点的CDN业务。CDN的总带宽容量超过50Tbps，支持免费测试！ Gcorelabs根据业务分，有2套后台，分别是： CDN、流媒体平台、DDoS高防业务、块存储、cloud云服务器、裸金属服务器...
易探云服务器怎么过户/转让?云服务器PUSH实操步骤
易探云服务器怎么过户/转让?易探云支持云服务器PUSH功能，该功能可将云服务器过户给指定用户。可带价PUSH，收到PUSH请求的用户在接收云服务器的同时，系统会扣除接收方的款项，同时扣除相关手续费，然后将款项打到发送方的账户下。易探云“PUSH服务器”的这一功能，可以让用户将闲置云服务器转让给更多需要购买的用户！易探云服务器怎么过户/PUSH?1.PUSH双方必须为认证用户：2.买家未接收前，卖家...
freehost为你推荐
-
网易网盘关闭入口网易网盘怎么用????brandoff香港购物在哪里杨紫别祝我生日快乐祝自己生日快乐内涵丰富的话lunwenjiance我写的论文，检测相似度是21.63%，删掉参考文献后就只有6.3%，这是为什么？嘀动网手机一键通用来干嘛呢？ip在线查询我要用eclipse做个ip在线查询功能，用QQwry数据库，可是我不知道怎么把这个数据库放到我的程序里面去，高手帮忙指点下，小弟在这谢谢了百度指数词为什么百度指数里有写词没有指数，还要购买javbibitreebibi是什么牌子的杨丽晓博客明星的最新博文www.36ybyb.com有什么网址有很多动漫可以看的啊?我知道的有www.hnnn.net.很多好看的！但是．．．都看了！我想看些别人哦！还有优酷网也不错．．．