Countersvmware

InformationGuideCopyright2008VMware,Inc.
Allrightsreserved.
1TimekeepinginVMwareVirtualMachinesVMwareESX3.
5/ESXi3.
5,VMwareWorkstation6.
5Becausevirtualmachinesworkbytime‐sharinghostphysicalhardware,avirtualmachinecannotexactlyduplicatethetimingbehaviorofaphysicalmachine.
VMwarevirtualmachinesuseseveraltechniquestominimizeandconcealdifferencesintimingbehavior,butthedifferencescanstillsometimescausetimekeepinginaccuraciesandotherproblemsinsoftwarerunninginavirtualmachine.
Thisinformationguidedescribeshowtimekeepinghardwareworksinphysicalmachines,howtypicalguestoperatingsystemsusethishardwaretokeeptime,andhowVMwareproductsvirtualizethehardware.
Thispaperisintendedforpartners,resellers,andadvancedsystemadministratorswhoaredeployingVMwareproductsandneedadeepunderstandingoftheissuesthatariseinkeepingaccuratetimeinvirtualmachines.
TheVMwareknowledgebasecontainsadditionalandmorefrequentlyupdatedinformation,includingbestpracticestoconfigurespecificguestoperatingsystemversionsforthemostaccuratetimekeeping,aswellasrecipesfordiagnosingandworkingaroundknownissuesinspecificversionsofVMwareproducts.
Thisdocumentincludesthefollowingtopics:"TimekeepingBasics"onpage1"TimeandFrequencyUnits"onpage4"PCTimerHardware"onpage4"VMwareTimerVirtualization"onpage6"TimekeepinginSpecificOperatingSystems"onpage10"SynchronizingVirtualMachinesandHostswithRealTime"onpage14"TimeandPerformanceMeasurementsWithinaVirtualMachine"onpage18"ResourcePressure"onpage21"Troubleshooting"onpage22"Resources"onpage26TimekeepingBasicsComputeroperatingsystemstypicallymeasurethepassageoftimeinoneoftwoways.
Tickcounting—Theoperatingsystemsetsupahardwaredevicetointerruptperiodicallyataknownrate,suchas100timespersecond.
Theoperatingsystemthenhandlestheseinterrupts(calledticks)andkeepsacounttodeterminehowmuchtimehaspassed.
Copyright2008VMware,Inc.
Allrightsreserved.
2TimekeepinginVMwareVirtualMachinesTicklesstimekeeping—Ahardwaredevicekeepsacountofthenumberoftimeunitsthathavepassedsincethesystembooted,andtheoperatingsystemsimplyreadsthecounterwhenneeded.
Ticklesstimekeepinghasseveraladvantages.
Inparticular,itdoesnotkeeptheCPUbusyhandlinginterrupts,anditcankeeptimeatafinergranularity.
However,ticklesstimekeepingispracticalonlyonmachinesthatprovideasuitablehardwarecounter.
Thecountermustrunataconstantrate,bereasonablyfasttoread,andeitherneveroverfloworoverflowinfrequentlyenoughthattheoperatingsystemcanreliablyextenditsrangebydetectingandcountingtheoverflows.
Besidesmeasuringthepassageoftime,operatingsystemsarealsocalledontokeeptrackoftheabsolutetime,oftencalledwall‐clocktime.
Generally,whenanoperatingsystemstartsup,itreadstheinitialwall‐clocktimetothenearestsecondfromthecomputer'sbattery‐backedrealtimeclockorqueriesanetworktimeservertoobtainamorepreciseandaccuratetimevalue.
Itthenusesoneofthemethodsdescribedabovetomeasurethepassageoftimefromthatpoint.
Inaddition,tocorrectforlong‐termdriftandothererrorsinthemeasurement,theoperatingsystemmayincludeadaemonthatrunsperiodicallytochecktheclockagainstanetworktimeserverandmakeadjustmentstoitsvalueandrunningrate.
TickCountingManyPC‐basedoperatingsystemsusetickcountingtokeeptime.
Unfortunately,supportingthisformoftimekeepingaccuratelyinavirtualmachineisdifficult.
Virtualmachinessharetheirunderlyinghardwarewiththehostoperatingsystem(oronVMwareESX,theVMkernel).
Otherapplicationsandothervirtualmachinesmayalsoberunningonthesamehostmachine.
Thus,atthemomentavirtualmachineshouldgenerateavirtualtimerinterrupt,itmaynotactuallyberunning.
Infact,thevirtualmachinemaynotgetachancetorunagainuntilithasaccumulatedabacklogofmanytimerinterrupts.
Inaddition,evenarunningvirtualmachinecansometimesbelateindeliveringvirtualtimerinterrupts.
Thevirtualmachinechecksforpendingvirtualtimerinterruptsonlyatcertainpoints,suchaswhentheunderlyinghardwarereceivesaphysicaltimerinterrupt.
Manyhostoperatingsystemsdonotprovideawayforthevirtualmachinetorequestaphysicaltimerinterruptatapreciselyspecifiedtime.
Becausetheguestoperatingsystemkeepstimebycountinginterrupts,timeasmeasuredbytheguestoperatingsystemfallsbehindrealtimewheneverthereisatimerinterruptbacklog.
AVMwarevirtualmachinedealswiththisproblembykeepingtrackofthecurrenttimerinterruptbackloganddeliveringtimerinterruptsatahigherratewheneverthebackloggrowstoolarge,inordertocatchup.
Catchingupismademoredifficultbythefactthatanewtimerinterruptshouldnotbegenerateduntiltheguestoperatingsystemhasfullyhandledthepreviousone.
Otherwise,theguestoperatingsystemmayfailtoseethenextinterruptasaseparateeventandmisscountingit.
Thisphenomenoniscalledalosttick.
Ifthevirtualmachineisrunningtooslowly,perhapsasaresultofcompetitionforCPUtimefromothervirtualmachinesorprocessesrunningonthehostmachine,itmaybeimpossibletofeedthevirtualmachineenoughinterruptstokeepupwithrealtime.
IncurrentVMwareproducts,ifthebacklogofinterruptsgrowsbeyond60seconds,thevirtualmachinegivesuponcatchingup,simplysettingitsrecordofthebacklogtozero.
Afterthishappens,ifVMwareToolsisinstalledintheguestoperatingsystemanditsclocksynchronizationfeatureisenabled,VMwareToolscorrectstheclockreadingintheguestoperatingsystemsometimewithinthenextminutebysynchronizingtheguestoperatingsystemtimetomatchthehostmachine'sclock.
Thevirtualmachinethenresumeskeepingtrackofitsbacklogandcatchingupanynewbacklogthataccumulates.
Anotherproblemwithtimerinterruptsisthattheycauseascalabilityissueasmoreandmorevirtualmachinesarerunonthesamephysicalmachine.
Evenwhenavirtualmachineisotherwisecompletelyidle,itmustrunbrieflyeachtimeitreceivesatimerinterrupt.
Ifavirtualmachineisrequesting100interruptspersecond,itthusbecomesreadytorunatleast100timespersecond,atevenlyspacedintervals.
Soroughlyspeaking,ifNvirtualmachinesarerunning,processingtheinterruptsimposesabackgroundloadof100*Ncontextswitchespersecond(evenifallthevirtualmachinesareidle).
Virtualmachinesthatrequest1,000interruptspersecondcreate10timesthecontextswitchingload,andsoforth.
TicklessTimekeepingAgrowingnumberofPC‐basedoperatingsystemsuseticklesstimekeeping.
Thisformoftimekeepingisrelativelyeasytosupportinavirtualmachineandhasseveraladvantages,buttherearestillafewchallenges.

Copyright2008VMware,Inc.
Allrightsreserved.
3TimekeepinginVMwareVirtualMachinesOnthepositiveside,whentheguestoperatingsystemisnotcountingtimerinterruptsfortimekeepingpurposes,thereisnoneedforthevirtualmachinetokeeptrackofaninterruptbacklogandcatchupifthenumberofinterruptsdeliveredhasfallenbehindrealtime.
Lateinterruptscansimplybeallowedtopileupandmergetogether,withoutconcernforclockslippagecausedbylostticks.
ThissavesCPUtimethatwouldotherwisebeconsumedinhandlingthelateinterrupts.
Further,theguestoperatingsystem'sviewoftimeismoreaccurate,becauseitsclockdoesnotfallbehindrealtimewhilethevirtualmachineisnotrunningorisrunningslowly.
Inordertoachievetheseadvantages,however,thevirtualmachineneedstoknowthattheguestoperatingsystemisusingticklesstimekeeping.
Thevirtualmachinemustassumetickcountingintheabsenceofknowledgetothecontrary,becauseiftheguestoperatingsystemisinfactcountingtimerinterrupts,itisincorrecttodropany.
VMwareproductsusemultiplemethodstodetectticklesstimekeeping.
First,iftheguesthasnotprogrammedanyofthevirtualtimerdevicestogenerateperiodicinterrupts,itissafetoassumethattickcountingisnotinuse.
However,someoperatingsystemsdoprogramoneormoretimerdevicesforperiodicinterruptsevenwhenusingticklesstimekeeping.
Insuchcasestheuseofticklesstimekeepingcanusuallybeinferredfromtheguestoperatingsystemtype.
Alternatively,softwareinthevirtualmachinecanmakeahypercalltoinformthevirtualmachinethatitistickless.
Anadditionalchallengeforbothformsoftimekeepingisthatvirtualmachinesoccasionallyrunhighlytime‐sensitivecode—forexample,measuringthenumberofiterationsofaspecificloopthatcanruninagivenamountofrealtime.
Insomecases,suchcodemaybehavebetterunderthetick‐countingstyleoftimekeeping,inwhichtheguestoperatingsystem'sviewoftimeappearstoslowdownorstopwhilethevirtualmachineisnotrunning.
InitializingandCorrectingWall-ClockTimeAguestoperatingsystemfacesthesamebasicchallengesinkeepingaccuratewall‐clocktimewhenrunningineitheravirtualorphysicalmachine:initializingtheclocktothecorrecttimewhenbooting,andupdatingtheclockaccuratelyastimepasses.
Forinitializingtheclock,aVMwarevirtualmachineprovidesmechanismssimilartothoseofaphysicalmachine:avirtualbattery‐backedCMOSclockandvirtualnetworkcardsthatcanbeusedtofetchthetimefromanetworktimeserver.
Onefurthermechanismisalsoprovided:VMwareToolsresetstheguestoperatingsystem'sclocktomatchthehost'sclockuponstartup.
TheinterfacebetweenguestandhostusesUTC(CoordinatedUniversalTime,alsoknownasGreenwichMeanTimeorGMT),sotheguestandhostdonothavetobeinthesametimezone.
Virtualmachinesalsohaveafurtherissue:whenthevirtualmachineisresumedfromsuspendorrestoredfromasnapshot,theguestoperatingsystem'swall‐clocktimeremainsatthevalueithadatthetimeofthesuspensionorsnapshotandmustbeupdated.
VMwareToolshandlesthisissuetoo,settingthevirtualmachine'sclocktomatchthehost'sclockuponresumeorrestore.
However,becauseuserssometimesneedavirtualmachinetohaveitsclocksettoafictitioustimeunrelatedtothetimekeptonthehost,VMwareToolscanoptionallybetoldnevertochangethevirtualmachine'sclock.
Updatingtheclockaccuratelyoverthelongtermischallengingbecausethetimerdevicesinphysicalmachinestendtodrift,typicallyrunningasmuchas100partspermillionfastorslow,withtheratevaryingwithtemperature.
Thevirtualtimerdevicesinavirtualmachinehavethesameamountofinherentdriftastheunderlyinghardwareonthehost,andadditionaldriftandinaccuracycanariseasaresultofsuchfactorsasround‐offerrorandlostticks.
Inaphysicalmachine,itisgenerallynecessarytorunnetworkclocksynchronizationsoftwaresuchasNTPortheWindowsTimeServicetokeeptimeaccuratelyoverthelongterm.
Thesameappliestovirtualmachines,andthesameclocksynchronizationsoftwarecanbeused,althoughitsometimesneedstobeconfiguredspeciallyinordertodealwiththelesssmoothbehaviorofvirtualtimerdevices.
VMwareToolscanalsooptionallybeusedtocorrectlong‐termdriftanderrorsbyperiodicallyresynchronizingthevirtualmachine'sclocktothehost'sclock,butthecurrentversionatthiswritingislimited.

Inparticular,inguestoperatingsystemsotherthanNetWare,itdoesnotcorrecterrorsinwhichtheguestclockisaheadofrealtime,onlythoseinwhichtheguestclockisbehind.
Copyright2008VMware,Inc.
Allrightsreserved.
4TimekeepinginVMwareVirtualMachinesTimeandFrequencyUnitsThefollowingtableprovidesaquicksummaryofunitsinwhichtimeandfrequencyaremeasured:PCTimerHardwareForhistoricalreasons,PCscontainseveraldifferentdevicesthatcanbeusedtokeeptrackoftime.
Differentguestoperatingsystemsmakedifferentchoicesaboutwhichofthesedevicestouseandhowtousethem.
Usingseveralofthedevicesincombinationisimportantinmanyguestoperatingsystems.
Sometimes,onedevicethatrunsataknownspeedisusedtomeasurethespeedofanotherdevice.
Sometimesafine‐grainedtimingdeviceisusedtoaddadditionalprecisiontothetickcountobtainedfromamorecoarse‐grainedtimingdevice.
Thus,itisnecessarytosupportallthesedevicesinavirtualmachine,andthetimesreadfromdifferentdevicesusuallymustappeartobeconsistentwithoneanother,evenwhentheyaresomewhatinconsistentwithrealtime.
AllPCtimerdevicescanbedescribedusingroughlythesameblockdiagram,asshowninFigure1.
Notallthedeviceshaveallthefeaturesshown,andsomehaveadditionalfeatures,butthediagramisausefulabstraction.

Figure1.
AbstractTimerDeviceTheoscillatorprovidesafixedinputfrequencytothetimerdevice.
Thefrequencymaybespecified,ortheoperatingsystemmayhavetomeasureitatstartuptime.
Thecountermaybereadableorwritablebysoftware.

Thecountercountsdownoneunitforeachcycleoftheoscillator.
Whenthecounterreacheszero,itgeneratesanoutputsignalthatmayinterrupttheprocessor.
Atthispoint,ifthetimerissettoone‐shotmode,itstops;ifsettoperiodicmode,itcontinuescounting.
Theremayalsobeacounterinputregisterwhosevalueisloadedintothecounterwhenitreacheszero;thisregisterallowssoftwaretocontrolthetimerperiod.
Somerealtimerdevicescountupinsteadofdownandhavearegisterwhosevalueiscomparedwiththecountertodeterminewhentointerruptandrestartthecountatzero,butbothcount‐upandcount‐downtimerdesignsprovideequivalentfunctionality.
Table1.
UnitsforMeasuringTimeandFrequencyAbbreviationDescriptionsSecondsmsMilliseconds(1/1000second)μsMicroseconds(10‐6seconds)nsNanoseconds(10‐9seconds)HzFrequency(cyclesorothereventspersecond)KHzKilohertz(1000cyclesoreventspersecond)MHzMegahertz(106cyclesoreventspersecond)GHzGigahertz(109cyclesoreventspersecond)OscillatorCounterCounterinput=0InterruptCopyright2008VMware,Inc.
Allrightsreserved.
5TimekeepinginVMwareVirtualMachinesCommonPCtimerdevicesincludetheprogrammableintervaltimer(PIT),theCMOSrealtimeclock(RTC),thelocaladvancedprogrammableinterruptcontroller(APIC)timers,theadvancedconfigurationandpowerinterface(ACPI)timer,thetimestampcounter(TSC),andthehighprecisioneventtimer(HPET).
PITThePITistheoldestPCtimerdevice.
Itusesacrystal‐controlled1.
193182MHzinputoscillatorandhas16‐bitcounterandcounterinputregisters.
Theoscillatorfrequencywasnotchosenforconvenienttimekeeping;itwassimplyahandyfrequencyavailablewhenthefirstPCwasdesigned.
(Theoscillatorfrequencyisone‐thirdofthestandardNTSCtelevisioncolorburstfrequency.
)ThePITdeviceactuallycontainsthreeidenticaltimersthatareconnectedindifferentwaystotherestofthecomputer.
Timer0cangenerateaninterruptandissuitableforsystemtimekeeping.
Timer1washistoricallyusedforRAMrefreshandistypicallyprogrammedfora15microsecondperiodbythePCBIOS.
Timer2iswiredtothePCspeakerfortonegeneration.
CMOSRTCTheCMOSRTCispartofthebattery‐backedmemorydevicethatkeepsaPC'sBIOSsettingsstablewhilethePCispoweredoff.
ThenameCMOScomesfromthelow‐powerintegratedcircuittechnologyinwhichthisdevicewasoriginallyimplemented.
Therearetwomaintime‐relatedfeaturesintheRTC.
First,thereisacontinuouslyrunningtimeofday(TOD)clockthatkeepstimeinyear/month/dayhour:minute:secondformat.

Thisclockcanbereadonlytothenearestsecond.
Thereisalsoatimerthatcangenerateperiodicinterruptsatanypower‐of‐tworatefrom2Hzto8192Hz.
ThistimerfitstheblockdiagrammodelinFigure1,withtherestrictionthatthecountercannotbereadorwritten,andthecounterinputcanbesetonlytoapoweroftwo.

Twootherinterruptscanalsobeenabled:theupdateinterruptandthealarminterrupt.
Theupdateinterruptoccursoncepersecond.
ItissupposedtoreflecttheTODclockturningovertothenextsecond.
Thealarminterruptoccurswhenthetimeofdaymatchesaspecifiedvalueorpattern.
LocalAPICTimerThelocalAPICisapartoftheinterruptroutinglogicinmodernPCs.
Inamultiprocessorsystem,thereisonelocalAPICperprocessor.
Oncurrentprocessors,thelocalAPICisintegratedontotheprocessorchip.
ThelocalAPICincludesatimerdevicewith32‐bitcounterandcounterinputregisters.
Theinputfrequencyistypicallytheprocessor'sbasefront‐sidememorybusfrequency(beforethemultiplicationbytwoorfourforDDRorquad‐pumpedmemory).
Thus,thistimerismuchfiner‐grainedandhasawidercounterthanthePITorCMOStimers,butsoftwaredoesnothaveareliablewaytodetermineitsfrequency.
Generally,theonlywaytodeterminethelocalAPICtimer'sfrequencyistomeasureitusingthePITorCMOStimer,whichyieldsonlyanapproximateresult.
ACPITimerTheACPItimerisanadditionalsystemtimerthatisrequiredaspartoftheACPIspecification.
Thistimerisalsoknownasthepowermanagement(PM)timerorthechipsettimer.
Ithasa24‐bitcounterthatincrementsat3.
579545MHz(threetimesthePITfrequency).
Thetimercanbeprogrammedtogenerateaninterruptwhenitshigh‐orderbitchangesvalue.
Thereisnocounterinputregister;thecounteralwaysrollsover.
(Thatis,whenthecounterreachesthemaximum24‐bitbinaryvalue,itgoesbacktozeroandcontinuescountingfromthere.
)TheACPItimercontinuesrunninginsomepowersavingmodesinwhichothertimersarestoppedorslowed.
TheACPItimerisrelativelyslowtoread(typically1–2μs).
TSCTheTSCisa64‐bitcyclecounteronPentiumCPUsandnewerprocessors.
TheTSCrunsofftheCPUclockoscillator,typically2GHzormoreoncurrentsystems.
Atcurrentprocessorspeedsitwouldtakeyearstorollover.
TheTSCcannotgenerateinterruptsandhasnocounterinputregister.
TheTSCcanbereadbysoftwareinoneinstruction(rdtsc).
Therdtscinstructionisnormallyavailableinusermode,butoperatingsystemsoftwarecanchoosetomakeitunavailable.
TheTSCis,byfar,thefinestgrained,widest,andmostconvenienttimerdevicetoaccess.
However,theTSCalsohasseveraldrawbacks:Copyright2008VMware,Inc.
Allrightsreserved.
6TimekeepinginVMwareVirtualMachinesAswiththelocalAPICtimer,softwaredoesnothaveareliablewaytodeterminetheTSC'sinputfrequency.
Generally,theonlywaytodeterminetheTSC'sfrequencyistomeasureitapproximatelyusingthePITorCMOStimer.
Severalformsofpowermanagementtechnologyvarytheprocessor'sclockspeeddynamicallyandthuschangetheTSC'sinputoscillatorratewithlittleornonotice.
Inaddition,AMDOpteronK8processorsdropsomecyclesfromtheTSCwhenenteringandleavingahaltstateifthehaltclockrampingfeatureisenabled,eventhoughtheTSCratedoesnotchange.
ThelatestprocessorsfromIntelandAMDnolongerhavetheselimitations,however.
SomeprocessorsstoptheTSCintheirlower‐powerhaltstates(theACPIC3stateandbelow).
Onshared‐busSMPmachines,alltheTSCsrunoffacommonclockoscillator,so(intheabsenceoftheissuesnotedabove)theycanbesynchronizedcloselywitheachotheratstartuptimeandthereaftertreatedessentiallyasasinglesystem‐wideclock.
ThisdoesnotworkonIBMx‐SeriesNUMA(non‐uniformmemoryaccess)machinesandtheirderivatives,however.
Inthesemachines,differentNUMAnodesrunoffseparateclockoscillators.
AlthoughthenominalfrequenciesoftheoscillatorsoneachNUMAnodearethesame,eachoscillatoriscontrolledbyaseparatecrystalwithitsowndistinctdriftfromthenominalfrequency.
Inaddition,theclockratesareintentionallyvarieddynamicallyoverasmallrange(2percentorso)toreducetheeffectsofemittedRF(radiofrequency)noise,atechniquecalledspread‐spectrumclocking,andthisvariationisnotinstepacrossdifferentnodes.
DespitethesedrawbacksoftheTSC,bothoperatingsystemsandapplicationprogramsfrequentlyusetheTSCfortimekeeping.
HPETTheHPETisadeviceavailableinsomenewerPCs.
ManyPCsystemsdonothavethisdeviceandoperatingsystemsgenerallydonotrequireit,thoughsomecanuseitifavailable.
TheHPEThasonecentralup‐counterthatrunscontinuouslyunlessstoppedbysoftware.
Itmaybe32or64bitswide.
Thecounter'speriodcanbereadfromaregister.
TheHPETprovidesmultipletimers,eachconsistingofatimeoutregisterthatiscomparedwiththecentralcounter.
Whenatimeoutvaluematches,thecorrespondingtimerfires.
Ifthetimerissettobeperiodic,theHPEThardwareautomaticallyaddsitsperiodtothecompareregister,thuscomputingthenexttimeforthistimertofire.
TheHPEThasafewdrawbacks.
Thespecificationdoesnotrequirethetimertobeparticularlyfine‐grained,tohavelowdrift,ortobefasttoread.
Sometypicalimplementationsrunthecounteratabout18MHzandrequireaboutthesameamountoftime(1–2μs)toreadtheHPETaswiththeACPItimer.
Implementationshavebeenobservedinwhichtheperiodregisterisoffby800partspermillionormore.
Adrawbackofthegeneraldesignisthatsettingatimeoutraceswiththecounteritself.
Ifsoftwaretriestosetashorttimeout,butforanyreasonitswritetotheHPETisdelayedbeyondthepointatwhichthetimeoutistoexpire,thetimeoutiseffectivelysetfarinthefutureinstead(about232or264counts).
Softwarecanstopthecentralcounter,butdoingsowouldspoilitsusefulnessforlong‐termtimekeeping.
TheHPETisdesignedtobeabletoreplacethePITandCMOSperiodictimersbydrivingtheinterruptlinestowhichthePITandCMOStimersarenormallyconnected.
MostcurrenthardwareplatformsstillhavephysicalPITandCMOStimersanddonotneedtousetheHPETtoreplacethem.
VMwareTimerVirtualizationVMwareproductsuseapatent‐pendingtechniquetoallowthemanytimerdevicesinavirtualmachinetofallbehindrealtimeandcatchupasneeded,yetremainsufficientlyconsistentwithoneanotherthatsoftwarerunninginthevirtualmachineisnotdisruptedbyanomaloustimereadings.
InVMwareterminology,thetimethatisvisibletovirtualmachinesontheirtimerdevicesiscalledapparenttime.
Generally,thetimerdevicesinavirtualmachineoperateidenticallytothecorrespondingtimerdevicesinaphysicalmachine,buttheyshowapparenttimeinsteadofrealtime.
Thefollowingsectionsnotesomeexceptionstothisruleandprovidesomeadditionaldetailsabouteachemulatedtimerdevice.
Copyright2008VMware,Inc.
Allrightsreserved.
7TimekeepinginVMwareVirtualMachinesVirtualPITVMwareproductsfullyemulatethetimingfunctionsofallthreetimersinthePITdevice.
Inaddition,whentheguestoperatingsystemprogramsthespeakertimertogenerateasound,thevirtualmachinerequestsabeepsoundfromthehostmachine.
However,thesoundgeneratedonthehostmaynotbetherequestedfrequencyorduration.
VirtualCMOSRTCCurrentVMwareproductsemulateallthetimingfunctionsoftheCMOSRTC,includingthetimeofdayclockandtheperiodic,update,andalarminterruptsthattheCMOSRTCprovides.
ManyguestoperatingsystemsusetheCMOSperiodicinterruptasthemainsystemtimer,soVMwareproductsrunitinapparenttimetobeconsistentwiththeothertimerdevices.
SomeguestoperatingsystemsusetheCMOSupdateinterrupttocountoffpreciselyonesecondtomeasuretheCPUspeedorthespeedofothertimerdevices,soVMwareproductsruntheCMOSupdateinterruptinapparenttimeaswell.
Incontrast,VMwareproductsbasethevirtualCMOSTODclockdirectlyontherealtimeasknowntothehostsystem,notonapparenttime.
ThischoicemakessensebecauseguestoperatingsystemsgenerallyreadtheCMOSTODclockonlytoinitializethesystemtimeatpoweronandoccasionallytocheckthesystemtimeforcorrectness.
OperatingsystemsusetheCMOSTODclockthiswaybecauseitprovidestimeonlytothenearestsecondbutisbatterybackedandthuscontinuestokeeptimeevenwhenthesystemlosespowerorisrestarted.

Specifically,theCMOSTODclockshowsUTCaskeptbythehostoperatingsystemsoftware,plusanoffset.
TheoffsetfromUTCisstoredinthevirtualmachine'snvramfilealongwiththerestofthecontentsofthevirtualmachine'sCMOSnonvolatilememory.
TheoffsetisneededbecausemanyguestoperatingsystemswanttheCMOSTODclocktoshowthetimeinthecurrentlocaltimezone,notinUTC.
Whenyoucreateanewvirtualmachine(ordeletethenvramfileofanexistingvirtualmachine)andpoweriton,theoffsetisinitialized,bydefault,tothedifferenceofthehostoperatingsystem'slocaltimezonefromUTC.
IfsoftwarerunninginthevirtualmachinewritesanewtimetotheCMOSTODclock,theoffsetisupdated.
YoucanforcetheCMOSTODclock'soffsettobeinitializedtoaspecificvalueatpoweron.
Todoso,settheoptionrtc.
diffFromUTCinthevirtualmachine's.
vmxconfigurationfiletoavalueinseconds.
Forexample,settingrtc.
diffFromUTC=0setstheclocktoUTCatpoweron,whilesettingrtc.
diffFromUTC=-25200setsittoPacificDaylightTime,sevenhoursearlierthanUTC.
TheguestoperatingsystemcanstillchangetheoffsetvalueafterpoweronbywritinganewtimetotheCMOSTODclock.
YoucanalsoforcetheCMOSTODclocktostartataspecifiedtimewheneverthevirtualmachineispoweredon,independentoftherealtime.
Todothis,settheconfigurationfileoptionrtc.
startTime.
ThevalueyouspecifyisinsecondssinceJan1,197000:00UTC,butitisconvertedtothelocaltimezoneofthehostoperatingsystembeforesettingtheCMOSTODclock(undertheassumptionthattheguestoperatingsystemwantstheCMOSTODclocktoreadinlocaltime).
IfyourguestoperatingsystemisrunningtheCMOSTODclockinUTCorsomeothertimezone,youshouldcorrectforthiswhensettingrtc.
startTime.
ThevirtualCMOSTODclockhasthefollowinglimitation:Becausetheclockisimplementedasanoffsetfromthehostoperatingsystem'ssoftwareclock,itchangesvalueifyouchangethehostoperatingsystemtime.

(Changingthehosttimezonehasnoeffect,onlychangingtheactualtime.
)Inmostcasesthiseffectisharmless,butitdoesmeanthatyoushouldneveruseavirtualmachineasatimeserverprovidingtimetothehostoperatingsystemthatitisrunningon.
Doingthiscancreateaharmfulpositivefeedbackloop,inwhichanychangemadetothehosttimeincorrectlychangestheguesttime,too,causingthehosttimetoappearwrongagain,whichcausesafurtherchangetothehosttime,etc.
WhetherthiseffectoccursandhowsevereitisdependsonhowtheguestoperatingsystemusestheCMOSTODclock.
SomeguestoperatingsystemsmaynotusetheCMOSTODclockatall,inwhichcasetheproblemdoesnotoccur.
SomeguestssynchronizetotheCMOSTODclockonlyatboottime,inwhichcasetheproblemdoesoccurbutthesystemgoesarounditsfeedbacklooponlyonceperguestboot.
Youcanusertc.
diffFromUTCtobreaksuchafeedbackloop,butitisbettertoavoidtheloopinthefirstplacebynotusingthevirtualmachineasatimeserverforthehost.
SomeguestoperatingsystemsperiodicallyresynchronizetotheCMOSTODclock(say,onceperhour),inwhichcasethefeedbackismorerapidandrtc.
diffFromUTCcannotbreaktheloop.
BecausethealarminterruptisdesignedtobetriggeredwhentheCMOSTODclockreachesaspecificvalue,thealarminterruptalsooperatesinrealtime,notapparenttime.
Copyright2008VMware,Inc.
Allrightsreserved.
8TimekeepinginVMwareVirtualMachinesThechoiceofrealorapparenttimeforeachfeatureoftheCMOSRTCdevicereflectsthewayguestoperatingsystemscommonlyusethedevice.
Guestoperatingsystemstypicallyhavenodifficultywithpartofthedeviceoperatinginapparenttimeandotherpartsoperatinginrealtime.
However,oneunsupportedguestoperatingsystem(USLUnixSystemVRelease4.
21)isknowntocrashifitseestheCMOSdevice'supdate‐in‐progress(UIP)bitsetwhilestartingup.
Itisnotknownwhetherthiscrashwouldoccuronrealhardwareorwhethertheguestoperatingsystemisconfusedbythefactthattheupdateinterrupt,theUIPbit,andtherolloveroftheCMOSTODclocktothenextseconddonotalloccuratthesamemoment,astheywouldonrealhardware.
Youcanworkaroundthisproblembysettingrtc.
doUIP=FALSEinthevirtualmachinesconfigurationfile,whichforcestheUIPbittoalwaysreturn0.
VirtualLocalAPICTimerVMwareproductsfullyemulatethelocalAPICtimeroneachvirtualCPU.
Thetimer'sfrequencyisnotdependentonthehost'sAPICtimerfrequency.
Inmostcases,thelocalAPICtimerrunsinapparenttime,matchingtheothertimerdevices.
However,someVMwareproductsareabletorecognizecasesinwhichtheguestoperatingsystemisusingticklesstimekeepingbuthasneverthelesssetupaperiodiclocalAPICtimerinterrupt.
Inthesecases,thelocalAPICtimerrunsina"lazy"mode,inwhichreadingitscounterreturnsthecurrentapparenttime,butlateAPICtimerinterruptsareallowedtopileupandmergeratherthanbeingaccumulatedasabacklogandcausingapparenttimetobeheldbackuntilthebacklogiscaughtup.
Also,APICtimerinterruptsondifferentvirtualCPUsareallowedtooccurslightlyoutoforder.
VirtualACPITimerVMwareproductsfullyemulatea24‐bitACPItimer.
Thetimerrunsinapparenttime,matchingtheothertimerdevices.
Itgeneratesaninterruptwhenthehigh‐orderbitchangesvalue.
VirtualTSCCurrentVMwareproductsvirtualizetheTSCinapparenttime.
ThevirtualTSCstaysinstepwiththeothertimerdevicesvisibleinthevirtualmachine.
Likethosedevices,thevirtualTSCfallsbehindrealtimewhenthereisabacklogoftimerinterruptsandcatchesupasthebacklogiscleared.
Thus,thevirtualTSCdoesnotcountcyclesofcoderunonthevirtualCPU;itadvancesevenwhenthevirtualCPUisnotrunning.
ThevirtualTSCalsodoesnotmatchtheTSCvalueonthehosthardware.
Whenavirtualmachineispoweredon,itsvirtualTSCisset,bydefault,torunatthesamerateasthehostTSC.
Ifthevirtualmachineisthenmovedtoadifferenthostwithoutbeingpoweredoff(thatis,eitherusingVMotionorsuspendingthevirtualmachineononehostandresumingitonanother),thevirtualTSCcontinuestorunatitsoriginalpower‐onrate,notatthehostTSCrateonthenewhostmachine.
YoucanforcethevirtualTSC'sratetoaspecificvalueN(incyclespersecondorHz)byaddingthesettingtimeTracker.
apparentHz=Ntothevirtualmachine's.
vmxconfigurationfile.
Thisfeatureisrarelyneeded.
Onepossibleuseistotestforbugsinguestoperatingsystems—forexample,Linux2.
2kernelshangduringstartupiftheTSCrunsfasterthan4GHz.
Notethatthisfeaturedoesnotchangetherateatwhichinstructionsareexecuted.
Inparticular,youcannotmakeprogramsrunmoreslowlybysettingthevirtualTSC'sratetoalowervalue.
YoucandisablevirtualizationoftheTSCbyaddingthesettingmonitor_control.
virtual_rdtsc=FALSEtothevirtualmachine's.
vmxconfigurationfile.
Thisfeatureisnolongerrecommendedforuse.
WhenyoudisablevirtualizationoftheTSC,readingtheTSCfromwithinthevirtualmachinereturnsthephysicalmachine'sTSCvalue,andwritingtheTSCfromwithinthevirtualmachinehasnoeffect.
Migratingthevirtualmachinetoanotherhost,resumingitfromsuspendedstate,orrevertingtoasnapshotcausestheTSCtojumpdiscontinuously.
SomeguestoperatingsystemsfailtobootorexhibitothertimekeepingproblemswhenTSCvirtualizationisdisabled.
Inthepast,thisfeaturehassometimesbeenrecommendedtoimproveperformanceNOTEDonotusethertc.
doUIP=FALSEsettingunlessyouarerunningaguestoperatingsystemthatrequiresit.
Settingthisvalueforotherguestoperatingsystemsmaypreventtimekeepingfromworkingcorrectly.
Copyright2008VMware,Inc.
Allrightsreserved.
9TimekeepinginVMwareVirtualMachinesofapplicationsthatreadtheTSCfrequently,butperformanceofthevirtualTSChasbeenimprovedsubstantiallyincurrentproducts.
Thefeaturehasalsobeenrecommendedforusewhenperformingmeasurementsthatrequireaprecisesourceofrealtimeinthevirtualmachine,butforthispurpose,thepseudoperformancecountersdiscussedinthenextsectionareabetterchoice.
PseudoperformanceCountersForcertainapplicationsitcanbeusefultohavedirectaccesstorealtime(asopposedtoapparenttime)withinavirtualmachine.
Forexample,youmaybewritingperformancemeasuringsoftwarethatisawareitisrunninginavirtualmachineanddoesnotrequireitsfine‐grainedtimertostayinstepwiththenumberofinterruptsdeliveredonothertimerdevices.
VMwarevirtualmachinesprovideasetofpseudoperformancecountersthatsoftwarerunninginthevirtualmachinecanreadwiththerdpmcinstructiontoobtainfine‐grainedtime.
Toenablethisfeature,usethefollowingconfigurationfilesetting:monitor_control.
pseudo_perfctr=TRUEThefollowingmachineinstructionsthenbecomeavailable:AlthoughtherdpmcinstructionnormallyisprivilegedunlessthePCEflagissetintheCR4controlregister,aVMwarevirtualmachinepermitstheabovepseudoperformancecounterstobereadfromuserspaceregardlessofthesettingofthePCEflag.
NotethatthepseudoperformancecounterfeatureusesatraptocatchaprivilegedmachineinstructionissuedbysoftwarerunninginthevirtualmachineandthushasmoreoverheadthanreadingaperformancecounterortheTSConphysicalhardware.
Therearesomelimitations.
SomeorallofthesecountersmaynotbeavailableonolderversionsofVMwareproducts.
Inparticular,elapsedrealtimeandelapsedapparenttimewerefirstintroducedinVMwareESX3.
5andVMwareWorkstation6.
5.
Thezeropointforthecountersiscurrentlyunspecified.
ThephysicalhostTSCmaychangeitscountingrate,jumptoadifferentvalue,orbothwhenthevirtualmachinemigratestoadifferenthostorisresumedfromsuspendorrevertedtoasnapshot.
Theelapsedrealtimecounterrunsataconstantratebutmayjumptoadifferentvaluewhenthevirtualmachinemigratestoadifferenthostorisresumedfromsuspendorrevertedtoasnapshot.
VirtualHPETCurrentVMwareproductsdonotprovideavirtualHPET,becausecurrentlysupportedguestoperatingsystemsdonotrequireone.
OtherTime-DependentDevicesComputergenerationofsoundistime‐sensitive.
Thesoundsthatavirtualmachinegeneratesarealwaysplayedbythehostmachine'ssoundcardatthecorrectsamplerate,regardlessoftimerbehaviorinthevirtualmachine,sotheyalwaysplayattheproperpitch.
Also,thereisenoughbufferingbetweenthevirtualsoundcardofthevirtualmachineandthehostmachine'ssoundcardsothatsoundsusuallyplaycontinuously.
However,therecanbegapsorstutteringifthevirtualmachinefallsfarenoughbehindthatthesupplyofbufferedsoundinformationavailabletoplayisexhausted.
PlaybackofMIDImusic(aswellassomeotherformsofmultimedia),however,requiressoftwaretoprovidedelaysforthecorrectamountoftimebetweennotesorotherevents.
Thus,playbackcanslowdownorspeedupiftheapparenttimedeviatestoofarfromrealtime.
Table2.
InstructionsAvailableWhenPseudoperformanceCountersAreEnabledInstructionTimeValueReturnedrdpmc0x10000PhysicalhostTSCrdpmc0x10001Elapsedrealtimeinnsrdpmc0x10002ElapsedapparenttimeinnsCopyright2008VMware,Inc.
Allrightsreserved.
10TimekeepinginVMwareVirtualMachinesVGAvideocardsproduceverticalandhorizontalblankingsignalsthatdependonamonitor'svideoscanrate.

VMwarevirtualmachinescurrentlymakenoattempttoemulatethesesignalswithaccuratetiming.
Thereisverylittlesoftwarethatusesthesesignalsfortiming,butafewoldgamesdousethem.
Thesegamescurrentlyarenotplayableinavirtualmachine.
VMIParavirtualTimerTheVirtualMachineInterface(VMI)isanopenparavirtualizationinterfacedevelopedbyVMwarewithinputfromtheLinuxcommunity.
VMIisanopenstandard,thespecificationforwhichisavailableathttp://www.
vmware.
com/pdf/vmi_specs.
pdf.
VMIiscurrentlydefinedonlyfor32‐bitguests.
VMwareproductsbeginningwithWorkstation6andESX3.
5supportVMI.
VMIincludesaparavirtualtimerdevicethattheguestoperatingsystemkernelcanuseforticklesstimekeeping.
Inaddition,VMIallowstheguestkerneltoexplicitlyaccountfor"stolentime";thatis,timewhentheguestoperatingsystemwasreadytorunbutthevirtualmachinewasdescheduledbythehostscheduler.
VMIcouldbeusedbyanyguestoperatingsystem,butcurrentlyonlyLinuxusesit.
See"Linux"onpage11.
TimekeepinginSpecificOperatingSystemsThissectiondetailssomeofthepeculiaritiesofspecificoperatingsystemsthataffecttheirtimekeepingperformancewhentheyarerunasguestsinvirtualmachines.
AfewoftheseissuesalsoaffecttimekeepingbehaviorwhentheseoperatingsystemsarerunashostsforVMwareWorkstationandotherVMwarehostedproducts.
MicrosoftWindowsMicrosoftWindowsoperatingsystemsgenerallykeeptimebycountingtimerinterrupts(ticks).
Systemtimeofdayispreciseonlytothenearesttick.
ThetimerdeviceusedandthenumberofinterruptsgeneratedpersecondvarydependingonwhichspecificversionofMicrosoftWindowsandwhichWindowshardwareabstractionlayer(HAL)areinstalled.
SomeuniprocessorWindowsconfigurationsusethePITastheirmainsystemtimer,butmultiprocessorHALsandsomeACPIuniprocessorHALsusetheCMOSperiodictimerinstead.
ForsystemsusingthePIT,thebaseinterruptrateisusually100Hz,althoughWindows98uses200Hz.

ForsystemsthatusetheCMOStimer,thebaseinterruptrateisusually64Hz.
MicrosoftWindowsalsohasafeaturecalledthemultimediatimerAPIthatcanraisethetimerratetoashighas1024Hz(or1000HzonsystemsthatusethePIT)whenitisused.
Forexample,ifyourvirtualmachinehastheAppleQuickTimeiconinthesystemtray,evenifQuickTimeisnotplayingamovie,theguestoperatingsystemtimerrateisraisedto1024Hz.
Thisfeatureisnotusedexclusivelybymultimediaapplications.
Forexample,someimplementationsoftheJavaruntimeenvironmentraisethetimerrateto1024Hz,sorunninganyJavaapplicationmayraiseyourtimerrate,dependingontheversionoftheruntimeyouareusing.
ThisfeatureisalsousedbyVMwarehostedproductsrunningonaWindowshostsystem,tohandlecasesinwhichoneormoreofthecurrentlyrunningvirtualmachinesrequireahighervirtualtimerinterruptratethanthehost'sdefaultphysicalinterruptrate.
MicrosoftWindowshasanadditionaltimemeasurementfeatureaccessedthroughtheQueryPerformanceCountersystemcall.
Thisnameisamisnomer,becausethecallneveraccessestheCPU'sperformancecounterregisters.
Instead,itreadsoneofthetimerdevicesthathaveacounter,allowingtimemeasurementwithafinergranularitythantheinterrupt‐countingsystemtimeofdayclock.
Whichtimerdeviceisused(theACPItimer,theTSC,thePIT,orsomeotherdevice)dependsonthespecificWindowsversionandHALinuse.
SomeversionsofWindows,especiallymultiprocessorversions,settheTSCregistertozeroduringtheirstartupsequence,inparttoensurethattheTSCsofalltheprocessorsaresynchronized.
MicrosoftWindowsalsomeasuresthespeedofeachprocessorbycomparingtheTSCagainstoneoftheothersystemtimersduringstartup,andthiscodealsosetstheTSCtozeroinsomecases.
SomemultiprocessorversionsoftheWindowsoperatingsystemprogramthelocalAPICtimerstogenerateoneinterruptpersecond.
OtherversionsofWindowsdonotusethesetimersatall.
Copyright2008VMware,Inc.
Allrightsreserved.
11TimekeepinginVMwareVirtualMachinesSomemultiprocessorversionsofWindowsroutethemainsystemtimerinterruptasabroadcasttoallprocessors.
Othersroutethisinterruptonlytotheprimaryprocessoranduseinterprocessorinterruptsforschedulertimeslicingonsecondaryprocessors.
Toinitializethesystemtimeofdayonstartup,MicrosoftWindowsreadsthebattery‐backedCMOSTODclock.
Occasionally,Windowsalsowritestothisclocksothatthetimeisapproximatelycorrectonthenextstartup.
WindowskeepstheCMOSTODclockinlocaltime,soinregionsthatturntheirclocksaheadbyanhourduringthesummer,WindowsmustupdatetheCMOSTODclocktwiceayeartoreflectthechange.
SomerarefailuremodescanputtheCMOSTODclockoutofstepwiththeWindowsregistrysettingthatrecordswhetherithasbeenupdated,causingtheWindowsclocktobeoffbyanhourafterthenextreboot.
IfVMwareToolsisinstalledinthevirtualmachine,itcorrectsanysucherroratboottime.
AdaemonpresentinWindowsNT‐familysystems(thatis,WindowsNT4.
0andlater)checksthesystemtimeofdayagainsttheCMOSTODclockonceperhour.
Ifthesystemtimeisoffbymorethan60seconds,itisresettomatchtheTODclock.
Thisbehaviorisgenerallyharmlessinavirtualmachineandmaybeusefulinsomecases,suchaswhenVMwareToolsorothersynchronizationsoftwareisnotinuse.
Onepossible(thoughrare)problemcanoccurifthedaemonsetstheclockaheadwhilethevirtualmachineisintheprocessofcatchinguponaninterruptbacklog.
Becausethevirtualmachineisnotawarethattheguestoperatingsystemclockhasbeenreset,itcontinuescatchingup,causingtheclocktoovershootrealtime.
IfyouturnonperiodicclocksynchronizationinVMwareTools,itdisablesthisdaemon.
ForadiscussionofW32TimeandotherWindowsclocksynchronizationsoftware,see"SynchronizingVirtualMachinesandHostswithRealTime"onpage14.
LinuxTimekeepinginLinuxhaschangedagreatdealoveritshistory.
Recently,thedirectionofkerneldevelopmenthasbeentowardbetterbehaviorinavirtualmachine.
However,alongtheway,anumberofkernelshavehadspecificbugsthatarestronglyexposedwhenruninavirtualmachine.
Somekernelshaveveryhighinterruptrates,resultinginpoortimekeepingperformanceandimposingexcessivehostloadevenwhenthevirtualmachineisidle.
Inmostcases,thelatestVMware‐supportedversionofaLinuxdistributionhasthebesttimekeepingbehavior.
SeeVMwareknowledgebasearticle1006427(http://kb.
vmware.
com/kb/1006427)forspecificrecommendations,includingworkaroundsforbugsandperformanceissueswithspecificdistributionvendorkernels.
TheremainderofthissectiondescribestheoveralldevelopmentandcharacteristicsoftheLinuxtimekeepingimplementationinmoredetail.
Linuxkernelversion2.
4andearlierversionsof2.
6usedtickcountingexclusively,withPIT0usuallyusedasthesourceofticks.
Morerecently,aseriesofchangesamountingtoarewriteofthetimekeepingsubsystemweremadetothekernel.
Thefirstmajorroundofchanges,whichwentintothe32‐bitkernelinversion2.
6.
18andthe64‐bitkernelin2.
6.
21,addedanabstractionlayercalledclocksourcetothetimekeepingsubsystem.

Thekernelcanselectfromseveralclocksourcesatboottime.
Generally,eachoneusesadifferenthardwaretimerdevicewithappropriatesupportcodetoimplementtheclocksourceinterface.
Almostalltheprovidedclocksourcesaretickless,includingallthecommonlyusedones.
Thenextmajorroundofchanges,completedin32‐bit2.
6.
21and64‐bit2.
6.
24,addtheclockeventsabstractiontothekernel.
ThesechangesaddtheNO_HZkernelconfigurationoption,which,whenenabledatkernelcompiletime,switchesthekerneltousinganaperiodic(one‐shot)interruptforsystemtimercallbacks,processaccounting,andscheduling.
VersionsofLinuxpriortotheintroductionofNO_HZrequireaperiodicinterruptforschedulertimeslicingandstatisticalprocessaccounting.
MostconfigurationsusethelocalAPICtimeroneachCPUtogenerateschedulerinterruptsforthatCPU,butinsomeuniprocessorconfigurations,theschedulerisdrivenfromthesameinterruptusedfortimekeeping.
UserapplicationsonLinuxcanrequestadditionaltimerinterruptsusingthe/dev/rtcdevice.
TheseinterruptscomeeitherfromtheCMOSperiodictimerortheHPET.
Thisfeatureisusedbysomemultimediasoftware.
ItisalsousedbyVMwarehostedproductsrunningonaLinuxhostsystem,tohandlecasesinwhichoneormoreofthecurrentlyrunningvirtualmachinesrequiresahighervirtualtimerinterruptratethanthehost'sdefaultphysicalinterruptrate.
SeeVMwareKnowledgeBasearticle892(http://kb.
vmware.
com/kb/892).

TheLinuximplementationofthisfeatureusingtheHPETcansometimesstopdeliveringinterruptsbecauseofthetimeout‐settingracementionedin"HPET"onpage6.
Copyright2008VMware,Inc.
Allrightsreserved.
12TimekeepinginVMwareVirtualMachinesMostLinuxdistributionsaresetuptoinitializethesystemtimefromthebattery‐backedCMOSTODclockatstartupandtowritethesystemtimebacktotheCMOSTODclockatshutdown.
Insomecases,LinuxkernelsalsowritethesystemtimetotheCMOSTODclockperiodically(onceevery11minutes).
YoucanmanuallyreadorsettheCMOSTODclockusingthe/sbin/hwclockprogram.
KernelsBeforeClocksourceLinuxkernelspriortotheintroductionoftheclocksourceabstractioncountperiodictimerinterruptsastheirbasicmethodoftimekeeping.
LinuxkernelsgenerallyusePIT0astheirmainsourceoftimerinterrupts.
Theinterruptrateuseddependsonthekernelversion.
Linux2.
4andearlierkernelsgenerallyprogramthePIT0timertodeliverinterruptsat100Hz.
Somevendorpatchesto2.
4kernelsincreasethisrate.
Inparticular,theinitialreleaseofRedHatLinux8andsomeupdatestoRedHatLinux7used512Hz,butlaterupdatesrevertedtothestandard100Hzrate.
SUSELinuxProfessional9.
0uses1000Hzwhenthedesktopboot‐timeoptionisprovidedtothekernel,andtheSUSEinstallationprogramsetsthisoptionbydefault.
EarlyLinux2.
6kernelsusedarateof1000Hz.
Thisratewaslatermadeconfigurableatkernelcompiletime,with100Hz,250Hz,and1000Hzasstandardchoicesand250Hzasthedefault;however,somevendors(includingRedHatintheRedHatEnterpriseLinux4andRedHatEnterpriseLinux5series)continuedtoshipkernelsconfiguredfor1000Hz.
ThelatestversionsinboththeRedHatEnterpriseLinux4andRedHatEnterpriseLinux5seriesincludeadivider=boot‐timeoptionthatcanreducetheinterruptrate—forexample,divider=10reducestheinterruptrateto100Hz.
Asmentionedabove,mostkernelsalsoprogramthelocalAPICtimeroneachCPUtodeliverperiodicinterruptstodrivethescheduler.
TheseinterruptsoccuratapproximatelythesamebaserateasthePIT0timer.

Thus,aone‐CPUvirtualmachinerunninganSMPLinux2.
4kernelrequiresatotalof200timerinterruptspersecondacrossallsources,whileatwo‐CPUvirtualmachinerequires300interruptspersecond.
Aone‐CPULinux2.
6kernelvirtualmachinethatusestickcountingfortimekeepingandthelocalAPICtimerforschedulingrequiresatotalof2000timerinterruptspersecond,whileatwo‐CPUvirtualmachinerequires3000interruptspersecond.
32‐bitLinuxkernel2.
4andearlierversionsinterpolatethesystemtime(asreturnedbythegettimeofdaysystemcall)betweentimerinterruptsusinganalgorithmthatissomewhatpronetoerrors.
First,thekernelcountsPITtimerinterruptstokeeptrackoftimetothenearest10milliseconds.
Whenatimerinterruptisreceived,thekernelreadsthePITcountertomeasureandcorrectforthelatencyinhandlingtheinterrupt.
ThekernelalsoreadsandrecordstheTSCatthispoint.
Oneachcalltogettimeofday,thekernelreadstheTSCagainandaddsthechangesincethelasttimerinterruptwasprocessedtocomputethecurrenttime.
Implementationsofthisalgorithmhavehadvariousproblemsthatresultinincorrecttimereadingsbeingproducedwhencertainraceconditionsoccur.
Theseproblemsarefairlyrareonrealhardwarebutaremorefrequentinavirtualmachine.
Thealgorithmisalsosensitivetolostticks(asdescribedearlier),andtheseseemtooccurmoreofteninavirtualmachinethanonrealhardware.
Asaresult,ifyourunaprogramthatloopscallinggettimeofdayrepeatedly,youmayoccasionallyseethevaluegobackward.
Thisoccursbothonrealhardwareandinavirtualmachinebutismorefrequentinavirtualmachine.
Most32‐bitversionsofLinuxkernel2.
6thatpredateclocksourceimplementseveraldifferentalgorithmsforinterpolatingthesystemtimeandletyouchooseamongthemwiththeclock=kernelcommandlineoption.
Unfortunately,alltheavailableoptionshavesomedrawbacks.
Twoofthealgorithmsincorporatecodethatattemptstodetectlostticksfromnonprocessedtimerinterruptsautomaticallyandaddextratickstocorrectforthetimeloss.
Unfortunately,thesealgorithmsoftenovercorrect:theyaddextra,spurioustickstotheoperatingsystemclockwhentimerinterrupthandlingisdelayedsuchthattwointerruptsarehandledinclosesuccessionbutneitherislost.
Suchbunchingofinterruptsoccursoccasionallyonrealhardware,usuallybecauseaCPUisbusyhandlingothertaskswhileinterruptsaretemporarilydisabled.
Thisproblemoccursmuchmorefrequentlyinavirtualmachinebecauseofthevirtualmachine'sneedtosharetherealCPUwithotherprocesses.
Thus,thisproblemcancausetheclocktoruntoofastbothonrealhardwareandinavirtualmachine,buttheeffectismuchmorenoticeableinavirtualmachine.
Thefollowingparagraphsdescribethetimeinterpolationalgorithmsof32‐bitLinux2.
6kernelspredatingclocksourcethatareusableinavirtualmachine:Copyright2008VMware,Inc.
Allrightsreserved.
13TimekeepinginVMwareVirtualMachinesTheoptionclock=tscselectsanalgorithmthatmakesuseofthePITcounterandtheTSCfortimeinterpolation.
ThisalgorithmissimilartothatofLinuxkernel2.
4butincorporateslosttickcorrection.
Aspreviouslynoted,themethodsusedtoadjusttimeforlostticksmayovercorrect,makingtheclockruntoofast.
Timegainsof10percentormorehavebeenobservedwhenrunningthisalgorithminavirtualmachine.
Thisoptionisthedefaultonsomekernels.
Theoptionclock=pmtmrselectsasimplerbutmorerobustalgorithmthatmakesuseoftheACPItimerforinterpolation.
Thisoptionalsoincludeslosttickcorrectioncodethatmaycausetimegains.
However,whenusedinavirtualmachine,timegainsfromusingthisoptionaremuchsmaller.
Thisoptionisusableinavirtualmachine,ifyoucantoleratethesmalltimegain.
Thisoptionisthedefaultonsomekernels.
Theoptionclock=pitusesonlythePITcounterforinterpolation.
Itdoesnotincludelosttickcorrectioncode,soitdoesnotgaintime,butitdoeslosetimewhenatickisactuallylost.
Unfortunately,thisoptionhasadifferentbugonmostkernels,suchthatifyouwriteaprogramthatcallsgettimeofdayrepeatedly,itfrequentlyjumpsbackwardbyabout1ms,thencorrectsitself.
The64‐bitLinuxkernelsthatpredateclocksourceimplementtimekeepingwithamethoddifferentfromthatusedinthecorresponding32‐bitkernels.
Unfortunately,thisimplementationalsohasproblemswithlosttickcorrectioncodethatovercorrectsandcausesatimegain.
Evensome64‐bitkernelsinthe2.
4serieshavelosttickovercorrection—forexample,thekernelincludedin64‐bitRedHatEnterpriseLinux3hasthisproblem.

The64‐bittimekeepingimplementationhasaslightlydifferentsetofavailablealgorithmsandiscontrolledbyadifferentboot‐timeoption.
Onealgorithmissimilartothe32‐bitclock=tscalgorithmdiscussedaboveandhasthesameproblemofseverelosttickovercorrection.
Thisoptionisthedefaultonmostkernelsandistheonlyoneavailableonolderkernels.
Analternativealgorithmissimilartothe32‐bitclock=pmtmralgorithmdiscussedabove.
Ithasthesameproblemofminorlosttickovercorrection.
Onsomekernelsthisalgorithmisnotavailableatall;onsomeitisselectedautomaticallyforAMDprocessorsbutcannotbeselectedmanually.
Onmostkernels,however,thisalgorithmcanbeselectedusingthenotsckernelcommandlineoption.
SeeVMwareknowledgebasearticle1006427(http://kb.
vmware.
com/kb/1006427)forrecommendedoptionstouseonspecificvendorkernels.
ClocksourceKernelsWiththenewclocksourceabstraction,thekernel'shigh‐leveltimekeepingcodebasicallydealsonlywithwall‐clocktimeandNTPratecorrection.
Itcallsintoalower‐levelclocksourcedrivertoreadacounterthatreflectstherawamountoftime(withoutratecorrection)thathaspassedsinceboot.
TheavailableclocksourcedriversgenerallydonotuseanyoftheproblematictechniquesfromearlierLinuxtimekeepingimplementations,suchasusingonetimerdevicetointerpolatebetweentheticksofanotherordoinglosttickcompensation.
Infact,mostoftheclocksourcedriversaretickless.
TheTSCclocksource(usuallythedefault)basicallyjustreadstheTSCvalueandreturnsit.
TheACPIPMtimerclocksourceissimilar,asthekernelhandlestimersthatwrap(whichoccursabouteveryfoursecondswitha24‐bitACPIPMtimer)andextendstheirrangeautomatically.
Theclocksourceabstractionisagoodmatchforvirtualmachines,thoughnotperfect.
TheTSCdoesnotrunatapreciselyspecifiedrate,sotheguestoperatingsystemhastomeasureitsrateatboottime,andthismeasurementisalwayssomewhatinaccurate.
RunningNTPorotherclocksynchronizationsoftwareintheguestcancompensateforthisissue,however.
TheACPIPMtimerdoesrunatapreciselyspecifiedratebutisslowertoreadthantheTSC.
Also,whenclocksourceisusedwithoutNO_HZ,theguestoperatingsystemstillprogramsatimertointerruptperiodically,sobydefault,thevirtualmachinestillkeepstrackofabacklogoftimerinterruptsandtriestocatchupgradually.
TheNO_HZoptionprovidesafurthersignificantimprovement.
Becausetheguestoperatingsystemdoesnotscheduleanyperiodictimers,thevirtualmachinecanneverhaveabackloggreaterthanonetimerinterrupt,soapparenttimedoesnotfallfarbehindrealtimeandcatchesupveryquickly.
Alsoimportant,NO_HZtendstoreducetheoverallaveragerateofvirtualtimerinterrupts,improvingsystemthroughputandscalabilitytolargernumbersofvirtualmachinesperhost.
Copyright2008VMware,Inc.
Allrightsreserved.
14TimekeepinginVMwareVirtualMachinesAlternatively,evenonkernelswithoutNO_HZ,softwarerunninginthevirtualmachinecanmakeahypercalltoinformthevirtualmachinethatitistickless.
Forexample,the64‐bitSUSELinuxEnterpriseServer10SP2kerneldoesthis.
ParavirtualKernelsWithsome32‐bitkernelversions,youcanuseVMI(see"VMIParavirtualTimer"onpage10)toobtainticklesstimekeepinginavirtualmachine.
TheVMIpatchesdevelopedforkernel2.
6.
20andearlierincludechangestothetimekeepingsubsystemthatusetheVMIparavirtualtimerdeviceforticklesstimekeepingandstolen‐timeaccounting.
SomedistributionvendorshaveshippedkernelsthatincludethisversionofVMI,soifyourunoneofthesedistributionsinavirtualmachine,yougetthebenefitsofthesechanges.
Inparticular,bothUbuntu7.
04(2.
6.
20)andSUSELinuxEnterpriseServer10SP2(2.
6.
16)shipwithVMIenabledinthedefaultkernel.
TheVMIpatcheswereacceptedintothemainline32‐bitkernelinversion2.
6.
21.
However,theclocksourceabstractionwasalsoaddedtothekernelinthisversion,makingticklesstimekeepingavailableusingthegenerichigh‐leveltimekeepingcode.
Accordingly,thetimekeepingportionoftheVMIpatcheswasdroppedatthispoint,becauseitwasnolongerneeded.
(Stolentimeaccountingwasalsodropped,butitmightbeaddedbackinadifferentwayinthefuture.
)SolarisTimekeepinginSolaris10istickless.
Theoperatingsystemreadsahardwarecounter(bydefault,theTSC)toobtaintherawamountoftimesincethesystembooted.
ThewallclocktimeatbootisreadfromtheCMOStimeofdayclock.
Inaddition,whilerunning,SolarisperiodicallychecksitsestimateofwallclocktimeagainsttheCMOSTODclockandusesthisinformationtocorrectandrefineitsboot‐timemeasurementoftheTSC'srunningrate.
TheSolaristimercallbackservicealsodoesnotuseaperiodicinterrupt.
Instead,itmaintainsaone‐shotinterruptsettowakethesystemupintimeforthenextscheduledcallback.
Thisinterruptisrescheduledastimercallbacksfire,areadded,orareremoved.
ThesecharacteristicsofSolarisaresimilartoLinuxwiththeclocksourceandNO_HZfeaturesandareagoodmatchforrunninginavirtualmachine.
Solarisdoescurrentlyexhibittwominorproblemswhenrunninginavirtualmachine.
First,whenrunningonmultipleprocessors,SolarisattemptstoresynchronizetheTSCsatboottimebymeasuringhowfaroutofsynctheyare,computingaTSCoffsetforeachprocessor,andapplyingthatoffsetinsoftwareasacorrectiontoeachsubsequentreadingofthecorrespondingTSC.
Unfortunately,whileavirtualmachineisbooting,allvirtualprocessorsarenotnecessarilysimultaneouslyscheduledonphysicalprocessorswhenSolaristakesitsmeasurement,causingSolaristomeasuretheTSCsasbeingslightlyoutofsyncandcomputeanunwantedoffset.
ThissortofproblemisnotuniquetoSolaris,butotheroperatingsystemsthatattempttoresynchronizetheTSCsdosoinhardware,bywritingtotheTSCs,whichgivesthevirtualmachinetheopportunitytodetecttheproblemandapplyaheuristictocorrectit.
VMwareisworkingwithSuntodisableTSCresynchronizationwhenSolarisrunsinavirtualmachine.
AsecondsmallissueisthatSolariscanoccasionallyfindtheCMOSTODclocktobetoofarofffromthevalueitexpectstoseeandconcludethattheCMOSclockisnotworkingproperly.
ThisresultsinaharmlesswarningmessageprintedtotheSolarisconsolelog.
SynchronizingVirtualMachinesandHostswithRealTimeAsdiscussedin"InitializingandCorrectingWall‐ClockTime"onpage3,forlong‐termaccuracy,bothphysicalandvirtualmachinesgenerallyneedtorunsoftwarethatperiodicallyresynchronizesthewallclocktimemaintainedbytheoperatingsystemtoanexternalclock.
Therearetwomainoptionsavailableforguestoperatingsystemclocksynchronization:VMwareToolsperiodicclocksynchronizationorthenativesynchronizationsoftwarethatyouwouldusewiththeguestoperatingsystemifyouwererunningitdirectlyonphysicalhardware.
SomeexamplesofnativesynchronizationsoftwareareMicrosoftW32TimeforWindowsandNTPforLinux.
Eachoptionhassomeadvantagesanddisadvantages.
Wediscusseachbrieflyhereandthenadddetailsinsubsequentsections.
Copyright2008VMware,Inc.
Allrightsreserved.
15TimekeepinginVMwareVirtualMachinesVMwareToolsperiodicclocksynchronizationhastheadvantagethatitisawareofthevirtualmachine'sbuilt‐incatch‐upandinteractsproperlywithit.
Iftheguestoperatingsystemclockisbehindrealtimebymorethantheknownbacklogthatisintheprocessofbeingcaughtup,VMwareToolsresetstheclockandinformsthevirtualmachinetostopcatchingup,whichsetsthebacklogtozero.
AnadditionaladvantageofVMwareToolsclocksynchronizationisthatitdoesnotrequirenetworkingtobesetupintheguest.
However,atthiswriting,VMwareToolsclocksynchronizationhasaseriouslimitation:itcannotcorrecttheguestclockifitgetsaheadofrealtime(exceptinthecaseofNetWareguestoperatingsystems).
Thislimitationappliesonlytoperiodicclocksynchronization.
VMwareToolsdoesaone‐shotcorrectionofthevirtualmachineclockthatmaysetiteitherbackwardorforwardintwocases:whentheVMwareToolsdaemonstarts(normallywhiletheguestoperatingsystemisbooting),andwhenausertogglestheperiodicclocksynchronizationfeaturefromofftoon.
Nativesynchronizationsoftwarehastheadvantagethatitisgenerallypreparedtodealwiththevirtualmachineclockbeingeitheraheadoforbehindrealtime.
Ithasthedisadvantagethatitisnotawareofthevirtualmachine'sbuilt‐incatch‐upandthustypicallydoesnotsynchronizetimeaswellinavirtualmachineasitdoeswhenrundirectlyonphysicalhardware.
Onespecificproblemoccursifnativesynchronizationsoftwarehappenstosettheguestoperatingsystemclockforwardtothecorrecttimewhilethevirtualmachinehasaninterruptbacklogthatitisintheprocessofcatchingup.
Settingtheguestoperatingsystemclockaheadisapurelysoftwareeventthatthevirtualmachinecannotbeawareof,soitdoesnotknowthatitshouldstopthecatch‐upprocess.
Asaresult,theguestoperatingsystemclockcontinuestorunfastuntilcatch‐upiscomplete,anditendsupaheadofthecorrecttime.
Fortunately,sucheventsareinfrequent,andthenativesynchronizationsoftwaregenerallydetectsandcorrectstheerrorthenexttimeitruns.
Anotherspecificproblemisthatnativesynchronizationsoftwaremayemploycontrolalgorithmsthataretunedforthetypicalratevariationofphysicalhardwaretimerdevices.
Virtualtimerdeviceshaveamorewidelyvariablerate,whichcanmakeitdifficultforthesynchronizationsoftwaretolockontothepropercorrectionfactortomaketheguestoperatingsystemclockrunatpreciselytherateofrealtime.
Asaresult,theguestoperatingsystemclocktendstooscillatearoundthecorrecttimetosomedegree.
Thenativesoftwaremayevendeterminethatthetimerdeviceisbrokenandgiveuponcorrectingtheclock.
Despitethesepotentialproblems,however,testinghasshownthatNTPinparticularbehavesfairlywellinavirtualmachinewhenappropriatelyconfigured(see"UsingNTPinLinuxandOtherGuests"onpage17).
NTPispreparedforsomeofitsreadingstobeanomalousbecauseofnetworkdelays,schedulingdelaysonthelocalhost,andotherfactorsandiseffectiveatfilteringoutsuchreadings.
Generally,itisbesttouseonlyoneclocksynchronizationserviceatatimeinagivenvirtualmachinetoensurethatmultipleservicesdonotattempttomakeconflictingchangestotheclock.
Soifyouareusingnativesynchronizationsoftware,wesuggestturningVMwareToolsperiodicclocksynchronizationoff.
UsingVMwareToolsClockSynchronizationVMwareToolsincludesanoptionalclocksynchronizationfeaturethatcanchecktheguestoperatingsystemclockagainstthehostoperatingsystemclockatregularintervalsandcorrecttheguestoperatingsystemclock.

VMwareToolsperiodicclocksynchronizationworksinconcertwiththebuilt‐incatch‐upfeatureinVMwarevirtualmachinesandavoidsturningtheclockaheadtoofar.
VMwareToolsalsoperformsone‐timecorrectionsoftheguestoperatingsystemclockaftercertainevents,evenifperiodicsynchronizationisturnedoff.
EnablingPeriodicSynchronizationToenableVMwareToolsperiodicclocksynchronizationinaguest,firstinstallVMwareToolsintheguestoperatingsystem.
YoucanthenturnonperiodicsynchronizationfromthegraphicalVMwareToolscontrolpanelwithintheguestoperatingsystem.
Alternatively,youcansetthe.
vmxconfigurationfileoptiontools.
syncTime=truetoturnonperiodicsynchronization.
SynchronizationinaLinuxguestworksevenifyouarenotrunningtheVMwareToolboxapplication.
AllthatisnecessaryisthattheVMwareguestdprocessisrunningintheguestoperatingsystemandtools.
syncTimeissettoTRUE.
Copyright2008VMware,Inc.
Allrightsreserved.
16TimekeepinginVMwareVirtualMachinesBydefault,thedaemoncheckstheguestoperatingsystemclockonlyonceperminute.
Youcanspecifyadifferentperiodbysettingthe.
vmxconfigurationfileoptiontools.
syncTime.
periodtothedesiredtimeperiod(valuespecifiedinseconds).
Whenthedaemoncheckstheguestoperatingsystemclock,ifitismuchfartherbehindthehosttimethanthevirtualmachine'sbuilt‐incatch‐upmechanismexpectsittobe,thedaemonresetstheguestoperatingsystemclocktohosttimeandcancelsanypendingcatch‐up.
Asdiscussedabove,thisperiodiccheckcurrentlycannotcorrecttheguestoperatingsystemtimeifitisaheadofthehosttimeexceptinNetWareguestoperatingsystems.
DisablingAllSynchronizationItisnormalforaguestoperatingsystem'sclocktobebehindrealtimewheneverthevirtualmachineisstoppedforawhileandthencontinuesrunning—inparticular,afterasuspendandresume,snapshotandreverttosnapshot,diskshrink,orVMotionoperation.
Therefore,ifVMwareToolsisinstalledinaguestoperatingsystem,theVMwareToolsdaemoncorrectstheguestoperatingsystemclockaftertheseeventsoccur,evenifperiodictimesynchronizationisturnedoff.
Occasionally,youmayneedtotestaguestoperatingsystemwithitsclocksettosomevalueotherthanrealtime.
Someexamplesincludesettingavirtualmachine'sdateto1999toworkaroundY2Kproblemsinlegacysoftwareorsettingavirtualmachinetovarioustimestotestdateprintingroutines.
Youmaywanttohavethevirtualmachineshowthesametimewheneveritispoweredon,tospecifyaconstantoffsetfromrealtime,ortosynchronizeavirtualmachinewithaMicrosoftWindowsdomaincontrollerwhosetimeisoutofsyncwiththehostmachineonwhichthevirtualmachineisrunning.
VMwareToolscansynchronizeguestoperatingsystemsonlytotherealtimeasmaintainedbythehostoperatingsystem,soyouneedtodisableVMwareToolsclocksynchronizationcompletelyifyouwanttomaintainafictitioustimeinaguestoperatingsystem.
VMwareToolsautomaticallyupdatestheguestoperatingsystem'stimetomatchthehostoperatingsystem'stimeinafewothercasesinwhichtheguestcanbeexpectedtohavelostalargeamountoftime,evenifperiodicclocksynchronizationisturnedoff.
Tomaintainafictitioustime,youneedtosetthefollowingoptionstoFALSE.
tools.
syncTime=FALSEtime.
synchronize.
continue=FALSEtime.
synchronize.
restore=FALSEtime.
synchronize.
resume.
disk=FALSEtime.
synchronize.
shrink=FALSEtime.
synchronize.
tools.
startup=FALSEInformationonthesesettingsisalsoavailableinVMwareknowledgebasearticle1189(http://kb.
vmware.
com/kb/1189).
Table3showswhateachoptioncontrols.
NOTEInsomeproductversions,youmayhavetouse0inplaceofFALSE.
Table3.
TimeSynchronizationSettingsintheConfigurationFileOptionEffecttools.
syncTimeIfsettoTRUE,theclocksyncsperiodically.
time.
synchronize.
continueIfsettoTRUE,theclocksyncsaftertakingasnapshot.
time.
synchronize.
restoreIfsettoTRUE,theclocksyncsafterrevertingtoasnapshot.
time.
synchronize.
resume.
diskIfsettoTRUE,theclocksyncsafterresumingfromsuspendandaftermigratingtoanewhostusingtheVMwareVMotionfeature.
time.
synchronize.
shrinkIfsettoTRUE,theclocksyncsafterdefragmentingavirtualdisk.
time.
synchronize.
tools.
startupIfsettoTRUE,theclocksyncswhenthetoolsdaemonstartsup,normallywhiletheguestoperatingsystemisbooting.
Copyright2008VMware,Inc.
Allrightsreserved.
17TimekeepinginVMwareVirtualMachinesBecauseguestoperatingsystemsgenerallygettheirtimefromthevirtualCMOSTODclockwhentheyarepoweredon,youneedtosetthisdevicetoyourfictitioustimeifyouwantthetimetopersistacrossguestoperatingsystemrestarts.
Ifyouwanttostartaguestoperatingsystemwiththesametimeoneverystartup,usethertc.
startTimeoptiondescribedin"VirtualCMOSRTC"onpage7.
If,instead,youwanttheguestoperatingsystemtohaveaconstantoffsetfromrealtimeasmaintainedbythehost,youcanusethertc.
diffFromUTCoption,orsimplysettheCMOSTODclockfromthevirtualmachine'sBIOSsetupscreenorfromwithintheguestoperatingsystem.
InMicrosoftWindows,settingthesystemtimeautomaticallyupdatestheCMOSclock.
InLinux,youcanusethe/sbin/hwclockprogramtosettheCMOSclock.
Alternatively,becausemostLinuxdistributionsareconfiguredtocopythesystemtimeintotheCMOSclockduringsystemshutdown,youcansimplysetthesystemtimeandshutdowntheguestoperatingsystembeforerestartingitagain.
UsingMicrosoftW32TimeinWindowsGuestsTheWindowsTimeService(W32Time),presentinWindows2000andlater,implementsasimplevariantoftheNetworkTimeProtocol(NTP).
ThesubsetiscalledSNTP.
W32TimeallowsyoutosynchronizeaWindowsmachine'sclockinseveraldifferentways,eachprovidingadifferentlevelofaccuracy.
LiketheCMOS‐basedtimedaemon,W32Timeisnotawareofanyattemptsbyavirtualmachinetoprocesstimerinterruptbacklogsandcatchthevirtualmachine'sclockuptorealtime,socorrectionsbyW32Timecanoccasionallyovershootrealtime,especiallyinolderversions.
However,W32Timeshouldgenerallycorrectsucherrorsonitsnextresynchronization.
NewerversionsofW32Time(inWindowsServer2003andlater)seemtoincorporateafullNTP‐likealgorithmandperformbetter,comparablytotheNTPimplementationusedinLinux.
TurningonVMwareToolsperiodicclocksynchronizationdoesnotdisableW32Time.
AtthiswritingwedonotyethavespecificrecommendationsonhowtoconfigureW32Timeforbestperformanceinavirtualmachine.
AfewcustomershavearequirementtouseavirtualmachineasaW32Timeserver,toprovidetimetoothersystems,butdonotwantthevirtualmachinetobeaW32Timeclient.
Instead,thevirtualmachinegetsitstimeusingVMwareToolsorrunsusingafictitioustimeasdescribedabove.
W32Timedoeshavetheabilitytoruninaserver‐onlymode.
ForinstructionsonsettingupW32Timetoruninthismode,refertoMicrosoftdocumentationontheWindowsTimeService—specifically,theNoSyncregistryoption.
UsingNTPinLinuxandOtherGuestsTheNetworkTimeProtocolisusableinavirtualmachinewithproperconfigurationoftheNTPdaemon.
Thefollowingpointsareimportant:Donotconfigurethevirtualmachinetosynchronizetoitsown(virtual)hardwareclock,notevenasafallbackwithahighstratumnumber.
Somesamplentpd.
conffilescontainasectionspecifyingthelocalclockasapotentialtimeserver,oftenmarkedwiththecomment"undisciplinedlocalclock.
"Deleteanysuchserverspecificationfromyourntpd.
conffile.
Includetheoptiontinkerpanic0atthetopofyourntp.
conffile.
Bydefault,theNTPdaemonsometimespanicsandexitsiftheunderlyingclockappearstobebehavingerratically.
Thisoptioncausesthedaemontokeeprunninginsteadofpanicking.
FollowstandardbestpracticesforNTP:Chooseasetofserverstosynchronizetothathaveaccuratetimeandadequateredundancy.
Ifyouhavemanyvirtualorphysicalclientmachinestosynchronize,setupsomeinternalserversforthemtouse,sothatallyourclientsarenotdirectlyaccessinganexternallow‐stratumNTPserverandoverloadingitwithrequests.
Copyright2008VMware,Inc.
Allrightsreserved.
18TimekeepinginVMwareVirtualMachinesThefollowingsamplentp.
conffileissuitableifyouhavefewenoughclientsthatitmakessenseforthemtoaccessanexternalNTPserverdirectly.
Ifyouhavemanyclients,adaptthisfilebychangingtheservernamestoreferenceyourinternalNTPservers.
#ntpd.
conftinkerpanic0restrict127.
0.
0.
1restrictdefaultkodnomodifynotrapserver0.
vmware.
pool.
ntp.
orgserver1.
vmware.
pool.
ntp.
orgserver2.
vmware.
pool.
ntp.
orgserver3.
vmware.
pool.
ntp.
orgHereisasample/etc/ntp/step-tickerscorrespondingtothesamplentp.
conffileabove.
#step-tickers0.
vmware.
pool.
ntp.
org1.
vmware.
pool.
ntp.
orgMakesurethatntpdisconfiguredtostartatboottime.
Onsomedistributionsthiscanbeaccomplishedwiththecommandchkconfigntpdon,butconsultyourdistribution'sdocumentationfordetails.
Onmostdistributions,youcanstartntpdmanuallywiththecommand/etc/init.
d/ntpdstart.
HostClockSynchronizationIfyouareusingVMwareToolstosynchronizeyourguestoperatingsystemclocktothehostclock,orifyourguestoperatingsysteminitializesitstimefromthevirtualCMOSTODclock,itisimportantforyourhostclocktohaveaccuratetime.
Inaddition,ifyouareusingnativeclocksynchronizationsoftwareintheguestoperatingsystem,youmightchoosetousethehostasatimeserverforthevirtualmachine.
Witheithersuchsetup,yourhostreceivesthecorrecttimefromthenetwork,andyourvirtualmachinesreceivethecorrecttimefromthehostoperatingsystem.
IfyouareusingMicrosoftWindowsasthehostforaVMwarehostedproduct,theWindowsTimeService(W32Time)canbeagoodwaytosynchronizethehostclock.
TherearemanywaystoconfigureW32Time,someofwhichgivemoreprecisesynchronizationthanothers.
SeeMicrosoftsdocumentationfordetails.
InadditiontoW32Time,therearealsomanyotherthird‐partyclocksynchronizationprogramsavailableforWindows.
IfyouareusingLinuxasthehostforaVMwarehostedproduct,NTPisagoodwaytosynchronizethehostclock.
TheNTPdaemoniscalledntpdoncurrentdistributions,xntpdonolderones.
VMwareESXandESXialsoincludeanNTPdaemon.
YoucanenableandconfigureNTPfromtheVirtualInfrastructureClient.
ForolderversionsofESX,youcanconfigureNTPusingtheinstructionsinVMwareknowledgebasearticle1339(http://kb.
vmware.
com/kb/1339).
Atthiswriting(withthelatestreleaseofESXbeingversion3.
5),theESXNTPdaemonrunsintheserviceconsole.
Becausetheserviceconsoleispartiallyvirtualized,withtheVMkernelindirectcontrolofthehardware,NTPrunningontheserviceconsoleprovideslessprecisetimethaninconfigurationswhereitrunsdirectlyonahostoperatingsystem.
Therefore,ifyouareusingnativesynchronizationsoftwareinyourvirtualmachines,itissomewhatpreferabletosynchronizethemoverthenetworkfromanNTPserverthatisrunningdirectlyonitshostkernel,nottotheNTPserverintheserviceconsole.
InVMwareESXi,thereisnoserviceconsoleandtheNTPdaemonrunsdirectlyontheVMkernel.
TimeandPerformanceMeasurementsWithinaVirtualMachineCustomersoftenasktowhatextenttheycantrusttimingandperformancemeasurementstakenwithinavirtualmachineandhowtosupplementthesemeasurementstofullyunderstandtheperformanceofapplicationsrunninginavirtualmachine.
Acompletetreatmentofthistopicisbeyondthescopeofthispaper,butwecangivesomegeneralguidancehere.
NOTEAnytinkercommandsusedmustappearfirst.
Copyright2008VMware,Inc.
Allrightsreserved.
19TimekeepinginVMwareVirtualMachinesTimeMeasurementsTimemeasurementstakenwithinavirtualmachinecanbesomewhatinaccuratebecauseofthedifficultyofmakingtheguestoperatingsystemclockkeepexacttime,asdiscussedatlengthabove.
Thereareseveralstepsyoucantaketoreducethisproblem.
Wherepossible,chooseaguestoperatingsystemthathasgoodtimekeepingbehaviorwhenruninavirtualmachine,suchasonethatusesticklessorVMItimekeeping.
Configuretheguestoperatingsystemtoworkaroundanyknowntimekeepingissuesspecifictothatguestversion.
SeetheVMwareknowledgebasefordetails.
Useclocksynchronizationsoftwareintheguest.
Therearealsoanumberofwaystoavoidusingtheguestoperatingsystem'sclock,ifyouarewritingormodifyingsoftwarespecificallytomaketimingmeasurementsinavirtualmachine.
Usethefeaturedocumentedin"PseudoperformanceCounters"onpage9.
Formeasuringrelativelylongtimeintervals,getthestartingandendingtimedirectlyfromanetworktimeserver,usingaprogramlikentpdateorrdate.
ReadthevirtualCMOSTODclock,becauseitrunsinrealtime,notapparenttime.
However,thisclockispreciseonlytothenearestsecond.
PerformanceMeasurementsPerformancemeasurementsreportedbyaguestoperatingsystemrunninginsideavirtualmachinearemeaningfulandinsomecasesjustasaccurateaswhentheoperatingsystemisrunningonphysicalhardware.
However,youmustinterprettheseresultswithcare,becausetheguestoperatingsystemdoesnothavethefullpictureofwhatishappeningonthehost.
Togetacompletepicture,youalsoneedtolookatstatisticstakenbytheVMkernel.
WithESXandESXi,VMkernelperformancestatisticsareavailablefromVirtualCenter,fromtoolssuchasesxtop,andalsoinsidetheguestoperatingsystemusingtheVMwareGuestSDK(alsoknownasguestliborvmGuestLib).
TogetacompletepicturewhenusingVMwarehostedproducts,usethehostoperatingsystem'sstandardtoolssuchasWindowsperfmonandLinuxtop.
Considerationsforseveralspecifictypesofperformancemeasurementfollow.
EventCountsGenerally,eventcountstakenbytheguestoperatingsystemareaccurate,buttheycountonlyeventshappeninginthatspecificvirtualmachine.
OnESX,forexample,theguestoperatingsystem'scountofcontextswitchesisaperfectlyaccuratecountofswitchesbetweenguestoperatingsystemprocesses,butitdoesnotcounthowoftentheVMkerneldescheduledthisvirtualmachineandranadifferentone.
ThelatterstatisticisavailablefromtheVMkernel.
Similarly,onhostedproducts,theguestoperatingsystemdoesnotknowhowmanytimesthehostoperatingsystemdescheduledthevirtualmachineandranadifferenthostprocess.
Asanotherexample,theguestoperatingsystem'scountofinterruptsaccuratelyreflectshowmanyvirtualinterruptsitreceived,butitdoesnotcountphysicalinterruptsonthehostsystem.
MemoryUsageMemoryusagecountstakenbytheguestoperatingsystemaccuratelyreflecttheguestoperatingsystem'susageofitsvirtualizedphysicalmemory.
Theydonot,however,showhowmuchrealphysicalmemorythevirtualmachinehasbeenabletosaveusingVMwaretechniquessuchasballooning,pagesharing,andlazilyallocatingpagesthattheguestoperatingsystemhasnevertouched,nordotheyreflectanymemorythathasbeenswappedtodiskatthehostlevel.
Copyright2008VMware,Inc.
Allrightsreserved.
20TimekeepinginVMwareVirtualMachinesCPUUsageCPUusagemeasurementstakenbyaguestoperatingsystemgenerallygiveanapproximatelycorrectmeasureoftherelativeCPUusageofvariousprocessesrunningintheguestoperatingsystem,buttheydonotreflecthowloadedthehostsystemis,becausemostguestoperatingsystemsareunawarethattheyarerunninginavirtualmachineandarethustime‐sharingthephysicalhardwarewithothervirtualmachines.
Thefollowingparagraphsclarifythispoint.
OperatingsystemsuseoneoftwobasicmechanismstochargeandaccountforCPUutilization:statisticalsamplingorexactmeasurement.
Thestatisticalsamplingmethodismorecommon.
Withstatisticalsampling,wheneveratimerinterruptoccurs,theoperatingsystemcheckswhatprocesswasinterrupted(whichmayhavebeentheidleprocess)andchargesthatprocessforthefullamountoftimethathaspassedsincethelasttimerinterrupt.
Thischargingisoftenincorrect,becausethecurrentprocessmaynothavebeenrunningfortheamountoftimereported,butoverthelongrun,theerrorsaverageouttonearzeroandthemethodprovidesausefulresult.
Withtheexactmeasurementmethod,ontheotherhand,theoperatingsystemusesaperformancecounterprovidedbytheCPU(oftentheTSC)tomeasuretheexactnumberofcyclesthatitgivestoeachprocess.
Iftheoperatingsystemisusingthestatisticalsamplingmethodandthetimerdeviceinuseisrunninginapparenttime(themostcommoncase),theguestoperatingsystemchargesalltheapparenttimethatpassestooneoranotherofitsprocesses.
Thisoccursbecausethevirtualmachinedeliversallofthetimerinterruptsthattheguestoperatingsystemhasrequested,butitshiftsthemintimesothattheyalloccurwhilethevirtualmachineisrunning—nonewhileitisdescheduled.
Asaresult,timewhenthevirtualmachinehasinrealitybeendescheduledischargedbytheguesttoguestprocesses.
Thischargingoccursrandomly,approximatelyinproportiontotheactualCPUusageofeachprocess.
Thustheguestoperatingsystem'sCPUusagestatisticsaccuratelyreflecttheCPUconsumptionofguestprocessesrelativetooneanotherbutnottheabsolutefractionofahostCPUtheyconsume.
Forexample,ifthevirtualmachineisgettingabout50percentofaphysicalCPUandhastwoprocesseseachconsumingaboutanequalamountoftime,withnoidletime,theguestoperatingsystemreportseachprocessasconsuming50percentofavirtualCPU.
Fromalargerpointofview,however,eachprocessisactuallyconsumingonly25percentofaphysicalCPU.
TheresultsaresimilarwiththeexactmeasurementmethodusingtheTSC.
TimeduringtheperiodthevirtualmachinewasdescheduledisrapidlycaughtupthenexttimeitrunsagainbyrunningtheTSCfaster,sothenextguestoperatingsystemprocesstorunischargedforthistime.
Thisexcesschargingisrandomlydistributedamongtheguestoperatingsystemprocesses,roughlyinproportiontohowmuchtimetheprocessisreallyconsuming.
Sotheguestoperatingsystem's"exact"CPUusagemeasurementsareinflatedandthusnolongerexact,butonaverageoverthelongterm,guestoperatingsystemCPUusagestatisticsdoaccuratelyreflecttheCPUconsumptionofguestoperatingsystemprocessesrelativetooneanother.
Theresultscanbesomewhatdifferentiftheguestoperatingsystemisusingthestatisticalsamplingmethodbutwithatimerdevicethatisin"lazy"mode(see"VirtualLocalAPICTimer"onpage8).
Inthiscase,ifmultiplevirtualtimerinterruptsarescheduledtooccurwhiletheguestoperatingsystemisdescheduled,theyeffectivelymergeintoone.
Thus,mostdescheduledtimeisnotchargedtoanyguestoperatingsystemprocess.

Withsomeguestoperatingsystems,idletimemaybecomputedasapparenttimeminusthesumoftimeschargedtootherprocesses,inwhichcasedescheduledtimeiscountedasidletime.
Inothers,theidleprocessischargedusingstatisticalsamplingaswithotherprocesses,sothetotalchargedtimedoesnotaddupto100percentofrealtime.
Finally,someguestoperatingsystemscanexplicitlyaccountfordescheduledtime.
AnoperatingsystemrunninginavirtualmachineandusingtheVMIparavirtualizationinterfacecanobtainanddisplaytheamountofstolentime—thatis,theamountoftimewhenthekernelwantedtorunanonidleprocessbutwasdescheduled.
Alternatively,aWindowsorLinuxguestoperatingsystemthathastheVMwareVMdescheddriver(alsocalledthe"timersponge")installedshowsmostdescheduledtimeashavingbeenusedbythevmdeschedprocessinsteadofarealguestoperatingsystemprocess.
VMdeschediscompatibleonlywithoperatingsystemsthatusestatisticalsamplingforprocessaccountingandtickcountingfortimekeeping,andthecurrentimplementationworksonlyonuniprocessorguests.
VMdeschedworksbymanipulatingtheCopyright2008VMware,Inc.
Allrightsreserved.
21TimekeepinginVMwareVirtualMachinestimingofvirtualtimerinterruptssothatmostcatch‐upinterruptsoccurwhilethevmdeschedprocessisrunning.
(Infact,theprocessdoesnothingmorethanrunbrieflywhentherearecatch‐upinterruptstobedelivered.
)Becausevmdeschedaddsoverheadandisnotcompatiblewithsomeofthenewerkerneltechnologies,itislikelynottobefurtherdevelopedandmaybedroppedinthefuture.
Becauseoftheseissues,totalCPUload(or,conversely,totalidletime)measuredfromwithinavirtualmachineisnotaverymeaningfulnumber,eventhoughCPUusageofnonidleguestoperatingsystemprocessesrelativetooneanotherismeaningful.
Therefore,ifyouarerunningsoftwareinavirtualmachinethatmeasuresandadaptstototalsystemload,youshouldexperimenttofindouthowthesoftwarebehaves,andyoumayfindthatyouneedtomodifythesoftware'smeasurementandadaptationalgorithms.
ResourcePressureBecausetimekeepingrequiressomeCPUresourcesandrequiressomeactivitiestobeperformedinatimelyway—especiallywhentickcountingisbeingused—theguestoperatingsystemclockinavirtualmachinethatdoesnotgetenoughresourcescanfallbehindorotherwisemisbehave.
Thissectiondiscussessomepotentialproblems.
CPUPressureIftheguestoperatingsystemisusingtickcountinganditdoesnotgetenoughCPUtimetohandlethenumberoftimerinterruptspersecondthatithasrequested,itsclockfallsbehindrealtime.
YoucandealwithCPUpressureissuesineitheroftwoways.
Wherepossible,configuretheguestoperatingsystemtousealowertimerinterruptrate.
WithLinuxguestoperatingsystems,chooseaticklesskernelifpossible,orakernelthatusesarelativelylowbasetimerinterruptrate,orakernelthathasthedivider=optionthatletsyoulowertherate.
SeetheLinuxbestpracticesguideinVMwareknowledgebasearticle1006427(http://kb.
vmware.
com/kb/1006427).
Avoidconfiguringyourvirtualmachineswithmorevirtualprocessorsthanneeded.
WithWindowsguestoperatingsystems,trytoavoidrunningsoftwareintheguestoperatingsystemthatraisesthetimerinterruptrate.
See"MicrosoftWindows"onpage10.
GivemoreCPUtimetothevirtualmachine.
OnbothVMwareESXandVMwarehostedproducts,avoidovercommittingthehostCPUbyrunningsomanyvirtualmachinesthatsomeorallofthemcannotgetenoughtimetohandlealltheirtimerinterrupts.
Theexactnumberofvirtualmachinesyoucanrundependsonwhatapplicationsyouarerunninginthemandhowbusytheyare,sowecannotgiveaguidelineinthispaper.
OtherVMwarepublicationsareavailabletohelpwithcapacityplanning,andfeaturessuchasVMwareDistributedResourceScheduler(DRS)andVMwareDistributedPowerManagement(VMwareDPM)areusefulinoptimizingtheplacementofvirtualmachinesonphysicalhostsandhelpingyoukeeptherightnumberofphysicalhostspoweredontohandlethecurrentresourceusagebyvirtualmachines.
Inaddition,onVMwareESX,ifaspecificvirtualmachineisnotgettingenoughCPUtimetohandleallitstimerinterrupts,youcangiveitaCPUreservationtoensureitgetsenoughtime.
Thiscomesattheexpenseofothervirtualmachinesthatmightotherwisehavebeenallocatedthattime.
MemoryPressureMemorypressurecanindirectlycauseCPUpressure.
YoucanovercommitmemoryonanESXhost—thatis,configurethevirtualmachinesonthathostwithatotalofmorememorythanphysicallyexistsonthehost—andESXisstillabletorunallthevirtualmachinesatonce.
ESXusesseveraltechniquestoconserveandsharememorysothatvirtualmachinescancontinuetorunwithgoodperformanceinthepresenceofmemoryovercommitment,aslongastheovercommitmentfactorisnottoohigh.
Incertaincases,memoryovercommitmentthatistoohighornotconfiguredproperlycancausetimekeepingproblems.
ThedetailsofESXmemorymanagementarebeyondthescopeofthispaper,butthisfollowingpointsprovideaquickoverview:Copyright2008VMware,Inc.
Allrightsreserved.
22TimekeepinginVMwareVirtualMachinesWhentheESXVMkernelstartsavirtualmachine,itdoesnotimmediatelygivethevirtualmachineenoughrealphysicalmemorytobackallofthevirtualphysicalmemorythatitwasconfiguredtohave.
Instead,theVMkernelallocatesmemoryasneeded.
TheVMkernelusespagesharing.
Itcontinuallyscansphysicalmemorytofindcaseswheremultiplememorypageshavethesamecontentandcollapsesthemdownintoasingleshared,copy‐on‐writepage.
TheVMkernelusesballooning.
IfaguestoperatingsystemhastheVMwareToolsvmmemsched(or"balloon")driverinstalled,whenevertheVMkernelneedstotakephysicalmemoryawayfromthevirtualmachine,itdoessobyaskingtheguestoperatingsystemtouseitsowninternalpagingmechanismstoswapoutmemory.
Asalastresort,whennoothermechanismshavereclaimedenoughmemory,theVMkernelchoosesvirtualmachinepagesandforciblyreclaimsthembycopyingthemtoaswapfileattheVMkernellevel.
AtimekeepingissuecanarisewhenpageshavebeenswappedoutbytheVMkernel.
BecausetheVMkernelhasnoinsightintowhattheguestoperatingsystemisdoingwithitspages,itcansometimesswapoutapagethattheguestoperatingsystemwillsoonneed.
Thenexttimetheguestoperatingsystemreferencesthatpage,theVMkernelhastoswapthepagebackintoRAMfromdisk.
Duringtheswap‐in,theguestoperatingsystemstopscompletely.
Itcannotruneventohandlevirtualinterrupts,suchastimerinterrupts.
(Incontrast,whenaguestoperatingsystemswapsoutitsownmemoryusingballooning,itusuallycanavoidswappingoutpagesthatwillbeneededagainsoon,andtheguestoperatingsystemcontinuestorunandprocessvirtualtimerinterruptswhileitisswappingmemorybackin.
)Asaresult,theguestoperatingsystemclockcanfallbehindwhilepagesarebeingreadinfromtheVMkernelswapfile.
Swap‐insfromdisktypicallytakearound10msperpage,andsometimesmanypagesmustbeswappedinbacktoback,sotheguestoperatingsystemclockcanfallbehindbymanysecondsduringaburstofswapping.
Theclockdoestypicallycatchupfullyoncetheguestoperatingsystem'sworkingsethasbeenswappedin,however.
Youcanavoidthismemorypressureissueineitheroftwoways:InstallVMwareToolsinyourvirtualmachinesandensurethatmemoryballooningisenabled.
Inmostcases,ballooningisabletoreclaimenoughmemorythatswappingattheVMkernellevelisnotrequired.
Thisisthepreferredapproach.
Ensurethatyourvirtualmachinesarebackedbyenoughphysicalmemorytoavoidswapping.
YoucanavoidtheneedforswappingbykeepingthememoryovercommitfactoronyourESXhostsloworavoidingovercommitentirely.
Oryoucanpreventswappingforaspecificvirtualmachinebysettingitsmemoryreservationto100percentofitsconfiguredmemorysize.
TroubleshootingThissectiondiscussessometroubleshootingtechniques.
Foradditionalinformation,searchfor"time"or"clock"intheVMwareknowledgebase(http://kb.
vmware.
com).
BestPracticesThefirststepindealingwithtimekeepingissuesispreventive:checkthatyourhostandvirtualmachineareconfiguredproperly.
Tosummarizethemainpoints:Ifpossible,usethemostrecentreleaseofyourVMwareproduct,oratleastthemostrecentminorreleaseofthemajorversionyouareusing.
Wearealwaysworkingonimprovingtimekeepingperformanceandfixingproblems.
Ifpossible,usethemostrecentsupportedminorversionoftheguestoperatingsystemineachofyourvirtualmachines.
Updatesandvendorpatchessometimesfixtimekeepingissues,especiallyinthecaseofLinuxguestoperatingsystems,inwhichthetimekeepingsystemhasbeenundergoingrapidevolution.
ChecktheVMwareknowledgebaseforarticlesaboutspecificconfigurationoptionsorworkaroundsforguestoperatingsystembugsthatmaybeneededfortheoperatingsystemversioneachofyourvirtualmachinesisrunning.
Inparticular,forLinuxguests,seethebestpracticesguideinknowledgebasearticle1006427(http://kb.
vmware.
com/kb/1006427).
Copyright2008VMware,Inc.
Allrightsreserved.
23TimekeepinginVMwareVirtualMachinesCheckthatyourhostsystemisconfiguredforthecorrecttimeandtimezone.
Checkthatitisrunningsuitableclocksynchronizationsoftware,asdescribedin"HostClockSynchronization"onpage18.
Checkthatyourvirtualmachinesaresettothecorrecttimezone.
Also,forLinuxguestoperatingsystems,itisbesttosettheoptioninyourLinuxdistributiontokeeptheso‐called"hardware"clock(thatis,thevirtualCMOSTODclock)inUTC,notlocaltime.
Thisavoidsanyconfusionwhenyourlocaltimechangesbetweenstandardanddaylightsavingtime(inEngland,"summertime").
Checkthatyouhaveappropriateclocksynchronizationsoftwareinstalledandconfiguredinyourvirtualmachines,asdescribedin"SynchronizingVirtualMachinesandHostswithRealTime"onpage14.
CheckthatVMwareToolsisinstalledinyourvirtualmachines.
EvenifyouarenotusingVMwareToolsperiodicclocksynchronization,theone‐timeclockcorrectionsdiscussedin"UsingVMwareToolsClockSynchronization"onpage15areimportant.
Inaddition,theVMwareToolspackageincludesspecializeddevicedriversthatimproveoverallperformanceofvirtualmachines,reducingCPUloadandthusindirectlyhelpingtimekeepingperformanceaswell.
GatheringInformationIfyoucontinuetohaveatimekeepingproblemaftercheckingtheabovebestpractices,thenextstepistoobserveyoursystembehaviorcarefullyandgatherdetailedinformationabouttheproblem.
Manydifferentproblemscanhavesimilarsymptomsorcanappearsimilarifnotobservedordescribedclearly.
Somespecificthingsyoucandoarecoveredinthefollowingsections.
ObserveSymptomsCarefullyNoteexactlyhowthevirtualmachinetimediffersfromrealtime,underwhatcircumstancesthetimekeepingproblemappears,andhowseveretheproblemis.
DoesthevirtualmachineshowtimethatisaheadofrealtimeorbehindDoestheerrorremainconstant,ordoesitincreaseorotherwisevaryovertimeHowfastdoestheerrorchangeCanyoucorrelateoccurrencesoftheproblemwithotheractivitiesthatcreateloadinthevirtualmachineoronthehostTestOperatingSystemClockagainstCMOSTODClockIfyouarerunningaLinuxguestoperatingsystem,runthefollowingscriptintheguest.
cat/etc/issueuname-adate/sbin/hwclockdatecat/proc/interruptssleep10cat/proc/interruptsdate/sbin/hwclockdateUsingtheoutputfromthescript,youcanseewhichtimerinterruptsareinuseandthefrequencywithwhichinterruptsaregenerated.
Checkhowmuchthevaluesshownin/proc/interruptschangeduringthe10secondsleepmeasuredbytheguest.
ThetimerinterruptsmostcommonlyusedbyLinuxare0or"timer"(thePIT)andLOC(thelocalAPICtimer).
Thisscriptalsoprovidesaroughwaytoobserveanylargedifferenceinrunningratebetweenthevirtualmachineandhostclocks.
Thedatecommandreturnstheguestoperatingsystemclocktime.
The/sbin/hwclockcommandreturnstheCMOSTODclocktime,whichVMwarevirtualizesatafixedoffsetfromthehost'sclock.
NOTEYoumayhavetorunthescriptasroot,because/sbin/hwclockrequiresrootprivilegeinsomeLinuxdistributions.
Whenyourunthescript,capturetheoutputtoafileandincludetheoutputifyoufileasupportrequestwithVMware.
Copyright2008VMware,Inc.
Allrightsreserved.
24TimekeepinginVMwareVirtualMachinesTurnonAdditionalLoggingYoucanturnonadditionalloggingoftimekeepingstatisticsinavirtualmachinebyaddingthefollowinglinestoits.
vmxconfigurationfileandrestartingthevirtualmachine:timeTracker.
periodicStats=TRUEtimeTracker.
statInterval=5Thesecondlinespecifiesthesamplinginterval(inseconds).
Thedefaultintervalis60seconds.
IfyouareplanningtofileasupportrequestwithVMware,pleaseenablethesesettings,dowhateverisnecessarytoreproducetheproblem,andruntheaffectedvirtualmachineinitsproblematicstateforabout30minutes.
Includetheresultingvmware.
logfilewithyourreport.
Thefollowinglistingshowsthetimetrackerstatisticsoutputfromatypicalvmware.
logfilefromarecentVMwareproduct.
Theformatofthisoutputissubjecttochange.
Jul3013:16:11.
044:vmx|TimeTrackerStatsbehindby2246us;runningat101%;mode0;catchuplimited4855us;0stops,0giveups,0numLargeBumps,maxLargeBump:0cyclesJul3013:16:11.
045:vmx|TimeTrackerStatsCMOS-P320ints,64.
00/sec,64.
01avg,64.
00req;5462tot,5461req;1808loprg,70756rtry;behind-15606usJul3013:16:11.
045:vmx|TimeTrackerStatstimer091ints,18.
20/sec,18.
21avg,18.
21req;1645tot,1644req;278loprg,14994rtry;behind-36140usJul3013:16:11.
045:vmx|TimeTrackerStatsPIIX4PMTT2ints,0.
40/sec,0.
42avg,0.
43req;34tot,34req;0loprg,0rtry;behind-1026844usThefollowingpointsdescribekeyphrasesinthisreport:behindby2246us—thevirtualmachine'sbuilt‐intimetrackerknowsthattheguestoperatingsystem'sclockisslightlybehindrealtime,by2246μs.
runningat101%—thevirtualmachinerantheguestoperatingsystem'sclockatanaverageof101percentofnormalspeedsincethelasttimestatisticswereprinted(roughlytimeTracker.
statInterval).
mode0—themodeinwhichthetimetrackerisoperating.
Currentlydefinedmodesincludethefollowing:0—aggressiveinterruptdelivery.
Thisisthenormalmode.
1—smoothinterruptdelivery.
Thisspecialmodespacesinterruptsoutmore.
Itisusedforcertainolderguestoperatingsystemsthatmayhavefragileinterrupthandlingcode.
2—smoothinterruptdelivery,withcatch‐upcurrentlyinprogress.
3—lazyinterruptdelivery.
Thisisusefulforticklessguestoperatingsystems.
4—timercalibrationmode.
ThisisusedbrieflywhileaLinuxguestoperatingsystemiscalibratingtimersduringboot.
catchuplimited4485us—duringthelaststatisticsinterval,thetimetrackerwassometimesnotabletoimmediatelycatchuptorealtimebecauseofaheuristicthatlimitstherateofcatch‐uptoavoidcausingproblemsintheguestoperatingsystem.
Atotalof4855μsofpotentialcatch‐upwasdelayedbythisheuristic.
Thisincludescatch‐upthatwasslightlydelayedbutdidoccurlaterintheinterval,notjustcatch‐upthatwasdelayedbeyondtheendoftheinterval.
Thus,itcanbemorethantheamountthetimetrackeriscurrentlybehind,asinthisexample.
0stops—VMwareToolshasnotaskedthetimetrackertostopcatch‐upsincethevirtualmachinewaspoweredon.
VMwareToolsstopscatch‐upwheneveritdetectsthattheguestoperatingsystem'sclockissignificantlybehindrealtimeandturnstheclockahead.
0giveups—thetimetrackeritselfhasnotdetectedthattheguestoperatingsystemclockistoofarbehindtocatchup.
0numLargeBumps,maxLargeBump:0cycles—therehavebeennolargeskewsbetweenapparenttimeondifferentvirtualCPUs.
Copyright2008VMware,Inc.
Allrightsreserved.
25TimekeepinginVMwareVirtualMachinesTheremaininglinesgivedetailsforspecifictimerdevices.
TheCMOS-PlinereferstotheCMOStimer'speriodicinterrupt,thetimer0linereferstoPITtimer0,andthePIIX4PMTTlinereferstotheACPIPMtimer.
OthernamesthatcanappearontheselinesincludeCMOS-U(theCMOStimerupdateinterrupt)andAPICn(thelocalAPICtimeronvirtualCPUn,wherenrangesfrom0upward).
TakingtheCMOS-Plineasanexample,thefollowingpointsexplainkeyphrases:320ints,64.
00/sec—therewere320virtualCMOSperiodictimerinterruptsdeliveredinthelaststatisticsinterval(fivesecondsinthisexample),orexactly64persecond.
64.
01avg,64.
00req—therewasanaverageof64.
01interruptspersecondovertheintervalsincetheguestoperatingsystemlastreprogrammedthistimertoadifferentrate.
Theguestoperatingsystemaskedfor64.
00interruptspersecond.
Ingeneral,theaveragecanbehigherthantherequestedratebecauseofcatch‐uporlowerbecausethetimetrackeriscurrentlybehind.
Inthisexample,thedifferenceisinsignificant.
5462tot,5461req—therehasbeenatotalof5462interruptssincetheguestoperatingsystemlastreprogrammedthetimer,whereasthereshouldhavebeen5461atthenominal,requestedrate.
1808loprg,70756rtry—on1808occasions,whenthevirtualmachinewantedtodeliveravirtualinterrupttotheguestoperatingsystem,itwasnotsafetodosobecausetheguestoperatingsystemhadmadetoolittleprogressrunningcodesincethelastvirtualinterruptofthistype.
Thetimetrackerdidatotalof70756retries,implyingthatitoftenrequiredmultipleretriestodeliverasingleinterrupt.
behind-15606us—howfarinthepastthenextinterruptfromthisdeviceisprogrammedtooccur,relativetoapparenttime.
Thisisnormallyanegativevalue,whichmeansthatthenextinterruptissettooccurinthefuture.
Itcansometimesbepositivebecauseofvariousunusualconditionsthatforceapparenttimetomoveaheadeventhoughatimerdevicehasnotyetdeliveredallofitsinterrupts.
Ifoneofthetimerdevicesintheguestoperatingsystemisnotcurrentlyprogrammedinaperiodicmodebuthasproducedinterruptsinthelastinterval,adifferentstyleofstatisticslineisloggedforit,withasubsetofthefieldsshownintheexampleabove.
Thelineusesthewordaperiodicbecauseitmostcommonlyoccurswhenthedeviceisbeingusedinone‐shotmode.
Anaperiodicstatisticslinelookslikethis:Aug1710:58:21.
264:vmx|TimeTrackerStatsAPIC0aperiodic12153ints,202.
54/sec;1092447tot;322loprg,326rtryThefollowinglistingshowstimetrackerstatisticsoutputfromanolderproductversionrunningadifferentguestoperatingsystem.
Manyfieldsaresimilar.
Thedifferencesaredescribedbelow.
Mar2117:17:36:vmx|TimeTrackerStatsbehindby104218351cycles(43668us);runningat100%;0stops,0giveupsMar2117:17:36:vmx|TimeTrackerStatsAPIC09972ints,997.
40/sec,1023.
94avg,1000.
49req;51188tot,50015req;59loprg,60rtryMar2117:17:36:vmx|TimeTrackerStatstimer09970ints,997.
20/sec,1023.
62avg,1000.
15req;51172tot,49998req;1395loprg,1400rtryThebehindbystatisticisgivenincycles(ofthevirtualTSC)aswellasmicroseconds.
Thereisnomodestatistic.
Instead,therunningatstatisticgivestherateatwhichthetimetrackeriscurrentlyattemptingtocatchuptheguestclock.
100%meansthatthereisnocatch‐upinprogress.
Atypicalvaluewhencatch‐upisinprogressis300%,buttheeffectivecatch‐uprateisgenerallymuchlower.