assist迈拓kvm

ExtendingKVMModelsTowardHigh-PerformanceNFVJunNakajima,JamesTsai,MesutErgin,YangZhang,andWeiWang14October2014LegalDisclaimerINFORMATIONINTHISDOCUMENTISPROVIDEDINCONNECTIONWITHINTELPRODUCTS.
NOLICENSE,EXPRESSORIMPLIED,BYESTOPPELOROTHERWISE,TOANYINTELLECTUALPROPERTYRIGHTSISGRANTEDBYTHISDOCUMENT.
EXCEPTASPROVIDEDININTEL'STERMSANDCONDITIONSOFSALEFORSUCHPRODUCTS,INTELASSUMESNOLIABILITYWHATSOEVER,ANDINTELDISCLAIMSANYEXPRESSORIMPLIEDWARRANTY,RELATINGTOSALEAND/ORUSEOFINTELPRODUCTSINCLUDINGLIABILITYORWARRANTIESRELATINGTOFITNESSFORAPARTICULARPURPOSE,MERCHANTABILITY,ORINFRINGEMENTOFANYPATENT,COPYRIGHTOROTHERINTELLECTUALPROPERTYRIGHT.
INTELPRODUCTSARENOTINTENDEDFORUSEINMEDICAL,LIFESAVING,ORLIFESUSTAININGAPPLICATIONS.
Intelmaymakechangestospecificationsandproductdescriptionsatanytime,withoutnotice.
Allproducts,dates,andfiguresspecifiedarepreliminarybasedoncurrentexpectations,andaresubjecttochangewithoutnotice.
Intel,processors,chipsets,anddesktopboardsmaycontaindesigndefectsorerrorsknownaserrata,whichmaycausetheproducttodeviatefrompublishedspecifications.
Currentcharacterizederrataareavailableonrequest.
IntelandtheIntellogoaretrademarksorregisteredtrademarksofIntelCorporationoritssubsidiariesintheUnitedStatesandothercountries.
*Othernamesandbrandsmaybeclaimedasthepropertyofothers.
Copyright2014IntelCorporation.
TheChallengeArchitectureProposalsforNFVforKVMCurrentStatusandSummary3Agenda4NFVVisionfromETSISource:http://portal.
etsi.
org/nfv/nfv_white_paper2.
pdfHighperformanceacrossallpacketsizes,includingsmallpackets(e.
g.
64B)Real-timeprocessing,includinglowlatencyandjitterRASSecurity.
.
.
5New/DifferentRequirementsforNFVComparedwithConventionalVirtualizationFocusonPerformanceTopicsTodayTheChallenge020,000,00040,000,00060,000,00080,000,000100,000,000120,000,000140,000,000160,000,000641281922563203844485125766407047688328969601024108811521216128013441408147210GbEPacketsPerSecond40GbEPacketsPerSecond100GbEPacketsPerSecond6.
72ns16.
8ns67.
2nsSaturationLineRate(MPPS)Source:DPDKSummit,VenkyVenkatesan,"ApplicationPerformanceTuningandFutureOptimizationsinDPDK",September8,2014https://www.
youtube.
com/watchv=qpfwDySweUADisclaimer:SoftwareandworkloadsusedinperformancetestsmayhavebeenoptimizedforperformanceonlyonIntelmicroprocessors.
Performancetests,suchasSYSmarkandMobileMark,aremeasuredusingspecificcomputersystems,components,software,operationsandfunctions.
Anychangetoanyofthosefactorsmaycausetheresultstovary.
Youshouldconsultotherinformationandperformanceteststoassistyouinfullyevaluatingyourcontemplatedpurchases,includingtheperformanceofthatproductwhencombinedwithotherproducts.
425593164.
925505010015020025030020092010201120122013SystemLevelL3Performance(MPPS)*Othernamesandbrandsmaybeclaimedasthepropertyofothers.
Disclaimer:SoftwareandworkloadsusedinperformancetestsmayhavebeenoptimizedforperformanceonlyonIntelmicroprocessors.
Performancetests,suchasSYSmarkandMobileMark,aremeasuredusingspecificcomputersystems,components,software,operationsandfunctions.
Anychangetoanyofthosefactorsmaycausetheresultstovary.
Youshouldconsultotherinformationandperformanceteststoassistyouinfullyevaluatingyourcontemplatedpurchases,includingtheperformanceofthatproductwhencombinedwithotherproducts.
IntelDPDKPerformanceAsnapshotofondifferentarchitecturesIntegratedMemoryControllerPCI-EGen2DataDirectI/OIntegratedPCI-EGen3AVX(integer,128-bit)4x10GbENICsPlatformFeaturesSource:DPDKSummit,VenkyVenkatesan,"ApplicationPerformanceTuningandFutureOptimizationsinDPDK",September8,2014https://www.
youtube.
com/watchv=qpfwDySweUALinuxKernelVMorUserProcessKernel(virt.
I/O)VM1VT-d,SR-IOVMiddleBox(e.
g.
virtualswitch)KVMKernel(virt.
I/O)VM2.
.
.
8FocusAreasforNFVPerformanceonKVMRecall67.
2ns,16.
8ns,…FastandEfficientInter-VMCommunicationGeneric:NetworkI/O,NUMA,NUMA-I/O,Caching,Affinity,…MorecoresMoremiddleboxespersocket,perserverServicechainingonserverLowerlatencyInter-VM(i.
e.
intra-node)vs.
Inter-nodeHigherBandwidthMemory(orcache)vs.
PCIebus9WhyInter-VMCommunicationFigure1.
TheIntelXeonprocessorE5-2600V2productfamilyMicroarchitectureSource(Figure1.
):https://software.
intel.
com/en-us/articles/intel-xeon-processor-e5-2600-v2-product-family-technical-overviewNotificationsforqueuecontrolKick,DoorBellVirtualSwitchPacketTransmissionCopy,etc.
TransitionsUser-KernelGuest-Host10Inter-VMCommunicationonKVMHostOSRxVMtapTxVMtapswitchappapphypervisorStackdrvdevdevStackdrvSwitchingpathcanbeabigperformancebottleneckXYX0.
712Mpps*Y0.
717Mpps**Intelinternalmeasurements64Bpackets,virtio-net+vhost-netTSCCycles(Haswell3.
2GHz),RoundTrip*:UserKernel(SystemCall)inVM(onKVM)E.
g.
getppid():1300(≈400ns)GuestHost(HyperCall)E.
g.
NullHypercall:1500-1600(≈500ns)ToreachSaturationLineRate(10GbE):Ifsystemcall/Hypercallisusedforeach64Bpackettransmission,wewouldneed:>6-7Cores**40GbE:>24-28Cores11CostofTransitions/IsolationPerspectiveofCPUCycles**:400/67.
2=5.
9,500/67.
2=7.
4*IntelinternalmeasurementsDisclaimer:SoftwareandworkloadsusedinperformancetestsmayhavebeenoptimizedforperformanceonlyonIntelmicroprocessors.
Performancetests,suchasSYSmarkandMobileMark,aremeasuredusingspecificcomputersystems,components,software,operationsandfunctions.
Anychangetoanyofthosefactorsmaycausetheresultstovary.
Youshouldconsultotherinformationandperformanceteststoassistyouinfullyevaluatingyourcontemplatedpurchases,includingtheperformanceofthatproductwhencombinedwithotherproducts.
Practically,thoseareratherlowerboundsbecausebatchingislimitedandactualpacketprocessinginhypercallsoverturnsgainofbatching.
TheChallengeArchitectureProposalsforNFVforKVMCurrentStatusandSummary12Agenda1.
Moveknowledgeandcontrolforinter-VMcommunicationtoVMs2.
AllowVMstoaccessotherVMstoshareoraccessmemoryinasafewayProvideVMswith"ProtectedMemoryView"Mappingitselfisprovidedbythehypervisor3.
AllowVMstouselow-latencynotificationmechanismsw/oVMexitsorinterruptsE.
g.
MONITOR/MWAIT,PostedInterrupt13Solutions:EmpowerGuestsinaSafeWayAvoidhypervisorinterventionsMotivation:WhydoesakernelmoduleneedtoknowaboutdatastructuresforPVdriversinguestsBecausewetrustkernelorkernelmodulesonly.
Whatifwetrustspecific(partof)guests…Vhost-netinguestcanavoidhypercallsifitcandirectlyaccessdestinationguests(virtqueue,etc.
)14Example:vhost-netFunctionalityinGuestsvhost-userisalreadythere15High-LevelArchitectureforFastInter-VMCommunication(w/oVT-d,SR-IOV)LinuxKernelKVMVM1Kernelvirto-netVM2Kernelvirto-netSharedmemoryforsynchronizationFastPathFastPathProtectedMemoryViewLow-LatencyNotification1.
DataTransmission2.
NotificationDirectAccesstoGuestsInProtectedMemoryViewVhost-netAPIFastPathcanworkwithvirtio-netorindependentlyvirtio-netvirtio-netLinuxKernelKVMVM1Kernelvirtio-netVM2Kernelvirtio-netSharedmemoryforsynchronizationFastPathFastPathMiddleBox(e.
g.
virtualswitch)VM0VT-d,SR-IOVFastPacketTransmissionSharedmemoryforsynchronization16High-LevelArchitectureforFastInter-VMCommunication(withVT-d,SR-IOV)FastPacketTransmissioncanbeinuser-levelVMFUNCinstructionwithEAX=0ValueinECXselectsanentryfromtheEPTP(Extended-Page-TablePointer)listAvailableinRing0-3,executedinguestNoVMexitCanbevirtualizedifnotavailable17IntroducingVMFunction0:EPTP*Switching…EPTP…ECX(index)EPTPlist(4KB)VMCS(perVCPU)*:Extended-Page-TablePointerEPTPSwitchingandTrampolineCodeGuestPhysicalPagesProtectedView(code,data)---DefaultEPTEPT:HostPhysicalPagesXWR18NoAccessTrampolinecodeforVMFUNCXWR-WRX-RX-RVMFUNCexecutedoutsideTrampolineCodewillcauseEPTviolationatnextinstructionHypervisorneedstorestoreDefaultEPTtodelivervirtualinterruptsEPTPswitchingDefaultViewAlternateViewstart_xmit(*skb,*dev){.
.
.
send(packets);}send(*packet){.
.
.
VMFUNC#0,EPTP;Tx(packets);VMFUNC#0,0}PageBoundaryTx(*packet){move_data();notify();}XWRstart_xmit(*skb,*dev){.
.
.
send(packets);}send(*packet){.
.
.
VMFUNC#0,EPTP;Tx(packets);VMFUNC#0,0}Tx(*packet){move_data();notify();}----WR-WRXWRXWRX-RX-REPTPerm.
ModifyqueuedescriptorsModifyqueuedescriptorsMoveDatabyTx()19MoreDetails:TransmittingPacketsDestinationVMSourceVMTrampolineCodeProtectedView24315DefaultViewAlternateViewPostedInterruptDelivervirtualinterruptsondestinationguestsw/oVMexits.
AlreadysupportedbyKVMStillrequiresVMexitonsourceguestMONITOR/MWAIT(Energy-EfficientPolling)betweenguestsThefeatureisnotadvertisedonKVMtodayUsevariablesonsharedmemorybetweensourceanddestinationPAUSELoop(Polling)betweenguestsLowestlatency,butnotenergyefficientInpractice,combineInterruptandPolling(likeNAPI)20Low-LatencyNotificationKnownmethodsMinimizeimpactofTLBmisses,cachemisses:Largepages(bothguest,EPT,VT-d),NUMA,IO-NUMA,DataDirectI/OE.
g.
LIFOmemorypoolZero-copyE.
g.
AddsourcebuffersmappingtoEPTofdestinationIfEPTPTEswerenotvalid,noINVEPTisrequired21PracticesforPerformanceGeneralDisclaimer:SoftwareandworkloadsusedinperformancetestsmayhavebeenoptimizedforperformanceonlyonIntelmicroprocessors.
Performancetests,suchasSYSmarkandMobileMark,aremeasuredusingspecificcomputersystems,components,software,operationsandfunctions.
Anychangetoanyofthosefactorsmaycausetheresultstovary.
Youshouldconsultotherinformationandperformanceteststoassistyouinfullyevaluatingyourcontemplatedpurchases,includingtheperformanceofthatproductwhencombinedwithotherproducts.
FrequencyofVMFUNCoperation:CostofVMFUNCisabout150TSCcycles(Haswell,3.
2GHz)*Around50ns,andsensitivetoTLB,cachesRecall67.
2ns,16.
8ns,…ToreachSaturationLineRate(10GbE):IfVMFUNCiscalledforeach64Bpackettransmission,we>1-2Cores(100nsforround-trip)40GbE:>4-8CoresThecostofVMFUNCwouldberelativelysmall,anditwouldprovidescalableperformance22PracticeforPerformanceEPTPSwitching*IntelinternalmeasurementsDisclaimer:SoftwareandworkloadsusedinperformancetestsmayhavebeenoptimizedforperformanceonlyonIntelmicroprocessors.
Performancetests,suchasSYSmarkandMobileMark,aremeasuredusingspecificcomputersystems,components,software,operationsandfunctions.
Anychangetoanyofthosefactorsmaycausetheresultstovary.
Youshouldconsultotherinformationandperformanceteststoassistyouinfullyevaluatingyourcontemplatedpurchases,includingtheperformanceofthatproductwhencombinedwithotherproducts.
getppid()inVM:1300(≈400ns)NullHypercall:1500-1600(≈500ns)Practically,thoseareratherlowerboundsbecausebatchingislimitedandactualpacketprocessingoverturnsgainofbatching.
TrampolineCodeisloadedbytheguest,buttheEPTpermission(X-R)issetbyKVMShouldbesignedtogetherwiththecodeintheProtectedViewinadvanceThesetofpages(inDestinationVM)accessedbycodeinProtectedViewneedtobecheckedandaddedbyKVMInaway,codeinProtectedViewisanextensionoftheKVM/hypervisorrunningincontrolledenvironment(stillinVXMnon-rootmode)23SecurityConsiderationTheChallengeArchitectureProposalsforNFVforKVMCurrentStatusandSummary24AgendaPoCinprogress:MeasuredcostofVMFUNC,memorybandwidthEnabledandmeasuredlatencyofMONITOR/MWAITinguestsMeasuringpathAWorkingonpathB25CurrentStatusPoCLinuxKernelKVMVM1Kernelvirto-netVM2Kernelvirto-netSharedmemoryforsynchronizationFastPathFastPathABvirtio-netvirtio-netBenefitsoftheArchitecture:ContainknowledgeandcontrolforInter-VMcommunicationinguestsAllowKVMtoenablemoreoptimizationandcustomizationforgueststohandlehighnetworkloadsefficientlyMoreefficientandscalablethanexistingonesWorkwithdirectI/OassignmentaswellNextStep:CompletePoCandgetmoredata26Summary27BackupCanoccuronlyinguest(vector20)SomeEPTviolationscangenerate#VEinsteadofVMexits(controlledbyhypervisor)Canvirtualizedifnotavailable#VE:VirtualizationException