sequentiallyyc8
WWW YC8 COM 时间:2021-03-02 阅读:()
IntegrationandAdvancedUsageBitfusionGuideWHITEPAPER–OCTOBER2019WHITEPAPER|2Bitfusion:IntegrationandAdvancedUsageTableofcontentsIntroduction3StartingFlexDirectDaemonsasServersviaCLI3RunClientApplicationswithFlexDirectviaflexdirectclient4ConfiguringIPAddressesasPartofClientConfiguration.
5Advanced:ClusterCommunications7Advanced:FlexibleDynamicGPUConfigurations7ExposingOneGPUoutoftheFourGPUsforApps8ExposingTwoGPUsoutoftheFourGPUsforApps8PartialGPUConfigurations9WHITEPAPER|3IntroductionForengineersintegratingBitfusiontechnologyintotheirownresourceschedulerorperhapsforadvancedusersneedingmorecontroloverGPUsareresourcing,thisguideshowshowtostartandinvokebothserverandclientprocesseswithlow-levelaccess.
YouwillstartaserverdaemonforaparticularGPUconfiguration(e.
g.
,partialmemory)andwriteaclient-sideconfigurationfile'adaptor.
conf'asshownintheexamplesbelow.
Wehavedoneintegrationsforseveraljobschedulersandresourcemanagers,socontactusifyou'relookingforhelp.
StartingFlexDirectDaemonsasServersviaCLIThedrawingbelowshowsthefourprocessesthatarerunningonaclient(orCPU)nodeandonaserver(orGPU)nodewhenyouareinteractingwiththeFlexDirectServer(Dispatcher).
Itshouldhelpyouunderstandtheconcepts,commandsandusagethatthismanualdiscusses.
Onlytwoprocessesaredirectlylaunchedbytheuser.
Thesearetheonesshowninafixedfontasyouwouldtypetheminacommandshell.
ThedrawingalsoshowstheTCPportsusedbytheGPUserverprocesses.
USERGPUGPUGPUGPUCOMPUTESERVERGPUSERVERALLOCATEINUSEflexdirectserver(Dispatcher)setupclients.
confforGPUserver:55001flexdirectclient--listeningonport55001(default)listeningonports45201+fordatapathmessageslinkedtoVMwareBitfusionCUDAlibCUDAServerCoolAppYoumuststartFlexDirectasaserver(whichiscalledDispatcher)onalltheinstancesthathaveGPUswhichyou'dliketomakeavailabletoyourclientnodesandapplications.
Shellflexdirectserver[-pport]WHITEPAPER|4YoucanalsostartaFlexDirectserver(Dispatcherprocess)fromtheclientmachinewiththerequest_gpuscommand.
However,thisrequiresthattheGPUserverisalreadyrunningtheresourcescheduler.
Advantagesinclude:PreventsmultipleusersfromtryingtoservethesameGPUsCreatesadaptors.
conffileforyouDoesnotautomaticallydeallocatetheGPUsafteraclientapplicationhasfinishedsoyoucanrunseveralapplicationssequentiallyHowever,thisdocumentcoversmanuallaunchesoftheFlexDirectserver.
RunClientApplicationswithFlexDirectviaflexdirectclientOncetheFlexDirectServersarerunning,runapplicationsusingflexdirectclient.
Passthe-lparameterasalistoftheIPaddressesofthenodesonwhichyouhaveFlexDirectServerrunning.
Usesemicolonstoseparatetheaddresses.
Replacewiththeapplicationyouwouldliketorun.
Useadoubledash--beforetheapplicationifitrequiresitsownarguments.
Shellflexdirectclient-l"172.
31.
51.
20;172.
31.
51.
26"[--]CPUServerCommandLineflexdirectclient-l172.
31.
51.
20:55002nvidia-smiGPUServerCommandLinenvidia-smi|NVIDIA-SMI375.
26DriverVersion:375.
26||GPUNamePersistence-M|Bus-IdDisp.
A|VolatileUncorr.
ECC||FanTempPerfPwr:Usage/Cap|Memory-Usage|GPU-UtilComputeM.
||0TeslaK80Off|0000:01:00.
0Off|N/A||N/A%53CP829W/149W|0MiB/11439MiB|0%Default||Processes:GPUMemory||GPUPIDTypeProcessnameUsage||Norunningprocessesfound|Youmayspecifyaportnumberwiththestandardcolonnotation:WHITEPAPER|5USERGPUGPUGPUGPUGPUGPUGPUGPUCOMPUTESERVERGPUSERVERGPUSERVERflexdirectclientflexdirectserverflexdirectserverConfiguringIPAddressesasPartofClientConfigurationIfyouwanttosimplifytheflexdirectclientcommand,youcanputyourBitfusionserverIPaddressesintothe/etc/bitfusionio/adaptor.
conffile.
Overridethedefaultportbyadding:.
CPUServerCommandLineCPUServerCommandLinecat/etc/bitfusionio/adaptor.
conf172.
31.
51.
20172.
31.
51.
26:57001flexdirectclientnvidia-smi|NVIDIA-SMI375.
26DriverVersion:375.
26||GPUNamePersistence-M|Bus-IdDisp.
A|VolatileUncorr.
ECC||FanTempPerfPwr:Usage/Cap|Memory-Usage|GPU-UtilComputeM.
||0TeslaK80Off|0000:01:00.
0Off|N/A||N/A%53CP829W/149W|0MiB/11479MiB|0%Default||Processes:GPUMemory||GPUPIDTypeProcessnameUsage||Norunningprocessesfound|Afterwritingadaptor.
conf,simplyrunflexdirectclientwithaGPUapplication.
Forexample,ifyourunflexdirectclientwithnvidia-smiitwilllisttheGPUsconfigured.
Typeflexdirecthelporflexdirecthelp[command]foradditionalhelpfulcommandsandinformation.
WHITEPAPER|6SampleOutput$flexdirecthelpNAME:flexdirect-RunapplicationwithBitfusionFlexDirectUSAGE:flexdirect"application"flexdirect--[application]flexdirecthelp[command]Formoreinformation,systemrequirements,andadvancedusagepleasevisithttps://www-review.
vmware.
com/solutions/business-critical-apps/hardwareaccelerators-virtualization.
htmlCOMMANDS:init,iInitializeconfiguration.
Requiresrootpriviledges.
version,vDisplayfullFlexDirectversion.
localhealth,LHRunhealthcheckoncurrentnodeonly.
upgrade,UUpgradeversion.
Requiresrootpriviledges.
uninstallUninstallFlexDirect.
Requiresrootpriviledges.
deallocDeallocatelicensecertificate.
Requiresrootpriviledges.
crashreportSendcrashreporttoBitfusion.
licenseChecklicensestatus.
list_gpusListtheavailableGPUsinasharedpool.
help,hShowsalistofcommandsorhelpforonecommand.
ClientCommands:client,cRunapplication.
health,HRunhealthcheckonallspecifiedserversandcurrentnode.
request_gpusRequestGPUsfromasharedpool.
release_gpusReleaseGPUsbackintoasharedpool.
Optionsmustmatchapreviousrequest_gpuscommand.
runRequestGPUsfromasharedpool,runaclientcommand,thenreleasetheGPUs.
statsGatherstatsfromallservers.
smiDisplaysmi-likeinfoforallservers.
localRunaCUDAapplicationlocally.
net_perfGathernetworkperformancedatafromallSRSservers.
ServerCommands:server,sRunserver.
resource_schedulerRunFlexDirectresourcescheduler(SRS)onGPUserverEXAMPLES:$sudoflexdirectinit-l$flexdirectresource_scheduler--srs_port50001$flexdirectrun-n4--Herearesomeflexdirectexampleswithexplanatorycomments.
WHITEPAPER|7TextInitializeflexdirectlicensebeforethefirstrunofserveronasystem$sudoflexdirectinit-lRunaflexdirectserverwithdefaultport55001$flexdirectserverRunaflexdirectserverwithadifferentport$flexdirectserver-p55010Runanapplicationwithaserverrunninglocalwithdefaultport55001$flexdirectclient-l"localhost"Runanapplicationwithmultipleservers,localorremote$flexdirectclient-l"192.
168.
0.
2:55010;192.
168.
0.
6:51234"Runanapplicationwithserversspecifiedinoneofthedefaultconfigfiles(~/.
bitfusionio/adaptor.
confand/etc/bitfusionio/adaptor.
confinpriorityorder)$flexdirectclientRunanapplicationwithserversspecifiedinaconfigfile$flexdirectclient-fRunaserverwitharesourcescheduleronacustomport$flexdirectresource_scheduler--srs_port50001--port55010Runanapplicationwith4sharedGPUs$flexdirectrun-n4Runanapplicationwith2sharedGPUs,usinghalftheavailablememory,andacustomservers.
conf$flexdirectrun-n2-p0.
5-sservers.
confRunanapplicationwith4sharedGPUswithInfiniBand$flexdirectrun-n4Runanapplicationlocally,restrictedtoonlyhalfthephysicalGPUmemory$flexdirectlocal-p0.
5Request8remoteGPUs$flexdirectrequest_gpus-sservers.
conf-fadaptor_8gpu.
conf-n8Runanapplicationwiththegeneratedconfigfile$flexdirectclient-fadaptor_8gpu.
confReleasethe8remoteGPUsaftertheapplicationhasfinished$flexdirectrelease_gpus-fadaptor_8gpu.
confGethelponaspecificcommand(theclientcommandinthisexample)$flexdirecthelpclientAdvanced:ClusterCommunicationsIfyouareunabletoopenupthedefault45201-46225portrangeforin-clustercommunication,youcanoverridethisrangebyexportingtheseenvironmentvariablesonyourGPUserversbeforerunningtheFlexDirectServer(alsocalledDispatcher):GPUServerCommandLine$exportBF_SERVER_PORT_MIN=$exportBF_SERVER_PORT_MAX=Advanced:FlexibleDynamicGPUConfigurationsTheexamplesbelowassumethatyouhaveafour-GPUserveratIPaddress123.
45.
67.
890.
WewillusethisoneGPUnodeforthreedifferentclientapplicationswithslightlydifferentresourceconfigurations,allsharingthesameGPUnode.
WHITEPAPER|8NOTENotehowasweprogressthroughtheexamples,weusedifferentportssothateachserverprocessisutilizinguniqueportsforcommunication.
BF_VISIBLE_DEVICESreferstotheIDnumberofeachGPUdevice,whichstartsat0.
Ifyouhavea4GPUinstance,theIDswouldbe0,1,2,and3respectively.
YoucanseethedevicesandtheirspecificIDsbyrunningnvidia-smi.
ExposingOneGPUoutoftheFourGPUsforAppsStarttheFlexDirectServer(alsocalledDispatcher)onthefirstGPUdevice(outofthefourweareassumingfortheseexamples)withthefollowingcommand:NowruntheFlexDirectClientonyourCPUnode.
Inthisexample,we'lldoitwithapplication"nvidia-smi",butyoucouldreplacethiswiththeapplicationyouwouldliketorunusingFlexDirectvirtualization.
ExposingTwoGPUsoutoftheFourGPUsforAppsStarttheFlexDirectServer(alsocalledDispatcher)onthefour-GPUnodewiththefollowingcommand:GPUServerCommandLineGPUServerCommandLineGPUServerCommandLineBF_VISIBLE_DEVICES=0flexdirectserver-p55001Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55001flexdirectclient-l123.
45.
67.
89:55001nvidia-smi|NVIDIA-SMI375.
26DriverVersion:375.
26||GPUNamePersistence-M|Bus-IdDisp.
A|VolatileUncorr.
ECC||FanTempPerfPwr:Usage/Cap|Memory-Usage|GPU-UtilComputeM.
||0TeslaK80Off|0000:01:00.
0Off|N/A||N/A%53CP829W/149W|0MiB/11479MiB|0%Default||Processes:GPUMemory||GPUPIDTypeProcessnameUsage||Norunningprocessesfound|BF_VISIBLE_DEVICES=0,1flexdirectserver-p55002WHITEPAPER|9RuntheFlexDirectClient.
Inthisexample,we'lldoitwithapplication"nvidia-smi:,butyoucouldreplacethiswiththeapplicationyouwouldliketorunusingFlexDirectvirtualization.
GPUServerCommandLineflexdirectclient-l123.
45.
67.
89:55002nvidia-smi|NVIDIA-SMI375.
26DriverVersion:375.
26||GPUNamePersistence-M|Bus-IdDisp.
A|VolatileUncorr.
ECC||FanTempPerfPwr:Usage/Cap|Memory-Usage|GPU-UtilComputeM.
||0TeslaK80Off|0000:01:00.
0Off|N/A||N/A%53CP829W/149W|0MiB/11479MiB|0%Default||||0TeslaK80Off|0000:01:00.
0Off|N/A||N/A%53CP829W/149W|0MiB/11479MiB|0%Default||Processes:GPUMemory||GPUPIDTypeProcessnameUsage||Norunningprocessesfound|PartialGPUConfigurations1/2-GPUavailableonport55001ThisisdonebysettingenvironmentalvariableBF_GPU_DEVICE_MEMORY_LIMITtohalfoftheGPUsmemory.
NVIDIAGPUSETTINGTOALLOWSHARINGWhenyoupartitionaGPU,presumablyyouwanttobeabletousebothpartitionssimultaneously.
NVIDIAGPUshaveacomputemodethatshouldbesetto"Default"(not"Exclusive")sothatmultipleapplicationscanshareaccess.
Usethenvidia-smi-acommandtoseethecurrentcomputemodesetting.
Andsetthemodeto"Default"withthecommandsudonvidia-smi-c0.
Server-sidecommandsshown,seeaboveonhowtoinvoketheclient.
GPUServerCommandLineBF_VISIBLE_DEVICES=0BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55001Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55001WHITEPAPER|10TwoGPUsAvailableonPort5500121/2-GPUsAvailableonPort55001Fortwohalf-sizedGPUs:161/2GPUsAssignedtoTwoDifferentClients(Acrosstwofour-GPUnodes).
EachclientseeseightpartialGPUs.
Usetwodifferentportnumbers,oneforeachclient.
Commentsareinterlacedwithcommands:GPUServerCommandLineGPUServerCommandLineBF_VISIBLE_DEVICES=0,1flexdirectserver-p55001Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55001BF_VISIBLE_DEVICES=0,1flexdirectserver-p55001Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55001GPUServerCommandLines#server1:$BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55001&$BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55002#server2:$BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55001&$BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55002SampleOutput#server1:Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55001Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55002#server2:Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55001Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55001ConfigurationFiles#client1adaptor.
conf:5500155001#client2adaptor.
conf:5500255002WHITEPAPER|11161/2GPUs(acrosstwofour-GPUnodes)available.
TwodifferentclientseachallocateonepartialGPU.
GPUServerCommandLines#Server1:BF_VISIBLE_DEVICES=0BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55001&BF_VISIBLE_DEVICES=1BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55002&BF_VISIBLE_DEVICES=2BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55003&BF_VISIBLE_DEVICES=3BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55004&BF_VISIBLE_DEVICES=4BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55005&BF_VISIBLE_DEVICES=5BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55006&BF_VISIBLE_DEVICES=6BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55007&BF_VISIBLE_DEVICES=7BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55008&#Server2:BF_VISIBLE_DEVICES=0BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55001&BF_VISIBLE_DEVICES=1BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55002&BF_VISIBLE_DEVICES=2BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55003&BF_VISIBLE_DEVICES=3BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55004&BF_VISIBLE_DEVICES=4BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55005&BF_VISIBLE_DEVICES=5BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55006&BF_VISIBLE_DEVICES=6BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55007&BF_VISIBLE_DEVICES=7BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55008&SampleOutput#Server1:Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55001Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55002Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55003Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55004Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55005Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55006Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55007Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55008#Server2:Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55001Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55002Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55003Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55004Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55005Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55006Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55007Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55008SampleOutput#Client1adaptor.
conf(firstpartialGPUofserver1):55001#Client2adaptor.
conf(secondpartialGPUofserver1):55002VMware,Inc.
3401HillviewAvenuePaloAltoCA94304USATel877-486-9273Fax650-427-5001vmware.
comCopyright2019VMware,Inc.
Allrightsreserved.
ThisproductisprotectedbyU.
S.
andinternationalcopyrightandintellectualpropertylaws.
VMwareproductsarecoveredbyoneormorepatentslistedatvmware.
com/go/patents.
VMwareisaregisteredtrademarkortrademarkofVMware,Inc.
anditssubsidiariesintheUnitedStatesandotherjurisdictions.
Allothermarksandnamesmentionedhereinmaybetrademarksoftheirrespectivecompanies.
ItemNo:VMW-0518-1843_VMW_CPBUTechnicalWhitePapers_BitfusionDocs_08IntegrationandAdvancedUsage_1.
5_YC8/19
5Advanced:ClusterCommunications7Advanced:FlexibleDynamicGPUConfigurations7ExposingOneGPUoutoftheFourGPUsforApps8ExposingTwoGPUsoutoftheFourGPUsforApps8PartialGPUConfigurations9WHITEPAPER|3IntroductionForengineersintegratingBitfusiontechnologyintotheirownresourceschedulerorperhapsforadvancedusersneedingmorecontroloverGPUsareresourcing,thisguideshowshowtostartandinvokebothserverandclientprocesseswithlow-levelaccess.
YouwillstartaserverdaemonforaparticularGPUconfiguration(e.
g.
,partialmemory)andwriteaclient-sideconfigurationfile'adaptor.
conf'asshownintheexamplesbelow.
Wehavedoneintegrationsforseveraljobschedulersandresourcemanagers,socontactusifyou'relookingforhelp.
StartingFlexDirectDaemonsasServersviaCLIThedrawingbelowshowsthefourprocessesthatarerunningonaclient(orCPU)nodeandonaserver(orGPU)nodewhenyouareinteractingwiththeFlexDirectServer(Dispatcher).
Itshouldhelpyouunderstandtheconcepts,commandsandusagethatthismanualdiscusses.
Onlytwoprocessesaredirectlylaunchedbytheuser.
Thesearetheonesshowninafixedfontasyouwouldtypetheminacommandshell.
ThedrawingalsoshowstheTCPportsusedbytheGPUserverprocesses.
USERGPUGPUGPUGPUCOMPUTESERVERGPUSERVERALLOCATEINUSEflexdirectserver(Dispatcher)setupclients.
confforGPUserver:55001flexdirectclient--listeningonport55001(default)listeningonports45201+fordatapathmessageslinkedtoVMwareBitfusionCUDAlibCUDAServerCoolAppYoumuststartFlexDirectasaserver(whichiscalledDispatcher)onalltheinstancesthathaveGPUswhichyou'dliketomakeavailabletoyourclientnodesandapplications.
Shellflexdirectserver[-pport]WHITEPAPER|4YoucanalsostartaFlexDirectserver(Dispatcherprocess)fromtheclientmachinewiththerequest_gpuscommand.
However,thisrequiresthattheGPUserverisalreadyrunningtheresourcescheduler.
Advantagesinclude:PreventsmultipleusersfromtryingtoservethesameGPUsCreatesadaptors.
conffileforyouDoesnotautomaticallydeallocatetheGPUsafteraclientapplicationhasfinishedsoyoucanrunseveralapplicationssequentiallyHowever,thisdocumentcoversmanuallaunchesoftheFlexDirectserver.
RunClientApplicationswithFlexDirectviaflexdirectclientOncetheFlexDirectServersarerunning,runapplicationsusingflexdirectclient.
Passthe-lparameterasalistoftheIPaddressesofthenodesonwhichyouhaveFlexDirectServerrunning.
Usesemicolonstoseparatetheaddresses.
Replacewiththeapplicationyouwouldliketorun.
Useadoubledash--beforetheapplicationifitrequiresitsownarguments.
Shellflexdirectclient-l"172.
31.
51.
20;172.
31.
51.
26"[--]CPUServerCommandLineflexdirectclient-l172.
31.
51.
20:55002nvidia-smiGPUServerCommandLinenvidia-smi|NVIDIA-SMI375.
26DriverVersion:375.
26||GPUNamePersistence-M|Bus-IdDisp.
A|VolatileUncorr.
ECC||FanTempPerfPwr:Usage/Cap|Memory-Usage|GPU-UtilComputeM.
||0TeslaK80Off|0000:01:00.
0Off|N/A||N/A%53CP829W/149W|0MiB/11439MiB|0%Default||Processes:GPUMemory||GPUPIDTypeProcessnameUsage||Norunningprocessesfound|Youmayspecifyaportnumberwiththestandardcolonnotation:WHITEPAPER|5USERGPUGPUGPUGPUGPUGPUGPUGPUCOMPUTESERVERGPUSERVERGPUSERVERflexdirectclientflexdirectserverflexdirectserverConfiguringIPAddressesasPartofClientConfigurationIfyouwanttosimplifytheflexdirectclientcommand,youcanputyourBitfusionserverIPaddressesintothe/etc/bitfusionio/adaptor.
conffile.
Overridethedefaultportbyadding:.
CPUServerCommandLineCPUServerCommandLinecat/etc/bitfusionio/adaptor.
conf172.
31.
51.
20172.
31.
51.
26:57001flexdirectclientnvidia-smi|NVIDIA-SMI375.
26DriverVersion:375.
26||GPUNamePersistence-M|Bus-IdDisp.
A|VolatileUncorr.
ECC||FanTempPerfPwr:Usage/Cap|Memory-Usage|GPU-UtilComputeM.
||0TeslaK80Off|0000:01:00.
0Off|N/A||N/A%53CP829W/149W|0MiB/11479MiB|0%Default||Processes:GPUMemory||GPUPIDTypeProcessnameUsage||Norunningprocessesfound|Afterwritingadaptor.
conf,simplyrunflexdirectclientwithaGPUapplication.
Forexample,ifyourunflexdirectclientwithnvidia-smiitwilllisttheGPUsconfigured.
Typeflexdirecthelporflexdirecthelp[command]foradditionalhelpfulcommandsandinformation.
WHITEPAPER|6SampleOutput$flexdirecthelpNAME:flexdirect-RunapplicationwithBitfusionFlexDirectUSAGE:flexdirect"application"flexdirect--[application]flexdirecthelp[command]Formoreinformation,systemrequirements,andadvancedusagepleasevisithttps://www-review.
vmware.
com/solutions/business-critical-apps/hardwareaccelerators-virtualization.
htmlCOMMANDS:init,iInitializeconfiguration.
Requiresrootpriviledges.
version,vDisplayfullFlexDirectversion.
localhealth,LHRunhealthcheckoncurrentnodeonly.
upgrade,UUpgradeversion.
Requiresrootpriviledges.
uninstallUninstallFlexDirect.
Requiresrootpriviledges.
deallocDeallocatelicensecertificate.
Requiresrootpriviledges.
crashreportSendcrashreporttoBitfusion.
licenseChecklicensestatus.
list_gpusListtheavailableGPUsinasharedpool.
help,hShowsalistofcommandsorhelpforonecommand.
ClientCommands:client,cRunapplication.
health,HRunhealthcheckonallspecifiedserversandcurrentnode.
request_gpusRequestGPUsfromasharedpool.
release_gpusReleaseGPUsbackintoasharedpool.
Optionsmustmatchapreviousrequest_gpuscommand.
runRequestGPUsfromasharedpool,runaclientcommand,thenreleasetheGPUs.
statsGatherstatsfromallservers.
smiDisplaysmi-likeinfoforallservers.
localRunaCUDAapplicationlocally.
net_perfGathernetworkperformancedatafromallSRSservers.
ServerCommands:server,sRunserver.
resource_schedulerRunFlexDirectresourcescheduler(SRS)onGPUserverEXAMPLES:$sudoflexdirectinit-l$flexdirectresource_scheduler--srs_port50001$flexdirectrun-n4--Herearesomeflexdirectexampleswithexplanatorycomments.
WHITEPAPER|7TextInitializeflexdirectlicensebeforethefirstrunofserveronasystem$sudoflexdirectinit-lRunaflexdirectserverwithdefaultport55001$flexdirectserverRunaflexdirectserverwithadifferentport$flexdirectserver-p55010Runanapplicationwithaserverrunninglocalwithdefaultport55001$flexdirectclient-l"localhost"Runanapplicationwithmultipleservers,localorremote$flexdirectclient-l"192.
168.
0.
2:55010;192.
168.
0.
6:51234"Runanapplicationwithserversspecifiedinoneofthedefaultconfigfiles(~/.
bitfusionio/adaptor.
confand/etc/bitfusionio/adaptor.
confinpriorityorder)$flexdirectclientRunanapplicationwithserversspecifiedinaconfigfile$flexdirectclient-fRunaserverwitharesourcescheduleronacustomport$flexdirectresource_scheduler--srs_port50001--port55010Runanapplicationwith4sharedGPUs$flexdirectrun-n4Runanapplicationwith2sharedGPUs,usinghalftheavailablememory,andacustomservers.
conf$flexdirectrun-n2-p0.
5-sservers.
confRunanapplicationwith4sharedGPUswithInfiniBand$flexdirectrun-n4Runanapplicationlocally,restrictedtoonlyhalfthephysicalGPUmemory$flexdirectlocal-p0.
5Request8remoteGPUs$flexdirectrequest_gpus-sservers.
conf-fadaptor_8gpu.
conf-n8Runanapplicationwiththegeneratedconfigfile$flexdirectclient-fadaptor_8gpu.
confReleasethe8remoteGPUsaftertheapplicationhasfinished$flexdirectrelease_gpus-fadaptor_8gpu.
confGethelponaspecificcommand(theclientcommandinthisexample)$flexdirecthelpclientAdvanced:ClusterCommunicationsIfyouareunabletoopenupthedefault45201-46225portrangeforin-clustercommunication,youcanoverridethisrangebyexportingtheseenvironmentvariablesonyourGPUserversbeforerunningtheFlexDirectServer(alsocalledDispatcher):GPUServerCommandLine$exportBF_SERVER_PORT_MIN=$exportBF_SERVER_PORT_MAX=Advanced:FlexibleDynamicGPUConfigurationsTheexamplesbelowassumethatyouhaveafour-GPUserveratIPaddress123.
45.
67.
890.
WewillusethisoneGPUnodeforthreedifferentclientapplicationswithslightlydifferentresourceconfigurations,allsharingthesameGPUnode.
WHITEPAPER|8NOTENotehowasweprogressthroughtheexamples,weusedifferentportssothateachserverprocessisutilizinguniqueportsforcommunication.
BF_VISIBLE_DEVICESreferstotheIDnumberofeachGPUdevice,whichstartsat0.
Ifyouhavea4GPUinstance,theIDswouldbe0,1,2,and3respectively.
YoucanseethedevicesandtheirspecificIDsbyrunningnvidia-smi.
ExposingOneGPUoutoftheFourGPUsforAppsStarttheFlexDirectServer(alsocalledDispatcher)onthefirstGPUdevice(outofthefourweareassumingfortheseexamples)withthefollowingcommand:NowruntheFlexDirectClientonyourCPUnode.
Inthisexample,we'lldoitwithapplication"nvidia-smi",butyoucouldreplacethiswiththeapplicationyouwouldliketorunusingFlexDirectvirtualization.
ExposingTwoGPUsoutoftheFourGPUsforAppsStarttheFlexDirectServer(alsocalledDispatcher)onthefour-GPUnodewiththefollowingcommand:GPUServerCommandLineGPUServerCommandLineGPUServerCommandLineBF_VISIBLE_DEVICES=0flexdirectserver-p55001Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55001flexdirectclient-l123.
45.
67.
89:55001nvidia-smi|NVIDIA-SMI375.
26DriverVersion:375.
26||GPUNamePersistence-M|Bus-IdDisp.
A|VolatileUncorr.
ECC||FanTempPerfPwr:Usage/Cap|Memory-Usage|GPU-UtilComputeM.
||0TeslaK80Off|0000:01:00.
0Off|N/A||N/A%53CP829W/149W|0MiB/11479MiB|0%Default||Processes:GPUMemory||GPUPIDTypeProcessnameUsage||Norunningprocessesfound|BF_VISIBLE_DEVICES=0,1flexdirectserver-p55002WHITEPAPER|9RuntheFlexDirectClient.
Inthisexample,we'lldoitwithapplication"nvidia-smi:,butyoucouldreplacethiswiththeapplicationyouwouldliketorunusingFlexDirectvirtualization.
GPUServerCommandLineflexdirectclient-l123.
45.
67.
89:55002nvidia-smi|NVIDIA-SMI375.
26DriverVersion:375.
26||GPUNamePersistence-M|Bus-IdDisp.
A|VolatileUncorr.
ECC||FanTempPerfPwr:Usage/Cap|Memory-Usage|GPU-UtilComputeM.
||0TeslaK80Off|0000:01:00.
0Off|N/A||N/A%53CP829W/149W|0MiB/11479MiB|0%Default||||0TeslaK80Off|0000:01:00.
0Off|N/A||N/A%53CP829W/149W|0MiB/11479MiB|0%Default||Processes:GPUMemory||GPUPIDTypeProcessnameUsage||Norunningprocessesfound|PartialGPUConfigurations1/2-GPUavailableonport55001ThisisdonebysettingenvironmentalvariableBF_GPU_DEVICE_MEMORY_LIMITtohalfoftheGPUsmemory.
NVIDIAGPUSETTINGTOALLOWSHARINGWhenyoupartitionaGPU,presumablyyouwanttobeabletousebothpartitionssimultaneously.
NVIDIAGPUshaveacomputemodethatshouldbesetto"Default"(not"Exclusive")sothatmultipleapplicationscanshareaccess.
Usethenvidia-smi-acommandtoseethecurrentcomputemodesetting.
Andsetthemodeto"Default"withthecommandsudonvidia-smi-c0.
Server-sidecommandsshown,seeaboveonhowtoinvoketheclient.
GPUServerCommandLineBF_VISIBLE_DEVICES=0BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55001Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55001WHITEPAPER|10TwoGPUsAvailableonPort5500121/2-GPUsAvailableonPort55001Fortwohalf-sizedGPUs:161/2GPUsAssignedtoTwoDifferentClients(Acrosstwofour-GPUnodes).
EachclientseeseightpartialGPUs.
Usetwodifferentportnumbers,oneforeachclient.
Commentsareinterlacedwithcommands:GPUServerCommandLineGPUServerCommandLineBF_VISIBLE_DEVICES=0,1flexdirectserver-p55001Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55001BF_VISIBLE_DEVICES=0,1flexdirectserver-p55001Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55001GPUServerCommandLines#server1:$BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55001&$BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55002#server2:$BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55001&$BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55002SampleOutput#server1:Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55001Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55002#server2:Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55001Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55001ConfigurationFiles#client1adaptor.
conf:5500155001#client2adaptor.
conf:5500255002WHITEPAPER|11161/2GPUs(acrosstwofour-GPUnodes)available.
TwodifferentclientseachallocateonepartialGPU.
GPUServerCommandLines#Server1:BF_VISIBLE_DEVICES=0BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55001&BF_VISIBLE_DEVICES=1BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55002&BF_VISIBLE_DEVICES=2BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55003&BF_VISIBLE_DEVICES=3BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55004&BF_VISIBLE_DEVICES=4BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55005&BF_VISIBLE_DEVICES=5BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55006&BF_VISIBLE_DEVICES=6BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55007&BF_VISIBLE_DEVICES=7BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55008&#Server2:BF_VISIBLE_DEVICES=0BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55001&BF_VISIBLE_DEVICES=1BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55002&BF_VISIBLE_DEVICES=2BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55003&BF_VISIBLE_DEVICES=3BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55004&BF_VISIBLE_DEVICES=4BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55005&BF_VISIBLE_DEVICES=5BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55006&BF_VISIBLE_DEVICES=6BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55007&BF_VISIBLE_DEVICES=7BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55008&SampleOutput#Server1:Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55001Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55002Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55003Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55004Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55005Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55006Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55007Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55008#Server2:Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55001Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55002Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55003Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55004Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55005Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55006Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55007Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55008SampleOutput#Client1adaptor.
conf(firstpartialGPUofserver1):55001#Client2adaptor.
conf(secondpartialGPUofserver1):55002VMware,Inc.
3401HillviewAvenuePaloAltoCA94304USATel877-486-9273Fax650-427-5001vmware.
comCopyright2019VMware,Inc.
Allrightsreserved.
ThisproductisprotectedbyU.
S.
andinternationalcopyrightandintellectualpropertylaws.
VMwareproductsarecoveredbyoneormorepatentslistedatvmware.
com/go/patents.
VMwareisaregisteredtrademarkortrademarkofVMware,Inc.
anditssubsidiariesintheUnitedStatesandotherjurisdictions.
Allothermarksandnamesmentionedhereinmaybetrademarksoftheirrespectivecompanies.
ItemNo:VMW-0518-1843_VMW_CPBUTechnicalWhitePapers_BitfusionDocs_08IntegrationandAdvancedUsage_1.
5_YC8/19
- sequentiallyyc8相关文档
- Statisticsyc8
- stackyc8
- 系统一级医院系统集成专业方案Com
- 调度使用COM—Express Nano工控板实现IP调度设备
- 接口基于COM技术的开放式数控软件系统的开发与研究
- 共产主义com开头词汇
RAKSmart VPS主机半价活动 支持Windows系统 包含香港、日本机房
RAKSmart 商家最近动作还是比较大的，比如他们也在增加云服务器产品，目前已经包含美国圣何塞和洛杉矶机房，以及这个月有新增的中国香港机房，根据大趋势云服务器算是比较技术流的趋势。传统的VPS主机架构方案在技术层面上稍微落后一些，当然也是可以用的。不清楚是商家出于对于传统VPS主机清理库存，还是多渠道的产品化营销，看到RAKSmart VPS主机提供美国、香港和日本机房的半价促销，当然也包括其他...
inlicloud48元/月，云主机,2核1G/200Mbps，可选安徽/上海联通/广州移动/江门移动NAT
inlicloud怎么样?inlicloud(引力主机)主要产品为国内NAT系列VPS，目前主要有：上海联通NAT(200Mbps带宽)、宿州联通NAT(200Mbps带宽)、广州移动NAT(200Mbps带宽)。根据官方的说法国内的NAT系列VPS不要求备案、不要求实名、对中转要求也不严格，但是，禁止任何形式的回国!安徽nat/上海联通/广州移动/江门移动nat云主机,2核1G/200Mbps仅...
RackNerd：特价美国服务器促销,高配低价,美国多机房可选择,双E526**+AMD3700+NVMe
racknerd怎么样？racknerd今天发布了几款美国特价独立服务器的促销，本次商家主推高配置的服务器，各个配置给的都比较高，有Intel和AMD两种，硬盘也有NVMe和SSD等多咱组合可以选择，机房目前有夏洛特、洛杉矶、犹他州可以选择，性价比很高，有需要独服的朋友可以看看。点击进入：racknerd官方网站RackNerd暑假独服促销：CPU：双E5-2680v3 (24核心，48线程)内存...
WWW YC8 COM为你推荐
-
淘宝客推广淘宝客推广有哪些途径云播怎么看片云播看不了视频pwpw域名的技巧百度手写百度手写显示镜像文件是什么什么是文件镜像?什么是镜像文件?镜像文件是什么什么叫镜像文件，作用是什么？童之磊网文大学很强吗？如何建立一个网站怎样能创建一个网站迅雷云点播账号求迅雷云播账号创维云电视功能什么是创维云电视啊?创维云电视是什么意思?