sequentiallyyc8
WWW YC8 COM 时间:2021-03-02 阅读:()
IntegrationandAdvancedUsageBitfusionGuideWHITEPAPER–OCTOBER2019WHITEPAPER|2Bitfusion:IntegrationandAdvancedUsageTableofcontentsIntroduction3StartingFlexDirectDaemonsasServersviaCLI3RunClientApplicationswithFlexDirectviaflexdirectclient4ConfiguringIPAddressesasPartofClientConfiguration.
5Advanced:ClusterCommunications7Advanced:FlexibleDynamicGPUConfigurations7ExposingOneGPUoutoftheFourGPUsforApps8ExposingTwoGPUsoutoftheFourGPUsforApps8PartialGPUConfigurations9WHITEPAPER|3IntroductionForengineersintegratingBitfusiontechnologyintotheirownresourceschedulerorperhapsforadvancedusersneedingmorecontroloverGPUsareresourcing,thisguideshowshowtostartandinvokebothserverandclientprocesseswithlow-levelaccess.
YouwillstartaserverdaemonforaparticularGPUconfiguration(e.
g.
,partialmemory)andwriteaclient-sideconfigurationfile'adaptor.
conf'asshownintheexamplesbelow.
Wehavedoneintegrationsforseveraljobschedulersandresourcemanagers,socontactusifyou'relookingforhelp.
StartingFlexDirectDaemonsasServersviaCLIThedrawingbelowshowsthefourprocessesthatarerunningonaclient(orCPU)nodeandonaserver(orGPU)nodewhenyouareinteractingwiththeFlexDirectServer(Dispatcher).
Itshouldhelpyouunderstandtheconcepts,commandsandusagethatthismanualdiscusses.
Onlytwoprocessesaredirectlylaunchedbytheuser.
Thesearetheonesshowninafixedfontasyouwouldtypetheminacommandshell.
ThedrawingalsoshowstheTCPportsusedbytheGPUserverprocesses.
USERGPUGPUGPUGPUCOMPUTESERVERGPUSERVERALLOCATEINUSEflexdirectserver(Dispatcher)setupclients.
confforGPUserver:55001flexdirectclient--listeningonport55001(default)listeningonports45201+fordatapathmessageslinkedtoVMwareBitfusionCUDAlibCUDAServerCoolAppYoumuststartFlexDirectasaserver(whichiscalledDispatcher)onalltheinstancesthathaveGPUswhichyou'dliketomakeavailabletoyourclientnodesandapplications.
Shellflexdirectserver[-pport]WHITEPAPER|4YoucanalsostartaFlexDirectserver(Dispatcherprocess)fromtheclientmachinewiththerequest_gpuscommand.
However,thisrequiresthattheGPUserverisalreadyrunningtheresourcescheduler.
Advantagesinclude:PreventsmultipleusersfromtryingtoservethesameGPUsCreatesadaptors.
conffileforyouDoesnotautomaticallydeallocatetheGPUsafteraclientapplicationhasfinishedsoyoucanrunseveralapplicationssequentiallyHowever,thisdocumentcoversmanuallaunchesoftheFlexDirectserver.
RunClientApplicationswithFlexDirectviaflexdirectclientOncetheFlexDirectServersarerunning,runapplicationsusingflexdirectclient.
Passthe-lparameterasalistoftheIPaddressesofthenodesonwhichyouhaveFlexDirectServerrunning.
Usesemicolonstoseparatetheaddresses.
Replacewiththeapplicationyouwouldliketorun.
Useadoubledash--beforetheapplicationifitrequiresitsownarguments.
Shellflexdirectclient-l"172.
31.
51.
20;172.
31.
51.
26"[--]CPUServerCommandLineflexdirectclient-l172.
31.
51.
20:55002nvidia-smiGPUServerCommandLinenvidia-smi|NVIDIA-SMI375.
26DriverVersion:375.
26||GPUNamePersistence-M|Bus-IdDisp.
A|VolatileUncorr.
ECC||FanTempPerfPwr:Usage/Cap|Memory-Usage|GPU-UtilComputeM.
||0TeslaK80Off|0000:01:00.
0Off|N/A||N/A%53CP829W/149W|0MiB/11439MiB|0%Default||Processes:GPUMemory||GPUPIDTypeProcessnameUsage||Norunningprocessesfound|Youmayspecifyaportnumberwiththestandardcolonnotation:WHITEPAPER|5USERGPUGPUGPUGPUGPUGPUGPUGPUCOMPUTESERVERGPUSERVERGPUSERVERflexdirectclientflexdirectserverflexdirectserverConfiguringIPAddressesasPartofClientConfigurationIfyouwanttosimplifytheflexdirectclientcommand,youcanputyourBitfusionserverIPaddressesintothe/etc/bitfusionio/adaptor.
conffile.
Overridethedefaultportbyadding:.
CPUServerCommandLineCPUServerCommandLinecat/etc/bitfusionio/adaptor.
conf172.
31.
51.
20172.
31.
51.
26:57001flexdirectclientnvidia-smi|NVIDIA-SMI375.
26DriverVersion:375.
26||GPUNamePersistence-M|Bus-IdDisp.
A|VolatileUncorr.
ECC||FanTempPerfPwr:Usage/Cap|Memory-Usage|GPU-UtilComputeM.
||0TeslaK80Off|0000:01:00.
0Off|N/A||N/A%53CP829W/149W|0MiB/11479MiB|0%Default||Processes:GPUMemory||GPUPIDTypeProcessnameUsage||Norunningprocessesfound|Afterwritingadaptor.
conf,simplyrunflexdirectclientwithaGPUapplication.
Forexample,ifyourunflexdirectclientwithnvidia-smiitwilllisttheGPUsconfigured.
Typeflexdirecthelporflexdirecthelp[command]foradditionalhelpfulcommandsandinformation.
WHITEPAPER|6SampleOutput$flexdirecthelpNAME:flexdirect-RunapplicationwithBitfusionFlexDirectUSAGE:flexdirect"application"flexdirect--[application]flexdirecthelp[command]Formoreinformation,systemrequirements,andadvancedusagepleasevisithttps://www-review.
vmware.
com/solutions/business-critical-apps/hardwareaccelerators-virtualization.
htmlCOMMANDS:init,iInitializeconfiguration.
Requiresrootpriviledges.
version,vDisplayfullFlexDirectversion.
localhealth,LHRunhealthcheckoncurrentnodeonly.
upgrade,UUpgradeversion.
Requiresrootpriviledges.
uninstallUninstallFlexDirect.
Requiresrootpriviledges.
deallocDeallocatelicensecertificate.
Requiresrootpriviledges.
crashreportSendcrashreporttoBitfusion.
licenseChecklicensestatus.
list_gpusListtheavailableGPUsinasharedpool.
help,hShowsalistofcommandsorhelpforonecommand.
ClientCommands:client,cRunapplication.
health,HRunhealthcheckonallspecifiedserversandcurrentnode.
request_gpusRequestGPUsfromasharedpool.
release_gpusReleaseGPUsbackintoasharedpool.
Optionsmustmatchapreviousrequest_gpuscommand.
runRequestGPUsfromasharedpool,runaclientcommand,thenreleasetheGPUs.
statsGatherstatsfromallservers.
smiDisplaysmi-likeinfoforallservers.
localRunaCUDAapplicationlocally.
net_perfGathernetworkperformancedatafromallSRSservers.
ServerCommands:server,sRunserver.
resource_schedulerRunFlexDirectresourcescheduler(SRS)onGPUserverEXAMPLES:$sudoflexdirectinit-l$flexdirectresource_scheduler--srs_port50001$flexdirectrun-n4--Herearesomeflexdirectexampleswithexplanatorycomments.
WHITEPAPER|7TextInitializeflexdirectlicensebeforethefirstrunofserveronasystem$sudoflexdirectinit-lRunaflexdirectserverwithdefaultport55001$flexdirectserverRunaflexdirectserverwithadifferentport$flexdirectserver-p55010Runanapplicationwithaserverrunninglocalwithdefaultport55001$flexdirectclient-l"localhost"Runanapplicationwithmultipleservers,localorremote$flexdirectclient-l"192.
168.
0.
2:55010;192.
168.
0.
6:51234"Runanapplicationwithserversspecifiedinoneofthedefaultconfigfiles(~/.
bitfusionio/adaptor.
confand/etc/bitfusionio/adaptor.
confinpriorityorder)$flexdirectclientRunanapplicationwithserversspecifiedinaconfigfile$flexdirectclient-fRunaserverwitharesourcescheduleronacustomport$flexdirectresource_scheduler--srs_port50001--port55010Runanapplicationwith4sharedGPUs$flexdirectrun-n4Runanapplicationwith2sharedGPUs,usinghalftheavailablememory,andacustomservers.
conf$flexdirectrun-n2-p0.
5-sservers.
confRunanapplicationwith4sharedGPUswithInfiniBand$flexdirectrun-n4Runanapplicationlocally,restrictedtoonlyhalfthephysicalGPUmemory$flexdirectlocal-p0.
5Request8remoteGPUs$flexdirectrequest_gpus-sservers.
conf-fadaptor_8gpu.
conf-n8Runanapplicationwiththegeneratedconfigfile$flexdirectclient-fadaptor_8gpu.
confReleasethe8remoteGPUsaftertheapplicationhasfinished$flexdirectrelease_gpus-fadaptor_8gpu.
confGethelponaspecificcommand(theclientcommandinthisexample)$flexdirecthelpclientAdvanced:ClusterCommunicationsIfyouareunabletoopenupthedefault45201-46225portrangeforin-clustercommunication,youcanoverridethisrangebyexportingtheseenvironmentvariablesonyourGPUserversbeforerunningtheFlexDirectServer(alsocalledDispatcher):GPUServerCommandLine$exportBF_SERVER_PORT_MIN=$exportBF_SERVER_PORT_MAX=Advanced:FlexibleDynamicGPUConfigurationsTheexamplesbelowassumethatyouhaveafour-GPUserveratIPaddress123.
45.
67.
890.
WewillusethisoneGPUnodeforthreedifferentclientapplicationswithslightlydifferentresourceconfigurations,allsharingthesameGPUnode.
WHITEPAPER|8NOTENotehowasweprogressthroughtheexamples,weusedifferentportssothateachserverprocessisutilizinguniqueportsforcommunication.
BF_VISIBLE_DEVICESreferstotheIDnumberofeachGPUdevice,whichstartsat0.
Ifyouhavea4GPUinstance,theIDswouldbe0,1,2,and3respectively.
YoucanseethedevicesandtheirspecificIDsbyrunningnvidia-smi.
ExposingOneGPUoutoftheFourGPUsforAppsStarttheFlexDirectServer(alsocalledDispatcher)onthefirstGPUdevice(outofthefourweareassumingfortheseexamples)withthefollowingcommand:NowruntheFlexDirectClientonyourCPUnode.
Inthisexample,we'lldoitwithapplication"nvidia-smi",butyoucouldreplacethiswiththeapplicationyouwouldliketorunusingFlexDirectvirtualization.
ExposingTwoGPUsoutoftheFourGPUsforAppsStarttheFlexDirectServer(alsocalledDispatcher)onthefour-GPUnodewiththefollowingcommand:GPUServerCommandLineGPUServerCommandLineGPUServerCommandLineBF_VISIBLE_DEVICES=0flexdirectserver-p55001Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55001flexdirectclient-l123.
45.
67.
89:55001nvidia-smi|NVIDIA-SMI375.
26DriverVersion:375.
26||GPUNamePersistence-M|Bus-IdDisp.
A|VolatileUncorr.
ECC||FanTempPerfPwr:Usage/Cap|Memory-Usage|GPU-UtilComputeM.
||0TeslaK80Off|0000:01:00.
0Off|N/A||N/A%53CP829W/149W|0MiB/11479MiB|0%Default||Processes:GPUMemory||GPUPIDTypeProcessnameUsage||Norunningprocessesfound|BF_VISIBLE_DEVICES=0,1flexdirectserver-p55002WHITEPAPER|9RuntheFlexDirectClient.
Inthisexample,we'lldoitwithapplication"nvidia-smi:,butyoucouldreplacethiswiththeapplicationyouwouldliketorunusingFlexDirectvirtualization.
GPUServerCommandLineflexdirectclient-l123.
45.
67.
89:55002nvidia-smi|NVIDIA-SMI375.
26DriverVersion:375.
26||GPUNamePersistence-M|Bus-IdDisp.
A|VolatileUncorr.
ECC||FanTempPerfPwr:Usage/Cap|Memory-Usage|GPU-UtilComputeM.
||0TeslaK80Off|0000:01:00.
0Off|N/A||N/A%53CP829W/149W|0MiB/11479MiB|0%Default||||0TeslaK80Off|0000:01:00.
0Off|N/A||N/A%53CP829W/149W|0MiB/11479MiB|0%Default||Processes:GPUMemory||GPUPIDTypeProcessnameUsage||Norunningprocessesfound|PartialGPUConfigurations1/2-GPUavailableonport55001ThisisdonebysettingenvironmentalvariableBF_GPU_DEVICE_MEMORY_LIMITtohalfoftheGPUsmemory.
NVIDIAGPUSETTINGTOALLOWSHARINGWhenyoupartitionaGPU,presumablyyouwanttobeabletousebothpartitionssimultaneously.
NVIDIAGPUshaveacomputemodethatshouldbesetto"Default"(not"Exclusive")sothatmultipleapplicationscanshareaccess.
Usethenvidia-smi-acommandtoseethecurrentcomputemodesetting.
Andsetthemodeto"Default"withthecommandsudonvidia-smi-c0.
Server-sidecommandsshown,seeaboveonhowtoinvoketheclient.
GPUServerCommandLineBF_VISIBLE_DEVICES=0BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55001Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55001WHITEPAPER|10TwoGPUsAvailableonPort5500121/2-GPUsAvailableonPort55001Fortwohalf-sizedGPUs:161/2GPUsAssignedtoTwoDifferentClients(Acrosstwofour-GPUnodes).
EachclientseeseightpartialGPUs.
Usetwodifferentportnumbers,oneforeachclient.
Commentsareinterlacedwithcommands:GPUServerCommandLineGPUServerCommandLineBF_VISIBLE_DEVICES=0,1flexdirectserver-p55001Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55001BF_VISIBLE_DEVICES=0,1flexdirectserver-p55001Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55001GPUServerCommandLines#server1:$BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55001&$BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55002#server2:$BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55001&$BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55002SampleOutput#server1:Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55001Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55002#server2:Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55001Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55001ConfigurationFiles#client1adaptor.
conf:5500155001#client2adaptor.
conf:5500255002WHITEPAPER|11161/2GPUs(acrosstwofour-GPUnodes)available.
TwodifferentclientseachallocateonepartialGPU.
GPUServerCommandLines#Server1:BF_VISIBLE_DEVICES=0BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55001&BF_VISIBLE_DEVICES=1BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55002&BF_VISIBLE_DEVICES=2BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55003&BF_VISIBLE_DEVICES=3BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55004&BF_VISIBLE_DEVICES=4BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55005&BF_VISIBLE_DEVICES=5BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55006&BF_VISIBLE_DEVICES=6BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55007&BF_VISIBLE_DEVICES=7BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55008&#Server2:BF_VISIBLE_DEVICES=0BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55001&BF_VISIBLE_DEVICES=1BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55002&BF_VISIBLE_DEVICES=2BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55003&BF_VISIBLE_DEVICES=3BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55004&BF_VISIBLE_DEVICES=4BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55005&BF_VISIBLE_DEVICES=5BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55006&BF_VISIBLE_DEVICES=6BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55007&BF_VISIBLE_DEVICES=7BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55008&SampleOutput#Server1:Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55001Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55002Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55003Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55004Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55005Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55006Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55007Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55008#Server2:Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55001Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55002Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55003Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55004Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55005Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55006Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55007Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55008SampleOutput#Client1adaptor.
conf(firstpartialGPUofserver1):55001#Client2adaptor.
conf(secondpartialGPUofserver1):55002VMware,Inc.
3401HillviewAvenuePaloAltoCA94304USATel877-486-9273Fax650-427-5001vmware.
comCopyright2019VMware,Inc.
Allrightsreserved.
ThisproductisprotectedbyU.
S.
andinternationalcopyrightandintellectualpropertylaws.
VMwareproductsarecoveredbyoneormorepatentslistedatvmware.
com/go/patents.
VMwareisaregisteredtrademarkortrademarkofVMware,Inc.
anditssubsidiariesintheUnitedStatesandotherjurisdictions.
Allothermarksandnamesmentionedhereinmaybetrademarksoftheirrespectivecompanies.
ItemNo:VMW-0518-1843_VMW_CPBUTechnicalWhitePapers_BitfusionDocs_08IntegrationandAdvancedUsage_1.
5_YC8/19
5Advanced:ClusterCommunications7Advanced:FlexibleDynamicGPUConfigurations7ExposingOneGPUoutoftheFourGPUsforApps8ExposingTwoGPUsoutoftheFourGPUsforApps8PartialGPUConfigurations9WHITEPAPER|3IntroductionForengineersintegratingBitfusiontechnologyintotheirownresourceschedulerorperhapsforadvancedusersneedingmorecontroloverGPUsareresourcing,thisguideshowshowtostartandinvokebothserverandclientprocesseswithlow-levelaccess.
YouwillstartaserverdaemonforaparticularGPUconfiguration(e.
g.
,partialmemory)andwriteaclient-sideconfigurationfile'adaptor.
conf'asshownintheexamplesbelow.
Wehavedoneintegrationsforseveraljobschedulersandresourcemanagers,socontactusifyou'relookingforhelp.
StartingFlexDirectDaemonsasServersviaCLIThedrawingbelowshowsthefourprocessesthatarerunningonaclient(orCPU)nodeandonaserver(orGPU)nodewhenyouareinteractingwiththeFlexDirectServer(Dispatcher).
Itshouldhelpyouunderstandtheconcepts,commandsandusagethatthismanualdiscusses.
Onlytwoprocessesaredirectlylaunchedbytheuser.
Thesearetheonesshowninafixedfontasyouwouldtypetheminacommandshell.
ThedrawingalsoshowstheTCPportsusedbytheGPUserverprocesses.
USERGPUGPUGPUGPUCOMPUTESERVERGPUSERVERALLOCATEINUSEflexdirectserver(Dispatcher)setupclients.
confforGPUserver:55001flexdirectclient--listeningonport55001(default)listeningonports45201+fordatapathmessageslinkedtoVMwareBitfusionCUDAlibCUDAServerCoolAppYoumuststartFlexDirectasaserver(whichiscalledDispatcher)onalltheinstancesthathaveGPUswhichyou'dliketomakeavailabletoyourclientnodesandapplications.
Shellflexdirectserver[-pport]WHITEPAPER|4YoucanalsostartaFlexDirectserver(Dispatcherprocess)fromtheclientmachinewiththerequest_gpuscommand.
However,thisrequiresthattheGPUserverisalreadyrunningtheresourcescheduler.
Advantagesinclude:PreventsmultipleusersfromtryingtoservethesameGPUsCreatesadaptors.
conffileforyouDoesnotautomaticallydeallocatetheGPUsafteraclientapplicationhasfinishedsoyoucanrunseveralapplicationssequentiallyHowever,thisdocumentcoversmanuallaunchesoftheFlexDirectserver.
RunClientApplicationswithFlexDirectviaflexdirectclientOncetheFlexDirectServersarerunning,runapplicationsusingflexdirectclient.
Passthe-lparameterasalistoftheIPaddressesofthenodesonwhichyouhaveFlexDirectServerrunning.
Usesemicolonstoseparatetheaddresses.
Replacewiththeapplicationyouwouldliketorun.
Useadoubledash--beforetheapplicationifitrequiresitsownarguments.
Shellflexdirectclient-l"172.
31.
51.
20;172.
31.
51.
26"[--]CPUServerCommandLineflexdirectclient-l172.
31.
51.
20:55002nvidia-smiGPUServerCommandLinenvidia-smi|NVIDIA-SMI375.
26DriverVersion:375.
26||GPUNamePersistence-M|Bus-IdDisp.
A|VolatileUncorr.
ECC||FanTempPerfPwr:Usage/Cap|Memory-Usage|GPU-UtilComputeM.
||0TeslaK80Off|0000:01:00.
0Off|N/A||N/A%53CP829W/149W|0MiB/11439MiB|0%Default||Processes:GPUMemory||GPUPIDTypeProcessnameUsage||Norunningprocessesfound|Youmayspecifyaportnumberwiththestandardcolonnotation:WHITEPAPER|5USERGPUGPUGPUGPUGPUGPUGPUGPUCOMPUTESERVERGPUSERVERGPUSERVERflexdirectclientflexdirectserverflexdirectserverConfiguringIPAddressesasPartofClientConfigurationIfyouwanttosimplifytheflexdirectclientcommand,youcanputyourBitfusionserverIPaddressesintothe/etc/bitfusionio/adaptor.
conffile.
Overridethedefaultportbyadding:.
CPUServerCommandLineCPUServerCommandLinecat/etc/bitfusionio/adaptor.
conf172.
31.
51.
20172.
31.
51.
26:57001flexdirectclientnvidia-smi|NVIDIA-SMI375.
26DriverVersion:375.
26||GPUNamePersistence-M|Bus-IdDisp.
A|VolatileUncorr.
ECC||FanTempPerfPwr:Usage/Cap|Memory-Usage|GPU-UtilComputeM.
||0TeslaK80Off|0000:01:00.
0Off|N/A||N/A%53CP829W/149W|0MiB/11479MiB|0%Default||Processes:GPUMemory||GPUPIDTypeProcessnameUsage||Norunningprocessesfound|Afterwritingadaptor.
conf,simplyrunflexdirectclientwithaGPUapplication.
Forexample,ifyourunflexdirectclientwithnvidia-smiitwilllisttheGPUsconfigured.
Typeflexdirecthelporflexdirecthelp[command]foradditionalhelpfulcommandsandinformation.
WHITEPAPER|6SampleOutput$flexdirecthelpNAME:flexdirect-RunapplicationwithBitfusionFlexDirectUSAGE:flexdirect"application"flexdirect--[application]flexdirecthelp[command]Formoreinformation,systemrequirements,andadvancedusagepleasevisithttps://www-review.
vmware.
com/solutions/business-critical-apps/hardwareaccelerators-virtualization.
htmlCOMMANDS:init,iInitializeconfiguration.
Requiresrootpriviledges.
version,vDisplayfullFlexDirectversion.
localhealth,LHRunhealthcheckoncurrentnodeonly.
upgrade,UUpgradeversion.
Requiresrootpriviledges.
uninstallUninstallFlexDirect.
Requiresrootpriviledges.
deallocDeallocatelicensecertificate.
Requiresrootpriviledges.
crashreportSendcrashreporttoBitfusion.
licenseChecklicensestatus.
list_gpusListtheavailableGPUsinasharedpool.
help,hShowsalistofcommandsorhelpforonecommand.
ClientCommands:client,cRunapplication.
health,HRunhealthcheckonallspecifiedserversandcurrentnode.
request_gpusRequestGPUsfromasharedpool.
release_gpusReleaseGPUsbackintoasharedpool.
Optionsmustmatchapreviousrequest_gpuscommand.
runRequestGPUsfromasharedpool,runaclientcommand,thenreleasetheGPUs.
statsGatherstatsfromallservers.
smiDisplaysmi-likeinfoforallservers.
localRunaCUDAapplicationlocally.
net_perfGathernetworkperformancedatafromallSRSservers.
ServerCommands:server,sRunserver.
resource_schedulerRunFlexDirectresourcescheduler(SRS)onGPUserverEXAMPLES:$sudoflexdirectinit-l$flexdirectresource_scheduler--srs_port50001$flexdirectrun-n4--Herearesomeflexdirectexampleswithexplanatorycomments.
WHITEPAPER|7TextInitializeflexdirectlicensebeforethefirstrunofserveronasystem$sudoflexdirectinit-lRunaflexdirectserverwithdefaultport55001$flexdirectserverRunaflexdirectserverwithadifferentport$flexdirectserver-p55010Runanapplicationwithaserverrunninglocalwithdefaultport55001$flexdirectclient-l"localhost"Runanapplicationwithmultipleservers,localorremote$flexdirectclient-l"192.
168.
0.
2:55010;192.
168.
0.
6:51234"Runanapplicationwithserversspecifiedinoneofthedefaultconfigfiles(~/.
bitfusionio/adaptor.
confand/etc/bitfusionio/adaptor.
confinpriorityorder)$flexdirectclientRunanapplicationwithserversspecifiedinaconfigfile$flexdirectclient-fRunaserverwitharesourcescheduleronacustomport$flexdirectresource_scheduler--srs_port50001--port55010Runanapplicationwith4sharedGPUs$flexdirectrun-n4Runanapplicationwith2sharedGPUs,usinghalftheavailablememory,andacustomservers.
conf$flexdirectrun-n2-p0.
5-sservers.
confRunanapplicationwith4sharedGPUswithInfiniBand$flexdirectrun-n4Runanapplicationlocally,restrictedtoonlyhalfthephysicalGPUmemory$flexdirectlocal-p0.
5Request8remoteGPUs$flexdirectrequest_gpus-sservers.
conf-fadaptor_8gpu.
conf-n8Runanapplicationwiththegeneratedconfigfile$flexdirectclient-fadaptor_8gpu.
confReleasethe8remoteGPUsaftertheapplicationhasfinished$flexdirectrelease_gpus-fadaptor_8gpu.
confGethelponaspecificcommand(theclientcommandinthisexample)$flexdirecthelpclientAdvanced:ClusterCommunicationsIfyouareunabletoopenupthedefault45201-46225portrangeforin-clustercommunication,youcanoverridethisrangebyexportingtheseenvironmentvariablesonyourGPUserversbeforerunningtheFlexDirectServer(alsocalledDispatcher):GPUServerCommandLine$exportBF_SERVER_PORT_MIN=$exportBF_SERVER_PORT_MAX=Advanced:FlexibleDynamicGPUConfigurationsTheexamplesbelowassumethatyouhaveafour-GPUserveratIPaddress123.
45.
67.
890.
WewillusethisoneGPUnodeforthreedifferentclientapplicationswithslightlydifferentresourceconfigurations,allsharingthesameGPUnode.
WHITEPAPER|8NOTENotehowasweprogressthroughtheexamples,weusedifferentportssothateachserverprocessisutilizinguniqueportsforcommunication.
BF_VISIBLE_DEVICESreferstotheIDnumberofeachGPUdevice,whichstartsat0.
Ifyouhavea4GPUinstance,theIDswouldbe0,1,2,and3respectively.
YoucanseethedevicesandtheirspecificIDsbyrunningnvidia-smi.
ExposingOneGPUoutoftheFourGPUsforAppsStarttheFlexDirectServer(alsocalledDispatcher)onthefirstGPUdevice(outofthefourweareassumingfortheseexamples)withthefollowingcommand:NowruntheFlexDirectClientonyourCPUnode.
Inthisexample,we'lldoitwithapplication"nvidia-smi",butyoucouldreplacethiswiththeapplicationyouwouldliketorunusingFlexDirectvirtualization.
ExposingTwoGPUsoutoftheFourGPUsforAppsStarttheFlexDirectServer(alsocalledDispatcher)onthefour-GPUnodewiththefollowingcommand:GPUServerCommandLineGPUServerCommandLineGPUServerCommandLineBF_VISIBLE_DEVICES=0flexdirectserver-p55001Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55001flexdirectclient-l123.
45.
67.
89:55001nvidia-smi|NVIDIA-SMI375.
26DriverVersion:375.
26||GPUNamePersistence-M|Bus-IdDisp.
A|VolatileUncorr.
ECC||FanTempPerfPwr:Usage/Cap|Memory-Usage|GPU-UtilComputeM.
||0TeslaK80Off|0000:01:00.
0Off|N/A||N/A%53CP829W/149W|0MiB/11479MiB|0%Default||Processes:GPUMemory||GPUPIDTypeProcessnameUsage||Norunningprocessesfound|BF_VISIBLE_DEVICES=0,1flexdirectserver-p55002WHITEPAPER|9RuntheFlexDirectClient.
Inthisexample,we'lldoitwithapplication"nvidia-smi:,butyoucouldreplacethiswiththeapplicationyouwouldliketorunusingFlexDirectvirtualization.
GPUServerCommandLineflexdirectclient-l123.
45.
67.
89:55002nvidia-smi|NVIDIA-SMI375.
26DriverVersion:375.
26||GPUNamePersistence-M|Bus-IdDisp.
A|VolatileUncorr.
ECC||FanTempPerfPwr:Usage/Cap|Memory-Usage|GPU-UtilComputeM.
||0TeslaK80Off|0000:01:00.
0Off|N/A||N/A%53CP829W/149W|0MiB/11479MiB|0%Default||||0TeslaK80Off|0000:01:00.
0Off|N/A||N/A%53CP829W/149W|0MiB/11479MiB|0%Default||Processes:GPUMemory||GPUPIDTypeProcessnameUsage||Norunningprocessesfound|PartialGPUConfigurations1/2-GPUavailableonport55001ThisisdonebysettingenvironmentalvariableBF_GPU_DEVICE_MEMORY_LIMITtohalfoftheGPUsmemory.
NVIDIAGPUSETTINGTOALLOWSHARINGWhenyoupartitionaGPU,presumablyyouwanttobeabletousebothpartitionssimultaneously.
NVIDIAGPUshaveacomputemodethatshouldbesetto"Default"(not"Exclusive")sothatmultipleapplicationscanshareaccess.
Usethenvidia-smi-acommandtoseethecurrentcomputemodesetting.
Andsetthemodeto"Default"withthecommandsudonvidia-smi-c0.
Server-sidecommandsshown,seeaboveonhowtoinvoketheclient.
GPUServerCommandLineBF_VISIBLE_DEVICES=0BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55001Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55001WHITEPAPER|10TwoGPUsAvailableonPort5500121/2-GPUsAvailableonPort55001Fortwohalf-sizedGPUs:161/2GPUsAssignedtoTwoDifferentClients(Acrosstwofour-GPUnodes).
EachclientseeseightpartialGPUs.
Usetwodifferentportnumbers,oneforeachclient.
Commentsareinterlacedwithcommands:GPUServerCommandLineGPUServerCommandLineBF_VISIBLE_DEVICES=0,1flexdirectserver-p55001Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55001BF_VISIBLE_DEVICES=0,1flexdirectserver-p55001Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55001GPUServerCommandLines#server1:$BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55001&$BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55002#server2:$BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55001&$BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55002SampleOutput#server1:Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55001Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55002#server2:Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55001Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55001ConfigurationFiles#client1adaptor.
conf:5500155001#client2adaptor.
conf:5500255002WHITEPAPER|11161/2GPUs(acrosstwofour-GPUnodes)available.
TwodifferentclientseachallocateonepartialGPU.
GPUServerCommandLines#Server1:BF_VISIBLE_DEVICES=0BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55001&BF_VISIBLE_DEVICES=1BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55002&BF_VISIBLE_DEVICES=2BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55003&BF_VISIBLE_DEVICES=3BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55004&BF_VISIBLE_DEVICES=4BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55005&BF_VISIBLE_DEVICES=5BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55006&BF_VISIBLE_DEVICES=6BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55007&BF_VISIBLE_DEVICES=7BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55008&#Server2:BF_VISIBLE_DEVICES=0BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55001&BF_VISIBLE_DEVICES=1BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55002&BF_VISIBLE_DEVICES=2BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55003&BF_VISIBLE_DEVICES=3BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55004&BF_VISIBLE_DEVICES=4BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55005&BF_VISIBLE_DEVICES=5BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55006&BF_VISIBLE_DEVICES=6BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55007&BF_VISIBLE_DEVICES=7BF_GPU_DEVICE_MEMORY_LIMIT=6291456000flexdirectserver-p55008&SampleOutput#Server1:Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55001Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55002Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55003Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55004Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55005Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55006Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55007Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55008#Server2:Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55001Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55002Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55003Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55004Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55005Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55006Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55007Dispatcherlistening.
.
.
Listeningon0.
0.
0.
0:55008SampleOutput#Client1adaptor.
conf(firstpartialGPUofserver1):55001#Client2adaptor.
conf(secondpartialGPUofserver1):55002VMware,Inc.
3401HillviewAvenuePaloAltoCA94304USATel877-486-9273Fax650-427-5001vmware.
comCopyright2019VMware,Inc.
Allrightsreserved.
ThisproductisprotectedbyU.
S.
andinternationalcopyrightandintellectualpropertylaws.
VMwareproductsarecoveredbyoneormorepatentslistedatvmware.
com/go/patents.
VMwareisaregisteredtrademarkortrademarkofVMware,Inc.
anditssubsidiariesintheUnitedStatesandotherjurisdictions.
Allothermarksandnamesmentionedhereinmaybetrademarksoftheirrespectivecompanies.
ItemNo:VMW-0518-1843_VMW_CPBUTechnicalWhitePapers_BitfusionDocs_08IntegrationandAdvancedUsage_1.
5_YC8/19
- sequentiallyyc8相关文档
- Statisticsyc8
- stackyc8
- 系统一级医院系统集成专业方案Com
- 调度使用COM—Express Nano工控板实现IP调度设备
- 接口基于COM技术的开放式数控软件系统的开发与研究
- 共产主义com开头词汇
hostkvm：美国VPS，三网强制CU-VIP线路，$5/月，1G内存/1核/15gSSD/500g流量
hostkvm在2021年3月新上线洛杉矶新VPS业务，强制三网接入中国联通优化线路，是当前中美之间性价比最高、最火热的线路之一，性价比高、速度非常好，接近联通AS9929和电信AS4809的效果，带宽充裕，晚高峰也不爆炸。 官方网站：https://hostkvm.com 全场优惠码：2021（全场通用八折，终身码，长期） 美国 US-Plan0【三网联通优化线路】 内存：1G CPU：...
PhotonVPS：$4/月,KVM-2GB/30GB/2TB/洛杉矶&达拉斯&芝加哥等
很久没有分享PhotonVPS的消息，最近看到商家VPS主机套餐有一些更新所以分享下。这是一家成立于2008年的国外VPS服务商，Psychz机房旗下的站点，主要提供VPS和独立服务器等，数据中心包括美国洛杉矶、达拉斯、芝加哥、阿什本等。目前，商家针对Cloud VPS提供8折优惠码，优惠后最低2G内存套餐每月4美元起。下面列出几款主机配置信息。CPU：1core内存：2GB硬盘：30GB NVm...
PIGYun中秋特惠：香港/韩国VPS月付14元起
PIGYun发布了九月份及中秋节特惠活动，提供8折优惠码，本月商家主推中国香港和韩国机房，优惠后最低韩国每月14元/中国香港每月19元起。这是一家成立于2019年的国人商家，提供中国香港、韩国和美国等地区机房VPS主机，基于KVM架构，采用SSD硬盘，CN2+BGP线路(美国为CUVIP-AS9929、GIA等)。下面列出两款主机配置信息。机房：中国香港CPU：1core内存：1GB硬盘：10GB...
WWW YC8 COM为你推荐
-
云播怎么看片云播看不了视频邮箱打不开怎么办我的邮箱打不开怎么办安装迅雷看看播放器如何用手机安装迅雷看看播放器云挂机快手极速版后台云挂机辅?助各位用了吗？在哪找的？什么是云平台什么是云系统？宽带接入服务器用wifi连不上服务器怎么办网站地图制作网站地图怎么做网页打不开的原因为什么我的有些网页打不开呢？手工杀毒手工杀毒与杀毒软件杀毒有什么区别？液晶显示器电源维修液晶显示器 电源板损坏 如何检修？ 好的加分！