Statisticsyc8

SystemRequirementsBitfusionGuideWHITEPAPER–OCTOBER2019WHITEPAPER|2VMwareBitfusion:SystemRequirementsTableofContentsSystemRequirements3HardwareRequirements3Networking4VerifyingSystemHealth5SoftwareDependencies6Ubuntu6CentOSandRedHat7ListCUDALibraries.
7VerifytheCUDAInstallation7CUDAInstallationonCPUNodes/VMs/Containers7WHITEPAPER|3SystemRequirementsFlexDirectcanbeinstalledfordifferentmodesofoperation.
Somemodesinclude,inwholeorinpart,theabilitiesofothermodes.
Thisonedocumentliststherequirementsforeachmode.
FLEXDIRECTCLIENTSFLEXDIRECTANALYTICSFLEXDIRECTSERVERFLEXDIRECTMANAGERUbuntuLTS16.
04CentOS7orRHEL7.
4+NVIDIAToolkitCUDA7.
5CUDA8CUDA9CUDA10UbuntuLTS16.
04CentOS7orRHEL7.
4+UbuntuLTS16.
04CentOS7orRHEL7.
4+UbuntuLTS16.
04CentOS7orRHEL7.
4+NVIDIAGPUdriverversion372orhigherNVIDIAGPUdriverversion372orhigherNVIDIAGPUdriverversion372orhigherNote:CanalsorunApplicationsunderFlexDirectClient,inwhichcaseNVIDIAToolkitprerequisitesareneeded.
Note:CanalsorunApplicationsunderFlexDirectClient,inwhichcaseNVIDIAToolkitprerequisitesareneeded.
FLEXDIRECTCLIENTSFLEXDIRECTANALYTICSFLEXDIRECTSERVERFLEXDIRECTMANAGERN/AAnygenerationCUDA-enabledNVIDIAGPU(s)AnygenerationCUDA-enabledNVIDIAGPU(s)AnygenerationCUDA-enabledNVIDIAGPU(s)10GbsormoreEthernet(TCP/IP)RoCEorInfinibandNetworkConnectivity10GbsormoreEthernet(TCP/IP)RoCEorInfiniband10GbsormoreEthernet(TCP/IP)RoCEorInfinibandSystemmemoryontheclientCPUmachinesshouldbe1.
5xthetotalaggregateGPUmemoryacrossalltheGPUsinthelargestGPUmachine.
N/AN/AN/AHardwareRequirementsWHITEPAPER|4NetworkingFlexDirectmakesuseofseveralportsorrangesofportsforprocesstoprocesscommunication.
Pleaseensurethatyourfirewallsdonotblocktheportsbelow.
Pleaseensuretherearenoportconflictswithotherapplications.
Someoftherangescanbemodifiedwithcommand-linearguments.
FLEXDIRECTCLIENTSFLEXDIRECTANALYTICSFLEXDIRECTSERVERFLEXDIRECTMANAGER56001ServerCommunicationOutboundto:–Servernodes–Managernodes56001ServerCommunicationInboundfrom:–Analyticsnodes–ManagernodesOutboundto:–Analyticsnodes–Managernodes56001ServerCommunicationInboundfrom:–Clientnodes56001ServerCommunicationInboundfrom:–Clientnodes–Analyticsnodes–ManagernodesOutboundto:–Analyticsnodes–ManagernodesNote:Overridewith--srs_port56008,forexample.
Note:Overridewith--srs_port56008,forexample.
Note:Overridewith--srs_port56008,forexample.
55001-55100DispatcherCommunicationOutboundto:–Servernodes–Managernodes55001-55100DispatcherCommunicationN/A55001-55100DispatcherCommunicationInboundfrom:–Clientnodes55001-55100DispatcherCommunicationInboundfrom:–Clientnodes45201-46225CUDACommunicationOutboundto:–Servernodes–Managernodes45201-46225CUDACommunicationN/A45201-46225CUDACommunicationInboundfrom:–Clientnodes45201-46225CUDACommunicationInboundfrom:–Clientnodes54000WebServerN/A54000WebServerInboundfrom:–Anyonewithabrowser54000WebServerN/A54000WebServerInboundfrom:–AnyonewithabrowserNote:Overridewith--web_port12345,forexample.
Note:Overridewith--web_port12345,forexample.
WHITEPAPER|5FLEXDIRECTCLIENTSFLEXDIRECTANALYTICSFLEXDIRECTSERVERFLEXDIRECTMANAGER54001StatisticsCollectionN/A54001StatisticsCollectionInbound/Outbound–localhostonly54001StatisticsCollectionN/A54001StatisticsCollectionInbound/Outbound–localhostonlyNote:Overridewith--collection_port54321,forexample.
Note:Overridewith--collection_port54321,forexample.
Ensureyourmachineshaveingress/egressinternetaccessforaccesstodownloads,licensing,etc.
Checkforothernetworkingpolicies,suchasoutboundproxyorinternalDNS,staticversusdynamicIPs.
ConsultwiththeBitfusionteamifyouareusingdynamicIPsorhaveanyproxiessetup.
VerifyingSystemHealthOnceyouhaveinstalledFlexDirect,youcanvalidateyourenvironmentforthebestresultsbyrunningtheFlexDirecthealthcheck.
Shell#Assumesflexdirectisinstalled;seeinstallationguide.
flexdirecthealthThehealthcheckwillrunchecksonthenodesofyourclusterappropriatetotheirhardware(e.
g.
,GPUs)ortotheirconfiguration(e.
g.
RoCE).
Soyouroutputmaydifferfromthatshownbelow.
Checkstrytofindsettingsorproblemsthatwilllimitorpreventthehigh-bandwidth,low-latencycommunicationneededforthebestperformance.
Theresultsshouldbeselfexplanatory.
CheckswillbeperformedonthelocalhostandonallconfiguredGPUservers.
Shownbelowistheoutputfromthecheckonthelocalnode.
WHITEPAPER|6Output$flexdirecthealthHealthreportforserver:192.
168.
10.
41:56001SoftwareVersionschecks:[PASS]Checklibrarydependency[PASS]CheckOFEDVersion:4.
3[PASS]CheckCUDAversion>=7050:Currentversion:9020[PASS]CheckGPUdriverversion>=367.
0:Currentversion:396.
37SystemResourceschecks:[PASS]Checkflexdirectinstall[PASS]CheckexternalconnectivitytoBitfusionlicenseserver.
Ignoreforon-premiseinstallations.
[PASS]Checkshadowmemory192076MBandtotalgpumem64640MBPerformancechecks:[PASS]CheckMTUSize:hi-speedinterfacesMTU>=4K[PASS]Checkmemops[PASS]Checkulimit-n>=4096:4096[MARGINAL]Checkmultinodesupport:GPUdirectnotsupported[MARGINAL]Checknv_peer_mem:nv_peer_memmodulenotloaded[SKIPPED]CannotperformPCIediagnosiswithoutrootaccess.
RunthebinarywithsudotogetdiagnosisStabilitychecks:[PASS]CheckNetworkErrors/Drops:foundnoerrorsorpacketdrops[PASS]CheckIBphysicallyup[PASS]CheckIBSMup[PASS]CheckMADagentregistrationerrors[PASS]CheckGPUAPImismatch[PASS]CheckGPUXiderrors[PASS]Checktemperaturecom/compute/cuda/repos/ubuntu1604/x86_64/cuda-repo-ubuntu1604_9.
1.
85-1_amd64.
deb$sudodpkg-icuda-repo-ubuntu1604_9.
1.
85-1_amd64.
deb$sudoapt-keyadv--fetch-keyshttp://developer.
download.
nvidia.
com/compute/cuda/repos/ubuntu1604/x86_64/7fa2af80.
pub$sudoapt-getupdate$sudoapt-getinstall-ycuda-toolkit-9-1$rmcuda-repo-ubuntu1604_9.
1.
85-1_amd64.
debListCUDALibrariesCUDAlibsmightbemissingifyourinstallationisaminimalone.
Copy/pastethefilesfrom/usr/local/cuda/lib64:VerifytheCUDAinstallationTheeasiestwaytodothiswillbetoinstalltheCUDAsamples.
HereistheRPMforCUDAsamples.
ItistheRPMforRHEL7andCUDA9.
1.
Afterinstallingthesamples,makesuredeviceQuerycanberuneitherdirectly(ifitisaGPUmachine)orwithFlexDirect(ifitisaClientCPUmachine)ScheduleatestjobusingCUDAdeviceQuery:CUDAInstallationonCPUNodes/VMs/ContainersInordertorunGPUapplicationswithFlexDirectonCPUnodes,VMs,orincontainerswhichdonothavedirectaccesstophysicalGPUhardware,youstillneedtoinstallCUDA—butthereisnoneedtoinstallanyNvidiadrivercomponents.
TheexamplebelowillustrateshowtotoinstallCUDA9.
1inafeweasysteps.
TheCUDAversionthatyouinstallontheClientshouldmatchtheCUDAversionwhichisrunningontheserver.
VMware,Inc.
3401HillviewAvenuePaloAltoCA94304USATel877-486-9273Fax650-427-5001vmware.
comCopyright2019VMware,Inc.
Allrightsreserved.
ThisproductisprotectedbyU.
S.
andinternationalcopyrightandintellectualpropertylaws.
VMwareproductsarecoveredbyoneormorepatentslistedatvmware.
com/go/patents.
VMwareisaregisteredtrademarkortrademarkofVMware,Inc.
anditssubsidiariesintheUnitedStatesandotherjurisdictions.
Allothermarksandnamesmentionedhereinmaybetrademarksoftheirrespectivecompanies.
ItemNo:VMW-0518-1843_VMW_CPBUTechnicalWhitePapers_BitfusionDocs_05SystemRequirements_1.
4_YC8/19