1.0ivybridge

LS-DYNAPerformanceBenchmarkandProfilingOctober20172NoteThefollowingresearchwasperformedundertheHPCAdvisoryCouncilactivities–Participatingvendors:LSTC,Huawei,Mellanox–Computeresource-HPCAdvisoryCouncilClusterCenterThefollowingwasdonetoprovidebestpractices–LS-DYNAperformanceoverview–UnderstandingLS-DYNAcommunicationpatterns–WaystoincreaseLS-DYNAproductivity–MPIlibrariescomparisonsFormoreinfopleasereferto–http://www.
lstc.
com–http://www.
huawei.
com–http://www.
mellanox.
com3LS-DYNALS-DYNA–Ageneralpurposestructuralandfluidanalysissimulationsoftwarepackagecapableofsimulatingcomplexrealworldproblems–DevelopedbytheLivermoreSoftwareTechnologyCorporation(LSTC)LS-DYNAusedby–Automobile–Aerospace–Construction–Military–Manufacturing–Bioengineering4ObjectivesThepresentedresearchwasdonetoprovidebestpractices–LS-DYNAperformancebenchmarkingMPILibraryperformancecomparisonInterconnectperformancecomparisonCompilerscomparisonOptimizationtuningThepresentedresultswilldemonstrate–Thescalabilityofthecomputeenvironment/application–Considerationsforhigherproductivityandefficiency5TestClusterConfigurationHuaweiFusionServerE9000withFusionServerCH121V516-node(640-core)"Skylake"cluster–Dual-Socket20-CoreIntelXeonGold6138@2.
00GHzCPUs–Memory:192GBmemory,DDR42666MHzRDIMMspernode–OS:RHEL7.
2,MLNX_OFED_LINUX-4.
1-1.
0.
2.
0InfiniBandSWstackMellanoxConnectX-5EDR100Gb/sInfiniBandAdaptersMellanoxSwitch-IBSB780036-portEDR100Gb/sInfiniBandSwitchCompilers:IntelParallelStudioXE2018MPI:IntelMPI2018,MellanoxHPC-XMPIToolkitv1.
9.
7,PlatformMPI9.
1.
4.
3Application:MPPLS-DYNAR9.
1.
0,build113698,singleprecisionMPIProfiler:IPM(fromMellanoxHPC-X)Benchmarks:TopCrunchbenchmarks–NeonRefinedRevised(neon_refined_revised),ThreeVehicleCollision(3cars),NCACMinivanModel(Caravan2m-ver10),odb10m(NCACTaurusmodel)6High-Performance2-SocketBladeUnlocksSupremeComputingPowerFull-seriesIntelXeonScalableProcessors,24DDR4DIMMs,AEPmemorysupported,1PCIeslot,2SFF/2NVMeSSDs/4M.
2SSDshigh-performancestorage,multi-planenetwork,LOMsupportedIntroducingHuaweiFusionServerE9000(CH121)V57LS-DYNAPerformance–CPUSKUsandGenerationLS-DYNAperformancegainbylargercorecountsandbettermemorythroughput–The"Gold6140"demonstratesa50%ofperformancegain(29%morecores)vsE5-2680v4–The"Gold6148"demonstratesa61%ofperformancegain(42%morecores)vsE5-2680v4–BaseclockarethesameonE5-2680v4andGold6148,whileGold6140runsslightlyslower–Skylakesupports6memorychannelsandfasterDIMMswhichimpactsonmemoryperformanceSingleNodePerformanceHigherisbetter61%50%8LS-DYNAPerformance–MemorySpeedMemoryspeedprovidessomebenefitstoLS-DYNAperformance–SkylakeplatformsupportsDIMMspeedupto2666MHzDIMMs–2666MHzDIMMsistheoretically~11%fasterthanthe2400MHzDIMMs–LS-DYNAreportsonlyabout~2-3%oftheimprovementonasinglenode–ItappearsonlypartofthespeeddifferenceistranslatedintoLS-DYNAperformancegain40MPIProcesses/NodeHigherisbetter9LS-DYNAPerformance–Sub-NUMAClusteringEnablingSNCprovidessomebenefitsforLS-DYNA–Sub-NUMAClustering(SNC)issimilartoacluster-on-die(COD)inHaswell/Broadwellgeneration–CPUcoresandmemorywouldbesplitinto2separateNUMAdomainswhenSNCisenabled–SNCgenerallyshoulddemonstratesomebenefitsforapplicationsthatrequiresgoodNUMAlocality–SNCdemonstratesaperformancegainof~2-3%onasinglenodebasis40MPIProcesses/NodeHigherisbetter10LS-DYNAPerformance–CPUInstructionsAVX2outperformsbothAVX-512andSSE2executablesonSkylakeCPU–Performancegainof17%byusingAVX2overAVX-512executables–AVX-512performsworsecomparedtoAVX2,despiteimprovedvectorization–AVX-512instructionsrunsatareducedclockfrequencyasAVX2andnormalclocks–BenefitofAVX2appearstobelargeronbiggerdataset(suchascar2car)40MPIProcesses/NodeHigherisbetter17%8%4%3%11LS-DYNAPerformance–CPUInstructionSetsSomevarianceinperformanceamongdifferentLS-DYNAversions/executables–AVX2performsbetterthanSSE2LS-DYNAexecutables–SmallvarianceinperformanceamongdifferentLS-DYNAreleases–R7.
1.
3appearedtoperformbetteronlargerdatasets40MPIProcesses/NodeHigherisbetter20%12LS-DYNAPerformance–MPILibrariesAllthreeMPIimplementationsshowsdecentperformanceatscale–PlatformMPIandHPC-Xperformssimilarly,whileIntelMPIshowsadropatsmalldatasetatscale40MPIProcesses/NodeHigherisbetter13LS-DYNAPerformance–SystemGenerationsCurrentSkylakesystemconfigurationoutperformspriorsystemgenerations–SkylakeplatformoutperformedBroadwellby21%,Haswellby51%,IvyBridgeby89%,SandyBridgeby132%,Westmereby222%,Nehalemby425%–Skylakeperforms41%betterthanBroadwellforthe3carsmodelonasingle-nodebasis–Systemcomponentsused:Skylake:2-socket20-coreXeonGold61382.
0GHz,2666MHzDIMMs,ConnectX-5EDRInfiniBandBroadwell:2-socket14-coreXeonE5-2690v42.
6GHz,2400MHzDIMMs,ConnectX-4EDRInfiniBandHaswell:2-socket14-coreXeonE5-2697v32.
6GHz,2133MHzDIMMs,ConnectX-4EDRInfiniBandIvyBridge:2-socket10-coreXeonE5-2680v22.
8GHz,1600MHzDIMMs,Connect-IBFDRInfiniBandSandyBridge:2-socket8-coreXeonE5-26802.
7GHz,1600MHzDIMMs,ConnectX-3FDRInfiniBandWestmere:2-socket6-coreXeonx56702.
93GHz,1333MHzDIMMs,ConnectX-2QDRInfiniBandNehalem:2-socket4-coreXeonx55702.
93GHz,1333MHzDIMMs,ConnectX-2QDRInfiniBandBestresultsshownHigherisbetter41%14LS-DYNASummaryLS-DYNAismulti-purposeexplicitandimplicitfiniteelementprogram–Utilizesbothcompute,memoryandnetworkcommunicationsforperformanceEffectofMPIonperformance–PlatformMPIandHPC-Xperformssimilarly,IntelMPIshowsadropatsmalldatasetEffectofSkylakegenerationonperformance–Providessubstantialperformancegainduetothelargercorecount,supportformemorychannels–Faster2666MHzDIMM(comparesto2400MHz)translatestoincrease2-3%inhigherperformanceEffortofCPUInstructionsonperformance–AVX-512performsworsecomparedtoAVX2,despitetheimprovedvectorization–AVX-512instructionsrunsatareducedclockfrequencyasAVX2andnormalclocksEffectofSNConperformance–EnablingSub-NUMAClusteringprovidessmalladvantage(~2-3%)onsinglenodeEffectfoLS-DYNAversiononperformance–SmallvarianceinperformanceamongdifferentLS-DYNAreleases;bestappearedtobeR7.
1.
31515ThankYouHPCAdvisoryCouncilAlltrademarksarepropertyoftheirrespectiveowners.
Allinformationisprovided"As-Is"withoutanykindofwarranty.
TheHPCAdvisoryCouncilmakesnorepresentationtotheaccuracyandcompletenessoftheinformationcontainedherein.
HPCAdvisoryCouncilundertakesnodutyandassumesnoobligationtoupdateorcorrectanyinformationpresentedherein