sufficientopteron

MellanoxTechnologiesInc.
2900StenderWay,SantaClara,CA95054Tel:408-970-3400Fax:408-970-3403http://www.
mellanox.
comRealApplicationPerformanceandBeyondWhitePaper:RealApplicationPerformanceandBeyond2006MellanoxTechnologiesInc.
2Scientists,engineersandanalystsinvirtuallyeveryfieldareturningtohighperformancecomputingtosolvetoday'svitalandcomplexproblems.
Simulationsareincreasinglyreplacingexpensivephysicaltesting,asmorecomplexenvironmentscanbemodeledandinsomecases,fullysimulated.
High-performancecomputingencompassesadvancedcomputationoverparallelprocessing,enablingfasterexecutionofhighlycomputeintensivetaskssuchasclimateresearch,molecularmodeling,physicalsimulations,cryptanalysis,geophysicalmodeling,automotiveandaerospacedesign,financialmodeling,dataminingandmore.
HPCclustershavebecomethemostcommonbuildingblocksforhigh-performancecomputing,notonlybecausetheyareaffordable,butbecausetheyprovidetheneededflexibilityanddeliversuperiorprice/performancecomparedtoproprietarysymmetricmultiprocessing(SMP)systems,withthesimplicityandvalueofindustrystandardcomputing.
MauiHighPerformanceComputingCenter1280servers,MellanoxInfiniBandinterconnect,42.
3TFlopsReal-worldapplicationperformancedependsontheperformanceofthevariouscluster'skeyelements–theprocessor,thememory,andtheinterconnect.
Theinterconnectcontrolsthedatatransferbetweenservers,andhasahighinfluenceontheCPUefficiencyandmemoryutilization.
Transportoffloadinterconnectarchitectures,unlikethe"on-loading"ones,eliminatetheneedofdealingwiththeprotocolprocessingwithintheCPUandthereforeincreasethenumberofcyclesavailableforcomputationaltasks.
IftheCPUisbusymovingdataandhandlingnetworkprotocolprocessing,itisunabletoperformcomputationalwork,andtheoverallproductivityofthesystemisseverelydegraded.
Thememorycopyoverheadincludestheresourcesrequiredtocopydatabuffersfromthenetworkdevicetothekernelmemoryandthenfromthekernelmemorytotheapplicationmemory.
Thisapproachrequiresmultiplememoryaccessesbeforethedataisplacedinitsfinaldestination.
Whileitisnotamajorproblemforsmalldatatransfers,itisabigproblemforlargerdatatransfers.
Thisiswheretheinterconnectzero-copycapabilitieseliminatesthememorybandwidthbottleneckwithoutinvolvingtheCPUinthenetworkdatatransfer.
WhitePaper:RealApplicationPerformanceandBeyond2006MellanoxTechnologiesInc.
3SandiaNationalLab4500servers,MellanoxInfiniBandinterconnect53TFlops,84.
66%LinpackefficiencyTheinterconnectbandwidthandlatencyhavetraditionallybeenusedastwometricsforassessingtheperformanceofthesystem'sinterconnectfabric.
However,thesetwometricsaretypicallynotsufficienttodeterminetheperformanceofrealworldapplications.
Typicalreal-worldapplicationssendmessagesrangingfrom64Byteto4Megabyteusingnotonlypoint-to-pointcommunicationbutadiversemixtureofcommunicationpatterns,includingcollectiveandreductionpatternsinthecaseofMPI.
Insomecases,interconnectvendorscreateartificialbenchmarks,suchasmessagerate,andapplybombasticmarketingsloganstothesebenchmarks–suchas"Hypermessaging".
Messagerateisyetanothersinglepointinthepoint-to-pointbandwidthgraph.
Ifthetraditionalinterconnectbandwidthindicatesthemaximumavailablebandwidth(singlepoint),messagerateindicatesthebandwidthformessagesizeofzeroor2bytes.
Thesinglepointsofdata,givesomeindicationfortheinterconnectperformance,butarefarfromdescribingtherealworldapplicationperformance.
Theinteractivecombinationofthosepoints,togetherwithothers(CPUoverhead,zerocopyetc.
),willdeterminetheoverallabilityoftheconnectivitysolution.
Thedifferencebetweentheoreticalpowerandwhatisactuallydeliveredismeasuredasprocessorefficiency.
ThemoreCPUcyclesusedtogetthedataoutthedoorby"fillingthewire"duetoprotocolanddatatransferinefficiencies,thelesscyclesareavailablefortheapplication.
Whencomparinglatenciesofdifferentinterconnects,oneneedstopayattentiontotheinterconnectarchitecture.
1useclatency"on-loading"interconnectversus2useclatency"off-load"solutionissimilartoacasewhenoneneedstodecidebetweentwocarsthatshowthesamehorsepower(i.
e.
CPU).
Bothenginesarecapableof200milesperhour,butthefirstcar,dueto"on-loading",limitstheactualenginepowerto75milesperhour(theenginepowermustbeusedforothertasks).
TheSecondcarhasnolimitationsontheengine,butitswheelscantolerateonly150milesWhitePaper:RealApplicationPerformanceandBeyond2006MellanoxTechnologiesInc.
4perhour.
Theknowledgeonthewheelstolerance(i.
e.
latency),asasinglepointofdata,isdefinitelymisleading.
Thereareattemptstoproviderealworldapplicationperformancewhilecomparingdifferentinterconnects,butinmostcasesthe"comparison"isbiasedandbyusingdifferentsystemsand/orconditions,whichmakesatruecomparisondifficult.
Therehavebeenrecentcasescomparing10-GigabitEthernettoInfiniBand.
WhileInfiniBandadaptersweretestedwithPCIex4(thatislimitedto~700MByte/secbandwidth(duetolimitationsincurrentavailablesystems),the10GigabitEthernetcardswerePCI-X,thatiscapabletohigherbandwidth(~850-900MByte/s).
OthercasescompareInfiniBandPCIex4tootherinterconnectswithPCIex8hostinterface(theonlyvalidconclusiononecanmakeisthatPCIex8hasmorelanesthanPCIex4).
AnotherpapercomparedQLogicInfiniPathonIntel3GHzCPUbasedsystemtoMellanoxInfiniBandon2.
2GHzOpteronbasedsystem.
Anyattempttocomparedifferentinterconnectsinthosemannersisdeceptive.
RealapplicationperformanceInfiniBandisaproveninterconnectforclusteredserversolutions,andoneoftheleadingconnectivitysolutionforhigh-performancecomputing.
InfiniBandwasdesignedasageneralI/Oandinpracticeprovideslow-latencyandthehighestlinkspeed.
ComputationalFluidDynamics(CFD)isoneofthebranchesoffluidmechanicsthatusesnumericalmethodsandalgorithmstosolveandanalyzeproblemsthatinvolvefluidflows.
ANSYS/FLUENTisaleadingcommercialsoftwareproviderforsolvingfluidflowproblems.
ThebroadphysicalmodelingcapabilitiesofFLUENThavebeenappliedtoindustrialapplicationsrangingfromairflowoveranaircraftwingtocombustioninafurnace,frombubblecolumnstoglassproduction,frombloodflowtosemiconductormanufacturing,fromcleanroomdesigntowastewatertreatmentplants.
Theabilityofthesoftwaretomodelin-cylinderengines,aeroacoustics,turbomachinery,andmultiphasesystemshasservedtobroadenitsreach.
AtthecoreofanyCFDcalculationisacomputationalgrid,usedtodividethesolutiondomainintothousandsormillionsofelementswheretheproblemvariablesarecomputedandstored.
InFLUENT,unstructuredgridtechnologyisused,whichmeansthatthegridcanconsistofelementsinavarietyofshapes:quadrilateralsandtrianglesfor2Dsimulations,andhexahedral,tetrahedral,prisms,andpyramidsfor3Dsimulations.
Theseelementsformaninterlockingnetworkthroughoutthevolumewherethefluidflowanalysistakesplace.
TheperformanceofaCFDcodedependsonseveralfactors,includingsizeandtopologyofthemesh,physicalmodels,numericsandparallelization,compilersandoptimization,inadditiontoperformancecharacteristicsofthehardwarewherethesimulationisperformed.
FLUENTprovidesasetofbenchmarkproblemswhichrepresenttypicalcurrentusageandcoveringawiderangeofmeshsizesandphysicalmodels.
Theproblemsselectedrepresentarangeofsimulationstypicalofthosewhichmightbefoundinindustry.
TheprincipalobjectiveofthisbenchmarksuiteistoprovidecomprehensiveandfaircomparativeinformationoftheperformanceofFLUENTonavailablehardwareplatforms.
ThefollowingchartscomparesMellanoxInfiniBandandQLogicInfiniPathinterconnectsonthesameplatform–dualcore,dualsocket,IntelXeon3GHz5100series(codenameWoodcrest)servers,usingFLUENTbenchmarks.
Whentestingrealworldapplications,theentirearchitecturemakesthedifference.
TheMellanoxarchitectureisafulltransport-offloadone,withhardwarecapabilitiesofRDMA,whileQLogicisafull"on-loading"architecture.
WhitePaper:RealApplicationPerformanceandBeyond2006MellanoxTechnologiesInc.
5InFluentFL5L3benchmark,aTurbulentflowofairthroughaductiscomputed.
Thecross-sectionalplanesoftheducttransitionfromacircleattheinlettoarectangleattheoutflowboundary.
TheReynolds-StressModelisusedforcomputingturbulence(numberofcells:9,792,512,celltypehexahedral,modelsRSMturbulence,solversegregatedimplicit).
FLUENTFL5L2benchmarkrepresentsthecomputationoftheexteriorflowfieldaroundasimplifiedmodelofapassengersedan.
ThesimulationgeometrywasusedfortheJapanExternalAerodynamicscompetition.
Aviscous-hybridgridwithprismaticcellsisusedtoadequatelyFluent6.
3,FL5L3case0200400600800100012001400160018002000020406080100120140CPUcoresRating(performance)QlogicMellanoxFluent6.
3,FL5L2case02000400060008000020406080CPUcoresRating(performance)QlogicMellanoxWhitePaper:RealApplicationPerformanceandBeyond2006MellanoxTechnologiesInc.
6modeltheboundarylayerregions(numberofcells3,618,080,celltypehybrid,modelsk-epsilonturbulence,solversegregatedimplicit).
ChoosingtherightinterconnectInbothcasesofFLUENTbenchmarks,MellanoxInfiniBandshowshigherperformanceandbettersuper-linearscalingcomparingtoQLogicInfiniPath.
FLUENT'sCFDapplicationisalatency-sensitiveapplication,andtheresultsshownherearegoodexamplesonhowpurelatencybenchmarkscanbemisleadingwhenchoosingtherightinterconnect.
Inordertodeterminethesystem'sperformance,oneshouldtakeintoconsiderationtheentireinterconnectarchitecture(suchasoff-loadingversuson-loading)andtheabilityofscaling,ratherthanjustsinglepointsofdata.
Inordertoprovidebetterapplicationssight,MellanoxhascreatedtheMellanoxClusterCenter.
TheMellanoxClusterCenteroffersanenvironmentfordeveloping,testing,benchmarkingandoptimizingproductsbasedonInfiniBandtechnology.
Thecenter,locatedinSantaClara,California,provideson-sitetechnicalsupportandenablessecuresessionsonsiteorremotely.
MoredetailscanbeachievedthroughMellanoxwebsite.