clusteringdl380g7

ORIGINALRESEARCHAnintelligentenergyoptimizationapproachforMPIbasedapplicationsinHPCsystemsBhavyasreeUnniNaziaParveenAnkitKumarB.
S.
BindhumadhavaReceived:4November2012/Accepted:26March2013/Publishedonline:16April2013CSIPublications2013AbstractEnergy-awarecomputingisgainingmoreandmoreattentioninhighperformancecomputing(HPC)environment.
Asanoutcomeofthis,variousenergy-awaretechniquesareexistingandmanyarebeingproposed.
Butitisdifculttohaveatechniquewhichsavesenergywithoutcompromisingtheperformance.
ThispapertalksaboutanovelenergyoptimizationapproachforMessagePassingInterface(MPI)applicationsrunningonHPCsystems.
OurapproachreliesonapplyingDynamicVoltageFrequencyScaling(DVFS)atnodelevelbyanoptimiza-tionagent.
WheneverMPIprocessesareidleorbusywithI/Ooperations,thecorrespondingCPUcoresrunathigherfrequencies,whichresultsinwastageofpower.
Duringthistime,CPUcoresfrequenciescanbereducedusingDVFSsothattheenergycanbesaved.
OurapproachisbasedonaMulti-agentbasedintelligentenergymanagementframe-work,whichusesanoptimizationagentforimplementingenergyoptimizationalgorithm.
Thekeyadvantageoftheproposedapproachisthattheperformancewillnotbecompromisedwhileachievingenergysavings.
KeywordsHPCEnergy-awarecomputingMPIDVFSMulti-agentsystemAutonomiccomputing1IntroductionEnhancingtheperformancewasthekeyconcernintheareaofhighperformancecomputing(HPC)duringpastyears,whereasenergymanagementwasinsecondplace.
ButnowthescenariohasreversedandtheenergymanagementhasemergedasthemostconsiderableaspectinHPCworld.
HPCsystemsconsumepowerinseveralmegawatts[1]andthishighpowerconsumptionmayleadtoproblemslikereducedreliability,increasedcost,lessstabilityetc.
Hencereducingpowerconsumptionforhighendcomputingbecomesacrucialissueatbothsystemlevelandapplica-tionlevel.
Generally,HPCsystemsareofdistributedmemoryarchitectureandMPIstandard[2]isoneofthemostcommonlyusedparallelprogrammingparadigminthistypeofmemoryarchitecture.
OurapproachisbasedonapplyingDVFSatnodelevelbyourenergyoptimizationtool,whichdependsonfollowingtwoconditions.
Firstoneis,wheneverthemasterprocessisexecutingandtheslaveprocessesareinwaitingstate.
Secondoneis,whenallprocessesarecarryingouttheI/Ooperations.
Inboththeconditions,thecorrespondingCPUcoresrunathigherfrequenciesthatsimplywastestheCPUcycleswhichinturn,resultsininefcientuseofpower.
Hence,ifwecanreducethepowerconsumptionduringthistime,thispowerwastagecanbeminimized.
Ourapproachusesthisprincipletominimisethepowerwastage.
Dynamicpowerconsumptionofprocessorispropor-tionaltotheproductofsquareofvoltageandfrequency[3].
Duringtheidleperiod,thedynamicpowerconsumptioncanbereducedbyusingDVFStechnique.
Themajorissues,whenthefrequencyofprocessoristobevariedandonwhichnodesarebeingaddressedbyourenergyB.
Unni(&)N.
ParveenA.
KumarB.
S.
BindhumadhavaRealTimeSystemsandSmartGridGroup,CentreforDevelopmentofAdvancedComputing,C-DACKnowledgepark,Bangalore,Indiae-mail:bhavyasreeu@cdac.
inN.
Parveene-mail:naziap@cdac.
inA.
Kumare-mail:ankitk@cdac.
inB.
S.
Bindhumadhavae-mail:bindhu@cdac.
in123CSIT(June2013)1(2):175–181DOI10.
1007/s40012-013-0012-6optimizationalgorithm.
Wehaveevaluatedouroptimiza-tionalgorithmusingIntelMPIBenchmarks(IMB)[4]andapseudocodewhichhasbeendevelopedbyus.
TheproposedMulti-agentbasedautonomicframeworkiscomposedofautonomiccomponents(agents)interactingwitheachother.
Anautonomiccomputingsystem[5]makesdecisionsonitsownandconstantlychecksandoptimizesitsstatusautomatically,adaptingitselftothechangingconditions.
Ouroptimizationagentisself-opti-mizing,i.
e.
itwillmonitorthesystemcontinuouslyandoptimizesdependingonthesystemstatusautomatically.
Thisframeworkprovidesintelligencetoouroptimizationapproachsothathumaninterventioncanbeavoided.
Inthispaper,wehaveusedbothtechnologiessothatenergyisintelligentlymanagedusingDVFSbyoptimiza-tionagentthatwilltakethedecisionwhenandwheretoapplyDVFSonnodes.
Therestofthepaperisorganizedasfollows.
Section2reviewstherelatedworks.
InSect.
3,weexplainedtheenergyoptimizationproblemandthealgorithmtosolveit.
WehavepresentedtheintelligentenergymanagementframeworkforenergyoptimizationinSect.
4.
InSect.
5,experimentalanalysisisdiscussed.
Finally,Sect.
6describestheconclusionsandfutureworks.
2RelatedworksRecently,therearelotsofresearchesbeingcarriedoutintheeldofpoweroptimizationinHPCsystems.
Inthissection,wefocusonpoweroptimizationintheareaofMPIapplicationsMostofthenodelevelenergymanagementtechniquesarebasedonDVFStechniques[6–8],becauseCPUisthemostpowerconsumingcomponentwithinanode[9].
In[10],DVFStechniqueisappliedtothenodeswithlesscomputationsoastoreducethepower.
TherearesomeresearchworkswhicharebasedontheenergyefcienttaskallocationofMPIjobs.
Y.
Maetal.
[11]explainshowefcientlytaskclusteringwithtaskduplicationcanbedonetoreduceenergyconsumption.
Thepaper[12]discussesabouttaskaggregationtosaveenergy.
SeveralresearchesarebeingcarriedoutinreducingtheCPUfrequencyduringthecommunicationphaseofMPIprograms.
Dongetal.
[13]focusedonscalingdowntheCPUfrequencyduringtheMPIcollectiveoperations.
Chenetal.
[14]presentedanAutomaticEnergyStatusControllingwhichcancontrolCPUfrequencyautomaticallybasedonthecommunicationlatencyinthenodes.
OurpaperisalsoconcernedaboutapplyingDVFStechniquesforenergyoptimizationinMPIapplications,butitmainlyfocusesonthestateoftheprocessesunderexecution.
3ProblemstatementMPIisacommonparallelprogramminginterfacewhichdistributesthetaskamongmultipleprocessors.
Processorsexecutethesetasksandcommunicatewitheachotherbymessagepassing.
MPIisbasedondistributedmemorymodelwhereeveryprocesshasitsownmemoryspacewhichcannotbeaccessedbyotherprocesses.
Basically,twotypesofMPIprogrammingmodelsareavailable,i.
e.
SPMD(SingleProgramMultipleData)andMPMD(MultipleProgramMultipleData)orMaster-Slave.
Ourtechniqueisapplicableintwosituations.
FirstoneisbasedonMPMDmodel,inwhich,usuallywhenthemasterprocessisexecutingitstask,alltheslaveprocessesareinwaitingstate.
Duringthisexecutiontime,alloftheslaveprocessesareidleandarewastingtheCPUcycles.
Eventhoughtheyareinwaitingstate,processorsoperateathighfrequencies.
Sinceprocessorsrunathighfrequencies,thisleadstohigherpowerconsumption.
Ourideaistoreducetheprocessorsfrequenciesonslavenodesaslongasslaveprocessesareidle,sothatthepowerwastagecanbeminimized.
SecondsituationiswhenprocessescarryouttheI/Ooperation,allthenodesonwhichtheseprocessesrun,operateathigherfrequencies.
ThepercentageofCPUutilizationislowduringexecution.
Hence,ifwecanreducethepowerconsumptionduringthistime,powerwastagecanbeminimized.
3.
1EnergymodelAllthemodernprocessorsareenabledwithDVFStech-nique.
ThissectiondescribesDVFSenabledsystemmodelintermsofenergyconsumption.
DVFSenabledprocessorcanworkonsetsofdifferentvoltageandfrequencyasgivenin(1)and(2).
Vvi;wheremin\i\max1Ffi;wheremin\i\max2viistheithoperatingvoltage.
fiistheithoperatingfrequency.
Theenergyconsumptionofaprocessoristhesumofstaticenergyanddynamicenergyconsumptionandisgivenin(3).
Energyconsumptionintermsofstaticanddynamicpowerisshownin(4).
EEdynamicEstatic3EPdynamicPstatic:Dt4Accordingto[15],totalenergyconsumptionequationcanbemodiedas(5).
EACv2fvIleak:Dt5176CSIT(June2013)1(2):175–181123where,Aisthepercentageofactivegates,Cisthecapacitanceloadofallgates,vistheoperatingvoltage,fistheprocessorfrequency,Ileakistheleakagecurrent,Dtisthetimeduration.
Unlikedynamicpower,staticpowerisnotactivitybased.
Byreducingtheprocessorsfrequencieswhentheyareinidlestate,thestaticpowerconsumptioncannotbedecreased.
Ontheotherhand,shuttingofftheinactivepartofthesystemdoeshelp,butitresultsinlossofstate.
DVFSmaybeusedtoreducethedynamicpowerconsumptionbychangingtheCPUclockfrequency-voltagesettingwithoutaffectingtheexecutiontime.
3.
2EnergyoptimizationalgorithmInthissection,weintroduceanovelenergyoptimizationalgorithmforMPIapplicationsinHPCenvironment.
Thisalgorithmmakesuseoftwofactors,i.
e.
thedifferenceintheexecutiontimeofmasterandslaveprocessesandthetimetakentocompletetheI/Otask.
Theamountofenergythatcanbesaveddependsonthetypeofapplication.
Usually,inHPCenvironmentnodesaregenerallynotsharedamongdifferentapplicationsthatiswholenodeisutilizedbyasinglejob.
TheworkowofthisalgorithmisdepictedinFig.
1.
Firststepofthealgorithmistoidentifythenodeswhichhasbeenallocatedforaparticularapplicationwiththehelpofscheduler.
Thisinformationcanbetakenfromthescheduler.
Inthenextstep,thealgorithmwillverifywhe-theralltheprocessesarecarryingoutI/Ooperations.
Ifitistrue,thenfrequencyofallthenodeswillbereduced.
Otherwise,itwillcheckforthesecondcondition,i.
e.
whethermasterisexecutingwithslavesonwaiting.
Ifthisconditionissatised,thenthenodeonwhichthemasterisrunningisfoundbyinteractingwiththeapplication.
Next,thefrequenciesoftheslaveprocessorsarereducedwhentheyareinwaitingstatebyusingDVFS.
WheneverslavesstartrunningortheI/Ooperationsareover,thefrequenciesoftheprocessorswillbeincreasedbyDVFSinthelaststep.
Thisalgorithmwillcontinuewiththeabovestepsandwillbeterminatedwhentheapplicationnishes.
4IntelligentenergymanagementframeworkforenergyoptimizationWehavedesignedandimplementedanIntelligentEnergyManagementframeworkwhichisbasedonMulti-agentsupportasshowninFig.
2.
AMulti-agentframework[16]consistsofloosely-coupledcomputationalautonomousagentsthatcanperformactions.
Thesehaveresourcesattheirdisposalandtheypossessknowledge.
Theyaresitu-atedinacommonenvironmentandtheycancommunicatethroughinteractionprotocols.
WehaveusedC-DACMulti-agentFramework(CMAF)[17]toprovidethesupportforagentexecutioninourarchitecture.
Forthisarchitecture,wehaveusedahybrid(reactiveandmobile)typeagent.
Areactiveagentreceivesinput,processesitandproducesoutput.
Amobileagentisacompleteself-containedbodyofcode,whichphysicallymovesfromonecomputertoanother.
Beforemigrating,themobileagentstopsexecutionatthesourceandresumesexecutionafterreachingthedestination.
ThisframeworkmainlyconsistsoftheTargetSystem(TS),whichisHPCSystem'scomputenodesandIntelli-gentEnergyManager(IEM).
TheframeworkisdeployedonHPCsystemwhereIEMisdeployedatHeadNodeandFig.
1FlowchartofenergyoptimizationapproachCSIT(June2013)1(2):175–181177123eachcomputenode(whichworksasTS)hostsanoptimi-zationagentthatexecutestheenergyoptimizationalgo-rithm.
OurIEMcomprisesofthreeparts,i.
e.
JobManager,LauncherandOptimizationagent.
JobManagerinteractswiththeschedulertogetdetailsaboutthejobs,i.
e.
onwhichnodeseachjobhasbeenallocated.
Itcollectsandupdatestheinformationregardingeachjobfromtheschedulerperiodically.
Foreveryjob,itpassesthecorrespondinginformationtotheLauncher.
Basedonthisinformation,Launcherinitiatestheoptimizationagentsonappropriatenodes.
Italsopassesthecorrespondingparameterstoeachagent.
Atnodelevel,thisagentinteractswiththeapplica-tionandcarriesouttheoptimizationaccordingtothealgorithmmentionedinSect.
3.
2.
Theoptimizationagentisterminatedwiththeendofapplication.
5ExperimentalanalysisInthissection,weevaluatedtheenergysavingsobtainedwithourenergyoptimizationalgorithm.
Theperformanceofenergyoptimizationalgorithmvariesaccordingtothenatureofapplicationi.
e.
whetheritisCPUboundorI/Obound,durationofexecution,no.
ofprocessesandnodesetc.
WehavecarriedoutourexperimentswithMPI-I/ObenchmarksofIMBpackageandourpseudocode.
Thefollowingsubsectionsdescribethedetailsoftheexperi-mentationdone.
5.
1ExperimentalenvironmentOurexperimentalplatformisequippedwiththreeHPDL380G7servers,eachhavingtwoIntelXeonE5645processorswithsixcores.
ThesethreesystemsareclusteredusingPBSresourcemanagerandMauischedulerwhereoneactsasaheadnodeandtheothertwoascomputenodes.
EachCPUcorehasmaximumfrequencyof2.
4GHzandminimumfrequencyof1.
6GHz.
EachnodehasRHEL6.
2operatingsystemandusesMPICH2-1.
4.
1libraryforMPI[18].
OurMulti-agentframeworkisloadedintotheheadnodeandcomputenodes.
Powermeasurementhasbeendoneusing''Watt-sUp.
NET''powermeter.
Theenergyconsumptionisestimatedbyintegratingtheactualpowermeasuresovertime.
TheexperimentalsetupforpowermeasurementisshowninFig.
3.
5.
2ExperimentalresultsWehaveevaluatedtheenergyoptimizationalgorithmusingtwoexperimentsinhighperformancemode.
Wecarriedoutrstexperimentwiththepseudocode,whichisbasedonMPMDmodelofMPI.
ItisCPUintensiveanddoesmatrixmultiplication.
Fig.
2ArchitectureofintelligentenergymanagementframeworkFig.
3Experimentalsetupforpowermeasurement178CSIT(June2013)1(2):175–181123Wehaveexecutedthisprogramontwoserverswith24processesbyrunning12processesoneachserver.
Wehaveexecutedtheprogramunderfourdifferentconditions,i.
e.
allprocessorsoperatingatmax.
frequency(2.
4GHz),allprocessorsoperatingatmin.
frequency(1.
6GHz),allprocessorsatmax.
frequencyandwithourenergyoptimi-zationtechnique,andallprocessorsatmax.
frequencyandwithourenergyoptimizationtechniquecombinedwithvaryingvoltagelevels.
Westeppeddownthevoltagelevelofprocessorsatmaximumfrequencybyonestep.
Inourenergyoptimizationtechnique,wheneverthemasterpro-cessisexecutingandtheslaveprocessesareinwaitingstate,thefrequencyofthenodes,onwhichslavesarerunning,isreducedfrom2.
4to1.
6GHz.
Thefrequencyisincreasedbackto2.
4GHzwhentheslaveprocessesstarttoexecute.
ThepowerconsumptionduringalltheconditionsforthisexperimentisshowninFig.
4.
TheenergyconsumptionandsavingsfortheaboveexperimentsareshownintheTable1.
Byutilizingourenergyoptimizationtechnique,weareabletoreducetheFig.
4PowerconsumptionwithpseudocodeTable1EnergyconsumptionandsavingsunderdifferentconditionsWithmaximumfrequencyWithminimumfrequencyWithenergyoptimizationtechniqueWithenergyoptimizationtechniqueandsteppeddownvoltageAvg.
Power(W)443.
33330.
66402.
11378.
96Energy(Ws)53,643.
355,881.
548,65647,370.
6Energysavings0%-4%9.
3%11.
7%Fig.
5PowerconsumptionwithP_write_privunderdifferentconditionCSIT(June2013)1(2):175–181179123energyconsumptionby9.
3%withoutaffectingtheper-formance.
Weachieved11.
7%energysavingswith3%increaseinexecutiontimebycombiningouroptimizationtechniquewithvaryingvoltagelevelsofprocessors.
ThesecondexperimentwasconductedwithP_write_-privbenchmark.
ItisoneoftheI/ObenchmarkofIMBpackage.
Inthiscase,allparticipatingprocessesperformconcurrentI/Otodifferent,privateles.
Wehaveexecutedthisprogramontwoserverswith24no.
ofprocesses,12oneachserver.
Thisexperimentwascarriedoutwithtwodifferentconditions,i.
e.
allprocessorswithmax.
operatingfrequency(2.
4GHz)andwithourenergyoptimizationtechnique.
WheneverallprocessesareperformingtheI/Oopera-tions,theprocessorfrequenciesofallthenodeswillbereducedfrom2.
4GHzto1.
6GHzinourenergyoptimi-zationtechnique.
TheexperimentresultswiththetwoconditionsduringtheexecutionofP_write_privbenchmarkisshowninFig.
5.
Theenergyconsumptionforbothexperimentsiscalculatedbyintegratingpowerreadingsovertheexecutiontime.
Thecorrespondingenergycon-sumptionandsavingsforthisexperimentareshowninTable2.
Byutilizingourenergyoptimizationtechnique,wecouldreducetheenergyconsumptionby11.
7%withoutaffectingtheperformance.
Asthenumberofprocessesandthenumberofnodesincrease,moreenergycanbesaved.
6ConclusionsandfutureworkInHPCenvironment,enormousamountofenergywastageoccursattheapplicationlevel.
Therefore,theneedforanefcientenergyoptimizationalgorithmisincreasingtre-mendously.
MostofthetechniquesarebasedonDVFS,whichisaproveneffectivewaytoreducepowerwastage.
OurresearchisbasedonminimizingtheenergywastageusingDVFSinMPIapplications.
Theproposedenergyoptimizationpolicyiseffectiveandcanautomaticallysetthefrequencyofprocessors,whichinturnleadstoreduc-tioninenergyconsumptionwithoutdegradingtheperfor-mance.
WehavealsodevelopedaMulti-agentbasedautonomicframeworkwhichhelpstoimplementouralgorithmonHPCsystems.
Infuture,wewilldeploythisalgorithmusingMulti-agentframeworkonliveHPCsystemsrunningMPIapplications.
AcknowledgmentsTheauthorswouldliketothankR.
K.
SenthilKumar,H.
V.
Raghu,SumitKumarSaurav,ManishaChauhanandB.
Jayanthfortheirvaluablesupportandsuggestionswhileconductingthisresearch.
References1.
GeR,FengX,PylaH,CameronK,FengKW(2007)PowermeasurementtutorialfortheGreen500ListJune27,20072.
TheMessagePassingInterface(MPI)Standard.
http://www-unix.
mcs.
anl.
gov/mpi/3.
GeR,FengX,CameronKW(2005)Performance-constraineddistributedDVSschedulingforscienticapplicationsonpower-awareclusters.
In:ProceedingsofACM/IEEEconferenceonsupercomputing(SC'05)'',20054.
IntelMPIBenchmarks-UsersGuideandMethodology.
http://www.
software.
intel.
com/en-us/articles/intel-mpi-benchmarks/5.
TewariV,MilenkovieM(2006)Standardsforautonomiccom-puting.
In:IntelTechnologyJournal,20066.
WangL,LaszewskiG,DayalJ,WangF(2010)TowardsenergyawareschedulingforprecedenceconstrainedparalleltasksinaclusterwithDVFS.
In:ProceedingsofIEEEsymposiumoncluster,cloudandgridComputing,CCGrid,20107.
LiD,SupinskiBR,SchulzM,CameronK,NikolopoulosDS(2010)HybridMPI/OpenMPpower-awarecomputing.
In:Pro-ceedingsofIEEEsymposiumonparallelanddistributedpro-cessing,IPDPS20108.
CameronKW,GeR,FengX(2005)High-performance,power-awaredistributedcomputingforscienticapplications.
In:IEEEcomputer,vol.
38,20059.
RoderoI,ChandraS,ParasharM,MuralidharR,SeshadriH,PooleS(2010)Investigatingthepotentialofapplication-centricaggressivepowermanagementforHPCworkloads.
In:Pro-ceedingsofIEEEconferenceonhighperformancecomputing,HiPC201010.
EtinskiM,CorbalanJ,LabartaJ,ValeroM,VeidenbaumA(2009)Power-AwareloadbalancingoflargescaleMPIapplica-tions.
In:ProceedingsofIEEEsymposiumonparallelanddis-tributedprocessing,IPDPS2009,doi:10.
1109/IPDPS.
2009.
516097311.
MaY,GongB,ZouL(2009)Energy-EfcientSchedulingAlgorithmofTaskDependentGraphonDVS-UnableClusterSystem.
In:ProceedingsofIEEE/ACMconferencegridcom-puting,doi:10/1109/GRID.
2009.
535305612.
LiD,NikolopoulosDS,CameronK,SupinskiBR,SchulzM(2010)Power-awareMPITaskAggregationPredictionforHigh-EndComputingSystems.
In:ProceedingsofIEEEsymposiumparallelanddistributedprocessingsymposium,IPDPS,201013.
DongY,ChenJ,YangX,YangCY,PengL(2008)LowPowerOptimizationforMPICollectiveOperations.
In:ProceedingsofIEEEconferenceyoungcomputerscientists,ICYCS2008,doi:10.
1109/ICYCS.
2008.
50014.
Y.
ChenandY.
Zeng(2011)''Automaticenergystatuscontrol-lingwithdynamicvoltagescalinginpowerawarehighperfor-mancecomputingcluster''.
Proc.
IEEEConf.
Parallelanddistributedcomputing,applicationsandtechnologies(PDCAT),Oct.
2011,pp.
412–416,doi:10.
1109/PDCAT.
2011.
2415.
KimNS,AustinTetal(2003)Leakagecurrent:Moore'slawmeetsstaticpower.
IEEEComputer,vol.
36,doi:10.
1109/MC.
2003.
1250885Table2EnergyconsumptionandsavingsunderdifferentconditionsWithmax.
frequencyWithourenergyoptimizationtechniqueAvg.
Power(W)317.
8280.
5Energy(Ws)54,35547,970.
7Energysavings0%11.
7%180CSIT(June2013)1(2):175–18112316.
AhmadHF(2002)Multi-agentsystems:overviewofanewpar-adigmfordistributedsystems.
InProceedingsofIEEEsympo-siumhighassurancesystemsengineering17.
VenkiteshS,BindhumadhavaBS,BhandariAA(2006)Imple-mentationofautomatedgridsoftwaremanagementtool:amobileagentbasedapproach.
InProceedingsofinternationalconferenceoninformationandknowledgeengineering18.
MPICH2-1.
4.
1.
http://www.
mcs.
anl.
gov/research/projects/mpich2/documentation/les/mpich2-1.
4.
1-userguide.
pdfCSIT(June2013)1(2):175–181181123