simulationsb2t

MultiGreen:Cost-MinimizingMulti-sourceDatacenterPowerSupplywithOnlineControlWeiDeng,FangmingLiu,andHaiJinServicesComputingTechnologyandSystemLabSchoolofComputerScienceandTechnologyHuazhongUniversityofScienceandTechnology,Wuhan,Chinafmliu@hust.
edu.
cnChuanWuDepartmentofComputerScienceTheUniversityofHongKongHongKongcwu@cs.
hku.
hkXueLiuSchoolofComputerScienceMcGillUniversityMontreal,Canadaxueliu@cs.
mcgill.
caABSTRACTFacedbysoaringpowercost,largefootprintofcarbonemis-sionandunpredictablepoweroutage,moreandmoremod-ernCloudServiceProviders(CSPs)begintomitigatethesechallengesbyequippingtheirDatacenterPowerSupplySys-tem(DPSS)withmultiplesources:(1)smartgridwithtime-varyingelectricityprices,(2)uninterruptedpowersupply(UPS)ofnitecapacity,and(3)intermittentgreenorre-newableenergy.
Itremainsasignicantchallengehowtooperateamongmultiplepowersupplysourcesinacomple-mentarymanner,todeliverreliableenergytodatacenterusersovertime,whileminimizingaCSP'soperationalcostoverthelongrun.
Thispaperproposesanecient,onlinecontrolalgorithmforDPSS,calledMultiGreen.
MultiGreenisbasedonaninnovativetwo-timescaleLyapunovoptimiza-tiontechnique.
Withoutrequiringaprioriknowledgeofsystemstatistics,MultiGreenallowsCSPstomakeonlinedecisionsonpurchasinggridenergyattwotimescales(inthelong-termmarketandinthereal-timemarket),leveragingrenewableenergy,andopportunisticallycharginganddis-chargingUPS,inordertofullyleveragetheavailablegreenenergyandlowelectricitypricesattimesforminimumop-erationalcost.
Ourdetailedanalysisandtrace-drivensim-ulationsbasedonone-monthreal-worlddatahavedemon-stratedtheoptimality(intermsofthetradeobetweenmin-imizationofDPSSoperationalcostandsatisfactionofdata-centeravailability)andstability(performanceguaranteeincasesofuctuatingenergydemandandsupply)ofMulti-Green.
CategoriesandSubjectDescriptorsD.
4.
7[OPERATINGSYSTEMS]:OrganizationandDe-sign—DistributedsystemsPermissiontomakedigitalorhardcopiesofallorpartofthisworkforpersonalorclassroomuseisgrantedwithoutfeeprovidedthatcopiesarenotmadeordistributedforprotorcommercialadvantageandthatcopiesbearthisnoticeandthefullcitationontherstpage.
Tocopyotherwise,torepublish,topostonserversortoredistributetolists,requirespriorspecicpermissionand/orafee.
e-Energy'13,May21–24,2013,Berkeley,California,USA.
Copyright2013ACM978-1-4503-2052-8/13/05.
.
.
$15.
00.
SmartGridwithMulti-marketsPowerControlUnitDatacenterPowerDemandGridTieUPSBatteryOn-SiteWind/SolarEnergygy-Figure1:Anillustrationofthedatacenterpowersupplysystem(DPSS).
KeywordsCloudComputing;Datacenter;PowerSupply;RenewableEnergy;EnergyEciency;OnlineControl;LyapunovOpti-mization1.
INTRODUCTIONTheproliferationofcloudcomputingserviceshaspro-motedmassive,geographicallydistributeddatacenters.
Cloudserviceproviders(CSPs)aretypicallyfacingthreemajorpower-relatedchallenges:(1)Skyrocketingpowerconsump-tionandelectricitybills,e.
g.
,Googlewithover1,120GWhand$67M,andMicrosoftwithover600GWhand$36M[31].
(2)Seriousenvironmentalimpact,asITcarbonfootprintscanoccupy2%oftheglobalCO2emissionsreportedly[18].
(3)Unexpectedpoweroutages,e.
g.
,AmazonexperiencedanoutageinOctober2012initsUS-East-1region,whichwastriggeredbyaseriesoffailuresinthepowerinfrastruc-ture[17].
Toaddressthesechallenges,modernCSPsbegintoequiptheirdatacenterpowersupplysystems(DPSS)withmul-tiplepowersourcesinacomplementarymanner,asillus-tratedinFig.
1.
First,moderndatacentersobtaintheirpri-marypowerfromasmartgrid.
Smartgridstypicallypro-videdierentpricingschemesatdierenttimescales,suchasalong-term-aheadpricingmarketandthereal-timemar-ket[1,15,19,21,37].
Next,datacentersareequippedwithuninterruptedpowersupplies(UPSs)toguardagainstpos-siblepowerfailures.
ThesupplyofUPSsmaymostlykeepadatacenterrunningfor530minutesuponapowerout-age[36].
Finally,CSPsarealsostartingtogreentheirdat-acenteroperationsbyintegratingon-siterenewableenergy,suchassolarandwindenergies[8,10,22–24,35,41].
Therenewableenergyisconnectedtothegridviaagrid-tiede-vice,whichcombineselectricityproducedfromtherenew-ablesourcesandthegridonthesamecircuitforpowersup-ply[4,10].
Theamountofrenewableenergyproducedcouldvarysignicantlyovertime[18].
UPScanbeusedtostoreenergyduringperiodsofhighlevelsofrenewableenergygen-erationand/orlowelectricitypricesinthegridmarkets.
Whentherenewableenergyisinsucientorpricesfromthegridarehigh,theUPSbatterycanbedischargedtoprovidepower[11–13,20,36,38].
AnimportantproblemfacedbyCSPsishowtominimizethelong-termcostofrunningtheirdatacenters.
Severalkeydecisionsneedtobemadeinanonlinefashionwhenoper-atingsuchaDPSS.
(1)Howmuchpowertobepurchasedfromthegrid'slong-termmarketandthereal-timemarket,respectively(2)Howtoecientlyutilizetheavailablere-newableenergy(3)HowtoopportunisticallyusetheUPSbatterytostoreexcesspowergenerated/purchasedandsup-plypowerwhenneededItischallengingtooptimallyutilizethemultiplesourcestoreliablypoweradatacenter,whileminimizingitsoperationalcost.
Onthedemandside,powerdemandinadatacenteristime-varying,duetovariantre-sourceusageofdierentapplicationsrunninginthedata-center;onenergysupplyside,thegridmayoerlong-termpricesandreal-timeprices,whichchangeovertime;further,theunpredictablenatureofrenewableenergyaddsontothesupplyuncertainties.
Therehavebeenanumberofworksinvestigatingdatacen-terpowersupplyincasesofvaryingpowerdemand,renew-ableenergysupplyandelectricitypricesfromsmartgrids.
Theseworkmayeitherassumeaprioriknowledgeofthepowerdemand[1,19,20,37],orrequireasubstantialamountofstatisticsofthesystemdynamics,inordertopredictthefuturedemandbasedondierentforecasttechniques[10,15,21,41].
Someonlyoptimizesingle-dayorsingle-householdpowersupply[14,21],whileothersdonotconsidertheinter-actionsamongrenewableenergyusage,multi-timescalegridpowerpurchasingandenergystoragefromtheprospectiveofadatacenteroperator[14,19,20,27,29,30,33].
Incontrast,weseektodesignanecientonlinecontrolstrategyforlong-termcostminimizingoperationoftheDPSSunderdynamicpowerdemandanduncertainrenewableenergysupplyinasynergeticmanner,withoutrequiringaprioriknowledgeorstationarydistributionofsystemstatistics.
Specically,basedonastochasticoptimizationmodelthatcharacterizestime-varyingpowerdemandandrenewableen-ergysupply,niteUPScapacityandtwo-timescalegridmar-kets,wederiveanonlineDPSScontrolalgorithm–MultiGreen–byapplyingatwo-stageLyapunovoptimizationtechnique[9,26,39].
MultiGreendecidestheamountofenergytopur-chasefromthegrid'slong-termmarketinintervalsoflongerperiodsoftime,asthebasicenergysupplytoaddressde-manddynamicsandreal-timepriceuctuationsinthefutureinterval;MultiGreenalsodecidestheamountofenergytopurchasefromthereal-timemarket,aswellastheamountofenergytostoreintoordischargefromtheUPSbatteries,inshortertimescales.
Theonlinedecisionsaresettobestuti-lizetheavailablerenewableenergyproducedovertimeandLong-term-aheadpowerpurchaseybef(t)ineachfine-grainedtimeslotτtkT(k+1)Tyrt(τ)D(β(τ))PowerDemandd(t)g(τ)0Powersupplyattimeslotτtomeetdemandτybef(t)NTybef(t)NTybef(t)NTybef(t)NTybef(t)NTFigure2:Anillustrationofthesystemmodel.
Eachcoarse-grainedtimeslotisdividedintoNT=5ne-grainedtimeslots.
theperiodswithlowelectricityprices,inordertominimizetheoperationalcostinthelongrunofthedatacenter.
AsalientfeatureofMultiGreenisthat,evenwithoutre-quiringanyaprioriknowledgeofthesystemdynamics,itcanarbitrarilyapproachtheoptimaloinecostwhichiscomputedwithfullknowledgeofthesystemoveritslongrunwithinaprovableO(1/V)gap.
Thealgorithmicparam-eterVservesasacontrolknob,byadjustingwhichCSPscancontrolthetradeobetweentheminimizationoftheDPSSoperationalcostandsatisfactionoftheconstraintsofdatacenteravailabilityandUPSlifetime.
Weanalyzetheperformanceofouronlinecontrolalgorithmwithrigoroustheoreticalanalysis.
Further,wedemonstratetheoptimality(intermsofthetradeobetweenminimizationofDPSSop-erationalcostandsatisfactionofdatacenteravailabilitycon-straint)andstability(performanceguaranteeincasesofuc-tuatingpowerdemandandsupply)achievedbyMultiGreen,usingextensivesimulationsbasedonone-monthworthoftracesfromlivepowersystems.
2.
SYSTEMMODELANDOBJECTIVEAsillustratedinFig.
2,weconsideraDPSSsystemoper-atinginadiscrete-timemodel.
TimeisdividedintoK(K∈N+)coarse-grainedslotsoflengthTinaccordancewiththeintervallengthofgrid'slong-termmarket,e.
g.
,daysorhours[21].
Eachcoarse-grainedtimeslotisfurtherdividedintoNT(NT∈N+)ne-grainedtimeslots,e.
g.
,NT=5inFig.
2.
Empirically,eachne-grainedtimeslotis15or60minuteslongperwhichthedatacentercanadjustpowercontrolstrategiesinamorepromptfashion[19,39].
2.
1OnlineControlDecisionsWeassumethatdatacenterenergydemandd(t)andre-newableenergygenerationg(t)arerandomvariables.
Wedon'tassumetheyfollowanyspecicprobabilitydistribu-tionfunctions.
Weassumeadatacenterprovidercanbuyelectricityfromthegrid'slong-termcontractsorbuyelec-tricityinspotmarkets.
Whileitisnotcommonintoday'smarketsfordatacenterstodirectlybuyenergyfromenergymarkets,thismaychangeinthefuture.
Infact,GoogleformedasubsidiaryGoogleEnergyLLC,andgetapprovalbyFERC(USFederalEnergyRegulatoryCommission)3yearsago[16].
ThisapprovesthatGoogleLLCcanbuyandevensellelectricity.
Aswecanimagine,inthefuture,moredatacenterproviderswillparticipateinelectricitymarkets,ascloud-scaledatacentersgrowrapidlyandcandrawtenstohundredsofmegawatts.
TheoperationofDPSSincludesfourkeycontroldecisionsintwotimescales:2.
1.
1Long-term-aheadEnergyPurchaseAtthebeginning(therstne-grainedtimeslot)ofeachcoarse-grainedtimeslott=kT(k=1,2,.
.
.
,K),theDPSSobservestheenergydemandd(t)andrenewableenergyg(t)generatedduringtimeslott.
Then,theDPSSmakesadecisiononhowmuchenergyybef(t)tobepurchasedatapriceplt(t)(withanupperboundpricePmax)inthelong-termmarket.
TheDPSSaveragelyschedulesenergyylt(t)=ybef(t)/NTtobeusedineachne-grainedtimeslotinthiscoarse-grainedtimeslot.
Forexample,iftheDPSSdecidestopurchase100KWintheday-aheadgridmarketaccordingtotheobservationofthecurrentdemandandre-newableenergyproduction,itwillschedule20KWforeachne-grainedtimeslotinthenextdaywhenNT=5.
2.
1.
2Real-timeEnergyBalancingSincetheprimarycostsforrenewableenergygenerationareconstructioncostssuchasdeployingsolarpanelsandwindturbines,theiroperationalcostisnegligible[35],andwefocusonoperationalcostminimizationinthispaper.
Thus,therenewableenergyisassumedtobeharvestedfreeafterdeployment,andwepreferentiallyuseit.
Whentherenewableenergyisgeneratedduringtimeslott,weuseittomeetenergydemandasmuchaspossible.
Ifthereisexcessrenewableenergy,weusethebatterytostoreit.
Specically,ateachned-grainedtimeslotτ∈[t,t+T1],theactualenergydemandd(τ)andavailablerenewableen-ergyg(τ)canbereadilyobservedbytheDPSS.
Ifthelong-term-aheadpurchasingandtherenewableenergyareenoughtomeetthecurrentenergydemand,i.
e.
,ylt(t)+g(τ)≥d(τ),thennoreal-timeenergydischarging/purchasingisneeded.
Otherwise,theDPSShastomakeadecisiononwhethertodischargeenergyD(β(τ))fromthebattery.
Ifthedischargedpowerisstillnotenough,theDPSSdecideshowmuchaddi-tionalenergyyrt(τ)tobepurchasedfromthegrid'sreal-timemarketatthereal-timepriceprt(τ)(≤Pmax)tofulllthecurrentdemand.
AnysuperuousenergyisusedtochargethebatteryR(τ).
Thus,wehave:ylt(t)+yrt(τ)+D(β(τ))+g(τ)R(τ)=d(τ),(1)whereD(β(τ))denotestheamountofUPSenergydischargedatthedepth-of-discharge(DoD)levelofβ(τ)(β(τ)∈[0,1]).
DoDisameasureofhowmuchenergyhasbeenwithdrawnfromthebattery,expressedasapercentageofthefulldis-chargingcapacity.
Forexample,letDmaxdenotethemaxi-mumenergythatwecandischargepertime,thenD(β(τ))=β(τ)Dmax.
Thebatteryiseitherchargedordischargedornotinuseateachtimeslot,i.
e.
,R(τ)·D(β(τ))=0.
2.
2OnlineControlConstraintsThereareaseriesofconstraintsthattheabovedecision-makingshouldsatisfy:2.
2.
1BalancingprocurementaccuracyandcostInpractice,thepriceofelectricityinthegrid'sreal-timemarkettendstobehigheronaveragethanthatinthegrid'slong-termmarket,i.
e.
,Eprt(τ)>Eplt(t)[1,21,37],asup-frontpaymentisassociatedwithcheapercontractpricesinthelong-termmarket.
Hence,whenprocuringenergyinthetwo-timescalemarkets,theDPSSshouldmakethebesttradeobetweenprocurementaccuracyandpowercost.
Ad-ditionally,weassumethatthemaximalamountofpowerthatthedatacentercandrawfromthegridateachtimeislimitedbyPgrid:0≤ylt(t)+yrt(τ)≤Pgrid.
(2)2.
2.
2GuaranteeingdatacenteravailabilityLetm(τ)denotetheUPSenergylevelattimeτ.
WeassumethattheecienciesofUPScharginganddischargingarethesame,denotedbyη∈[0,1],e.
g.
,η=0.
8meansthatonly80%ofthechargedordischargedenergyisusefulwhenchargingordischarging.
ThedynamicsofUPSlevelm(τ)canbeexpressedas:m(τ+1)=m(τ)+ηR(τ)D(β(τ))/η.
(3)Weassumethatunderanyfeasiblecontrolalgorithm,theUPSbatteryalwaysreservesaminimumenergylevelMmin,toguaranteereliabledatacenteroperationincaseofpoweroutage.
Forinstance,theenergybuereBu[11]alwaysre-tainsve-minute-worthofreservedenergyinUPStoensuredatacenteravailability.
WeassumethatUPShasacapacityofMUPS,thuswehave:Mmin≤m(τ)≤MUPS.
(4)Typically,Mmincanpowerthepeakdemandofadatacenterforaboutaminute,whileMUPScanfor530minutes[36].
2.
2.
3UPSlifetimeandoperationalcostInpractice,UPSisconstrainedbythemaximumamountsofenergyforrechargeanddischargepertime:0≤D(β(τ))≤Dmax,0≤R(τ)≤Rmax,(5)whereDmaxandRmaxarethemaximumenergythatwecanrechargeanddischargeUPSpertime,respectively.
IthasbeenpracticallyshownthattheUPSlifetimeisadecreasingfunctionofDoDandcharge/dischargecycles[11].
Thecostofoperatingthebatteryisafunctionofhowof-ten/muchitischargedanddischarged.
Weassumethatthecostsofeachrecharginganddischargingoperationarethesame,denotedasCr.
IfanewUPScostsCupstopur-chaseanditcansustainLupstotalcyclesofcharginganddischargingatthemaximumDoD,thenCr=Cups/Lups.
IfthelifetimeoftheUPSisLifeups,thenthemaximumal-lowabledischargingandchargingnumberoverthelongrun[0,t1]wheret∈KTisNmax=Lups·KTLifeups.
Nmaxsatises:0≤t1τ=0a(τ)≤Nmax,(6)wherea(τ)denoteswhethertheUPSisusedintimeslotτornot,thatisa(τ)=1ifD(τ)>0orR(τ)>0,a(τ)=0otherwise.
Hence,attimeslott,theoperationalcostofUPSoperationisa(t)Cr.
2.
3StochasticCostMinimizationFormulationAteachne-grainedtimeslotτ,theDPSSoperationalcostisthesumofcostsforgridenergypurchasingandUPScharging/discharging.
Therefore,Cost(τ)ylt(t)plt(t)+yrt(τ)prt(τ)+a(τ)Cr.
WeseektodesignanonlineDPSScontrolalgorithmthatcansystematicallymakeonlinedeci-sionsbysolvingthefollowingstochasticcostminimizationproblemP1:minybef,yrt,D(β),RCostavliminft→∞1tt1τ=0E[Cost(τ)](7)s.
t.
t:constraints(1)(2)(4)(5)(6).
Sincethebatterycanbechargedtostoreenergyordis-chargedtoservedemand,thecurrentcontroldecisionsarecoupledwiththefuturedecisions.
Forexample,currentde-cisionsmayleaveinsucientbatterycapacityforstoringrenewableenergyproducedinthenearfuture,oroverusethebatteryandthreatendatacenteravailability.
Tosolvethisoptimizationproblem,thecommonlyuseddynamicpro-grammingtechniqueandMarkovdecisionprocesssuerfromacurseofdimensionalityandrequiresignicantknowledgeofthedemandandsupplyoverthelongterm[2,15].
Incon-trast,therecentlydevelopedLyapunovoptimizationframe-work[9,26]isshowntoenablethedesignofonlinecontrolalgorithmsforsuchconstrainedoptimizationoftime-varyingsystemswithoutrequiringaprioriknowledgeofthefutureworkloadandcosts.
Inparticular,ourabovetwo-timescalepowerdeliverymodeltswellthetwo-stageLyapunovop-timizationframework[39],thatenablesustoperformtwolevelsofcontrolstrategiesattwolevelsoftimegranularity.
Therefore,wedesignouronlinecontrolalgorithmbasedonthetwo-timescaleLyapunovoptimizationtechnique.
2.
4AnOptimalOfineAlgorithmNowwepresentapolynomial-timeoptimaloinesolu-tionforproblemP1asabenchmarkforcomparison.
Inthetheoreticallyoptimalscenario,DPSSknowsallfuturesystemstatisticsincludingenergydemandd(t),renewableenergyproductiong(t)andgridenergypricesplt(t),prt(t),t∈[0,1,.
.
.
,KT].
First,wepresentthefollowingstraight-forwardLemma1abouttheoptimalreal-timeenergypur-chasingwithoutproofforbrevity.
Lemma1.
IneveryoptimalsolutionoftheoptimizationproblemP1,itholdsτ,yrt(τ)≡0orprt(τ)≡0.
Theabovelemmaimpliesthatreal-timeenergypurchas-ingisunnecessaryintheoptimalsolution,whereallthefuturestatisticsareknowninadvance.
Thus,solvingtheoptimizationproblemP1isequivalenttosolvingKsingletime-slotproblemsP2asfollows,t∈[0,1,.
.
.
,KT],attherstne-grainedtimeslotofeachcoarse-grainedtimeslotoverthelonghorizonKT.
minybef,D(β),Rylt(t)plt(t)+a(t)Cr(8)s.
t.
ylt(t)+g(t)+D(β(t))R(t)=d(t),0≤ylt(t)≤Pgrid,t:constraints(4)(5)(6).
P2onlyincludeslinearterms,andhencecanbesolvedinpolynomialtimeusingstandardlinearprogrammingtech-niques,e.
g.
,interiorpointmethods[3].
3.
ONLINEALGORITHMDESIGNNowwedevelopanonlinealgorithmtoachievenear-optimalsolutionwithoutaprioriknowledgeofpowerdemandandrenewableenergygeneration.
3.
1ALyapunovOptimizationSolutionToguaranteedatacenteravailabilityanddeliverreliableenergytodatacenterpowerdemand,weshouldguaranteeconstraint(4)ofbatterylevel.
WeuseanauxiliaryvariableX(t)totrackthebatterylevel,denedasfollows:X(t)=m(t)VPmax/TMminDmax/η,(9)whereV≥0isacontrolparametertobespeciedlater,whichaectsthedistancetotheoptimalvalueandisrelatedtothebatterycapacity.
TheintuitionbehindX(t)isthatbycarefullytuningtheweightsVfordecision-making,wecanensurethatwhenevercharging/dischargethebattery,theenergylevelinthebatteryalwaysliesinthefeasibleregion[Mmin,MUPS].
Recallthatm(t)istheactualbatterylevelintimeslottandevolvesaccordingtoEq.
(3).
ThedynamicsofX(t)isgivenas:X(t+1)=X(t)+ηR(τ)D(β(τ))/η.
(10)InTheorem2,wewillprovethatforanytimeslott,thebatterylevelm(t)isalwaysinthesaferange,sinceX(t)isdeterministicallylowerandupperbounded.
Withthequeue,wetransformtheinequalityconstraint(4)intoaqueuestabilityproblem[26].
Asascalarmeasureofthequeuelength,wedeneaquadraticLyapunovfunctionas:L(t)12X2(t).
(11)Thisrepresentsascalarmetricofqueuecongestioninthesystem.
TokeepthesystemstablebypersistentlypushingtheLyapunovfunctiontowardsalowercongestionstate,weintroducetheT-slotconditionalLyapunovdriftas:T(t)E[L(t+T)L(t)|X(t)].
(12)ThenfollowingtheLyapunovdrift-plus-penaltyframework[9],weaddafunctionoftheexpectedoperationalcostoverTslots(i.
e.
,thepenaltyfunction)to(12)toobtainthedrift-plus-penaltyterm.
Ourcontrolalgorithmisdesignedtomakedecisionsonybef(t),yrt(t),D(β(t))andR(t)tomin-imizeanupperboundonthefollowingdrift-plus-penaltytermineverytimeframeoflengthT:T(t)+VE{t+T1τ=tCost(τ)|X(t)},(13)wherethecontrolparameterVischosenbytheCSPtotunethetradeobetweenDPSScostminimizationanddatacen-teravailability(batterylevel).
Forinstance,ifVissettobelargerandmoreemphasisisputtocostminimization,thenUPSwillbeoverlyusedforcertaintimesandthusdat-acenteravailabilityonlyachievesaweaksatisfaction.
Akeyderivationstepistoobtainanupperboundonthisterm.
ThefollowingTheorem1givestheanalyticalboundonthedrift-plus-penaltyterm.
Theorem1.
(Drift-plus-PenaltyBound)LetV>0andt=kT(k∈Z+).
ConsideringthequadraticLyapunovfunc-tionEq.
(11),weassumethatE[L(0)]0andt=kTforsomenonnegativeintegerk.
Replac-ingtheconcatenatedqueueX(τ)withX(t),thedrift-plus-penaltysatises:T(t)+VE{t+T1τ=tCost(τ)|X(t)}(15)≤B2T+E{t+T1τ=tX(t)[R(τ)ηD(β(τ))/η]|X(t)}whereB2=B1+T(T1)[R2maxη2D2max/η2]/2,andB1isgiveninTheorem1.
Proof.
AccordingtoEq.
(10),foranyτ∈[t,t+T1],wecangetthat:X(t)(τt)Dmax/η≤X(τ)≤X(t)+(τt)ηRmax,Therefore,recallingeachterminEq.
(14),wehave:t+T1τ=tX(τ)[R(τ)ηD(β(τ))/η]≤t+T1τ=t[X(t)+(τt)ηRmax]R(τ)ηt+T1τ=t[X(t)(τt)Dmax/η]D(β(τ))/η=t+T1τ=tX(t)[R(τ)ηD(β(τ))/η]+t+T1τ=t(τt)[RmaxR(τ)η2DmaxD(β(τ))/η2]≤t+T1τ=tX(t)[R(τ)ηD(β(τ))/η]+T(T1)[R2maxη2D2max/η2]/2.
Therefore,bydeningB2=B1+T(T1)[R2maxη2D2max/η2]/2,substitutingtheaboveinequalityinto(14),weprovethethe-orem.
3.
3Two-timescaleOnlineControlAlgorithmComparingRHSof(14)withRHSof(15),wecanseethattheRHSof(15)givesalargerupperboundthantheRHSof(14).
WeseektominimizetheRHSof(15),toderivetheonlinedecisions.
Thecontroldecisionybef(t)shouldbemadeatthebeginningofeachcoarse-grainedtimeslotwhileyrt(τ),D(β(τ)),andR(τ)aremadeateachne-grainedtimeslot.
Thus,wecanseparatetheproblemintotwoindepen-dentsub-problemsP3andP4asgiveninourMultiGreenAlgorithm1,tomakedecisionsinthetwotimescales,re-spectively.
Ateachcoarse-grainedtimeslott=kT,Multi-Greendecideshowmuchenergyybef(T)=NTylt(t)tobepurchasedfromthegrid'slong-termenergymarket.
Thedecisionshouldmakesurethatthecurrentenergydemandismetandthebatterystoresenoughenergyforthefutureneed.
Ateachreal-timeslotτ∈[t,t+T1],MultiGreendecidesreal-timemarketprocurementyrt(τ),andUPSbat-terydischargingD(β(τ))andchargingR(τ)tosupplyen-ergywhenneededorstoreadditionalenergy,soastomatchthepowerdemandandsupply.
Attheendofeachtimeslot,MultiGreenupdatesitsqueuestatistics.
Remarks:MultiGreeniscomputationallyecient.
Eachtimeitonlyneedstosolvetwolinearprogramswithfourvariables(ybef(t),yrt(τ),D(β(τ)),R(τ))andfourlinearcon-straintsin(1)(2)(3)(5)(6).
Wecaneasilysolvethetwosub-problemsP4andP5usingclassicallinearprogrammingAlgorithm1:TheOnlineAlgorithmMultiGreen.
1)Long-term-aheadEnergyPlaning:Ateachcoarse-grainedtimeslott=kT(k∈Z+),theDPSSdecidestheoptimalpowerprocurementybef(t)inthegrid'slong-termmarkettominimizethefollowingproblemP3:minE{t+T1τ=tVylt(t)plt(t)+yrt(τ)prt(τ)|X(t)}+E{t+T1τ=tX(t)[R(τ)ηD(β(τ))/η]|X(t)}s.
t.
(1)(2).
2)Real-timeEnergyBalancing:ThentheDPSSaveragelyschedulesenergyylt(t)=ybef(t)/NTtobeusedforeachne-grainedtimeslotτ∈[t,t+T1].
TheDPSSdecidesreal-timeenergyprocurementyrt(τ),andUPSbatterydischargingD(β(τ))andchargingR(τ)tominimizethefollowingoptimizationproblemP4:minVyrt(τ)prt(τ)+X(t)R(τ)ηD(β(τ))ηs.
t.
(1)(2)(5)(6)3)QueueUpdate:UpdatetheactualandvirtualqueuesusingEq.
(3)andEq.
(10).
approaches,e.
g.
,interiorpointmethods[3].
MultiGreenmakesonlinecontroldecisionsybef(t),yrt(τ),D(β(τ))andR(τ)solelybasedonthecurrentavailablestatisticsateachtimeslot,includingqueuestatistics,energydemand,vol-umeoftheavailablerenewableenergy,energypricesandUPSenergylevel.
Thesestatisticstypicallyonlyrequireafewbitstostore,andtakeverylittletimetocalculateandtransmit.
Besides,interiorpointmethodshavealowcom-putationalcomplexity(usuallypolynomialtime)inprac-tice[3].
Thoughadvancedpredictiontechniquescancom-plementMultiGreentomakemoreaccuracydecisions,atradeoexistsbetweenthebenetsofdecisionaccuracyandcomplexityofimplementingtheforecasttechniques.
Notethat,MultiGreenismoresuitablefordelay-sensitiveenergydemandthandelay-tolerantdemand.
Thatis,MultiGreenseekstoaddressenergydemandwhenitisgeneratedimme-diately.
Noenergydemandshouldbedelayedtoafuturetimetoaddress.
Weleaveitasthefutureworktodesignasmartpowersupplysystemformixedworkloads.
4.
PERFORMANCEANALYSISInthissection,weanalyzeourMultiGreenalgorithmintermsofperformanceboundandrobustness.
4.
1PerformanceBoundWerstanalyzethegapbetweentheresultachievedbyMultiGreen,ifaccurateknowledgeofX(τ)inthefuturecoarse-grainedintervalisemployedratherthanourapprox-imation.
Weassumethatthetheoreticaloineoptimalob-jectivefunctionvalueisφoptofthecostminimizationprob-lemP1.
Theorem2.
(PerformanceBound):Thetime-averagedcostCostGreenavachievedbytheMultiGreenalgorithmbasedonaccurateknowledgeofX(τ)inthefuturecoarse-grainedintervalsatisesthefollowingboundwithanygivencontrolparameterV(V>0):(1)Thetime-averagecostCostGreenavachievedbyMulti-Greensatisesthefollowingbound:CostGreenavliminft→∞1tt1τ=0E[Cost(τ)]≤φopt+B2V,(16)whereB2isgiveninCorollary1.
(2)TheUPSbatterylevelm(t)isalwaysintherange[Mmin,MUPS].
Datacenteravailabilityissatised.
(3)Allcontroldecisionsarefeasible.
Proof.
(1)Lett=kT(k∈Z+)andτ∈[t,t+T1].
FromtheoptimaloinepolicyinSec.
2.
4,weknowthatthereisanoptimalsolutionφopt.
SinceMultiGreenminimizestheRHSofEq.
(15),pluggingthepolicyπintotheRHSofEq.
(15),wehave:(t)+VE{t+T1τ=tCostGreenav(τ)|X(t)}≤B2T+Vφopt.
Takingtheexpectationofbothsidesandrearrangingtheterms,weget:E{L(t+T)L(t)}+VTE{CostGreenav(t)}≤B2T+VTφopt.
Summingtheaboveovert=kT,k=0,1,2,.
.
.
K1,usingthefactthatL(t)>0,anddividingbothsidesbyVKT,weobtain:1KTE{KT1τ=0CostGreenav(τ)}≤φopt+B2V.
TakingthelimitasK→∞,wecompletetheproof.
(2)WerstobservethatsubproblemP4hasthefollowingpropertiesrelatedtobatteryoperation:Lemma2.
IfX(t)>0,thenR(t)=0;ifX(t)7.
48,duetomorefrequentbatterycharging/discharging,MultiGreenhasalowercostandalowerlevelofconstraintsatisfactionthanGreen.
Bychoos-inganappropriatevalueofV,e.
g.
,V=10,MultiGreencanachieveasignicantlylowercostcomparedwithGreenwhileguaranteeingacceptablesatisfactionofconstraintsondatacenteravailabilityandUPSlifetime.
5.
2.
2ImpactofCoarse-grainedTimeFrameTInFig.
5,wexVtobe10andvaryTfrom3timeslots(3hours)to144timeslots(6days),whichisasucient-longrangeforexploringtheimpactofdierenttimescalesofthegrid'slong-termmarket.
WeobservethatThasrel-0369121518212427305.
0x1031.
0x1041.
5x1042.
0x1042.
5x1043.
0x1043.
5x1044.
0x1044.
5x104OfflineOptimalAlgorithmGreenAlgorithmTotalCost($)Day(V=10,T=24)GreenAlgorithmTM,NBTM,MUPS=0.
25MWhRTM,MUPS=0.
5MWhTM,MUPS=0.
5MWhTM,MUPS=1MWhOfflineOptimalFigure6:Impactofbatterycapacityandgridmarketstructure.
TwoMarkets—TM,Real-TimeMarket—RTM,NoBattery—NB.
ativelylessimpactonthecostofoperatingtheDPSS.
Theuctuationofthetime-averagedcostismorenotablewhenTbecomeslonger.
TherationaleisthatthetermBεinTheorem3isproportionaltoT,whichmeansthattheun-certaintiesofenergydemandandrenewableenergyincreasewiththeincreaseofT.
Nevertheless,thetime-averagedcostonlyuctuateswithin[9.
7%,+8.
5%].
ThiscorroboratesTheorem3that,evenwithinfrequentdecisionsoftheDPSSoperations,MultiGreencanstillachievesignicantcostre-duction.
5.
2.
3ImpactofBatteryCapacityandGridMarketsInFig.
6,wecomparethetime-averagedtotalcostunderdierentbatterysizes(MUPS∈{0,0.
25,0.
5,1}MWh)overthe31-dayperiodwithV=10andT=24.
Itshowsthatthetime-averagedtotalcostdecreaseswiththeincreaseoftheUPSbatterycapacity.
TherationaleisthatanUPSwithlargercapacitycanstoremoresuperuousrenewableenergygenerated,ormoreenergypurchasedfromthegridwhenthepriceislow,toservethedemand,resultinginloweroverallcosts.
InFig.
6,wealsocomparethecasewithenergypurchaseintwo-timescalemarketswiththecasewhereonlythereal-timemarketexists,bothwithV=10,T=24,MUPS=0.
5MWh.
Wecanobservethattheexistenceofthegrid'slong-termmarketcanbringinadditionalcostreduction.
ThereasonisthatDPSScanpurchasecertainamountofenergybeforehandinthegrid'slong-termmarketwithrela-tivelowerpricesInaddition,wecanobservethatevenwithouttheUPSbattery,theMultiGreenalgorithmwiththetwo-timescalemarketscanreducethecostby10.
06%,comparedtotheGreenalgorithm.
Withtwomarkets,whenweincreasethebatterysizefrom0to1MWh,theaverageoperationalcostreductionrangesfrom10.
06%to34.
21%.
Thebenetbroughtbyenergystorageishigherthanthatofexploitingthetwomarkets.
Whenthebatterysizeislargeenough,MultiGreencanapproachtheoptimaloinealgorithm.
5.
3CharacterizingAlgorithmRobustnessAsmentionedinSec.
3,ourMultiGreenalgorithmapprox-imatesthefuturequeuestatisticsasthecurrentvalues.
Nowweexploretheinuenceofapproximationerrorsontheper-formanceofMultiGreen.
Weaddarandomapproximationerrortothedatacenterenergydemand,solarenergygener-ationandenergyprices,e.
g.
,uniformlydistributed±50%Figure7:DPSSoperationalcostatvariouslevelsofrenewableenergypenetrationandenergydemandvariation.
errors[39].
WeletMultiGreenmakeallthecontroldeci-sionsbasedonthedatasetwithsuchrandomerrorsunderdierentvaluesofV.
InFig.
8,weshowthedierencesinpercentagebetweentheDPSSoperationalcostsachievedwithapproximatedvaluesandtheresultsweobtainedusingtheoriginaltraces.
Weobservethatthedierenceuctuateswithin[1.
3%,2.
1%]forallvaluesofV.
Thus,MultiGreenisrobusttoinaccuratefutureinformation.
Further,westudytheimpactsofrenewableenergypen-etration(thepercentageofrenewableenergyinthetotaldatacenterenergysupply)andthevariationofdatacenterenergydemandonthetotalcost.
InFig.
7,x-axisrepresentsrenewableenergypenetrationintherangeof[0,100%].
Yaxisrepresentsthestandarddeviationofthedemand,i.
e.
,KT1t=0[d(t)E(→d)]2*pd(t),whereE(→d)istheexpec-tationoftheseriesofdemandd(t)overtimelengtht∈[0,KT],andpd(t)isthedistributionprobabilityofd(t).
Weassumethattherandomvariableofthedatacenterenergydemandisuniformlydistributed(pd(t)=1/KT).
Asex-pected,Fig.
7showsthatwiththeincreaseofpenetrationofrenewableenergy,theDPSSoperationalcostdecreasessignicantly.
Therationaleisthatrenewableenergyishar-vestedcost-free(wedonotconsidertheconstructioncost).
Incontrast,asthevariationofdemandincreases,theopera-tionalcostincreasesslightly.
Therationaleisthatintensivevariationincurslargeapproximationerrors.
6.
RELATEDWORKInthissecton,wediscusstheresearchmostpertinenttothisworkasfollows.
Therstcategoryofworksisexploitingrenewableenergyindatacenters.
ManylargeITcompaniesrecentlyconsidergreeningtheirdatacenterswithrenewableenergy[7,8,10,22–24,35,41].
However,theintermittentna-tureofrenewableenergyposessignicantchallengestomakeuseofthem.
Someworksmakethetrac"followtherenew-ables"toexecuteworkloadwhen/whererenewableenergyisavailable[10,22–24,41]orcarbonfootprintislow[7].
How-ever,theseapproachesrequirepredictionofrenewableen-ergyproductionwhenschedulingworkload,orsacriceper-formancetoavoidwastingrenewableenergy.
Otherworkssupplyrenewableenergytodeferrableloadstoaligndemandwithintermittentavailablerenewableenergy[27,29,30].
Buttheyarefromtheprospectiveofrenewableenergyprovidersanddonotconsiderenergystorageandmultiplemarketsinthesmartgrid.
0102030405060708090100-2-1012DifferenceinOperationalCost(%)ControlParameterVFigure8:Theimpactofapproximationerrorsinoperationalcostreduction.
Thesecondcategoryofworksisleveragingenergystor-ageindatacenters.
Recently,UPSshowsitsbenetstoreduceelectricitycostsindatacenters[11–13,36,38].
Dat-acenterscanstoreenergyintheUPSwhenenergypricesarelowanddischargeUPSwhenpricesarehigh,toreducethepowerdrawnfromthegrid[13,36].
Moreover,UPScanshavepeaks[11,12].
Duringperiodsoflowdemand,UPSbatteriesstoreenergy,whilestoredenergycanbeusedtotemporarilyaugmentthegridsupplyduringhoursofpeakload.
However,theseworksfocusonstudyingthebenetsofUPSbatteryforpowercostreduction,andnorenewableenergyandgridmarketsareconsidered.
Onthecontrary,weleverageUPStostudyhowtomanagemultiplepowersuppliesofadatacenterinanintegratedway.
Thirdstreamofworksisonmultipletimescaledispatch,pricingandschedulinginsmartgrid.
Nairetal.
[1]studiedtheoptimalenergyprocurementfromlong-term,interme-diate,andreal-timemarketsunderintermittentrenewableenergysupplies.
Jiangetal.
[19]proposedanoptimalmulti-periodpowerprocurementanddemandresponsealgorithmwithoutenergystorage.
"Risk-limiting-dispatching"ispro-posedin[37]tomanageintegratedrenewableenergy.
How-ever,theabovethreeapproachesassumethatthedemandcanbeknownahead.
Jiangetal.
[21]solvedtheoptimalday-aheadprocurementandreal-timedemandresponseproblemusingdynamicprogramming,whileHeetal.
[15]formulatedthemulti-timescalepowerdispatchandschedulingproblemasaMarkovdecisionproblem.
Boththeseapproachesneedsubstantialsystemstatisticsandarecomputationallyexpen-sive.
Wemitigatethesedisadvantagesbyapplyingtwo-stageLyapunovoptimizationthatmakesonlinecontroldecisionswithoutaprioriknowledgeoranystationarydistributionofenergyprices,demandandsupply.
Recentlytheauthorsin[13,32,33,39,41]distributedrequestsacrossmultipledatacenterstoreduceelectricitycostsbyleveragingbothtimediversityandlocationdiversityofelectricitypricesinthesmartgrid.
Incontrast,westudyhowtoreducetheop-erationalcostinadatacenterpoweredbymultiplepowersourcesratherthanhowtodistributerequestsacrossdata-centers.
Inaddition,interesthasbeengrowinginpowermanage-mentinsmartgridsanddatacentersusingLyapunovopti-mization[5,6,9,26,42].
Onsmartgrids,severalworkshaveproposedoptimalpowermanagementbasedonsingle-stageLyapunovoptimization.
However,theyeitherfocusedonmanagingindividualhouseholddemand[14]ordidnotcon-sidertheinteractionbetweenrenewableenergyandenergystorage[14,19,20,27].
Incontrast,wemanagetheuncer-taindatacenterdemandandmulti-sourceenergysupplyinasystematicfashionusingtwo-stageLyapunovoptimiza-tion.
Although[34,39]haveusedtwo-stageLyapunovtodesignatwo-timescalealgorithmandaT-StepLookaheadalgorithm,bothofthemstudyhowtoschedulejobsordis-tributerequestsinsolelygrid-poweredgeographicaldata-centersratherthanhowtosupplymulti-sourceenergyinadatacenterwithuncertaindemand.
7.
CONCLUSIONInthispaper,westudyanimportantproblemofhowtominimizetheoperationalcostofdatacentersbyusingmuti-pleenergyresouces.
WeproposeMultiGreen,anonlinecon-trolalgorithmapplyingthetwo-stageLyapunovoptimiza-tiontechnique,whichoptimallyschedulesmultipleenergysupplysourcestopoweradatacenter,inacostminimizingfashion.
Withoutrequiringaprioriknowledgeofsystemstatistics,MultiGreencandeliverreliableenergytodata-centerswhileminimizingtheoperationalcostinthedat-acenter'slong-runoperation.
Bothmathematicalanalysesandtrace-drivenevaluationsdemonstratetheoptimalityandrobustnessofMultiGreen.
Especially,itcanapproachtheoineoptimalcostwithinadiminishinggapofO(1/V),whichismainlydecidedbytheUPSbatterycapacity,gridmarketstructureandDPSSoperationfrequencyforenergypurchasingandUPScharging/discharging.
8.
ACKNOWLEDGEMENTSTheCorrespondingAuthorisFangmingLiu.
TheresearchwassupportedbyagrantfromTheNationalNaturalScienceFoundationofChina(NSFC)undergrantNo.
61133006.
9.
REFERENCES[1]S.
AdlakhaandA.
Wierman.
EnergyProcurementStrategiesinthePresenceofIntermittentSources.
Technicalreport,2012.
[2]D.
Bertsekas.
DynamicProgrammingandOptimalControl,volume1.
AthenaScienticBelmont,MA,1995.
[3]G.
Dantzig.
LinearProgrammingandExtensions.
Princetonuniversitypress,1998.
[4]N.
Deng,C.
Stewart,andJ.
Li.
ConcentratingRenewableEnergyinGrid-tiedDatacenters.
InProc.
ofIEEEInternationalSymposiumonSustainableSystemsandTechnology(ISSST),May2011.
[5]W.
Deng,F.
Liu,H.
Jin,andX.
Liao.
OnlineControlofDatacenterPowerSupplyUnderUncertainDemandandRenewableEnergy.
InProc.
ofIEEEICC,June2013.
[6]W.
Deng,F.
Liu,H.
Jin,andC.
Wu.
SmartDPSS:Cost-MinimizingMulti-sourcePowerSupplyforDatacenterswithArbitraryDemand.
InProc.
ofICDCS,July2013.
[7]P.
Gao,A.
Curtis,B.
Wong,andS.
Keshav.
It'sNotEasyBeingGreen.
InProc.
ofACMSIGCOMM,Aug.
2012.
[8]Y.
Gao,Z.
Zeng,X.
Liu,andP.
R.
Kumar.
TheAnswerIsBlowingintheWind:AnalysisofPoweringInternetDataCenterswithWindEnergy.
InProc.
ofINFOCOM(MiniConference),Apr.
2013.
[9]L.
Georgiadis,M.
Neely,M.
Neely,andL.
Tassiulas.
ResourceAllocationandCross-layerControlinWirelessNetworks.
NowPub,2006.
[10]I.
Goiri,K.
Le,T.
Nguyen,J.
Guitart,J.
Torres,andR.
Bianchini.
GreenHadoop:LeveragingGreenEnergyinData-ProcessingFrameworks.
InProc.
ofEuroSys,Apr.
2012.
[11]S.
Govindan,A.
Sivasubramaniam,andB.
Urgaonkar.
BenetsandLimitationsofTappingintoStoredEnergyforDatacenters.
InProc.
ofACMISCA,June2011.
[12]S.
GovindanandB.
Urgaonkar.
LeveragingStoredEnergyforHandlingPowerEmergenciesinAggressivelyProvisionedDatacenters.
InProc.
ofACMASPLOS,Mar.
2012.
[13]Y.
Guo,Z.
Ding,Y.
Fang,andD.
Wu.
CuttingDownElectricityCostinInternetDataCentersbyUsingEnergyStorage.
InProc.
ofGLOBECOM,Dec.
2011.
[14]Y.
Guo,M.
Pan,andY.
Fang.
OptimalPowerManagementofResidentialCustomersintheSmartGrid.
IEEETransactionsonParallelandDistributedSystems,23(9):1593–1606,2012.
[15]M.
He,S.
Murugesan,andJ.
Zhang.
MultipleTimescaleDispatchandSchedulingforStochasticReliabilityinSmartGridswithWindGenerationIntegration.
InProc.
ofINFOCOM,Apr.
2011.
[16]http://en.
wikipedia.
org/wiki/GoogleEnergy.
[17]http://www.
datacenterknowledge.
com/archives/2012/10/22/amazon-cloud-outage-aecting-manysites/.
[18]http://www.
greenpeace.
org.
HowCleanisYourCloud.
2012.
[19]L.
Huang,J.
Walrand,andK.
Ramchandran.
OptimalPowerProcurementandDemandResponsewithQuality-of-usageGuarantees.
ArxivpreprintarXiv:1112.
0623,2011.
[20]L.
Huang,J.
Walrand,andK.
Ramchandran.
OptimalDemandResponsewithEnergyStorageManagement.
ArxivpreprintarXiv:1205.
4297,2012.
[21]L.
JiangandS.
Low.
Multi-periodOptimalProcurementandDemandResponsesinthePresenceofUncrtainSupply.
InProc.
ofIEEEConferenceonDecisionandControl(CDC),2011.
[22]C.
Li,A.
Qouneh,,andT.
Li.
iSwitch:CoordinatingandOptimizingRenewableEnergyPoweredServerClusters.
InProc.
ofISCA.
ACM,June2012.
[23]Z.
Liu,Y.
Chen,C.
Bash,A.
Wierman,D.
Gmach,Z.
Wang,M.
Marwah,andC.
Hyser.
RenewableandCoolingAwareWorkloadManagementforSustainableDataCenters.
InProc.
ofSIGMETRICS,June2012.
[24]Z.
Liu,M.
Lin,A.
Wierman,S.
Low,andL.
Andrew.
GeographicalLoadBalancingwithRenewables.
InProc.
ofACMGreenMetrics,June2011.
[25]MIDC.
http://www.
nrel.
gov/midc/.
[26]M.
Neely.
StochasticNetworkOptimizationwithApplicationtoCommunicationandQueueingSystems.
SynthesisLecturesonCommunicationNetworks,3(1):1–211,2010.
[27]M.
Neely,A.
Tehrani,andA.
Dimakis.
EcientAlgorithmsforRenewableEnergyAllocationtoDelayTolerantConsumers.
InProc.
ofSmartGridComm,Oct.
2010.
[28]NYISO.
http://www.
nyiso.
com.
[29]A.
Papavasiliou.
CouplingRenewableEnergySupplywithDeferrableDemand.
PhDthesis,UniversityofCalifornia,Berkeley,2012.
[30]A.
PapavasiliouandS.
Oren.
SupplyingRenewableEnergytoDeferrableLoads:AlgorithmsandEconomicAnalysis.
InProc.
ofEnergySocietyGeneralMeeting,July2010.
[31]A.
Qureshi.
Power-DemandRoutinginMassiveGeo-DistributedSystems.
PhDthesis,MassachusettsInstituteofTechnology,2010.
[32]L.
Rao,X.
Liu,L.
Xie,andW.
Liu.
MinimizingElectricityCost:OptimizationofDistributedInternetDataCentersinaMulti-electricity-marketEnvironment.
InProc.
ofINFOCOM,Mar.
2010.
[33]L.
Rao,X.
Liu,L.
Xie,andZ.
Pang.
HedgingAgainstUncertainty:ATaleofInternetDataCenterOperationsUnderSmartGridEnvironment.
IEEETransactionsonSmartGrid,2(3):555–563,2011.
[34]S.
Ren,Y.
He,andF.
Xu.
Provably-EcientJobSchedulingforEnergyandFairnessinGeographicallyDistributedDataCenters.
InProc.
ofICDCS,June2012.
[35]C.
StewartandK.
Shen.
SomeJoulesAreMorePreciousThanOthers:ManagingRenewableEnergyintheDatacenter.
InProc.
oftheWorkshoponPowerAwareComputingandSystems,2009.
[36]R.
Urgaonkar,B.
Urgaonkar,M.
Neely,andA.
Sivasubramanian.
OptimalPowerCostManagementUsingStoredEnergyinDataCenters.
InProc.
ofACMSIGMETRICS,June2011.
[37]P.
Varaiya,F.
Wu,andJ.
Bialek.
SmartOperationofSmartGrid:Risk-limitingDispatch.
inProc.
oftheIEEE,99:(1):40–57,2011.
[38]D.
Wang,C.
Ren,A.
Sivasubramaniam,B.
Urgaonkar,andH.
Fathy.
EnergyStorageinDatacenters:What,Where,andHowmuchInProc.
ofSIGMETRICS,June2012.
[39]Y.
Yao,L.
Huang,A.
Sharma,L.
Golubchik,M.
Neely,etal.
DataCentersPowerReduction:ATwoTimeScaleApproachforDelayTolerantWorkloads.
InProc.
ofIEEEINFOCOM,Mar.
2012.
[40]R.
Zavadil,J.
King,N.
Samaan,J.
Lamoree,M.
Ahlstrom,B.
Lee,D.
Moon,C.
Finley,D.
Savage,R.
Koehnen,etal.
FinalReport–2006MinnesotaWindIntegrationStudy.
TheMinnesotaPublicUtilitiesCommission,2006.
[41]Y.
Zhang,Y.
Wang,andX.
Wang.
GreenWare:GreeningCloud-ScaleDataCenterstoMaximizetheUseofRenewableEnergy.
InProc.
ofACMMiddleware,Dec.
2011.
[42]Z.
Zhou,F.
Liu,H.
Jin,B.
Li,B.
Li,andH.
Jiang.
OnArbitratingthePower-PerformanceTradeoinSaaSClouds.
InProc.
ofIEEEINFOCOM,Apr.
2013.